Streamlining EM into Auto-Encoder Networks

Abstract: We present a new deep neural network architecture, named EDGaM, for deep clustering. This architecture can seamlessly learn deep auto-encoders and capture common group features of complex inputs in the encoded latent space. The key idea is to introduce a differentiable Gaussian mixture neural network between an encoder and a decoder. In particular, EDGaM streamlines the iterative Expectation-Maximum (EM) algorithm of the Gaussian mixture models into network design and replaces the alternative update with a forward-backward optimization. Being differentiable, both network weights and clustering centroids in EDGaM can be learned simultaneously in an end-to-end manner through standard stochastic gradient descent. To avoid preserving too many sample-specific details, we use both the clustering centroid and the original latent embedding for decoding. Meanwhile, we distill the soft clustering assignment for each sample via entropy minimization such that a clear cluster structure is exhibited. Our experiments show that our method outperforms state-of-the-art unsupervised clustering techniques in terms of both efficiency and clustering performance. 

Reviews and author responses:

This paper proposes a deep clustering method called EDGaM. Clustering algorithms often suffer from cluster collapse or sample-specific details overestimation. To balance both challenges, the authors propose a differentiable GMM network in the latent space between encoder and decoder. The network is designed with the scheme of skip connection and several loss terms are applied.

Pros:
+ The idea of streamlining EM into a differentiable network to learn its parameters by backpropagation is interesting. 
+ Writing is clear.

Cons:
- Does the assumption of setting covariance $\Sigma_k$ as $I$ and weight $\pi_k$ as uniform for simplicity too strong? Is there a way to update these two parameters?
- There seems to lack an ablation study to individually illustrate the function of each loss term in Eq. (9).
- The authors mentioned three types of deep clustering approaches in related work. But it seems the authors only compare with the second type which they follow in their experiments. I wonder if the results can be better or comparable with the other two types?

Overall I think it's a good paper but maybe more experiments need to be done to make it comprehensive.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------

After reading the rebuttal, I think the authors have well addressed my concern, thus this paper is good to be accepted for ICLR and I raise my rating from 6 to 7.  


The most common approach for deep clustering is to train a deep auto encoder (DAE) that is used to find latent features which are utilised for clustering with a classic clustering approach such as the K-means. The K-means is not differentiable and therefore the clustering was not part of the network. This caused alternative update of the parameters. 

In this work the same pipeline is used yet, the clustering is carried out by EM-GMM algorithm. It is a nice idea since it is known that the K-means is a special case of the EM-GMM algorithm which is softer and not hard-decisioned like the K-means. The authors present an iterative EM algorithm which is part of the training procedure. 

High clustering results are presented on 4 different datasets.  The proposed skip connection was also found beneficial.

It seems like the authors are not familiar with the recursive EM algorithm proposed by Titterington [1]. In addition, in [2] it is presented that the recursive EM algorithm  is almost identical to the training (Feed-forward and back-propagation) of the mixture-of-deep-experts’ architecture.  It is interesting to see the link of the proposed algorithm to these methods.

Experiments:
It is common to use also the adjusted rand index (ARI) as an objective for the clustering. Interesting to see the performance of the proposed data also on this measurement. 
All datasets in the experiment are images. What is the performance on different kind of datasets such as texts ?

There is a repetition (almost identically paragraphs) between the introduction and the related work section. 

[1] Titterington, D.M.: Recursive parameter estimation using incomplete data. J. Roy. Statist. Soc. Ser. B 46, 257–267 (1984)

[2] Chazan, S.E., Gannot, S. and Goldberger, J., 2018, July. Training strategies for deep latent models and applications to speech presence probability estimation. In International Conference on Latent Variable Analysis and Signal Separation (pp. 319-328). Springer, Cham.


Thanks for your valuable comments. 

Concern 1: The unrolling methods are conceptually less elegant than a top-down generative model with learned amortized inference, which may also enable us to determine the number of clusters in a principled way.
   - A1: We partially agree with the reviewer. First of all, we prefer to call the unrolling method an alternative inference approach for the ELBO of a top-down generative model, rather than a top-down generative model directly. This is why we motivate our EDGaM starting from mixture of Gaussians instead of soft k-means, which is closer to our final model. From this perspective, the unrolling method is conceptually equivalent to the ELBO, both of which are derived for the top-down generative model. Consequently, we believe that, based on our framework, the corresponding unrolling solution for non-parametric models would also be readily available.

Concern 2: My main concern is that this paper does not break new ground conceptually. 
   - A2: Yes, we agree our EDGaM does not break new ground conceptually. However, our EDGaM is a non-trivial combination of unrolling and deep auto-encoder, which still brings some innovative optimization solution and new insights in the field of deep clustering: 
      - (1) No iteration:  although originated from EM, EDGaM avoids the alternative update as most clustering methods. Consequently, our EDGaM achieves superior efficiency as all parameters in EDGaM are updated simultaneously. 
      - (2) No need clustering over the whole batch: due to the conflicts between the clustering objective and the reconstruction task, other deep clustering models usually require clustering over the full batch to ensure reliability, which is not memory efficient for large-scale applications. Through a skip connection, our EDGaM unifies the goal of clustering and self-reconstruction such that all parameters in EDGaM can be updated end-to-end by SGD.

Q5: It is not very clear to me why using more EM layers ($T\ge7$) hurts the performance, and why $T=3$ seems to be the best. Probably more analysis and explanation are needed here, in addition to the discussions in Section 5.1.
   - A5: The number of EM iterations affects differently from the number of unfolded layers due to different optimization paradigms. Since EM is optimized per iteration, more iterations would contribute to better performance. Whereas multiple layers in EDGaM are optimized simultaneously via gradient descent, more layers would summarize the embedding excessively and lead to gradient vanishing. However, this should not be a big issue since we empirically verified that 3-layers can consistently achieve good performance on all four datasets. 

Q6: In addition to the entropy regularization on individual $\lambda_n$ in equation (9), entropy regularization on $\lambda$ averaged over the mini-batch is also used in other work (e.g., IMSAT). The regularization on the averaged $\lambda$ could drive the solution away from a degenerated solution, where all samples are clustered into one or a few clusters. I wonder whether the same technique can also benefit this work.
   - A6: No, AE-based method deep clustering methods do not encounter the degeneration issue, so this technique will not benefit our work but may harm the performance. We analyze the scenes of these two techniques which can be used. To the best of our knowledge, there are three types of entropy regularization: 
      - (A) individual entropy minimization: to squeeze out uncertainty in pursuit of a (nearly) one-hot probability vector, widely used in the semi-supervised setting, like VAT[5]. 
      - (B) individual entropy maximization: to avoid overfitting(supervised)/degeneration(unsupervised) and promote uncertainty, can also be understood as label smoothing, like [6].
      - (C) mini-batch average entropy maximization: similar to (B), but more moderate, like [3].

     According to our analysis, the target scope of (C) conflicts with our motivation, which would not benefit our setting. 

Q7: There are quite a few deep clustering algorithms, e.g.,IMSAT[7], IIC[3], Kingdra[4], proposed in recent years. It would be nice if the authors can compare with them as well.
   - A7: As we have clarified in the related work and experiment, the third type of deep clustering method, which is based on self-supervised learning, such as IMSAT[7] and IIC[3], can achieve state-of-the-art clustering results on various (image) datasets. However, the performance of these methods highly depends on the choice of invariant transformations (extra expert knowledge), which requires domain knowledge and usually varies from dataset to dataset. This is the reason why we do not compare these methods in our experiment. To justify our claim, we also collected the result of our EDGaM and other self-supervised based deep clustering on a text dataset, named Reuters10K, where the invariant transformations are either unavailable or invalid.

     In the following, we compared some deep clustering methods published in recent years with AE-based deep clustering in the following. For the best comparison, we present the best reported
results from their original papers or from [10].
***
Clustering Acc (Higher is better) $\quad$ MNIST $\quad$ Reuters10K
***
Self-Supervsied   $\quad$   IMSAT[7]          $\quad$  0.984    $\quad$      0.710
***
Self-Supervsied   $\quad$ IIC[3]      $\quad$               0.992     $\quad$            --
***
Graph-based      $\quad$          Kindar[4]           $\quad$      0.985      $\quad$      0.705 
***
AE-based        $\quad$          AE+k-means[8]       $\quad$    0.818       $\quad$     0.705
***
AE-based        $\quad$          IDEC[9]      $\quad$              0.881      $\quad$     0.756   
***
AE-based        $\quad$          VaDE[10]       $\quad$           0.945     $\quad$        0.794
***
AE-based        $\quad$          EDGaM           $\quad$          0.967     $\quad$        0.807
***

    From the above table, we find that: (a) self-supervised based deep clustering approaches highly depend on the choice of invariant transformations. If no effective invariant transformations are valuable, like the text data, their clustering performance would degenerate as Valina AE + k-means. (b) Kindar[4], based on graph and ensemble, is highly dependent of the original latent embedding output by vanilla AE. If a good latent embedding is not available, its clustering performance would also degenerate as Valina AE + k-means. (c) Joint learning of deep embedding and clustering can mutually promote each other, especially when the vanilla AE embedding is not good enough. (d) Graph-based Kindar need to respectively build multiple graphs with the whole dataset, which is not suitable for the large-scale application that we target. 

    All the above discussion will merge into the updated main content and then detail in the appendix. 


Thank you very much for your comments. Below is a report for how we address your concern. 

Q1:  is it possible to directly optimize the GMM log-likelihood function in equation (1), together with the reconstruction loss and some entropy regularization? This way the EM algorithm can be avoided in the first place. 
   - R1: No, the direct minimization of equation (1) within the AE network is not numerically stable, since the latent code $z$ and cluster centroids $\mu$ will change simultaneously. 
One possible solution is to split equation (1) into two sub-equations by introducing the auxiliary variable $\lambda$ as [1][2]. Then, it would expand into two objectives, optimizing w.r.t. to the latent code $z$ and cluster centroids $\mu$, respectively, which is exactly the one-layer unrolling methods for the EM algorithm of GMM. Our EDGaM further unrolling it into multiple layers, which is shown to improve clustering performance.

Q2: I assume that $N$ in equation (7) stands for the mini-batch size, and in the denominators the summation over $\lambda$ to get $N_k$ is over the mini-batch instead of the full dataset.
   - R2: Yes, $N_k$ and $N$ are calculated on the fly with regards to the mini-batch data during the experiment. When encountering an cluster imbalanced dataset, we suggest adding a smoothing variable to avoid sampling bias, i.e., $N_k = \sum_n \lambda_{nk}+ a$, $N = \sum_k N_k + Ka$, where $a$ is set to $3$ in the experiment.  
Thanks for pointing this out. We already explicitly stated this point in Sect.3. Moreover, a more detailed explanation of the reweighing strategy  had been discussed in Appendix.B. 

Q3: $\pi$ is uniform and the covariance matrix is the identity matrix for all clusters. Hence this is more of a deep K-means model than a full GMM model. More details for updating the covariance matrix and cluster weight are needed if the authors would want to extend it to a deep GMM model.
   - R3: Yes, the simplified EDGaM is more likely a k-means, compared to a mixture of Gaussians (GMM). But we insist motivate our method from GMM for the following reasons: 
      - (A) the introduction of the auxiliary variable $\lambda$ is more natural for GMM; 
      - (B) the soft assignment, instead of hard assignment, is adopted for the sake of feasible gradient descent. Meanwhile, it also justifies the introduction of the minimum entropy term, which helps us to distillate the reliable cluster assignment gradually.
      - (C) The reconstruction of latent representation is naturally derived following the generative model, i.e., GMM, which motivates us to streamline the iterative EM algorithm of GMM between the encoder and the decoder.
We agree that extra technique is needed for updating the covariance matrix and cluster weight. However, we think the current setting is sufficient to capture the clustering structure in the well-separated latent space, even assuming an identity covariance matrix and a uniform cluster weight. Please refer to A1 to Reviewer\#2 for a detailed discussion. 

Q4: It is a bit unclear to me the approximate equality part (i) in equation (7), especially the re-weighting with $1/N_k$. 
   - A4: Sorry to confuse the reviewer. The content on the advantage of cluster reweigh strategy using $N_k$ was discussed in Appendix.B. We will add the detailed derivation of equation (7) in the Appendix for better understanding.
$$
\mathcal{L}\_{cluster}(\mu) = -\sum_{n=1}^N\sum_{k=1}^K \lambda_{nk}\ln [\frac{1}{K} \mathcal{N}(z_n |\mu_k, \mathbb
{I})] =\sum_{n=1}^N\sum_{k=1}^K \lambda_{nk}\ln K -\sum_{n=1}^N\sum_{k=1}^K \lambda_{nk}\ln \frac{1}{\sqrt{(2 \pi)^d|\mathbb{I}|}} \mathrm{e}^{-\frac{1}{2}(z_n-\mu_k)' \mathbb{I}^{-1}(z_n-\mu_k)}
$$
$$
 =(\ln K + \frac{d}{2}\ln(2\pi))\sum_{n=1}^N\sum_{k=1}^K \lambda_{nk} + \frac{1}{2}\sum_{n=1}^N\sum_{k=1}^K \lambda_{nk}\|z_n-\mu_k\|\_2\^2 
$$
$$
\approx \boxed{N\ln K + \frac{Nd}{2}\ln(2\pi)} + \frac{1}{2}\sum_{n=1}^N \boxed{\frac{N}{KN_k}}\sum_{k=1}^K \lambda_{nk}\|z_n-\mu_k\|\_2\^2
$$
where $N = \sum_{n=1}^{N}\sum_{k=1}^K \lambda_{nk}$. $N\ln K + \frac{Nd}{2}\ln(2\pi)$ is constant. Note that we suggest to reweigh the within-group variance with $w_k = \frac{N}{KN_k} = \frac{\bar{N}}{N_k}$, where $\bar{N}$ is the average number of samples for each cluster and $N_k$ is the number of samples belonging to cluster $k$. Regarding a cluster balance dataset, i.e., $w_k \approx 1$, the reweigh strategy becomes invalid and equation (7) reduces to regular cluster loss. Otherwise, equation (7) will adjust the penalty of within-group variance to lend an ear to small clusters when clusters are imbalanced.

We are grateful for your efforts and time. You have made this work better.

Q1: Concerns about the simple setting $\Sigma_k = I$ and $\pi_k = 1/K$. Is there a way to update these two parameters?
   - A1: The current setting is sufficient to capture the clustering structure in the latent space for the following reasons: 
      * (1) the covariance matrix $\Sigma$ is useful for improving clustering when clusters are overlapped. However, entropy regularization helps to promote a well-separated latent embedding (Fig.5a). Under such a scenario, the centroids alone are sufficient to capture the clustering structure. 
      * (2), The cluster weight $\pi$, denoting the size of each cluster, was designed to give higher preference to the large cluster when a sample is close to multiple cluster centroids simultaneously. Similarly, it is only useful for disambiguation when clusters are overlapped but is redundant for our well-separated latent embedding. 
      * (3) Moreover, weight $\pi_k$ would lead to cluster collapse, when clusters are imbalance, since it naturally suppresses small clusters. We tackled this issue via reweighing the within-group variance using $\frac{N}{KN_k}$ (Eq.7) with the estimated clustering assignment $\lambda$. Note that $\frac{N}{KN_k}$ is inverse to the size of the cluster, which would promote the small cluster.

   Meanwhile, it is difficult to incorporate the estimation of these two parameters into the current optimization which adopts stochastic gradient descent. In particular, it requires extra techniques to ensure the covariance matrix being positive-definite, and also ensure the weight being positive and normalized. However, referring to the online EM literature[1], it would be feasible if we resort to moment updates on the sufficient statistics instead of stochastic gradient descent to update the cluster parameters, i.e., centroids, covariance matrix, and the weight.  

Q2: There seems to lack an ablation study to individually illustrate the function of each loss term in Eq. (9).
   - A2: For the sake of presentation, we only showcase the result on MNIST dataset (1) Full loss. ACC: 0.967, NMI: 0.922; (2)AE loss. ACC: 0.346, NMI: 0.23; (3) AE+cluster. ACC: 0.805, NMI: 0.781; (4)AE+MinEnt ACC: 0.871, NMI: 0.757; (5) AE+clustering+correction ACC: 0.911, NMI: 0.845. 

   From the above results, we find that (1) correction step is necessary; (2) Either cluster or minimum entropy only cannot get desired performance; (3) without minimum entropy loss, EDGaM can still get high clustering performance. 

Q3: The authors only compare with the second type which they follow in their experiments. I wonder if the results can be better or comparable with the other two types? 
   - A3:  In Table 3, we already compared with the first type methods, e.g., DEC and Deepcluster, which demonstrate that EDGaM is significantly better than the first type method. In terms of the third type method, e.g., IMSAT and IIC, they require a collection of predefined semantics-invariant transformations to construct semantics pairs for each sample. Their cluster performance highly depends on the choice of invariant transformations, which usually varies from dataset to dataset. To justify our claim, we collected the result of EDGaM and other self-supervised based deep clustering on the text dataset Reuters10K, where invariant transformations are either unavailable or invalid. For the best comparison, we present the best-reported results from their original papers.
***
Clustering Acc (Higher is better) $\quad$ MNIST $\quad$ Reuters10K
***
Self-Supervsied   $\quad$   IMSAT[3]   $\quad$  0.984    $\quad$      0.710
***
Self-Supervsied   $\quad$ IIC[2]      $\quad$    0.992     $\quad$            --
***
AE-based        $\quad$  AE+k-means[4]   $\quad$    0.818       $\quad$     0.705
***
AE-based        $\quad$          EDGaM           $\quad$          0.967     $\quad$        0.807

From the above table, we find that: (1) self-supervised based deep clustering approaches highly depend on the choice of invariant transformations. If no effective invariant transformations are valuable, like the text data, their clustering performance would degenerate as Valina AE + k-means. (2) Joint learning of deep embedding and clustering can mutually promote each other, especially when the vanilla AE embedding is not good enough. 

* [1] Olivier Capp ́e and Eric Moulines. Online expectation-maximization algorithm for latent data models.Journal of the Royal Statistical Society: Series B,2009.
* [2] Xu Ji,  Joao F Henriques,  and Andrea Vedaldi.   Invariant information clustering for unsupervised image classification and segmentation. ICCV, 2019.
* [3] Hu, Weihua, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. "Learning Discrete Representations via Information Maximizing Self-Augmented Training." ICML, 2017
* [4] Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering, 2013.

Yes, we agree with the reviewer. MoE-based [2] can also be extended for deep clustering, e.g., [9] and [10], by modeling each expert as an auto-encoder network. It can be classified as the second type of deep clustering. In particular, the soft assignment is formulated as the output of a mixture assignment network, whose input can be the concatenated latent features as [9] or the raw sample as [10].

According to the [9] and [10], the MoE-based deep clustering is inferior to our EDGaM in terms of the following aspects: 
   * EDGaM achieves much higher ACC (0.967) on MNIST, compared to [9] (0.856) and [10] (0.89).
   * EDGaM has fewer parameters (1 AE), compared to [9] and [10] (K AE + 1 assignment network).
   * [9] and [10] may suffer from cluster collapse, where only partial AE network is well trained. 
  * MoE based deep clustering methods are inevitably affected by sample-specific details that are unrelated to semantics. 

We will incorporate the suggested references in the updated manuscripts. 

Thanks,
 The literature review is better now. 

One comment regarding the following:

"(3) MoDE proposed in [2] is designed for the supervised mixture model, which would encounter cluster collapse as the first type of deep clustering method (See our related work) for the unsupervised clustering task."

It should be mentioned that the model in [2] can be designed also  for clustering - see [9],[10].  


[9]  Zhang, D., Sun, Y., Eriksson, B., & Balzano, L. (2017). Deep unsupervised clustering using mixture of autoencoders. arXiv preprint arXiv:1712.07788.
[10] Chazan, S. E., Gannot, S., & Goldberger, J. (2019, October). Deep Clustering based on a Mixture of Autoencoders. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) (pp. 1-6). IEEE.


Thanks for your valuable comments. You have made this work better.

Q1: Additional background information on the iterative EM algorithm is required.
   - A1: Thanks for bringing this to our attention. We summarize the connection and difference between the two suggested recursive EM algorithms [1][2] and our EDGaM in the following. 

   Connection: the same motivation for enabling the EM algorithm differentiable as well as avoiding full-batch updates in EM. 
      * (1) the simpliﬁed Titterington’s EM scheme [1] is the same as the one-layer unrolling method, namely updating the model parameters via differentiating the objective directly with the introduced auxiliary variable (i.e., $\lambda$ in our paper and w in [2]).
      * (2) Our motivation for the moment update in equation 4 is also inspired by the online EM literature [3].  
      * (3) All three methods ([1],[2], and EDGaM) enable the stochastic update. 

   Difference: 
      * (1) Both [1] and [2] cannot be used for the deep clustering problem we considered in this paper. 
      * (2) [1] is designed for a generalized EM algorithm, which can be applied to conduct clustering in the raw features space. How to combine it with deep learning is still unknown? That is exactly our motivation to propose EDGaM. 
      * (3) MoDE proposed in [2] is designed for the supervised mixture model, which would encounter cluster collapse as the first type of deep clustering method (See our related work) for the unsupervised clustering task. 

   We have already merged the above discussion into our main content. See Sect.3.2 for more details. 

Q2:  It is common to use also the adjusted rand index (ARI) as an objective for the clustering. Interesting to see the performance of the proposed data also on this measurement.
   - A2:  The ARI performance for our EDGaM on MNIST is 0.929. We did not report ARI since the majority of the recent baselines do not report the ARI score on the considered dataset in their original papers. We cannot find the ARI score in other recent papers that refer to them either. 

Q3: What is the performance on different kinds of datasets such as texts?
   - A3: In the following, we collected the result of our EDGaM on a text dataset Reuters10K. The detailed construction of Reuters10K can be found in IDEC[4]. Tips: we find the general reconstruction loss, such as MSE, L1, SmoothL1, and CrossEntroy does not work very well or is not stable. We chose one minus cosine similarity, which works pretty well and stable.

   In the following, we collected some deep clustering methods published in recent years with AE-based deep clustering in the following. For the best comparison, we present the best-reported results from their original papers or from [4].
***
Clustering Accuracy on Reuters10K
***
IMSAT[5]: 0.710; $\quad$ Kindar[6]: 0.705; $\quad$ AE+k-means[7]: 0.705; $\quad$ IDEC[4]:  0.756; $\quad$ VaDE[8]: 0.794;  $\quad$ EDGaM: 0.807

   From the above results, we find that: (1) self-supervised based deep clustering approaches highly depend on the choice of invariant transformations. If no effective invariant transformations are valuable, like the text data, their clustering performance would degenerate as Valina AE + k-means. (2) Joint learning of deep embedding and clustering can mutually promote each other, especially when the vanilla AE embedding is not good enough. 

Q4: There is a repetition (almost identically paragraphs) between the introduction and the related work section.
   - A4: Thanks for the suggestion. We have removed the redundant sentences from related work.

Reference:
* [1] Titterington DM. Recursive parameter estimation using incomplete data. 1984.
* [2] Chazan, Shlomo E., Sharon Gannot, and Jacob Goldberger. Training strategies for deep latent models and applications to speech presence probability estimation. 2018.
* [3] Percy Liang and Dan Klein. Online em for unsupervised models. 2009.
* [4] Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. IJCAI, 2017.
* [5] Hu, Weihua, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. "Learning Discrete Representations via Information Maximizing Self-Augmented Training." ICML, 2017.
* [6]  Divam Gupta, Ramachandran Ramjee, Nipun Kwatra, and Muthian Sivathanu.  Unsupervised clustering using pseudo-semi-supervised learning.  ICLR, 2019.
* [7] Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering, 2013.
* [8] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou.   Variational deep embedding: an unsupervised and generative approach to clustering. IJCAI, 2017.


* [1]Klaus Greff, Sjoerd Van Steenkiste, and Jurgen Schmidhuber.  Neural expectation maximization.  In Advances in Neural Information Processing Systems, pages 6691–6701, 2017.
* [2] Shlomo E Chazan, Sharon Gannot, and Jacob Goldberger. Training strategies for deep latent models and applications to speech presence probability estimation. In International Conference on LatentVariable Analysis and Signal Separation, pages 319–328. Springer, 2018
* [3]  Xu Ji,  Joao F Henriques,  and Andrea Vedaldi.   Invariant information clustering for unsupervised image classification and segmentation.   InProceedings of the IEEE International Conference on computer Vision, pages 9865–9874, 2019.
* [4]  Divam Gupta, Ramachandran Ramjee, Nipun Kwatra, and Muthian Sivathanu.  Unsupervised clustering using pseudo-semi-supervised learning.  In International Conference on Learning Representations, 2019.
* [5] Miyato T, Maeda SI, Koyama M, Ishii S. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence. 2018 Jul 23;41(8):1979-93.
* [6] Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR workshops, 2017.
* [7] Hu, Weihua, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. "Learning Discrete Representations via Information Maximizing Self-Augmented Training." In International Conference on Machine Learning, pp. 1558-1567. 2017.
* [8] Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering. In Iberoamerican Congress on Pattern Recognition, pp. 117–124. Springer, 2013.
* [9] Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. InProceedings of the 26th International Joint Conference on ArtificialIntelligence, pp. 1753–1759. AAAI Press, 2017a.
* [10] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou.   Variational deep embedding: an unsupervised and generative approach to clustering.  InProceedings of the 26thInternational Joint Conference on Artificial Intelligence, pp. 1965–1972, 2017.

This paper proposed a streaming EM algorithm to learn latent representations and cluster in the latent space simultaneously, through a deep GMM model. The EM algorithm was unfolded into network layers as the forward pass, and the global parameters (GMM, encoder, decoder) was optimized through the backward propagation. Another novelty of this work is using both the original latent representation and its GMM reconstruction as the decoder input. Experimental results were provided to evaluate the proposed algorithm. The paper was written with clarity and insightful remarks.

Here are some detailed comments on the technical aspects of the paper:
1) Some readers would ask, is it possible to directly optimize the GMM log-likelihood function in equation (1), together with the reconstruction loss and some entropy regularization? This way the EM algorithm can be avoided in the first place. I think some explanation is needed here to motivate the (streaming) EM algorithm.

2) I assume that N in equation (7) stands for the mini-batch size, and in the denominators the summation over \lambda to get N_k is over the mini-batch instead of the full dataset. It would be nice to state that explicitly in Section 3, to reflect the streaming nature of the algorithm. 

3) The proposed algorithm assumes that \pi is uniform and covariance matrix is identity matrix for all clusters. Hence this is more of a deep K-means model than a full GMM model. More details for updating the covariance matrix and cluster weight are needed if the authors would want to extend it to a deep GMM model.

4) It is a bit unclear to me the approximate equality part (i) in equation (7), especially the re-weighting with 1/N_k. More derivation or explanation would be helpful to explain and justify that step.

5) In Figure 3 we see that more EM layers actually hurt the performance. This is in contrast with the original EM algorithm, which will increase the log-likelihood function monotonically with more EM iterations,  until convergence to a local optima. It is not very clear to me why using more EM layers (T>=7) hurts the performance, and why T=3 seems to be the best. Probably more analysis and explanation are needed here, in addition to the discussions in Section 5.1.

6) In addition to the entropy regularization on individual \lambda_n in equation (9), entropy regularization on \lambda averaged over the mini-batch is also used in other work (e.g., IMSAT). The regularization on the averaged \lambda could drive the solution away from degenerated solution, where all samples are clustered into one or a few clusters. I wonder whether the same technique can also benefit this work.

7) There are quite a few deep clustering algorithms proposed in recent years. It would be nice if the authors can compare with them as well. Here are a few examples:       Ji, Xu, João F. Henriques, and Andrea Vedaldi. "Invariant information clustering for unsupervised image classification and segmentation." In Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874. 2019.      Gupta, Divam, Ramachandran Ramjee, Nipun Kwatra, and Muthian Sivathanu. "Unsupervised Clustering using Pseudo-semi-supervised Learning." In International Conference on Learning Representations. 2019.


This paper consists of two components. One is to unroll the EM algorithm for fitting a mixture of Gaussian model into a differentiable multi-layer network. The other is to insert this unrolled network between the encoder and decoder in the latent space. The whole network can be trained end-to-end for the purpose of clustering. Experiments show that the learned network achieves strong performances on clustering tasks. 

I find the proposed method sound, useful, and efficient. I like Figure 2, which illustrates the whole scheme very well. 

My main concern is that this paper does not break new ground conceptually. Both the idea of unrolling an iterative algorithm and the idea of using auto-encoder for deep clustering are well known. In fact, I have some reservations about unrolling an inference or learning algorithm and embedding it into a bigger network so that it can be trained by an overall loss. There is no doubt that it is convenient and useful, and it is in general consistent with learned computation, but it is conceptually less elegant than a top-down generative model with learned amortized inference. A top-down model may also enable us to determine the number of clusters in a principled way. 

One may interpret the residual network and transformer network etc as unrolled iterative updating algorithms. But they do not assume any top-down model. In the case of a given top-down model such as mixture of Gaussian, embedding the model fitting algorithm into a bigger network appears less than compelling conceptually. 

Incidentally, the unrolled EM-like update appeared in an early version of capsule net. 



Meta-review: This paper presents a method for unrolling the iterative expectation-maximizing steps in the EM algorithm for a Gaussian mixture model into layers in a neural network. Then, the proposed method is applied in the latent space of an autoencoder to allow deep clustering using autoencoder features. The reviewers raised concerns regarding the novelty, lack of systematic ablation experiments, and the efficacy of mimicking iterative optimization steps. Moreover, the training objective does not *exactly* correspond to the variational lower bound that EM often follows. 

The additional experimental results in the revised version address some of the concerns regarding the empirical studies. However, the relationship between the proposed method and incremental EM algorithms and top-down generative models is not properly discussed. Moreover, backpropagating through unrolled iterations does not seem elegant compared to principled approaches such as variational/amortized inference that are commonly used. For example, increasing the number of unrolled steps is observed to hurt the performance due to "vanishing gradients" whereas in the EM algorithm increasing the quality of optimization in the M steps should provide a better fit to training data. 

Given the current presentation, I believe that the paper is not ready to be presented at ICLR. However, I encourage the authors to take the review feedback into consideration to improve the paper.

Decision: withdrawn-rejected-submissions