Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes combining the latent space of a variational autoencoder with two losses that regularize the latent space. The first loss is the cluster hardening loss in Aljalbout et. al [https://arxiv.org/pdf/1801.07648.pdf]. This loss attempts to convert from a  soft-assignments of points (in latent space) to cluster centers (where the assignments are based on similarities computed via a Student t kernel) to a hard assignments of points in latent space to cluster centers. The transformation is posed as the minimization of a KL divergence.
In the model under consideration there is assumed to be a grid of "cluster centers" or centroids that all points must cluster along.

The second loss (new in this paper), penalizes the similarities between a point and a centroid from being far away from the similarities of the points to the neighbors of the centroid (on the grid). i.e. this loss tries to ensure that neighboring centroids on the grid correspond to similar points in latent space.

Finally, an extension to temporal data is proposed. The temporal model is as follows: an encoder used at each time step to obtain the latent representation and a third loss to encourage that the latent representations across consecutive time steps stays close to each other is incorporated into the learning algorithm. The latent representation is presented as input to an RNN which is used to reconstruct the data.

On NMI and cluster purity (evaluated on MNIST, fashionMNISt), the model outperforms two closely related models (the SOM-VAE and DESOM). Similarly on clustering time series data from physionet, the proposed method outperforms the SOM-VAE. The model also compares results on mean squared error (in predicting the time series 6 hours before ICU dispatch) but their baseline is an LSTM model *without* a latent variable -- a fairer baseline would be against their own model with only the variational bound used for learning. The paper also visualizes what is encoded in the centroids.

Overall this paper's contribution is the use of a VAE (rather than an autoencoder as in related work) that contains a latent space regularized to favour learning cluster structure. The paper provides good empirical evidence to suggest that the combination of the proposed losses alongside the VAE does yield better clustering performance. However, I find the addition of the two losses somewhat ad-hoc and little in the way of explanation is provided for when we should expect such a model to work and when it may not. There is not much of a discussion regarding the complexity of learning with the proposed losses but it looks like a simple algorithm for learning with Equation (1) would have to use the entire dataset to compute it. Could you comment on the scalability of the learning algorithm?

Minor comments, there are several places that need editing for grammar and context:
 * The equation at the top of page 4 needs editing within the subscripts of the summation since i is overloaded
 * line 29 talks about "the observed centroids", but centroids are not mentioned until much later in the paper
 * expand AE the first time it is used as an acronym

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper aims to build a deep clustering algorithm with interpretable latent topology. The authors combine VAE, clustering assignment hardening and SOM. The authors further extend the model to deal with sequence input, by adopting temporal smoothness regularization and language model. 
Good performance in clustering is achieved, experimental results also show that the topological latent structure is captured and the full (sequence) model captures more temporal information than the basic model.
However, the effect of SOM seems to be complicated and not fully explored. 
Considering that VAE+SOM has been introduced before, the novelty of this paper is to combine a few techniques/tricks to strengthen the clustering performance.

Some concerns:
1.	Representation
Good clustering does not guarantee to capture a generally good representation. For example, clustering algorithms lead to a good interpretation of “color” of images that do not need to be good at clustering the semantics of images. If we consider all the factors that generate the data, it is not easy to design metrics to cluster all these factors. 
2.	Vector Quantization/K-means
Can you review, compare and contrast your work with some recent discrete representation learning works? There are some works (e.g. VQ-VAE (-2)) pursing discrete representations via vector quantization or the Gumbel-softmax trick. The learned representations are also highly structured, can be used for visualization and can be used for further clustering (though might not be as good as clustering-driven approaches).
How’s the qualitative difference in the topology of centroids between your method and vector-quantization-based approaches? 
3.	The role of SOM
The clustering performance mostly comes from clustering assignment hardening, while SOM does not clearly benefit the performance of clustering according to table one. 
In figure three, you show that SOM’s row in affecting the clustering performance. As tuning SOM would lead to different trends in NMI and purity, what is your strategy in determining your beta?
Have you ever checked what happened to the centroid topology when varying beta? 
4.	Some suggestions on experiments
.	To fully understand the benefit of each component of your loss, you can try a variant of VarTPSOM without enforcing “smoothness” 
.	How is the beta affect VarTPSOM in sequence evaluation?
.	In your table one, it might be good to compare with some recent discrete latent variable models (as mentioned in bullet point one). 
.       It might be good to try the number of clusters other than 64 in your experiments, towards understanding how it affects the performance.

Minor:
Denote N as the number of samples, and K as the number of centroids. At the bottom of page 3, you need O(K) to calculate each s_{i,j}. At the beginning of page 4, based on {s_{i,j}}, you either need to store intermediate computational results for acceleration, or you might need O(NK) to calculate each t_{i,j}. The whole process would be much slower than k-means/vector quantization, which seems to be not good for large enough data set.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes VarPSOM, a method which utilizes variational autoencoders (VAEs) and clustering techniques based on self-organizing maps (SOMs) to learn clustering of image data (MNIST and Fashion MNIST in particular). An LSTM-based extension termed VarTPSOM is also evaluated on medical time series data. For the most part, the experimental results are promising, and the visualizations are particularly nice.

One of my main points of confusion is with the exposition of the method. To start, the objective presented in Eq. 3 is simply a sum of a variational lower bound and the PSOM clustering loss. Does this have a probabilistic interpretation, e.g., is it a lower bound for a particular generative model? If so, this would be useful to discuss prominently in the paper. If not, it is not clear to me what the authors are gaining from the variational framework. The paragraph at the bottom of page 4 that discusses the "advantages of a VAE over an AE" is not convincing to me. The authors claim that "points with a higher variance in the latent space could be identified as potential outliers and therefore treated as less precise and trustworthy". This isn't demonstrated in the experiments, and to the best of my knowledge, this has not been shown in prior work. If I am mistaken, a citation would be appreciated and should be included in the paper. Additionally, the claim that "the regularization term of the VAE prevents the network from scattering the embedded points discontinuously in the latent space" can also be accomplished with AEs with simple regularization, a standard technique for a wide range of AEs.

Similar comments can be made for the VarTPSOM objective in Eq. 6. Prior work in variational inference for time series, e.g., [1, 2] define a probabilistic time series generative model, from which variational inference naturally prescribes a learning objective. In my opinion, this stands in stark contrast to this work, which takes the VarPSOM objective and simply adds a time series loss on top. This is also a viable approach to building models, but why emphasize variational so much if the method is hardly motivated by anything variational?

I believe that the authors need more thorough experimental comparisons if they wish to demonstrate that their method actually benefits from the variational pieces. Most obviously, I do not believe that any of the comparisons represent the proposed method but with the VAE swapped out for some type of AE? It is my understanding that AE+SOM, SOM-VAE, and DESOM do not represent this exact ablation. VarIDEC performing better than IDEC is a data point in support of this hypothesis, however, this is a comparison of a prior method and not the proposed method.

The related work section mentions that SOM-VAE and DESOM are "likely limited by the absence of techniques used in state-of-the-art clustering methods". Is it possible to address this limitation of prior work? If so, how would this approach compare to the proposed method in terms of implementation and performance? I am not necessarily interested in an actual empirical evaluation, but including this in the related work section would likely be interesting for the reader.

The authors claim in the implementation details that "[s]ince the prior in the VAE enforces the latent embeddings to be compact, it also requires more dimensions to learn a meaningful latent space". Is there a citation for this? My understanding is that posterior collapse leads to VAEs not using additional dimensions even when they are provided, which seems to contradict this claim.

Table 2 seems to have very low NMI numbers across the board, am I reading this incorrectly? Are there prior SOTA numbers that can be included?

Finally, it seems that some of the ideas and motivation in the paper are related to learning discrete structures with variational approaches, e.g., [3, 4]. If the authors agree, it may be appropriate to include some discussion in related work.

[1] Johnson et al, "Composing graphical models with neural networks for structured representations and fast inference". NIPS 2016.
[2] Fraccaro et al, "Sequential neural models with stochastic layers". NIPS 2016.
[3] Tomczak and Welling, "VAE with a VampPrior". AISTATS 2018.
[4] Vikram et al, "The LORACs prior for VAEs: Letting the trees speak for the data". AISTATS 2019.

------

To elaborate on my "Experience Assessment" of "I have read many papers in this area": "this area" in my case refers to amortized variational inference and VAEs, not clustering techniques and SOM.
review_title: Official Blind Review #1

Meta-review: The authors present a deep model for probabilistic clustering and extend it to handle time series data.   The proposed method beats existing deep models on two datasets and  the representations learned in the process are also interpretable.

Unfortunately, despite detailed responses by the authors, the reviewers felt that some of their main concerns were not addressed. For example, the authors and the reviewers are still not converging on whether SOM-VAE uses a VAE or an autoencoder. Further, the discussion about the advantages of VAE over AE is still not very convincing. Currently the work is positioned as a variational clustering method but the reviewers feel that it is a clustering method which uses a VAE (yes, I understand that this difference is subtle but needs to be clarified). 

The reviewers read the responses of the author and during discussions with the AC suggested that there were still not convinced about some of their initial questions. Given this, at this point I would prefer going by the consensus of the reviewers and recommend that this paper cannot be accepted.
Decision: Reject