Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper presents a clustering method in latent space. The work extends a previous approach (Shah & Koltun 2017) which employs a continuous relaxation of the clustering assignments. The proposed method is tested on several image and text data sets.

However, the work has a number of problems and unclear points.

1) There is no theoretical guarantee that RCC or DCC can give good clusterings. The second term in Eq. 2 will pull z's closer but it can also wrongly place data points from different clusters nearby.

2) The method uses an autoencoder with elementwise least square loss. This is not suitable for data sets such as images and time series.

3) Please elaborate "redesending M-estimator" in Section 2. Also, please explicitly write out what are rho_1 and rho_2 in the experiments.

4) The method requires many extra hyperparameters lambda, delta_1, delta_2. Users have to set them by ad hoc heuristics.

5) In each epoch, the method has to construct the graph G (the last paragraph in Page 4) over all z pairs.  This is expensive. The author didn't give any running time estimation in theory or in experiments.

6) The experimental results are not convincing. For MNIST its best accuracy is only 0.912. Existing methods for this data set have achieve 0.97 accuracy. See for example [Ref1,Ref2,Ref3]. For RCV1, [Ref2] gives 0.54, but here it is only 0.495.

7) Figure 1 gives a weird result. There is no known evidence that MNIST clusters intrinsically distribute like snakes. They must be some wrong artefacts introduced by the proposed method. Actually t-SNE with MNIST pixels is not bad at all. See [Ref4].

8) It is unknown how to set the number of clusters in proposed method.


[Ref1] Zhirong Yang, Tele Hao, Onur Dikmen, Xi Chen, Erkki Oja. Clustering by Nonnegative Matrix Factorization Using Graph Random Walk. In NIPS 2012.
[Ref2] Xavier Bresson, Thomas Laurent, David Uminsky, James von Brecht. Multiclass Total Variation Clustering. In NIPS 2013.
[Ref3] Zhirong Yang, Jukka Corander and Erkki Oja. Low-Rank Doubly Stochastic Matrix Decomposition for Cluster Analysis. Journal of Machine Learning Research, 17(187): 1-25, 2016.
[Ref4] https://sites.google.com/site/neighborembedding/mnist

Confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature
review_title: A continuous relaxation for clustering with deep autoencoder

final_score: rating: 6: Marginally above acceptance threshold
review_content: As authors stated, the proposed DCC is very similar to RCC-DR (Shah & Koltun, 2007). The only difference in (3) from RCC-DR is the decoding part, which is replaced by autoencoder instead of linear transformation used in RCC-DR. Authors claimed that there are three major differences. However, due to the highly nonconvex properties of both formulations, the last two differences hardly support the advantages of the proposed DCC comparing with RCC-DR because the solutions obtained by both optimization approaches are local solutions, unless authors can claim that the gradient-based solver is better than alternating approach in RCC-DR. Hence, DCC is just a simple extension of RCC-DR.

In Section 3.2, how does the optimization algorithm handle the equality constraints in (5)? It is unclear why the existing autoencoder solver can be used to solve (3) or (5). It seems that the first term in (5) corresponds to the objective of autoencoder, but the last two terms added lead to different objective with respect to variables y. It is better to clarify the correctness of the optimization algorithm.

Authors claimed that the proposed method avoid discrete reconfiguration of the objective that characterize prior clustering algorithms, and it does not rely on a priori knowledge of the number of ground-truth clusters. However, it seems not true since the graph construction at every epoch depends on the initial parameter delta_2 and the graph is constructed such that f_{i,j}=1 if distance is less than delta_2. As a result, delta_2 is a fixed threshold for graph construction, so it is indirectly related to the number of clusters generated. In the experiments, authors set it as the mean of the bottom 1% of the pairwise distances in E at initialization, and clustering assignment is given by connected component in the last graph. This parameter might be sensitive to the final results.

Many terms in the paper are not well explained. For example, in (1), theta are treated as parameters to optimize, but what is the theta used for? Does the Omega related to encoder and decoder of the parameters in autoencoder. What is the scaled Geman-McClure function? Any reference? Why should this estimator be used?

From the visualization results in Figure 1, it is interesting to see that K-means++ can achieve much better results on the space learned by DCC than that by SDAE from Table 2. In Figure 1, the embedding by SDAE (Figure 1(b)) seems more suitable for kmeans-like algorithm than DCC (Figure 1(c)). That is the reason why connected component is used for cluster assignment in DCC, not kmeans. The results between Table 2 and Figure 1 might be interesting to investigate. 

review_title: Authors of this paper presented a clustering algorithm by jointly solving deep autoencoder and clustering as a global continuous objective. Experiments demonstrate better results than state-of-the-art clustering schemas.

final_score: rating: 7: Good paper, accept
review_content: The authors proposed a new clustering algorithm named deep continuous clustering (DCC) that integrates autoencoder into continuous clustering. As a variant of  continuous clustering (RCC), DCC formed a global continuous objective for joint nonlinear dimensionality reduction and clustering. The objective can be directly optimized using SGD like method. Extensive experiments on image and document datasets show the effectiveness of DCC. However, part of experiments are not comprehensive enough. 

The idea of integrating autoencoder with continuous clustering is novel, and the optimization part is quite different. The trick used in the paper (sampling edges but not samples) looks interesting and seems to be effective. 

In the following, there are some detailed comments:
1. The paper is well written and easy to follow, except the definition of Geman-McClure function is missing. It is difficult to follow Eq. (6) and (7).
2. Compare DCC to RCC, the pros and cons are obvious. DCC does improve the performance of clustering with the cost of losing robustness. DCC is more sensitive to the hyper-parameters, especially embedding dimensionality d. With a wrong d DCC performs worse than RCC on MNIST and similar on Reuters. Since clustering is one unsupervised learning task. The author should consider heuristics to determine the hyper-parameters. This will increase the usability of the proposed method.
3. However, the comparison to the DL based partners are not comprehensive enough, especially JULE and DEPICT on image clustering. Firstly, the authors only reported AMI and ACC, but not NMI that is reported in JULE. For a fair comparison, NMI results should be included. Secondly, the reported results do not agree with the one in original publication. For example, JULE reported ACC of 0.964 and 0.684 on MNIST and YTF. However, in the appendix the numbers are 0.800 and 0.342 respectively. Compared to the reported number in JULE paper, DCC is not significantly better.

In general, the paper is interesting and proposed method seems to be promising. I would vote for accept if my concerns can be addressed.

The author's respond address part of my concerns, so I have adjusted my rating.
review_title: The authors proposed a new clustering algorithm named deep continuous clustering (DCC) that integrates autoencoder into continuous clustering. The paper is interesting and could be improved by increasing the usability of the method.

Meta-review: After careful consideration, I think that this paper in its current form is just under the threshold for acceptance. Please note that I did take into account the comments, including the reviews and rebuttals, noting where arguments may be inconsistent or misleading.

The paper is a promising extension of RCC, albeit too incremental. Some suggestions that may help for the future:

1) Address the sensitivity remark of reviewer 2. If the hyperparameters were tuned on RCV1 instead of MNIST, would the results across the other datasets remain consistent?

2) Train RCC or RCC-DR in an end-to-end way to gauge the improvement of joint optimization over alternating, as this is one of the novel contributions.

3) Discuss how to automatically tune \lambda and \delta_1 and \delta_2. These may appear in the RCC paper, but it's unclear if the same derivations hold when going to the non-linear case (they may in fact transfer gracefully, it's just not obvious). It would also be helpful for researchers building on DCC.
Decision: Reject