Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: - The idea of combining learning representations and ensemble clustering is interesting and possibly promising. I think this paper in its present form is not ready to be published though.

- Cited literature on consensus clustering is old and outdated (up to 2005) and does not capture the state of the art. More recent work exists that outperforms the ones cited in this paper. The authors should also consider comparing their method with the state of the art on consensus clustering which is not deep learning based.

- The description of the algorithm is often too vague and is fragmented in mini-sections, and the flow that brings them together is unclear. The individual pieces are not novel and no novel technical challenges or contributions are presented. Several steps and settings are ad-hoc and not well-motivated. What does it mean exactly: "Performing a forward pass on the two views results in feature vectors f1,f2."? How are f1 and f2 obtained? What's the motivation for using a multi-layer perceptron to further reduce f1 and f2? Why should we expect this to produce a good representation for clustering? How were the dimensionalities of the hidden and output layers chosen?

- The notation in Eq. (2) is not clearly defined. What is a transportation polytope?

- Writing and organization need major work. There are many vague statements that are not well-supported or motivated. 

- How does the set of observations X defined in 3.1 relate with the input batch X_b in 3.1.2?

- The process of generating two views of the input images is not clearly defined. It seems that this process mainly consists in adding random noise to the images? This is quite different from the notion of different *views* of a data object (e.g. image) in the literature. Also, why 2 views and not 3 or more?

- I don't find Figure 1 particularly useful. Part (a) does not help at all in understanding the approach.

- What's the size of a typical ensemble being generated? The paper mentions M transformations. What is the typical M value. In the experiments M ranges from 10 to 100. How sensitive is the approach to the specific transformations being applied and to the number of transformations?

- The relationship and distinction between the views and the ensemble need to be better clarified.

- I find the explanation above Eq (4) unconvincing. How diverse are the probability estimations generated by each component of the ensemble? Have the authors performed any analysis on this?

- The authors should discuss assumptions and limitations of the proposed methodology. What assumptions does the proposed methodology make? Under which scenarios and data distributions is the methodology expected to perform well? What are its limitations?

- No insight or analysis on the results is provided. Often PICA outperforms the proposed method. I understand that PICA uses both training and test data, but since we are dealing with clustering and unsupervised data, I am not sure this is a real advantage for PICA.



review_title: The authors propose a deep learning approach to clustering that combines learning representations and ensemble learning. The aim is to learn a unifying data representation by creating a consensus on multiple clustering outputs. The type of clustering being pursued is a standard centroid-based clustering.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the effect of combining ensemble learning approaches with deep clustering. The paper wants to show that ensemble learning methods, in particular consensus clustering, can improve the clustering accuracy when combined with general representation learning/clustering blocks. However, I am not sure that the results presented in the paper are enough to support the claims.

The paper's  pros are:
(+) It Is the first to combine ensemble methods with deep clustering models. Although ensemble methods have been widely applied, studying consensus clustering in the current problem setting is novel.
(+) It Is the first to be able to have an ensemble deep clustering algorithm that gains empirically over other state-of-the-art models, showing the ideas to be potentially effective.
(+) The writing is in general clear and undestandable.

The paper's cons are:
(-) The wording in the abstract is a bit confusing in the sense that after reading it one might think the algorithm does consensus clustering first and uses the clustering to learn better representations of the input data. Although this is clarified later in the main body.
(-) The description of the main algorithm seems to be more intuitive than innovative. Some algorimic design choices are not very convincing. For example, the choice of using random projections on embedding to produce different clusterings, although an interesting idea, makes me wonder why it is necessarily a good way to introduce randomness into the whole framework. The authors can expand their discussion on this. I also remain dubious about why different representation instead of different clustering methods 
Also, since the authors meant the idea to be applicable to general representation learning/deep clustering blocks, I'm not sure the current experimental data in the paper can lead to that conclusion.
(-) Using the performance metrics provided by the authors, I find it a bit hard to conclude that the proposed algorithm has a significant advantage over state of the art methods, especially PICA. Also, the fluctuation in performance metrics caused by different parameter settings seems to be, in magnitude, at least comparable to the margin of ConCURL over the baselines.

Overall, I think this paper contains  interesting seed ideas such as combining consensus clustering with representation learning and making use of the learned representation to generate multiple clusterings. These seed ideas could be good for this venue. However, the work is still premature and flawed by crude algorithm/experiment design.  The quality can be significantly improved if the authors can give a more general algorithmic framework (since the authors meant the ideas to be applicable to general representation learning/clustering algorithms), equipped with more thorough experimental investigation to support the applicability and superiority of the current approach.
review_title: A good attempt at combining ensemble learning approaches with deep clustering

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a learning-based approach for image clustering. In particular, similarly to recent algorithms fro unsupervised representation learning, such a DeepCluster, they propose to iterate between clustering the images in the feature space of the network and updating the network weights to respect the clusters. Two main differences with respect to DeepCluster-like algorithms is that they target the task of clustering itself, and do not evaluate the generalizability of the leaned representations for other tasks, and that they propose to use cluster ensembles to improve training robustness. In particular, they generate 2 clusterings of the images at every iteration by applying different sets of data augmentations and feature transformations to the input images. The objective is then not only to respect these clusterings but also to enforce consistency between them over time, thus improving representation invariance to irrelevant image details. In an experimental evaluation on a set of standard image clustering benchmarks they outperform prior work in most scenarios. 

The idea is reasonable and the method seems sound, however a similar approach has been proposed in Zhuang et al., ICCV'19 (Local Aggregation for Unsupervised Learning of Visual Embeddings). In contrast to this work, the authors of Zhuang et al., used different runs of the clustering algorithm to obtain diverse clusterings, instead of transforming the images, but the overall approach is very similar. The authors seem to be not aware of that work.

In the current from it is not clear whether the proposed approach has any advantages over Zhuang et al. In the rebuttal the authors need to provide a discussion of their novelty with respect to that method as well as an experimental comparison on ImageNet.


I appreciate the authors' efforts to fairly compare to Zhuang et al. in the rebuttal, and I do find the preliminary evidence sufficient to establish that their approach outperforms LA on clustering metrics. However, the authors seem to miss the the point that LA is not just another consensus clustering approach which they forgot to include into literature review since it does not report clustering metrics. The main contribution of their work is combining consensus clustering with representation learning, which is exactly what the authors of LA had done before. It does seems that the particular approach proposed in this paper results in a better clustering performance, so the submission contains a valid contribution, but the relationship between the two methods needs to be discussed in a lot more detail, and they have to be throughly compared experimentally. I encourage the authors to improve the manuscript in this direction and resubmit to a different venue.
review_title: Valid, but a very similar prior work ignored by the authors

Meta-review: This paper proposes a model for learning using ensemble clustering. The reviewers found the general idea promising.
However, while promising, all reviewers noted that in its curent form the paper is not fit for publication. The reviewers pointed out missing references, issues with the abstract, lack of motivation for some of the algorithmic choices, limited novelty over clarity in the description of difference w.r.t. previous work.
Because of all these reasons, this paper does not meet the bar of acceptance. I recommend the authors take into account the feedback provided in the reviews and discussion and resubmit to another venue.
Decision: Reject