Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes to use mixture of experts for image clustering.  The individual expert for each cluster adopts an instance discrimination approach for training.  The proposed method has shown superior clustering performance compared to an extensive number of clustering methods on a reasonable collection of data sets.

Though the overall idea of the proposed method is clear, the paper does not seem to explain some technical details clear enough.  In Equation, the probability P(Y,Z|X) does not seem to be correct. The product term over k appears to be valid only when it uses a hard assignment, i.e. P(z_n|x_n) = 1 for exactly one of the clusters.  The student and teacher embeddings are used before Equation 4 but have not been explained until a later part of the paper.  It is also unclear why an instance discrimination approach would lead to a better clustering performance.

Although an extensive number of clustering methods have been included in the experiments, it has omitted some strong competitors including Variational Deep Embedding (Jiang et al. 2016), Latent Tree Variational Autoencoder (Li et al. 2019), Deep clustering via a Gaussian mixture VAE with Graph embedding (Yang et al. 2019), etc.  Those methods appear to perform better on some of the data sets.  For example, those methods have been reported to yield over 85% of accuracy on the STL-10 data set but the proposed method yields only 75% on the same data set.  It would be better if the paper could include those methods in the experiments or justify the selection of baseline methods.

Overall, the proposed method appears to be an interesting combination of some existing methods.  However, the technical details need better clarity and the experiments should include more relevant methods.
review_title: Combination of existing ideas with concerns on technical details and experimental results

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: Authors present “mixture of experts” type of method to solve a clustering with unsupervised learning problem. Method is called as Mixture of Contrastive Experts (MiCE) which uses contrastive learning as a base module and combines it with latent mixture models. Authors develop a scalable algorithm for MiCE and empirically evaluate the proposed method for image clustering. 

Recommendation: I am tending towards accepting the paper (rating 6). Reason for the acceptance is the novel method supported by empirical evidence. Reason for score not being too high are some weaknesses mentioned in the details later. 
Strengths: 
1) Authors address an image clustering algorithm given number of clusters. Submission is clear, technically correct and present novel findings. 
2) The proposed approach is well motivated. 

Weakness/Questions:
1) All recent papers have ImageNet-10 as one of the five common datasets [1-2]. Why was it omitted in the current paper? 
2) It looks like method is very tied up with MoCo and developed on top of it. Is there an easy/quick way to use other backbones like SimCLR instead of MoCo and still preserve all the steps in the method?
3) Why were images in ImageNet-Dog resized to 96 x 96 x3?
4) Almost all prior methods and proposed method MiCE assume that a number of clusters are known (which shouldn’t ideally be the case). But it looks like the proposed method MiCE uses the information in a better way by assuming number of experts = number of clusters. Can one use more or less number of experts (than number of clusters K) and still partition the sample data into K clusters? Can one easily use over clustering as presented in IIC? 
5) How does the proposed method MiCE fare in terms of computation complexity when compared to MoCo? 
6) How do \mu, w (the expert and gating prototypes) differ during the training? Since the prediction value is the sum of the expert probability weighted by the gating function, one would expect the gating prototypes and expert prototypes to be similar? Is this true? How is consistency maintained (is there a clear correspondence, prototype 1 in expert matches to prototype 2 in gating function)?
7) Can authors elaborate more about the usage of Max-Mahalanobis distribution (MMD) and how exactly does it solve the issue of “unnecessary difficulties in partitioning the dataset if some of them are crowded together.”
8) “Since the images of CIFAR-10 and CIFAR-100 are smaller than ImageNet images, following (Chen et al., 2020), we replace the first 7x7 Conv of stride 2 with a 3x3 Conv of stride 1 for all experiments on CIFAR-10 and CIFAR-100. The first max-pooling operation is removed as well. For fair comparisons”: Is removing  first max-pooling operation standard practice? Is there any performance loss when max pooling is not removed? 

Minor:
1) What does NMI and ARI of > 1 mean? Don’t they have to be in the range of [0,1]? 


[1] Huang, Jiabo, Shaogang Gong, and Xiatian Zhu. "Deep Semantic Clustering by Partition Confidence Maximisation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8849-8858. 2020.
[2] Wu, Jianlong, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. "Deep comprehensive correlation mining for image clustering." In Proceedings of the IEEE International Conference on Computer Vision, pp. 8150-8159. 2019. 



review_title: Novel methodology to solve a clustering problem with scope of improvement in the paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary and Contributions: Inspired by the mixture of experts, authors propose an image clustering algorithm using a mixture of contrastive experts where, each of the conditional models is an expert in discriminating a subset of instances based on contrastive learning.  To this end they  use a gating function to partition an unlabeled dataset into subsets according to the latent semantics and discriminative distinct, where the gating function performs a soft partitioning of the dataset
based on the cosine similarity between the image embeddings and the gating prototypes. The authors carry out experiments on four widely adopted natural image datasets to evaluate the performance of the method in these tasks and compare it to competing methods and baselines.


Correctness and Clarity: The paper is well-written, with informative figures and tables. The paper presents the idea in a clear and straight-forward manner, and is solidly built on top of the current literature. Authors convincingly tested the method with multiple SOTA and baseline and the results look correct to me.  

Reproducibility: The details of the experiments, implementation, and the public datasets are included in the paper. Thanks also for sharing the code.

Additional Feedback and Suggestions:  Since the goal of the paper is image clustering, providing some visual results is appreciated. Also, I am curious to see the performance of the method when we have large number of clusters in our dataset e.g. ImageNet. 

Decision: The idea of using  a scalable variant of the Expectation-Maximization (EM) algorithm to help with the nontrivial inference and learning problems caused by the latent variables seems interesting to me.  And overall, the technical novelty together with the fine evaluation are good enough for ICLR, in my opinion.

review_title:   

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an image clustering methodology based on Mixture of Experts (MoE) for image clustering. 
Although MoE has been proposed for supervised learning problems, the authors exploit the instance discrimination framwork to apply the MoE idea for image clustering. 

This is a novel aspect of the proposed method.

The MoCo framework (unsupervised) for contrastive learning of image representations is employed to define a mixture of MoCo experts model where each expert additionally includes a cluster prototype vector to facilitate clustering.

This unified approach for simultaneous MoCo-based representation learning and clustering seems to provide better results that the two-stage approach of first applying MoCo and then using k-means clustering on the obtained representations.

A probabilistic formulation of the method is presented, along with a training approach based on EM algorithm for likelihood maximization.

There are several concerns related to presentation and clarity.

Comments to be addressed:
1) It would be easier to understand the contribution of the paper, if the MoCo approach were initially described and then the proposed method was presented as a mixture of MoCo experts. The paper in its current form (section 3) is difficult to follow, since several MoCo ideas are mentioned (eg. student and teacher network,  EMA, etc) without been intuitively explained.
2) In section 3 that describes the method, there is no reference about image augmentation, although it is a critical aspect of the approach. Use of image augmentation is only mentioned at the end of the Appendix.
3) A pseudocode descibing the exact steps of the proposed method is imperative.
4) Due to some approximations made, is it possible to prove convergence of the proposed EM procedure?
5) Gating prototypes \omega remain fixed during training. It is important to provide more details on the MMD method used to specify them.
6) It seems strange that, while \omega are specified using embeddings from the the initial network g(x), they are not involved during training.
7) A bad specification of \omega is expected to have strong negative influence on the results that cannot be recovered.
8) What is the size of minibatch B? (eq. (10)).       

review_title: An extension of the MoCo idea in the form of a mixture of MoCo experts for image clustering  

Meta-review: Thanks for your submission to ICLR!

This paper considers a novel unsupervised image clustering framework based on a mixture of contrastive experts framework.  Most of the reviewers were overall positive about the paper.  On the positive side, they noted that the paper had an interesting idea, was well motivated, written well, and had solid results.  Also, the authors provided detailed and useful responses to the reviews, which further strengthened the case for accepting the paper.  On the negative side, one reviewer felt that the paper seemed a bit preliminary and its presentation could improve.  Also, there was some concern about missing comparisons / discussion to previous work (including from a public comment) or data sets (e.g. ImageNet-10).  Again, the authors responded well to these concerns.

Given that the overall response was quite positive with the paper, I'm happy to recommend accepting it.


Decision: Accept (Poster)