Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new clustering method called Twin-Contrast Clustering(TCC), which extends the mainstream contrastive learning paradigm to a cluster-level scheme. TCC simultaneously learns instance- and cluster-level representations by leveraging cluster assignment variables. And TCC uses Gumbel Softmax to reparametrize the assignment variables, which makes TCC is able to be trained end-to-end without auxiliary steps.
main_review: Pros:
1.	This paper is well-written and easy to follow. Most of the details are fully illustrated and the ablation study is quite adequate.
2.	The network structure is very cleverly designed. The authors give in-depth thoughts on how to calculate the clustering level representation and how to introduce clustering information into the instance level contrastive learning. Gumbel Softmax is also a reasonable approach to obtain a more discrete category representation.

Cons:
1.	Is TCC more limited on large-scale data with more categories? According to the description in the methods section, batch_size=32xK and queue_size L=100xK, so the complexity of the algorithm and the memory requirements increase significantly when there are more categories. It would be helpful to experiment with data such as TinyImageNet with 200 classes in CC or ImageNet-100 for the main conclusion of the paper.
2.	From the experimental results, cluster-level augmentation has a large impact on the final performance, but the explanation of cluster-level aug(b) is not sufficient. From the descriptions of Line113-115 and Line299-301, aug(b) plays the role of using the default soft assignment instead of hard assignment, so how does hard assignment work? Is only the maximum probability feature retained?
3.	The paper lacks a comparison of TCC with the best unsupervised methods, especially DeepCluster-V2 and SwAV, which introduce the concept of clustering. From aug(b) and the larger $\lambda$ used in Gumbel Softmax, TCC still needs a soft semantic representation to help learning, which plays a similar effect to the prototype used in SwAV, so SwAV is a worthwhile baseline for comparison.
4.	Finally a small question, given that most methods deal with balanced category data, and TCC introduces a 1/K posterior probability prior, does this mean that TCC does not perform as robust as methods like CC when dealing with category unbalanced data?

limitations_and_societal_impact: Please refer to Cons in Main Review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: For image clustering, this paper extends the mainstream contrastive learning paradigm to a cluster-level scheme, and proposes twin-contrast clustering. Heuristic cluster augmentation equivalents are presented to enable cluster-level contrastive learning. Besides, this paper also derives the evidence lower-bound of the instance-level contrastive objective with the assignments. Experiments on several datasets demonstrate its superiority.
main_review: Pros:

(1) The overall writing and presentation are good.

(2) The idea for cluster-level augmentation and contrastive learning is interesting.

(3) Experimental results are good.

Cons:

(1) How to achieve the irrelevant minorities augmentation? Where does the small proportion of irrelevant data come from? Please give more explanation.

(2) The backbone might be different from other baselines. Please analyze the difference and its influence.

(3) Please analyze the influence of training epochs. I notice that at least 1,000 epochs are needed for training, which is much larger than other methods. 

(4) Compared with the improvement on CIFAR and ImageNet-Dog, the results on STL-10 and ImageNet-10 are not satisfying. The explanation in subsection 5.3 is so simple. Please give more analysis about this.

(5) In experiments, the largest dataset only contains 20 clusters. Will the proposed method be effective on datasets with more categories? More experiments are suggested.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a method for end-to-end deep clustering. Authors utilizes NCE loss and proposes a method to produce cluster level loss. They motivate their approach and supply empirical evaluation.
main_review: The authors propose a method for end-to-end deep clustering. Authors utilizes NCE loss and proposes a method to produce cluster level loss. They motivate their approach and supply empirical evaluation. The paper is very clearly written and the problem under is considerations very important topic. As the title of the paper and introduction hints clustering is depending on the representation of one single point but representation of the group of point. Unfortunately, authors didn’t consider appropriate state-of-the art methods as baselines. A small list is given below:
a)	Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation (available in ICLR webpage since October 2020)
b)	Consensus Clustering with Unsupervised Representation Learning (available in arxiv since October 2020)
c)	Representation Learning for Clustering via Building Consensus (available in arxiv since May 2021)
I would like to be fair to authors and I will only consider some parts of c) in the next steps. However, I don’t see any reason why a) and b) are not considered as  baselines or discussed in their paper. Both papers are end-to-end and contains relevant results. The method given in a) is outperforming the proposed method on CIFAR-10, STL-10, ImageNet-10. On ImageNet-10 the difference is app. 10%. 
On the other hand, the method b) has comparable results however the main motivation of the paper under review is very similar to it. Please note that b) doesn’t use NCE type loss but it builds on BYOL.
Although the paper c) is a direct baseline to the paper under review (NCE is used, different types of clustering constraints are handled) and it improves state of the art, I will ignore the clustering results. However, c) also proposes a new method of evaluating deep clustering method and points out several drawbacks of the clustering methods e.g., sensitivity to hyper-parameter tuning. I would like to see similar analysis from authors in the author response.

All these points bring questions about the novelty of the method and performance of the method. I have some concerns; I would like to see significant amount of improvements.

<Please see the discussion below. After the rebuttal and discussion with authors, I have modified my score>
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends the mainstream contrastive learning paradigm to a cluster-level scheme. Specifically, the author propose twin-contrastive clustering (TCC),  which links the instance-level learning track with the cluster-level one. Extensive experiments show the advantages of TCC.
main_review: The motivation of this paper is very intuitive, and in fact, some relevant papers (as described in the paper) have been explored . But interestingly, the authors compile contrastive learning while partially borrow from deep set representations, realizing the two-level contrastive learning in a novel way.
Some issues:
(1) In 'Instance-Level Contrastive Learning' part, for the term e, why use f_\theta(x) + NN_\theta(c)? How about  using concatenate operation？
(2) In 'Cluster-Level Augmentation Equivalents', why consider 'irrelevant minorities'? How to generate and use such augmentation?
(3) In fact, SwAV [a] is also a kind of contrastive learning with both instance-level and cluster-level, so it should be added as a baseline.
(4) Compared with CC, will the improvement of TCC performance benefit from model capacity increase? and in fact TCC has more trainable parameters.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper proposes a novel extension to contrastive learning, using both cluster level and instance level tasks in representation learning. The bulk of the reviewers found this method interesting and well-motivated, and from my perspective as well using this additional structure is a good direction in contrastive learning / representation learning. The authors felt their work should be viewed independently from SwAV, but the reviewers clearly saw the strong overlap between what the two models do. The reviewers added SwAV as a baseline, and their model performed favorably on their benchmark tasks. However, I encourage the authors to be generous on the connection between their work and SwAV in particular in the final draft, while the downstream applications are similar, I believe the motivations are very similar. Finally, the reviewers requested more benchmarks, such as TinyImagenet, and this satisfied the reviewers. Therefore I recommend acceptance as a poster.

A note that I do believe that [a] "Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation" should be included in the final draft. I do not believe that the significance of the submitted work should be judge by [a], as it is quite new and could have feasibly fallen under the radar with no ill intent. But I believe both papers are doing similar things and as such this paper could do the community a service by including results in [a] and providing this as additional context to their results. I strongly encourage the authors to do this.
Decision: Accept (Poster)