Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel deep clustering method with non-contrastive representation motivation. The authors provide detailed experimental results to show the superior performance of the proposed method.
main_review: 1. strengths

-the authors provide a sufficient literature review.
-the paper is well organized.
-detailed appendix, including ELBO, convergence analysis, experimental setup, etc.
-abundant experimental results.
-large performance improvement.

2. weaknesses

-the non-contrastive representation motivation maybe overstated. L_{aug-ins} is a non-contrastive loss, but L_{pcl} is a typical contrastive loss. From Tab.4, L_{pcl} may be more important than L_{aug-ins}.
-the proposed L_{pcl} is not novel enough. This loss is actually class-level contrastive loss, which is proposed in the work (Contrastive clustering, AAAI 2021)
-the EM-based framework is not novel enough. ProtoNCE firstly adopt this framework in representation clustering related area.
-the reviewer would like to see more clustering results in some challenging datasets, such as Tiny-ImageNet, ImageNet-50/100/200 subsets (in SCAN, ECCV 2020).

summary_of_the_review: Although the reviewer provides some negative feedback in Main Review, the reviewer still likes this paper. Therefore, I will give a positive rating.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: For deep clustering, this paper explores the non-contrastive representation learning based on BYOL to handle the issue of the class collision caused by inaccurate negative samples. ProtoCL is proposed to encourage prototypical alignment between two augmented views and prototypical uniformity, hence maximizing the inter-cluster distance. Experiments on various datasets demonstrate the superiority of the proposed method.
main_review: Pros:

(1) The writing and organization are good, which makes the paper easy to read and follow.

(2) The authors conduct extensive experiments, including the ImageNet to show the superiority. The results are good.

Cons:

(1) My main concern lies in the novelty, which is very limited. This paper simply combines two existing unsupervised learning methods, BYOL and PCL, and then applies it to the clustering task. The class collision issue for negative samples contrastive learning methods is well addressed by BYOL instead of this paper. The difference between the proposed ProtoCL and PCL comes from the BYOL. The EM framework is also presented by the PCL. Besides, the neighbor examples based positive sampling strategy has been well investigated in both unsupervised feature learning and deep clustering areas [1,2]. Therefore, the only contribution is the combination and application to the clustering task, which is obviously insufficient.

(2) Two important references [1,2] are missing. These two methods thoroughly study the positive sampling strategy for clustering and unsupervised feature learning to handle the issue of inaccurate negative samples. Therefore, the positive sampling strategy in NCC is not new. Please also compare the results with these methods.

(3) Though the reported results are very high, the comparison might be unfair. Some existing contrastive learning based clustering methods, such as CC and GCC [1], can also adopted the framework of BYOL and PCL to improve the results. Therefore, the results might be unconvincing.

[1] Graph Contrastive Clustering, ICCV 2021
[2] Weakly Supervised Contrastive Learning, ICCV 2021
summary_of_the_review: This paper proposes the non-contrastive representation learning method for deep clustering based on BYOL. It is a combination of two existing methods. The novelty is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel method of self-supervised representation learning.
To circumvent the class collision issue arose from building a large set of negative samples in contrastive SSL based methods, it is built from non-contrastive SSL methods, such as BYOL.
The goal is to handle the weaknesses of non-contrastive SSL methods, training instability and representation collapse.
The two proposed methods for alleviating the two factors are i) augmented positive sampling and ii) optimizing uniformity of representation space via prototypical cluster features computed from k-means clustering.

main_review: ### Strengths
* The comparison with the prior works and the motivation of the proposed method are well described and reasonable.
* The efficacy of the proposed methods are well proven with the experimental results.
* A lot of experimental results on hyperparameters search show that the proposed method is quite robust.


### Weaknesses
* Class-collision issue arose from k-means clustering

    I think it's kind of a chicken and egg situation, cause if the space is not well uniformly distributed, the result of the k-means clustering would be quite degenerated which would result in a severe class collapse issue.
Have you checked the k-means clustering results with the ground truth class labels?

* Lack of analysis on the over-clustering results

    Based on the results in Table 4 and Figure A5, it seems that the pre-defined value of the number of clusters, K, has the largest impact on the performance (more than 10% in terms of accuracy score). It is quite intuitive to think that the more the clusters are set, the better the classification performance (cause there are more exemplars to infer the class), but the results are the opposite of this idea for the CIFAR-10 dataset. Since it is the factor that impacts the performance most, I think more elaboration of this result is required.
summary_of_the_review: I think this paper is well written in that the motivation, the description of methodology and the analysis of the results are all reasonable.
Though I feel more details are needed for some factors that i described in the main review section, I think this work is good since the proposed method has pointed out the weaknesses of the previous methods and improved it experimentally.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes that negative samples are risky, therefore, let's cluster samples in the latent space to form prototypical clusters, and apply contrastive loss only on the cluster centres and not on the features themselves, unlike in the case of ProtoNCE where all samples are optimized. This however risks clusters becoming meaningless, hence the cluster grouping loss which maximizes the likelihood of each sample being from a nearby cluster with a Gaussian assumption. The method is tested on multiple datasets outperforming all compared methods.
main_review: ## Strength

### Good performance

The paper demonstrates a clear improvement over existing methods on various datasets. It is unfortunate that the method is not tested on the same settings as PCL in Table 2, but Table 3 does seem to show that there is a large enough gap between the two.

### Well motivated

The motivation for the paper is really on the fact that negatives harm, and the paper shows how to effectively avoid this from happening. Have your clusters well spread, but don't ask for individual samples to be further away, simply ask them to be closer to each center. And by enforcing the centres to be spread, you naturally avoid asking two different samples close by to be further away, which may not be nice.

## Weaknesses

### Missing PCL in Table 2

PCL is perhaps the exact antithesis of the proposed method. The method already shows to work better than PCL in Table 3, so why not include it in Table 2 to make it VERY clear that the proposed way is the way to go compared to PCL? This is personally a must-have experiment for me.

### Ablation study

The ablation study can be improved by perhaps also showing how the ProtoNCE's formulation of negative losses harms performance. For example, what happens if you simply add that term to the NCC formulation? 

Also, from the ablation study, it is slightly questionable whether the alignment loss is necessary. In fact, shouldn't the positive sampling also do the same thing as the alignment loss? 

### Presentation

The presentation can be improved. The paper is not too hard to follow but could be presented in a more top-down way. This might be a presentation style issue, and it does come to personal taste, so please do take this into account only as a suggestion.

The abstract does not read well. There are multiple novel terms introduced, "a positive sampling strategy" and "prototypical contrastive loss" that are not explained, which makes it hard to follow. For example,

> First, we propose a positive sampling strategy to align one augmented view of instance with the neighbors of another view so that we can avoid the class collision issue caused by the negative examples and hence improve the within-cluster compactness.

Can be simply written as

> First, we relate between neighbouring instances in the learned latent space to improve within-cluster compactness leading to less collision of non-related classes.

Prototypical contrastive loss also can be presented in a similar way, without necessarily introducing the term. In fact, I would personally argue that it is easier to understand the method the other way around. The main idea of the paper seems to be that negative samples are risky, therefore, let's cluster samples in the latent space to form prototypical clusters, and apply contrastive loss only on the cluster centres and not on the features themselves, unlike in the case of ProtoNCE where all samples are optimized. This however risks clusters becoming meaningless, hence the cluster grouping loss which maximizes the likelihood of each sample being from a nearby cluster with a Gaussian assumption.

This problem persists also in the introduction, as the reader is unaware of what the paper means by "positive sampling" until Section 3.2, and Prototypical contrastive loss until 3.3. This problem is exacerbated since it is unclear what 3.2 and 3.3 are trying to achieve until you reach 3.4, which provides the picture in which all of these fit in place.

summary_of_the_review: The paper shows promising results, with a great motivation. However, there are some issues that can be easily fixed to greatly improve the quality of the paper. Specifically, ProtoNCE, or PCL, should be included in Table 2, and an additional baseline that empirically shows the negative loss harming performance would be great. Presentation could also be improved to be more clear.
review_title: null

Meta-review: This paper received the initial scores with large variance. During the intensive discussion (Number of Forum replies is up to 60), the opinions reached the consensus. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *rejection*. Here are the comments that I summarized, which include my opinion and evidence.

**Research Problem and Motivation**

(1) It seems that the authors aimed to address the question that “are negative examples necessary for deep clustering?” This research problem has been proposed and addressed in BYOL and SimSiam (If I remembered correctly, some reviewer pointed this out). What the authors actually did is to add two more components, positive sampling strategy and prototypical contrastive loss, on the top of BYOL. In my eyes, it is like putting two patches on BYOL, where one of them does not work (I will explain later). 

(2) Moreover, the authors failed to clearly illustrate the drawback of BYOL. In the last sentence of the third paragraph in the Introduction part, the authors mentioned that “BYOL only optimize the alignment term, leading to unstable training and suffering from the representation collapse.” This sentence is too general, which lacks strong motivation. 

Therefore, the research problem addressed here is an incremental problem over BYOL, rather than brings new insights into the contrastive learning community.

**Philosophy**

Without a clear motivation, it is difficult to catch the philosophy of this paper, i.e., how their proposed components tackle BYOL’s drawbacks. Moreover, the relationship between two components is also unclear. 

**Novelty**

I believe Reviewer Na7g has a thorough analysis of the novelty of this paper. I will not go into details here. The difference does not mean novelty. 

**Technique**

Positive sampling strategy does not work. If we take a closer look at Table 4, the rows of BYOL and NCC with PS, there is no significant performance gain. The p-values of t-test on ACC results on CIFAR-10 and CIFAR-20 are 0.92 and 0.32, respectively. Actually, the prototypical contrastive loss is the key element to boost performance over BYOL.

**Misleading Title**

Based on the above point, the title is misleading. Although no negative data sample pairs are used in the training, the contractiveness on the cluster-level should also belong to the scope of contrastive learning.

**Experiments**

(1) In the Introduction part, the authors mentioned that SimSiam is in the same non-contrastive category with this paper. However, this paper is not included in the comparison.

(2) The competitive methods in Table 2 and 3 are not consistent. The authors even did not report the performance of BYOL on ImageNet-1K.

(3) Positive sampling strategy does not work. See the above Technique point.

(4) The authors only reported the running time on CIFAR-10 and CIFAR-20.

Therefore, the experimental results are not very convincing and solid to me.

**Presentation**

I believe the presentation also needs many efforts to smooth the logic. For example, “Even though Grill et al. (2020); Richemond et al. (2020) have proposed to use some tricks such as SyncBN (Ioffe & Szegedy, 2015) and weight normalization (Qiao et al., 2019) to alleviate this issue, the additional computation cost is significant.” Actually, Grill et al. (2020) is BYOL, where the authors added their components on. The computational cost of the proposed method should be heavier than BYOL. 

Based on the above points, this paper suffers from several severe issues, which makes it not self-standing.
Decision: Reject