Title: Contextually Affinitive Neighborhood Refinery for Deep Clustering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Clustering

Self-supervised learning

re-ranking
Abstract: Previous endeavors in self-supervised learning have enlightened the research of deep clustering from an instance discrimination perspective. Built upon this foundation, recent studies further highlight the importance of grouping semantically similar instances. One effective method to achieve this is by promoting the semantic structure preserved by neighborhood consistency. However, the samples in the local neighborhood may be limited due to their close proximity to each other,
  which may not provide substantial and diverse supervision signals. Inspired by the versatile re-ranking methods in the context of image retrieval, we propose to employ an efficient online re-ranking process to mine more informative neighbors in a Contextually Affinitive (ConAff) Neighborhood, and then encourage the cross-view neighborhood consistency. To further mitigate the intrinsic neighborhood noises near cluster boundaries, we propose a progressively relaxed boundary filtering strategy to circumvent the issues brought by noisy neighbors. Our method can be easily integrated into the generic self-supervised frameworks and outperforms the state-of-the-art methods on several popular benchmarks.

Contextually Affinitive Neighborhood Refinery for Deep Clustering

Abstract

Previous endeavors in self-supervised learning have enlightened the research of deep clustering from an instance discrimination perspective. Built upon this foundation, recent studies further highlight the importance of grouping semantically similar instances. One effective method to achieve this is by promoting the semantic structure preserved by neighborhood consistency. However, the samples in the local neighborhood may be limited due to their close proximity to each other, which may not provide substantial and diverse supervision signals. Inspired by the versatile re-ranking methods in the context of image retrieval, we propose to employ an efficient online re-ranking process to mine more informative neighbors in a Contextually Affinitive (ConAff) Neighborhood, and then encourage the cross-view neighborhood consistency. To further mitigate the intrinsic neighborhood noises near cluster boundaries, we propose a progressively relaxed boundary filtering strategy to circumvent the issues brought by noisy neighbors. Our method can be easily integrated into the generic self-supervised frameworks and outperforms the state-of-the-art methods on several popular benchmarks. Code is available at: https://github.com/cly234/DeepClustering-ConNR .

1 Introduction

Fueled by the expressive power of neural networks, deep clustering has emerged as a prominent solution in the low-label regime, where traditional clustering can be significantly enhanced by utilizing latent data representations. Recent advancements in self-supervised learning have further expanded the possibilities for deep clustering [38, 17, 41], providing a solid foundation for grouping similar instances. Among the various self-supervised recipes, contrastive learning methods ensure instance discrimination backed up with sufficient negative samples, while non-contrastive learning methods circumvent the class collision issue [37, 17] by simply pulling two augmented views closer. However, capturing the intrinsic semantic structure with clear decision boundaries remains a challenge in deep clustering, which has led researchers to focus on more group-aware and discriminative approaches.

Group-aware methods commonly enforce the prediction consistency between samples and their nearest neighborhoods, measured by cosine similarity or Euclidean distance for deep clustering[42, 21, 55, 56]. Although these methods are simple and effective, a significant portion of the retrieved neighborhoods may contain semantically redundant information. This is primarily due to the fact that the local neighboring samples are already close to the anchor sample in the feature space, and as such, may not provide sufficient supervision signals for deep clustering. To address this limitation, previous works have added memory banks with historical features to enlarge the search space [21, 49], or labeled the proximity between samples over the entire dataset [55, 56]. Building on the foundation of

t'

t

t'

t

A

B

0

1

1

1

1

0

0

0

Similar

A

B

Dissimilar

Euclidean Neighborhood

with Cross-view Consistency

Euclidean Distance

Contextual Distance

ConAff Neighborhood

with Cross-view Consistency

Euclidean Distance

t

t'

Contextual Distance

Augmented Pairs

Pull Force

Reciprocal Relation

Different Classes

(a)

Top-10 Neighbors Visualization in ConAff Neighborhood

(b)

previous group-aware methods, a key research question arises: Can we excavate more informative and reliable neighbors from intra-batch samples?

Inspired by the versatile re-ranking techniques that have shown remarkable performance in image retrieval tasks [57, 45, 32, 44, 45], we propose a novel approach to deep clustering that involves retrieving neighborhoods in a contextually affinitive metric space. Typically, the neighborhood information is retained in the ranking list of pairwise similarities between a query sample and other samples. However, such a list may neglect the contextual statistics that exist between the samples. When two images are contextually affinitive, they may share similar relationships to a set of reference images. In this study, the reference images are defined as the current batch, and the relationship between two images is established based on whether they are among each other's topk neighbors. Consequently, distant image pairs in the original feature space may be contextually affinitive, and therefore considered as more informative positive pairs. Additionally, the contextually affinitive neighborhood in one view may align with the query sample in the other view, thus enabling cross-view neighborhood consistency. Figure 1 (a) presents a comparison between the conventional Euclidean Neighborhood and the proposed ConAff Neighborhood. Additionally, Figure 1 (b) illustrates the results obtained when retrieving the nearest neighbors using both types of neighborhoods.

In the context of learning with neighborhoods, intrinsic noises can exist that are less observable compared to external noises caused by label contamination. These noises originate from the inferiority of the intrinsic feature space and taking these noisy samples as candidates for clustering can lead to the accumulation errors and degrade overall performance. To address this challenge, we propose a progressive boundary filtering strategy based on the ConAff neighborhood that further improves the robustness of clustering. Our strategy filters out boundary samples in a self-paced learning paradigm and gradually involves more complex samples for clustering.

Our contributions are summarized as follows:

We propose a Co ntextually affinitive N eighborhood R efinery framework (CoNR) for deep clustering, which attempts to excavate more semantically relevant neighbors in a contextually affinitive space, termed as ConAff neighborhood.

We devise a progressive boundary filtering strategy to combat the intrinsic noises existing in the ConAff neighborhood, further improving the robustness of clustering.

Extensive experiments demonstrate that CoNR improves upon the SSL baseline by a significant margin, proving to be competitive with state-of-the-art methods on five widely-used benchmarks. CoNR's simplicity and effectiveness allow it to be easily integrated into other SSL frameworks, resulting in consistent performance gains.

2 Methodology

Our goal is to group a dataset containing n unlabeled samples into k semantic clusters, where k refers to the cardinality of class label sets. Since it is non-trivial to directly obtain clean cluster assignments from scratch, we adopt an unsupervised pre-text task as the initialization for image clustering, which provides a decent neighborhood structure. However, directly utilizing the neighboring information could lead to sub-optimal results as i) the nearest-neighbor searching schema based on cosine similarity may not favor hard positives that are important for clustering, and ii) the top-ranked neighbors of samples located at a cluster boundary contain inevitable noise when the learned clusters have not been not mutually separated. We, therefore, propose our main algorithm for clustering in a contextually affinitive neighborhood, with progressive relaxation of boundary sample filtering, in an attempt to alleviate the above issues.

2.1 Preliminary

In general, given a dataset D = { x 1 , x 2 , · · · , x n } , an image x i uniformly sampled from D , and two augmentations t ∼ T , t ′ ∼ T ′ , we aim to first encourage the instance-aware concordance based on BYOL [11] for early pre-training, and then encourage the group-aware concordance for later finetuning, in which the cornerstone of our method lies. Following [11], the model is equipped with an online network q ( F o ( · )) and a target network F t ( · ) , where each of F o ( · ) , F t ( · ) consists of an encoder cascaded by a projector, and q ( · ) constitutes an additional non-linear predictor.

2.1.1 Instance-aware Concordance

The commonly-used instance discrimination approach for non-contrastive self-supervised learning enforces the similarity between a referenced feature and its corresponding feature in another augmented view, which can be formulated as:

where ⟨·⟩ denotes the cosine similarity. This can be interpreted as the promotion of instance-aware concordance under different augmentations and hidden transformations, which could help bootstrap an over-clustered representation via the stop-gradient and momentum-update techniques. Notably, this instance-aware concordance produces a general-purpose representation that can be well transferred to various downstream tasks, as well as a solid basis for neighborhood discovery and higher-order contextual exploration.

2.1.2 Group-aware Concordance

In order to develop the discrimination framework from instance level to group-aware concordance, we consider regarding the local neighborhood as a semantic group that involves more diverse positive samples, sharing the same spirit with previous works [10, 21, 42, 29]. Typically, group-aware concordance can be formulated as:

where N ( x i ) denotes the set of neighborhood samples of x i preserved in a Euclidean metric space, and x i ∈ B reveals that all samples in the batch are uniformly considered to encourage the cross-view

neighborhood consistency. While the de-facto methods [10, 21, 42, 29] exploit neighborhoods to enrich the latent representation, they somehow ignore the quality of neighborhoods themselves. On the one hand, good neighbors should be clean, which means that we should select proper candidates from B to retrieve clean neighbors. On the other hand, good neighbors should be rich, which motivates us to replace N ( x i ) with more informative neighborhoods in higher-order space. Therefore, we seek to further improve the group-aware concordance paradigm from two perspectives, which will be introduced in Sec. 2.2 and Sec. 2.3.

2.2 Contextually Affinitive Neighborhood Discovery

As discussed in Sec. 2.1, previous works retrieve the nearest neighbors based on pairwise cosine similarity, where the distance metric depends solely on two involved data points on the hypersphere. On the contrary, our solution is to redefine how two data points are similar by considering contextual relations among the whole batch data. Concretely, we first transform the data points into a contextually affinitive space, where we assume that the more similar two transformed data points are, the more high-order contextual information they share, and thus we can utilize this information to excavate more meaningful neighbor pairs, which we term as the contextually affinitive neighborhood discovery.

Formally, given a batch of normalized features B f = [ f 0 , · · · , f |B f |-1 ] T , our goal is to first transform B f into a contextually refined feature space H f = [ h r 0 , · · · , h r |B f |-1 ] T , and then retrieve the nearest neighbors set using the distance metric defined by H f H T f . Inspired by [53, 57, 34] which build the contextual relations over the entire dataset, we efficiently retrieve the contextually affinitive neighborhood via topology graphs in an online manner following [53].

Building Contextual k -NN Graph . The underlying principle of building the contextual k -NN graph is to inject contextual information into nodes V and edges E so that the message propagation can move beyond the local region [53]. First, an online reciprocal adjacency matrix A ∈ R |B f |×|B f | is computed as follows:

where f j ∈ N ( f i , k 1 ) denotes f j is among the k 1 nearest neighbors of f i in B f , and each row of A can be interpreted as the k -reciprocal encoding [57]. Then, the graph G = ( V , E ) can be defined as:

where f j ∈ N ( f i , k 2 ) denotes that f j lies within the k 2 nearest neighbors of f i in B f .

Message Propagation . After building the contextual relational graph, the message propagation can further obtain high-order relations among k -reciprocal encoding acquired at the previous stage, similar to the query expansion technique [35].

where α is a fixed value to realize the α -weighted query expansion ( α -QE), and e ij ∈ E ensures that the most confident k 2 nearest neighbors are selected for aggregation. Normally, k 2 should be far less than k 1 . For clarity, if we assume the number of layers for message propagation is L , the initial input for the message propagation stage, h (0) i , is derived from v i from the previous stage. After propagation across all L layers, the output h ( L ) i becomes our desired contextually refined feature, represented as h r i .

Affinitive Neighborhood Retrieval . So far, the intra-batch data points have been mapped to a contextual metric space via a two-stage transformation. The underlying ranking statistics within these transformed data points should be more accurate and informative. Therefore, we reveal the hidden properties by explicitly retrieving the affinitive neighborhood using the refined node features, which we term as the contextually affinitive neighborhood discovery. Concretely, given the original features [ f 0 , · · · , f |B f |-1 ] T and the refined features [ h r 0 , · · · , h r |B f |-1 ] T , we define the affinitive neighborhood N a ( i, k ) as:

Then the group-aware discrimination can be reformulated as:

Weremark that the contextually affinitive neighborhood discovery ensures more informative neighbors are involved for training at every iteration, and the superior contextual neighbor information lying between the non-differentiable transformed data points is synchronously propagated to the original features to improve the representation quality.

2.3 Progressive Boundary Filtering with Relaxation

In this section, we are focused on the inevitable noise existing in the neighborhood. Although the contextually affinitive neighborhood could offer more contextually relevant samples to boost training, there still exist some unreliable samples among the top-ranked neighbors. Such a phenomenon becomes more evident when the sampled data points lie on the decision boundary. As a consequence, the inclusion of contaminated neighbor information will cause noise accumulation, to the detriment of later training iterations. Therefore, we propose an online boundary sample filtering with progressive relaxation to alleviate the issue.

2.3.1 Online Boundary Sample Detection

In order to better assess the current clustering quality and detect the samples located at the boundary, we apply k-means after each training epoch, which is a common practice in former works [17, 38]. Instead of directly utilizing the consistency of pseudo labels among neighbors, we place more emphasis on the holistic cluster structure, which is to distinguish farther samples located at the cluster boundary with controlled degrees. Inspired by the silhouette score which measures the ratio between the mean intra-cluster distance and the mean nearest-cluster distance in an offline manner, we devise an approximate online version to serve a similar purpose.

Formally, given a sample x i with pseudo label π ( x i ) , and K cluster centroids C = [ c 1 , · · · , c K ] T obtained from last epoch, we first approximate the mean intra-cluster distance d I and nearest mean inter-cluster distance d N by:

where

And then the boundary ratio can be computed as:

Normally, a large boundary ratio indicates the given sample is more likely to be located at the boundary, while a small ratio implies the sample is close to its cluster center, and it is safer to exploit non-boundary samples for training since they exhibit cleaner neighborhoods.

2.3.2 Progressive Relaxation

While boundary samples may possess inferior neighborhoods, it is barely true to ignore these samples during the entire clustering process. To make a balance, we progressively relax the restrictions of samples to promote neighborhood consistency. The motivations are two folds. First, we set a strict criteria to select clean samples at the initial stage to avoid the confirmation bias brought by noisy samples. Second, after the neighborhoods have been consistently refined using cleaner samples, the boundary samples can further improve the overall clustering quality without affecting the holistic structure, which can be validated in our ablation experiments.

Formally, given a batch B sampled from the dataset, we adopt a linearly increased fraction ratio fr ( t ) to dynamically control a filtering threshold σ ( t ) which is used to filter out the hard boundary samples at epoch t .

Algorithm 1 The proposed algorithm CoNR

where t is the current epoch, t 0 is the epoch to start clustering, and T is the maximum number of epochs for group-aware concordance clustering. Then σ ( t ) can be obtained by retrieving the largest ratio among the top |B|∗ fr ( t ) smallest boundary ratios in { r 0 , · · · , r |B|1 } . According to the σ ( t ) , we filter out these samples whose boundary ratios are too large to be involved in clustering at the current epoch t :

Notably, our progressive boundary filtering strategy is essentially different from the self-labeling step that selects the most confident samples for fine-tuning in prior work [42, 29]. First, their selection strategy is completely based on prediction confidence or pseudo-label consistency, neglecting the intrinsic cluster structure to evaluate the difficulty of samples. Second, their self-labeling step is based on cross-entropy loss, which heavily relies on stronger augmentations to avoid overfitting. By contrast, our filtering strategy is proposed to search a robust neighborhood and avoid accumulation errors during the group-aware concordance, which does not require additional modules or augmentations.

2.4 The Overall Objective and Optimization

Based on Sec. 2.2 and Sec. 2.3, our final objective for group-aware concordance can be further formulated as:

where x i ∈ {B\B ( t ) } controls whether the samples should be selected as candidates to retrieve neighborhood, and j ∈ N ( a ) ( i, k ) determines whether the samples are contextually affinitive to the candidates. We do not exclude B ( t ) from N ( a ) ( i, k ) as our aim is to filter out unreliable candidates lying around the boundaries, which means that for each selected candidate, we take its ConAff neighborhood as a whole to preserve its original structure. The overall training pipeline is shown in Algorithm 1.

As discussed in Sec. 2.1, our framework comprises two training stages: instance-aware concordance and group-aware concordance, and the final objective for optimization can be expressed as:

3 Related Work

3.1 Deep Clustering

Deep clustering, an integration of deep learning and traditional clustering, has emerged as a de facto paradigm to learn feature representations and cluster unlabelled data simultaneously. Originated from seminal works [48], deep clustering has seen a paradigm shift from traditional methodologies [6, 36, 28, 43, 52] to more sophisticated models [4, 12, 14, 15, 30, 39]. The advent of contrastive learning in deep clustering [23, 24, 38] has brought new perspectives but also introduced challenges, mainly in defining contrastive losses and balancing instance pairs. Concurrently, self-supervised learning and multi-modal clustering techniques are pushing the boundaries further, with hybrid models combining generative and discriminative aspects, leading to richer representations and improved performance in deep clustering.

3.2 Self-Supervised Learning

Historically, self-supervised learning (SSL) approaches for representation learning have primarily utilized generative models [9] or relied on uniquely crafted pretext tasks such as solving jigsaw puzzles [31] and colorization [51] to the general-purpose representations. However, the emergence of contrastive learning methods [13, 5, 47] has proven highly effective for both representation learning and succeeding tasks. Despite their effectiveness, these methods necessitate a vast array of negative examples to ensure instance-aware discrimination in the embedding space. Apart from instance discrimination, group-aware discrimination focuses more on the semantic structure. While Deep Clustering [2], ODC [50] and COKE [33] disentangle the clustering stage and the learning stage, SeLa [1] and SwAV [3] attempt to solve the grouping problem via optimal transport. Another line of methods [21, 10, 27] focuses on neighborhood consistency, consistently bootstrapping the representation by discovering abundant semantically similar instances.

3.3 Re-ranking in Image Retrieval

Image Retrieval aims to retrieve the gallery images that are the most similar to the query image from a large corpus of images. Re-ranking is a training-free technique that is used to reorder and improve the initial ranking result using higher-order similarity metrics. Generally, k -NN-based re-ranking represents the fashion in image re-ranking. k -NN-based re-ranking views the k -reciprocal nearest neighbors of an image as highly relevant candidates [54, 34, 20]. Jegou et al. [18] iteratively correct distance estimates based on local vector distributions, Liu et al .[25] utilize graph convolutional networks (GCN) to encode neighbor information directly into image descriptors. Zhong et al . [57] encodes the reciprocal information of a query image into a vector and computes similarities using Jaccard distance. Zhang et al . [53] accelerate [57] by using k -reciprocal encodings as the node features of a GNN and enhancing the features via graph propagation. Specifically, our ConAff neighborhood is inspired by [53, 57] which considers contextual similarity with query expansion.

4 Experiments

4.1 Datasets and Settings

Following [17, 23], we report the deep clustering results on five widely-wised benchmarks, including CIFAR-10 [22], CIFAR-20 [22], STL-10 [7], ImageNet-10 [4], ImageNet-Dogs [4]. CIFAR-10 and CIFAR-20 both contain 60,000 images, and CIFAR-20 uses 20 superclasses from CIFAR-100 following prior practice [17, 23]. STL contains 100,000 unlabeled images and 13,000 labeled images. ImageNet-10 selects 10 classes from ImageNet-1k, containing 13,000 images while ImageNet-Dogs selects 15 different breeds of dogs from ImageNet-1k, containing 19,500 images. For image size, we use 32x32 for CIFAR-10 and CIFAR-20, 96x96 for STL-10 and ImageNet-10, 224x224 for ImageNet-Dogs, following the prior work [17]. For the dataset split, both train and test data are used for CIFAR-10 and CIFAR-20, both labeled and unlabeled data are used for STL-10, and only training data of ImageNet-10 and ImageNet-Dogs are used, which is strictly the same setting with [17, 38, 23, 24]. Also, all the experiments are conducted with a known cluster number.

4.2 Implementations

For data augmentations, we strictly follow [17], which uses ResizedCrop, ColorJitter, Grayscale, and HorizontalFlip, for a fair comparison with previous works [17, 38, 41]. For loss computation, we use a symmetric loss by swapping the two augmentations and computing the asymmetric losses twice. For architecture, we use ResNet-18 for small-scale datasets CIFAR-10 and CIFAR-20 and ResNet-34 for the other datasets, following [23, 17, 40]. For CIFAR-10 and CIFAR-20, we replace the first convolution filter of size 7x7 and stride 2 with a filter of size 3x3 and stride 1, and remove the first max-pooling layer, following [17]. All datasets are trained with 1000 epochs, where the first 800 epochs are trained with standard BYOL loss L I sim , and the remaining 200 epochs are trained with our proposed L GAF sim . We adopt the stochastic gradient descent (SGD) optimizer and the cosine decay learning rate schedule with 50 epochs of warmup. The base learning rate is 0.05 with a batch size of 256. For instance-aware concordance, we directly follow [11, 17] to establish a fair baseline. For group-aware concordance, we set k, k 1 , k 2 to 20,30,10 for ImageNet-Dogs and 10,10,2 for other datasets, since ImageNet-Dogs is a fine-grained dataset compared with previous ones.

4.3 Comparison with the State-of-the-Art

Comparison Settings . In this section, we compare our methods ConNR with the current state-ofthe-art methods, as shown in Table 1. Specifically, IIC, DCCM, and PICA are clustering methods without contrastive learning. SCAN and NMM are multi-stage methods that require pre-training and fine-tuning. DivClust is a recently proposed method based on concensus clustering by controlling the degree of diversity, and we report their best performance among different diversity levels in Table 1 for comparison. The other methods mostly learn representations based on contrastive learning or noncontrastive learning in an end-to-end fashion. For these methods that learn a generic representation rather than directly output cluster assignments, we use k -means for evaluation. In particular, for ProPos, BYOL, and ConNR (ours), the produced features output by the target encoder are used for k -means for a fair comparison.

Comparison Results . Regarding the quantitative results, we achieve consistent performance gains across the board. Specifically, for relatively small datasets such as ImageNet-10 and STL-10 with 13K images, our method reaches satisfying results, which not only surpass our baseline BYOL by at least +10.1% ACC on STL-10 in only 200 epochs, but also further achieve a new state-of-the-art performance with 96.4% ACC on ImgeNet-10. For moderate-scale datasets such as CIFAR-10 and CIFAR-20 with 60K images, our method yields stable performance improvement with an average gain of +1.6% and +2.6% ACC compared with ProPos [17] on CIFAR-10 and CIFAR-20, respectively. More importantly, for the challenging fine-grained dataset ImageNet-Dogs, we achieve +6.2% and +1.9% performance boosts in terms of ACC on our baseline and the most advanced competitors. We ascribe this significant improvement to the superiority of our ConAff neighborhood, especially on the fine-grained task. Additionally, we provide results on Tiny ImageNet and performance comparisons in Appendix A.3.

4.4 Ablation Study

In this section, we perform a comprehensive ablation experiment in Table 2 to validate the effectiveness of our proposed method. Specifically, InC represents the baseline, CnAffN, OBD, PR are techniques proposed by our method, and LN is a standard choice used for performance comparison.

Validation of Contextually Affinitive Neighborhood . We make the following observations: First, our proposed contextually-affinitive neighborhood (ConAffN) consistently outperforms the local neighborhood approach, irrespective of whether a progressive boundary filtering strategy is utilized or not. However, ConAffN achieves its peak performance when both online boundary filtering and progressive relaxation are utilized. Instead, the local neighborhood meets with performance saturation when combined with progressively relaxed boundary filtering. Also, as illustrated by the purity curve in Figure 2, ConAffN identifies more reliable neighbors, particularly when the value of K is larger.

Validation of Progressive Boundary Filtering with Relaxation . As previously discussed, both LN and ConAffN combined with OBD experience consistent performance gains, with +1.9% and +1.4%,

NeighborhoodPurityCurve

100

90

Ratio

80

70

60

Boundary Filtering

ConAff Neighborhood

Local Neighborhood

0

5

10

15

20

25

30

35

K

respectively. We speculate that the boundary filtering strategy uniformly improves the robustness of neighborhood clustering. Moreover, the online boundary detection exhibits a much cleaner neighborhood (Figure 2), which supports our motivation. Despite the efficacy of OBD, we find progressive relaxation (PR) drives the model towards less biased clustering as it gradually involves the difficult samples for clustering, which further improves the ACC by +0.7%.

0.70

0.75

0.80

0.85

0.90

0.95

1.00



75

80

85

90

95

100





ACC

NMI

ARI

(a)

0

10

20

30

40

50

k1

-5

0

5

10

15

20

25

30

79.9

79.7

79.6

79.4

79.4

78.8

77.4

76.8

ACC (%) with different k1 and k2 on ImageNet-Dogs

(b)

0.73

0.74

0.75

0.76

0.77

0.78

0.79





ConNR

BYOL

(c)

More Discussions .We conducted an analysis of three parameters: k 1 and k 2 , which represent the number of neighbors used to construct the graph, and fr , the fraction ratio set at the initial training stage. Figure 3(a) presents the results with varying fr values. The outcomes exhibit relative stability within the range of [0.70, 0.80, 0.90] for fr , however, a noticeable performance decline is observed when fr = 1 . This indicates a lack of boundary filtering applications. We provide the results with different choices of k 1 and k 2 in Figure 3(b). For non-aggressive strategy where k 1 and k 2 are smaller than 10, there are consistent gains compared with the baseline, while for more aggressive choices where k 1 , k 2 are larger, the performance can be improved even further, validating the design of contextually affinitive neighborhood. In Figure 3(c), we present the ACC curves generated during the clustering process. Here, we can see that ConNR not only converges considerably faster than BYOL, but also provides notable improvements when implemented as an add-on module. A detailed exploration of how ConNR benefits other self-supervised benchmarks is provided in Appendix A.2.

5 Conclusion and Limitations

In this paper, we proposed a novel method to improve deep clustering in self-supervised learning by promoting the semantic structure preserved by neighborhood consistency. Our approach, the Contextually Affinitive (ConAff) Neighborhood, employs an efficient online re-ranking process to mine more informative neighbors and encourages cross-view neighborhood consistency. We also

introduced a progressively relaxed boundary filtering strategy to mitigate the intrinsic neighborhood noises near cluster boundaries. Our method outperformed state-of-the-art methods on several popular benchmarks and could be easily integrated into generic self-supervised frameworks. One potential limitation of our approach is that ConNR generally assumes the labels induce an equipartition of the whole data and thus does not contain specific mechanisms for handling unbalanced or long-tailed datasets. We believe that our approach can be further extended for this purpose. While our major goal in this paper is dedicated to learning a clustered and well-separated representation for deep clustering, the proposed ConAff neighborhood s a generic design choice that could be viewed from a broader perspective, hopefully benefiting the task of self-supervised learning that focuses more on downstream applications.

References

Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR) , 2019.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) , 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2017.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In nternational Conference on Machine Learning (ICML) , 2020.

Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 1995.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2011.

Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations (ICLR) , 2017.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Jean-Bastien Grill et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Philip Haeusser, Johannes Plapp, Vladimir Golkov, Elie Aljalbout, and Daniel Cremers. Associative deep clustering: Training a classification network with no labels. In German Conference on Pattern Recognition (GCPR) , 2019.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neighbourhood discovery. In International Conference on Machine Learning (ICML) , 2019.

Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning via affinity diffusion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2020.

Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , 2020.

Zhizhong Huang, Jie Chen, Junping Zhang, and Hongming Shan. Learning representation for clustering via prototype scattering and positive sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2022.

Herve Jegou, Hedi Harzallah, and Cordelia Schmid. A contextual dissimilarity measure for accurate and efficient image search. In 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2007.

Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Self-taught metric learning without labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for selfsupervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2021.

Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision (IJCV) , 2022.

Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, and Ioannis Patras. Divclust: Controlling diversity in deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

KL Navaneet, Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Kossar Pourahmadi, Akshayvarun Subramanya, and Hamed Pirsiavash. Constrained mean shift using distant yet related neighbors for representation learning. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems (NeurIPS) , 2001.

Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing (TIP) , 2022.

Chuang Niu, Jun Zhang, Ge Wang, and Jimin Liang. Gatcluster: Self-supervised gaussianattention network for image clustering. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision (ECCV) , 2016.

Jianbo Ouyang, Hui Wu, Min Wang, Wengang Zhou, and Houqiang Li. Contextual similarity aggregation with self-attention for visual re-ranking. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Danfeng Qin, Stephan Gammeter, Lukas Bossard, Till Quack, and Luc Van Gool. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2011.

Filip Radenovi´ c, Giorgos Tolias, and Ondˇ rej Chum. Fine-tuning cnn image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2018.

Anant Ram, Sunita Jalal, Anand S Jalal, and Manoj Kumar. A density based algorithm for discovering density varied clusters in large spatial databases. International Journal of Computer Applications , 2010.

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning (ICML) , 2019.

Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip Torr, and Ling Shao. You never cluster alone. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Guy Shiran and Daphna Weinshall. Multi-modal deep clustering: Unsupervised partitioning of images. In 2020 25th International Conference on Pattern Recognition (ICPR) , 2021.

Yaling Tao, Kentaro Takagi, and Kouta Nakata. Clustering-friendly representation learning via instance discrimination and feature decorrelation. In International Conference on Learning Representations (ICLR) , 2021.

Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations (ICLR) , 2021.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020.

Joe H Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association , 58(301):236-244, 1963.

Hui Wu, Min Wang, Wengang Zhou, and Houqiang Li. A general rank preserving framework for asymmetric image retrieval. In International Conference on Learning Representations (ICLR) .

Hui Wu, Min Wang, Wengang Zhou, Houqiang Li, and Qi Tian. Contextual similarity distillation for asymmetric image retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF international conference on computer vision (ICCV) , 2019.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In nternational Conference on Machine Learning (ICML) , 2016.

Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV) , 2016.

Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmod record , 1996.

Xuanmeng Zhang, Minyue Jiang, Zhedong Zheng, Xiao Tan, Errui Ding, and Yi Yang. Understanding image retrieval re-ranking: A graph neural network perspective. arXiv preprint arXiv:2012.07620 , 2020.

Liang Zheng, Yi Yang, and Qi Tian. Sift meets cnn: A decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 2017.

Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple yet effective method to learn class-sensitive image feature representations by exploring local neighbourhood structures, which are shown discriminative and beneficial to image clustering with the help of off-the-shelves clustering algorithms like k-means. This is achieved by mining \"rich\" (less visually redundant to the anchors) neighbours in a contextual metric space embedding high-order relations among $k$-reciprocal encoding of samples. A boundary sample detection strategy is also introduced to avoid the distractions from false positive sample pairs involved in the discovered neighbourhoods. The effectiveness of the proposed model was validated on 5 standard image clustering benchmark datasets.\n\nSTRENGTHS:\n+ The paper is generally well-written with clear motivations and sufficient discussions of the rationale behind each model design\n+ The proposed model is simple yet effective, which presents a new state-of-the-art for image clustering even though it doesn't explicitly learn the target cluster assignment as conventional deep clustering approaches\n\nWEAKNESSES:\n+ The paper is generally well-written, however, there are still rooms to improve:\n    - I'd suggest clarifying at an early stage that the outputs of the proposed model are the feature representations of images rather than their cluster assignments to one of $k$ classes (L70) given that the paper claimed to propose a ``clustering'' model, which is a bit confusing to me until the experiment section mentions that K-means is adopted.\n    - Is $h^{(0)}$ defined before it is used in Eq.(6)? I failed to find the definition and assumed it is $v_i$ in Eq.(4). Please correct me if I'm wrong.\n    - Is $r$ in Eq.(7) the same as the bounary ratio discussed in Eq.(11)?\n    - How many times the message propagation is conducted for refining the features to obtain $h^r$? \n+ Given that the key idea of the proposed model is to benefit global data structure modelling (clustering) by exploring local data structure (local neighbourhoods), such an idea is closely related to \"Unsupervised Deep Learning by Neighbourhood Discovery\" (ICML'19) and \"Unsupervised deep learning via affinity diffusion\" (AAAI'20) which are missing in the discussion.\n+ Some model designs need to be further justified\n    - The authors claimed that the proposed online boundary sample detection method is better than directly using the consistency of pseudo labels among neighbours to identify boundary labels, I'd suggest adding a simple ablation study to support this claim, e.g. replace boundary ratio with the consistency of sample's pseudo labels to its neighbours and keep all the other designs unchanged.\n    - To add a visualisation of features and highlight those boundary samples identified by the online boundary detection method (e.g. samples with a larger boundary ratio are shown as smaller dots) might be more intuitive for understanding how the data selection is conducted and why it is beneficial\n    - as the neighbourhoods are discovered according to the refined features obtained by multiple times of message propagation (Eq.(6)), it will be interesting to see whether the purity of neighbourhoods is positive correlated to the message propagation times, i.e. neighbourhood mined by $h^{(i+1)}$ is more reliable than that by $h^{(i)}$. \n    - what's the time complexity of mining contextual affinitive neighbourhood.\n    - why not test on Tiny-ImageNet which is another popular benchmark dataset for image clustering and is larger than those datasets adopted here\n\nQUESTIONS:\nGiven that the proposed model is trained to produce class-sensitive features of images, it will be interesting to further explore whether such discriminative features can also benefit other downstream tasks beyond image clustering\n\nLIMITATIONS:\nYes, the authors have addressed the limitation in the conclusion section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper improves self-supervised learning for deep clustering. The self-supervision is based on non-contrastive loss functions, which aim to reduce distance between similar samples. Previous algorithms have proposed both the instance level loss functions and group level cost functions. The current paper improves the group level cost function where the group is neighbours of an anchor sample.  The current work proposes a different way to compute neighbours of each anchor sample. It also proposes to clean the neighbours by removing samples at the cluster boundaries. A scheme to proposed to start training with samples closer to the cluster centre as neighbours and then at the later iterations allowing more noisy samples to contribute in the loss computation.\n\nSTRENGTHS:\n1.\tThis paper is well-written and easy to follow. Most of the details are fully illustrated and the ablation study is quite adequate.\n2.\tGood empirical results were reported on different clustering benchmarks.\n3.\tThe model structure is carefully designed, and the paper provides a clear explanation of each component along with their respective justifications.\n4.\tThe idea of using k-NN graph for affinitive neighbourhood discovery is simple and efficient.\n\nWEAKNESSES:\n1. Motivation of defining a graph as in Eq (3) and then message passing as in Eq (6) is completely missing. Theoretical justification and ablation study to justify both of these steps is essential for this paper.\tWhat if these steps are removed or added with existing group based method? \n\n2. Given the rapid progress in the field, the baseline BYOL used here  is outdated. Compared methods are mostly up to 2021. Many methods have been proposed in 2022 and 2023. Will the proposed steps improve a recent baseline? Also please include some recent methods in comparisons. \n\n3. There is no connection between Eqs (3)-(4) and Eqs (5)-(6). The node representations v_i are never used. Using max in Eq (11) has no advantage because by definition d^N>d^i. In Eq (10) a condition k not equal to \\pi(x_i) must be imposed.  \n\n4. In experiments, the largest dataset only contains 20 clusters. Will the proposed method be effective on datasets with more categories? Many SOTA methods have shown performance at CIFAR100, therefore authors should give performance for at least one such dataset.\n\n5.\tPlease analyse the influence of dropping the initial training with BYOL loss on performance.\n\n6.\tThe backbones are different from other baselines. Please analyse the difference of baselines and its influence on performance.\n\n7.\tHow to choose values of k1, k2 and fr? Is there any data driven scheme?\n\n8.\tIn fIgure 3b,  the performance degrades for k2=20. Please provide explanation why is this happening.\n\n9. There is no details of message passing in the paper. Value of alpha and how many message passing iterations are used? \n\n10. Also some terms like `employs an efficient online re-ranking process' and 'encourages cross-view neighborhood consistency' in the conclusion remain unexplained. Where are these terms in the methodology section?  Similarly, in the abstract, following sentence is not clear: 'However, the samples in the local neighborhood may be limited due to their close proximity to each other, which may not provide substantial and diverse supervision signals.'. Similarly following motivation is missing: `Inspired by the\nversatile re-ranking methods in the context of image retrieval'. Please improve the abstract to represent the contributions of the paper and remove redundant claims which are not materialised in the paper.\n\nQUESTIONS:\nThe same as in Weaknesses section. Authors please respond to the points raised in the weaknesses section.\n\nLIMITATIONS:\nI don't see the proposed algorithm assumes balanced cluster sizes as authors mentioned. The main limitation may be fewer number of classes in the dataset.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task of deep clustering. The paper introduces a new method, Contextually affinitive Neighborhood Refinery framework (CoNR) that is based on sampling among a relevant set of samples, the contextually affinitive space named ConAff neighborhood. Specifically the method uses top-k re-ranking along with $\\alpha$-query extension to select informative neighbours. It further reduces noise among neighbours using a maximum boundary ratio, _i.e._ based on k-means clustering this ratio indicates the proximity of a sample to the cluster boundary. Finally the paper proposes a strategy to progressively relax the condition for sampling, _i.e._ the maximum boundary ratio that is acceptable. The paper validates CoNR on standard deep clustering benchmarks, _i.e._ five relatively small scale dataset: CIFAR-10/20, Imagenet-10/dogs, STL-10. The proposed method shows encouraging results against other deep-clustering state-of-the-art methods. The paper conducts ablation studies of the method on CIFAR-10, and analysis of some hyper-parameters on CIFAR-10 and Imagenet-dogs.\n\nSTRENGTHS:\nS1. The paper is well written and easy to follow.\n\nS2. The CoNR method is well motivated and each component is well introduced.\n\nS3. Experimental results are encouraging when compared to previous state-of-the-art methods for deep clustering.\n\nS4. Ablation studies are present. Ablation studies are really convincing on CIFAR-10: each element seems to participate to the final performance with non-trivial gains.\n\nS5. The paper conducts analysis on the robustness of the method with respect to some of the hyper-parameters used to train CoNR. The method seems robust to variation of hyper-parameters in wide ranges.\n\nS6. Code is present as supplementary material which will help reproducibility. Although I have not tried to use it.\n\nWEAKNESSES:\nW1. The paper lacks discussing a closely related paper from the unsupervised metric learning community that uses intra-batch relations and online re-ranking: Self-Taught Metric Learning without Labels, CVPR 2022. The paper should compare results with the STML method.\n \nW2. The paper should discuss relations to other self-supervised method such as ReSSL: Relational Self-Supervised Learning with Weak Augmentation (NeurIPS 2021) and With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations (ICCV 2021), that use relations with neighbour samples.\n\nW3. The method requires pre-training the network with another self-supervised learning method (BYOL, Bootstrap your own latent: A new approach to self-supervised Learning - NeurIPS 2020). Although this seems to be a common practice in deep-clustering, I believe this reduces the impact of the method, _e.g._ its applicability in large scale setting.\n\nW4. The paper does not conduct experiments on large scale dataset. The two largest datasets in the paper have 60k samples at most. It is not clear whether this type of method will outperform state-of-the-art self-supervised methods methods and k-means clustering on large scale datasets, _e.g._ Imagenet.\n\nW5. The experiments conducted in the paper involved relatively low complexity dataset, _i.e._ with few classes: 20 classes at most. Even though it seems to be, again, a common benchmark, this does not allow to benchmark the method and its effectiveness on complex and realistic tasks.\n\nW6. The paper reports only clustering centered metrics (ACC, NMI and ARI), the paper could also measure common metrics such as knn-accuracy to compare to other method in unsupervised representation learning.\n\nW7. The paper uses the train and test split to train and evaluate the proposed method on the different datasets. The paper should further motivate this beyond it being a common benchmark.\n\nW8: The paper should motivate the use of the 20 subclasses to evaluate CIFAR-100 beyond it being a common benchmark.\n\nQUESTIONS:\nQ1. Line 70: the paper defines the deep clustering task. However it is unclear to me how $k$, the number of clusters, can be set, _i.e._ it is the number of classes on the dataset which should be unknown as it is an unsupervised task. Is there another way to automatically find $k$?\n\nQ2. Eq 10: gives $\\pi’$. At the beginning of the epoch will $\\pi=\\pi’$? The paper could present studies that show the evaluation of $r_i$ during a single epoch and the different epochs.\n\nQ3. Eq 14: gives the final objective for group-aware concordance. Does $\\mathcal{N}^{(a)}(I, k)$ include $\\mathcal{B}^{(t)}$?\n\nQ4. Line 184: the paper defines $\\sigma^{(t)}$ as the boundary ratio, however the boundary ratio was defined in Eq. 11, $r_i$. Maybe $\\sigma^{(t)}$ should be referred to as the maximum boundary ratio in line 184.\n\nLIMITATIONS:\nThe paper proposes a well motivated and introduced method. The experiments presented in the paper are encouraging.\n\nHowever the experiments are limited to small scale benchmarks and low complexity datasets. I believe this necessary to compare CoNR to other self-supervised methods (e.g. Dino, Emerging Properties in Self-Supervised Vision Transformers - ICCV 2021) that shines on large scale datasets.\nFurthermore the paper should compare the proposed CoNR method to the STML method from Self-taught metric learning without labels (CVPR 2022). Both methods are closely related and I believe this is necessary to validate the CoNR against the state-of-the-art.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce the Contextually affinitive Neighborhood Refinery (CoNR) framework, which presents a novel approach to deep clustering. CoNR utilizes a simple and efficient online re-ranking process to identify informative neighbors within the ConAff Neighborhood, promoting cross-view neighborhood consistency. Additionally, CoNR incorporates a progressive boundary filtering strategy that effectively tackles intrinsic noises near cluster boundaries. Overall, the CoNR framework offers a promising solution for deep clustering, enabling improved clustering performance and easy integration into existing SSL frameworks.\n\nSTRENGTHS:\n1) The paper introduces a novel framework called Contextually affinitive Neighborhood Refinery (CoNR) for deep clustering, which allows for the mining of more semantically relevant neighbors.\n\n2) One of the key advantages of CoNR is its seamless integration into other SSL frameworks. Its simplicity and effectiveness enable researchers and practitioners to easily incorporate it into their existing frameworks, leading to consistent performance improvements.\n\n3) To tackle the challenge of intrinsic noises near cluster boundaries, the paper presents a progressive boundary filtering strategy. This strategy adopts a self-paced learning paradigm to gradually filter out boundary samples, thereby enhancing the robustness of the clustering process by involving more complex samples.\n  \n4) Extensive experiments conducted demonstrate that CoNR surpasses state-of-the-art methods on five widely-used benchmarks, establishing its competitiveness in the field of deep clustering.\n\nWEAKNESSES:\n1) The paper does not thoroughly analyze the limitations of the CoNR framework. A deeper exploration of the potential drawbacks or scenarios where CoNR may not perform optimally would have provided a more balanced perspective on its applicability.\n\n2) While the paper briefly mentions the simplicity and effectiveness of CoNR, it could benefit from a more explicit discussion on potential future directions for improvement and extension of the proposed framework\n\nQUESTIONS:\n1) Compared with ConAff neighborhood, why might the local neighborhood samples provide limited supervision signals, and what are the implications of this limitation?\n2) How does the proposed method promote cross-view neighborhood consistency? Need more explanation.\n\nLIMITATIONS:\nThe potential utilization of the ConAff neighborhood in generic SSL frameworks beyond deep clustering is an interesting prospect. However, the paper does not thoroughly explore this possibility, which can be considered a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a Contextually affinitive Neighborhood Refinery framework for deep clustering which employs an online re-ranking process to mine more informative neighbors and encourage the cross-view neighborhood consistency. A progressive relaxed boundary filtering strategy is present to reduce the influence of neighborhood noise near cluster boundaries.

STRENGTHS:
1. The paper is well-written, and the motivation is clear. 
2. Multiple baseline methods are included for comparison.

WEAKNESSES:
1. Multiple terms are used without explanation. For example, what does $l$ in Eq. 6 refer to? And $r$ in Eq. 7?
2. Knowing the cluster number is generally an impractical assumption. What if the actual number of classes is unknown? Will the proposed method be able to estimate the number of classes?

QUESTIONS:
See weakness

LIMITATIONS:
When class-distribution is imbalanced, the neighborhood could be unreliable for identifying hard positives.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06