SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As discussed in Sec. 2.1, our framework comprises two training stages: instance-aware concordance and group-aware concordance, and the final objective for optimization can be expressed as:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathbb {I} (t <   t _ {0}) \mathcal {L} _ {\text {s i m}} ^ {I} + \mathbb {I} (t \geq t _ {0}) \mathcal {L} _ {\text {s i m}} ^ {G A F} \tag {15}
$$
3.1 Deep Clustering
Deep clustering, an integration of deep learning and traditional clustering, has emerged as a de facto paradigm to learn feature representations and cluster unlabelled data simultaneously. Originated from seminal works [48], deep clustering has seen a paradigm shift from traditional methodologies [6, 36, 28, 43, 52] to more sophisticated models [4, 12, 14, 15, 30, 39]. The advent of contrastive learning in deep clustering [23, 24, 38] has brought new perspectives but also introduced challenges, mainly in defining contrastive losses and balancing instance pairs. Concurrently, self-supervised learning and multi-modal clustering techniques are pushing the boundaries further, with hybrid models combining generative and discriminative aspects, leading to richer representations and improved performance in deep clustering.
3.2 Self-Supervised Learning
Historically, self-supervised learning (SSL) approaches for representation learning have primarily utilized generative models [9] or relied on uniquely crafted pretext tasks such as solving jigsaw puzzles [31] and colorization [51] to the general-purpose representations. However, the emergence of contrastive learning methods [13, 5, 47] has proven highly effective for both representation learning and succeeding tasks. Despite their effectiveness, these methods necessitate a vast array of negative examples to ensure instance-aware discrimination in the embedding space. Apart from instance discrimination, group-aware discrimination focuses more on the semantic structure. While Deep Clustering [2], ODC [50] and COKE [33] disentangle the clustering stage and the learning stage, SeLa [1] and SwAV [3] attempt to solve the grouping problem via optimal transport. Another line of methods [21, 10, 27] focuses on neighborhood consistency, consistently bootstrapping the representation by discovering abundant semantically similar instances.
3.3 Re-ranking in Image Retrieval
Image Retrieval aims to retrieve the gallery images that are the most similar to the query image from a large corpus of images. Re-ranking is a training-free technique that is used to reorder and improve the initial ranking result using higher-order similarity metrics. Generally,  $k$ -NN-based re-ranking represents the fashion in image re-ranking.  $k$ -NN-based re-ranking views the  $k$ -reciprocal nearest neighbors of an image as highly relevant candidates [54, 34, 20]. Jegou et al. [18] iteratively correct distance estimates based on local vector distributions, Liu et al.[25] utilize graph convolutional networks (GCN) to encode neighbor information directly into image descriptors. Zhong et al. [57] encodes the reciprocal information of a query image into a vector and computes similarities using Jaccard distance. Zhang et al. [53] accelerate [57] by using  $k$ -reciprocal encodings as the node features of a GNN and enhancing the features via graph propagation. Specifically, our ConAff neighborhood is inspired by [53, 57] which considers contextual similarity with query expansion.
4.1 Datasets and Settings
Following [17, 23], we report the deep clustering results on five widely-wised benchmarks, including CIFAR-10 [22], CIFAR-20 [22], STL-10 [7], ImageNet-10 [4], ImageNet-Dogs [4]. CIFAR-10 and CIFAR-20 both contain 60,000 images, and CIFAR-20 uses 20 superclasses from CIFAR-100 following prior practice [17, 23]. STL contains 100,000 unlabeled images and 13,000 labeled images. ImageNet-10 selects 10 classes from ImageNet-1k, containing 13,000 images while ImageNet-Dogs selects 15 different breeds of dogs from ImageNet-1k, containing 19,500 images. For image size, we use  $32 \times 32$  for CIFAR-10 and CIFAR-20,  $96 \times 96$  for STL-10 and ImageNet-10,  $224 \times 224$  for ImageNet-Dogs, following the prior work [17]. For the dataset split, both train and test data are used for CIFAR-10 and CIFAR-20, both labeled and unlabeled data are used for STL-10, and only training data of ImageNet-10 and ImageNet-Dogs are used, which is strictly the same setting with [17, 38, 23, 24]. Also, all the experiments are conducted with a known cluster number.
4.2 Implementations
For data augmentations, we strictly follow [17], which uses ResizedCrop, ColorJitter, Grayscale, and HorizontalFlip, for a fair comparison with previous works [17, 38, 41]. For loss computation, we use a symmetric loss by swapping the two augmentations and computing the asymmetric losses twice. For architecture, we use ResNet-18 for small-scale datasets CIFAR-10 and CIFAR-20 and ResNet-34 for the other datasets, following [23, 17, 40]. For CIFAR-10 and CIFAR-20, we replace the first convolution filter of size  $7 \times 7$  and stride 2 with a filter of size  $3 \times 3$  and stride 1, and remove the first max-pooling layer, following [17]. All datasets are trained with 1000 epochs, where the first 800 epochs are trained with standard BYOL loss  $\mathcal{L}_{sim}^{I}$ , and the remaining 200 epochs are trained with our proposed  $\mathcal{L}_{sim}^{GAF}$ . We adopt the stochastic gradient descent (SGD) optimizer and the cosine decay learning rate schedule with 50 epochs of warmup. The base learning rate is 0.05 with a batch size of 256. For instance-aware concordance, we directly follow [11, 17] to establish a fair baseline. For group-aware concordance, we set  $k$ ,  $k_{1}$ ,  $k_{2}$  to 20,30,10 for ImageNet-Dogs and 10,10,2 for other datasets, since ImageNet-Dogs is a fine-grained dataset compared with previous ones.
Table 1: Clustering result comparison (in percentage %) with the state-of-the-art methods on five benchmarks.
CIFAR-10	CIFAR-20	STL-10	ImageNet-10	ImageNet-Dogs
NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
IIC [19]	51.3	61.7	41.1	-	25.7	-	43.1	49.9	29.5	-	-	-	-	-	-
DCCM [46]	49.6	62.3	40.8	28.5	32.7	17.3	37.6	48.2	26.2	60.8	71.0	55.5	32.1	38.3	18.2
PICA [16]	56.1	64.5	46.7	29.6	32.2	15.9	-	-	-	78.2	85.0	73.3	33.6	32.4	17.9
SCAN [42]	79.7	88.3	77.2	48.6	50.7	33.3	69.8	80.9	64.6	-	-	-	-	-	-
NMM [8]	74.8	84.3	70.9	48.4	47.7	31.6	69.4	80.8	65.0	-	-	-	-	-	-
CC [23]	70.5	79.0	63.7	43.1	42.9	26.6	76.4	85.0	72.6	85.9	89.3	82.2	44.5	42.9	27.4
MiCE [41]	73.7	83.5	69.8	43.6	44.0	28.0	63.5	75.2	57.5	-	-	-	42.3	43.9	28.6
GCC [56]	76.4	85.6	72.8	47.2	47.2	30.5	68.4	78.8	63.1	84.2	90.1	82.2	49.0	52.6	36.2
TCL [24]	81.9	88.7	78.0	52.9	53.1	35.7	79.9	86.8	75.7	87.5	89.5	83.7	62.3	64.4	51.6
IDFD [40]	71.1	81.5	66.3	42.6	42.5	26.4	64.3	75.6	57.5	89.8	95.4	90.1	54.6	59.1	41.3
TCC [38]	79.0	90.6	73.3	47.9	49.1	31.2	73.2	81.4	68.9	84.8	89.7	82.5	55.4	59.5	41.7
ProPos [17]	85.1	91.6	83.5	58.2	57.8	42.3	75.8	86.7	73.7	89.6	95.6	90.6	73.7	77.5	67.5
DivClust [26]	72.4	81.9	68.1	44.0	43.7	28.3	-	-	-	89.1	93.6	87.8	51.6	52.9	37.6
BYOL [11]	79.4	87.8	76.6	55.5	53.9	37.6	71.3	82.5	65.7	86.6	93.9	87.2	63.5	69.4	54.8
CoNR (Ours)	86.7	93.2	86.1	60.4	60.4	44.3	85.2	92.6	84.6	91.1	96.4	92.2	74.4	79.4	66.7
4.3 Comparison with the State-of-the-Art
Comparison Settings. In this section, we compare our methods ConNR with the current state-of-the-art methods, as shown in Table 1. Specifically, IIC, DCCM, and PICA are clustering methods without contrastive learning. SCAN and NMM are multi-stage methods that require pre-training and fine-tuning. DivClust is a recently proposed method based on consensus clustering by controlling the degree of diversity, and we report their best performance among different diversity levels in Table 1 for comparison. The other methods mostly learn representations based on contrastive learning or non-contrastive learning in an end-to-end fashion. For these methods that learn a generic representation rather than directly output cluster assignments, we use  $k$ -means for evaluation. In particular, for ProPos, BYOL, and ConNR (ours), the produced features output by the target encoder are used for  $k$ -means for a fair comparison.
Comparison Results. Regarding the quantitative results, we achieve consistent performance gains across the board. Specifically, for relatively small datasets such as ImageNet-10 and STL-10 with 13K images, our method reaches satisfying results, which not only surpass our baseline BYOL by at least  $+10.1\%$  ACC on STL-10 in only 200 epochs, but also further achieve a new state-of-the-art performance with  $96.4\%$  ACC on ImageNet-10. For moderate-scale datasets such as CIFAR-10 and CIFAR-20 with 60K images, our method yields stable performance improvement with an average gain of  $+1.6\%$  and  $+2.6\%$  ACC compared with ProPos [17] on CIFAR-10 and CIFAR-20, respectively. More importantly, for the challenging fine-grained dataset ImageNet-Dogs, we achieve  $+6.2\%$  and  $+1.9\%$  performance boosts in terms of ACC on our baseline and the most advanced competitors. We ascribe this significant improvement to the superiority of our ConAff neighborhood, especially on the fine-grained task. Additionally, we provide results on Tiny ImageNet and performance comparisons in Appendix A.3.
In this section, we perform a comprehensive ablation experiment in Table 2 to validate the effectiveness of our proposed method. Specifically, InC represents the baseline, CnAffN, OBD, PR are techniques proposed by our method, and LN is a standard choice used for performance comparison.
Validation of Contextually Affinitive Neighborhood. We make the following observations: First, our proposed contextually-affinitive neighborhood (ConAffN) consistently outperforms the local neighborhood approach, irrespective of whether a progressive boundary filtering strategy is utilized or not. However, ConAffN achieves its peak performance when both online boundary filtering and progressive relaxation are utilized. Instead, the local neighborhood meets with performance saturation when combined with progressively relaxed boundary filtering. Also, as illustrated by the purity curve in Figure 2, ConAffN identifies more reliable neighbors, particularly when the value of K is larger.
Validation of Progressive Boundary Filtering with Relaxation. As previously discussed, both LN and ConAffN combined with OBD experience consistent performance gains, with  $+1.9\%$  and  $+1.4\%$ ,
Table 2: Ablation Results (in percentage %) on CIFAR-10. (InC: Instance-aware Concordance, LN: Local Neighborhood, CnAffN: Contextually Affinitive Neighborhood, OBD: Online Boundary Detection, PR: Progressive Relaxation)
InC	LN	CnAffN	OBD	PR	CIFAR-10
NMI	ACC	ARI
✓	79.4	87.8	76.6
✓	✓	81.9	89.6	78.7
✓	✓	84.6	91.1	82.4
✓	✓	✓	83.9	91.1	82.3
✓	✓	✓	✓	84.8	91.6	84.2
✓	✓	✓	85.7	92.5	85.1
✓	✓	✓	✓	86.7	93.2	86.1
Neighborhood Purity Curve
Figure 2: The curve of the purity (in percentage  $\%$  ) with the changes of K.
respectively. We speculate that the boundary filtering strategy uniformly improves the robustness of neighborhood clustering. Moreover, the online boundary detection exhibits a much cleaner neighborhood (Figure 2), which supports our motivation. Despite the efficacy of OBD, we find progressive relaxation (PR) drives the model towards less biased clustering as it gradually involves the difficult samples for clustering, which further improves the ACC by  $+0.7\%$ .
(a)
Figure 3: (a) Performance comparison with different initial fraction ratios on CIFAR-10. (b) Performance with a different selection of  $k_{1}$ ,  $k_{2}$  on ImageNet-Dogs. (c) Clustering performance comparison with ConNR and BYOL on ImageNet-Dogs.
More Discussions. We conducted an analysis of three parameters:  $k_{1}$  and  $k_{2}$ , which represent the number of neighbors used to construct the graph, and  $fr$ , the fraction ratio set at the initial training stage. Figure 3(a) presents the results with varying  $fr$  values. The outcomes exhibit relative stability within the range of [0.70, 0.80, 0.90] for  $fr$ , however, a noticeable performance decline is observed when  $fr = 1$ . This indicates a lack of boundary filtering applications. We provide the results with different choices of  $k_{1}$  and  $k_{2}$  in Figure 3(b). For non-aggressive strategy where  $k_{1}$  and  $k_{2}$  are smaller than 10, there are consistent gains compared with the baseline, while for more aggressive choices where  $k_{1}, k_{2}$  are larger, the performance can be improved even further, validating the design of contextually affinitive neighborhood. In Figure 3(c), we present the ACC curves generated during the clustering process. Here, we can see that ConNR not only converges considerably faster than BYOL, but also provides notable improvements when implemented as an add-on module. A detailed exploration of how ConNR benefits other self-supervised benchmarks is provided in Appendix A.2.
5 Conclusion and Limitations
In this paper, we proposed a novel method to improve deep clustering in self-supervised learning by promoting the semantic structure preserved by neighborhood consistency. Our approach, the Contextually Affinitive (ConAff) Neighborhood, employs an efficient online re-ranking process to mine more informative neighbors and encourages cross-view neighborhood consistency. We also
introduced a progressively relaxed boundary filtering strategy to mitigate the intrinsic neighborhood noises near cluster boundaries. Our method outperformed state-of-the-art methods on several popular benchmarks and could be easily integrated into generic self-supervised frameworks. One potential limitation of our approach is that ConNR generally assumes the labels induce an equipartition of the whole data and thus does not contain specific mechanisms for handling unbalanced or long-tailed datasets. We believe that our approach can be further extended for this purpose. While our major goal in this paper is dedicated to learning a clustered and well-separated representation for deep clustering, the proposed ConAff neighborhood s a generic design choice that could be viewed from a broader perspective, hopefully benefiting the task of self-supervised learning that focuses more on downstream applications.
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (21YF1429400, 22YF1428800), Shanghai Local College Capacity Building Program (23010503100), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
[1] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations (ICLR), 2019.
[2] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[4] Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In international Conference on Machine Learning (ICML), 2020.
[6] Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 1995.
[7] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[8] Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[9] Jeff Donahue, Philipp Krajhenbuhl, and Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations (ICLR), 2017.
[10] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[11] Jean-Bastien Grill et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[47] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[48] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In international Conference on Machine Learning (ICML), 2016.
[49] Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[50] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[51] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[52] Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmoid record, 1996.
[53] Xuanmeng Zhang, Minyue Jiang, Zhedong Zheng, Xiao Tan, Errui Ding, and Yi Yang. Understanding image retrieval re-ranking: A graph neural network perspective. arXiv preprint arXiv:2012.07620, 2020.
[54] Liang Zheng, Yi Yang, and Qi Tian. Sift meets cnn: A decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017.
[55] Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[56] Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Xian-Sheng Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[57] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.