SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Motivated by the strong  $k$ -NN performance of CARP and following the previous evaluation protocol by Caron et al. [8], we assess the performance of ImageNet pre-trained CARP encoders on image retrieval and copy detection downstream tasks. We took the officially released weights from the competing methods, used the frozen encoders as feature extractors, and performed retrieval using  $k$ -NN. For both tasks, the nearest neighbor computation is done on the 2048-dim representation from the ResNet-50 encoder. We report the top-3 best-performing methods. For an extended evaluation, see Appendix A.2.
Table 1: Transfer learning evaluation. We report top-1 accuracy ( $k = 20$ ) for individual datasets and averages over all datasets for  $k \in \{10, 20, 100, 200\}$ . Top performing in **bold**, top-2 underlined.
Methods	Ep	Pets	Flowers	Aircraft	Cars	Country	Food	STL	GTSRB	Avg @k
results for k = 20	10	20	100
oBoW (mc) [19]	200	57.3	61.9	18.1	11.5	12.0	47.4	96.6	50.6	44.3	44.4	43.5
SeLa-v2 (mc) [1]	400	66.8	58.6	20.7	13.3	10.5	46.8	94.0	59.0	46.1	46.2	45.5
InfoMin [41]	800	77.8	61.9	18.2	14.4	11.6	52.4	96.4	54.8	48.6	48.4	47.3
DeepC-v2 (mc) [5]	800	78.3	76.3	32.0	25.0	13.6	62.3	95.6	63.4	56.0	55.8	54.4
SwAV (mc) [7]	800	77.0	75.2	29.0	22.7	13.8	59.1	95.2	63.2	54.5	54.4	53.1
DINO (mc) [8]	800	80.9	81.6	35.3	30.1	14.4	62.0	95.6	62.9	57.9	57.8	56.9
Triplet [45]	980	83.5	77.7	33.4	25.2	14.1	61.5	95.6	63.5	56.5	56.8	56.2
BarlowT [48]	1000	82.9	78.8	32.7	26.3	13.3	61.4	94.8	65.6	56.8	57.0	56.4
MoCo-v3 [12]	1000	86.4	79.0	36.9	29.3	12.4	60.0	96.7	72.8	59.2	59.2	58.4
CARP	400	86.8	80.0	42.1	33.5	12.3	58.4	95.9	75.3	60.4	60.5	59.7
CARP (mc)	400	83.9	80.3	34.8	27.1	14.2	62.9	95.5	62.8	57.6	57.7	56.8
Table 2: Clustering evaluation. We report (NMI) normalized mutual information, (AMI) adjusted mutual information, and (ARI) adjusted rand index. Top performing in bold, top-2 underlined.
Method	ImageNet-1M	CIFAR-10	CIFAR-100	GTSRB
NMI	AMI	ARI	NMI	AMI	ARI	NMI	AMI	ARI	NMI	AMI	ARI
PCL v2 [29]	69.7	47.5	22.2	46.7	46.6	34.8	49.1	42.2	17.1	44.1	44.1	13.0
SeLa-v2 [1]	68.7	45.5	21.3	42.0	41.9	30.6	49.7	42.9	18.2	45.7	43.2	12.0
DeepC-v2 [5]	69.7	47.1	22.4	47.0	46.9	35.5	53.2	46.7	21.8	48.1	45.7	13.5
SwAV [7]	68.5	45.1	20.5	46.8	46.8	37.0	52.1	45.5	20.0	51.0	48.8	15.0
DINO [8]	69.2	46.2	21.7	39.6	39.5	28.0	47.6	40.4	16.2	52.0	49.8	15.4
MIRA [28]	68.9	45.7	21.2	39.5	39.4	28.8	49.0	42.1	17.6	51.6	49.4	15.8
CoKe [33]	68.9	45.6	21.3	45.9	45.8	34.2	51.9	45.2	19.5	49.4	47.1	13.7
CARP	70.3	48.0	23.9	49.0	48.9	38.7	54.5	48.2	23.1	54.8	52.7	19.6
Image retrieval. We consider the revisited Oxford and Paris landmark image retrieval datasets [34]. Given a query image of a landmark, the objective is to retrieve all database images depicting the same landmark. Each dataset contains three different difficulty levels. In Table 3, we report the mean Average Precision (mAP) on the Medium and Hard subsets for various SSL algorithms. CARP's representations perform well on Oxford 5k and take second place for Paris 6k. See Appendix D.5 for the evaluation protocol.
Copy detection. We benchmark self-supervised ResNet-50 encoders on the INRIA Copydays dataset [16]. In Table 4, we report the mean Average Precision (mAP) on the "strong" subset and compare CARP's performance against other state-of-the-art methods.
Table 3: Image retrieval evaluation. We report mAP on the revisited Oxford and Paris for the (M) Medium and (H) Hard subsets.
Method	ep	ROx	Rpar
M	H	M	H
Supervised [35]	100	49.8	8.5	74.0	52.1
Random	-	1.6	0.7	4.1	2.5
DINO (mc) [8]	800	35.4	11.1	55.9	27.5
Triplet [45]	980	35.3	12.0	58.2	28.7
MoCo-v3 [12]	1000	33.1	10.9	59.1	31.3
CARP	200	38.8	15.5	58.8	30.4
Table 4: Copy-detection evaluation. We report mAP for the Copydays dataset on the "strong" subset.
Method	Ep	mAP
Random	-	25.7
DINO (mc) [8]	800	78.8
Triplet [45]	980	81.7
VICReg [3]	1000	83.7
MoCo-v3 [12]	1000	80.6
CARP (mc)	400	84.0
Table 5: Few-shot classification on VOC07 and INat2018. We report mAP for VOC07 and top-1 accuracy for INat2018, at  $n$ , across 5 independent runs, where  $n$  denotes the number of training examples. Top performing in bold, top-2 underlined.
Method	Ep	Pascal VOC07	INat2018
n=1	n=2	n=4	n=8	n=16	full	n=1	n=2	n=4	n=8	n=16	full
PCL v2 [29]	200	47.9	59.6	66.2	74.5	78.3	85.4	1.4	1.6	2.3	2.9	4.8	2.1
DINO (mc) [8]	800	45.6	58.4	66.6	74.8	79.6	88.2	6.5	12.0	20.4	29.6	35.9	30.4
Triplet [45]	980	43.6	56.2	64.6	73.8	79.6	88.3	11.4	19.1	28.9	37.6	44.0	41.4
MoCo-v3 [12]	1000	46.6	59.6	67.0	75.4	80.2	87.4	8.1	12.2	18.5	27.2	33.5	28.0
CARP (mc)	200	46.0	58.3	66.5	75.5	79.5	88.0	8.6	14.4	23.6	32.7	38.2	33.9
400	47.1	59.8	67.3	75.8	80.0	88.2	11.5	19.6	29.6	39.1	45.1	42.6
5.4 Few-shot classification
Table 5 compares the few-shot classification performance of SSL methods on the VOC07 [17] and INat2018 [43] datasets. We train linear SVMs following Li et al. [29] and linear classifiers on top of the frozen representations from the self-supervised ResNet-50 encoders for VOC07 and INat2018, respectively.
The INat2018 dataset is especially challenging for low-shot classification. It contains 8142 classes with a long tail distribution, where the number of images per class varies between 1000 maximum and 2 minimum. CARP remains a strong feature extractor for both tasks while competitors oscillate between datasets. CARP pre-trained for 400 epochs demonstrates an efficient learning performance and wins most configurations. We report a complete evaluation with standard deviations in Appendix A.3.
5.5 Linear evaluation
Following the linear evaluation protocol proposed by Zhou et al. [49], we trained linear classifiers on top of CARP's frozen 2048-dim representations for 100 epochs.
We assess the linear evaluation performance of CARP's pre-trained representations on ImageNet-1M for three pre-training configurations, 100, 200, and 400 epochs, varying the utilization of multi-crop (mc) augmentation, Table 6. CARP surpasses SwAV on all pre-trained configurations,  $+0.4\%$  on 100 epochs,  $+0.4\%$  on 200 epochs,  $+0.4\%$  on 400 epochs w/o multi-crop, and  $+0.4\%$  on 400 epochs with multi-crop. Indeed, CARP's 400 epochs pre-trained representations perform on par with DINO and SwAV, both pre-trained for 800 epochs.
In addition, we evaluated CARP's representations using a weighted  $k$ -Nearest Neighbor ( $k$ -NN) classifier. CARP achieves better  $k$ -NN performance than SwAV ( $+1.4\%$ ), DINO ( $+0.2\%$ ), and BYOL ( $+1.1\%$ ), all trained for 800 epochs or more. These results emphasize the efficiency of the proposed random partition pretext task.
In this section, we assess whether a consistent assignment of views over random partitions benefits the learned representations and improves training stability. We ablate CARP's main hyperparameters to establish a good baseline for pre-training on ImageNet-1M. For ablations, we trained CARP using the full ImageNet-1M dataset for 50 epochs. The batch size was set to 256, the number of prototypes  $K = 65536$ , and the number of random partition blocks  $N_{\mathcal{P}} = 128$ . Hence, each block contains  $N_B = 512$  prototypes. We report results for single runs. See Appendix B for more results and Appendix C for implementation details.
6.1 Training CARP with different batch sizes
Most SSL methods [7, 8, 22, 48] report their best results when using substantially large batch sizes. In Table 7, we observe a similar pattern when training CARP. Our default configuration of 1024
Table 6: Linear evaluation. We report top-1 linear and  $k$  -NN  $\left( {k = {20}}\right)$  accuracy for the ImageNet-1M dataset.  ${}^{ \dagger  }$  Results computed by us using the officially released pre-trained models. Top performing in bold, top-2 underlined.
Method	Ep	Linear	k-NN
Supervised	100	76.5	-
SwAV (mc) [7]	100	72.1	61.8†
200	73.9	63.7†
400	74.6	65.0†
800	75.3	66.3†
DINO (mc) [8]	800	75.3	67.5
BYOL [22]	1000	74.3	66.6
MoCo-v3 [24]	1000	74.6	68.9†
CARP	400	73.0	67.6
CARP (mc)	100	72.5	63.5
200	74.2	66.5
400	75.3	67.7
Table 7: CARP learns better representations when larger batch sizes (bs) are employed.
bs	128	256	512	1024	2048	4096
k-NN	46.56	51.32	54.23	56.63	57.0	58.5
Table 8: Exploring different strategies to create the partitions.
Epochs	25	50	75	100
Constant	45.15	49.89	52.81	53.32
Random	48.68	53.98	55.93	56.38
observations yields a  $k$ -NN top-1 performance  $10\%$  higher than a batch size of 128. Table 7 confirms that training with large batch sizes benefits the learned representations. However, training with smaller batch sizes requires further tuning of other hyperparameters, such as the block size  $N_B$ . Specifically, we observed that reducing the block size  $N_B$  improves the learned representations when training with small batch sizes, which makes CARP robust to low-resource training.
6.2 Exploring different strategies to build partitions
Table 8 explores different ways of creating random partition blocks from the learnable prototypes. CARP's default strategy recreates the random partitions at every training step. In other words, for each iteration, we assign  $N_B$  randomly chosen prototypes to the  $N_{\mathcal{P}}$  partition blocks. Table 8 contrasts CARP's default strategy with one in which the partition blocks are created only once, in a sequential manner, and kept fixed throughout training. We observe that training CARP with fixed partition blocks still produce useful representations. However, as measured by  $k$ -NN performance, randomly recreating the partition blocks at each iteration further benefits the learned representations. Since the partition blocks are randomly recreated at every iteration of gradient descent, the classification subproblems are always different. In practice, this variance allows for many unique pretext tasks at each iteration, which provides a positive regularization effect on CARP.
Even though CARP's representations transfer well to many downstream tasks, our experiments showed that CARP's representations do not transfer well to dense prediction tasks such as detection and segmentation. We hypothesize this limitation is due to the architectural characteristics of ConvNets that collapse local feature maps to an average global representation in the last layer, combined with our classification-like loss function. Refer to Appendix A.4 for quantitative results.
We presented consistent assignment of views over random partitions (CARP), a self-supervised clustering-based method for visual feature learning. CARP learns prototypes in an online fashion end-to-end using gradient descent by minimizing a cost function that optimizes consistency between views' assignments and uniform distribution across prototypes within a random partition. Our experiments demonstrate that posing the optimization problem at the level of random partitions of learnable prototypes stabilizes training by avoiding trivial solutions in joint-embedding architectures and increases the performance of the learned representation. We compared the performance of CARP against the state-of-the-art ResNet-based SSL methods across multiple pretext tasks and datasets. The results demonstrated that the representations learned by CARP performed well on many visual downstream tasks.
The computations were performed in part on resources provided by Sigma2—the National Infrastructure for High Performance Computing and Data Storage in Norway—through Project NN8104K. This work was funded in part by the Research Council of Norway, through its Centre for Research-based Innovation funding scheme (grant no. 309439), and Consortium Partners. This study was financed in part by the Coordenação de Aperfeicoamento de Pessoal de Nível Superior—Brasil (CAPES)—Finance Code 001
[1] Y.M. Asano, C. Rupprecht, and A. Vedaldi. Self-labelling via simultaneous clustering and representation learning. In Inter. Conf. Learn. Represent. (ICLR), 2020.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 8443-8452, 2021.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In Inter. Conf. Learn. Represent. (ICLR), 2022.
[4] Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a “siamese” time delay neural network. In Adv. Neural Inf. Process. Sys. (NeurIPS), pages 669–688. World Scientific, 1993.
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conf. Comput. Vis. (ECCV), pages 132-149, 2018.
[6] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pretraining of image features on non-curated data. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 2959-2968, 2019.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 9650-9660, 2021.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter. Conf. Mach. Learn. (ICML), 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 15750-15758, 2021.
[45] Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 9505-9515, 2021.
[46] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 3733-3742, 2018.
[47] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
[48] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Inter. Conf. Mach. Learn. (ICML), 2021.
[49] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In Inter. Conf. Learn. Represent. (ICLR), 2022.
A Extended results
In this section, we expand the evaluation experiments from the main text. We use the officially released pre-trained methods across all benchmarks to represent each SSL method. We did not retrain/reimplement any competing method. In total, we pre-trained 4 instances of CARP. To evaluate CARP's performance in a low training regime, we trained two 200-epoch models, one with multi-crop (mc) and the other without it. Similarly, to evaluate longer training performance, we trained two 400-epoch models, one with multi-crop and the other without it.
A.1 Transfer learning evaluation
In Table A.1, we report detailed results for the transfer learning  $k$ -NN experiments. We evaluate SSL methods for values of  $k \in \{10, 20, 100, 200\}$ , including all instances of CARP. Cf. Appendix D.3 for the evaluation protocol.
CARP achieved either top-1 or top-2 performance in seven out of 8 datasets. STL-10 is the only dataset where CARP is neither the top-1 nor top-2. Moreover, in the FGVCAircraft, Stanford Cars, and GTSRB datasets, CARP archived top-1 and top-2 performances with large margins. The average  $k$ -NN performance per value of  $k$  across all datasets is reported in Table 1 in the main text.
A.2 Image retrieval and copy detection
Tables A.2 and A.3 show additional results for image retrieval and copy detection evaluations. For both tasks, we compare CARP's performance against nine SSL methods. We report mAP on the Medium and Hard splits of the revisited Oxford and Paris datasets for image retrieval. We report mAP on the "strong" set of the Copydays dataset for copy detection.
A.3 Few-shot evaluation
Table A.4 shows additional few-shot evaluation results on the Pascal VOC07 and INat-2018 datasets. We report mAP and top-1 accuracy @k, averaged over 5 independent runs, for VOC07 and INat-2018, respectively. We include CARP's 200 and 400 multi-crop models. Cf. Appendix D.2 for details on the evaluation protocol.
A.4 Dense prediction evaluation
We evaluated our CARP's multi-crop model on the object detection downstream tasks using the Pascal VOC07 dataset. We followed the guidelines from He et al. [24]. We fine-tuned for 24k iterations on PASCAL VOC trainval07+12 and evaluated on test2007. CARP performs at AP=44.2, AP50=79.4, and AP75=47.6; results are averaged over 5 trials. Compared to other SSL methods, such as MoCo-v2 [11] (AP=57.4 AP50=82.5 AP75=64.0), CARP underperforms significantly.
Due to a limited execution budget, the ablations and the main experiments differ slightly in some hyperparameters. Here, we describe the configurations used for the ablations—for the main experiments, see Appendix C.1. For ablations, we trained CARP using the full ImageNet-1M dataset for 50 epochs. The projection head learns a latent representation of 128-dim. The batch size is set to 256 observations, and the projection head hidden layers contain 2048 neurons. We set the number of learnable prototypes  $K = 65536$ , and the number of random partition blocks  $N_{\mathcal{P}} = 128$ . Hence, each block contains  $N_B = 512$  prototypes. We report results for single runs.
B.1 Does the number of learnable prototypes affect the learned representations?
Table B.1 examines the effect of training CARP with different configurations of prototypes  $K$ . Similar to other clustering-based SSL methods [7, 8, 29], CARP also benefits from over-clustering. As the number of trainable prototypes  $K$  grows, the  $k$ -NN performance of the learned representations increases. In addition, note that if the number of prototypes  $K$  is smaller than the number of actual
Table A.1: Transfer learning evaluation. We compare CARP's performance against nine SSL methods on eight datasets. We report results for  $k \in  \{ {10},{20},{100},{200}\}$  . Top methods in bold, top-2 underlined. Methods with (mc) use multi-crop.
Method	ep	Oxford-IIIT Pet	Oxford Flowers-102	FGVCAircraft	Stanford Cars
10	20	100	200	10	20	100	200	10	20	100	200	10	20	100	200
oBoW (mc) [18]	200	55.8	57.3	57.2	57.5	63.5	61.9	58.9	59.4	19.1	18.1	15.8	14.9	11.9	11.5	10.9	10.5
SeLa-v2 [1]	400	66.7	66.8	65.8	66.0	59.5	58.6	56.8	57.2	21.3	20.7	18.1	16.4	13.3	13.3	13.5	13.4
InfoMin [41]	800	78.0	77.8	76.6	76.2	63.6	61.9	60.1	60.8	18.9	18.2	15.8	13.4	14.7	14.4	13.2	12.9
SWAV (mc) [7]	800	77.2	77.0	74.9	74.5	76.4	75.2	73.7	74.6	29.6	29.0	27.5	25.1	22.7	22.7	21.8	21.0
DINO (mc) [8]	800	81.5	80.9	79.0	78.9	82.3	81.6	80.8	81.2	36.1	35.3	33.5	31.1	30.0	30.1	28.9	27.5
DeepC-v2 (mc) [5]	800	79.0	78.3	76.3	75.4	78.3	76.3	75.3	76.0	32.5	32.0	28.9	26.5	25.2	25.0	23.4	22.1
Triplet [45]	980	83.3	83.5	82.4	82.4	78.5	77.7	76.9	77.3	33.3	33.4	31.7	29.6	24.4	25.2	25.5	25.0
MoCo-v3 [12]	1000	86.6	86.4	85.8	85.8	79.8	79.0	78.3	78.6	37.7	36.9	33.5	32.1	28.6	29.3	28.4	27.2
BarlowT [48]	1000	82.5	82.9	82.2	82.3	79.8	78.8	77.9	78.1	32.9	32.7	30.6	29.2	25.9	26.3	26.1	25.2
CARP	200	86.8	86.8	86.2	85.9	79.0	78.2	77.4	77.7	40.5	38.9	36.0	34.4	29.4	29.8	29.6	29.6
400	86.4	86.8	86.2	86.0	81.0	80.0	79.3	79.6	42.5	42.1	39.3	38.6	32.6	33.5	32.6	31.5
CARP (mc)	200	78.7	78.7	77.1	76.8	80.6	79.7	78.7	78.8	35.7	35.0	32.4	30.6	26.4	26.6	25.4	24.3
400	83.9	83.9	83.6	83.2	81.4	80.3	79.0	79.6	35.2	34.8	32.4	30.7	26.6	27.1	26.1	24.9
Country-211	Food-101	STL-10	GTSRB
Method	ep	10	20	100	200	10	20	100	200	10	20	100	200	10	20	100	200
oBoW (mc) [18]	200	11.7	12.0	11.8	11.4	45.8	47.4	47.3	46.0	96.6	96.6	96.3	95.7	50.1	50.6	49.9	48.2
SeLa-v2 [1]	400	10.1	10.5	11.0	11.0	45.7	46.8	46.6	45.5	94.0	94.0	93.5	93.3	58.1	59.0	58.7	57.9
InfoMin [41]	800	11.2	11.6	11.9	11.9	51.5	52.4	51.4	49.5	96.5	96.4	96.2	96.0	54.9	54.8	53.5	52.3
SwAV (mc) [7]	800	13.6	13.8	13.1	12.7	57.9	59.1	58.2	57.0	95.5	95.2	94.0	93.0	62.9	63.2	61.6	60.0
DINO (mc) [8]	800	14.1	14.4	14.2	13.6	60.9	62.0	61.4	60.0	95.9	95.6	94.7	93.8	62.7	62.9	62.6	61.8
DeepC-v2 (mc) [5]	800	13.3	13.6	12.5	12.1	61.2	62.3	61.4	60.1	95.7	95.6	94.5	93.3	62.9	63.4	62.4	61.3
Triplet [45]	980	13.7	14.1	14.3	14.2	60.1	61.5	61.0	60.0	95.7	95.6	94.9	94.5	63.4	63.5	62.9	62.0
MoCo-v3 [12]	1000	12.4	12.4	13.2	13.2	59.0	60.0	59.1	57.7	96.9	96.7	96.2	95.9	72.4	72.8	72.6	71.7
BarlowT [48]	1000	12.8	13.3	13.7	13.6	60.3	61.4	60.7	59.4	94.8	94.8	94.2	93.8	65.3	65.6	65.6	64.4
CARP	200	11.9	12.2	12.7	12.8	57.4	58.4	57.7	56.3	95.5	95.5	94.6	93.9	73.1	73.7	73.5	72.6
400	11.9	12.3	12.8	12.8	57.6	58.4	57.6	56.3	96.1	95.9	95.0	94.3	74.7	75.3	75.2	74.4
CARP (mc)	200	14.2	14.5	14.3	13.9	60.5	61.8	60.7	59.3	95.8	95.5	94.1	93.4	64.6	64.7	64.2	63.0
400	14.1	14.2	14.3	13.9	61.7	62.9	62.1	60.7	95.9	95.5	94.3	93.4	62.2	62.8	62.4	61.4
classes in the dataset, the  $k$ -NN performance of the learned representations degrades. Based on these experiments, we set the default number of prototypes  $K = 65536$  for the ImageNet-1M dataset.
B.2 Does the number of partition blocks matter?
To better understand the effect of the hyperparameters  $N_{\mathcal{P}}$  and  $N_B$  on the learned representations and in the training stability, the first row of Table B.2 demonstrates the performance of CARP using different configurations for the number of partition blocks  $N_{\mathcal{P}}$  and their sizes  $N_B$ . For completeness, we analyze the effect of removing the momentum encoder in Appendix B.3. We also present an ablation on the effect of the momentum update in Table B.3.
Specifically, as the partition sizes grow and the number of partition blocks  $N_{\mathcal{P}}$  decreases, the quality of the learned representations tends to decline and eventually collapse. Note that setting a partition size  $N_B = 65536$  produces a single partition block  $N_{\mathcal{P}}$  containing all prototypes. Precisely, the setup in the last row and last column of Table B.2 is equivalent to CARL [38]. It shows that a naive implementation leads to a collapsed solution, and the divide-and-conquer approach of devising random partitions from the learnable prototypes avoids such trivialities.
Note that as smaller the block size  $N_B$ , more stable the algorithm will be. However, the quality of the learned representation might decrease since the pseudo-classification tasks, posed by the random partitions, becomes easier with fewer prototypes. On the other hand, a larger block size  $N_B$  poses a more challenging consistency task at the expense of contributing to mode collapse.
For most cases, however, for block sizes ranging from  $N_B = 128$  to  $N_B = 4096$ , CARP learns useful representations and shows robustness to this hyperparameter. By default we set the partition block size  $N_B = 512$ .
Table A.2: Image retrieval evaluation. We report mAP performance of various self-supervised methods for the image retrieval downstream task on the revisited Oxford and Paris datasets. All SLL methods were pre-trained on ImageNet-1M. We used the officially released pre-trained models from respective methods for evaluation. Top-1 performers in bold, top-2 underlined.
Method	Ep	ROx	Rpar
Medium	Hard	Medium	Hard
Supervised	100	49.8	18.5	74.0	52.1
Scratch	1.6	0.7	4.1	2.5
oBoW (mc) [18]	200	20.4	4.4	40.6	16.2
SeLa-v2 (mc) [1]	400	20.1	4.9	37.1	13.6
InfoMin [41]	800	24.4	5.7	44.6	18.8
SwAV (mc) [7]	800	31.1	10.1	48.9	20.6
DINO (mc) [8]	800	35.4	11.1	55.9	27.5
DeepC-v2 (mc) [5]	800	32.6	10.9	50.0	20.2
Triplet [45]	980	35.3	12.0	58.2	28.7
VICReg [3]	1000	32.7	8.5	57.5	29.0
MoCo-v3 [12]	1000	33.1	10.9	59.1	31.3
CARP	200	38.8	15.5	58.8	30.4
400	38.9	15.1	58.5	30.2
CARP (mc)	200	32.8	10.4	53.6	24.9
400	33.7	11.6	54.0	26.5