SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A.3: Copy detection evaluation. We report mAP on the "strong" subset of the Copydays dataset and compare CARP's performance against seven SSL methods. Top-1 performers in bold, top-2 underlined.
Method	Ep	mAP
Scratch	25.7
oBoW (mc) [18]	200	61.5
SeLa-v2 (mc) [1]	400	76.6
InfoMin [41]	800	67.5
SwAV (mc) [7]	800	76.1
DINO (mc) [8]	800	78.8
DeepC-v2 (mc) [5]	800	76.0
Triplet [45]	980	81.7
VICReg [3]	1000	83.7
MoCo-v3 [12]	1000	80.6
CARP	200	82.3
400	82.6
CARP (mc)	200	80.8
400	84.0
B.3 The importance of the momentum encoder
Table B.2 contrasts CARP's joint-embedding architectures with and without a momentum encoder, which is equivalent to setting  $\eta = 0$  in the momentum encoder update equation. Different from other SSL methods [8, 22], CARP works with both setups. However, we observe that using a momentum encoder significantly boosts the performance of the learned representations. Table B.2 shows that regardless of block sizes, representations learned using a momentum encoder-based architecture consistently outperform the siamese counterpart.
B.4 Who provides the best features for downstream evaluation?
One way to understand CARP's joint-embedding architecture with a momentum encoder is through the teacher-student framework, where the momentum encoder is the teacher that guides the learning student. The addition of the momentum encoder raises the question of which module produces the
Table A.4: Few-shot classification on VOC07 and INat-2018. We report mAP at  $n$  on VOC07 and top-1 accuracy for INat-2018 across 5 independent runs, where  $n$  denotes the number of training examples. Standard deviations rounded to the first decimal place.
Method	Ep	Pascal VOC07
n=1	n=2	n=4	n=8	n=16	full
PCL v2 [29]	200	47.9 ± 4.1	59.6 ± 2.7	66.2 ± 2.2	74.5 ± 0.5	78.3 ± 0.4	85.4
SeLa-v2 (mc) [1]	400	42.0 ± 2.2	54.5 ± 3.2	62.2 ± 1.5	71.4 ± 0.5	76.9 ± 0.4	85.3
DeepC-v2 (mc) [5]	800	46.5 ± 2.4	58.4 ± 2.9	66.5 ± 1.6	74.5 ± 0.9	79.5 ± 0.4	87.6
SwAV (mc) [7]	800	42.9 ± 2.1	54.9 ± 4.4	64.0 ± 2.1	72.9 ± 1.1	78.7 ± 0.6	88.1
DINO (mc) [8]	800	45.6 ± 2.4	58.4 ± 3.2	66.6 ± 2.1	74.8 ± 0.8	79.6 ± 0.6	88.2
Triplet [45]	980	43.6 ± 3.3	56.2 ± 3.5	64.6 ± 1.8	73.8 ± 0.1	79.6 ± 0.7	88.3
MoCo-v3 [12]	1000	46.6 ± 3.7	59.6 ± 2.9	67.0 ± 2.4	75.4 ± 0.7	80.2 ± 0.6	87.4
BarlowT [48]	1000	42.6 ± 3.7	55.5 ± 3.2	63.5 ± 1.8	72.6 ± 0.1	77.6 ± 0.5	86.3
CARP (mc)	200	46.0 ± 3.2	58.3 ± 3.3	66.5 ± 2.4	75.5 ± 0.1	79.5 ± 0.6	88.0
400	47.1 ± 3.2	59.8 ± 3.2	67.3 ± 2.2	75.8 ± 1.1	80.0 ± 0.7	88.2
INat-2018
Method	Ep	n=1	n=2	n=4	n=8	n=16	full
PCL [29]	200	1.4 ± 0.1	1.6 ± 0.1	2.3 ± 0.2	2.9 ± 0.1	4.8 ± 0.1	2.1
SeLa-v2 (mc) [1]	400	2.9 ± 0.2	4.2 ± 0.1	6.3 ± 0.1	10.0 ± 0.1	13.5 ± 0.1	8.2
DeepC-v2 (mc) [5]	800	7.6 ± 0.2	13.0 ± 0.8	20.9 ± 0.5	29.6 ± 0.4	36.4 ± 0.2	32.8
SwAV (mc) [7]	800	5.3 ± 0.1	9.2 ± 0.5	15.6 ± 0.1	23.1 ± 0.2	29.4 ± 0.2	24.2
DINO [8]	800	6.5 ± 0.1	12.0 ± 0.5	20.4 ± 0.5	29.6 ± 0.3	35.9 ± 0.3	30.4
Triplet [45]	980	11.4 ± 0.2	19.1 ± 0.7	28.9 ± 0.8	37.6 ± 0.3	44.0 ± 0.1	41.4
MoCo-v3 [12]	1000	8.1 ± 0.1	12.2 ± 0.3	18.5 ± 0.3	27.2 ± 0.3	33.5 ± 0.1	28.0
BarlowT [48]	1000	8.8 ± 0.1	12.2 ± 0.5	17.2 ± 0.2	24.6 ± 0.1	30.8 ± 0.1	25.3
CARP (mc)	200	8.6 ± 0.2	14.4 ± 0.1	23.6 ± 0.3	32.7 ± 0.3	38.2 ± 0.2	33.9
400	11.5 ± 0.1	19.6 ± 0.1	29.6 ± 0.9	39.1 ± 0.3	45.1 ± 0.2	42.6
Table B.1: CARP benefits from over-clustering. Setting a small number of prototypes may hurt the learned representations.
K	1024	2048	4096	16384	65536	262144
k-NN	48.81	49.98	50.69	50.81	51.2	51.31
best representations. To answer this question, Figure B.1 explores the  $k$ -NN performance when extracting features from the momentum encoder (teacher) versus the student. We observe that teachers' representations constantly outperform the students' during training. However, by the end of the training, the student catches up with the teacher.
We train CARP on the ImageNet-1M unlabeled dataset using ResNet50 [23] encoders. We take the output representation of the last global average pooling layer (a 2048-dim vector) and project it to a 256-dim vector. Following Caron et al.'s [8] work, our MLP projection head contains 3 dense layers with batch normalization and the GELU activations. The hidden units of the projection head contain 2048 neurons. The 256-dim representation vector is fed to an assigner MLP that outputs unnormalized probabilities w.r.t. the learnable prototypes. By default, the assigner function is implemented as a linear layer and trains  $K = 65536$  prototypes. To generate the random partitions, we set the number of partitions  $N_{\mathcal{P}} = 128$ , which creates subsets containing  $N_B = 512$  randomly chosen prototypes. We use the same data augmentations proposed by Grill et al. [22] to generate synthetic views. The protocol creates three data augmentation pipelines, the first two to generate global views and the last to generate multi-crops. CARP is pre-trained with the LARS [47] optimizer, end to end, with weight decay of  $1 \times 10^{-6}$ . For models training up to 200 epochs, the learning rate starts from 0.6 and decays
Table B.2: CARP with and without a momentum encoder. Without the random partition strategy (last column), training collapses regardless of using a momentum encoder or a pure siamese architecture.
NB	32	64	128	256	512	1024	2048	4096	16384	65536
w/mom. enc.	49.56	50.75	51.19	51.20	51.32	51.06	51.31	51.08	49.67	0.11
w/o mom. enc.	48.95	49.28	48.81	47.37	46.16	44.68	44.29	44.39	47.25	0.11
Table B.3: The effect of the hyperparameter  $\eta$  on the momentum encoder updates. In the last column,  $\eta$  starts as  $\eta = 0.99$  and it is annealed to  $\eta = 1.0$  following a cosine schedule.
η	0	0.5	0.9	0.99	0.999	0.99 → 1.0
k-NN	51.0	50.2	50.3	51.1	50.1	51.3
to 0.006 with a cosine scheduling [30] without warmups. For models pre-trained for more than 400 epochs, the learning rate starts at 0.3 and decays to 0.003 using the same cosine scheduler. We train the system with a global batch size of 4096 observations. For all experiments, we used 4 A100 40GB GPUs and gradient accumulation to simulate large batch sizes. Cf. to Appendix E for a PyTorch style pseudo-code.
D Evaluation Protocols
D.1 Linear evaluation
For ImageNet-1M evaluation, we trained a linear classifier on top of the frozen representations extracted from the last average pooling layer of the ResNet50 encoder for 100 epochs, following Zhou et al.'s [49] protocol. The evaluation script performs a grid search hyperparameter tuning on the learning rate, weight decay regularization, and optimizer. For each input image, we take a random crop followed by a resize to  $224 \times 224$  and an optional horizontal flipping. For testing, images are resized to  $256 \times 256$  and center-cropped to  $224 \times 224$ .
D.2 Few-shot evaluation
We measure the few-shot learning capabilities of SSL methods on the Pascal VOC07 and iNaturalist 2018 datasets. For VOC07, we are interested in the multi-label classification performance. We closely follow Li et al.'s [29] protocol and train Linear SVNs on fixed 2048-dim representations from many SSL ResNet-50 encoders.
For INat-2018, we expand the few-shot evaluation challenge to a complex scenario containing more than 8k classes. We train linear classifiers for 20 epochs on fixed 2048-dim representations. We use
Figure B.1: During training, the representations extracted from the teacher outperform the representations from the student network.
the SGD optimizer. The learning rate starts at 0.03 and decays by a factor of 10 in epochs 12 and 18, respectively.
For both datasets, we vary the number  $n$  of labeled examples per class and report the average results across 5 independent runs.
D.3  $k$ -NN evaluation
To perform the  $k$ -NN evaluation, we use pre-trained SSL ResNet50 encoders as feature extractors to compute and store the representations from many vision datasets. Following Caron et al.'s [8] setup, the representation vector for a test image is compared against all representations from the training split and a prediction is made via weighted voting. If one of the closest neighbors has the same class as the test image, it contributes to the final voting as  $\alpha_{i} = \exp \left(\frac{M_{i}z}{\tau}\right)$  where  $M$  is a memory bank containing representations from the training data,  $z$  is the representation from the test data, and  $\tau$  is the temperature hyper-parameter. For all experiments, we run  $k$ -NN with configurations of  $K_{\mathrm{near}} \in \{10, 20, 100, 200\}$ . For most experiments, a value of  $k = 20$  is consistently the best setup across all methods.
D.4  $k$ -means evaluation
Similar to  $k$ -NN evaluation, we take self-supervised pre-trained encoders and use them to extract 2048-dim feature vectors from the training set of datasets like CIFAR-10/100 and ImageNet-1M. For ImageNet-1M, we use only  $10\%$  of the training data following the  $10\%$  subset from Chen et al. [9]. We fit  $k$ -means classifiers on the learned representations of the training set and use the validation split to assess the quality of the learned prototypes. We report three metrics to assess clustering performance: Normalized Mutual Information (NMI), Adjusted Mutual Information (AMI), and Adjusted Rand Index (ARI). The number of prototypes  $k$  is set to be the number of true classes of each dataset. We use the faiss library [25] for fast  $k$ -means. For each experiment, we run  $k$ -means for 100 iterations, 20 redos, and spherical normalization. To measure the clustering performance of CARP, we observed that a 400 epoch model with a learning rate of 0.3 slightly outperformed the other instances; therefore, we use this model to report results in Table 2. This instance of CARP uses two views and is only used for clustering evaluation.
D.5 Image retrieval evaluation
We strictly follow the evaluation script eval_image_retrieval.py provided by Caron et al. [8], for the image retrieval evaluation task. The script uses the revisited Oxford and Paris image retrieval datasets [34]. The dataset contains three protocols of varying difficulty levels. We take the ImageNet pre-trained ResNet-50 encoder from CARP, freeze the weights, and apply  $k$ -NN evaluation directly to the frozen 2048-d features for retrieval, conditioned on a query image.
D.6 Copy detection evaluation
We strictly follow the evaluation script eval_copy_detector.py provided by Caron et al. [8] for copy detection evaluation. The evaluation is performed on the INRIA Copydays dataset [16]. The dataset contains holiday pictures in the format query/database. Each image has suffered three kinds of artificial attacks: JPEG, cropping, and "strong." We report performance evaluation on the "strong" subset. Images in the "strong" subset were intentionally distorted by blur, insertions, print, and scan. The task is to recognize these images despite distortion. We take the frozen CARP ResNet-50 encoder and extract 2048-dim vectors from query and database images at resolution  $320^2$ . Then, we perform copy detection with cosine similarity between query and database features. We report mean average precision (mAP) as a performance metric. Unlike the benchmark described by Caron et al. [8], we do not utilize additional distractors, nor do we centralize the data using statistics learned in a different set on images.
E Pseudocode of CARP in a PyTorch-like Style
NB: number of random prototypes within a block
K: number of prototypes
NP: number of blocks in the partition, i.e. K // NB # N: batch size
for x1, x2 in loader: # student and teacher branches z1, w1 = enc(x1), mom_enc(x1) # [N, K] z2, w2 = enc(x2), mom_enc(x2) # [N, K] s_logits, t_logits = [z1, z2], [w1, w2]
sample cluster indices with no replacement rand.proto_ids = multinomial(ones(K), K, False) split.proto_ids = stack(split(rand.proto_ids, NB))  
preds_list, targets_list = [], []
for s_log, t_log in zip(s_logits, t_logits):
    preds = get_logits_gr(s_log, splitProto_ids)
    targets = get_logits_gr(t_log, splitProto_ids)
    preds_list.append(preds)
    targets_list.append(target)
loss = loss_fn(preds_list, targets_list)
def loss_fn(s_list, t_list):  
    c_loss = consistency_loss(s_list[0], t_list[1])  
    c_loss += consistency_loss(s_list[1], t_list[0])  
    s = cat(s_list, dim=1)  
    t = cat(t_list, dim=1)  
    probs = cat([s, t], dim=1).transpose(0, 1)  
    e_loss = kl_div(mean(probs, dim=0))  
    return c_loss + e_loss
def consistency_loss(s, t):
    loss = einsum("knc,knc->kn", [s, t])
    return -log(loss).mean()
def kl_div(p):
    return mean(log(K) + sum(p * log(p), dim=-1))
def get_logits_gr(logits, proto_ids):  
    logits_gr = logits[:, proto_ids_FLatten()]  
    logits_gr = logits_gr.split(NB, dim=1)  
    logits_gr = stack(logits_gr, dim=0)  
return softmax(logits_gr, dim=-1) # [NP, N, NB]