Representation Learning via Consistent Assignment of Views over Random Partitions
Authors: Thalles Santos Silva
Adín Ramírez Rivera
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
unsupervised learning
self-supervised learning
computer vision
TL;DR: CARP is a self-supervised method for representation learning of visual features. We evaluate CARP's representations in 17 datasets across many standard protocols. We compare CARP performance to 11 existing self-supervised methods.

Abstract: We present Consistent Assignment of Views over Random Partitions (CARP), a self-supervised clustering method for representation learning of visual features. CARP learns prototypes in an end-to-end online fashion using gradient descent without additional non-differentiable modules to solve the cluster assignment problem. CARP optimizes a new pretext task based on random partitions of prototypes that regularizes the model and enforces consistency between views' assignments.  Additionally, our method improves training stability and prevents collapsed solutions in joint-embedding training. Through an extensive evaluation, we demonstrate that CARP's representations are suitable for learning downstream tasks. We evaluate CARP's representations capabilities in 17 datasets across many standard protocols, including linear evaluation, few-shot classification, $k$-NN, $k$-means, image retrieval, and copy detection. We compare CARP performance to 11 existing self-supervised methods. We extensively ablate our method and demonstrate that our proposed random partition pretext task improves the quality of the learned representations by devising multiple random classification tasks.
In transfer learning tasks, CARP achieves the best performance on average against many SSL methods trained for a longer time.

Representation Learning via Consistent Assignment of Views over Random Partitions
Thalles Silva
Adín Ramírez Rivera Department of Informatics University of Oslo adinrQuio. no
Institute of Computing
University Campinas
thalles. silva@studenta.. ic. unicamp br
Abstract
We present Consistent Assignment of Views over Random Partitions (CARP), self-supervised clustering method for representation learning of visual features. CARP learns prototypes an end-to-end online fashion using gradient descent without additional non-differentiable modules to solve the cluster assignment problem. CARP optimizes new pretext task based on random partitions of prototypes that regularizes the model and enforces consistency between views' assignments. Additionally, our method improves training stability and prevents collapsed solutions in joint-embedding training. Through an extensive evaluation, we demonstrate that CARP's representations are suitable for learning downstream tasks. We evaluate CARP's representations capabilities in 17 datasets across many standard protocols, including linear evaluation, few-shot classification, j-NN, k- means, image retrieval, and copy detection. We compare CARP performance to 11 cxisting self-supervised methods. We extensively ablate our method and demonstrate that proposed random partition pretext task improves the quality the learned representations by devising multiple random classification tasks. transfer learning tasks, CARP achieves the best performance average against many SSL methods trained for a longer time.
Introduction
Learning from unlabeled data has been of the main challenges in computer vision. Recent approaches based on self-supervised learning (SSL) have significantly reduced the gap between supervised and unsupervised pre- pre-trained representations. Nowadays, self-supervised pre-training vast quantities of unlabeled data, prior to learning a downstream supervised task of interest, can be more effective than supervised pre-training for many tasks [7, 19, 22].
Current SSL methods can be divided into two classes: (1) self-supervised embedding prediction [9, 24, 39, 41] and (2) clustering [1, 5, 6, 29]. As the name suggests, embedding prediction methods work directly in the representation space. They are trained with either contrastive or non non-contrastive loss functions. Instead of reconstructing the input signal, their loss function maximizes agreement between embeddings of the same view and optionally pushes representations from different views apart. On the other hand, clustering methods discretize the representation space by learning finite set of prototypes. These prototypes aggregate representations from different images that are similar enough to be assigned together. Nevertheless, recent SSL methods build joint- ceint-embedding architecture that can be pure siamese [4] or use different encoders.
Code httts:://staalllet wwwpshhhwllee....hhbaeeeeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The most challenging and significant difference among these methods is how they avoid trivial solutions when training toint-embedding architectures. Contrastive methods avoid trivial solutions by explicitly pushing representations of negative samples away from the anchor representation, while non-contrastive methods avoid trivial solutions by regularization or architectural designs [10, 22].
Among self-supervised clustering methods, recent work proposed avoiding trivial solutions by using non-differentiable modules such as the Sinkhorn- Knopp algorithm [l. 7, 14] and classic machine learning methods [29, 42] such as k-Means clustering or k-Nearest Neighbor, to solve the cluster assignment problem.
Consistent assigmments [37, 38] were recently proposed as a way to learn prototypes to improve the representations. Similarly, our learning objective imposes two constraints (1) consistent assignment of views over learnable prototypes and (2) a uniform distribution for the average predictions within a batch. However, we show that such strategy does not scale well to enormous datasets containing millions of classes. In such situations, we need to model large numher of prototypes while enforcing consistent assignment between views and avoiding collapsed solutions. We show that naive implementation of this strategy makes the learning problem challenging from a training stability perspective, where the model quickly settles for trivial solution by assigning all views' embeddings to the same prototype.
To overcome these issues, propose novel sulf-super approach based the consistent assigmment of views over random partition sets (CARP). We train CARP to minimize a consistency loss, which encourages the model to assign different views of the same unlabeled example to the same prototype. We solve the dimensionality problem by enforcing smaller problems through the introduction of random partitions that enforce consistency and regularize the model. The energy between the views' representations and the trainable prototypes (within random partitions) allows us to automatically bootstrap predictions and targets our consistency loss. Our contributions are three-fold:
1. A novel and entirely online ciit-embedding learning strategy based on self- self-supervised clustering, see Figure 1. We propose a divide-and and---uu-- pretext task based on randomly generated partitions of learnable prototypes. Our loss function allows stable training of joint-embedding architectures in a self-supervised context.
2. A framework that simplifies self-supervised training and does not require normalization tech- niques [7, 9] or the necessity of mining negatives for contrastive training [9, 32].
3. A differentiable assigner module that generates soft pseudo-labels by comparing the representa- tions of image views to prototypes within random partition subsets. To avoid trivial solutions, we enforce the average predictions over batch to be non-informative over the set of prototypes within a random subset.
2 Related work
Self-supervised embedding prediction methods operate directly in the representation space by learning a metric such that embeddings from views of the same image are closer to one another while embeddings from views of different images are far away in the feature space. These methods can be trained using contrastive or con-contrattive loss functions. Methods that minimize a loss function with a contrastive term date back to 1990s [4, 13, 21]. They must explicitly find representations from non-correlated images to use negatives. Recent contrastive methods include InstDise [46], CPC [32], SimCLR [9] and MoCo [11, 24]. These methods learn unsupervised representations by minimizing nearly the same contrastive loss function, i.e., the InfoNCE [32]. CARP does not directly optimize the views' embeddings, nor is it a contrastive method. Instead, we learn a set of general prototypes using random partition strategy that stabilizes the learning process and avoids trivial solutions commonly found when training joint- coint-embediig SSL models.
On the other hand, non-contrastive methods work by approximating embeddings of views from the same image. The main advantage is not requiring explicit opposing representations in the loss formu- lation. To avoid trivial solutions, a common approach is to implement "stop- "roppggrddeet" operation that prevents the gradient signal from flowing to the two branches of the join-embedding architecture simultaneously. BYOL [22] and SimSiam [10] are examples of such methods. CARP takes advan- tage of non-contrastive training since it does not require mining negatives for optimization. Also, different from Grill et al. 's [22] work, CARP does not require momentum encoder, though using

---

Partition
Encoder
Projector
Assigner
Predictions
Bo
fo
99
go
LLJ.
II
dls.
EMA
EMA
EMA
$22
Targets
w
Bo
B
fc
9
Figure 1: CARP's training architecture. From two views, we train an encoder f(), followed by a projection 9()) produce representation vectors 2" and w", respective the parameters 0 and & for each branch, for each view indexed by v. The representations are fed to an assigner function (() that produces normalized distributions of views w.r.t. the learnable prototypes. Note that 0 are the trainable weights, and & are an exponential moving average of A. We create partitions by randomly arranging the prototypes into predefined number of blocks. E.g., from a total of K prototypes, we create Np 2 blocks, each containing NB = 3 prototypes. Then, we enforce consistent assignment of views over prototypes within the blocks.
significantly improves the learned representations. CARP trains a joint-embedding architecture and uses the "stop-gradd grrop--grdiett operation in conjunction with a regularized pretext task based on random partitions of prototypes to avoid mode collapse.
Self-supervised clustering methods do work directly on the views' embeddings. Instead, they learn a set of prototypes that are used to solve subsequent pretext tasks in different ways. Caron al. [5], for instance, used k-Means clustering every epoch cluster the representations from the entire dataset and produce pseudo-labely labels for the training images. Then, classifier head is trained to predict the spcudo-labels, Caron et [6] proposed a method combine the rotation prediction pretext task [18] with clustering. Li et al. [29] presented a method based on expectation-maxiiizztine maximization (EM) that merges clustering with the contrastive learning framework from He al. [24]. Recent work by Caron et al. [7] and Asano et al. [1] combine SSL. with clustering. They utilize the non- differentiable Sinkhorn- Sinkhorn-Knopp algorithm solve the cluster assignment problem without falling into collapsed solutions. Silva and Ramírez Rivera [38] proposed, CARL, an online clustering method that does not require non-differential algorithms to avoid trivial solutions.
Contrast to previous approaches. Instead of solving the cluster assignment problem using non- differentiable algorithm such as the Sinkhorn-Knopp. CARP is trained end-to-end with gradient descent. Different from Caron et al. s [8] work, our model does not require extra momentum encoders or data structures store previous predictions as way to avoid trivial solutions. Unlike CARL [37, 38], CARP avoids trivial solutions by posing the optimization problem at the level of random partitions of prototypes that solve the high-dimensionality nature of the task. Unlike Caron et al.'s [6] work, our method does not require clustering the entire dataset every epoch to generate pseudo-labels Instead, CARP generates soft pseudo- labels- an online fashion from examples in single minibatch.
Currently, the best self-supervised methods [8, 12] vision transformers [15, 44] their backbones. Recent methods [20] employ masking pretext task where some patches of the views are manually hidden, and the network is tasked infer the missing pieces. In this paper, we consider transformer- based methods to be out-of-scope of our compared backbones. Thus, we do not include them in our results to maintain fair comparison.
3 Consistent Assignment of Views
From an image Fi, we create two views, # = T(x;) and 47 = T(x;), using a stochastic function that applies set of random image transformations to z; (cf. Appendix C.1). CARP a joint- embedding architecture with two modules: differentiable (student) and non-differentiable (teacher)

---

I.I.
I.
Bo
I.
Prototypes
B
Figure 2: Instead of posing a pseudo-lassiifcattin problem overall prototypes (circles and squares), the views are assigned (colored dashed lines) to subset of prototypes (blocks), devising multiple escudo-classification problems. Then, we contrast their distributions of assignments. (For this example, K' = 8, NB = 4 with Np 2.)
branch. Each module has its own set of weights and the same architectural design. Both contain an encoder f(-) and projection head 9(-)- The differentiable student receives a pair of views and produces embedding vectors z = fo(fo((;) for v E {1, Similarly, the non-differentiable teacher produces target embeddings "." D8 (f,
The objective to learn set of prototype vectors to discretize the embedding space. These prototypes are not meant to represent the true classes of the data. Instead, they may be interpreted as anchors attract views given image commonplace in the embedding space. The function ((:-)) receives the views' representations, z! and w. E Iixdd as input and outputs normalized probability vectors relating the views' embeddings with the prototypes such that s = q<Y, C)) and f (w" C), where sy and t R are the normalized probabilities of view, 46, w.r.t. the prototypes C Kxd Note that d is the dimensionality of the embedding vector, K is the number of prototypes, and the assigner q(,, C*) = softmax (/ CT).
To avoid trivial solutions in the joint- embedding training, need a loss function that prevents the assignment of all representation vectors 2, to a unique prototype. Unlike previous work, we seek a method that solves the cluster assignment problem in an online fashion using gradient descent.
We propose a loss function composed of two terms: consistency and entropy. The consistency term learns the relations between embedding vectors and prototypes. It enforces different views of the same image to be assigned to the same prototype with high confidence. For normalized probability vectors and b, we define the consistency term as
Co(a,b) log (),b),
(I)
where - >) a dot product.
The consistency loss is optimized when the two views at! and ii are assigned to the same prototype with maximal confidence, i.e., when the probability distributions of the two views s! and sf resemble equal one-hot vectors.
If optimize the consistency loss, Les training collapses state where all views assigned to the same prototype. common approach to avoid such failure is to ensure that all the prototypes get roughly the same number of assignments. Let define the function
avg ((ar,bb))
1 I.
Σ
+ by
as the average probability across the representations within batch size L. For our distributions, we define = {s) If we maximize the entropy of the mean probabilities of batch, (Ø). we will encourage the average predictions to be closer to a uniform distribution. Previous work [2, 26, 42] has used this entropy term in various scenarios, ranging from discriminative unsupervised clustering to semi-supervised learning. In our case, this term enforces non-informative prior over the prototypes with a particular schedule learning. Thus, the final proposed objective to

---

minimize is
(o(o,))) + (o
L L- NI
where A > trades off consistency at the view level with the average uniform assignment at the batch level. Note that the contribution of the entropy term decays as training progresses.
3.1 Limitations of Consistent Assignments
The formulation of the consistent assignments (3) is similar to CARL [37, 38], except for the additional teacher stream that stabilizes training and improves performance. Nevertheless, training an unsupervised system by minimizing the loss (3) is challenging. The main limitation how to avoid trivial solutions in unsupervised training of joint-embedding architectures. For some cases, tuning the contribution of the entropy term might be enough to optimize the loss (3) stably. However, adjusting such hyperparameter is difficult because one configuration does not hold for all training conditions. For instance, if the value of Xe is too small, the consistency term wins the arms race, and the average distribution over the batch, H (p) becomes one-hot alike, i.c., all views end up assigned to the same prototype. the value of Xe is too large, the entropy term gets the upper hand, and collapse is avoided. However, the process of view assignment is neglected over the policy of distributing views uniformly, which results in poor performance of the learned representations, as shown by Table B.2.
For small number of general prototypes, training is more stable, and the model avoids collapse with a simple tuning of the Xe parameter. However, for larger number of general prototypes, stability becomes an issue. The main problem lies with the entropy term. When the distribution is larger, i.e., K > N, regular batch sizes, such as N 64 or N 128, become too small to properly model the distribution, i.e., the signal is too weak for most prototypes. Consequently, to avoid collapse, we need to increase the strength of the entropy term or increase the batch size, which in turn decreases performance.
To address such limitation, we propose to decouple the loss function (3) into smaller sub-problems. Instead of enforcing both consistency and uniform assignments over all the general prototypes, we propose a pretext task over subsets or blocks of a random partition of the general prototype set C.
Assignment based on Random Partitions
Given the set of K trainable prototypes C = f2,, Cx), we define partition of C as P {B, C. C) such that 0 & P. B. B, P where B, P. and B, n B, 0 for all B, B, E P. and i ( j. We refer to B, as block or subset of the partition. We are interested in a partition set P = {B.},r of size Np, i.e., IP| Np.
Using the concept of partition of set, blocks that satisfies the learning problem defined in Section 3. If the size of a partition block, B,, equals the number of prototypes, Ng K then the partition P is trivial, i.c., P {B1} {C). If the size of the partition blocks equals NB = 1, then we have K blocks in P, and each block has a unique prototype. Here, the learning task is equivalent multiple binary classification problems, where each output score, if normalized, expresses the likelihood of a data point x; to independently belong to each prototype.
define framework of pretext tasks over partition
However, if the block size < NB < K, and NB divides K, then the partition P will be composed of Np |//N] J blocks. We define P by randomly assigning Ng prototypes cy. for = 0, 1,...,,. NB. to each block B, = {cyls. where = 0, 0.1...., Np.
Instead of mapping a single representation 2 as linear combination of all prototypes in C, we compare the view's representations =! and w against all the prototypes in the j-th block. That is, ,Y = q(2,, B,) and f q(w,, By), for every block in the partition P. to obtain the normalized probability distribution relating view from image with the prototypes of the j-th block of the random partition, where "i,, and f E RIXIXNP.
To ensure that views are consistent among the blocks, we optimize the views' distributions si and t over the prototypes of a block indexed by j. so that the two distributions are consistent with one another. Thus, the consistency term of our partition loss is Co(st,y .. {. where for each block
5

---

B,, the loss is minimized when the pair of views, 2 and L gets assigned to the same prototypes across blocks. In other words, we look for the agreement between student and teacher assignments' probabilities across views of given sample.
Following similar reasoning, the block-wise entropy term is defined H(p)), where Pj avg( avg(((s), (i is the average prediction over each block Bj for batch of size N. Thus, the final objective for consistent assignment of random partition is,
N No
ΣΣ (Cd H(p,). NNP
1
Note that to fully use the pair of views at each iteration, we symmetrically use the L. consistency function. The probability vectors f come from the momentum teacher and are used as target distributions.
We can view the random partition pretext task as posing multiple pscudo-lassiitiatiion problems over subsets of prototypes. At each iteration, the pseuuo-classfficaioon tasks change because the partitions are recreated with different prototypes, cf. Figure 2. One of the benefits of such a strategy is that we no longer require tuning the hyperparameter Xe avoid trivial solutions. The stochastic nature of the random partition pretext task, blended with the multiple prediction tasks over subsets of prototypes, provides a regularization effect that improves the learned representations and training stability.
Other clustering- basedd methods [7, 8] rely on sharpening the distributions to improve their self- supervised signals used targets in cross-entropy loss. On the contrary, formulation does not require the temperature parameter for sharpening the predictions and guiding the learning of the student. We can think of the consistency loss implicitly learning the temperature parameter to make the predictions sharper at each iteration. This is an important advantage of our consistency loss in contrast to previous methods.
5
Main results
5.1 Transfer learning evaluation
Table shows that CARP's pre-trained representations are suitable for learning new downstream tasks over multiple datasets with distinct difficulty levels. We compare CARP's i:-NN performance against nine SSL methods across eight datasets and report average results for k= [10, 20, 100, 200), over all datasets. We advocate for k-NN instead of linear evaluation, where a linear classifier trained on top of the frozen features, for the following reasons: (1) k-NN is faster, (2) k-NN demands fewer resources, and (3) k-NN requires less hyperparameter tuning. For individual references, Table I shows top-1 I accuracy for h 20 on each dataset. For fixed k = 20, CARP outperforms competitors out of datasets, and in the Flowers and Country datasets, Moreover, the average performances at & show that CARP performs comparably well on all datasets without significant performance differencees when varying the number labeled examples k. We show the detailed results in Appendix A.
a close second.
5.2 Clustering evaluation
Table reports clustering performance metrics of various clustering-based SSL methods the ImageNet-1N 1M [36], CIFAR-10/100 [27], and the GTSRB [40] datasets. For the ImageNet- IM, only I % of the labeled data is used following the subset provided by Chen al. [9]. See Appendix D.4 for the detailed k-meaas evaluation protocol.
5.3 Image retrieval and copy detection
Motivated by the strong k-NN performance of CARP and following the previous evaluation protocol by Caron al. [8], we assess the performance of ImageNet pre-trained CARP encoders on image retrieval and copy detection downstream tasks. We took the officially released weights from the competing methods, used the frozen encoders as feature extractors, and performed retrieval using k-NN. For both tasks, the nearest neighbor computation is done on the 2048-dim representation from the ResNet-50 encoder. We report the top-3 best-performing methods. For an extended evaluation, Appendix A.2.

---

Table Transfer learning evaluation. We report top-1 accuracy 20) for individual datasets and averages over all datasets for k E {10, 20, 100, 200). Top performing in bold, top-2 underlined.
Pets Plowers Aircraft Cars Country Food STL. GTSRB
Avg @k
Methods
Ep
results for k
20
10
20
100
200
aBoW (me) [19]
200
57.3
61.9
18.1
11.5
12.0
47.4
96.6
50.6
44.3
44.4
43.5
43.0
SeLa-v2 (mc) [1]
400
66.8
58.6
20.7
13.3
10.5
46.8
94.0
59.0
46.1
46.2
45.5
45.1
InfoMin [41]
800
77.8
61.9
18.2
14.4
11.6
52.4
96.4
54.8
48.6
48.4
47.3
46.6
DeepC-v2 (me) [5]
800
78.3
76.3
32.0
62.3
63.4
56.0
55.8
54.4
SwAV (mc) [7]
800
77.0
75.2
29.0
22.7
13.8
59.1
95.2
63.2
54.5
54.4
53.1
52.2
DINO (mc) [8]
800
80.9
81.6
35.3
30.1
14.4
62.0
95.6
62.9
57.9
57.8
56.9
56.0
Triplet [45]
980
83.5
77.7
33.4
25.2
14.1
61.5
95.6
63.5
56.5
56.8
56.2
55.6
BarlowT [48]
1000
82.9
78.8
32.7
26.3
13.3
61.4
94.8
65.6
56.8
57.0
56,4
55.7
McCo-v3 [12]
1000
86.4
79.0
36.9
29.3
12.4
60.0
96.7
72.8
59.2
59.2
58.4
57.8
CARP
400
86.8
80.0
42.1
33.5
12.3
58.4
95.9
75.3
60.4
60.5
59.7
59.2
CARP (mc)
400
83.9
80.3
34.8
27.1
14.2
62.9
95.5
62.8
57.6
57.7
56.8
56.0
Table Clustering evaluation. We report (NMI) normalized mutual information, (AMI) adjusted mutual information, and (ARI) adjusted rand index. Top performing in bold, top-2 underlined.
ImageNet-IM
CIFAR-10
CIFAR-100
GTSRB
Method
NMI AMI
ARI
NMI
AMI
ARI
NMI
AMI
ARI
NMI
AMI
ARI
PCL v2 [29]
69.7
47.5
22.2
46.7
46.6
34.8
49.1
42.2
17.1
44.1
44.1
13.0
SeLa-v2 [1]
45.5
21.3
42.0
41.9
30.6
49.7
42.9
18.2
45.7
43.2
DeepC-v2 [5]
69.7
47.1
22.4
47.0
46.9
35.5
53.2
46.7
21.8
48.1
45.7
13.5
SwAV [7]
68.5
45.1
20.5
46.8
46.8
37.0
52.1
45.5
20.0
51.0
48.8
15.0
DINO [8]
69.2
46.2
21.7
39.6
39.5
28.0
47.6
40.4
16.2
52.0
49.8
15.4
MIRA [28]
68.9
45.7
21.2
39.5
39.4
28.8
49.0
42.1
17.6
51.6
49.4
15.8
CoKe [33]
68.9
45.6
21.3
45.9
45.8
34.2
51.9
45.2
19.5
49.4
47.1
13.7
CARP
48.0
23.9
49.0
48.9
38.7
54.5
48.2
23.1
54.8
52.7
19.6
Image retrieval. We consider the revisited Oxford and Paris landmark image retrieval datasets [34]. Given a query image of landmark, the objective is retrieve all database images depicting the same landmark. Each dataset contains three different difficulty levels. In Table 3, we report the mean Average Precision (mAP) the Medium and Hard subsets for various SSL algorithms. CARP's representations perform well on Oxford 5k and take second place for Paris 6k. See Appendix D.5 for the evaluation protocol.
Copy detection. We benchmark self-supervised ResNet-50 encoders on the INRIA Copydays dataset [16]. In Table report the mean Average Precision (mAP) the "strong" subset and compare CARP's performance against other state-of-the-ard methods.
Table Image retrieval evaluation. We report mAP on the revisited Oxford and Paris for the (M) Medium and (H) Hard subsets.
Table 4: Copy-detection evaluation. We report mAP for the Copydays dataset on the "strong" subset.
ROx
Rpar
Method
Ep
mAP
Method
ep
M
H
M
H
Random
25.7
Supervised [35]
100
49.8
8.5
74,0
52,1
DINO (me) [8]
800
78.8
Random
1.6
0.7
4.1
2.5
Triplet [45]
980
81.7
VICReg [3]
1000
83.7
DINO (mc) [8]
800
35.4
11.1
55.9
27.5
MoCo-v3 [12]
1000
80.6
Triplet [45]
980
35.3
12.0
58.2
28.7
MoCo-v3 [12]
1000
33.1
10.9
59.1
31.3
CARP (mc)
400
84.0
CARP
200
38.8
15.5
58.8
30.4

---

Table Few-shot classification on VOC07 and INat2018. We report mAP for VOCO7 and top-1 accuracy for INat2018, at n, across 5 independent runs, where denotes the number of training examples. Top performing bold, top-2 underlined.
Pascal VOCO7
INat2018
Method
Ep
I=1
n=2
n=4
n=8
n=16
full
|==
n=2
n=4
n=8
n=16
full
PCL v2 [29]
200
47.9
59.6
66.2
74.5
78.3
85.4
1.4
1.6
2.3
2.9
4.8
2.1
DINO (mc) [8]
800
45.6
58.4
66.6
74.8
79.6
88.2
6.5
12.0
20.4
29.6
35.9
30.4
Triplet [45]
980
43.6
56.2
64.6
73.8
79.6
88.3
11.4
19.1
28.9
37.6
44.0
41.4
MoCo-v3 [12]
1000
75.4
87.4
12.2
18.5
27.2
33.5
28.0
CARP (mc)
200
46.0
58.3
66.5
75.5
79.5
88.0
8.6
14.4
23.6
32.7
38.2
33.9
67.3
75.8
80.0
88.2
11.5
19.6
29.6
45.1
42.6
5.4 Few-shot classification
Table 5 compares the few-shot classification performance of SSL methods on the VOCO7 [17] and INat2018 [43] datasets. We train linear SVMs following Li et al. [29] and linear classifiers on top of the frozen representations from the self-supervised ResNet-50 encoders for VOCO7 and INat2018 18, respectively.
The INat2018 dataset is especially challenging for low-shot classification. contains 8142 classes with long tail distribution, where the number of images per class varies between 1000 maximum and 2 minimum. CARP remains a strong feature extractor for both tasks while competitors oscillate between datasets. CARP pre-trained for 400 epochs demonstrates efficient learning performance and wins most configurations. We report a complete evaluation with standard deviations Appendix A.3.
5.5 Linear evaluation
Following the linear evaluation protocol proposed by Zhou et al. [49], trained linear classifiers top of CARP's frozen 2048-dim representations for 100 epuchs.
We assess the linear evaluation performance of CARP's pre-trained representations on ImageNet- 1M for three pre-training configurations, 100, 200, and 400 epochs, varying the utilization of multi-crop (mc) augmentation, Table 6. CARP surpasses SwAV on all pre-trained configurations, +0.4% on 100 epochs, +0.4% on 200 epochs, +0.4% on 400 epochs w/o multi-crop, and +0.4% on 400 epochs with multi-crop. Indeed, CARP's 400 epochs pre-trained representations perform on par with DINO and SwAV, both pre-trained for 800 epochs.
In addition, we evaluated CARP's representations using weighted k-Nearest Neighbor (k-NN) classifier. CARP achieves better k-NN performance than SwAV (+1.4%), DINO (+0.2%), and BYOL (+1.1%), all trained for 800 epochs or more. These results emphasize the efficiency of the proposed random partition pretext task.
Ablations
In this section, we assess whether a consistent assignment views over random partitions benefits the learned representations and improves training stability. We ablate CARP's main hyperparameters to establish a good baseline for pre- pre-training on ImageNet- 1M. For ablations, we trained CARP using the full ImageNet- 1M dataset for 50 epochs. The batch size set to 256, the number of prototypes K = 65 536, and the number of random partition blocks Np = 128. Hence, each block contains NB 512 prototypes. We report results for single runs. See Appendix B for more results and Appendix C for implementation details.
6.1 Training CARP with different batch sizes
Most SSL methods [7, 22, 48] report their best results when using substantially large batch sizes. In Table 7, observe similar pattern when training CARP. Our default configuration of 1024

---

Table Linear evaluation. We report top-1 linear and k-NN (k: 20) aceuracy for the ImageNet-
IM dataset. Results computed by us using the officially released pre-trained models. Top performing bold, top-2 underlined.
Method
Ep
Linear
K-NN
Supervised
100
76.5
SWAV (mc) [7]
100
72.1
61.8
200
73.9
63.7
400
74.6
65.0
800
75.3
66.3
DINO (mc) [8]
800
75.3
67.5
BYOL [22]
1000
74.3
66.6
MoCo-v3 [24]
1000
74.6
68.91
CARP
400
73.0
67.6
CARP (mc)
100
72.5
63.5
200
74.2
66.5
400
75.3
67.7
Table 7: CARP learns better representations when larger batch sizes (bs) are employed.
Table 8: Exploring different strategies to create the partitions.
bs
128
256
1024
2048
4096
Epochs
50
100
K-NN
46.56
51.32
54.23
56.63
57.0
58.5
Constant
45.15
49.89
52.81
53.32
Random
48.68
53.98
55.93
56.38
observations yields a k-NN top-1 performance 10% higher than batch size of 128. Table confirms that training with large batch sizes benefits the learned representations. However, training with smaller batch sizes requires further tuning of other hyperparameters, such as the block size NB- Specifically, we observed that reducing the block size NB improves the learned representations when training with small batch sizes, which makes CARP robust to low-resource training.
6.2 Exploring different strategies build partitions
Table 8 explores differennt ways of creating random partition blocks from the learnable prototypes. CARP's default strategy recreates the random partitions at every training step. In other words, for each iteration, we assign NB randomly chosen prototypes the Ny partition blocks. Table 8 contrasts CARP's default strategy with one in which the partition blocks are created only once, in a sequential manner, and kept fixed throughout training. We observe that training CARP with fixed partition blocks still produce useful representations. However, as measured by k-NN performance, randomly recreating partition blocks at each iteration further benefits the learned representations. Since the partition blocks are randomly recreated at every iteration of gradient descent, the classification subproblems are always different. practice, this variance allows for many unique pretext tasks at each iteration, which provides positive regularization effect CARP.
7 Limitations
Even though CARP's representations transfer well to many downstream tasks, our experiments showed that CARP's representations do not transfer well to dense prediction tasks such as detection and segmentation. We hypothesize this limitation is due to the architectural characteristics of ConvNets that collapse local feature maps to an average global representation in the last layer, combined with classification-lie like loss function. Refer to Appendix A.4 for quantitative results.

---

8 Conclusion
We presented consistent assignment of views over random partitions (CARP), self-supervised clustering-bace based method for visual feature learning. CARP learns prototypes in an online fashion end-to-end using gradient descent by minimizing cost function that optimizes consistency between views' assignments and uniform distribution across prototypes within a random partition. Our experiments demonstrate that posing the optimization problem at the level of random partitions of learnable prototypes stabilizes training by avoiding trivial solutions in joint-embedding architectures and increases the performance of the learned representation. We compared the performance of CARP against the state-of-th tee-oofthhe-rt ResNet-based SSL methods across multiple pretext tasks and datasets. The results demonstrated that the representations learned by CARP performed well on many visual downstream tasks.
Acknowledgements
The computations were performed in part resources provided by Sigma2-the National Infras- tructure for High Performance Computing and Data Storage in Norway- Project NN8 144.. This work was funded in part by the Research Council of Norway, through its Centre for Research- based Innovation funding scheme (grant no. 309439), and Consortium Partners. This study was financed in part by the Coordenação de Aperfeigoamento de Pessoal de Nível Superior- Brasil (CAPES)-Finane Code 001
References
[1] Y.M. Asano, C. Rupprecht, and A. Vedaldi. Self-labelling via simultancous clustering and representation learning. In Inter. Conf. Learn. Represent. (ICLR), 2020.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In JEEE Inter. Conf. Comput. Vis. (ICCV), pages 8443 8452, 2021.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Yarimace-ivvarancc-cowaaaae regular- ization for self-supervised learning. In Inter. Conf. Learn. Represent. (ICLR), 2022.
[4] Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In Adv. Neural Inf. Process. Sys. (NeurIPS), pages 669-688. World Scientific, 1993.
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. European Conf. Comput. Vis. (ECCV), pages 132-149, 2018
[6] Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- training of image features on non-curated data. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 9959-2968, 2019.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Adv. Neural Inf. Process. Sys. (NeurIPS), 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervieed vision transformers. IEEE Inter. Conf. Comput. Vis. (ICCV), pages 9650-9660, 2021.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter. Conf. Mach. Learn. (ICML), 2020.
[10] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. IEEF/VVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 15750-15777 2021.
10

---

[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0422 2020.
[12] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE Inter. Conf. Compul. Vis. (ICCV), 2021.
[13] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metrie discriminatively, with application to face verification. In IEEE/CVE Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), volume 1. pages 539-546. IEEE, 2005.
[14] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 26, pages 2292-2300, 2013.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In Inter. Conf. Learn Represent. (ICLR), 2020.
[16] Matthijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg, and Cordelia Schmid. Evaluation of gist descriptors for web-scale image search. In ACM Inter. Conf. Image Video Ret. (CIVR), pages 1 1-8, 2009.
[17] Mark Everingham, Luc Van Gool, Christopher KI Williams. John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inter. J. Compus. Vis., 88(2):303-338 2010.
[18] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Inter. Conf. Learn. Represent. (ICLR), 2018.
[19] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-of- viieof--vsullwworte generation for unsupervised representation learning. JEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), 2020.
[20] Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog, (CVPR), 2022.
[21] Jacob Goldherger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In Adv. Neural Inf. Process. Sys. (NewrIPS), volume 17, 2004.
[22] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach self-supervised learning. In Adv. Neural Inf. Process. Sys. (NeurIPS), 2020.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Inter. Conf. Comput. Vis. Pallern Recog. (CVPR), pages 770-778,
2016.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVE Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 9729-9738, 2020.
[25] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Trans. Big Data, 733)5555-577, 2019.
[26] Armand Joulin and Francis Bach. A convex relaxation for weakly supervised classifiers. In Inter. Conf. Mach. Learn. (ICML), 2012.
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[28] Dong Hoon Lee, Sungik Choi, Hyunwoo J Kim, and Sae- Young Chung. Unsupervised visual representation learning via mutual information regularized assignment. In Adv. Neural Inf. Process. Sys. (NeurIPS), volume 35, pages 29610-29622 2022.

---

[29] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. Inter. Conf. Learn. Represent. (ICLR), 2021.
[30] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter. Conf. Learn. Represent. (ICLR). 2016.
[31] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian repre- sentations. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 6707-6717, 2020.
[32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018.
[33] Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means, In /EEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 16640-16649, 2022.
[34] Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 5706-5715, 2018.
[35] Jerome Revaud, Jon Almazán, Rafael Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with a listwise loss. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 5107-51 16,,,16666666 2019.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Inter. J. Comput. Vis., (5533)22112553 2015. doi: 10. 1007// 1663-015-88111 -01---
[37] Thalles Silva and Adín Ramírez Rivera. Representation learning via consistent assignment of views to clusters. In ACM/SIGAPP Symp. Appl. Comp. (SAC), page 987-994, 2022. ISBN 9781450387132. doi: 10. 10.1145/34773144350777
[38] Thalles Santos Silva and Adín Ramírez Rivera, Consistent assignment for representation learning. In Energy Based Models Wksp. (ICLRW), 2021.
[39] Thalles Santos Silva, Hélio Pedrini, and Adín Ramírez Rivera. Self-supervised learning of contextualized local visual embeddings. In Vis. Induct. Priors Data-Eff. Deep Learn. Wksps. (ICCVW), 2023.
[40] Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IEEE Inter. Joint Conf. Neural Netw. (IJCNN), pages
IEEE, 2011.
[41] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Adv. Neural Inf. Process. Sys. (NeurIPS), volume 33, pages 6827-6839. Curran Associates, Inc., 2020.
[42] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Lue Van Gool, Scan: Learning to classify images without labels. In European Conf. Comput. Vis. (ECCV), pages 268-285. Springer, 2020.
[43] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In IEEF/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 8769-8778 2018.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. In Ady, Neural Inf. Process. Sys. (NeurIPS), volume 30, 2017.
12

---

[45] Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In IEEE Inter. Conf. Comput. Vis. (ICCV), pages 9505-9515, 2021.
[46] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR), pages 3733-3744 2018.
[47] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprini arXiv: / 7708.00088.00 00880888, 2017.
[48] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning redundancy reduction. In Inter. Conf. Mach. Learn. (ICML), 2021.
[49] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ihot: Image bert pre-training with online tokenizer. In Inter. Conf. Learn. Represent. (ICLR), 2022.
13

---

A Extended results
In this section, we expand the evaluation experiments from the main text. We use the officially released pre-trained methods across all benchmarks to represent each SSL method. We did not retrain/reimplement any competing method. In total, we pre-trained 4 instances of CARP. To evaluate CARP's performance in a low training regime, we trained two 200-epoch models, one with multi-crop (me) and the other without it. Similarly, to evaluate longer training performance, we trained two 400-epoch models, one with multi-crop and the other without it.
A.1 Transfer learning evaluation
In Table 1, report detailed results for the transfer learning k-NN experiments. We evaluate SSL methods for values E {10, 20, 100, 200), including all instances of CARP. Cf. Appendix D.3 for the evaluation protocol.
CARP achieved either top-1 or top-2 performance seven out datasets. STL-10 is the only dataset where CARP is neither the top-1 nor top-2. Moreover, in the FGVCAireraft, Stanford Cars, and GTSRB datasets, CARP archived top-1 and top-2 performances with large margins. The average k-NN performance per value of l across all datasets is reported in Table I in the main text.
A.2 Image retrieval and copy detection
Tables A.2 and A.3 show additional results for image retrieval and copy detection evaluations. For both tasks, we compare CARP's performance against nine SSL methods. We report mAP on the Medium and Hard splits of the revisited Oxford and Paris datasets for image retrieval. We report mAP the "strong" set of the Copydays dataset for copy detection.
A.3 Few-shot evaluation
Table A.4 shows additional few-shot evaluation results on the Pascal VOCO7 and INat-2018 datasets. We report mAP and top-1 accuracy @k, averaged over 5 independent runs, for VOCO7 and INat-2018. respectively. We include CARP's 200 and 400 multi-crop models. Cf. Appendix D.2 for details on the evaluation protocol.
A.4 Dense prediction evaluation
We evaluated our CARP's multi-crop model on the object detection downstream tasks using the Pascal VOCO7 dataset. We followed the guidelines from He et al. [24]. We fine-tuned for 24k iterations on PASCAL VOC trainval07+12 and evaluated on test2007. CARP performs at AP=44.2, AP50=79.4, and AP75=47.66 66; results are averaged over 5 trials. Compared to other SSL methods, such as MoCo-v2 [11] (AP=57.4 AP50=82.5 AP75=64.0), CARP underperforms significantly.
B Ablations
Due to a limited execution budget, the ablations and the main experiments differ slightly in some hyperparameters. Here, we describe the configurations used for the ablations-for the main experi- ments, see Appendix C. For ablations, 50 epochs. The projection head learns a latent representation of 128-dim. The batch size is set to 256 observations, and the projection head hidden layers contain 2048 neurons. We set the number of learnable prototypes = 65 536, and the number of random partition blocks Ny = 128. Hence, each block contains Ng 512 prototypes. We report results for single runs.
trained CARP using the full ImageNet- 1M dataset for
B.1 Does the number of learnable prototypes affect the learned representations?
Table B.1 examines the effect of training CARP with different configurations of prototypes K. Similar to other clustering-basee SSL methods [7, 8, 29]. CARP also benefits from over-clustering. As the number of trainable prototypes K grows, the k--N performance of the learned representations increases. In addition, note that if the number of prototypes K is smaller than the number of actual
14

---

Table A.1: Transfer learning evaluation. We compare CARP's performance against nine SSL methods on eight datasets. We report results for & E {10, 20, 100, 200). Top methods in bold, top-2 underlined, Methods with (me) use multi-crop.
Oxford-IITT Pet
Oxford Flowers- 102
FGVCAircraft
Stanford Cars
Method
ep
10
20
100
200
10
20
100
200
10
20
100
200
10
20
100
200
oBoW (me) [18]
200
55.8
57.3
57.2
57.5
63.5
61.9
58.9
59.4
19.1
18.1
15.8
14,9
11.9
11.5
10.9
10.5
ScLa-v2 [1]
400
66.7
66.8
65.8
66.0
59.5
58.6
56.8
57.2
21.3
20.7
18.1
16.4
13.3
13.3
13.5
13.4
InfoMIn [41]
800
78.0
77.8 76.6
76.2
63.6
61.9
60.1
60.8
18.9
18.2
15.8
13.4
14.7
14.4
13.2
12.9
SWAV (mc) [7]
800
77.2 77.0 74.9
74.5
76.4
75.2
73.7
74,6
29.6
29.0
27.5
25.1
22.7
22.7
21.8
21.0
DINO (mc) [8]
800
81.5 80.9
79.0
78.9
82.3
81.6
80.8
81.2
36
35.3
33.5
31,1
30.0
30.1
28.9
27.5
DeepC-v2 (mc) [5]
800
79.0 78.3
76.3
75.4
78.3
76.3
75.3
76.0
32.5
32.0
28.9
26.5
25.2
25.0
23.4
22.1
Triplet [45]
980
83.3 83.5
82.4
82.4
78.5
77.7
76.9
77.3
33.3
33.4
31.7
29.6
24.4
25.2
25.5
25.0
MoCo-v3 [12]
1000
86.6
86.4
85.8
85.8
79.8
79.0
78.3
78.6
37.7
36.9
33.5
32.1
28.6
29.3
28.4
27.2
BarlowT [48]
1000
82.5
82.9
82.2
82.3
79.8
78.8
77.9
78.1
32.9
32.7
30.6
29.2
25.9
26.3
26.1
25.2
CARP
200
86.8
86.8
86.2
85.9
79.0
78.2
77.4
77.7
40.5
38,9
36.0
34.4
29.4
29.8
29.6
29.6
400
86.4
86.8
86.2
86.0
81.0
80.0
79.3
79.6
42.5
42,1
39.3
38.6
32.6
33.5
32.6
31.5
CARP (mc)
200
78.7
78.7
77.1
76.8
80.6
79.7
78.7
78.8
35.7
35.0
32.4
30.6
26.4
26.6
25,4
24.3
400
83.9
83,9
83.6
83,2
81.4
80.3
79.0
79.6
35.2
34.8
32,4
30.7
26.6
27.1
26.1
24.9
Country-211
Food-101
101
STL-10
GTSRB
Method
10
20
100
200
10
20
100
200
10
20
100
200
10
20
100
200
oBoW (mc) [18]
200
11.7
12.0
11.
11.4
45.8
47.4
47.3
46.0
96.6
96.6
96.3
95.7
50.1
50.6
49.9
48.2
SeLa-v2 [1]
400
10.1
10.5
11.0
11.0
45.7
46.8
46.6
45.5
94.0
93.5 93.3
58.1
59.0
57.9
InfoMin [41]
800
11.6
11.9
11.9
51.5
52.4
51.4
49.5 96.5 96.4 96.2 96.0
54,9
54.8
52.3
SwAV (mc) [7]
800
13.6
13.8
13.1
12.7
57.9
59.1
58.2
57.0 95.5 95.2 94.0 93.0
62.9
63.2 61.6
60.0
DINO (mc) [8]
800
14.4
14.2
13.6
60.9
62.0
61.4
95.9 95.6 94.7 93.8
62.7
62.9
62.6
61.8
DeepC-v2 (me) [5]
800
13.3
13.6
12.5
12.1
61.2
62.3
61.
60.1
95.7 95.6 94.5 93.3
62.9
63.4
62.4 61.3
Triplct [45]
980
13.7
14.1
14.3
14.2
60.1
61.5
61.0
60.0
95.7 95.6
94.9 94.5
63.4
63.5
62.9
62.0
MoCo-v3 [12]
1000
12.4
12.4
13.2
13.2
59.0
60.0
59.1
57.7
96.9
96.7 96.2 95.9
72.4
72.8
72.6
71.7
BarlowT [48]
1000
12.8
13.3
13.7
13.6
60.3
61.4
60.7
59.4
94.8
94.8
94.2
93.8
65.3
65.6
65.6
64.4
CARP
200
11.9
12.2
12.7
12.8
57.4
58.4
57.7
56.3
95.5
95.5
94.6
93.9
73.1
73.7
73.5
72.6
400
11.9
12.3
12.8
12.8
57.6
58.4
57.6
56.3
96.
95.9
95.0
94.3
74.7
75.3
75.2
74.4
CARP (mc)
200
14.2
14.5
14.3
13.9
60.5
61.8
60.7
59.3
95.8
95.5
94.1
93.4
64.6
64.7
64.2
63.0
400
14.1
14.2
14.3
13.9
61.7
62.9
62.1
60.7
95.9
95.5
94.3
93.4
62.2
62.8
62.4
61.4
classes in the dataset, the N-NN performance of the learned representations degrades. Based on these experiments, set the default number of prototypes K 65536 for the ImageNet-INM 1M dataset.
B.2 Does the number of partition blocks matter?
To better understand the effect of the hyperparameters Np and NB on the Iearned representations and in the training stability, the first row of Table B.2 demonstrates the performance of CARP using different configurations for the number of partition blocks Np and their sizes Ng. For completeness, we analyze the effect of removing the momentum encoder in Appendix B.3. We also present an ablation on the effect of the momentum update in Table B.3.
Specifically, as the partition sizes grow and the number of partition blocks Np decreases, the quality of the learned representations tends to decline and eventually collapse. Note that setting a partition size NB 65536 produces single partition block Np containing all prototypes. Precisely, the setup in the last row and last column of Table B.2 is equivalent to CARL [38]. It shows that a naive implementation leads a collapsed solution, and the wnvide-ndd-conuer approach of devising random partitions from the learnable prototypes avoids such trivialities.
Note that as smaller the block size Ng. more stable the algorithm will be. However, the quality of the learned representation might decrease since the partitions, becomes easier with fewer prototypes. On the other hand, a larger block size Ng poses more challenging consistency task at the expense of contributing to mode collapse.
tasks, posed by the random
For most cases, however, for block sizes ranging from NB = 128 to NB 4096, CARP learns useful representations and shows robustness to this hyperparameter. By default we set the partition block size Ng = 512.
15

---

Table A.2: Image retrieval evaluation. We report mAP performance of various self-supervised methods for the image retrieval downstream task on the revisited Oxford and Paris datasets. All SLL methods were pre-trained on ImageNet-IM. We used the officially released pre-trained models from respective methods for evaluation. Top-1 performers in bold, top-2 underlined.
ROx
Rpar
Method
Ep
Medium
Hard
Medium
Hard
Supervised
100
49.8
18.5
74.0
52.1
Scratch
1.6
4.1
2.5
oBoW (me) [18]
200
20.4
4.4
40.6
16.2
SeLa-v2 (mc) [1]
400
20.1
4.9
37.1
13.6
InfoMin [41]
800
24.4
5.7
44.6
18.8
SwAV (mc) [7]
800
31.1
10.1
48.9
20.6
DINO (mc)
800
35.4
11.1
55.9
27.5
DeepC-v2 (mc) [5]
800
32.6
10.9
50.0
20.2
Triplet [45]
980
35.3
12.0
58.2
28.7
VICReg [3]
1000
32.7
8.5
57.5
29.0
MoCo-v3 [12]
1000
33.1
10.9
59.1
31.3
CARP
200
38.8
15.5
58.8
30.4
400
38,9
15.1
58.5
30.2
CARP (mc)
200
32.8
10.4
24.9
400
33.7
11.6
54.0
26.5
Table A.3: Copy detection evaluation. We report mAP on the "strong" subset of the Copydays dataset and compare CARP's performance against seven SSL methods. Top-1 performers in bold, top-2 underlined.
Method
Ep
mAP
Scratch
25.7
oBoW (mc) [18]
200
61.5
ScLa-v2 (me) [1]
400
76.6
InfoMin [41]
800
67.5
SwAV (mc) [7]
800
76.1
DINO (mc) [8]
800
78.8
DeepC-v2 (me) [5]
800
76.0
Triplet [45]
980
81.7
VICReg [3]
1000
83.7
MoCo-v3 [12]
1000
80.6
CARP
200
82.3
400
82.6
CARP (mc)
200
80.8
400
84.0
B.3
The importance of the momentum encoder
Table B.2 contrasts CARP's joint-embedding architectures with and without momentum encoder, which is equivalent to setting 7 = 0 in the momentum encoder update equation. Different from other SSL methods [8, 22], CARP works with both setups. However, observe that using momentum encoder significantly boosts the performance of the learned representations. Table B.2 shows that regardless of block sizes, representations learned using momentum encoder-batd based architecture consistently outperform the siamese counterpart.
B.4
Who provides the best features for downstream evaluation?
One way to understand CARP's joint-embedding architecture with momentum encoder is through the teacher-student framework, where the momentum encoder is the teacher that guides the learning student. The addition of the momentum encoder raises the question of which module produces the
16

---

Table A.4: Few-shot classification on VOCO7 and INat-2018. We report mAP on VOCO7 and 1op-1 accuracy for INat-2018 across 5 independent runs, where n denotes the number of training examples. Standard deviations rounded to the first decimal place.
Pascal VOCO7
Method
Ep
n=1
n==
n=4
n=
n=16
full
PCL v2 [29]
200
47.9
59.6 2.7
66.2
74.5
+
0.4
85.4
SeLa-v2 (mc) [1]
400
4.2
5.2
6..2
0.5
7..9
85.3
DeepC-v2 (mc) [5]
800
2..
28.4
66.5
7.9
79.
87.6
SwAV (me) [7]
800
42.9
4..9
64.0
7.19
78.7
DINO (mc) [8]
800
4..6
58.4
66.622.1
74.8
79.6
88.2
Triplet [45]
980
43,6
5..2
6..6
73.8
79.6
88.3
McCo-v3 [12]
1000
4..6
59.6
6..0
7.74
8.6
87.4
BarlowT [48]
1000
3.7
1.8 3.2
72.6#0.1
0.5
86.3
CARP (mc)
200
4.2
3.3
6.4
0.1
0.6
88.0
400
47.1 3.2
5.2
+ 2.2
75.8 1.1
80.0 0.7
88.2
INat-2015
Method
Ep
n=1
11=2
n=4
n=8
n=16
full
PCL [29]
200
1.4
1.6
2.3
2.1
4.1
2.1
SeLa-v2 (mc) [1]
400
0.2
0.1
6.1
10.0
13.5
8.2
DeepC-v2 (mc) [S]
800
7.6
13.0
2.59
29.6#0.4
36.4+0.2
32.8
SwAV (mc) [7]
800
5.3
9.2
1..6
23.1
29.4
24.2
DINO [8]
800
6.5
1..0
2..4
29.6
3..
30.4
Triplet [45]
980
11.4
19.1
2..9
37.6
44.0
41.4
MoCo-v3 [12]
1000
8.1
12.
183
27.2
33.5
28.0
BarlowT [48]
1000
8.8
0.5
1722
24.6#0.1
30.8
25.3
CARP (mc)
200
0.6
04.4
2..6
32.
38.
400
0.1
1..6
2.9
0,3
45.1
42.6
Table B.I: CARP benefits from over-clustering. Setting a small number prototypes may hurt the learned representations.
1024
2048
4096
16384
65536
262144
K-NN 48.81
49.98
50.69
50.81
51.2
51.31
best representations. To answer this question, Figure B.1 explores the k-NN performance when extracting features from the momentum encoder (teacher) versus the student. We observe that teachers' representations constantly outperform the students' during training. However, by the end of the training, the student catches up with the teacher.
C Implementation Details
C.1 Experimental Setup
We train CARP the ImageNet- unlabeled dataset using ResNet50 [23] encoders. We take the output representation of the last global average pooling layer (a 2048-dim vector) and project it to a [8] work, our MLP projection head contains 3 dense layers
256-dim vector. Following Caron with batch normalization and the GELU activations. The hidden units of the projection head contain 2048 neurons. The 256-dim representation vector is fed to an assigner MLP that outputs unnormalized probabilities w.r.t. the learnable prototypes. By default, the assigner function is implemented as linear layer and trains K 65 536 prototypes. To generate the random partitions, we the number of partitions Np = 128, which creates subsets containing Ng = 512 randomly chosen prototypes. use the same data augmentations proposed by Grill et [22] to generate synthetic views. The protocol creates three data augmentation pipelines, the first two to generate global views and the last to generate multi-crops. CARP is pre-trained with the LARS [47] optimizer, end to end, with weight decay of I 10-6, For models training up 200 epochs, the learning rate starts from 0.6 and decays
17

---

Table B.2: CARP with and without momentum encoder. Without the random partition strategy (last column), training collapses regardless of using a momentum encoder or pure siamese architecture.
Ng
32
64
128
256
512
1024
2048
4096
16384
65536
w/ mom. enc.
49,56
50.75
51.19
51.20
51.32
51.06
51.31
51.08
49.67
0.11
w/o mom. enc.
48.95
49.28
48.81
47.37
46.16
44.68
44.29
44.39
47.25
0.11
Table B.3: The effect the hyperparameter on the momentum encoder updates. the last column,
7 starts as 17 0.99 and it is annealed to r) 1.0 following cosine schedule.
7
0.5
0.9
0.99
0.999
0.99 + 1.0
K-NN 51.0
50.2
50.3
51.1
51.3
to 0.006 with a cosine scheduling [30] without warmups. For models pre-trained for more than 400 epochs, the learning rate starts 0.3 and decays 0.003 using the same cosine scheduler. We train the system with global batch size 4096 observations. For all experiments, we used A100 40GB GPUs and gradient accumulation to simulate large batch sizes. Cf. to Appendix E for PyTorch style pseudo-code.
D Evaluation Protocols
D.1 Linear evaluation
For ImageNet-IM evaluation, trained linear classifier on top of the frozen representations extracted from the last average pooling layer of the ResNet50 encoder for 100 epochs, following Zhou et al.'s [49] protocol. The evaluation script performs a grid search hyperparameter tuning the learning rate, weight decay regularization, and optimizer. For each input image, we take a random crop followed by resize 224 224 and optional horizontal flipping. For testing, images resized to 256 256 and center-cropped to 224 x 224.
D.2 Few-shot evaluation
We measure the few-shot learning capabilities of SSL methods on the Pascal VOCO7 and iNaturalist 2018 datasets. For VOCO7, we are interested in the multi-label classification performance. We closely follow Li et al.'s 's [29] protocol and train Linear SVNs on fixed 2048-dim representations from many SSL ResNet-50 encoders.
For 1Nat-2018, we expand the few-shot evaluation challenge to a complex scenario containing more than 8k classes. We train linear classifiers for 20 epochs on fixed 2048-dim representations. We
62
k-NN top-1 accuracy
60
58
56
54
Student (0)
52
Teacher (E)
25
50
75
100
Training epochs
Figure B. During training, the representations extracted from the teacher outperform the representa- tions from the student network.
18

---

the SGD optimizer. The learning rate starts at 0.03 and decays by factor of 10 in epochs 12 and 18, respectively.
For both datasets, we vary the number n of labeled examples per class and report the average results across 5 independent runs.
D.3 N-NN evaluation
To perform the k-NN evaluation, we use pre- rre-trained SSL ResNet50 encoders as feature extractors to compute and store the representations from many vision datasets. Following Caron et al.'s 's [8] setup, the representation vector for a test image is compared against all representations from the training split and prediction is made via weighted voting. If one of the closest neighbors has the same class as the test image, it contributes to the final voting a; exp M2z) where M is a memory bank containing representations from the training data, 2 is the representation from the test data, and is the temperature hyper-parameter For all experiments, Knear E { (10, 20, 100, 200}. For most experiments, a value = 20 is consistently the best setup across all methods.
run N-NN with configurations of
D.4 k-means evaluation
Similar to k-NN evaluation, we take self-supervised pre-trained encoders and use them to extract 2048-dim feature vectors from the training set of datasets like CIFAR-10/100 and ImageNet-IM: For ImageNet-IM, we use only 10% of the training data following the 10% subset from Chen et al. [9]. We fit k-means classifiers split to assess the quality of the learned prototypes. We report three metrics to assess clustering performance: Normalized Mutual Information (NMI), Adjusted Mutual Information (AMI), and Adjusted Rand Index (ARI). The number of prototypes k is set to be the number of true classes of each dataset. We the faiss library [25] for fast k-means. For each experiment,
the learned representations of the training set and use the validation
run k-means for 100 iterations, 20 redos, and spherical normalization. To measure the clustering performance of CARP, we observed that 400 epoch model with a learning rate of 0.3 slightly outperformed the other instances; therefore, we use this model to report results in Table 2. This instance of CARP uses two views and is only used for clustering evaluation.
D.5 Image retrieval evaluation
We strictly follow the evaluation script wwwiiiaeaarieeeeeeeeeee py provided by Caron et al. [8], for the image retrieval evaluation task. The script uses the revisited Oxford and Paris image retrieval datasets [34]. The dataset contains three protocols of varying difficulty levels. We take the ImageNet pre-trained ResNet-50 encoder from CARP, freeze the weights, and apply k-NN evaluation directly to the frozen 2048-d features for retrieval, conditioned on a query image.
D.6 Copy detection evaluation
We strictly follow the evaluation script eval_ copy wew.ectiiioonn.mm py provided by Caron al. [8] for copy detection evaluation. The evaluation is performed on the INRIA Copydays dataset [16]. The dataset contains holiday pictures in the format query/database. Each image has suffered three kinds of artificial attacks: JPEG, cropping, and "strong." We report performance evaluation on the "strong" subset. Images in the "strong" subset were intentionally distorted by blur, insertions, print, and scan. The task is to recognize these images despite distortion. We take the frozen CARP ResNet-50 encoder and extract 2048-dim vectors from query and database images at resolution 3202. Then, we perform copy detection with cosine similarity between query and database features. We report mean average precision (mAP) as performance metric. Unlike the benchmark described by Caron al. [8], do not utilize additional distractors, nor do we centralize the data using statistics learned in different set on images.
E Pseudocode of CARP in a PyTorch-like Style
UB: number of random prototypes within block
K: number of prototypes
19

---

NP: number of blocks in the partition, e.e. K // NB
N: batch size
for x1, x2 in loader:
student and teacher branches
21, w1 enc(x1), mom_enc(x1) # [N, K]
22, #2 exc(x2) enc_enc(a (x2) # [N, K]
s_logits, t_logits [z1, z2] [w1, w2]
sample le cluster indices with replacement rand_proto_idd ids multinomiallomess split_proto_ids stack (rond_proto__de NB))
K, False)
preds_list, targeta_list [], [
for s_log, t_log in zip(s, sip(s_logits, t_logits):
preds get_logita_grre_te targets get_logits_grrt_log aplit_proto_ids.
idd))
preds_list apssnd (preds) targets_ bistt. append (targets)
loss loss_ faas_fn(predd, _list, targeta_list) perform gradient descent steps
def loss_fn(s_lists t_list):
c_loss comsistency_Loor(e__lee000 [0], t_list[1]
c_loss +
t_list[0]
cat (s_list, dim=1)
cat (t_list, dim=1)
probs cat([a, t], dim=1) transpose 1)
e_loss kl_div (mean dim=O))
return c_loss e_loss
def consistency (osaa,s, t):
loss = einsum ("kne, knc->kn", [s, t])
-log(1oas) .maooaa))mam())))
def kl_div(p):
return mean (Kog(K) + sum(p log(p), dim=-1))
def get get_logits..rrlogttt. proto_ids).
logits_gr logits(;, proto_ids.flattenn) logits_gr logits_gr. (NB,t dim-1)
logits_gr stack (logita_gr, dim=0)
softmar (logita..,, dim=-1) [NP, NB]
20


Reviews:
[{"review_id": "HVNGljX27U", "reviewer": "Reviewer_VZFy", "summary": "This paper addresses a collapsing problem that arises in clustering-based contrastive learning. To resolve the problem, the paper proposes an improved version of consistent assignment in CARL, utilizing a strategy of random partitioning. In particular, the original consistent assignment loss exhibits a stability issue when a large number of prototypes are employed; thus, it is challenging to determine proper hyperparmeters to ensure both stabilization and strong performance. This paper introduces a method of random partitioning among the trainable prototypes and applies the consistent assignment loss across each partition. Such approach effectively mitigates the instabilty problem associated with a large number of clusters while maintaining the benfits from using numerous clusters. Importantly, the proposed method does not introduce any additional hyper-parameters, such as trade-off parameter or sharpening temperature. The experimental results demonstrate the effectiveness of the proposed method in various downstream tasks such as transfer learning, clustering, image retrieval, copy-detection, few-shot classification, and linear/k-NN evaluation.", "questions": "### Suggestion\n* Please address the concerns listed in the weaknesses.\n* Given that all experiments are conducted using ResNet-50, experimental results with Vision Transformers (ViTs) shall strengthen the contributions of the paper.\n\n\n### Question\n* Why is it important or beneficial to avoid using non-differentiable modules, such as Sinkhorn-Knopp, for generating target cluster assignments? (L4-5, L99-100, L101-103)", "limitations": "The paper describes a limitation of the proposed method that the representations learned through CARP do not transfer well to dense prediction tasks. This limitation seems logical, as the proposed method simulates smaller pseudo-classification problems, which might be more suitable for downstream tasks such as classification, clustering, and retrieval.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The method seems simple yet effective, achieving robust performance and preventing collapse without the need for additional hyperparameters.\n* The experiments are conducted across various downstream tasks, including transfer learning, clustering, image retrieval, copy-detection, few-shot classification, and linear/k-NN evaluation; the method consistently demonstrates strong performance.\n* The method has been meticulously ablated (in main text and supplementary) with various implementation choices thoroughly examined.\n* The paper is well-composed and easy to follow.", "weaknesses": "* The discussion and comparison to other related baseline methods [1, 2, 3], which similarly employ the entropy of the mean probabilities of a batch while addressing a stabilization of it, seem to be missing.\n\n[1] Self-Supervised Learning by Estimating Twin Class Distributions, arxiv 2021 \\\n[2] Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment, NeurIPS 2022 \\\n[3] Masked Siamese Networks for Label-Efficient Learning, ECCV 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The discussion and comparison to other related baseline methods [1, 2, 3], [...], seem to be missing.\n\nTo address the concerns regarding a proper comparison to the other suggested methods, we took the pre-trained models from their respective official repositories and ran the same benchmark used in the paper for all methods. *These results support the conclusion that CARP learns representations that generalize to future tasks.* We detail the results below.\n\nWe took the 400 epoch multi-crop MIRA (github.com/movinghoon/MIRA) and the 850 epoch multicrop TWIST (github.com/bytedance/TWIST/tree/main), both with ResNet backbones. We did not benchmark MSN (github.com/facebookresearch/msn) because it only provides ViTs backbones which invalidates a fairer comparison with CNN-based encoders. We display the results in the table below. \n\nClustering evaluation, paper's Table 2\n\n\\begin{array}{lrrrrrrrrrrrr}\n\\hline\nMethod&ImgNet&&&Cifar10&&&Cifar100&&&GTSRB&&\\\\\\\\\n& NMI&AMI&ARI&NMI&AMI&ARI&NMI&AMI&ARI&NMI&AMI&ARI\\\\\\\\\n\\hline\nTWIST&\\textbf{72.7}&\\textbf{52.4}&\\textbf{28.6}&41.8&41.7&30.4&50.4&43.6&18.5&48&45.6&13.3\\\\\\\\\nMIRA&68.9&45.7&21.2&39.5&39.4&28.8&49&42.1&17.6&51.6&49.4&15.8\\\\\\\\\nCARP&70.3&48&23.9&\\textbf{49}&\\textbf{48.9}&\\textbf{38.7}&\\textbf{54.5}&\\textbf{48.2}&\\textbf{23.1}&\\textbf{54.8}&\\textbf{52.7}&\\textbf{19.6}\\\\\\\\\n\\hline\n\\end{array}\n\nOverall, CARP outperforms MIRA and TWIST on clustering evaluation, mainly in transfer scenarios with more considerable differences in scores. Interestingly, TWIST outscores CARP on the ImageNet dataset. However, **its performance decreased significantly on transfer datasets/tasks such as Cifar-10/100 and GTSRB**. **CARP**, on the other hand, **scores strongly on all 4 datasets.** These results support the conclusion that CARP learns representations that generalize to future tasks.\n\nk-NN transfer learning (k=20), paper's Table 1\n\n\\begin{array}{lrrrrrrrrrrrrr} \\hline\n&& Pets&Flowers&Aircraft&Cars&Country&Food&STL&GTSRB&Avg @k\\\\\\\\\nMethod&Ep&&&&&&&&&10&20&100&200\\\\\\\\\n\\hline\nTWIST&850&83.9&73.4&23.1&20.4&13.9&60.4&\\textbf{96.6}&59.1&53.8&53.9&52.8&52.1\\\\\\\\\nMIRA&400&83.4&\\textbf{81.4}&35.6&26.6&\\textbf{14.3}&\\textbf{64.2}&95.6&64.2&58.2&58.2&56.9&56\\\\\\\\\nCARP&400&\\textbf{86.8}&80&\\textbf{42.1}&\\textbf{33.5}&12.3&58.4&95.9&\\textbf{75.3}&\\textbf{60.4}&\\textbf{60.5}&\\textbf{59.7}&\\textbf{59.2}\\\\\\\\\n\\hline\n\\end{array}\n\nOn average, *CARP k-NN transfer performance outweighs both MIRA and TWIST on the 8 transfer datasets.* Individually, CARP wins 4 out of 8 datasets.\n\nAs pointed out by the reviewer, both MIRA and TWIST use the entropy term to avoid collapse during training. One important distinction, however, is that CARP maximizes the entropy over the random partitions and not over the entire set of prototypes. *We hypothesize (and show practical results in Table B2 in the appendix) that the random partition pretext task has a regularization effect that stabilizes training, avoids collapse, and improves final performance.* Because the subtasks constantly change, the model is less prone to collapsing all the embeddings into a single prototype. As a result, the regularization effect from the random partition diminishes the importance of entropy maximization. Hence, we do not need to consider tuning the entropy term's contribution to avoid trivial solutions. TWIST, on the other hand, needs to tune it carefully.\n\n> Given that all experiments are conducted using ResNet-50, experimental results with Vision Transformers (ViTs) shall strengthen the contributions of the paper.\n\nWe thank the reviewer for the suggestion. It is in our plans to explore how the proposed random partition pretext task would behave with other architectures, such as ViTs. However, we chose ResNets as the main architecture mainly because most SSL methods provide baselines with ResNet backbones. Indeed, recent SSL methods are switching to ViTs. However, given our computing budget, running so many long experiments with more than one architecture would be impractical. For this reason, we decided to pick one base architecture and do the best we could regarding training, ablations, and evaluations to make our proposal clear and robust.\n\n> Why is it important or beneficial to avoid using non-differentiable modules, such as Sinkhorn-Knopp, for generating target cluster assignments? (L4-5, L99-100, L101-103)\n\nWe apologize that the main text did not convey this matter clearly. We do not propose to avoid these types of non-differentiable modules, but rather we propose to explore a different solution. We will update our writing to convey this tone better.\n\nOur work provides an alternative solution to the cluster assignment problem with benefits and drawbacks. One benefit of having an end-to-end differentiable architecture might be performance-wise. The Sinkhorn-Knopp is an iterative algorithm that may require extra computation. Moreover, it also increases the number of hyperparameters that must be appropriately tuned. Nevertheless, solutions that employ the Sinkhorn-kopp deal with the extra compute and tuning and produce strong results. On the other hand, our approach shows that one can learn equally good, or even better, generalizable representations using a novel strategy that can avoid collapse in a deep learning integrated architecture. \n"}, {"review_id": "fJnsOlLrJU", "reviewer": "Reviewer_FHva", "summary": "This paper works on self-supervised representation learning. Under the setting of consistent clustering assignment between augmented views (SwAV-like), the authors found that when the number of prototypes is significantly larger than the batch size, the commonly used technique for avoiding trivial solutions fails. And they thus propose to randomly partition the prototypes into multiple subgroups to avoid the trivial solution. Experiments on extensive benchmarks, especially retrieval-based ones, validate the effectiveness of the proposed method.", "questions": "- I noticed that different tables may refer to different variants of CARP (or CARP w/ mc). Why is it and why multiple cropping does not help in e.g., Tab. A.1?\n- One advantage of clustering-based methods is that they tend to suit ViTs better (e.g., in DINO). Why is this work restricted in ResNets?", "limitations": "limitations are discussed in Sec. 7", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*Originality:* The major insight of this paper: entropy regularization fails when the prototype number is significantly greater than the batch size ($K \\gg N$) is novel, and the solution that divides the prototypes into multiple subgroups, is straightforward and verified to be useful. Other components of this framework resemble multiple previous works (*e.g.*, SwAV, DINO, MSN).\n\n*Quality:* The idea is clearly formulated and presented, and the method is well evaluated in extensive experiments, making it in shape of a solid paper.\n\n*Clarity:* The idea is straightforward and the delivery is clear enough, reading it is smooth and I do not find trouble in understanding.\n\n*Significance:* It tackles the assignment strategy in clustering-based self-supervised learning, and provides a solution for $K \\gg N$, which is somehow useful.\n\n*Reproducibility:* Code is provided in the supplementary material to facilitate reproduction.", "weaknesses": "*Motivation*: Why do we need so many prototypes during pre-training? The major stream of self-supervised learning is to enlarge the batch size for better representation learning. The prototype number, in contrast, has not shown the necessity for very large numbers yet. In fact, in DINOv2 they have reduced $K$ from 65536 in DINOv1 to 4096, which is competent for learning good representations. Moreover, large number of prototypes means high computational cost in pre-training. And after pre-training, they are simply dropped and not used in downstream tasks. So why do we need so many prototypes?\n\n*Technical Contribution:* In terms of technical contribution, this work proposes to randomly divide the prototypes into subgroups. The idea is clear and straightforward but the contribution might be limited.\n\n*Evaluation*: The evaluation is mainly focused on $k$-NN or linear probing on retrieval-based benchmarks. However, for one thing, a good deep representation does not have to be linear, and performance under full fine-tuning (transfer learning) might be of higher interest. For another, one might be more interested in generic classification benchmarks (ImageNet) and dense prediction downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Why do we need so many prototypes during pre-training? [...] \n\nBased on our practical experience, **the optimal number of prototypes highly depends on the number of hidden classes of the dataset.**  Due to ImageNet's high number of classes (1000), in practice, a higher number of prototypes produces better outcomes.\n\nWe ablated the number of prototypes in Section B.1 in the appendix. Practical results suggest that over-clustering is beneficial. **However, CARP proved to be robust to the number of prototypes.** Indeed, the **difference in k-NN performance when learning 65536 prototypes as opposed to 4096 prototypes is only 0.5%, which agrees with the reviewer's point about Dino-v2**.\n\nWhile we agree with the reviewer's concerns about the large number of prototypes, our empirical evidence suggests that there is a trade-off that must be considered.  However, to provide conclusive evidence and best practices, more work understanding this trade-off is needed.  However, this is out of the scope of our current proposal.\n\n> In terms of technical contribution, this work proposes to randomly divide the prototypes into subgroups. The idea is clear and straightforward but the contribution might be limited.\n\nIn summary, our main contributions are the proposal of a stochastic partitioning process to improve the learned representations' performance on downstream tasks, as demonstrated in our comprehensive evaluation protocol.  Moreover, we empirically show that this process reduces the noise on the learned representations, stabilizes the training, and avoids its collapse.\n\n> The evaluation is mainly focused on k-NN or linear probing on retrieval-based benchmarks. However, for one thing, a good deep representation does not have to be linear, and performance under full fine-tuning (transfer learning) might be of higher interest. For another, one might be more interested in generic classification benchmarks (ImageNet) and dense prediction downstream tasks.\n\nWe strived to design an evaluation protocol focused on two main aspects.  First, we wanted to evaluate the learned representation in transfer learning scenarios.  Second, the evaluation protocol should be diverse, including many distinct datasets with varying difficulty levels.  \nOur evaluation protocol includes over 15 datasets across many standard protocols, such as linear evaluation, few-shot classification, k-NN, k-means, image retrieval, and copy detection.\n\nWe agree with the reviewer that a good representation does not need to be linear. Nevertheless, **linear evaluations test the representations power** by using a simpler classifier, **giving a better intuitive sense of how good the representation is since it does not rely on other components**, such as the backbone architecture or the myriad of hyperparameters used to fine-tune.  A good example is linear probing.  Even though it is a linear protocol, many hyperparameters can influence the final result.   Consequently, comparing linear performance between methods that used different training strategies becomes unreliable.\n\nA good example is NNCLR [1], where the logistic regressor on top of the frozen representation is trained with the LARS optimizer and large batch sizes.  Based on our experiments, this configuration produces significantly better final accuracy than the approach proposed by MoCo (vanilla SGD with small batch sizes).  **With Linear evaluations based on k-NN** of k-Means, **the effect of external hyperparameters, choices of optimizers, or even the batch size, is reduced**, which in our view, **makes the comparison fairer.**  Nevertheless, we support evaluation protocols that test different aspects of the representation. \n\n[1] Dwibedi, Debidatta, et al. \"With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.\" ICCV. 2021.\n\n> I noticed that different tables may refer to different variants of CARP (or CARP w/ mc). \n> Why is it[…]\n\nFor our evaluation, we trained 4 instances of CARP in total.  CARP 200 and 400 epochs without multi-crop augmentation and CARP 200 and 400 epochs with multi-crop (w/ mc).  In the tables in the main text, we report the instances of CARP that performed best in the downstream tasks.  Indeed, we did the same for the competing methods.  In total, we compared CARP against 11 existing SSL methods.  However, in the main text, we reported only a subset that performed well on a given downstream task due to space constraints.  **We showed the full evaluation for all instances of CARP and all competing SSL methods in the appendix.** \n\n>  and why multiple cropping does not help in e.g., Tab. A.1?\n\nAs for the performance using multi-crop, it also caught our attention.  In short, **we suspect that multi-crop augmentation may be causing many SSL methods to overfit to ImageNet's training data distribution.**  As a result, we may see higher performance scores for linear probing on the same ImageNet, but modest transfer learning performance to other datasets and tasks.  However, further experiments are required to fully understand the impact of multi-crop and the decrease in performance seen in our experiments.  These evaluations are out of the scope of our current proposal, though. \n\n> One advantage of clustering-based methods is that they tend to suit ViTs better (e.g., in DINO). Why is this work restricted in ResNets?\n\nIt is in our plans to explore how the proposed random partition pretext task would behave with other architectures, such as ViTs.  However, we chose ResNets as the main architecture mainly because most SSL methods provide baselines with ResNet backbones.  Indeed, recent SSL methods are switching to ViTs. However, given our computing budget, running so many long experiments with more than one architecture would be impractical.  For this reason, we decided to pick one base architecture and do the best we could regarding training, ablations, and evaluations to make our proposal clear and robust.\n"}, {"review_id": "K8b7wQiHTz", "reviewer": "Reviewer_osfH", "summary": "This work extends the existing work about consistent assignment for representation learning by introducing random partitions. Specifically, when the number of prototypes is large in clustering-based self-supervised learning, the work shows that CARL [36] cannot handle the loss and the regularization well. To mitigate the problem, CARP is proposed to obtain subsets of prototypes and then apply CARL on the partitions of prototypes. Standard evaluation protocol is included to compare the proposed method to benchmark methods.", "questions": "Besides the weakness, my major concern is about the limited contribution compared to CARL.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The clustering-based method shows the potential for self-supervised learning. \n2. The strategy of random partition can avoid the problem from a large number of prototypes. \n3. Experiments are extensive on diverse data sets.", "weaknesses": "1. As a closely-rated work, the comparison to CARL lacks, which makes the contribution of this work unconvincing. Moreover, the original paper of CARL shows that it works well with 10000 prototypes, which contradicts the claim of this work. \n2. The performance of clustering-based method in the comparison is out of date. CoKe [1] as an online clustering method demonstrates 74.9% and 76.4% linear probing performance on ImageNet with two-view and multi-view augmentations, respectively. Moreover, CoKe does not request an additional momentum encoder and a sharpening temperature for the pseudo-label from the other view, which makes the contribution of this work less significant. \n3. While CARP shows a competitive performance with less training epochs, it is better to show if it can achieve a better performance with the long training strategy. \n\n[1] Unsupervised Visual Representation Learning by Online Constrained K-Means. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> [...] my major concern is about the limited contribution compared to CARL.\n\nThe main difference between CARL and CARP is the random partition pretext task and its positive effects on training SSL models. As described in Section 3.1, training a system like CARL is difficult due to stabilities, requiring tuning the entropy term and dealing with the negative trade-off. **Our pretext task diminishes the trade-off effect between the two losses and allows stable training with better outcomes.** Moreover, the entropy regularizer in CARP is also optimized over the prototypes within the subsets (which differs from CARL), contributing to stable training and high-quality representations (high scores in transfer learning benchmarks). Another part of our contribution is the **understanding of how we can stabilize training and avoid collapsed solutions through a stochastic sampling process** that is achieved with the proposed random partition pretext task.\n\n> [...] the comparison to CARL lacks, which makes the contribution of this work unconvincing. [...] CARL [...] works well with 10000 prototypes, which contradicts the claim of this work.\n\nEven though CARL can be trained with many prototypes, **one needs to carefully tune the weight contribution for the entropy term to avoid training collapse.** As discussed in Section 3.1, **representation quality decreases if the entropy term is too high, and collapse might happen if it is too low.** \n\nWe show evidence of this effect in Table B.2 in the appendix. As we increase the size of the partitions, performance decreases. The last column (Table B.2) is equivalent to **not having random partitions** (i.e., CARL), where we see collapse due to a large number of prototypes. The random partition pretext task addresses these limitations by improving overall performance and increasing training stability.\n\nCARL is focused on small-scale datasets such as Cifar-10/100 and STL-10, while our work's primary pre-training data is ImageNet. Nevertheless, to address the reviewers' concerns, we took CARL's official code (gitlab.com/mipl/carl), and pre-trained it on ImageNet for 200 and 400 epochs using 3000 prototypes (we experienced collapse with 10000). The results below show that CARP outperforms CARL on linear evaluation by considerable margins.\n\n\\begin{array} {rr}\\hline\nMethod&Ep&Acc&kNN\\\\\\\\\n\\hline\nCARL&200&65.3&59.4\\\\\\\\\n&400&70.4&62.8\\\\\\\\\nCARP&200&74.0&66.5\\\\\\\\\n&400&75.0&67.7\\\\\\\\\n\\hline \n\\end{array}\n\n> The performance of clustering-based method in the comparison is out of date. CoKe [1] as an online clustering method demonstrates 74.9% and 76.4% linear probing performance [...] \n\nTo address the reviewer's concern, we took the officially released 800 epoch model from CoKe's repository and ran our benchmark evaluations. We present the results below.\n\nk-NN transfer learning (k=20), paper's Table 1\n\n\\begin{array}{lrcrrrrrrrrrrr}\n\\hline\nMethod&Ep&Pets&Flowers&Aircraft&Cars&Country&Food&STL&GTSRB&Avg @k&&&\\\\\\\\\n&&&&&&&&&&10&20&100&200\\\\\\\\\n\\hline\nCoKe&800&79.5&79.5&27&22.4&14.6&58.9&95.7&60.4&57.1&57.4&57.1&56.3\\\\\\\\\nCARP&400&86.8&80&42.1&33.5&12.3&58.4&95.9&75.3&60.4&60.5&59.7&59.2\\\\\\\\\n\\hline\n\\end{array}\n\nClustering evaluation, paper's Table 2\n\n\\begin{array}{lrrrrrrrrrrrr}\n\\hline\nMethod&ImgNet&&&Cifar10&&&Cifar100&&&GTSRB&&\\\\\\\\\n&NMI&AMI&ARI&NMI&AMI&ARI&NMI&AMI&ARI&NMI&AMI&ARI\\\\\\\\\n\\hline\nCoKe&68.9&45.6&21.3&45.9&45.8&34.2&51.9&45.2&19.5&49.4&47.1&13.7\\\\\\\\\nCARP&70.3&48&23.9&49&48.9&38.7&54.5&48.2&23.1&54.8&52.7&19.6\\\\\\\\\n\\hline\n\\end{array}\n\nCoKe performs well when trained and evaluated on ImageNet. However, on transfer tasks, **CARP outperforms CoKe by large margins on both k-NN and k-means evaluations.** \n\n> Moreover, CoKe does not request an additional momentum encoder and a sharpening temperature for the pseudo-label from the other view, which makes the contribution of this work less significant.\n\nWe'd like to emphasize that CoKe does sharpen the probability distributions in its loss function. If we look closer at CoKe's code (github.com/idstcv/CoKe/blob/main/main_multi_view.py#L351C33-L351C33), we can see two losses (1) loss_pred and (2) loss_proj. The first is a hard loss, where predictions are sharpened. The second is a soft loss, where predictions and target distributions are sharpened before softmax (refer to parameter $coke_t$).\n\nIn fact, CARP and CoKe's loss functions differ significantly. While CoKe uses a sharpened cross-entropy loss for both hard and soft losses, **CARP proposes a consistency loss where the goal is to minimize the negative log of the product of the views probability distributions**. In the general rebuttal PDF, we show evidence that the sharpening operation is built-in CARP's loss. During training, the probability distributions for predictions and targets are automatically sharpened toward one-hot vectors.\n\nAlso, CARP's contributions **go beyond architectural and implementation details such as momentum encoders and use of temperatures** and mainly **focus on a novel approach to learning representations through prototypes that stabilizes training, avoids collapses, and yields transferable representations.** \n\n> While CARP shows a competitive performance with less training epochs, it is better to show if it can achieve a better performance with the long training strategy.\n\nTo address the limitations pointed out by the reviewer, we trained CARP for 800 epochs with multi-crop. CARP archives 75.9% linear probing on ImageNet, which beats methods such as DINO and SwAV (75.3%). While we understand the reviewer's point of view, we emphasize that *an efficient method is cheaper, allows faster training with reduced hardware costs and emissions (democratizing research), and has strong inductive biases* (allowing the model to extract patterns quickly).\n"}, {"review_id": "OGuI9vsxL2", "reviewer": "Reviewer_fyYo", "summary": "The authors propose a new method for self-supervised learning based on cluster assignments. The method is based on a consistent assignments approach that assigns the same prototype to different views of the same image. To overcome issues of the previous methods that are not well scalable, a divide and conquer approach is proposed. Radom partitions of the learnable prototypes are created each iteration/epoch that allows not only make the training to be stable but also faster. The authors show the efficiency of the approach across a wide range of different datasets and settings. ", "questions": "Q1: what if to have hard pseudo-labels instead of soft assignments?   \nQ2: L218 - it is an interesting observation, could the authors make an analysis and confirmation of that?  \nQ3: what is the value of \\lambda_e, no information in implementation details? how does the performance change if to change this parameter?  \nQ4: could the authors also add a discussion on the differences to SwAV?   \n", "limitations": "The authors address limitations in the form of discussion wrt the previous work based on which they propose the new approach. The section provides nice insights into the limitations of the previous method but also shows rather limited novelty of the current approach over the previous one. The method section is rather describing the previous approach as a new one that can be confusing. However, I find the proposed divide-and-conquer approach very interesting and helpful based on the results. It would be helpful to rewrite partially the paper making more focus on the proposed contribution rather than on the previous method with additional analysis/confirmation of all the statements.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper is well written, it’s clearly stated what is novel and what is borrowed from the previous methods, and the explanations of the approach are easy to follow.   \nS2: The idea of creating subtasks to reduce computation time and avoid collapse sounds novel.  \nS3: As it is empirically demonstrated, the proposed idea improves the performance over multiple benchmarks on kNN and retrieval tasks while being on-par with linear probe.   \n", "weaknesses": "W1: would be interesting to see how the methods work with transformer based architectures. e.g. within moco-v3 framework?   \nW2: the contributions are not clearly stated apart from the first one: citation [9] in contribution is wrong, it’s SimCLR contrastive approach that should belong to negatives. Moreover, the authors use entropy maximization as a form of normalization which the authors even state as a separate contribution. But this is cannot be a separate contribution because it was used before as a technique to avoid trivial solutions e.g. in [39].  \nW3: the Fig.7 is not helpful at all. It was not possible to understand what it is depicted there. The text (not caption) is clear though.   \nW4: In the intro, the authors introduce two classes of SSL. It is too restrictive division and does not cover all SSL methods. Where e.g. MAE lies?   \nW5: [36] is not really recent while the authors refer to it as “were recently proposed” on L41  \n  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: would be interesting to see how the methods work with transformer-based architectures?\n\nWe thank the reviewer for the suggestion. It is in our plans to explore how the proposed random partition pretext task would behave with other architectures, such as ViTs. However, we chose ResNets as the main architecture mainly because most SSL methods provide baselines with ResNet backbones. While recent SSL methods are switching to ViT, given our computing budget, running experiments with more than one architecture would be impractical. Thus, we decided to pick one base architecture.\n\n> W2: the contributions are not clearly stated apart from the first one [...] the authors use entropy maximization as a form of normalization [...] But this is cannot be a separate contribution because it was used before as a technique to avoid trivial solutions e.g. in [39].\n\n**Our main contribution is the introduction of the random partition pretext task.** In addition, CARP does not require normalizing the embeddings to a hypersphere (required by other methods), nor does it require mining negatives during training. As the reviewer pointed out, the mean entropy maximization of the probabilities has been used to avoid collapse. However, **CARP does not use it as other methods have**, such as TWIST and PAWS. In CARP, the mean entropy is maximized over the random partitions, i.e., we maximize it over many subproblems constantly changing during training. *We believe that this process removes noisy patterns from the processed data and recovers stable representations due to the stochastic process of the partitions*. The importance of our pretext task to avoid training collapse and improve representations' power is shown in Table B1, appendix.\n\n> W3: the Fig.7 is not helpful at all. It was not possible to understand what it is depicted there. The text (not caption) is clear though.\n\nThere is no Fig. 7 in the paper. \n \nIf you referred to Fig. 2, it was meant to describe the random partition pretext task. Other clustering-based SSL methods pose the view assignment task over the entire set of learnable prototypes. Conversely, CARP's random partitions break the prototypes into smaller subgroups. Each subset of prototypes is used to optimize a view classification task in parallel. In Fig 2, we have 2 views and 8 prototypes. The prototypes are randomly divided into 2 groups. For each group, (1) we enforce consistent assignment of views and (2) maximize entropy over the batch.\n\n> W4: In the intro, the authors introduce two classes of SSL. It is too restrictive division and does not cover all SSL methods. Where, e.g., MAE lies?\n\nWe apologize for overseeing the taxonomy and commit to updating it. Our motivation was to provide a simple and not exhaustive categorization for SSL methods based on joint-embedding architectures. We agree that MAE is an SSL method. However, MAE is an autoencoder (encoder/decoder) model based on pixel reconstructions, which makes it different from joint-embedding methods that operate in the embedding space that we were interested in presenting.\n\n> W5: [36] is not really recent while the authors refer to it as “were recently proposed” on L41\n\nWe thank the reviewer for the suggestion. We will reword this phrase in the final version of the manuscript.\n\n> What if to have hard pseudo-labels instead of soft assignments?\n\nCARP’s consistency loss is optimized when the views' probability distributions are identical one-hot vectors (hard assignments). However, sharpening the distributions toward hard assignments is an iterative process and converges in the limit. To address the reviewer's concern, we trained CARP using hard assignments instead of soft ones. We noticed that training becomes unstable and collapses right in the beginning. We hypothesize that the soft assignments have a regularization effect and prevent the embeddings from being assigned to the same prototypes. \n\n> Q2: L218 - it is an interesting observation, could the authors make an analysis and confirmation of that?\n\nIn the general rebuttal PDF, we show evidence that the sharpening operation is built-in CARP's loss function. During training, the probability distributions for predictions and targets behave like sharpened one-hot vectors, as shown by the maximum and minimum values of the distribution converging towards one and zero, respectively. The distributions of different sharpening values produce a similar trend.\n\n> what is the value of $\\lambda_e$, no information in implementation details? how does the performance change if to change this parameter?\n\nIn practice, the value of $\\lambda$ is 1. One of the benefits of our proposed random partition pretext task is a regularization effect that helps prevent collapsed solutions. CARP does not require tuning the $\\lambda$ parameter that defines the contribution of the entropy term to the final loss.\n\nWe thank the reviewer for pointing out this missing piece. *We will add a description of this parameter in the final version of the manuscript.*\n\n> Could the authors also add a discussion on the differences to SwAV?\n\nThe most important difference between CARP and SwAV is the way they avoid trivial solutions. While SwAV uses a non-differential iterative module (Sinkhorn–Knopp) to balance the target predictions over the learned prototypes, CARP uses the random partition pretext task combined with the mean entropy maximization. Plus, SwAV requires normalized embeddings to avoid NaNs during training; CARP does not. SwAV enforces consistent assignment between views using the cross-entropy loss and requires extra hyperparameters to sharpen the probability distributions. CARP, on the other hand, **has the sharpening of the probabilities built into its consistency loss**. CARP's consistency loss is optimized when the probability distributions of the views are identical one-hot vectors. CARP's consistency loss avoids extra hyperparameters and sharpens the distributions by design."}]

Author rebuttal:
We thank the reviewers for their time reviewing our work and their valuable feedback. We will incorporate the additional results presented in this rebuttal as well as the suggestions in the final version of our manuscript.

In summary, our main contributions are the proposal of a stochastic partitioning process to improve the learned representations' robustness, as demonstrated in our comprehensive out-of-domain downstream evaluation. Moreover, we empirically show that our random partition pretext task produces a regularization effect that stabilizes the training and avoids collapse.

Our comprehensive evaluation protocol assesses the representation power of more than 11 SSL methods over 16 different datasets. We increased this evaluation, as requested, and included evaluation results for three additional methods, TWIST [a], MIRA [b], and CoKe [c]. The additional results are presented in the reviewers' individual replies.
In short, **CARP pre-trained representations remained top performant, mainly on transfer learning scenarios, which speaks for the power of the proposed random partition pretext task**.

In addition, we are providing an extra PDF document containing plots that strengthen the intuition about the sharpening effect of the consistency loss used by CARP. 

We highlight that we evaluated our method against other ResNet-based models due to their prevalence in the existing SSL methods. In this light, our evaluation is fairer and demonstrates the improvements of the proposed partitioning while maintaining the backbones. While we share the desire to have more tests as proposed by the reviewers, due to our limited computing budget, we could not evaluate several backbones on our setup and ablations. 

[a] Self-Supervised Learning by Estimating Twin Class Distributions, arxiv 2021

[b] Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment, NeurIPS 2022

[c] Unsupervised Visual Representation Learning by Online Constrained K-Means. CVPR 2022.


Meta-review: Summary
The paper presents an approach to improve clustering-based contrastive learning methods. The authors address the issue of collapse wherein all the data points map to the same point during training. The proposed idea (CARL) is simple and uses consistent assignments by way of random partitioning.

Strengths
- The proposed method addresses an important issue in self-supervised learning on collapse. The idea to achieve this by tweaking the partitioning is simple.
- The experimental setup in this work is solid. The results across image classification, retrieval, copy detection, few-shot evaluation show consistent improvements

Reviews & Justification
The paper received positive reviews overall. The only negative reviewer has concerns about relative contributions over CARL. After reading the rebuttal thoroughly, the AC believes that the authors have resolved this concern to the best extent possible. Since it is non-trivial to extend the original CARL method to ImageNet, the authors contributions over this work are significant.
Decision: Accept (poster)