Title: Representation Learning via Consistent Assignment of Views over Random Partitions
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation learning

unsupervised learning

self-supervised learning

computer vision
Abstract: We present Consistent Assignment of Views over Random Partitions (CARP), a self-supervised clustering method for representation learning of visual features. CARP learns prototypes in an end-to-end online fashion using gradient descent without additional non-differentiable modules to solve the cluster assignment problem. CARP optimizes a new pretext task based on random partitions of prototypes that regularizes the model and enforces consistency between views' assignments.  Additionally, our method improves training stability and prevents collapsed solutions in joint-embedding training. Through an extensive evaluation, we demonstrate that CARP's representations are suitable for learning downstream tasks. We evaluate CARP's representations capabilities in 17 datasets across many standard protocols, including linear evaluation, few-shot classification, $k$-NN, $k$-means, image retrieval, and copy detection. We compare CARP performance to 11 existing self-supervised methods. We extensively ablate our method and demonstrate that our proposed random partition pretext task improves the quality of the learned representations by devising multiple random classification tasks.
In transfer learning tasks, CARP achieves the best performance on average against many SSL methods trained for a longer time.

Representation Learning via Consistent Assignment of Views over Random Partitions

Abstract

We present Consistent Assignment of Views over Random Partitions (CARP), a self-supervised clustering method for representation learning of visual features. CARP learns prototypes in an end-to-end online fashion using gradient descent without additional non-differentiable modules to solve the cluster assignment problem. CARP optimizes a new pretext task based on random partitions of prototypes that regularizes the model and enforces consistency between views' assignments. Additionally, our method improves training stability and prevents collapsed solutions in joint-embedding training. Through an extensive evaluation, we demonstrate that CARP's representations are suitable for learning downstream tasks. We evaluate CARP's representations capabilities in 17 datasets across many standard protocols, including linear evaluation, few-shot classification, k -NN, k -means, image retrieval, and copy detection. We compare CARP performance to 11 existing self-supervised methods. We extensively ablate our method and demonstrate that our proposed random partition pretext task improves the quality of the learned representations by devising multiple random classification tasks. In transfer learning tasks, CARP achieves the best performance on average against many SSL methods trained for a longer time.

1 Introduction

Learning from unlabeled data has been one of the main challenges in computer vision. Recent approaches based on self-supervised learning (SSL) have significantly reduced the gap between supervised and unsupervised pre-trained representations. Nowadays, self-supervised pre-training on vast quantities of unlabeled data, prior to learning a downstream supervised task of interest, can be more effective than supervised pre-training for many tasks [7, 19, 22].

Current SSL methods can be divided into two classes: (1) self-supervised embedding prediction [9, 24, 39, 41] and (2) clustering [1, 5, 6, 29]. As the name suggests, embedding prediction methods work directly in the representation space. They are trained with either contrastive or non-contrastive loss functions. Instead of reconstructing the input signal, their loss function maximizes agreement between embeddings of the same view and optionally pushes representations from different views apart. On the other hand, clustering methods discretize the representation space by learning a finite set of prototypes. These prototypes aggregate representations from different images that are similar enough to be assigned together. Nevertheless, recent SSL methods build a joint-embedding architecture that can be pure siamese [4] or use different encoders.

Code at https://sthalles.github.io/carp/ .

Adín Ramírez Rivera

The most challenging and significant difference among these methods is how they avoid trivial solutions when training joint-embedding architectures. Contrastive methods avoid trivial solutions by explicitly pushing representations of negative samples away from the anchor representation, while non-contrastive methods avoid trivial solutions by regularization or architectural designs [10, 22].

Among self-supervised clustering methods, recent work proposed avoiding trivial solutions by using non-differentiable modules such as the Sinkhorn-Knopp algorithm [1, 7, 14] and classic machine learning methods [29, 42] such as k -Means clustering or k -Nearest Neighbor, to solve the cluster assignment problem.

Consistent assignments [37, 38] were recently proposed as a way to learn prototypes to improve the representations. Similarly, our learning objective imposes two constraints (1) consistent assignment of views over learnable prototypes and (2) a uniform distribution for the average predictions within a batch. However, we show that such a strategy does not scale well to enormous datasets containing millions of classes. In such situations, we need to model a large number of prototypes while enforcing consistent assignment between views and avoiding collapsed solutions. We show that a naive implementation of this strategy makes the learning problem challenging from a training stability perspective, where the model quickly settles for a trivial solution by assigning all views' embeddings to the same prototype.

To overcome these issues, we propose a novel self-supervised approach based on the consistent assignment of views over random partition sets (CARP). We train CARP to minimize a consistency loss, which encourages the model to assign different views of the same unlabeled example to the same prototype. We solve the dimensionality problem by enforcing smaller pseudo-classification problems through the introduction of random partitions that enforce consistency and regularize the model. The energy between the views' representations and the trainable prototypes (within random partitions) allows us to automatically bootstrap predictions and targets to our consistency loss. Our contributions are three-fold:

A novel and entirely online joint-embedding learning strategy based on self-supervised clustering, see Figure 1. We propose a divide-and-conquer pretext task based on randomly generated partitions of learnable prototypes. Our loss function allows stable training of joint-embedding architectures in a self-supervised context.

A differentiable assigner module that generates soft pseudo-labels by comparing the representations of image views to prototypes within random partition subsets. To avoid trivial solutions, we enforce the average predictions over a batch to be non-informative over the set of prototypes within a random subset.

A framework that simplifies self-supervised training and does not require normalization techniques [7, 9] or the necessity of mining negatives for contrastive training [9, 31, 32].

2 Related work

Self-supervised embedding prediction methods operate directly in the representation space by learning a metric such that embeddings from views of the same image are closer to one another while embeddings from views of different images are far away in the feature space. These methods can be trained using contrastive or non-contrastive loss functions. Methods that minimize a loss function with a contrastive term date back to 1990s [4, 13, 21]. They must explicitly find representations from non-correlated images to use as negatives. Recent contrastive methods include InstDisc [46], CPC [32], SimCLR [9] and MoCo [11, 24]. These methods learn unsupervised representations by minimizing nearly the same contrastive loss function, i.e., the InfoNCE [32]. CARP does not directly optimize the views' embeddings, nor is it a contrastive method. Instead, we learn a set of general prototypes using a random partition strategy that stabilizes the learning process and avoids trivial solutions commonly found when training joint-embedding SSL models.

On the other hand, non-contrastive methods work by approximating embeddings of views from the same image. The main advantage is not requiring explicit opposing representations in the loss formulation. To avoid trivial solutions, a common approach is to implement a 'stop-gradient' operation that prevents the gradient signal from flowing to the two branches of the join-embedding architecture simultaneously. BYOL [22] and SimSiam [10] are examples of such methods. CARP takes advantage of non-contrastive training since it does not require mining negatives for optimization. Also, different from Grill et al.'s [22] work, CARP does not require a momentum encoder, though using it

ˆ

x

1

ˆ

x

2

f

θ

Encoder

f

ξ

g

θ

Projector

g

ξ

z

v

w

v

q

θ

q

ξ

Predictions

Targets

EMA

EMA

EMA

Partition

P

B

0

B

1

B

0

B

1

significantly improves the learned representations. CARP trains a joint-embedding architecture and uses the 'stop-gradient' operation in conjunction with a regularized pretext task based on random partitions of prototypes to avoid mode collapse.

Self-supervised clustering methods do not work directly on the views' embeddings. Instead, they learn a set of prototypes that are used to solve subsequent pretext tasks in different ways. Caron et al. [5], for instance, used k -Means clustering at every epoch to cluster the representations from the entire dataset and produce pseudo-labels for the training images. Then, a classifier head is trained to predict the pseudo-labels. Caron et al. [6] proposed a method to combine the rotation prediction pretext task [18] with clustering. Li et al. [29] presented a method based on expectation-maximization (EM) that merges clustering with the contrastive learning framework from He et al. [24]. Recent work by Caron et al. [7] and Asano et al. [1] combine SSL with clustering. They utilize the nondifferentiable Sinkhorn-Knopp algorithm to solve the cluster assignment problem without falling into collapsed solutions. Silva and Ramírez Rivera [38] proposed, CARL, an online clustering method that does not require non-differential algorithms to avoid trivial solutions.

Contrast to previous approaches. Instead of solving the cluster assignment problem using a nondifferentiable algorithm such as the Sinkhorn-Knopp, CARP is trained end-to-end with gradient descent. Different from Caron et al.'s [8] work, our model does not require extra momentum encoders or data structures to store previous predictions as a way to avoid trivial solutions. Unlike CARL [37, 38], CARP avoids trivial solutions by posing the optimization problem at the level of random partitions of prototypes that solve the high-dimensionality nature of the task. Unlike Caron et al.'s [6] work, our method does not require clustering the entire dataset every epoch to generate pseudo-labels. Instead, CARP generates soft pseudo-labels in an online fashion from examples in a single minibatch.

Currently, the best self-supervised methods [8, 12] use vision transformers [15, 44] as their backbones. Recent methods [20] employ a masking pretext task where some patches of the views are manually hidden, and the network is tasked to infer the missing pieces. In this paper, we consider transformerbased methods to be out-of-scope of our compared backbones. Thus, we do not include them in our results to maintain a fair comparison.

3 Consistent Assignment of Views

From an image x i , we create two views, ˆ x 1 i = T ( x i ) and ˆ x 2 i = T ( x i ) , using a stochastic function T that applies a set of random image transformations to x i (cf. Appendix C.1). CARP is a jointembedding architecture with two modules: a differentiable (student) and a non-differentiable (teacher)

Prototypes

ˆ

x

1

ˆ

x

2

z

1

w

2

B

0

B

1

branch. Each module has its own set of weights and the same architectural design. Both contain an encoder f ( · ) and a projection head g ( · ) . The differentiable student receives a pair of views and produces embedding vectors z v i = g θ ( f θ (ˆ x v i )) , for v ∈ { 1 , 2 } . Similarly, the non-differentiable teacher produces target embeddings w v i = g ξ ( f ξ (ˆ x v i )) .

The objective is to learn a set of prototype vectors C to discretize the embedding space. These prototypes are not meant to represent the true classes of the data. Instead, they may be interpreted as anchors to attract views of a given image to a commonplace in the embedding space. The function q ( · , · ) receives the views' representations, z v i and w v i ∈ R 1 × d , as input and outputs normalized probability vectors relating the views' embeddings with the prototypes such that s v i = q ( z v i , C ) and t v i = q ( w v i , C ) , where s v i and t v i ∈ R 1 × K are the normalized probabilities of a view, ˆ x v i , w.r.t. the prototypes C ∈ R K × d . Note that d is the dimensionality of the embedding vector, K is the number of prototypes, and the assigner q ( h, C ) = softmax ( h · C T ) .

To avoid trivial solutions in the joint-embedding training, we need a loss function that prevents the assignment of all representation vectors z i to a unique prototype. Unlike previous work, we seek a method that solves the cluster assignment problem in an online fashion using gradient descent.

We propose a loss function composed of two terms: consistency and entropy. The consistency term learns the relations between embedding vectors and prototypes. It enforces different views of the same image to be assigned to the same prototype with high confidence. For normalized probability vectors a and b , we define the consistency term as

where ⟨· , ·⟩ is a dot product.

The consistency loss is optimized when the two views ˆ x 1 i and ˆ x 2 i are assigned to the same prototype with maximal confidence, i.e., when the probability distributions of the two views s 1 i and s 2 i resemble equal one-hot vectors.

If we optimize the consistency loss, L c , training collapses to a state where all views are assigned to the same prototype. A common approach to avoid such failure is to ensure that all the prototypes get roughly the same number of assignments. Let us define the function

as the average probability across the representations within a batch of size L . For our distributions, we define ¯ p = avg ( { ( s v i , t v i ) } N i =1 ) . If we maximize the entropy of the mean probabilities of a batch, H (¯ p ) , we will encourage the average predictions to be closer to a uniform distribution. Previous work [2, 26, 42] has used this entropy term in various scenarios, ranging from discriminative unsupervised clustering to semi-supervised learning. In our case, this term enforces a non-informative prior over the prototypes with a particular schedule learning. Thus, the final proposed objective to

minimize is

where λ e > 0 trades off consistency at the view level with the average uniform assignment at the batch level. Note that the contribution of the entropy term decays as training progresses.

3.1 Limitations of Consistent Assignments

The formulation of the consistent assignments (3) is similar to CARL [37, 38], except for the additional teacher stream that stabilizes training and improves performance. Nevertheless, training an unsupervised system by minimizing the loss (3) is challenging. The main limitation is how to avoid trivial solutions in unsupervised training of joint-embedding architectures. For some cases, tuning the contribution of the entropy term might be enough to optimize the loss (3) stably. However, adjusting such a hyperparameter is difficult because one configuration does not hold for all training conditions. For instance, if the value of λ e is too small, the consistency term wins the arms race, and the average distribution over the batch, H (¯ p ) becomes one-hot alike, i.e., all views end up assigned to the same prototype. If the value of λ e is too large, the entropy term gets the upper hand, and collapse is avoided. However, the process of view assignment is neglected over the policy of distributing views uniformly, which results in poor performance of the learned representations, as shown by Table B.2.

For a small number of general prototypes, training is more stable, and the model avoids collapse with a simple tuning of the λ e parameter. However, for a larger number of general prototypes, stability becomes an issue. The main problem lies with the entropy term. When the distribution is larger, i.e., K ≫ N , regular batch sizes, such as N = 64 or N = 128 , become too small to properly model the distribution, i.e., the signal is too weak for most prototypes. Consequently, to avoid collapse, we need to increase the strength of the entropy term or increase the batch size, which in turn decreases performance.

To address such limitation, we propose to decouple the loss function (3) into smaller sub-problems. Instead of enforcing both consistency and uniform assignments over all the general prototypes, we propose a pretext task over subsets or blocks of a random partition of the general prototype set C .

4 Assignment based on Random Partitions

̸

Given the set of K trainable prototypes C = { c 1 , c 2 , ..., c K } , we define a partition of C as P = { B i ⊂ C } N P i =1 , such that ∅ / ∈ P , ⋃ i B i = P where B i ∈ P , and B i ∩ B j = ∅ for all B i , B j ∈ P , and i = j . We refer to B i as a block or subset of the partition. We are interested in a partition set P = { B i } N P i =1 of size N P , i.e., |P| = N P .

Using the concept of a partition of a set, we can define a framework of pretext tasks over partition blocks that satisfies the learning problem defined in Section 3. If the size of a partition block, B i , equals the number of prototypes, N B = K then the partition P is trivial, i.e., P = { B 1 } = { C } . If the size of the partition blocks equals N B = 1 , then we have K blocks in P , and each block has a unique prototype. Here, the learning task is equivalent to multiple binary classification problems, where each output score, if normalized, expresses the likelihood of a data point x i to independently belong to each prototype.

However, if the block size 1 < N B < K , and N B divides K , then the partition P will be composed of N P = ⌊ K/N P ⌋ blocks. We define P by randomly assigning N B prototypes c j , for j = 0 , 1 , . . . , N B , to each block B i = { c j } j , where i = 0 , 1 , . . . , N P .

Instead of mapping a single representation z v i as a linear combination of all prototypes in C , we compare the view's representations z v i and w v i against all the prototypes in the j -th block. That is, s v i,j = q ( z v i , B j ) and t v i,j = q ( w v i , B j ) , for every block in the partition P , to obtain the normalized probability distribution relating a view from image i with the prototypes of the j -th block of the random partition, where s v i,j and t v i,j ∈ R 1 × 1 × N P .

To ensure that views are consistent among the blocks, we optimize the views' distributions s v i,j and t v i,j over the prototypes of a block indexed by j , so that the two distributions are consistent with one another. Thus, the consistency term of our partition loss is L c ( s 1 i,j , t 2 i,j ) , where for each block

B j , the loss is minimized when the pair of views, ˆ x 1 i and ˆ x 2 i , gets assigned to the same prototypes across blocks. In other words, we look for the agreement between student and teacher assignments' probabilities across views of a given sample.

Following similar reasoning, the block-wise entropy term is defined as H (¯ p j ) , where ¯ p j = avg ( { ( s v i,j , t v i,j ) } N i =1 ) is the average prediction over each block B j for a batch of size N . Thus, the final objective for consistent assignment of random partition is,

Note that to fully use the pair of views at each iteration, we symmetrically use the L c consistency function. The probability vectors t v i,j come from the momentum teacher and are used as target distributions.

We can view the random partition pretext task as posing multiple pseudo-classification problems over subsets of prototypes. At each iteration, the pseudo-classification tasks change because the partitions are recreated with different prototypes, cf. Figure 2. One of the benefits of such a strategy is that we no longer require tuning the hyperparameter λ e to avoid trivial solutions. The stochastic nature of the random partition pretext task, blended with the multiple prediction tasks over subsets of prototypes, provides a regularization effect that improves the learned representations and training stability.

Other clustering-based methods [7, 8] rely on sharpening the distributions to improve their selfsupervised signals used as targets in a cross-entropy loss. On the contrary, our formulation does not require the temperature parameter for sharpening the predictions and guiding the learning of the student. We can think of the consistency loss as implicitly learning the temperature parameter to make the predictions sharper at each iteration. This is an important advantage of our consistency loss in contrast to previous methods.

5 Main results

5.1 Transfer learning evaluation

Table 1 shows that CARP's pre-trained representations are suitable for learning new downstream tasks over multiple datasets with distinct difficulty levels. We compare CARP's k -NN performance against nine SSL methods across eight datasets and report average results for k = {10, 20, 100, 200}, over all datasets. We advocate for k -NN instead of linear evaluation, where a linear classifier is trained on top of the frozen features, for the following reasons: (1) k -NN is faster, (2) k -NN demands fewer resources, and (3) k -NN requires less hyperparameter tuning. For individual references, Table 1 shows top-1 accuracy for k = 20 on each dataset. For fixed k = 20 , CARP outperforms competitors in 5 out of 8 datasets, and in the Flowers and Country datasets, it is a close second . Moreover, the average performances at k show that CARP performs comparably well on all datasets without significant performance differences when varying the number of labeled examples k . We show the detailed results in Appendix A.1.

5.2 Clustering evaluation

Table 2 reports clustering performance metrics of various clustering-based SSL methods on the ImageNet-1M [36], CIFAR-10/100 [27], and the GTSRB [40] datasets. For the ImageNet-1M, only 1% of the labeled data is used following the subset provided by Chen et al. [9]. See Appendix D.4 for the detailed k -means evaluation protocol.

5.3 Image retrieval and copy detection

Motivated by the strong k -NN performance of CARP and following the previous evaluation protocol by Caron et al. [8], we assess the performance of ImageNet pre-trained CARP encoders on image retrieval and copy detection downstream tasks. We took the officially released weights from the competing methods, used the frozen encoders as feature extractors, and performed retrieval using k -NN. For both tasks, the nearest neighbor computation is done on the 2048-dim representation from the ResNet-50 encoder. We report the top-3 best-performing methods. For an extended evaluation, see Appendix A.2.

Image retrieval. We consider the revisited Oxford and Paris landmark image retrieval datasets [34]. Given a query image of a landmark, the objective is to retrieve all database images depicting the same landmark. Each dataset contains three different difficulty levels. In Table 3, we report the mean Average Precision (mAP) on the Medium and Hard subsets for various SSL algorithms. CARP's representations perform well on Oxford 5k and take second place for Paris 6k. See Appendix D.5 for the evaluation protocol.

Copy detection. We benchmark self-supervised ResNet-50 encoders on the INRIA Copydays dataset [16]. In Table 4, we report the mean Average Precision (mAP) on the 'strong' subset and compare CARP's performance against other state-of-the-art methods.

5.4 Few-shot classification

Table 5 compares the few-shot classification performance of SSL methods on the VOC07 [17] and INat2018 [43] datasets. We train linear SVMs following Li et al. [29] and linear classifiers on top of the frozen representations from the self-supervised ResNet-50 encoders for VOC07 and INat2018, respectively.

The INat2018 dataset is especially challenging for low-shot classification. It contains 8142 classes with a long tail distribution, where the number of images per class varies between 1000 maximum and 2 minimum. CARP remains a strong feature extractor for both tasks while competitors oscillate between datasets. CARP pre-trained for 400 epochs demonstrates an efficient learning performance and wins most configurations. We report a complete evaluation with standard deviations in Appendix A.3.

5.5 Linear evaluation

Following the linear evaluation protocol proposed by Zhou et al. [49], we trained linear classifiers on top of CARP's frozen 2048-dim representations for 100 epochs.

We assess the linear evaluation performance of CARP's pre-trained representations on ImageNet-1M for three pre-training configurations, 100, 200, and 400 epochs, varying the utilization of multi-crop (mc) augmentation, Table 6. CARP surpasses SwAV on all pre-trained configurations, +0.4% on 100 epochs, +0.4% on 200 epochs, +0.4% on 400 epochs w/o multi-crop, and +0.4% on 400 epochs with multi-crop. Indeed, CARP's 400 epochs pre-trained representations perform on par with DINO and SwAV, both pre-trained for 800 epochs.

In addition, we evaluated CARP's representations using a weighted k -Nearest Neighbor ( k -NN) classifier. CARP achieves better k -NN performance than SwAV (+1.4%), DINO (+0.2%), and BYOL (+1.1%), all trained for 800 epochs or more. These results emphasize the efficiency of the proposed random partition pretext task.

6 Ablations

In this section, we assess whether a consistent assignment of views over random partitions benefits the learned representations and improves training stability. We ablate CARP's main hyperparameters to establish a good baseline for pre-training on ImageNet-1M. For ablations, we trained CARP using the full ImageNet-1M dataset for 50 epochs. The batch size was set to 256, the number of prototypes K = 65536 , and the number of random partition blocks N P = 128 . Hence, each block contains N B = 512 prototypes. We report results for single runs. See Appendix B for more results and Appendix C for implementation details.

6.1 Training CARP with different batch sizes

Most SSL methods [7, 8, 22, 48] report their best results when using substantially large batch sizes. In Table 7, we observe a similar pattern when training CARP. Our default configuration of 1024

observations yields a k -NN top-1 performance 10% higher than a batch size of 128. Table 7 confirms that training with large batch sizes benefits the learned representations. However, training with smaller batch sizes requires further tuning of other hyperparameters, such as the block size N B . Specifically, we observed that reducing the block size N B improves the learned representations when training with small batch sizes, which makes CARP robust to low-resource training.

6.2 Exploring different strategies to build partitions

Table 8 explores different ways of creating random partition blocks from the learnable prototypes. CARP's default strategy recreates the random partitions at every training step. In other words, for each iteration, we assign N B randomly chosen prototypes to the N P partition blocks. Table 8 contrasts CARP's default strategy with one in which the partition blocks are created only once, in a sequential manner, and kept fixed throughout training. We observe that training CARP with fixed partition blocks still produce useful representations. However, as measured by k -NN performance, randomly recreating the partition blocks at each iteration further benefits the learned representations. Since the partition blocks are randomly recreated at every iteration of gradient descent, the classification subproblems are always different. In practice, this variance allows for many unique pretext tasks at each iteration, which provides a positive regularization effect on CARP.

7 Limitations

Even though CARP's representations transfer well to many downstream tasks, our experiments showed that CARP's representations do not transfer well to dense prediction tasks such as detection and segmentation. We hypothesize this limitation is due to the architectural characteristics of ConvNets that collapse local feature maps to an average global representation in the last layer, combined with our classification-like loss function. Refer to Appendix A.4 for quantitative results.

8 Conclusion

We presented consistent assignment of views over random partitions (CARP), a self-supervised clustering-based method for visual feature learning. CARP learns prototypes in an online fashion end-to-end using gradient descent by minimizing a cost function that optimizes consistency between views' assignments and uniform distribution across prototypes within a random partition. Our experiments demonstrate that posing the optimization problem at the level of random partitions of learnable prototypes stabilizes training by avoiding trivial solutions in joint-embedding architectures and increases the performance of the learned representation. We compared the performance of CARP against the state-of-the-art ResNet-based SSL methods across multiple pretext tasks and datasets. The results demonstrated that the representations learned by CARP performed well on many visual downstream tasks.

References

Y.M. Asano, C. Rupprecht, and A. Vedaldi. Self-labelling via simultaneous clustering and representation learning. In Inter. Conf. Learn. Represent. (ICLR) , 2020.

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In IEEE Inter. Conf. Comput. Vis. (ICCV) , pages 8443-8452, 2021.

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In Inter. Conf. Learn. Represent. (ICLR) , 2022.

Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a 'siamese' time delay neural network. In Adv. Neural Inf. Process. Sys. (NeurIPS) , pages 669-688. World Scientific, 1993.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conf. Comput. Vis. (ECCV) , pages 132-149, 2018.

Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pretraining of image features on non-curated data. In IEEE Inter. Conf. Comput. Vis. (ICCV) , pages 2959-2968, 2019.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Adv. Neural Inf. Process. Sys. (NeurIPS) , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE Inter. Conf. Comput. Vis. (ICCV) , pages 9650-9660, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter. Conf. Mach. Learn. (ICML) , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 15750-15758, 2021.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In IEEE Inter. Conf. Comput. Vis. (ICCV) , 2021.

Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , volume 1, pages 539-546. IEEE, 2005.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Adv. Neural Inf. Process. Sys. (NeurIPS) , volume 26, pages 2292-2300, 2013.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Inter. Conf. Learn. Represent. (ICLR) , 2020.

Matthijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg, and Cordelia Schmid. Evaluation of gist descriptors for web-scale image search. In ACM Inter. Conf. Image Video Ret. (CIVR) , pages 1-8, 2009.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inter. J. Comput. Vis. , 88(2):303-338, 2010.

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Inter. Conf. Learn. Represent. (ICLR) , 2018.

Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-of-visual-words generation for unsupervised representation learning. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , 2020.

Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , 2022.

Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In Adv. Neural Inf. Process. Sys. (NeurIPS) , volume 17, 2004.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Adv. Neural Inf. Process. Sys. (NeurIPS) , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 9729-9738, 2020.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Trans. Big Data , 7(3):535-547, 2019.

Armand Joulin and Francis Bach. A convex relaxation for weakly supervised classifiers. In Inter. Conf. Mach. Learn. (ICML) , 2012.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Dong Hoon Lee, Sungik Choi, Hyunwoo J Kim, and Sae-Young Chung. Unsupervised visual representation learning via mutual information regularized assignment. In Adv. Neural Inf. Process. Sys. (NeurIPS) , volume 35, pages 29610-29623, 2022.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In Inter. Conf. Learn. Represent. (ICLR) , 2021.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Inter. Conf. Learn. Represent. (ICLR) , 2016.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 6707-6717, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 16640-16649, 2022.

Filip Radenovi´ c, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondˇ rej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 5706-5715, 2018.

Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza. Learning with average precision: Training image retrieval with a listwise loss. In IEEE Inter. Conf. Comput. Vis. (ICCV) , pages 5107-5116, 2019.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Inter. J. Comput. Vis. , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y.

Thalles Silva and Adín Ramírez Rivera. Representation learning via consistent assignment of views to clusters. In ACM/SIGAPP Symp. Appl. Comp. (SAC) , page 987-994, 2022. ISBN 9781450387132. doi: 10.1145/3477314.3507267.

Thalles Santos Silva and Adín Ramírez Rivera. Consistent assignment for representation learning. In Energy Based Models Wksp. (ICLRW) , 2021.

Thalles Santos Silva, Hélio Pedrini, and Adín Ramírez Rivera. Self-supervised learning of contextualized local visual embeddings. In Vis. Induct. Priors Data-Eff. Deep Learn. Wksps. (ICCVW) , 2023.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IEEE Inter. Joint Conf. Neural Netw. (IJCNN) , pages 1453-1460. IEEE, 2011.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Adv. Neural Inf. Process. Sys. (NeurIPS) , volume 33, pages 6827-6839. Curran Associates, Inc., 2020.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conf. Comput. Vis. (ECCV) , pages 268-285. Springer, 2020.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 8769-8778, 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inf. Process. Sys. (NeurIPS) , volume 30, 2017.

Guangrun Wang, Keze Wang, Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self-supervised representation learning. In IEEE Inter. Conf. Comput. Vis. (ICCV) , pages 9505-9515, 2021.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE/CVF Inter. Conf. Comput. Vis. Pattern Recog. (CVPR) , pages 3733-3742, 2018.

Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In Inter. Conf. Mach. Learn. (ICML) , 2021.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In Inter. Conf. Learn. Represent. (ICLR) , 2022.

A Extended results

In this section, we expand the evaluation experiments from the main text. We use the officially released pre-trained methods across all benchmarks to represent each SSL method. We did not retrain/reimplement any competing method. In total, we pre-trained 4 instances of CARP. To evaluate CARP's performance in a low training regime, we trained two 200-epoch models, one with multi-crop (mc) and the other without it. Similarly, to evaluate longer training performance, we trained two 400-epoch models, one with multi-crop and the other without it.

A.1 Transfer learning evaluation

In Table A.1, we report detailed results for the transfer learning k -NN experiments. We evaluate SSL methods for values of k ∈ { 10 , 20 , 100 , 200 } , including all instances of CARP. Cf. Appendix D.3 for the evaluation protocol.

CARP achieved either top-1 or top-2 performance in seven out of 8 datasets. STL-10 is the only dataset where CARP is neither the top-1 nor top-2. Moreover, in the FGVCAircraft, Stanford Cars, and GTSRB datasets, CARP archived top-1 and top-2 performances with large margins. The average k -NN performance per value of k across all datasets is reported in Table 1 in the main text.

A.2 Image retrieval and copy detection

Tables A.2 and A.3 show additional results for image retrieval and copy detection evaluations. For both tasks, we compare CARP's performance against nine SSL methods. We report mAP on the Medium and Hard splits of the revisited Oxford and Paris datasets for image retrieval. We report mAP on the 'strong' set of the Copydays dataset for copy detection.

A.3 Few-shot evaluation

Table A.4 shows additional few-shot evaluation results on the Pascal VOC07 and INat-2018 datasets. We report mAP and top-1 accuracy @ k , averaged over 5 independent runs, for VOC07 and INat-2018, respectively. We include CARP's 200 and 400 multi-crop models. Cf. Appendix D.2 for details on the evaluation protocol.

A.4 Dense prediction evaluation

We evaluated our CARP's multi-crop model on the object detection downstream tasks using the Pascal VOC07 dataset. We followed the guidelines from He et al. [24]. We fine-tuned for 24k iterations on PASCAL VOC trainval07+12 and evaluated on test2007. CARP performs at AP=44.2, AP50=79.4, and AP75=47.6; results are averaged over 5 trials. Compared to other SSL methods, such as MoCo-v2 [11] (AP=57.4 AP50=82.5 AP75=64.0), CARP underperforms significantly.

B Ablations

Due to a limited execution budget, the ablations and the main experiments differ slightly in some hyperparameters. Here, we describe the configurations used for the ablations-for the main experiments, see Appendix C.1. For ablations, we trained CARP using the full ImageNet-1M dataset for 50 epochs. The projection head learns a latent representation of 128-dim. The batch size is set to 256 observations, and the projection head hidden layers contain 2048 neurons. We set the number of learnable prototypes K = 65536 , and the number of random partition blocks N P = 128 . Hence, each block contains N B = 512 prototypes. We report results for single runs.

B.1 Does the number of learnable prototypes affect the learned representations?

Table B.1 examines the effect of training CARP with different configurations of prototypes K . Similar to other clustering-based SSL methods [7, 8, 29], CARP also benefits from over-clustering. As the number of trainable prototypes K grows, the k -NN performance of the learned representations increases. In addition, note that if the number of prototypes K is smaller than the number of actual

classes in the dataset, the k -NN performance of the learned representations degrades. Based on these experiments, we set the default number of prototypes K = 65536 for the ImageNet-1M dataset.

B.2 Does the number of partition blocks matter?

To better understand the effect of the hyperparameters N P and N B on the learned representations and in the training stability, the first row of Table B.2 demonstrates the performance of CARP using different configurations for the number of partition blocks N P and their sizes N B . For completeness, we analyze the effect of removing the momentum encoder in Appendix B.3. We also present an ablation on the effect of the momentum update in Table B.3.

Specifically, as the partition sizes grow and the number of partition blocks N P decreases, the quality of the learned representations tends to decline and eventually collapse. Note that setting a partition size N B = 65536 produces a single partition block N P containing all prototypes. Precisely, the setup in the last row and last column of Table B.2 is equivalent to CARL [38]. It shows that a naive implementation leads to a collapsed solution, and the divide-and-conquer approach of devising random partitions from the learnable prototypes avoids such trivialities.

Note that as smaller the block size N B , more stable the algorithm will be. However, the quality of the learned representation might decrease since the pseudo-classification tasks, posed by the random partitions, becomes easier with fewer prototypes. On the other hand, a larger block size N B poses a more challenging consistency task at the expense of contributing to mode collapse.

For most cases, however, for block sizes ranging from N B = 128 to N B = 4096 , CARP learns useful representations and shows robustness to this hyperparameter. By default we set the partition block size N B = 512 .

B.3 The importance of the momentum encoder

Table B.2 contrasts CARP's joint-embedding architectures with and without a momentum encoder, which is equivalent to setting η = 0 in the momentum encoder update equation. Different from other SSL methods [8, 22], CARP works with both setups. However, we observe that using a momentum encoder significantly boosts the performance of the learned representations. Table B.2 shows that regardless of block sizes, representations learned using a momentum encoder-based architecture consistently outperform the siamese counterpart.

B.4 Who provides the best features for downstream evaluation?

One way to understand CARP's joint-embedding architecture with a momentum encoder is through the teacher-student framework, where the momentum encoder is the teacher that guides the learning student. The addition of the momentum encoder raises the question of which module produces the

best representations. To answer this question, Figure B.1 explores the k -NN performance when extracting features from the momentum encoder (teacher) versus the student. We observe that teachers' representations constantly outperform the students' during training. However, by the end of the training, the student catches up with the teacher.

C Implementation Details

C.1 Experimental Setup

We train CARP on the ImageNet-1M unlabeled dataset using ResNet50 [23] encoders. We take the output representation of the last global average pooling layer (a 2048-dim vector) and project it to a 256-dim vector. Following Caron et al.'s [8] work, our MLP projection head contains 3 dense layers with batch normalization and the GELU activations. The hidden units of the projection head contain 2048 neurons. The 256-dim representation vector is fed to an assigner MLP that outputs unnormalized probabilities w.r.t. the learnable prototypes. By default, the assigner function is implemented as a linear layer and trains K = 65536 prototypes. To generate the random partitions, we set the number of partitions N P = 128 , which creates subsets containing N B = 512 randomly chosen prototypes. We use the same data augmentations proposed by Grill et al. [22] to generate synthetic views. The protocol creates three data augmentation pipelines, the first two to generate global views and the last to generate multi-crops. CARP is pre-trained with the LARS [47] optimizer, end to end, with weight decay of 1 × 10 -6 . For models training up to 200 epochs, the learning rate starts from 0.6 and decays

to 0.006 with a cosine scheduling [30] without warmups. For models pre-trained for more than 400 epochs, the learning rate starts at 0.3 and decays to 0.003 using the same cosine scheduler. We train the system with a global batch size of 4096 observations. For all experiments, we used 4 A100 40GB GPUs and gradient accumulation to simulate large batch sizes. Cf. to Appendix E for a PyTorch style pseudo-code.

D Evaluation Protocols

D.1 Linear evaluation

For ImageNet-1M evaluation, we trained a linear classifier on top of the frozen representations extracted from the last average pooling layer of the ResNet50 encoder for 100 epochs, following Zhou et al.'s [49] protocol. The evaluation script performs a grid search hyperparameter tuning on the learning rate, weight decay regularization, and optimizer. For each input image, we take a random crop followed by a resize to 224 × 224 and an optional horizontal flipping. For testing, images are resized to 256 × 256 and center-cropped to 224 × 224 .

D.2 Few-shot evaluation

We measure the few-shot learning capabilities of SSL methods on the Pascal VOC07 and iNaturalist 2018 datasets. For VOC07, we are interested in the multi-label classification performance. We closely follow Li et al.'s [29] protocol and train Linear SVNs on fixed 2048-dim representations from many SSL ResNet-50 encoders.

For INat-2018, we expand the few-shot evaluation challenge to a complex scenario containing more than 8k classes. We train linear classifiers for 20 epochs on fixed 2048-dim representations. We use

25

50

75

100

52

54

56

58

60

62

Training epochs

k

-NN top-

1

accuracy

Student (

θ

)

Teacher (

ξ

)

the SGD optimizer. The learning rate starts at 0.03 and decays by a factor of 10 in epochs 12 and 18, respectively.

For both datasets, we vary the number n of labeled examples per class and report the average results across 5 independent runs.

D.3 k -NN evaluation

To perform the k -NN evaluation, we use pre-trained SSL ResNet50 encoders as feature extractors to compute and store the representations from many vision datasets. Following Caron et al.'s [8] setup, the representation vector for a test image is compared against all representations from the training split and a prediction is made via weighted voting. If one of the closest neighbors has the same class as the test image, it contributes to the final voting as α i = exp ( M i z τ ) where M is a memory bank containing representations from the training data, z is the representation from the test data, and τ is the temperature hyper-parameter. For all experiments, we run k -NN with configurations of K near ∈ { 10 , 20 , 100 , 200 } . For most experiments, a value of k = 20 is consistently the best setup across all methods.

D.4 k -means evaluation

Similar to k -NN evaluation, we take self-supervised pre-trained encoders and use them to extract 2048-dim feature vectors from the training set of datasets like CIFAR-10/100 and ImageNet-1M. For ImageNet-1M, we use only 10% of the training data following the 10% subset from Chen et al. [9]. We fit k -means classifiers on the learned representations of the training set and use the validation split to assess the quality of the learned prototypes. We report three metrics to assess clustering performance: Normalized Mutual Information (NMI), Adjusted Mutual Information (AMI), and Adjusted Rand Index (ARI). The number of prototypes k is set to be the number of true classes of each dataset. We use the faiss library [25] for fast k -means. For each experiment, we run k -means for 100 iterations, 20 redos, and spherical normalization. To measure the clustering performance of CARP, we observed that a 400 epoch model with a learning rate of 0.3 slightly outperformed the other instances; therefore, we use this model to report results in Table 2. This instance of CARP uses two views and is only used for clustering evaluation.

D.5 Image retrieval evaluation

We strictly follow the evaluation script eval_image_retrieval.py provided by Caron et al. [8], for the image retrieval evaluation task. The script uses the revisited Oxford and Paris image retrieval datasets [34]. The dataset contains three protocols of varying difficulty levels. We take the ImageNet pre-trained ResNet-50 encoder from CARP, freeze the weights, and apply k -NN evaluation directly to the frozen 2048-d features for retrieval, conditioned on a query image.

D.6 Copy detection evaluation

We strictly follow the evaluation script eval_copy_detection.py provided by Caron et al. [8] for copy detection evaluation. The evaluation is performed on the INRIA Copydays dataset [16]. The dataset contains holiday pictures in the format query/database. Each image has suffered three kinds of artificial attacks: JPEG, cropping, and 'strong.' We report performance evaluation on the 'strong' subset. Images in the 'strong' subset were intentionally distorted by blur, insertions, print, and scan. The task is to recognize these images despite distortion. We take the frozen CARP ResNet-50 encoder and extract 2048-dim vectors from query and database images at resolution 320 2 . Then, we perform copy detection with cosine similarity between query and database features. We report mean average precision (mAP) as a performance metric. Unlike the benchmark described by Caron et al. [8], we do not utilize additional distractors, nor do we centralize the data using statistics learned in a different set on images.

E Pseudocode of CARP in a PyTorch-like Style

# NB: # K: number of prototypes

number of random prototypes within a block

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for self-supervised learning based on cluster assignments. The method is based on a consistent assignments approach that assigns the same prototype to different views of the same image. To overcome issues of the previous methods that are not well scalable, a divide and conquer approach is proposed. Radom partitions of the learnable prototypes are created each iteration/epoch that allows not only make the training to be stable but also faster. The authors show the efficiency of the approach across a wide range of different datasets and settings.\n\nSTRENGTHS:\nS1: The paper is well written, it’s clearly stated what is novel and what is borrowed from the previous methods, and the explanations of the approach are easy to follow.   \nS2: The idea of creating subtasks to reduce computation time and avoid collapse sounds novel.  \nS3: As it is empirically demonstrated, the proposed idea improves the performance over multiple benchmarks on kNN and retrieval tasks while being on-par with linear probe.\n\nWEAKNESSES:\nW1: would be interesting to see how the methods work with transformer based architectures. e.g. within moco-v3 framework?   \nW2: the contributions are not clearly stated apart from the first one: citation [9] in contribution is wrong, it’s SimCLR contrastive approach that should belong to negatives. Moreover, the authors use entropy maximization as a form of normalization which the authors even state as a separate contribution. But this is cannot be a separate contribution because it was used before as a technique to avoid trivial solutions e.g. in [39].  \nW3: the Fig.7 is not helpful at all. It was not possible to understand what it is depicted there. The text (not caption) is clear though.   \nW4: In the intro, the authors introduce two classes of SSL. It is too restrictive division and does not cover all SSL methods. Where e.g. MAE lies?   \nW5: [36] is not really recent while the authors refer to it as “were recently proposed” on L41\n\nQUESTIONS:\nQ1: what if to have hard pseudo-labels instead of soft assignments?   \nQ2: L218 - it is an interesting observation, could the authors make an analysis and confirmation of that?  \nQ3: what is the value of \\lambda_e, no information in implementation details? how does the performance change if to change this parameter?  \nQ4: could the authors also add a discussion on the differences to SwAV?\n\nLIMITATIONS:\nThe authors address limitations in the form of discussion wrt the previous work based on which they propose the new approach. The section provides nice insights into the limitations of the previous method but also shows rather limited novelty of the current approach over the previous one. The method section is rather describing the previous approach as a new one that can be confusing. However, I find the proposed divide-and-conquer approach very interesting and helpful based on the results. It would be helpful to rewrite partially the paper making more focus on the proposed contribution rather than on the previous method with additional analysis/confirmation of all the statements.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extends the existing work about consistent assignment for representation learning by introducing random partitions. Specifically, when the number of prototypes is large in clustering-based self-supervised learning, the work shows that CARL [36] cannot handle the loss and the regularization well. To mitigate the problem, CARP is proposed to obtain subsets of prototypes and then apply CARL on the partitions of prototypes. Standard evaluation protocol is included to compare the proposed method to benchmark methods.\n\nSTRENGTHS:\n1. The clustering-based method shows the potential for self-supervised learning. \n2. The strategy of random partition can avoid the problem from a large number of prototypes. \n3. Experiments are extensive on diverse data sets.\n\nWEAKNESSES:\n1. As a closely-rated work, the comparison to CARL lacks, which makes the contribution of this work unconvincing. Moreover, the original paper of CARL shows that it works well with 10000 prototypes, which contradicts the claim of this work. \n2. The performance of clustering-based method in the comparison is out of date. CoKe [1] as an online clustering method demonstrates 74.9% and 76.4% linear probing performance on ImageNet with two-view and multi-view augmentations, respectively. Moreover, CoKe does not request an additional momentum encoder and a sharpening temperature for the pseudo-label from the other view, which makes the contribution of this work less significant. \n3. While CARP shows a competitive performance with less training epochs, it is better to show if it can achieve a better performance with the long training strategy. \n\n[1] Unsupervised Visual Representation Learning by Online Constrained K-Means. CVPR 2022.\n\nQUESTIONS:\nBesides the weakness, my major concern is about the limited contribution compared to CARL.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on self-supervised representation learning. Under the setting of consistent clustering assignment between augmented views (SwAV-like), the authors found that when the number of prototypes is significantly larger than the batch size, the commonly used technique for avoiding trivial solutions fails. And they thus propose to randomly partition the prototypes into multiple subgroups to avoid the trivial solution. Experiments on extensive benchmarks, especially retrieval-based ones, validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n*Originality:* The major insight of this paper: entropy regularization fails when the prototype number is significantly greater than the batch size ($K \\gg N$) is novel, and the solution that divides the prototypes into multiple subgroups, is straightforward and verified to be useful. Other components of this framework resemble multiple previous works (*e.g.*, SwAV, DINO, MSN).\n\n*Quality:* The idea is clearly formulated and presented, and the method is well evaluated in extensive experiments, making it in shape of a solid paper.\n\n*Clarity:* The idea is straightforward and the delivery is clear enough, reading it is smooth and I do not find trouble in understanding.\n\n*Significance:* It tackles the assignment strategy in clustering-based self-supervised learning, and provides a solution for $K \\gg N$, which is somehow useful.\n\n*Reproducibility:* Code is provided in the supplementary material to facilitate reproduction.\n\nWEAKNESSES:\n*Motivation*: Why do we need so many prototypes during pre-training? The major stream of self-supervised learning is to enlarge the batch size for better representation learning. The prototype number, in contrast, has not shown the necessity for very large numbers yet. In fact, in DINOv2 they have reduced $K$ from 65536 in DINOv1 to 4096, which is competent for learning good representations. Moreover, large number of prototypes means high computational cost in pre-training. And after pre-training, they are simply dropped and not used in downstream tasks. So why do we need so many prototypes?\n\n*Technical Contribution:* In terms of technical contribution, this work proposes to randomly divide the prototypes into subgroups. The idea is clear and straightforward but the contribution might be limited.\n\n*Evaluation*: The evaluation is mainly focused on $k$-NN or linear probing on retrieval-based benchmarks. However, for one thing, a good deep representation does not have to be linear, and performance under full fine-tuning (transfer learning) might be of higher interest. For another, one might be more interested in generic classification benchmarks (ImageNet) and dense prediction downstream tasks.\n\nQUESTIONS:\n- I noticed that different tables may refer to different variants of CARP (or CARP w/ mc). Why is it and why multiple cropping does not help in e.g., Tab. A.1?\n- One advantage of clustering-based methods is that they tend to suit ViTs better (e.g., in DINO). Why is this work restricted in ResNets?\n\nLIMITATIONS:\nlimitations are discussed in Sec. 7", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a collapsing problem that arises in clustering-based contrastive learning. To resolve the problem, the paper proposes an improved version of consistent assignment in CARL, utilizing a strategy of random partitioning. In particular, the original consistent assignment loss exhibits a stability issue when a large number of prototypes are employed; thus, it is challenging to determine proper hyperparmeters to ensure both stabilization and strong performance. This paper introduces a method of random partitioning among the trainable prototypes and applies the consistent assignment loss across each partition. Such approach effectively mitigates the instabilty problem associated with a large number of clusters while maintaining the benfits from using numerous clusters. Importantly, the proposed method does not introduce any additional hyper-parameters, such as trade-off parameter or sharpening temperature. The experimental results demonstrate the effectiveness of the proposed method in various downstream tasks such as transfer learning, clustering, image retrieval, copy-detection, few-shot classification, and linear/k-NN evaluation.\n\nSTRENGTHS:\n* The method seems simple yet effective, achieving robust performance and preventing collapse without the need for additional hyperparameters.\n* The experiments are conducted across various downstream tasks, including transfer learning, clustering, image retrieval, copy-detection, few-shot classification, and linear/k-NN evaluation; the method consistently demonstrates strong performance.\n* The method has been meticulously ablated (in main text and supplementary) with various implementation choices thoroughly examined.\n* The paper is well-composed and easy to follow.\n\nWEAKNESSES:\n* The discussion and comparison to other related baseline methods [1, 2, 3], which similarly employ the entropy of the mean probabilities of a batch while addressing a stabilization of it, seem to be missing.\n\n[1] Self-Supervised Learning by Estimating Twin Class Distributions, arxiv 2021 \\\n[2] Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment, NeurIPS 2022 \\\n[3] Masked Siamese Networks for Label-Efficient Learning, ECCV 2022\n\nQUESTIONS:\n### Suggestion\n* Please address the concerns listed in the weaknesses.\n* Given that all experiments are conducted using ResNet-50, experimental results with Vision Transformers (ViTs) shall strengthen the contributions of the paper.\n\n\n### Question\n* Why is it important or beneficial to avoid using non-differentiable modules, such as Sinkhorn-Knopp, for generating target cluster assignments? (L4-5, L99-100, L101-103)\n\nLIMITATIONS:\nThe paper describes a limitation of the proposed method that the representations learned through CARP do not transfer well to dense prediction tasks. This limitation seems logical, as the proposed method simulates smaller pseudo-classification problems, which might be more suitable for downstream tasks such as classification, clustering, and retrieval.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}