Clustering Properties of Self-Supervised Learning
Authors: Xi Weng
Jianing An
Xudong Ma
Binhang Qi
Jie Luo
Xi Yang
Jin Song Dong
Lei Huang
Venue: ICML 2025 poster (2025)
Keywords: self-supervised learning
joint embedding architectures
clustering properties
positive-feedback learning
Primary area: deep_learning->selfsupervised_learning
TL;DR: This paper proposes a novel positive-feedback SSL method, which leverages the model's clustering properties to promote learning in a self-guided manner.

Abstract: Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output *encoding* exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed **Re**presentation **S**elf-**A**ssignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful.

Clustering Properties of Self-Supervised Learning
Xi Weng 12 Jie Luo!
Jianing An Xi Yang?
Xudong Ma Jin Song Dong?
Binhang Qi Lei Huang ter
Abstract
model
Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich represen- tations with strong clustering properties, magi- cally in the absence of label supervision. De- spite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metries that the encoder's 's output encod- ing exhibits superior and more stable clustering properties compared to other components. Build- ing this insight, we propose a novel positive- feedback SSL method, termed Representation Self-Assignment (ReSA), which leverages the model's clustering properties to promote learn- ing self-guided manner. Extensive experi- ments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-ard SSL methods by significant mar- gin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shap- ing representations that are inherently more struc- tured and semantically meaningful.
generate
guide
Positive feedback
fo-)
design
loss function
Representations
Figure I. The positive-feedback SSL framework. It involves the model generating representations that possess semantically clus- tering information. This clustering information is leveraged design self-supervised loss function, which is then employed to more effcetively guide the model's learning process.
under different transformations and noise (Bachman et al., 2019; He et al., 2020; Chen et al., 2020a), with demonstrated exceptional effectiveness in visual representation learning. Although such pretext task may intuitively seem unrelated to capturing semantic relationships. extensive studies (Caron et al., 2018; 2021; Assran et al., 2022b) have demonstrated the strong correlation between its learned representations and semantic information.
Ben-Shaul et al. (2023) take further step characterize the semantic structures learned by JEA into hierarchic clus- tering properties, indicating SSL-trained representations exhibit a centroid-like geometric structure and induce three levels of semantic clustering: augmentation sample level, semantic classes, and superclass level. This intriguing find- ing reveals that SSL methods based on JEA can facilitate strong clustering capabilities during training, but also raises the question of whether these properties hold untapped po- tential that can be further leveraged to improve SSL itself.
1. Introduction
Self-supervised learning (SSL) has emerged as transfor- mative paradigm in universal representation learning (Oord et al., 2018; Bachman et al., 2019; He et al., 2020; Chen et al., 2020a; Bao et al., 2021; Oquab et al., 2023; Assran et al., 2023), consistently surpassing supervised learning in downstream performance. Joint embedding architectures (JEA), particular, aim to learn invariance of the same data
Contributions. In this paper, we aim to investigate the design of SSL methods by leveraging the inherent cluster- ing properties of representations, enabling closed-loop mositive-feedhace SSI. framework, as illustrated in Figure 1. To achieve this goal, we propose three key questions and our main contributions are summarized as follows:
'SLCCCSE, School of Artificial Intelligence, Beihang Uni- versity" 'School of Computing. National University of Singapore Beijing Academy of Artificial Intelligence 'Hangzhou Interna- tional Innovation Institute, Beihang University. Correspondence to: Lei Huang <huanglciAI buaa.uaad.. edu.en>. Our code available at htttss:/// /github.... com/ winci-aii resa
Where to extract clustering properties from? We demonstrate through various metrics that the encoder's output, referred to as encoding, exhibits superior and more stable clustering properties compared to other
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Clustering Properties of Self-Supervised Learning
components, such embedding and the hidden layer outputs within the projector.
SimCLR MoCaVa
BYOL SimSiam
DINO SWAV
BowlowTiwins VICReg
How to leverage clustering properties? We pro- pose a novel SSL method, termed Representation Self- Assignment (ReSA), which employs an online self- clustering mechanism to leverage the model's inher- ent clustering properties, thereby facilitating positive- feedback learning. Standard experiments demonstrate that ReSA surpasses existing state-of-ther methods in both performance and training efficiency.
0.5
0.06
0.16
0.03
0.4
0.14
0.00
0.12
-0.03
Value
0.10
Whether it facilitates better clustering properties? We examine whether and how ReSA facilitates better clustering properties, demonstrating that it excels at both lineggrained and coarse- grasing learning, shaping representations that are inherently more structured and semantically meaningful.
0.2
-0.06
0.08
-0.09
0.06
0..12
0.04
0.0
ARI 0.15
2. Background, Related Work, & Notation
5Cmean
SCat
Figure 3. Comparison of clustering metrics in encoding H and embedding Z across various self- superviee pretrained models. methods utilize a ResNet-18 encoder pretrained on CIFAR-10 for 1000 epochs. Circular markers represent metries computed using encodings, while cross markers correspond to metrics derived from embeddings. All metrics are computed on the entire training set, and similar trends can be observed in the validation set.
2.1. Self-Supervised Learning
H encoding
Z: embedding
Z
Encoder
Projector
B
l
T
Encoder
Projector
we have Fa(-) Go (Ee. with learnable parameters = (0.. 0g). is worth noting that extensive work (Chen et al., 2020a; Gupta al., 2022) has demonstrated that us- ing the encoding as the representation for downstream tasks achieves much better performance than using embedding, that the projector is only used during the pre-training process and discarded in inference.
Z'
Figure 2. The basic notations for joint emhedding architectures
(JEA) in SSL.
Joint embedding architectures (JEA). Let B denote mini-batch input sampled uniformly from a set of images D. and T denote the set of data transformations available for augmentation. We consider pair of neural networks Fa and F² parameterized by 0 and 0' respectively. They take as input two randomly augmented views, X T(B) and X' (B), where T, T' E T;; and they output the embeddings Z F& (X) and Z' (X)). The networks trained with an objective function that minimizes the distances between embeddings obtained from different (two) views of the same images:
Self-supervised paradigms. The main challenge with JEA is representation collapse, where both branches pro- duce identical and constant outputs regardless of the inputs. Numerous paradigms have been proposed to avoid collapse, including contrastive learning methods (Chen et al., 2020a; He et al., 2020; Chen al., 2020b; 2021; Liu et al., 2022) that attract different views from the same image (positive pairs) while pushing apart different images (negative pairs), and non non-contrastive approaches (Grill et al., 2020; Caron et al., 2020; 2021; Weng et al., 2024) which directly align positive targets without incorporating negative pairs. Al- though Ben-Shaul et al. (2023) had demonstrated that the encodings learned through SSL are highly correlated with semantic classes and exhibit strong clustering capabilities, few methods have leveraged this clustering ability to facili- tate
C(B.0) 0) = EB-D, TI ((Fo(T (T
(1)
where ((.,)) is a loss function, which aims to learn invari- ance data transformations.
In particular, the JEA usually consist of shared encoder Eg, (.) and projector Go (.), commonly referred to as Siamese Network (Chen & He, 2021). As shown in Figure 2, their outputs H = Ee, (X) E dd. Xm and Z = Go, (H) E Rd xm are referred to as encoding and embedding, respec- tively (where m is the mini-batch size, de and dg are their corresponding feature dimensions). Under this notation,
learning. A closely related work (Ma et al., 2023) exploits the encoding' 's augmentation robust- ness to re- reigei the positive alignment the SSL objective functions; however, it still overlooks the rich clustering in- formation inherent in the encodings.
2

---

Clustering Properties of Self-Supervised Learning
SimCLR
1212
022
601
Spach
Epech
Epeck
8pasth
SWAV
Spach
Speckh
Epoch
vICReg
Linee
Spach
Spoch
Figure Comparison of linear evaluation accuracy and clustering metrics of encoding H. embedding Z, and the hidden layer oulputs within the projector P during the training process. The experiments are conducted using SimCLR, VICReg, and SwAV, employing ResNet-18 encoder pretrained on CTFAR- 100 for 500 epochs. The projector a standard three-layer ML. BN ReLU activations, containing two hidden linear layers, so their outputs are denoted as Po and P1.
2.2. The Information Distinction in JEA
3. Exploring Clustering Properties of SSL
The projector has become an indispensable component of JEA-based SSL. However, the theoretical dynamics of optimization and the reasons behind its success remain an open question within the community. Some works have attempted explain these principles. For example, Jing et al. (2022) empirically discovered that applying SSL loss either encoding or embedding led to a significant decrease in the rank of the corresponding features. They argued that this rank reduction indicates a loss of diverse information, which, in turn, reduces generalization capability. This expla- nation aligns with the hypothesis in SimCLR (Chen al., 2020a), where the additional projector acts as a buffer prevent information degradation of the encoding caused the invariance constraint. Additionally, Gupta et al. (2022)'s 's null space analysis for the projector posited that the pro- jector might implicitly learn to select a subspace of the encoding. which is then mapped into the embedding. In this way, only subspace the encoding encouraged to be dhyle-invariant, while the other subspace can retain more useful information.
Ben-Shaul et al. (2023) find that within the encoder of JEA, the clustering ability of features improves progressively as intermediate layers get deeper. However, it remains un- clear whether the projector exhibits a similar trend. To quantitatively evaluate the clustering performance of these components, we employ widely recognized metrics such the Silhouette Coefficient (SC) (Rousseeuw, 1987) and Adjusted Rand Index (ARI) (Hubert & Arabie, 1985). In particular, larger mean value of SC (SCmean) indicates stronger local clustering ability in the representation, and a smaller standard deviation (SCau) reflects better stabil- ity local clustering ! Meanwhile, higher ARI values correspond enhanced global clustering properties. The detailed introduction of these metrics can be found in the Appendix A.
Using these metrics, we first evaluate the clustering abili- ties of encoding H and embedding Z across various self- supervised pretrained models in CIFAR-10 10 (Krizhevsky et al., 2009) dataset, which contains only classes and commonly used for cluster analysis (Ben-Shaul al., 2023). It evident in Figure 3 that, across most SSL models, encod- ings achieve visibly higher ARI and SCmcan as well lower SCaa values, compared to embeddings. These observations reflect common grounds of SSL models: Encodings not only
Therefore, the SSL constraint can cause the embedding to lose information, which may include not only clustering- irrelevant features such as background information, but also class-relevant information, making it difficult to determine which encoding or embedding exhibits better clustering performance in this context. such cases, this paper first an- alyzes the differences between the in terms of clustering properties empirically.
'The 'clustering ability' refers to how well the vectors represent the underlying structure of the data, while 'stability' sig- nifies that clustering results are more consistent across data points, meaning fewer outliers and more stable cluster assignments.
3

---

Clustering Properties of Self-Supervised Learning
Assignment (ReSA) guide the loss function among em- beddings Z and Z'. See Figure for the clear framework.
sug.
grad.
cross entropy
X
H
Z
B
weak
Encoder
Proj
4.1. Online Self-Clusteing
H
Following notations in Section 2.1, we apply the encod- ing H as the representation to perform clustering. Unlike previous approaches, e.g. SwAV (Caron et al.. 2020) and DINOv2 (Oquab et al., 2023) employing learnable proto- types to map features into the clustering space, we treat samples in H = [h..... hm] simultaneously as points to be clustered and as anchors. In details, we first calculate the cosine self-similarity matrix by Sμ HH' H, where samples in H are nornonallee as h:/ || l|,, Vi. Then the online clustering assignment Aи is computed upon SH using the iterative Sinkhorn- Knopp (Cuturi, 2013) as shown in Algorithm 1.
Ah
H
z'
Kiectors
standard
Encoder
Proj
X
H
Z'
Figure 5. The framework of Representation Self- Soilgmmsignmnnt (ReSA). Here, no grad. denotes that the operation does not in- volve gradient propagation, norn signifies that each sample is Lz-normalized compute cosine similarities, and sinkhorn refers to the Sinkhom-Knopp algorithm used for clustering assignment.
possess richer semantic information but also demonstrate high-quality clustering properties. These include excellent local clustering ability (SCmean) and stability (SCsu), global clustering capability and similarity measure effectiveness (ARI). An exception to this pattern is observed with SWAV and BYO,, whose embeddings also perform well across these metrics. We speculate that this may be due to the design of their loss functions, which enables the embed- dings to learn effective clustering properties. For instance, SWAV learns by predicting cluster assignments directly on the embeddings.
Algorithm 1 Sinkhorn- Algorithm
Require: Cosine self-similarity matrix Sμ E Rm reg- ularization parameter e > 0, all-ones column vector 1 iteration count T, Hadamard product o
Ensure: Doubly stochastic matrix Ah
Initialize Q
exp(SH /c)
Σ., exp(SSxr4)
Initialize marginals: c f m 4m
for 1 to T do Compute row sums: <- Q1m
Normalize rows: Q x Qo (9] I
To gain deeper insights into the evolution of clustering prop- erties of each component during training, we conduct exper- iments on the CIFAR-100 100 (Krizhevsky et al., 2009) dataset, which features a more complex set of categories, using three methods: SimCLR, VICReg, and SwAV. The results are shown in Figure 4. Overall, the encodings demonstrate a consistent improvement in all clustering metrics through- out training. In contrast, the ones of embeddings degrade significantly during the later stages of training. Moreover, the clustering metrics of the hidden layer oulputs within the projector show notable differences and weaknesses com- pared to those of the encodings, despite Po being separated by only a single linear layer and achieving nearly the same linear evaluation accuracy as the encodings.
Compute column sums: v 4- Q 1m
Normalize columns: Q f Q ($) end for
Normalize columns again: Q < Q o 1m ( Return Ah Q
We follow SWAV which uses only 3 iterations and sets the regularization parameter 0.05. This algorithm does not involve gradient propagation, enabling it to be efficiently implemented on GPUs (Caron et al., 2020). After obtaining the doubly stochastic matrix Aи as the assignment, it can naturally be utilized to guide relationship between the em- beddings Z and Z'. Specially, we use the cross-entropy loss to promote the learning process:
In summary, the experiments above demonstrate that encod- ings exhibit superior and more stable clustering properties compared to embeddings and the hidden layer outputs within the projector across various SSL models. This finding high- lights the potential of leveraging encoding as the optimal representation for clustering, providing a foundation for de- signing assitive-feedback SSL systems that capitalize these robust clustering properties.
CReSA
1 2mn
Σ i,j
Ah o log D(Z
(2)
Σ f,j
A o log D(Z" 2)
4. Leverage Clustering Properties for Positive-Feedback Learning
exp(Z Zz/ where ZZ) exp Z//r1mm and D Z) are prob ability distributions derived through the softmax function. T is a scalar temperature hyperparameter, o stands for Hadamard product, and 1 m is the all-ones column vector.
Based on above analyses, we design a novel positive- feedback SSL method, which derives Representation Self-
4

---

Clustering Properties of Self-Supervised Learning
Comparison to SwAV. As pioneering SSL method based on online clustering, SwAV (Caron al., 2020) em- ploys 'swapped" prediction mechanism (which is also adopted by DINOv2 (Oquab et al., 2023)), where the cluster assignment of one view is predicted from the embedding of another view. This is achieved by minimizing the following objective:
observed by Wang & Isola (2020), the hardness-aware prop- erty discovered by Wang & Liu (2021) through gradient analysis, and the probabilistic model of InfoNCE derived by Bizeul et al. (2024) from the perspective of mutual infor- mation. Since the motivation for our proposed ReSA does not have a direct connection with the evidence lower bound of mutual information, in this section, we provide an intu- itive gradient analysis to further understand the optimization mechanism of ReSA.
ESWAV=
1 2m
ΣΦ's
log D(Z C)+
+(+
Σ Q o log D(Z'TC) CI) 1.
Given the Lz-normalized embedding vectors Z [Z....., Zmd and Z' = [z'....,,zi. zn the InfoNCE formula on 2; can be writen as (omitting the symmetric terms):
where C' RRd, is the prototype matrix learned by back- propagation, and Q sinkhorn(Z C) is the cluster assign- ment using Brnthhrnn-noope algorithm. The key differences and advantages of our ReSA compared to SWAV (and DI- NOv2) can be summarized as follows: (1) ReSA computes clustering assignments on encoding with high-quality clus- tering properties, whereas SwAV performs it on the less stable emhedding. (2) AWAV requires learnable prototypes, which often necessitate complex design strategies, such as freeziig prototypes during the early stages of training and use a large number of prototypes K' to ensure stability, whereas ReSA directly extracts clustering information from the representations. (3) SwAV executes the Sinkhorn-Knopp algorithm multiple times, corresponding to the number of global augmented views. In contrast, ReSA only requires single execution of this algorithm regardless of the number of augmented views. This highlights the efficiency of ReSA, particularly under multi-crop scenarios.
exp($1./T) Ex= exp(3L,//T)
(IfoNCC(Zz)) -logg
(5)
where 8LJ = 2, z/y, V. Defining the probability Pi.j
exp(s;,/T) Ers exp(st,a/T)* the gradients of InfoNCE with respect to the positive similarity 81, and the negative similarity sijj (i + j) are formulated as (Wang & Liu, 2021):
1 T
Σ Pa
Pi.
0sis
Øsi.
(6)
Similarly, our ReSA formula z; can be writen as:
m
exp($15/T) log E exp($sx/T)
ERESA (zi) = Σ Ah (i.j)
Comparison to InfONCE As a well-known contrastive loss function, InfoNCE (Oord al., 2018) aims maximize the similarity between positive pairs while minimizing the similarity between the negative pairs, thereby approximating the optimization of mutual information as follows:
By contrast, the gradient of ReSA with respect to the sim- ilarity 8ty for any pair of samples (Vi,j) takes exactly the same analytical form:
OeResA(zi) 1 (Pa, AH (.D)
1 2m
Σ log Z') +
Σ log D(Z 2))
Osi.j
=
(8)
InfoNCE =
Based the gradient analysis above, know that In- foNCE explicitly distinguishes between the gradient forms of positive and negative similarities. This restrictive mecha- nism naturally leads to harmful gradient updates for negative sample pairs within the same class. contrast, ReSA elimi- nates the distinction between positive and negative samples and adapts optimize the similarity of all sample pairs by leveraging self-clustering of the encodings, thereby address- ing key challenge in contrastive learning.
(4)
It is evident that when Aи equals the identity matrix, ReSA and InfoNCE are entirely equivalent. In other words, ReSA guides the relationship among embeddings through assign- ments obtained via self-clustering, whereas InfoNCE em- ploys the identity matrix as hard matching target, strictly enforcing the maximization of distances between all nega- tive pairs, which may inadvertently push samples belonging to the same category further apart during training, thereby disrupting the underlying semantic cluster structure (Wang & Liu, 2021; Huang et al., 2024).
4.3. Impact of Image Augmentation on ReSA
Having introduced the learning process of ReSA, it is es- sential to consider another critical aspect of SSL: image augmentation, which has long been acknowledged as a key factor in enhancing the performance of sulf-supe meth- ods (Chen et al., 2020b; Grill et al., 2020). Standard prac-
4.2. A Gradient Perspective on ReSA
Many works have conducted in-depth theoretical studies on InfoNCE, such as the alignment and uniformity properties
5

---

Clustering Properties of Self-Supervised Learning
10:36
3:35
in improvements across various metries, whereas remov- ing HorizontalFlip causes slight drop in val accuracies. Based on these findings, we design a weak augmentation for ReSA, which includes only ResizedCrop and Horizon- talFlip, and discover that this design not only significantly enhances the clustering performance of the representations but also improves their overall quality. We note that these findings align with the results observed in ReSSL (Zheng et al., 2021), where weak augmentation enables the model to better capture the relationships among samples.
ARI SCmwn
10.26
3.25
ARI (x100-)
10.16
3.15
10.06
3.05
5C
9.96
2.95
9.86
2.85
9.76
2.75
79.1
70.8
78.6
70.4
In summary, we have completed the introduction of the ReSA framework as shown in Figure 5, which leverages clustering information extracted from the encoding to guide the design of the loss function, thereby achieving positive- feedback self-supervised learning.
linear val acc./%)
78.1
69.9
77.6
69.5
77.1
69.0
76.6
68.6
76.1
68.1
linear
75.6
knn
67.7
5. Experiments on Standard SSL Benchmark
75.1
67.3
ResisedCrop
Cologittur
Grayscale
GausslanBlur
Horizontal Flllp
Weak
In this section, we conduct extensive experiments on stan- dard SSL benchmarks evaluate the effectiveness of ReSA. We perform pretraining from scratch variety of datasets, including CIFAR- 10/100, ImageNet-100 and Ima- geNet (Deng et al., 2009), utilizing diverse encoders such as ConvNets and the ViT. Furthermore, we compare the perfor- mance of ReSA with sfate-of-the-art SSL methods across a range of downstream tasks, e.g. linear probe evaluation and transfer learning. The full PyTorch-style algorithm as well as details of implementation is provided in Appendix B.
Figure 6. Investigate the impact of image augmentation ReSA. The experiments are conducted employing a ResNet-18 encoder pretrained on ImageNet- 100 for 200 epochs. The starting posi- tions of bars represent results of the standard augmentation. The symhols* "_' on the a--axis indicate the removal of specific trans- formation from the standard augmentation. "Weak' denotes the weak augmentation that includes only ResizedCrop and Horizon- talFlip.
tices involve employing a variety of complex transforma- tions with random probabilities, such as ResizedCrop. Col- orJitter, Grayscale, GaussianBlur, and HorizzntallFipp to increase the task's complexity and improve the robustness of the learned representations. However, for clustering-basee basedd SSL methods, overly aggressive augmentations can distort the original image information, making it harder for the model to discern meaningful patterns (Zheng et al., 2021), which may result in incorrect clustering assignments. To address this, we systematically evaluate the effect of each transformation technique on ReSA's clustering performance during training and its linear evaluation accuracy.
5.1. Evaluation for Classification
Evaluation on small and medium size datasets. Follow- ing the benchmark in solo-learn (da Costa et al., 2022), we first perform pretraining and classification evaluations on CIFAR- 100, and ImageNet- 100, strictly adher- ing to the same experimental settings as other methods with- out introducing any additional tricks. The results in Table I reveal that ReSA consistently outperforms state-of-the-an methods, even those with carefully optimized parameters, across all datasets. Particularly noteworthy is ReSA's per- formance in k-nearest neighbors classification, surpassing other methods with absolute accuracy improvements of ap- proximately 3% to 8% on CIFAR-10 and 5% to 13% on CIFAR-100. These findings highlight that ReSA captures representations with superior clustering structures.
Since ReSA only requires extracting clustering information from the encodings of one single view, we only need to adjust the image augmentation for that specific view, while the standard augmentation setting can be applied to the other (s)w(s). Subsequently, we conduct experiments us- ing the high-resolution ImageNet- 100 (Tian et al., 2020) dataset, and the results are presented in Figure 6. It evi- dent that removing any single transformation improves the clustering performance of the representations, with Resized- Crop (replaced with fixed CenterCrop) having the most significant impact. However, its removal leads to substan- tial decline in representation quality, indicating the critical role of ResizedCrop in learning invariance. The removal of ColorJitter, Grayscale, or GaussianBlur each results
Evaluation on ImageNet. Following the ImageNet evalu- ation protocol commonly used by SSL methods, we pretrain ResNet-50 encoders with ReSA for varying numbers of epochs. As shown in Table 2, ReSA consistently outper- forms other methods on the large-scale ImageNet dataset. Remarkably, after only 100-epoch training, ReSA surpasses the performance of SimCLR, SwAV, and SimSiam trained for 800 epochs. With 200 epochs, ReSA exceeds of the-art methods such as M3CoV3, Barlow Twins, VICReg,
6

---

Clustering Properties of Self-Supervised Learning
Table 1. Classification top-1 accuracies of linear and k-nearest neighbors (k = 5) classifier for different loss functions and datasets. The table is mostly inherited from solo-learn (da Costa et al., 2022). methods are based on ResNet- with two augmented views generated from per sample and are trained for 1000-epoch with a batch size 128. The bold values indicate the best performance, and underlined values represent the second highest accuracy.
ZIFAR-10/100 with batch size of 256 and 400-epoch on ImageNet-100
CIFAR-10
CIFAR-100 100
ImageNet- linear
Method
linear
k-nn
linear
k-nn
k-nn
SimCLR (Chen al., 2020a)
90.74
85.13
65.78
53.19
77.64
65. 55.78
BYOL (Grill al., 2020)
92.58
87.40
70.46
56.46
80.32
68.94
SwAV (Caron et al., 2020)
89.17
84.18
64.88
53.32
74.28
63.84
SimSiam (Chen & He, 2021)
90.51
86.82
66.04
55.79
78.72
67.92
MoCoV3 (Chen et al., 2021)
93.10
89.47
68.83
58.23
80.36
72.76
W-MSE (Ermolov et al., 2021)
91.55
89.69
66.10
56.69
76.23
67.72
DINO (Caron al., 2021)
89.52
86.13
66.76
56.24
74.92
64.30
Barlow Twins (Zbontar et al., 2021)
92.10
88.09
70.90
59.40
80.16
72.14
VICReg (Bardes et al., 2022)
92.07
87.38
68.54
56.32
79.40
71.94
CW-RGP (Weng et al., 2022)
92.03
89.67
67.78
58.24
76.96
68.46
INTL (Weng al., 2024)
92.60
90.03
70.88
61.90
81.68
73.46
ReSA (ours)
93.53
93.02
72.21
66.83
82.24
74.56
Table ImageNet classification top-1 accuracy a linear classi- fier based on ResNet-50 encoder. All methods are pretrained with two 224 augmented views generated from per sample. Given that one of the objectives of SSL methods to achieve high per- formance with small batch sizes (Chen et al., 2020b; Chen He, 2021), it's worth noting that ReSA also perform effectively when trained with a small batch size of 256.
Table 3. ImageNet classification top-1 accuracy a linear and a k-nearest neighbors (k 20) classifier based standard ViT- S/16 encoder. All models are pretrained for 300-epoch with two 224 views.
Classifier
BYOL
SwAV
MoCoV3
DINO
ReSA
linear k-nn
71.4
68.5
72.5
72.5
72.7
66.6
60.5
67.7
67.9
68.3
pretrained epochs
Method
batch size
100
200
800
Table 4. Comparison of Computational overhead among various SSL methods. For faimess, 224² augmented views pretraining on ImageNct, and perform all measurements including peak memory (GB per GPU) and training time (hours per epoch) on the same environment and machine equipped with 1100-PCIE-40GB GPUs using dataloading workers under mixed-precision.
SimCLR
256 4096
57.5
62.0
66.5
the batch size 1024 with
66.5
68.3
70.4
256
65.5
67.7
SwAV
4096
66.5
69.1
71.8
1024
67.4
71.0
72.4
MoCoV3
4096
68.9
73.8
BYOL
4096
66.5
70.6
74.3
Barlow Twins
2048
67.7
70.2
73.2
Method
encoder
memory
time
VICReg
2048
68.6
70.8
73.1
ResNet-50
13.7
0.19
SWAV
256
68.1
70.0
71.3
ViT-S/16
14.8
0.17
SimSiam
1024
68.0
69.9
71.1
ResNet-50
14.6
0.19
McCov3
256
70.1
ViT-S/16
15.5
0.22
МEC
1024
70.6
71.9
74.0
ResNet-50 ViT-S/16
15.5
0.20
DINO
256
69.5
71.1
73.1
16.4
0.21
INTL
1024
69.7
71.2
73.3
ResNet-50 ViT-S/16
14.6
0.16
ReSA (ours)
256
71.9
73.4
ReSA (ours)
15.6
0.12
1024
71.3
73.8
75.2
5.2. Analysis on Computational Overhead
and INTI,, all trained for 800 epochs. When extended to 800 epochs, ReSA achieves linear classification accuracy of 75.2%. level that methods like SwAV and DINO only reach by employing the multi-crop (Caron et al., 2020) trick. These results underscore ReSA's exceptional potential for training on large-scale datasets. Additionally in Table 3, we conduct preliminary evaluations of ReSA's training capabil- ity on the Vision Transformer, using a standard VÝT-S/16, which has comparable number of parameters ResNet-50. We do not incorporate extensive training tricks, yet ReSA still outperform DINO in both linear and k-nn classification.
In Table 4, we provide fair comparison of the training costs among ReSA and several SSL methods. The results show that ReSA has memory consumption comparable to MoCoV3 but achieves faster training speeds, especially on ViTs, where it is nearly twice as fast. This improvement is attributed to ReSA's simpler image augmentation settings and the removal of batch normalization (BN) from the pro- jector and predictor MLPs. Additionally, ReSA outperforms DINO in both memory usage and training time. This ad- vantage stems from DINO's 's reliance on an extremely high-

---

Clustering Properties of Self-Supervised Learning
Table 5. Transfer Learning to COCO detection and instance seg- mentation. All competitive methods are based on ResNet-50 with 200-epoch pretraining ImageNet. We follow MoCo (He et al., 2020) to apply Mask R-CNN (1 x schedule) fine-tur sieettnndddde in COCO 2017 train, evaluated in COCO 2017 val.
free
bir
hanee
crol
ailee on dirkereeeed hores
car
bol head
harte heas
SImCLR
VICReg
SwAV
COCO detection
COCO
instance seg.
Method
APs0
AP
APrs
AP50
AP
AP7s
Scratch
44.0
26.4
27.8
46.9
29.3
30.8
Supervised
58.2
38.2
41.2
54.7
33.3
35.2
SimCLR
57.7
37.9
40.9
54.6
33.3
35.3
McCoV3
DINO
ReSA (ours)
McCoV2
58.8
39.2
42.5
55.5
34.3
36.6
BYOL
57.8
37.9
40.9
54.3
33.2
35.0
SWAV
57.6
37.6
40.3
54.2
33.1
35.1
SimSiam
59.3
39.2
42.1
56.0
34.4
36.7
Barlow Twins
59.0
39.2
42.5
56.0
34.3
36.5
MEC
59.8
39.8
43.2
56.3
34,7
36.8
INTL
60.9
40.7
43.7
57.3
35,4
37.6
ReSA (ours)
61.1
41.0
44.3
57.7
35.7
38.4
dimensional prototype (e.g., output dimension = 65536),
which significantly impacts training efficiency. Finally, while SwAV exhibits the smallest memory footprint among the methods due to its absence of momentum networks, its training speed remains slower than ReSA. This is because SwAV also requires a performs the Sinkhorn-Knopp algorithm twice per iteration.
prototype and
Figure 7. T-SNE visualization of SSL representations on CIFAR- 10. All methods are pretrained for 1000 epochs on CIFAR-10 using ResNet-18, with encodings utilized as representations to visualize all training data. For the multiple centroids observed in the bird and horse categories, we enclose points of each subelass with convex polygons and display the corresponding images.
5.3. Transfer to Downstream Tasks
To evaluate the quality of representations learned by ReSA, we transfer our pretrained model to downstream tasks, in- cluding COCO (Lin et al., 2014) object detection and in- stance segmentation. For these tasks, we adopt the baseline codebase from MoCo (He et al., 2020). Most results re- ported in Table 5 are inherited from SimSiam paper (Chen & He, 2021). Notably, ReSA also achieves better performance compared other methods on both tasks, highlighting its strong potential for downstream applications.
also acquire robust clustering performance.
6.1. ReSA Excels at Fine-grained Learning
Furthermore, in Figure 7, we visualize the representation distributions learned by ReSA and other SSL methods on 1I using T-SNE (van der Maaten & Hinton, 2008). Notably, the representations learned by ReSA exhibit clear separations between different classes, whereas those learned by other methods show varying degrees of overlap, making it difficult to discern distinct boundaries.
6. How ReSA Shapes Better Clustering Properties?
In this section, utilize visualizations and additional ex- periments illustrate the differences among the represen- tations learned by ReSA and other SSL methods, as well as to investigate whether and how ReSA facilitates better clustering properties.
Another intriguing observation is the presence of multiple centroids within the bird and horse categories in the rep- resentations. Upon further investigation of the samples corresponding to these centroids, we find that, unlike the neural collapse (Papyan et al., 2020) in supervised learning (where samples of the same class collapse to a single point), SSL models are capable of capturing more fine- -eeiggriinedd fea- tures, such as color distinctions (e.g., cassowary vs. ostrich), structural differences (c.g., whole horse vs. horse head), and the presence of multiple objects (e.g., a rider on a horse). Moreover, compared to other methods, ReSA demonstrates a superior ability to distinguish these fine- grained features.
Firstly, we track the evolution of clustering metrics for each component during ReSA training, as shown in Figure 9. It can be clearly observed that while ReSA exhibits relatively slow performance improvement in the early stages of train- ing, it significantly outperforms other methods in the later stages. Interestingly, all components of ReSA demonstrate strong clustering properties, suggesting that leveraging high- quality clustering information from the encodings to guide the learning of embeddings enables the projector layers to
8

---

Clustering Properties of Self-Supervised Learning
Table 6. Transfer learnngg fine- tne-grained datasets based on ResNet-50 pretrained on ImageNet. We employ a k-nearest neighbors classifier
(k: = 5, 10, 20), without requiring additional training or parameter tuning. The model weights for all other methods are soureed directly from their respective official codebases. i indicates that these methods employ the multi-crop trick, i.c. generating two 224a views and six 962 views for each image, which can enhance performance but comes at the cost of additional computational overhead.
pretrained epochs
ImageNet-IK
IK
CUB LUB-200-200
220--0001
Pets-37
Food- 101
Flowers-
102
Method
5
10
20
5
10
20
5
10
20
5
10
20
5
10
20
MoCoV3
1000
67.9
68.9
68.9
46.8
48.8
50.4
85.4
86.5
86.5
56.3
58.6
59.7
83,4
81.6
80.9
VICReg
1000
64.3
65.2
65.6
33.4
35.4
36.3
81.5
82.0
82.3
56.9
59.6
61.0
83.4
83.2
82.6
INTL
800
63.6
64.8
65.1
26.7
28.0
29.4
78.4
79.5
79.7
55.6
58.1
59.2
78.8
77.6
77.2
ReSA (ours)
800
69.2
69.9
69.9
56.5
58.5
59.9
85.8
87.2
87.5
58.3
60.4
61.3
84,4
83.6
83.6
SWAVT
800
64.3
65.5
65.7
26.2
27.3
28.4
77.2
77.3
77.1
54.7
57.4
58.7
79.3
79.9
78.4
DINO
800
66.4
67.4
67.6
33.8
35.5
36.8
81.1
81.6
80.9
58.2
60.8
61.8
84.8
84.1
83.7
Dener
Table 7. CIFAR-100 classification top-1 accuracy of linear and a k-nearest neighbors (A- 5) classifier based on 100 fine-grained classes and 20 coarse-grained superclasses.
Rpm
SimCLit
VicReg
SwAV
fine-grained
coarse-grained k-nn
Method
linear
k-nn
linear
SimCLR
65.8
53.2
72.5
67.2
SwAV
64.9
53.3
70.0
66.3
MoCoV3
68.8
58.2
76.4
68.6
MoCoVS
DINO
ReSA (ours)
DINO
66.8
56.2
72.9
70.2
VICReg
68.5
56.3
74.3
69.9
ReSA (ours)
72.2
66.8
79.8
78.8
that ReSA consistently achieves much higher accuracies than its counterparts. These experimental results confirm that ReSA also demonstrates exceptional performance in coarse-graing learning.
Figure& 8. T-SNE visualization of SSL representations on CIFAR- 100. We enclose points of cach subclass with convex polygons.
To further substantiate this, we transfer models pretrained on ImageNet to fine-graines datasets for evaluation. As shown in Table 6, ReSA consistently outperforms other SSL meth- ods on fine- finengrained datasets, with particularly considraable improvements observed the CUB-200-2011
We also present ablation studies, along with a discussion of potential future research presented in Appendix C.
7. Conclusion
In this work, we demonstrate the feasibility of leveraging the rich clustering properties inherent in SSL models, particu- larly within encodings, to enable positive-feedbact mecha- nism. Building upon this, we propose ReSA, which exhibits exceptional performance across a wide range of benchmarks and excels at both fine-grained and coarse-graing learning, We believe this dual capability would take a step toward addressing the long-standing challenge of reconciling the seemingly conflicting demands of fine-grained and coarse- grained visual representations within unified framework. thereby advancing the development of large-scale visual foundation models.
6.2. ReSA also Stands Out in Coarse-grained Representations
Finally, we explore ReSA's performance at the coarse- grained level. Specifically, we select CIFAR-100 for vi- sualization, as its 100 classes can be grouped into 20 coarse- grained superclasses. For clarity, we randomly selected 5 superclasses for T-SNE visualization, as shown in Figure 8. is evident that all other methods exhibit dense overlap on the CIFAR- 100 dataset, resulting in numerous indistinguish- able outliers. In contrast, our ReSA effectively identifies these hard samples, clustering them correctly within their re- spective groups. We believe this capability of ReSA is key factor behind its superior accuracy with the k-nn classifier, substantially exceeding other SSL methods. Furthermore, as shown in Table 7, we evaluate the performance of var- ious SSL models using coarse- grained labels and observe
A Acknowledgment
This work was supported by the National Science and Tech- nology Major Project (2022ZD01 16306), National Natural

---

Clustering Properties of Self-Supervised Learning
Science Foundation of China (Grant No. 62476016 and 62441617), the Fundamental Research Funds for the Cen- tral Universities.
Bendidi, I., Bardes, A., Cohen, E., Lamiable, A., Bollot, G., and Genovesio, A. No free lunch in self supervised representation learning, arXiv preprint arXiv:2304.11718. 2023.
Impact Statement
Bizeul, A., Schölkopf, B., and Allen, C. A probabilistic model behind self-supervised learning. arXiv preprint arXiv:2402.01 2024.
This paper presents work aimed at advancing the field of self- supervised learning. By improving representation learning, methods contribute to wide range of applications across various domains, including computer vision, natural language processing, and beyond. While our work has po- tential societal implications, such as enabling more efficient use of data and reducing reliance on labeled datasets, we do not identify any specific ethical concerns or negative consequences that require particular attention at this stage.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. Computer vision-ECCI 2014: 13th European con- ference, zurich, Switzerland, September 6- 6-1 2014, pro- ceedings, part VI 13. pp. 446 446-461. Springer, 2014.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
References
Asano, Y. M., Rupprecht, C., and Vedaldi, A. critical analysis of self-supervision, what we can learn from single image. arXiv preprint arXiv:1904.1333, 2019.
Caron, M., Misra, Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Assran, M., Balestriero, R., Duval, Q., Bordes, F., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., and Ballas, N. The hidden uniform cluster prior in self-supervised learning. arXiv preprint arXiv:2210.07277, 2022a.
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self- supervised vision transformers. ICCV, 2021.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In ICML, 2020a.
Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P. Joulin, A., Rabbat, M., and Ballas, N. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pp. 456-473. Springer, 2022b.
Chen, X. and He, K. Exploring simple siamese representa- tion learning. In CVPR, 2021.
Chen, X., Fan, H., Girshick, R., and He, K. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint- enbeddinbdding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619- 15629, 2023.
Chen, X., Xie. S., and He, K. An empirical study of training self-supervised vision transformers. In CVPR, 2021.
Cuturi, M. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
da Costa, V. G. T. Fini, E., Nabi, M., Sebe, N., and Ricci, E. solo-learn: A library of self-supervised meth- ods for visual representation learning. Journal of Ma- chine Learning Research, 23(56): 2022. URL http: / / ammlr.org www.orrggppppaa..e wwwpp/pppssssssem v23 / -1-1155..111 html.
Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint AXiv:2106608254, 2021.
Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Variance- invariance-covaria regularization for supe-vupe learning. In ICLR, 2022.
Deng. J., Dong, W., Socher, R., Li. L.-J., Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
Ben-Shaul, I., Shwartz-Ziv, R., Galanti, T., Dekel, S., and LeCun, Y. Reverse engineering self-supervised learning. Advances in Neural Information Processing Systems, 36: 2023.
Devlin, Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv: 1ele:8/000000 810.04805 2018
10

---

Clustering Properties of Self-Supervised Learning
Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N. Whitening for self-supervised representation learning. In ICML, 2021.
Singhal, K., Green, B., al. Augmentations algo-
rithms: What works in self-supervised learning. arXiv preprint urXiv:2403.05722 2024.
Geiping, J., Goldblum, M., Somepalli, G., Shwartz-Ziv, R., Goldstein, T., and Wilson, A. G. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint DrXiv:2210006441, 2022.
Moutakanni, T., Oquab, M., Szafraniec, M., Vakalopoulou, M., and Bojanowski, P. You don't need data- augmentation in self-superrised learning. arXiv preprint arXiv:2406.09220 092944 2024.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & iimage processing, pp. 722-729. IEEE, 2008.
Grill, J.. Strub, F,, Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., and Valko, M. Bootstrap your own latent a new approach self-supervised learning. In NeuralPS, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint htlv:880703775 2018.
Gupta, K., Ajanthan, T., Hengel, A. v. d., and Gould, S. Understanding and improving the role of projec- tion head in self-supervised learning. arXiv preprint arXiv:2212. 1491, 2022.
Oquab, M. Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07199. 2023.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Papyan, V., Han, X., and Donoho, D. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, 117(40):4452-24666 2020.
Huang, L., Ni, Y., Weng. X., Anwer, R. M., Khan, S., Yang, M.-H., and Khan, F. S. Understanding whitening loss in self-supervised learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1-12, 2024.
Parkhi, O. M., Vedaldi. A., Zisserman, A., and Jawahar, Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3-498-3501 IEEE, 2012.
Hubert, L. and Arabie, P. Comparing partitions. Journal of classification, 2:193-218, 1985.
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand- ing dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
Purushwalkam, S. and Gupta, A. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, Advances in Neural Information Process- ing Systems, 33:3407-3418, 2020.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Rousseeuw, P. J. Silhouettes: graphical aid to the inter- pretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
Lin, T.-.,, Maire, M. Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, L. Microsoft coco: Common objects in context. In ECCV, 2014.
Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In European conference on computer vision, 2020.
Liu, X., Wang, 7., Li, Y.-L.,, and Wang, S. Self-supervised learning via maximum entropy coding. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
van der Maaten, L. and Hinton, G. E. Visualizing data using t-sne. Journal of Machine Learning Re- search, 9:2579-2605, 2008. URL https://api. semanticscholas wttp//orpSSD:S:555550...
Loshchilov, I. and Hutter, F. SGDR: stochastic gradient descent with restarts. In ICLR, 2017.
Wagner, D., Ferreira, F., Stoll, D., Schirrmesster, R. T., Müller, S., and Hutter, F. On the importance of hyper- parameters and data augmentation for self-supervised learning. arXiv preprint arXiv:2207.07875, 2022.
Ma, J.. Hu, T. and Wang, W. Deciphering the projection head: Representation evaluation self- supervede learning. arXiv preprint 2rXiv:2301.12188) 2023.
Morningstar, W., Bijamov, A., Duvarney, C., Friedman, L., Kalibhat, N., Liu, L.,, Mansfield, P., Rojas-Gomez, R.,
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd 2irds-200-2011 dataset. 2011.

---

Clustering Properties of Self-Supervised Learning
Wang, F. and Liu, H. Understanding the behaviour of con- trastive loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2495- 2504, 2021.
Wang, T. and Isola, Understanding contrastive represen- tation learning through alignment and uniformity the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Weng, X., Huang, L., Zhao, L., Anwer, R. M., Khan, S., and Khan, F. An investigation into whitening loss for self-supervised learning. In NeurlPS, 2022.
Weng, X., Ni, Y., Song, T., Luo, J., Anwer, R. M., Khan, S., Khan, S., and Huang. Modulate your spectrum in self-supervised learning. In ICLR, 2024.
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimina- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Zhontar, J., Jing, L.,, Misra, I., Lecun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In JCML, 2021.
Zheng, M., You, S., Wang, F., Qian, C., Zhang, C., Wang, X., and Xu, C. Ressl: Relational self-supervised learning with weak augmentation, Advances in Neural Information Processing Systems, 34:2543 -5555, 2021.
12

---

Clustering Properties of Self-Supervised Learning
A. Details of Clustering Metrics
To quantitatively compare the clustering ability the encoding, embedding, and the hidden layer outputs within the projector, we first define the following metrics in supervised settings (assign samples with the same true label to the same cluster). Let X' labels.
{01, (2: ...., IN be set of N data points, and y {31, 1 ................................ yN3 be the corresponding set of true
Definition A.1. (Silhouette Coefficient, SC) The Silhouette Coefficient (Rousseeuw, 1987) is measure of how similar a sample its own cluster compared to its nearest cluster. For given data point v;s sc(2,) defined as:
b(x;) a(2;)
sc(x,) max(a(x;)) b(x;))
(9)
where a(x;)) is the average distance from point z; to all other points z that share the same true label ys yys and b(x4) the minimum average distance from point x; to all points zi that have different true label yi * y/-
Based on this definition, we know that SC focuses on measuring the local clustering ability of features, with higher values indicating better local clustering ability. Although the SC ranges from [-1, 1]. the diversity of features learned through SSL can cause large value of max(a(z;), b(z;)), leading to a smaller effective range for SC. Therefore, in this paper, note that SC > 0 indicates that the sample has been assigned to the correct cluster. For population statistics, we can compute the mean SCmean and standard deviation SCau over all data points in X.
Definition A.2. (Adjusted Rand Index, ARI) The Adjusted Rand Index (Hubert & Arabie, 1985) is measure of the agreement between two partitions of data, adjusted for chance grouping. Given the true labels y and set of predicted labels )" = {V1. 1/2. ... U' J. the ARI is defined as:
RI - R
ARI
(10)
max(RI) E[Ri]
where RI is the Rand Index, and E[RI] is its expected value under random labeling.
In practice, the ARI can be computed using a contingency table between y and y Let naj denote the number of data points assigned to the i-th cluster y and the j-th cluster in y" Defining a; = E3 nay and bj = E Najs then the ARI is calculated as:
Eis (22)
(E, ( E,
ARI
Σ. (2) Σ, (2) Σ, ( Σ, (3) (7)
where (
x(n=1) is the binomial coefficient. The ARI ranges from 0.5, 1]. where an ARI close to 1 indicates a perfect agreement between the true and predicted labels, and an ARI close to suggests that the prediction is no better than random assignment.
2
Typically, we apply k-means (k set to the true number of classes) clustering on X to obtain the pseudo labels )" Subsequently, ARI is used to measure the agreement between true and pseudo labels, thereby reflecting global clustering ability of the features and the extent to which similarity measures effectively capture the data structure.
B. Details of Implementation
In this section, we provide the details and hyperparameters for ReSA pretraining and downstream evaluation.
B.1. Datasets
CIFAR-10 and CIFAR- 100 (Krizhevsky et al., 2009), two small-scale datasets composed of 32 x 32 images with 10 and 100 classes, respectively.
ImageNet- (Tian et al., 2020), random 100-class subset of ImageNet (Deng et al., 2009).
13

---

Clustering Properties of Selt-Supervised Learning
ImageNet (Deng et al., 2009), the well-known largescale dataset with about 1.3M training images and 50K test images, spanning 1000 classes.
COCO2017 (Lin al., 2014), a large-scale object detection, segmentation, and captioning dataset with 330K images containing 1.5 million object instances.
We also evaluate on aiue-griiedd datasets including CUB-200-2011 (Wah et al., 2011), Oxford- -ilrdd--II--eets (Parkhi et al., 2012), Food-101 (Bossard al., 2014), and Oxford-Flowers (Nilsback & Zisserman, 2008).
B.2. Implementation Details of ReSA Pretraining
For clarity, we first provide the algorithm of ReSA in PyTorch-style pseudo-code:
E, Em: encoder, momentum encoder
G, Gm: projector mlp, momentum projector mlp
# P: predictor mlp (optional)
Tw, weak, standard augmentation
# temp: temperature 0.4
for x in loader: # load a minibatch x with a samples
x1, Tu(x), T(x) two augmentation views
h1, h2 E(x1), E(x2) # encodings
z1, G(h1), G(h2) embeddings (mxd)
if P: z1, z2 = P(z1), P(z2) # predicted embeddings
with torch. gr
updaate_momentum_parrana
1)
exponential moving average
h1m, h2m Em(x1), Em(x2) nomentum encodings
z1m, z2m Gm(h1m), Gm(h2m) momentum embeddings (mxd)
assign sinkhorn(coo__immh11] h1m)) # compute representation assignment
Loss cooss_entropycoosssiic,,, 22m) / temp, assign) \
cross_eetropy(cossssn((. z1m) temp, assign)
return loss / 2
def cos_sin(x, y):
return norm(x) @ norm(y).T L2-normalize
def cross_entropyxx, y):
Loss mean(y log_softmax(x, dim=1) + y.T log_softmaxxx.T, dim=1)) return loss
def sinkhornsscores, eps=0.05, niters=3); # Here 'scores" should be square matrix
Q = exp(scores / eps)
Q /= sum(
m = shape
ones(m)
for in vange(niters)
u Q Q.sum(dim=])
*= (c / u). unsqueeze(1)
(c / (.sum(din==)) unspueezesuuueee
return (Q / Q sumddnn= keepdin=True)),
14

---

Clustering Properties of Self-Supervised Learning
Universal settings. In all experiments conducted in Section 5, we adopt a momentum network, consistent with the practices of most existing self-supervised learning (SSL.) methods (Grill al., 2020; Chen et al., 2021; Caron al., 2021; Liu et al., 2022; Weng et al., 2024). While the momentum network is not necessary to prevent collapse in ReSA, it has been shown to effectively promote long-term learning in SSL models (He et al., 2020; Chen & He, 2021). The momentum coefficient, temperature, and Sinkhorn-Knopp parameters in ReSA are configured in accordance with the pseudo-code provided earlier, without requiring further tuning. Furthermore, a standard three-layer MLP is employed as the projector, featuring a hidden layer dimension of 2048 and an output embedding dimension of 512. For training on large-scale datasets, such as ImageNet- IK, we follow the practices of MoCoV3 (Chen et al.. 2021) and MEC (Liu al., 2022), appending a two-layer MLP predictor to the projector in ReSA. The hidden layer and embedding dimensions of the predictor are kept identical to those of the projector. Additionally, we adopt a conventional configuration: when using ConvNets as the encoder, batch normalization (BN) and ReLU activation are incorporated into the hidden layers of both the projector and predictor. However, for ViT-based encoders, we draw inspiration from DINO (Caron et al., 2021), omitting BN and replacing ReLU with the gaussian error linear units (GELU) activation. This modification ensures that ReSA operates as a BN-free system during ViT training, eliminating the need for BN synchronization and offering an improvement in training efficiency.
Building on the aforementioned settings, the only modifications required pertain to optimizer-related parameters, including the learning rate, weight decay, and the number of warmup epochs. These adjustments are made in accordance with the specific encoder architecture and dataset used. Nonetheless, certain settings remain fixed. For instance, we adopt the linear scaling rule, setting the learning rate as Ir = base Ir x batch size/256. After the warmup phase, the learning rate follows a cosine decay schedule (Loshchilov & Hutter, 2017).
Details in training ConvNets. We follow the exact same optimization settings and parameters as INTL (Weng et al., 2024) when pretraining ConvNets. The SGD optimizer is used and detailed parameters are provided in Table 8. The only exception is when training ResNet-50 on ImageNet for 800 epochs, where we reduce the base Ir to 0.4 to ensure training stability. Additionally, it is worth noting that we observe a slight performance drop in ReSA when the learning rate decreases to very small value during the later stages of training. We hypothesize that an excessively small learning rate may amplify the regularization effect of weight decay in SGD, causing the weights diverge from the optimal solution. To address this, set the minimum learning rate in the cosine decay schedule to 0.1 Ir.
Table Optimizer-related parameters ReSA pretraining.
Method
dataset
encoder
predictor
optimizer
batch size
base Ir
weight decay
warmup
CIFAR-10
R8
X
SGD
256
0.3
10-4
2 epochs
CIFAR-100
ResNet-18 18
X
SGD
256
0.3
10-4
2 epochs
ImageNet-100 100
ResNet-18
X
SGD
128
0.5
2.5 x 10-5
2 epochs
ReSA
256
0.5
10-5
2 epochs
ResNet-50
SGD
ImageNet
1024
0.5
10-5
10 epochs
ViT-S/16
AdamW
1024
x 10-4
0.1
40 epochs
Details in training ViTs. Vision Transformer (ViT) pre-training involves numerous intricate settings, such as initialization methods and optimization parameters, which have a significant impact on training outcomes. Notably, representative SSL methods on ViTs, such as DINO and MoCoV3, adopt totally distinct designs in both architecture and training strategies. DINO (Caron et al., 2021) leverages a range of training tricks, including weight decay scheduling, gradient clipping, and stochastic depth, among others. To stabilize training, it avoids mixed--rrccisoon training, which substantially reduces computational efficiency and increases memory requirements. MoCoV3 (Chen al., 2021), on the other hand, proposes freezing the patch embedding layer to enhance training stability while enabling mixed- -ieeed-preiision training. However, it still incorporates batch normalization (BN) in the projector and predictor MIPs, which reduces training efficiency and complicates its application in multi-view scenarios (Morningstar et al., 2024). Taking these considerations into account, we adopt the VIT design and initialization approach of MoCoV3 for ReSA, but remove the BN layers from the MLPs. This ensures that ReSA functions as a BN-free system during ViT training, eliminating the need for BN synchronization and improving overall training efficiency. In this paper, we set the optimizer-related parameters as shown in Table 8. We
15

---

Clustering Properties of Self-Supervised Learning
believe that further exploration of more suitable initialization methods and related parameters for ReSA could enhance its performance, evidenced by its outstanding results on ConvNets.
B.3. Implementation Details of ReSA Evaluating
Details in evaluating CIFAR-10/100. When evaluating on CIFAR-10/100, we adopt the same linear evaluation protocol as in W-MSE (Ermolov et al., 2021) and INTL (Weng et al., 2024): training linear classifier for 500 epochs each labeled dataset using the Adam optimizer, without data augmentation. The learning rate is exponentially decayed from 10-2 2 to 10-6 and the weight decay is 5 x 10-6. Under these settings, a single-GPU evaluation takes under one minute minute-substanee substantially faster than the protocol in solo-learn (da Costa et al., 2022), which can take tens of minutes. We also apply this evaluation protocol to models provided by solo-learn; however, their performance degrades noticeably, so we report the official results in Table 1. In addition, following W-MSE and INTL, we evaluate a simple 5-nn classifier (A: = 5) on these datasets for completeness. We track both linear and k-m elassifier accuracies for ReSA throughout training and observe that at certain checkpoints, ReSA achieves even higher performance than the final values reported in Table 1 (e.g., linear accuracies of 93.89% on CIFAR-10 and 72.5% on CIFAR-100) Nevertheless, we report only the final checkpoint's 's results for consistency.
Table 9, Optimal learning rate (Ir) for training linear classifiers.
pretrained settings
linear eval.
Method
dataset
encoder
batch size
ImageNet-
R
128
256
10
ReSA
ResNet-50
ImageNet
1024
40
ViT-S/16
1024
0.03
Details in evaluating ImageNet-100/1K. For linear evaluation, we train the linear classifier for 100 epochs with SGD optimizer and using MultiStepLR scheduler with 7 = 0.1 dropping at the last 40 and 20 epochs. In all our linear classifier training, we fix the batch size 256 and set the weight decay to O. However, when using different pretraining datasets, encoders, or batch sizes, the optimal learning rate for evaluation varies accordingly. The specific optimal values for each setting are provided in Table 9. In addition, when training the linear classifier with ViT-S/16, we follow BERT (Devlin, 2018) and DINO (Caron et al., 2021) by concatenating the [CLS] tokens from the last four layers.
Table 10. Low-shot evaluation. All models are pretrained on ImageNet with ResNet-50,, and then fine-tuned with a linear classifer on 1% or 10% subset of ImageNet for 20 epochs. i indicates employing the multi-crop trick during pretraining.
top-1
top-5
Method
1%
10%
1%
10%
SimCLR
48.3
65.6
75.5
87.8
BYOL
53.2
68.8
78.4
89.0
SwAVi
53.9
70.2
78.5
89.9
Barlow Twins
55.0
69.7
79.2
89.3
VICReg
54.8
69.5
79.4
89.5
INTL
55.0
69.4
80.8
89.8
ReSA (ours)
56.4
70.4
81.0
90.1
We further evaluate the low-shot learning capability of ReSA in semi-supervised classification. Specifically, we fine-tune the pre-trained ReSA encoder and train a linear classifier for 20 epochs, using 1% and 10% subsets of ImageNet, following the same splits as SimCLR (Chen et al., 2020b). The optimization conducted using the SGD optimizer with a learning rate of 0.0002 for the encoder and 40 for the classifier, under a batch size of 256. along with a cosine decay schedule. The results, presented in Table 10, demonstrate that ReSA also performs effectively in low-shot learning scenarios.
16

---

Clustering Properties of Self-Supervised Learning
e/fice
H
520
2004
Po
529
fpach
Spock
Epock
Spach
P1
0:0
0.0
Epeth
0.5
Z
Figure Comparison of evaluation accuracies clustering metrics among various methods during the training process. The experiments are conducted using SimCLR, SwAV, VICReg, and ReSA. The settings and notations are consistent with ones in Figure 4.
Details in evaluating fine-grained datasets. In the evaluation experiments on fine-grained datasets presented in Table 6, we apply a weighted k-nearest neighbors classifier following (Wu et al., 2018). We freeze the pretrained model to compute and store the features the training data and use these features to select the nearest neighbors for the data in the test set. Based on the top k-nearest neighbors (W&),, predictions are made using a weighted voting mechanism. Specifically, the class receives a total weight of ΣieN, ae where a represents the contribution weight. We compute a, exp with set to 0.07 as described in (Wu et al., 2018) and used by DINO, without tuning this value.
C. Additional analyses on ReSA
C.1. Ablation Studies
As presented in Table 11, we conduct ablation studies to investigate the impact of network architecture design and temperature hyperparameter selection ReSA performance. Our analysis reveals that employing momentum network yields accuracy improvement of approximately 2%, albeit at the cost of increased computational overhead. In contrast, the integration of an additional predictor demonstrates comparable accuracy gain of around 1% while maintaining near-identical computational efficiency, exhibiting negligible impact on runtime performance.
Meanwhile, in contrast to contrastive learning methods and other approaches such as SwAV and DINO, which typically require small temperature value (e.g. т = 0.1), ReSA achieves favorable performance with a higher temperature value 0.4). This indicates that the optimization process of ReSA can effectively incorporate broader range of samples with better tolerance, rather than focusing exclusively hard samples (Wang & Liu, 2021), is the case with other methods.
Additionally, we conduct experiments test the extraction of clustering information from embedding to obtain the self assignment Aи. The results shown in Figure 10 indicate that, under this condition, the loss struggles to converge, and the model's s accuracy significantly declines compared to ReSA. This finding consistent with the analyses in Section 3, where we note that the clustering properties of embedding are less stable and inferior to those of encoding, making it challenging for the model to effectively learn high-quality clustering information.
17

---

Clustering Properties of Self-Supervised Learning
Table 1. Ablation studies on the network architecture and temperature hyperparameter of ReSA ImageNet using ResNet-50 the encoder. When evaluating the impact of different network architectures on ReSA, we set the batch size to 256 and perform pretraining using a single GPU.
Method
momentum
predictor
linear acc.
memory (GB)
time (h/epoch)
71.9
25.2
0.78
X
70.8
25.2
0.78
ReSA
69.7
24.3
0.58
x
X
68.7
24.3
0.58
temperature
Method
batch size
0.2
0.3
0.5
256
71.2
71.4
71.9
71.7
ReSA
1024
70.9
71.1
71.3
71.2
uncoding ambolding
2.7
0.5
2.6-
Linear val acc.
Lose
4.3
0.4
3.5
ambedding
embedding
200
Epoch
Epech
Figure 10. Ablation extraction of clustering information from encoding embedding obtain the self-assignment An. Both pretrained for 1000 epochs on CIFAR- 100 using ResNet-18 18 under totally the same experimental settings provided in Appendix B.
331
122
initialization
early stages of pretraining
Figure 11. Visualization of the self-assignment matrix Aн during the early stages of training.
C.2. How ReSA Avoids Feature Collapse and Early Clustering Error?
Although ReSA has demonstrated good performance in various experiments, remains unclear how the model avoids feature collapse and clustering error in the early stages of training when has yet learned any clustering information.
To answer these questions, we first consider the loss formula of ReSA: (RoSA(2))
- Ah (i.7) log
exp(si./T)
m
18

---

Clustering Properties of Self-Supervised Learning
Here, Ah (i.f) denotes the (i.j)-th element of Ан, where Aн Siinkhom H). Given that the Sinkhorn-Knopp algorithm exhibits strict monotonicity, we have the property that h, " h > h, hs, then Ah (1-) AH (i,A), Since the vectors in H are Lg-normalized, the diagonal elements of H H are all maximized 1. This implies that for any i and j, we have Ah (6,4) 2 Ah (t.). Furthermore, due the sharp distribution employed in Sinkhorn, the diagonal elements are significantly larger than the off-diagonal elements. This ensures that the optimization focus of ReSA remains on the alignment of augmented views from the same image, substantially reducing the impact of early assignment errors (off-diagonal elements) on the training process. Throughout training, the continual aligmment enables the model to learn meaningful representations, which in turn facilitates correct cluster assignments the later stages, further promoting learning. In contrast, other clustering-bee assignment, making more challenging to avoid early clustering errors. We speculate that this one of the key reasons why ReSA outperforms these methods in terms of overall effectiveness.
methods such as SwAV and DINO require an initialized prototype for cluster
We then visualize the self-assignment matrix Aи during the early stages of training. As shown in Figure 11, at model initialization, the assignment already exhibits a dominance of diagonal elements, reflecting an optimization trend that pulls augmented views from the same image closer together. As training progresses, we observe that the dominance of diagonal elements gradually strengthens, effectively preventing feature collapse in the model.
C.3. Pretraining on Long-tailed Dataset
To further evaluate the training performance of ReSA on imbalanced datasets, we conduct experiments using four self- supervised learning methods ReSA, MoCoV3, INTL, and ViCteeg-peetrained and evaluated on the long-tailed CIFAR100- LT dataset. follow the setup on https: : /twgginuuggggggg wwwggginnacccee co wt//oaa.aaoooo htt//ataseettt /etas/tttttttssssesseeesssss gajarsky//aaaaaaa hat//ifarrr wwrffarrr11111 00-1t. setting an imbalance factor of 1/20, resulting in the CIFARI00-LT dataset containing 15,907 images. We train these four SSL models for 1000 epochs with ResNet-18 as the encoder and evaluate it on the full CIFARIOO in Figure 12, the loss values of these four methods converge well, but the final evaluation accuracy is significantly lower compared training on the full CIFAR 100 dataset in Table 1. Interestingly, we notice that ReSA's loss decreases more slowly in the early stages, and its accuracy improves more gradually than other methods. We hypothesize that this may be due to noisy initial clusters in the early stages of training, causing clustering errors. However, we find that as training progresses, ReSA's accuracy continues to rise in the mid-phase, surpassing all other methods. This suggests that ReSA is able to gradually resolve these issues and learn the correct clustering patterns as training advances, rather than amplifying errors. Overall, this experiment demonstrates that ReSA can also learn effective representations on long-tailed datasets.
set. As observed
15-
252
ReSA
MoCoV3
225
INTL
ViCReg
30
Epoch
Epoch
Epech
Epoch
0.00
Linear top-1 DiCC
1.4
Linear Eop-5 atCC
Kon
RecA PuCIVS
MCA Mcco/3
405
204
100
Epoch
Epoch
Epech
Figure 12. Pretraining the long-tailed dataset. Here We report the training losses of four self-supervised learning methods the CIFAR100-LT dataset, along with their evalutaion performance on the full CIFAR-100 test set as measured by a linear probe and a k-NN classifier.
19

---

Clustering Properties of Self-Supervised Learning
C.4. The impact of weak augmentation on other methods
Since the design of weak augmentation (identical to that in ReSSL (Zheng et al., 2021)) provides certain degree of performance improvement for ReSA, also examine its impact other self-supervised learning methods. Specifically, select SwAV, VICReg, and MoCoV3, strictly following the experimental configurations in solo-learn (da Costa et al., 2022), with the only modification being the replacement of the image augmentation settings. As shown in Table 12, weak augmentation do not enhance the performance of these methods. This is likely because the standard augmentation settings have been extensively tuned through numerous experiments, ensuring optimal training conditions for these approaches.
Table 12. Investigate the impact of weak augmentation on other methods.
CIFAR-10
CIFAR- 100
Method
weak aug.
linear
k-nn
linear
k-nn
x 89.17
84.18
64.88
53.32
SwAV
V
88.79
84.01
64.21
53.07
x 92.07
87.38
68.54
56.32
VICReg
91.35
86.75
67.27
55.79
x 93.10
89.06
68.83
58.09
MoCoV3
V 93.05
89.09
68.72
58.02
C.5. Discussion and Future Work
The relationship between image augmentation and SSL via joint embedding architectures has grown increasingly intricate. Over the past few years, many studies (Chen et al., 2020b; Grill et al., 2020; Wagner et al., 2022; Morningstar al., 2024) have emphasized the critical role of image augmentation JEA, demonstrating that making subtle modifications image augmentations, such merely adjusting the parameters of ResizedCrop and ColorJitter, the performance of SSL models. However, recent works (Assran et al., 2023; Moutakanni et al., 2024) have begun to challenge this paradigm by exploring new self-supervised learning frameworks that climinate the reliance on hand-crafted data augmentations. These efforts argue that specific augmentations may introduce strong biases that could be detrimental to certain downstream tasks (Assran et al., 2022a) and that the most effective augmentations are often task-specifie, depending on the domain, rather than adhering to universally hand-crafted settings (Bendidi et al., 2023; Asano et al., 2019; Geiping al., 2022; Purushwalkam Gupta, 2020).
significantly impact
Interestingly, Moutakanni et al. (2024) successfully demonstrate that hand-crafted domain-speciff data augmentations not essential for training state-of-the-art joint embedding architectures when scaling self-supervised learning. Their findings reveal that, with sufficiently large datasets, simple crop of images alone can achieve remarkable results. Notably, this observation aligns perfectly with the characteristics of ReSA. As we show in Section 4.3, removing any single transformation enhances the clustering properties of the representations learned during training, enabling ReSA to better capture the inherent clustering information within the data. When the dataset size is sufficiently large, eliminating all hand-crafted data augmentations perfectly unleashes the innate potential of ReSA. We look forward future research validating this hypothesis and applying ReSA to large-scale pretraining scenarios.
20


Reviews:
[{"review_id": "IhJridbU4v", "reviewer": "Reviewer_69BS", "summary": "This paper investigates the clustering properties inherent in self-supervised learning (SSL) through joint embedding architectures. The authors empirically demonstrate that encoder outputs (encodings) exhibit superior clustering quality compared to projector embeddings, as measured by silhouette coefficients and adjusted Rand indices. Building on this insight, They propose Representation Soft Assignment (ReSA), a novel SSL framework that leverages online self-clustering on encodings to guide representation learning through soft assignment targets. ReSA doesn't require learnable prototypes and executes Sinkhorn-Knopp clustering only once per iteration, achieving computational efficiency while preserving semantic structure.   A weak augmentation strategy that balances clustering stability and invariance learning further enhances the method's effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 69BS,\n\nWe sincerely thanks for your time and the valuable feedback for this paper. \n\n**Why encodings inherently possess better clustering capability**:\nWe would like to thank the reviewers for their insightful discussion regarding the distinction between encoding and embedding. To the best of our knowledge, the projector has become an indispensable component of JEA-based SSL. However, the dynamics of its optimization and the reasons behind its success remain an open question within the community. Some works have attempted to explain these principles. For example, Jing et al. (2022) [1] empirically discovered that applying SSL loss either to encoding or embedding led to a significant decrease in the rank of the corresponding features. They argued that this rank reduction indicates a loss of diverse information, which, in turn, reduces generalization capability. This explanation aligns with the hypothesis in SimCLR [2], **where the additional projector acts as a buffer to prevent information degradation of the encoding caused by the invariance constraint**. Additionally, Gupta et al. (2022) [3] 's null space analysis for the projector posited that the projector might implicitly learn to select a subspace of the encoding, which is then mapped into the embedding. In this way, only a subspace of the encoding is encouraged to be style-invariant, while the other subspace can retain more useful information.\n\nTherefore, the SSL constraint can cause the embedding to lose information, which may include not only clustering-irrelevant features such as background information, but also class-relevant information, making it difficult for us to determine which—encoding or embedding—exhibits better clustering properties in this context. Based on this, our paper empirically demonstrates that the projector indeed filters out information that could support clustering, while encoding achieves better clustering performance. \n\n**theoretical analysis on ReSA**: We are truly grateful for your expectations and support for this work. As clarified above, the theoretical investigation of the projector remains an open research challenge. The community currently lacks definitive theoretical evidence explaining why the encoding can achieve superior downstream performance compared to the embedding. Building on your insights, we can first conduct interpretability experiments to examine which class-relevant and class-irrelevant features are captured by both the encoding and the embedding, thereby demonstrating how the encoding may possess better clustering properties. We also appreciate your suggestion to quantify this semantic preservation difference through methods such as information bottleneck theory. This could be a valuable approach to theoretically investigate why encodings, as opposed to projector embeddings, may lead to better clustering. We will look into incorporating this analysis in the revised version of the paper to provide a more comprehensive theoretical foundation.\n\n**error risk**: We agree that the paper could benefit from a more rigorous analysis of how the ReSA loss function handles error propagation. In the context of self-supervised learning, early clustering errors are inevitable, as there is no label information or other prior knowledge to guide the process. This is a challenge faced by most clustering-based SSL methods. However, numerous experiments have shown that ReSA exhibits relatively slow performance improvement in the early stages of training, but achieve significantly better results in the later stages compared to other methods, as shown in Figure 9 of our paper and https://anonymous.4open.science/r/long-tailed-evaluation-2843. This suggests that while ReSA may suffer from incorrect cluster assignments in the early training stages, which slows down learning, it is able to gradually suppresses error propagation and learn the correct clustering patterns as training progresses, rather than amplifying the errors. \n\n**projector-free SSL baselines**: To the best of our knowledge, the projector has become an indispensable component of JEA-based SSL. Traditional projector-free baselines, such as the MoCo (He et al. 2020), no longer achieve competitive results, while newer versions like MoCo V2 and V3 have demonstrated the projector’s crucial role in enhancing representational generalization.\n\n**long-tailed data**: Dear Reviewer, please refer to https://anonymous.4open.science/r/long-tailed-evaluation-2843.\n\n**Transfer to Downstream Tasks**: In addition to the experiments described in Section 5.3, we also perform transfer learning on fine-grained datasets (Table 6) and conduct a low-shot evaluation (Table 10). We will include these details in Section 5.3 as well.\n\n[1] Understanding dimensional collapse in contrastive self-supervised learning.\n\n[2] A Simple Framework for Contrastive Learning of Visual Representation.\n\n[3] Understanding and Improving the Role of Projection Head in Self-Supervised Learning.", "claims_and_evidence": "Some important claims lack justification:\n1. The core premise \"encodings inherently have better clustering properties\"—is presented as an empirical observation (Sec 3) but lacks theoretical grounding. The paper empirically demonstrates that encoder outputs (H) exhibit stronger clustering properties than projector embeddings (Z) via metrics like SC/ARI (Fig 3-4). However, no theoretical analysis explains why encodings inherently possess better clustering capability. Does the encoder’s position (pre-projector) naturally filter augmentation noise or preserve semantic information? It is suggested to quantify the semantic preservation difference of H/Z through methods such as information bottleneck theory.\n2. There is an error propagation risk that exists: The positive feedback of ReSA depends on the quality of cluster assignment, If the initial clusters are noisy (such as in the early stages of training), the doubly stochastic matrix AH reinforces incorrect associations.\nWhile Figure 11 suggests that Sinkhorn-Knopp mitigates early noise, there is no proof that the ReSA loss function suppresses error propagation. The paper lacks quantitative evidence that ReSA robustly handles noisy clusters. No analysis shows whether initial noisy assignments (e.g., low-ARI epochs) get amplified via the feedback loop.", "methods_and_evaluation_criteria": "In the section \"3. Exploring Clustering Properties of SSL\"， the evaluation exclusively focuses on SSL methods with projector heads, omitting architectures without projectors. This introduces selection bias, as the observed encoding superiority (Sec 3) might be contingent on projector-augmented frameworks rather than intrinsic to encodings. \nSuggested Improvement: Validate clustering metrics on projector-free SSL baselines to disentangle whether the encoding advantage stems from architectural constraints or fundamental representation properties.", "theoretical_claims": "The paper does not present formal theoretical proofs so it is suggested that providing the theoretical proof of the core premise—\"encodings inherently have better clustering properties\"", "experimental_designs_or_analyses": "Experiments only use balanced datasets (e.g., CIFAR, ImageNet). There is no validation on long-tailed or domain-shifted data. It is suggested that test ReSA under imbalanced data distributions** to assess whether clustering-guided learning amplifies biases.", "relation_to_broader_scientific_literature": "The paper is highly related to these deep clustering methods based on SSL such as contrastive clustering.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. The theoretical basis of this paper is insufficient, for the core premise-“encodings inherently have better clustering properties”, even though the authors try to prove it from an experimental point of view, I still think the experimental results are not significant enough.\n2. The paper is well presented in terms of iconography and text, with a good summary of previous work and clear claims.\n3. The innovation of this paper is okay, and the use of soft assignment does solve the problem that samples belonging to the same category may be unintentionally pushed farther away during the training process, thus destroying the underlying semantic clustering structure, and making full use of the hidden information contained in the encoding. It would be an excellent work if the authors could add the theoretical basis for the important points in the paper and optimize the noise robustness of the model.", "other_comments_or_suggestions": "It is recommended that the section “5.3. Transfer to Downstream Tasks” be more detailed, as this is an advantage of SSL to better assess the validity of the ReSA methodology proposed in this paper.", "questions_for_authors": "See the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aeZznE0Bx5", "reviewer": "Reviewer_PEEd", "summary": "The paper presents a new self-supervised learning method that encourages clustering in the output \"embedding\" layer based on clustering found in the earlier \"encoding\" layer used as representations for downstream tasks. \n\nThe paper analyses clustering metrics on representations extracted at different layers of the NN architecture (encoder + projector), to identify where clustering might be better applied and thereby derive a training objective that combines concepts used previously in SSL in a new, arguably simpler way.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PEEd,\n\nWe would like to begin by sincerely thanking you for the insightful and valuable comments. Below, we address each of the comments in detail.\n\n**Relation to Broader...**: \nWe have revisited these papers, and will cite and discuss them in the new manuscript. \n- We will further investigate the ELBO proposed by Bizeul et al. which may be helpful to derive ReSA's bounds. \n- MuGS and MSN are both excellent works that aim to enhance SSL performance in ViTs. In contrast, this work is grounded in traditional SSL benchmarks to demonstrate the effectiveness of encoding-based positive feedback SSL, which is why our experiments primarily use ResNet models. In the past two months, we also explored the effectiveness of ReSA pretraining in ViTs using framework of DINOv2, achieving **77.8%** k-NN accuracy with ViT-B after just 100 epochs, compared to DINOv2's **76.0%** after 100 epochs and MuGS’s **78.0%** after 400 epochs. \n\n**087R**:  Here, we refer to SSL methods that apply clustering constraints on the embedding (i.e., the output of the projector). Since DeepCluster (Caron et al., 2018) does not include a projector, this limitation was addressed and improved as DeepCluster-v2 noted in the SwAV (Caron et al., 2020). We will clarify this point.\n\n**086L**: In the definition of joint embedding architectures (JEA), it is feasible to use either two identical (i.e., one shared) or different neural networks. In practice, using the same architecture without sharing parameters (i.e. the teacher-student distillation with momentum updates as used in BYOL, DINO, and MoCoV3) often yields the best performance. For generality, we do not emphasize the detailed choices of neural networks here, as different methods may adopt different strategies. \n\n**083R**: Thanks again for this constructive question. Due to word count limitations, we kindly ask you to refer to our response to Reviewer 69BS, specifically under the section **Why encodings inherently possess better clustering capability**. We sincerely appreciate your understanding and attention to this matter.\n\n**085R**: As we clarify above, the invariance constraint imposed by the SSL loss can cause the embedding to lose diverse information, which may include clustering-irrelevant features such as background information, color redundancy, viewpoint variations and so on. In such cases, fewer clustering-irrelevant features contribute to better clustering performance. However, the invariance constraint may also lead to the loss of class-relevant information, making it difficult for us to determine which—encoding or embedding—exhibits better clustering properties in this context.\n\n**100R**: \n- Local clustering capability refers to how well the representation partitions the data into meaningful local clusters. A higher mean silhouette coefficient for the representation indicates better separation and tighter grouping of data points within their respective clusters, which demonstrates stronger clustering capability.\n- Local clustering stability is quantified by the standard deviation of the silhouette coefficient. A lower standard deviation signifies that the clustering results are more consistent across the data points, meaning fewer outliers and more stable cluster assignments. This indicates that the clustering representation is robust and less sensitive to variations or noise in the data. We would clarify these more clearly in appendix A.\n\n**142L**: When we talk about the 'ability' of a set of vectors, we are referring to how well they can represent the underlying structure of the data. In this case, 'Global clustering capability' looks at how well the entire set of data is grouped into clusters as a whole, considering the overall structure of the data. As we use k-mean when calculating ARI, so a higher ARI value also indicates that the relationships between the vectors can be well captured and measured using the similarity.\n\n**208L**: \n- We sincerely appreciate your valuable feedback. We will explicitly revise $\\mathbf{S} _\\mathbf{Z}$ to $\\mathbf{Z}^\\top \\mathbf{Z}'$ for better clarity.\n- Although $\\mathbf{S} _\\mathbf{H}$ is a symmetric matrix, the iterative row and column normalization inside the Sinkhorn-Knopp algorithm does not preserve symmetry. This is because each row and column can receive different normalization factors during each iteration. As a result, even a symmetric input matrix can become asymmetric after being processed by the Sinkhorn updates.\n\n**214L**: \n- Yes, they indeed represent the same softmax functions. Here we stated them twice to emphasize that these distributions are respectively computed row-wise and column-wise. We appreciate your feedback and will revise it to convey this more clearly.\n- $\\mathbf{1} _m \\in \\mathbb{R}^{m\\times 1}$ stands for all-ones column vector as we note in (144R) of Algorithm 1. Here, we express the softmax functions in the form of matrix multiplication, where the denominator corresponds to the sum of each row.", "claims_and_evidence": "The paper claims to achieve superior performance to existing methods, which seems well justified, subject to including all relevant comparators. (see below)", "methods_and_evaluation_criteria": "The method is based on appropriate analysis (of clustering in NN layers) and applies clustering, used extensively in prior SSL works, in a novel way that makes intuitive sense.", "theoretical_claims": "The paper includes no theoretical claims.\n\nThe paper would be improved if it considered how the proposed training objective fits with current theoretical rationale for SSL (see below).", "experimental_designs_or_analyses": "The protocol seems standard for SSL papers", "relation_to_broader_scientific_literature": "* (025R) Caron et al. (2018) should be cited here as one of the first to do this with modern NNs\n* (029R) the introduction links clustering and representation learning empirically and theoretically (029R-038R), the latter should include Bizeul et al. (https://arxiv.org/pdf/2402.01399), which proposes a latent variable model that unifies various SSL methods, including clustering methods so seems directly relevant. \n  - The clustering nature of ReSA and close relationship shown to InfoNCE suggest ReSA appears consistent with their model. Discussion of this would improve the paper, perhaps adding theoretical justification for why ReSA works/what is implicitly assumed about the data.\n* (037L) \"Representation Learning with Contrastive Predictive Coding\" van den Oord et al. (InfoNCE)  would seem appropriate to cite here as it was a key initiator of recent interest in SSL models.\n* Recent works: MuGS and MSN (https://arxiv.org/pdf/2204.07141, https://arxiv.org/pdf/2203.14415) are not referenced or compared to. \n    - Please discuss these and reconcile their results, e.g. for ImageNet, with those in the paper.", "essential_references_not_discussed": "See \"Relation to Broader..\" (above)\n- MUGS and MSN (https://arxiv.org/pdf/2204.07141, https://arxiv.org/pdf/2203.14415)\n- Bizeul et al. (https://arxiv.org/pdf/2402.01399)", "other_strengths_and_weaknesses": "* Well written and clear paper. \n* Prepared to raise score if concerns are appropriately addressed.", "other_comments_or_suggestions": "087R: The citation is DeepCluster, text says DeepCluster2?", "questions_for_authors": "* 086L: Do you train 1 neural network function or 2 independent functions with distinct parameters? This gives the impression of 2, but later it seems only 1. If 1, then this presentation is misleading and should be made more clear and consistent with the paper and model that follows. This is important for a clear understanding of how representations are a function of the data.\n* 083R: \"it remains unclear..\" - Why would this *not* continue through all layers of the network (a layer doesn't \"know\" which part its in)?\n* 085R: \"embedding tends to be less influenced by clustering irrelevant features\" - what does this mean?\n* 100R: what is clustering \"capability\" and \"stability\" in local clustering? (e.g. stability with respect to what?)\n* 142L: \"These include excellent local clustering ability .. and stability ..., global clustering capability and similarity measure effectiveness (ARI).\" Can you put this in clearer every day language?   What is the \"ability\" of a set of vectors, etc?\n* 208L: $S_M$ for a matrix $M=H$ is already defined (137R) to be symmetric so taking transpose in Eq 2 appears to make no sense. But you then ***re**define* $S_M$ below (for $M=Z$). \n    - It would be clearer to explicitly put $Z^TZ'$ in Eq 2, which is then directly comparable to Eq 3.\n* 208L: Since $S_H$ is symmetric, is $A_H$ not symmetric also? If so, why transpose? (confusing)\n* 214L: no need to define the same function twice for different arguments (confusing)\n* 214L: what is the normalisation constant? Is it not the sum over all of S?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWdfDFVNui", "reviewer": "Reviewer_kGg9", "summary": "This paper studies the clustering properties of self-supervised learning. This paper finds that the encoder's output exhibits superior and more stable clustering properties than other components. Based on insight, this paper proposes a novel positive feedback method to improve the representation ability further. \n\nThe idea of positive-feedback is interesting. Extensive experiments validate the feasibility of this idea. \n\nThe finding on the clustering properties of encoding is interesting. However, the underlying reason is not well explained.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kGg9,\n\nWe sincerely thanks for your time and the valuable suggestions to improve this paper. Here we address each of the comments in detail.\n\n**Why encoding and embedding have different clustering properties**: Thanks for this constructive question. Due to word count limitations, we kindly ask you to refer to our response to Reviewer 69BS, specifically under the section **Why encodings inherently possess better clustering capability**. We sincerely appreciate your understanding and attention to this matter.\n\n**How to get the ARI values**: In Definition A.2 of Appendix A, we provide a detailed definition and formula for the Adjusted Rand Index (ARI). For practical computation, you may also refer to \"sklearn.metrics.adjusted_rand_score\".\n\n**Theoretical foundations of the findings**: Thanks sincerely for this insightful suggestion. The theoretical exploration of the distinction between encoding and embedding remains a challenge in SSL. We will follow your suggestion and continue to investigate how ReSA facilitates positive feedback learning, optimizes the model's noise robustness, and whether it offers improved bounds compared to other methods. \n\n**$\\mathbf{A} _\\mathbf{H}$ seems computationally heavy.**: We note that the Sinkhorn-Knopp algorithm, which is used to compute $\\mathbf{A} _\\mathbf{H}$, does **not** involve gradient propagation and requires only three iterations, making it both efficient and practical as reported in Caron et al. (2020). To quantitatively describe the computation time of the Sinkhorn-Knopp Algorithm, we conducted experiments and found that using an A100-PCIE-40GB GPU, the time for a single computation of a 1024 * 1024 tensor matrix (as used in our experiments) is **0.001578** seconds. This means that when training ResNet-50 on ImageNet for 1 epochs (1251 iterations), the total time for Sinkhorn-Knopp algorithm is about 1.974 seconds. As shown in Table 4, the overall training time is 0.16 × 3600 = 576 seconds, so the proportion of time spent on Sinkhorn-Knopp algorithm is approximately 1.974 / 576, which is about **0.3%**. Moreover, when using larger models, the time spent on the Sinkhorn-Knopp algorithm will represent an even smaller proportion of the total computation time.\n\n[1] Caron et al. (2020). Unsupervised learning of visual features by contrasting cluster assignments.\n\n**What if an error exists (some samples belonging to different classes are in the same cluster)? Will the proposed method boost this error?**: Thank you for this insightful question. In the context of self-supervised learning, the issue of some samples from different classes being assigned to the same cluster is quite likely, especially when there is no label information guiding the process. This is a challenge faced by most clustering-based SSL methods. However, numerous experiments have shown that ReSA exhibits relatively slow performance improvement in the early stages of training, but achieve significantly better results in the later stages compared to other methods, such as contrastive or decorrelation-based SSL, as shown in Figure 9 of our paper and https://anonymous.4open.science/r/long-tailed-evaluation-2843. This suggests that while ReSA may suffer from incorrect cluster assignments in the early training stages, which slows down learning, it is able to gradually address these issues and learn the correct clustering patterns as training progresses, rather than boosting the errors. Furthermore, clustering-based SSL methods like SwAV, DINO and ReSA typically employ clustering assignments with sharp distribution, which help mitigate the impact of incorrect assignments. Investigating the optimization dynamics of clustering-based SSL is an intriguing theoretical direction, and we hope to explore this area in the future. \n\n**The assignment matrix and the stochastic matrix**: We note that $\\mathbf{A} _\\mathbf{H}$ is the clustering assignment matrix obtained by applying the Sinkhorn-Knopp algorithm to $\\mathbf{S} _\\mathbf{H}$. The purpose of the Sinkhorn-Knopp algorithm is to ensure that the output is a doubly stochastic matrix, where each row and column sums to 1. Therefore, $\\mathbf{A} _\\mathbf{H}$ is a doubly stochastic clustering assignment matrix, and the matrices $\\mathbf{A} _\\mathbf{H}$ on Line 139 and 146 are exactly identical.\n\n**If the encoding replaces the embedding, will Eq. (3) produce better performance?**: Thank you for this thought-provoking question. In Eq. (3), SwAV requires learnable prototypes to perform clustering. If the encoding is used in place of the embedding—namely $\\mathbf{Q}=sinkhorn(\\mathbf{H}^\\top \\mathbf{C})$—we must ensure that the encoding and embedding output dimensions match. We tested this approach in early experiments on ImageNet-100 and found that, although training proceeded normally, the resulting performance was quite poor. We speculate that this may be due to the difficulty of effectively optimizing the prototypes under this condition.", "claims_and_evidence": "The clustering properties of encoding are interesting. \nHowever, I cannot understand why encoding and embedding have different clustering properties. To my knowledge, both embedding and encoding can be regarded as features of neural networks with various depths. So why will adding more layers (from encoding to embedding) degrade the clustering properties? \n\nIn Figure 3, how to get the ARI values?", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This paper lacks a theoretical analysis. I suggest the author analyze the theoretical foundations of the findings.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "This paper is related to self-supervised learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nThe experiment is through. \n\nWeakness:\nThis paper needs to calculate the $A_H$, which seems computationally heavy.", "other_comments_or_suggestions": "No.", "questions_for_authors": "The idea of positive feedback is interesting. In line 117, we can see that this paper aims to enhance the clustering structure in the encoding. But my question is, what if an error exists (some samples belonging to different classes are in the same cluster)? Will the proposed method boost this error? \n\nIn 139, $A_H$ is a clustering assignment matrix. But in line 146, it becomes a doubly stochastic matrix. The assignment matrix and the stochastic matrix have different sizes. \n\nIn Eq. (3), if the encoding replaces the embedding, will Eq. (3) produce better performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work identifies an interesting observation: encoder outputs in joint embedding architectures display stronger and more stable clustering properties than projector embeddings. Building on this insight, the authors propose a new self-supervised learning method, ReSA, which introduces a positive-feedback mechanism to guide learning via soft clustering assignments. ReSA avoids the use of learnable prototypes, relies on a computationally efficient Sinkhorn-Knopp-based formulation, and achieves consistently strong results across standard SSL benchmarks.

While several reviewers initially expressed reservations about the lack of theoretical analysis and the potential risks of reinforcing early clustering noise, the authors provided thoughtful and comprehensive responses. They clarified the empirical basis for the encoder's superior clustering properties and outlined relevant directions for developing theoretical support in future work. They also presented evidence suggesting that the proposed method is robust to early-stage noise, gradually improving clustering quality as training progresses. Additional clarifications regarding the model architecture, efficiency of the Sinkhorn step, and practical advantages over baseline clustering methods were well-received by the reviewers.

Following the rebuttal, all reviewers either maintained or increased their scores, and acknowledged that their key concerns had been addressed. The paper presents a novel and well-motivated approach that is both simple and effective, and is likely to be of interest to the community studying SSL and clustering-based learning dynamics.
Decision: Accept (poster)