Reverse Engineering Self-Supervised Learning
Authors: Ido Ben-Shaul
Ravid Shwartz-Ziv
Tomer Galanti
Shai Dekel
Yann LeCun
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-Supervised Learning
Deep Learning
Representation Learning

Abstract: Understanding the learned representation and underlying mechanisms of Self-Supervised Learning (SSL) often poses a challenge. In this paper, we ‘reverse engineer’ SSL, conducting an in-depth empirical analysis of its learned internal representations, encompassing diverse models, architectures, and hyperparameters. Our study reveals an intriguing process within the SSL training: an inherent facilitation of semantic label-based clustering, which is surprisingly driven by the regularization component of the SSL objective. This clustering not only enhances downstream classification, but also compresses the information. We further illustrate that the alignment of the SSL-trained representation is more pronounced with semantic classes rather than random functions. Remarkably, the learned representations align with semantic classes across various hierarchical levels, with this alignment intensifying when going deeper into the network. This ‘reverse engineering’ approach provides valuable insights into the inner mechanism of SSL and their influences on the performance across different class sets.

Reverse Engineering Self-Supervised Learning
Ido Ben-Shaul
Ravid Shwartz-Ziy"
Department of Applied Mathematics Tel-Aviv University & eBay Research
New York University
swwarts. zi zivōnyu. edu
ido. Sarchauu@@maill com
Tomer Galanti'
Shai Dekel
Massachusetts Institute of Technology gal ant 0mit..cm edu
Department of Applied Mathematics
Tel-Aviv University
shaideekel6@gmaii.ccom com
Yann LeCun
New York University & Meta AI, FAIR yann@cs.my nyu. edu
Abstract
Self-supervised learning (SSL) is a powerful tool in machine learning, but under- standing the learned representations and their underlying mechanisms remains a challenge. This paper presents an in-depth empirical analysis of SSL-trained representations, encompassing diverse models, architectures, and hyperparameters. Our study reveals an intriguing aspect of the SSL training process: it inherently facilitates the clustering of samples with respect to semantic labels, which is surpris- ingly driven by the SSL objective's regularization term. This clustering process not only enhances downstream classification but also compresses the data information. Furthermore, we establish that SSL-traines representations align more closely with semantie classes rather than random classes. Remarkably, we show that learned rep- resentations align with semantic classes across various hierarchical levels, and this alignment increases during training and when moving deeper into the network. Our findings provide valuable insights into SSL's representation learning mechanisms and their impact on performance across different sets classes.
1 Introduction
Self-supervised learning (SSL) [12, 5] has made significant progress over the last few years, almost reaching the performance of supervised baselines on many downstream tasks [42, 4, 31, 30, 47, 34, 37, 66, 16]. However, understanding the learned representations and their underlying mechanisms remains a persistent challenge due to the complexity of the models and the lack of labeled training data. Moreover, the pretext tasks used in sef-suupevviiion frequently lack direct relevance to the specific downstream tasks, further complicating the interpretation of the learned representations.
In supervised classification, however, the structure of the learned representations is often very simple. For instance, a recent line work [50, 36] has identified a universal phenomenon called neural collapse, which occurs at the terminal stages of training. Neural collapse reveals several properties of the top-layer embeddings, such as mapping samples from the same class to the same point, maximizing the margin between the classes, and certain dualities between activations and the top- layer weight matrix. Later studies [10, 28, 26, 56, 62] characterized related properties at intermediate layers of the network.
Equaa contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Initialization
End of train
Sample
Original
Super
SSL training induced semantic clustering, UMAP visualization of the SSL representa- tions before and after training in different hierarchies. (top) Augmentations five different samples, each sample colored distinctly. (middle) Samples from five different classes within the standard CIFAR-100 100 dataset. (bottom) Samples from five different superelasses within the dataset.
Figure I:
Compared to traditional classification tasks that aim to accurately categorize samples into spe- cific classes, modern SSL algorithms typically minimize loss functions that combine two essen- tial components: one for clustering augmented samples (invariance constraint) and another for preventing representation collapse (regularization constraint). For example, contrastive learning methods [14, 48, 63, 46] train representations to be indistinguishable for different augmentations of the same sample and at the same time to distinguish augmentations of different samples. On the other hand, non-contrastive approaches [6. 66, 34] use regularizers to avoid representation collapse.
Contributions. In this paper, provide in-depth analysis of representation learning with SSL through a series of carefully designed experiments. Our investigation sheds light on the clustering process that occurs during training. Specifically, our findings reveal that augmented samples exhibit highly clustered behavior, forming centroids around the mean embedding of augmented samples sharing the same original image. More surprisingly, we observe that clustering also occurs with respect to semantic labels, despite the absence of explicit information about the target task. This demonstrates the capability of SSL capture and group samples based on semantic similarities.
Our main contributions are summarized as follows:
Clustering: We investigate the clustering properties of SSL-trained representations. In Figure we show that akin to supervised classification [50], SSL-trained representation functions exhibit a centroid-lik geometric structure, where feature embeddings of augmented samples belonging to the same image tend to cluster around their respective means. Interestingly, at later stages of training, a similar trend appears with respect to semantic classes.
The role of regularization: In Figure 3 (left) we show that the accuracy of extracting (i.e., linear probing) semantic classes from SSL-trained representations continuously improves even after the model accurately clusters the augmented training samples based on their sample identity. As can be seen, the regularization constraint plays key role in inducing the clustering of data into semantic attributes at the embedding space.
Impact of randomness: We argue that SSL-trained representation functions are highly correlated with semantic classes. To support this claim, we study how the degree of randomness in the targets affects the ability to learn them from pretrained representation. As we show in Figure 4, the alignment between the targets and the learned representations substantially improves as the targets become less random and more semantically meaningful.

---

Learning hierarchic representations: We study how SSL algorithms learn representations that exhibit hierarchic classes. We demonstrate that throughout the training process. the ability to cluster and distinguish between samples with respect to different levels of semantic targets continually improves all layers of the model. We consider several levels of hierarchy, such as the sample level (the sample identity), semantic classes (such as different types of animals and vehicles), and high-level classes (c.g., animals, and vehicles).
Furthermore, we analyze the ability of each layer capture distinct semantic classes. As shown in Figure 5, we observe that the clustering and separation ability of each layer improves as we move deeper into the network. This phenomenon closely resembles the behavior observed in supervised learning [10, 26, 56, 62], despite the fact that SSL-trained models are trained without direct access to the semantic targets.
In Section 4, we investigate various clustering properties in the top layer of SSL-trained networks. In Section 5, we conduct several experiments to understand what kinds of functions are encoded in SSL.-trained neural networks and can extracted from their features. In Section 6, we study how SSL al algorithms learn different hierarchies and how this occurs at intermediate layers.
2 Background and Related Work
2.1 Self-Supervised Learning
SSL is family of techniques that leverages unlabeled data to learn representation functions that be easily adapted to new downstream tasks. Unlike supervised learning. which relies on labeled data, SSL employs self-defined signals to establish proxy objective. The model is pre-trained using this proxy objective and then fine-tuned on the supervised downstream task. However, a major challenge of this approach is to prevent 'representation collapse', where the model maps all inputs to the same output [39].
Several methods have been proposed to address this challenge. Contrastive learning methods such as SimCLR [14] and its InfoNCE criterion [64] learn representations by maximizing the agreement between positive pairs (augmentations of the same sample) and minimizing the agreement between negative pairs (augmentations of different samples). In contrast, non-contrastive learning methods [6, 66, 34] replace the dependence on contrasting positive and negative samples, by applying certain regularization techniques to prevent representation collapse. For instance, some methods use stop-gradients and extra predictors to avoid collapse [16, 34J, while others use an additional clustering constraints [13].
Variance- Invariane-vaaieeeeeee www.eaaannnne---oaaiaaneee Regularization (VICReg). A widely used method for SSL training [6], which generates representations for both an input and its augmented counterpart. It aims optimize two key aspects:
Invariance loss The squared Euclidean distance between pairs of embedding, serving to ensure consistency between the representation of the original and augmented inputs.
Regularization Comprising two elements, the variance loss, which promotes increased variance across the batch dimension through a hinge loss restraining the standard deviation, and the covari- ance loss, which penalizes off-diagonal coefficients of the covariance matrix of the embeddings to foster decorrelation among features.
This leads to the VICReg loss function:
L(f) As Az(Z + v(Z)) plc(Z) cZZ'),
(1)
Invariance
Regularization
where L(f) is minimized over batches of samples I. s(Z, Z'), v(2), and c(2) denote the invariance, variance, and covariance losses respectively, and X, μ4, v > are hyperparameters controlling the balance between these loss components.
A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). The SimCLR method [14], another popular approach, minimizes the contrastive loss function [35]. Given two batches of embeddings Z, Z', can also decompose the objective into an invariance term and
3

---

regularization term:
B
((f)
(im Zi, Z,)TT)
log
exp(sim( 2j)/T),
Invariancs
Jiht
Regularization
where sim(a,
Tb is the cosine similarity between vectors a and b, and is ||6]
"temperature" parameter that controls the sharpness of the distribution. Intuitively, minimizing the loss encourages the representations Zi and Z of the same input sample to be similar (i.e., have high cosine similarity) while pushing away the representations Z; and Zj of different samples (i.e..
have a low cosine similarity) for all j E [B].
2.2 Neural Collapse and Clustering
In a recent paper [50], was demonstrated that deep networks trained for classification tasks exhibit notable behavior: the top-layer feature embeddings of training samples of the same class tend to concentrate around their respective class means, which are maximally distant from each other. This behavior is generally regarded desirable since max- max-mnrgin classifiers tend to exhibit better generalization guarantees (c.g., [2, 7, 33J,, and clustered embedding spaces are useful for few-shot transfer learning (e.g., [32, 28, 27, 29])).
In this paper, we aim to explore whether similar clustering tendencies occur in SSL-trained represen- tation functions at both the sample level (with respect to augmentations) and at the semantic level. To address this question, we first introduce some key aspects of neural collapse. Specifically, we will focus on class- feasure variance collapse and nearest class-center (NCC) separability.
To measure these properties, we the following metrics. Suppose we have representation function f Rd > R" and a collection of unlabeled datasets $I....,Su Sc C Idd (each corresponding to different class). The variibiiit collapse measures to what extent the samples from the same class are mapped to the same vector. To quantify this property we use the averaged class- distance normalized variance (CDNV) [28] (V,,, = Avg=,y ( (S,) where (S) and (S) are the mean and variance of the uniform distribution U[(((z) SH.
(5)
A weaker notion of clustering is the nearest class-center (NCC) separability [26], which sures to what extent the feature embeddings of training samples form centroid-like geometry. The NCC separability asserts that, during training, the penultimate-layer's feature embeddings of training samples can be accurately classified with the 'nearest class-center (NCC) classifier', h(x)= = arg min [lf()) (S.))l To measure this property, compute the NCC train and test accuracies, which simply the accuracy rates of the NCC classifier.
A recent paper [22] argued that idealized SSL-trained representations simultaneously cluster data into multiple equivalence classes. This implies that there are various ways categorize the data that result in high NCC accuracy. However, achieving a very small CDNV for multiple different categorizations impossible since a zero CDNV would mean that feature embeddings of two samples from the same category are identical. Consequently, an intriguing question arises: with respect to which sets of classes do the learned feature embeddings cluster?
2.3 Reverse Engineering Neural Networks
Reverse engineering neural networks have recently garnered attention as an approach to explain how neural networks make predictions and decisions. While substantial work has been dedicated to understanding the functionalities of intermediate layers of neural network classifiers [23, 9, 17, 50, 51, 60, 56, 3], characterizing the functionalities of representation functions generated by SSL algorithms remains an open problem.
major source of complexity is the reliance on a pretext task and image augmentations. For instance, to fully understand the success of SSL it is essential to understand the relationship between the pretext task and the downstream task [58, 65). with SSL, and therefore, SSL algorithms are quite complicated and include many different types of losses [14, 66, 6] and optimization tricks [14, 16, 37]. Of note, not obvious what are the differences between representations that are learned with contrastive learning and non-contrastive
yet unclear how to properly train representations

---

Morm alted Linesrr Accuracy
1.6
Normskzod itatto
NDC Acrumnty
18
1-4
0.2
Hierarchy
Hienarehy Original
Hierarchy Oriiinal Swmple
Sample
Original
Sample
Super
100
1000
10 100
1000
10
100
1000
Epoch
Epoch
Epoch
Figure 2: SSL algorithms cluster the data with respect to semantic targets. (left) The normalized NCC train accuracy, computed by dividing the accuracy values by their value at initialization. (middle) The ratio between the NCC test accuracy and the linear test accuracy for per-sample and semantic classes normalized by its value at initialization. (right) The linear test accuracy rates, normalized by their values at initialization. All experiments are conducted on CIFAR-100 with VICReg training.
0.7
0.5
Motual Information
Accuracy
N0C Accurosy
2..0
0.2
Act:
4.6
Epoch
Epoch
Figure 3: (left) The regularization and invariance losses together with the original target linear test accuracy of an SSL-trained model over the course of training. (middle) Compression the mutual information between the input and the representation during the course of training. (right) SSL training learns clustered representations. The NCC train and test accuracy in supervised (with/without augmentation), and SSL. All experiments are conducted on CIFAR-100 100 with VICReg training.
learning methods [52]. this work, we make new strides to understand SSL-trained representations, by investigating their clustering properties with respect various types of classes.
3 Problem Setup
Self-supervised learning (SSL) is frequently used to pretrain models, preparing them for adaptation to downstream tasks. This raises essential question: How does SSL training influence the learned representation? Specifically, what underlying mechanisms are at work during this training, and which classes can be learned from these representation functions? To investigate this, train SSL networks across various settings and analyze their behavior using different techniques. All the training details can be found in Appendix A.
Data and augmentations. Throughout all of the experiments (in the main text) used the CIFAR-100 [41] image classification dataset, which contains 100 classes of semantic objects (original classes), which are also divided into 20 superelasses, with each superclass containing classes (for example, the 'fish' superclass contains the 'aquarium fish', 'flatfish', 'ray', 'shark" and 'trout' categories). In order to train our models used the image augmentation protocol introduced in SimCLR [14]. Each SSL training session is carried out for 1000 epochs, using the SGD optimizer with momentum. Usually, SSL-trained models tested on a variety of downstream datasets assess their performance. To simplify our analysis and minimize the influence of potential distribution shifts, we specifically evaluate the models' performance on the CIFAR- 100 dataset. Such evaluations are typically done in SSL research [6, 34, 66], usually training and evaluating on ILSVRC (ImageNet). Additional evaluations on various datasets (CIFAR-10 [41], FOOD101 [11], Aircrafts [45], Tiny Imagnet [43], and Imagenet [19]) be found in Appendix B.2.

---

0.58
Epech 20
Metric
1.8
NOC
CONY
1 Aroumny
Lipear Aocuracy
Dicanca Sqg
0,45
400
Varsince
LOC0
1.2
0.30
0.8
0.95
Pgoch
Figure 4: SSL continuously learns semantic targets random ones. (lfft) The linear test accuracy for targets with varying levels of randomness from the last layers at different epochs. (middle) The linear accuracy for targets with varying levels of randomness for the trained model. (right) The ratios between non-random and random targets for various clustering metrics. All experiments are conducted CIFAR-100 with VICReg training.
Epoch
Egoch
Epoch
1.24
+-40
Scouray Rabio
-010
Lineee Acounry
100
Lining ACCUSSO
100
100
-4000
400
1.3
400
1000
1000
10000
0.5
6.0
2.10
1.13
1.0
19
Layer
Figure 5: SSL efficiently learns semantic classes throughout intermediate layers. The linear test accuracy of different layers of the model at various epochs (left) With respect to the 100 original classes. (middle) With respect to the 20 superclasses. (right) The ratio between the superclass and the original classes. All experiments are conducted on CIFAR-100 with VICReg training.
Backbone architecture. In all our experiments, we utilized the RES-L-H architecture as backbone, coupled with a two-layer multi-layer perceptron (MLP) projection head. The RES-L-H architecture a variant of the ResNet architecture [38] where the width remains constant across all L residual blocks as introduced in [26].
Linear probing. To evaluate the effectiveness of extracting given discrete function (e.g., categories) from a representation function, we employ linear probing. In this approach, we train linear classifier, also known as linear probe, on top of the representation, using set of training samples. To train the linear classifier, we used the Linear Support Vector Classification (LSVC) algorithm from the scikit-learn library [54]. The "linear accuracy" is determined by measuring the performance of trained linear classifier on validation set.
Sample level classification. We aim to evaluate the capacity of given representation function to correctly classify different augmentations originating from the same image and distinguish them from augmentations different images. To facilitate this, we constructed new dataset specifically designed to evaluate sample-level separability.
The training dataset comprises 500 random images sourced from the CIFAR-100 training set. Each image represents particular class and undergoes 100 different augmentations. As such, the training dataset contains 500 classes with total of 50000 samples. For the test set, utilize the same 500 images but apply 20 different augmentations drawn from the same distribution. This results in test consisting of 10000 samples. To measure the linear or NCC accuracy rates of given representation function at the sample level, we first compute relevant classifier using the training data and subsequently evaluate its accuracy rates on the corresponding test set.
Unraveling the Clustering Process in Self-Supervised Learning
The clustering process has long played a significant role in aiding the analysis of deep learning models [1, 17, 50]. To gain intuition behind the SSL training, in Figure I we present UMAP [57]

---

visualization of the network's 's embedding space for the training samples, both pre-and post-training
across different hierarchies.
As anticipated, the training process successfully clusters together samples at the per-sample level, mapping different augmentations of the same image (as depicted in the first row). This outcome is unsurprising given that the objective inherently encourages this behavior (through the invariance loss term). Yet, more remarkably, the training process also clusters the original 'semantic classes' from the standard CIFAR-100 dataset, even though the labels were absent during the training process. Interestingly, the high-level hierarchy (the superclasses) is also effectively clustered. This example demonstrates that while training procedure directly encourages clustering at the sample level, the SSL-trained representations of data are additionally clustered with respect to semantic classes across different hierarchies.
To quantify this clustering process further, we train a RES-10-250 network using VICReg. We measure the NCC train accuracy, which is a clustering measure (as described in Section 2.2), both at the sample level and with respect to the original classes. Notably, the SSL-trained representation exhibits neural collapse at a sample level (with an NCC train accuracy near 1.0), yet the clustering with respect to the semantic classes is also significant (<s 0.41 on the original targets).
As shown in Figure 2 (left), most of the clustering process with respect to the augmentations (on which the network was directly trained) occurs at the beginning of the training process and then plateaus, while the clustering with respect to semantic classes (which is not directly featured in the training objective) continues to improve throughout the training process.
A key property observed in [50] is that the top-layer embeddings of supervised training samples gradually converge towards a centroid-liks structure. In our quest to better understand the clustering properties of SSL-trained representation functions, we investigated whether something similar occurs with SSL. The NCC classifier's 's performance, being a linear classifier, is bounded by the performance of the best-performing linear classifier. By evaluating the ratio between the accuracy of the NCC classifier and linear classifier trained on the same data, we can investigate data clustering at various levels of granularity. In Figure (middle), we monitor this ratio for the sample-level classes and for the original targets, normalized with respect to its value at initialization. As SSL training progresses, the gap between NCC accuracy and linear accuracy narrows, indicating that augmented samples progressively cluster more based on their sample identity and semantic attributes. Furthermore, the plot shows that the sample-level ratio is initially higher, suggesting that augmented samples cluster based their identity until they converge centroids (the ratio between the NCC accuracy and the linear accuracy is 2 0.9 at epoch 100). However, as training continues, the sample-level ratio saturates while the class-level ratio continues to grow and converges to Re 0.75. This implics that the augmented samples are initially clustered with respect to their sample identities, and once achieved, they tend to cluster with respect high-level semantic classes.
Implicit information compression in SSL training. Having examined the clustering process within SSL training, we next turn our attention to analyzing information compression during the learning process. As outlined in Section 2.2, effective compression often yields representations that both beneficial and practical. Nevertheless, it is still largely uncharted territory whether such compression indeed occurs during the course of SSL. training [59].
To shed light on this, we utilize the Mutual Information Neural Estimation (MINE) [8], a method designed to estimate the mutual information between the input and its corresponding embedded repre- sentation during the course of training. This metric serves as an effective gauge of the representation's 's complexity level, essentially demonstrating how much information (number of bits) it encodes.
In Figure 3 (middle) we report the average mutual information computed across different MINE initialization seeds. As demonstrated, the training process manifests significant compression, cul- minating in highly compact trained representations. This observation aligns scamlessly with our previous findings related to clustering, where we noted that more densely clustered representations correspond to more compressed information.
The role of regularization loss. To gain deeper understanding of the factors influencing this clustering process, we study the distinct components of the SSL objective function. As we discussed in section 1, the SSL objective function is composed of two terms: invariance and regularization. The invariance term's ''s main function is to enforce similarity among the representations of augmentations of the same sample. In contrast, the regularization term helps to prevent representation collapse.
7

---

To investigate the impact of these components on the clustering process, we dissect the objective function into the invariance and regularization terms and observe their behavior during the course of training. As part of this exploration, we trained RES-5-250 network using VICReg with μ = X = 25 and V 1, as proposed in [6]. Figure (left) presents this comparison, charting the progression of loss terms and the linear test accuracy with respect to the original semantic targets. Contrary to popular belief, the invariance loss does not significantly improve during the training process. Instead, the improvement in loss (and downstream semantic accuracy) is achieved due to minimizing the regularization loss. This observation is consistent with our earlier finding that the per-sampling clustering process (which is driven by the invariance loss) saturates early in the training process.
We conclude that the majority of the SSL training process is geared towards improving the semantic accuracy and clustering of the learned representations, rather than the per-sample classification accuracy and clustering. This is consistent with the observed trend of the regularization loss improving over the course of training and the early saturation of the invariance loss.
In essence, our findings suggest that while self-supervised learning directly targets sample-level clustering, the majority of the training time is spent on orchestrating data clustering according to semantic classes across various hierarchies. This observation underscores the remarkable versatility of SSL methodologies in generating semantically meaningful representations through clustering and sheds light on its underlying mechanisms.
Comparing supervised learning and SSL clustering. As mentioned in Section 4, deep network classifiers tend to cluster training samples into centroids based on their classes. However, the learned function is truly clustering the classes only if this property holds for the test samples, which is expected to occur but to a lesser degree.
An interesting question is to what extent SSL implicitly clusters the samples according to their semantic classes, compared to the clustering induced by supervised learning. In Figure 3 (right), we report the NCC train and test accuracy rates at the end of training for different scenarios: supervised learning with and without augmentations and SSL. For all cases, used the RES-10- 10-250 architecture.
While the NCC train accuracy of the supervised classifier is 1.0, which is significantly higher than the NCC train accuracy of the SSL-trained model, the NCC test accuracy of the SSL model is slightly higher than the NCC test accuracy of the supervised model. This implies that both models exhibit a similar degree clustering behavior with respect to the semantic classes. Interestingly, when training the supervised model with augmentation slightly decreases the NCC train accuracy, yet significantly improves the NCC test accuracy.
Exploring Semantic Class Learning and the Impact of Randomness
Semantic classes define relationships between inputs and targets based on inherent patterns in the input. On the other hand, mappings from inputs to random targets lack discernible patterns, resulting in seemingly arbitrary connections between inputs and targets.
In this section, we study the influence of randomness on a model's proficiency in learning desired targets. To do so, we build series of target systems characterized by varying degrees of randomness and examine their effect on the learned representations. We train a neural network classifier on the same dataset for classification and use its target predictions at different epochs as targets with varying degrees of randomness. At epoch O, the network is random, generating deterministic yet seemingly arbitrary labels. As training advances, the functions become less random, culminating in targets that align with ground truth targets (considered non-random). We normalize the degree of randomness between 0 (non-random, end of training) and 1 (random, initialization). Utilizing this methodology, we investigate the classes SSL learns during training.
Initially, our emphasis is on exploring the impact of the SSL training process on the learned targets. We train RES-5-250 network with VICReg and employ a ResNet-18 to generate targets with different levels of randomness. Figure 4 (left) showcases the linear test accuracy for varying degrees of random targets. Each line corresponds to the accuracy at different stage of SSL training for different levels of randomness. As we can see, the model captures classes closer to "semantic" ones (lower degrees of randomness) more effectively throughout training while not showing significant performance improvement on highly random targets. For results with additional random target generators, see Appendix B.5.
8

---

A key question in deep learning revolves around understanding the functionalities and the impact of intermediate layers on classifying different types of classes. For example, do different layers learn different kinds of classes? We explore this by evaluating the linear test accuracy of different layer representations for various degrees of target randomness at the end of the training. As shown in Figure 4 (middle), the linear test accuracy consistently improves at all layers as randomness decreases, with deeper layers outperforming across all class types and the performance gap broadening for classes closer to semantic ones.
Beyond classification accuracy, a desirable representation exhibits a high degree of clustering with respect to the targets [50, 24, 32, 28]. We assess the quality of clustering using several metrics: NCC accuracy, CDNV, average per-class variance, and average squared distance between class means [28]. To gauge the improvement in representation over training, we calculate the ratios of these metrics for semantic vs. random targets. Figure 4(right) displays these ratios, indicating that the representation increasingly clusters the data with respect to semantic targets compared to random ones, as the ratio of the NCC, CDNV, increases during training. Interestingly, we see that the decrease in the CDNV, which is the variance divided by the squared distance, is caused solely by the increase in squared distance. The variance ratio stays fairly constant during training. This phenomenon of encouraging large margins between clusters has been shown to improve the performance [24, 17].
6 Learning Class Hierarchies and Intermediate Layers
Previous studies have given evidence that in the context of supervised learning, intermediate layers gradually capture features at varying levels of abstraction [1, 17, 50, 26, 10]. The initial layers tend to focus on low-level features, while deeper layers capture more abstract features. In Section we showed that SSL implicitly learns representations highly correlated with semantic attributes. Next, we investigate whether the network learns higher-level hierarchical attributes and which layers are better correlated with these attributes.
In order measure what kinds of targets are associated with the learned representations, we trained RES-5-250 network using VICReg. We compute the linear test accuracy (normalized by its value at initialization) with respect to three hierarchies: the sample-level, the original 100 classes, and the 20 superclasses. In Figure 2 (right), we plot these quantities computed for the three different sets of classes. We observed that, in contrast the sample-level classes, the performance with respect to both the original classes and the superclasses significantly increased during training. The complete details are provided in Appendix A.
As a next step, we investigate the behavior of intermediate layers of SSL-trained models and their ability to capture targets of different hierarchies. To this end, we trained a RES-1O-250 with VICReg. In Figure 5 (left and middle), we plot the linear test accuracy across all intermediate layers at various stages of training, measured for both the original targets and the superclasses. In Figure 5 (right) we plot the ratios between the superclasses and the original classes.
We draw several conclusions from these results. First, we observe consistent improvement in clustering as we move from earlier to deeper layers, becoming more prominent during the course of training. Furthermore, similar to supervised learning scenarios [62, 26, 10], we find that the linear accuracy of the network improves in each layer during SSL training. Of note, we find that for the original classes, the final layers are not the optimal ones. Recent studies in SSL [49, 55] showed that the performance of the different algorithms are highly sensitive to the downstream task domains. Our study extends this observation and suggests that also different parts of the network may be more suitable for certain downstream tasks and class hierarchies. Figure 5(right), it is evident that relatively, the accuracy of the superclasses improves more than that of the original classes in the deeper layers of the network.
7 Conclusions
Representation functions that cluster data based on semantic classes are generally favored due to their ability to classify classes accurately with limited samples [32, 28, 27, 29]. this paper, we conduct a comprehensive empirical exploration of SSL-trained representations and their clustering properties concerning semantic classes. Our findings reveal an intriguing impact of the regularization constraint in SSL. algorithms. While regularization is primarily used to prevent representation collapse, it also enhances the alignment between learned representations and semantic classes, even after accurately clustering augmented training samples. We investigate the emergence of different class types during

---

training, including targets with various hierarchies and levels of randomness, and examine their aligmment different intermediate layers. Collectively, these results provide substantial evidence that SSL algorithms learn representations that effectively cluster based on semantic classes.
Despite the similarities observed between the supervised setting and the SSL setting in our paper, several questions remain unanswered. Foremost among them is the inquiry into why SSL algorithms learn semantic classes. Although we present compelling evidence linking this phenomenon the regularization constraint, the explanation of how this term manages to cluster data with respect to semantic classes, and the specific types of classes being learned, remains unclear. deeper understanding of the types of classes to which data clusters, coupled with the connection between neural collapse and transfer learning [28. 27, 29], may be helpful understanding the transferability of SSL-trained representations to downstream tasks.
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017.
[2] Martin Anthony and Peter Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, USA, 1st edition, 2009.
[3] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
[4] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, l'Alché-Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[5] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Gold- stein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning, 2023.
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: ization for self-supervised learning. International Conference on Learning Representations, 2022.
regular-
[7] Peter I.. Bartlett, Dylan Foster, and Matus Telgarsky. pnnetrally-normalizee margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30. Curran Associates Inc., 2017.
[8] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv: 1801.00000002 2018
[9] Ido Ben-Shaul and Shai Dekel. Sparsity-probe: Analysis tool for deep learning models. ArXiv, abs/2105.068449 2021.
[10] Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, volume 196 of Proceedings of Machine Learning Research, pages 37-47. PMLR, 2022.
[11] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 - mining discriminative components with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 446-461. Springer International Publishing, 2014.
[12] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In J. Cowan, G. Tesauro, and Alspector, editors, Advances in Neural Information Processing Systems, volume Morgan- Kaufmann, 1993.
10

---

[13] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICMI.), 2020.
[15] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong cemi-supervised learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22243-22555. Curran Associates, Inc., 2020.
[16] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758, June 2021.
[17] Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Dnn or k-nn; That is the generalize vs. memorize question, 2018.
(18) Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoe V. Le. Autoaug- ment: Learning augmentation strategies from data. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 113-123, 2019.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021, abs/2010.1 1990090..1999,.., 2020.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[22] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self- supervised learning by characterizing idealized representations. In Advances in Neural Informa- tion Processing Systems, volume 35, pages 11279-1 11296, 2022.
[23] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Doddss Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
[24] Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[25] William
Falcon et al.
Pytorch lightning. lightning. 3, 2019.
GitHub.
Note:
[26] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the Implicit Bias Towards Minimal Depth of Deep Neural Networks. arXiv, 2022.
[27] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. ICMI. Workshop on Pre-training: Perspectives, Pitfalls, and Puths Forward, 2022.
11

---

[28] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations. 2022.
[29] Tomer Galanti, András György, and Marcus Hutter. Generalization bounds for transfer learning with pretrained classifiers, 2022.
[30] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6830 6840, June 2021.
generation for self-supervised learning. In Proceedings
[31] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations. 2018.
[32] Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3607-3616. PMLR, 13-18 Jul 2020.
[33] Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA, 9()):733-54, 05 2020.
[34] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2020. Curran Associates Inc.
[35] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Putlern Recognition (CVPR'06) volume 2, pages 1735-1742, 2006.
[36] X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations, 2022.
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[39] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse contrastive self-supervised learning. International Conference on Learning Representa- lions, 2022.
[40] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[41] Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical Report, 2009.
[42] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016, pages 577-593, Cham, 2016. Springer International Publishing.
[43] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[44] Nya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference Learning Representations, 2017.
12

---

[45] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-granne line-grained visual classification of aircraft, Technical report, 2013.
[46] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[47] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6706-6716 16, 2019.
[48] Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise- contrastive estimation. In C.J. Burges, L. Bottou, M. Welling, Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[49]
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[50] Vardan Papyan, X. Y. Han, and David L.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24522-2465) 2020.
[51] Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res., 18:83:1-83:52, 2017.
[52] Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? In The Eleventh International Conference on Learning Representations, 2023.
[53] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[54] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-288 2830, 2011.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, volume 139, pages 8748-8763, 2021.
[56] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2023.
[57] Tim Sainburg. Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation, 33(1 1):2881-2907, 2021.
[58] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceed- ings of the 36th International Conference on Machine Learning, volume 97, pages 5628-5637, 2019.
13

---

[59] Ravid Shwartz-Ziv and Yann LeCun. To Compress or Not to Compress- Self-Superviser Learning and Information Theory: Review. arXiv, 2023.
[60] Ravid Shwartz-ZZ) Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. ArXiv, abs// 7077000001000. 10, 2017.
[61] Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner al. Lightly. 2020.
GitHub, Note: htpps//giihub..comtighhtttyy-tt
[62] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In Proceedings of the 39th International Conference on Machine Learning, volume 162. pages 21478-21505, PMLR, 2022.
[63] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv, 2018.
[64] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748. 2018.
[65] Julius Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16451-164667 Curran Associates, Inc., 2021.
[66] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing. 2021.
14

---

A Training Details
A.1 Data augmentation
We follow the image augmentation protocol first introduced in SimCLR [14] and now commonly used by similar approaches based on siamese networks [13, 34, 14, 66). Two random crops from the input image cropped and resized 32 32, followed by random horizontal flip, color jittering of brightness, contrast, saturation and hue, Gaussian blur, and random grayscale. Each crop is normalized in each color channel using the ImageNet mean and standard deviation pixel values. The following operations are performed sequentially to produce each view:
Random cropping with area uniformly sampled with size ratio between 0.08 to 1.0, followed by resizing to size 32 x 32. RandomResi oeteeeeiiieeoooppe (32, scale= (8,08, 1. 1.0)) in PyTorch.
Random horizontal flip with probability 0.5.
Color jittering of brightness, contrast, saturation and huc, with probability 0.8. ColorJitter 40.4, 0 0.4, 0.2, 0.1) in PyTorch.
Grayscale with probability 0.2.
Gaussian blur with probability 0.5 and kernel size 23.
Solarization with probability 0.I.
Color normalization with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225)
(ImageNet standardization).
Supervised learning Figure 3 (right). For the supervised learning setting, we used AutoAug- ment [18] with policy created for CIFAR-10 [41].
A.2 Network Architectures
In this section, we describe the network architectures used in our experiments.
SSL backbone. The main architecture used the SSL backbone is convolutional residual network, denoted RES-L L-H. consists of a stack of two 2 x convolutional layers with stride 2, batch normalization, and ReLU activation, followed by L residual blocks. The ith block computes g' (z) (x= (C? (a B) (C) (z)))))), where C²¹ is 3 3 convolutional layer with H channels, stride 1, and padding B is a batch normalization layer, for j E (1, and is the ReLU activation.
Random target functions. Throughout the paper. we use two different random target functions, ResNet-18 [38] and a visual transformer (ViT) [21] (Appendix B.5). For the transformer architecture, used 10 layers, with 8 attention heads and 384 hidden dimensions. The classification was done using a classification token.
A.3
Optimization
SSL. We trained all of our SSL. models for 1000 epochs using batch size of 256. We used the SGD optimizer with learning rate of 0.002, momentum value of 0.9, and weight decay value of 1c-6. Additionally, we used a Aoncaannn learning rate scheduler [44].
Random targets. The ResNet-18 18 targets were trained using the Cross Eross-Entropy loss, with the Adam optimizer [40], using learning rate of 0.05 and the Cosine Annealing learning rate scheduler. For the ViT training, we used the Adam optimizer with 0.001 learning rate, weight decay of 5e-5, and Cosine Annealing learning rate scheduler. We used the model's class predictions as the targets.
A.4 Implementation Details
Our experiments were implemented in Py PoTch [53], utilizing the Lightly [61] library for SSL models and PoTch Lightning [25] for training. All of our models were trained on V100 GPUs
B Additional Experiments
Clustering SSL. Figure 2 in the main text, we reported behaviors of extracting classes of different hierarchies from SSL-trained models. In this experiment, we provide the un-normalized version of the results in Figure Specifically, Figure (left), we show the un-normalized results, along with the normalized plots shown in the main text Figure 6 (right).
15

---

1.8
0.9
Narmalized Accuracy
1.6
Linear Accuragy
0.7
1.4
0.5
Hlerarchy
Hierarcy
Sample
Sample
Oraginal
Original
Super
10
100
1000
10
100
1000
Epoch
Epoch
2.6
0.8
Normaltaed NCC Accuray
NDC Acgunicy
1.8
0.4
Hlerarchy Samsple Original
Hierarchy
Sample
Original
10
100
1000
100
1000
Epoch
Epoch
1.4
Ratto
Normalized
0.7
Hierarchy Sampole Original
Hierarchy
0.5
Sample Original
10
100
1000
100
1000
Epoch
Egoch
Figure 6: SSL algorithms cluster the data with respect to semantic targets. (right) The results in
Figure 2. (left) The unnormalized version of the same results.
B.1 Network Architectures
SSL research primarily focuses on utilizing single ResNer-50 backbone for experimental purposes. However, given the similarities between SSL. and supervised learning, as demonstrated in the paper, it is worth exploring how the choice of backbone architecture affects the clustering of representations with respect to semantic targets. In Figure 7, we present the intermediate layer clustering for different network architectures for extracting the original classes (left) and the superclasses (right). In Figure (top), we display the linear test accuracy of RES-5-50, RES-5-250, and RES-5-1000 networks different epochs (20, 40, 100, 400, 1000). As can be seen, the network's width has significant impact on the results for all intermediate layers and at all training epochs.
In Figure 7 (bottom), we present the linear test accuracy of networks RES-5-250,
throughout the training epochs (left) and intermediate layers (right). It is generally seen that in the later epochs of training, the additional layers benefit the linear test performance of the network. However, it is also interesting to note that for the initial layers of the network, the shallow RES-5-250 model gets more clustered representations both for the original classes and for the superclasses. This hints at the fact that deeper networks may need intermediate layer losses to encourage clustering in initial layers [24, 10].
Experiments with Vision Transformers. We extend our experimentation to different backbones to validate the results illustrated in the main paper. A Vision Transformer (ViT) is adopted as per the architecture detailed in [20], employing a 5-layer transformer with 4 heads and a hidden dimension of 384. In Figure 8, we examine the clustering degree produced by various models trained using different SSI algorithms on the Tiny Imagenet dataset. Specifically, we report the NCC train accuracy across intermediate layers for both the VICReg and SimCLR algorithms.
16

---

Width
Width
+500
230
Linear Accyrney
1000
Linear Accuracy
1000
0.6
R
4
Layer
Layer
Depth
Depth Shallow
Shallow
Deep
Deep
Lineer Accurady
Linear Accustty
4
10
2
3
4
Layar
Loyer
Original classes
Superclasses
Figure 7: The influence of width and depth learning semantic classes at intermediate layers. (top) Linear test accuracy at different epochs for neural networks of varying widths. (bottom) Linear test accuracy of neural networks with different depths. (left) The performance is measured in relation to the original classes. (right) The performance with respect to the superclasses.
VICReg
VICReg Normalized
SimCLR
SimCLR Normalized
Figure 8: VICReg and SimCLR with VIT [20] backbone clusters the data with respect to semantic targets in intermediate layers. The NCC train accuracy (normalized and un-normalized) of an SSL-trained network on Tiny Imagenet, measured intermediate layers.
B.2 Experiments with Additional Datasets
CIFAR-10. We run similar experiments using RES-5-250 on the CIFAR-10 dataset. In Fig- 10 (top), report the NCC test accuracy for recovering both the sample level and the class labels. Similar to the results in the main text, the samples progressively cluster around their augmentation class means early in training. However, during the majoring of the training process, the augmented samples tend to cluster with respect to the semantic labels. In Figure 10 (bottom), we report the NCC and linear (from left to right) test accuracies of intermediate layers with respect to the semantic labels.
SWaV, Tiny Imagenet
SWaV. Aircrafts
Barlow Twins, Aircrafts
Figure 9: Different algorithms cluster the data with respect to semantic targets in intermediate layers. The (normalized) NCC train accuracy of an SSL-trained RES-5-250 network on different datasets, measured at intermediate layers.
17

---

Normalizen leed NOC Aucurmey
NCC Accuracy
1.2
0.6
1.1
lllerarcyy Sacaple Original
0.5
Sample
Original
100
Epuch
Epoch
0.65
Epoch 20
NCC Accuracy
Lineer Amouraty
0.7
100
1030
0.55
0.5
6.6
0.45
20
100
800
Epoch
Figure 10: SSL algorithms cluster the data with respect to semantic targets. (top) The NCC train accuracy of an SSL-trained network on CIFAR-10, measured at the sample level and original classes (both un-normalized and normalized). (bottom) The NCC train accuracy of the model at different layers and epochs.
As can be seen, the degree of clustering monotonically improves at all layers during the course of training.
FOOD-101. Similar to CIFAR-10, we experiment the FOOD101 dataset, consisting of 75,750 train samples and 25,250 test samples across 101 food classes (e.g. 'pizza'. 'misso soup', 'waffles') ) using VICReg and the RES-5-250 architecture. In Figure 13 (left), plot the NCC test accuracy for both the sample level, and the semantic classes level, and in Figure 13 (right) we plot the NCC test accuracy of the intermediate layers with respect the sample level labels. The results similar to the ones shown in Section 4. Additionally, we present the sample level clustering in the intermediate layers. Here, also
improvement throughout the layers and the epochs.
Aircrafts. We applied the SWaV and Barlow Twins algorithms with the RES-5-250 architecture to the Aircrafts dataset, which consists of 10,000 images from 100 aircraft types. In Figure 9 we show the NCC train accuracy the various intermediate layers.
Tiny Imagenet. We also experiment on more varied datasets, like the Tiny ImageNet dataset, consisting of 100,000 train samples and 10,000 test samples across 200 classes (e.g. 'nematode' 'harmonica', 'syringe'), ) using VICReg and the RES-10-255 architecture. In Figure 12 we plot the NCC test accuracy for both the sample level, and the semantic classes level, both unormalized and
normalized resp. In Figure 9 we show the NCC train accuracy using the SWav SSL algorithm. The results are similar the ones shown in the clustering section.
Imagenet. We extend our experimentation to the more complex ImageNet [19] dataset. In Figure 11, show the linear test accuracy and NCC train accuracy of both SwAV and SimCLR on the ImageNet dataset, along training epochs, with respect to the semantic labels on the final layer of a ResNet-50., Similarly to the supervised setting, as shown in section Section 4, both the linear accuracy and the NCC accuracy dramatically improves during training.
B.3 Hyperparameters
In Figure 14, we present the training losses and linear accuracies for three different hyperparameter selections trained using VICReg and RES-5-250 network. Specifically, we vary the l (variance regularization) parameter and display the losses and linear accuracy for 5, 25, 100, from left to
18

---

Linear and NCC Accuracy
0.8
SwAV NCC
SimCLR NCC
SwAV Linear
0.1
SimCLR Linear
0.01
100
200
Epochs
Figure 11: SwAV and SimCLR cluster ImageNet data with respect to semantic targets. The last layer Linear Test accuracy and NCC train accuracy of an SSL-trained ResNet-50 on Imagenet, on the 1000 original classes.
1.8
NOC Accarecy
0.4
lnndl NOL
1.4
0.2
Hlerarchy Sample Original
Hlerarchy Sample
Original
128 Epoch
256
512
1024
16
32
61
128
256
512
1021
Epoch
Figure 12: VICReg clusters the data with respect semantic targets The NCC train accuracy of an SSL-trained RES-10-250 on Tiny Imagenet, measured at the sample level and original classes (both an-normalized and normalized).
right, respectively. Both loss terms are normalized by μ.. In all cases, we used v = 1 and À 25 by default.
Interestingly, the configuration with /a = 5 exhibits significantly lower performance compared to the default setting (/4 25). However, the configuration with f 100 achieves performance comparable to the default, despite the network having considerably higher invariance loss term. This observation further supports our claim that the regularization term plays a crucial role learning semantic features.
B.4 Experiments with Different SSL Algorithms
In our main text, we primarily focus on the VICReg SSL algorithm. However, to ensure the robustness of our findings across different algorithms, also trained network using the SimCLR algorithm [14] with our default RES-5-250 architecture. In Figure 15, we present several experiments that compare the network trained with VICReg the one trained with SimCLR.
Firstly, Figure 15 (top) illustrates the linear test accuracies of intermediate layers for both algorithms throughout the training epochs with respect the original classes (left) and the superclasses (right). The darkness of the lines indicates the progression of epochs, with lighter shades representing later epochs. Despite the significant differences between the algorithms, particularly SimCLR's contrastive nature, we observe similar clustering behavior across different layers and epochs with VICReg consistently achieving better performance. This finding demonstrates that various SSL. algorithms yield comparable clustering characteristics.
19

---

0.9
Layer
Normalized NOC Accuraoy
Sumple NCC Arcuracy
0.8
1.1
Itlerarchy Sacaple
0.6
Original
Epuch
Epoch
Figure 13: SSL. algorithms cluster the data with respect to semantic targets and invariance to augmentations at intermediate layers. (left) The normalized NCC test accuracy of VICReg- trained network on FOOD101 with respect to the sample level labels and the original class labels. (right) The NCC test accuracy of the model with respect to the sample level labels of intermediate layers.
0.0
0.0
0.0
Accuracy
0,5
3itcous Accurscy
30.5
$0.4
0.25
0.5
0.2
Tn
TeM
100
0.45
100
1003
20
1000
1000
Epoch
Tppcch
Tipoch
11
11 25
14
100
Figure 14: The role of the regularization term in SSL training. Each plot depicts the regularization and invariance losses, along with the linear test accuracy, throughout the training process of VICReg with μ 5, 25, 100 respectively.
Furthermore, in addition to clustering, we aim to assess the similarity of representations between the two algorithms. In Figure (bottom left), we present the correspondence the representations with random targets, similar to the approach used in Figure 4 (left), across different training epochs. The darkness of the lines represents the progression of epochs, with lighter shades indicating later epochs. Additionally, in Figure 15 (bottom right), depict the same correspondence for different intermediate layers, similar to Figure 4 (middle), at the end the SSL train. Across various layers and epochs, observe similar behavior between the algorithms. However, it is worth noting that VICReg demonstrates a slightly higher performance extracting the random targets.
B.5 Random Target Function Architectures
Neural network architectures possess inherent biases that aid modeling complex functions. Con- volutional networks, for example, implicitly encode biases through their structure, which includes locality and translation invariance. As result, these networks tend to prioritize local patterns over global features. On the other hand, the ViT architecture takes different approach by treating images as sequences of patches and utilizing self-attention mechanisms. This design introduces a bias that enables long-range dependencies and global context.
An interesting question arises regarding how the choice the SSL backbone architecture influences the learned representations. To explore this, we examine the alignment of representations with different types of random targets. Specifically, we conduct experiments similar to those illustrated in Figure (left and middle) but employ different random targets based the VIT architecture [52].
In Figure 16, we monitor the linear test accuracy of VICReg-trained RES-5-250 in recovering both the ResNet-13 18 targets and the VIT targets. Figure 16 (left) presents the linear test accuracy at different training epochs (20, 40, 100, 400, 1000) from dark to light resp. and Figure (right) shows the linear test accuracy at different intermediate layers (1-5) from dark to light resp. at the end of SSL
20

---

SSL Algorithm VICReg
SST Algorithm ViCRea
1inear Acourney
Linnar Accuracy
2
Layer
SSL Algorithu ViCRag
SSL Algorithm ViCRRg
Lineer Arcurace 20.4
SimCLR
SimCLR
Liceer Accustsy
0.35
0.75
0.95
0.99
0.75
0.95
0.99
Randomness Level
Acndomness Level
Figure 15: SimCLR and VICReg have similar performance. (top) Linear test accuracy different training epochs, function of the intermediate layer, for original classes and superclasses, from left to right resp. (bottom) (left) Linear test accuracy in different training epochs (from dark to light) with respect different randomness levels. (right) Linear test accuracy in different intermediate layers, at the end training with respect to different randomness levels.
0.55
Generator
Target Generator
RooNeet18
de RessNet-18
VIT
Liner Accurery
20.45
Libear Accuracy
0.45
Lisear
0.35
0.35
0.25
0.25
Randomness
Figure 16: The implicit bias of the backhone architecture the learned representations. (left) Linear test accuracy of an SSL-trained RES-5-250 network for extracting ResNet- 18 and ViT random target functions with varying degrees of randomness (x-axis) different epochs (color-coded from dark to bright). (right) Linear test accuracy of an SSL-trained RES-5-250 network for extracting ResNet-18 and ViT random target functions with varying degrees of randomness (x-axis) different intermediate layers (color-coded from dark to bright).
training. As can be seen, the results with ViT targets are consistent with the claims in Section 5. In other words, the SSL representations exhibit improved alignment during training epochs and across intermediate layers.
However, it is evident that the linear accuracy with respect to the ResNet-18 targets consistently outperforms the accuracy with respect to ViT targets at all training stages, except for the end of supervised training phase ("0" randomness). This indicates that during training process, the SSL-trained model acquires representations that are more aligned with those achieved by training convolutional network architectures. This behavior can attributed to the implicit biases introduced by the backbone, which share similarities with the ones present in the ResNet- 18 architecture.
21

---

C Limitations
While our study has yielded significant findings, it not without its limitations. Firstly, our experi- ments were conducted on select datasets, which inherently possess their unique characteristics and biases. As such, applying our analysis to different datasets may potentially give different results. This reflects the inherent variability and diversity of real-world data and the possible influence of dataset-specific factors on our findings. Secondly, our analysis primarily focuses on the vision domain. While we believe that findings have substantial implications in this area, the generalizability of our findings to other domains, such as natural language processing or audio processing, remains unverified.
D Broader Impact
In this paper, our primary objective is characterize the different properties and aspects of SSL, with the aim of deepening our understanding of the learned representations. SSL has demonstrated its versatility wide range practical downstream applications, including image classification, image segmentation, and various language tasks. While our main focus lies in unraveling the underlying mechanisms of SSL,, the insights gained from this research hold the potential to enhance SSL algorithms. Consequently, these improved algorithms could have significant impact on diverse set of applications. However, crucial to acknowledge the potential risks associated with the misuse of these technologies.
22


Reviews:
[{"review_id": "k4OAmfgv2n", "reviewer": "Reviewer_7nAJ", "summary": "To reveal the underlying mechanism of self-supervised learning methods, this paper mainly investigate their clustering characteristics. Empirical analyses show that these methods cluster samples based on their semantic classes. Additionally, the use of regularization terms, introduced to prevent representation collapse, proves beneficial even after the clustering process.", "questions": "One of my major concerns is about the significance of the findings, since the paper primarily focuses on describing the properties of contrastive and non-contrastive learning methods. ", "limitations": "I cannot find any ethical concerns. For technical limitations, see the Weaknesses section above.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Understanding the mechanism of self-supervised learning is one of the most important goals acknowledged by the machine learning community. \n2. This paper provides not only qualitative but also quantitative results. \n3. This paper is well-written and well-organized. Notably, the introduction section effectively summarizes the key takeaways", "weaknesses": "1. While the findings are interesting, I'm not fully convinced of their importance. A comparison with other pre-training methods or the proposal of a novel self-supervised learning method based on the conclusions drawn would emphasize the significance of the paper's findings.\n2. The scope of the setup is limited:\n    - The experiments solely utilize RES-L-H architectures.\n    - The primary downstream task focuses on image classification using CIFAR-100. This not only neglects dense prediction tasks but also relies on a small dataset compared with ImageNet.\n    - Many experiments mainly focuses on linear probing accuracy rather than fine-tuning performance. However, a high linear probing accuracy does not necessarily indicate a high fine-tuning accuracy [a]. In practice, fine-tuning accuracy is sometimes more crucial than linear probing accuracy.\n    - The title and the term “self-supervised learning” in this paper are potentially misleading, since this paper mainly focuses on contrastive and non-contrastive learning, rather than on masked image modeling. Moreover, it seems that some claims are only valid for contrastive or non-contrastive learning.\n\n[a] Park, Namuk, et al. \"What Do Self-Supervised Vision Transformers Learn?.\" ICLR (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We've taken your insights into account and have expanded our analyses and clarifications as detailed below.\n\n# Weaknesses: \n## W1: \nWhile we agree that deriving a novel SSL method based on these findings could be a compelling avenue for future exploration, we believe our current work lays the foundational understanding of the principal mechanisms through which SSL methods learn semantic labels. This parallels the work shown in [1], where the phenomenon of Neural Collapse was first introduced and subsequently utilized to improve generalization and performance in [2,3]. Shedding light on the connection between the learning patterns of supervised algorithms with those of SSL, as presented in this paper, is a significant leap toward improving SSL. For example, in L347, \"Our study extends..\". This outcome could guide decisions regarding selecting intermediate layers when leveraging representations from an SSL algorithm. While this is outside the scope of our current work, it certainly presents a promising avenue for future research!\n\n## W2:\n\nIn this work, our primary objective was to study the relationship between supervised classification and SSL. Our findings underscore that the Neural Collapse framework [1] and linear probing—across diverse architectures, datasets, SSL algorithms, and levels of randomness — are intrinsically linked to the supervised context. We have demonstrated that SSL algorithms can deduce semantic labels during training in a manner reminiscent of the supervised setting. Although we did not incorporate ImageNet in our analyses due to a lack of computational resources, we have attempted to address the reviewer's concerns by extending our studies to several other datasets, including Tiny-ImageNet [8], Aircrafts [9], Oxford Pets [10], and Oxford Flowers [11]. Future research may study how clustering in SSL algorithms impacts different downstream applications, such as Object Detection and Segmentation.\n\nWe agree that linear probing might not fully capture the performance realized during fine-tuning. However, the primary focus of this paper was to analyze the representations acquired during SSL within the same dataset. The influence of clustering degree on performance during fine-tuning indeed presents an intriguing avenue for exploration. In most cases, linear probing serves as a measure to quantify the representation by decoupling it from downstream fine-tuning [4-7]. Our specific interest lies in understanding how SSL algorithms cluster data in relation to previously unseen labels and how this process diverges from the findings in the supervised case [1]. Given this objective, it's more pertinent for our study to examine the representations prior to any fine-tuning.\n\nWe acknowledge that the primary focus of our study revolves around contrastive and non-contrastive methods, and we will ensure this is clearly communicated in the paper. It's important to highlight that most SSL algorithms [4-7], could be categorized under our framework. Investigating how MAE-based methods fit into our framework is an interesting avenue for further exploration.\n\nTo further extend our analysis, we've added:\n\n- **Datasets** - Additional to CIFAR100, CIFAR10, and FOOD101, we included results on Tiny-ImageNet [8], Aircrafts [9], Oxford Pets [10], and Oxford Flowers [11]. These datasets are much more diverse and contain different features, different numbers of training examples, and different numbers of classes.\n- **SSL Methods** - To further enrich our study, we've added to VICReg and SimCLR, the clustering-based SSL method, SWaV, and Barlow Twins.\n- **Integration with Recent Architectures as Backbones** - Additional to RES-L-W of different widths and depths, we included Vision Transformer (VIT) [12] as an additional backbone.\n\nThank you again for your thoughtful review. We made an effort to address your feedback including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n## References\n[1] V. Papyan, et al., \"Prevalence of neural collapse during the terminal phase of deep learning training\" 2020.\n\n[2] Z. Zhu, et al., \"A Geometric Analysis of Neural Collapse with Unconstrained Features\", 2021. \n\n[3] I. Ben-Shaul and S. Dekel, \"Nearest Class-Center Simplification through Intermediate Layers\", 2022\n\n[4] A. Bardes, et al., \"VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning\", 2022. \n\n[5] T. Chen, et al., \"A Simple Framework for Contrastive Learning of Visual Representations\", 2020.\n\n[6] J.-B. Grill, et al., \"Bootstrap Your Own Latent a New Approach to Self-Supervised Learning\", 2020.\n\n[7] M. Caron, et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", 2020.\n\n[8] Y. Le et al., \"Tiny ImageNet Visual Recognition Challenge,\" 2015.\n\n[9] S. Maji et al., \"Fine-Grained Visual Classification of Aircraft\", 2013. \n\n[10] Zhang, H. et al. \"0/1 Deep Neural Networks via Block Coordinate Descent\", 2022.\n\n[11] M.-E. Nilsback et al., \"Automated Flower Classification over a Large Number of Classes\", 2008.\n\n[12] A. Dosovitskiy et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\", 2021. \n\n[13] M. Assran et al., \"Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture\", 2023"}, {"review_id": "oaXbGoknmb", "reviewer": "Reviewer_YeWL", "summary": "This paper focuses on an empirical analysis of how self-supervised learning (SSL) works. While training artificial neural networks for classification under supervision has been extensively explored, the authors propose to shed more light on SSL process using VicReg + ResNet-L-H and a series of experiments on the CIFAR-100 dataset. This work presents that during the training process, we can observe improvements in clustering for the original classes and in super-classes, that presents some emerging properties of hierarchical clustering during SSL training. What is important, this happens while the expected instance-level clustering stops improving. Although the authors do not provide an answer as to why this is the case, they attribute this phenomenon to the regularization term in the SSL loss function (what is presented in one of the experiments). In work we can find few more analysis.", "questions": "1. One of the insightful outcome of this work is that the clustering property can be attributed to regularization term. But this was not analyzed. Why not increase the term for that component and check the behavior?  \n\n1. Looking at the figure 3 (right). The SSL method seems to work significantly worse than the SL method (41 vs 61% accuracy). This looks like a massive difference, maybe there is something wrong with the training run?\n\n1. Can we see figure 2 without normalized accuracies? I think the default option would be to show absolute values, so I wonder what we gain by showing normalized accuracies. One culprit is that in your “Sample” experiment you have 500 classes and in “original” 100 classes, but I would still like to see absolute values.\n\n1. The randomness experiment is not clear to me. The paper says: „We train a neural network classifier on the same dataset for classification and use its target predictions at different epochs as targets with varying degrees of randomness”, which I found hard to understand. Are the labels simply random at training time as in [1]?\n \nAlso: “We normalize the degree of randomness between 0 (non-random, end of training) and 1 (random, initialization)” - how do you do this? Does 0.9 degree of randomness correspond to 0.9 fraction of total training epochs? Or does it correspond to 0.9 of final performance of the network?\nFinally, looking at the fig. 4. Is there something surprising in the plot? (Increased randomness = reduced accuracy).\n\n1. I think there is something wrong with the following sentence: L.210: “Notably, the SSL-trained representation exhibits neural collapse at a sample level (with an NCC train accuracy near 1.0), yet the clustering with respect to the semantic classes is also significant (≈ 0.41 on the original targets).“ Maybe it is about Fig. 3 (right) but the NCC train accuracy for the SSL method is above 0.4 (not “near 1.0”)? \nAlso, it is not clear what is the message that this figure shows as i) SL and SSL have totally different accuracies so they are not comparable, ii) the fact that we get worse results on validation is also expected.\n\n1. “As shown in Figure 5, we observe that the clustering and separation ability of each layer improves as we move deeper into the network” - it is not true. For classification of original semantic classes the linear accuracy actually drops at the last layer (or several last layers in case of not fully trained networks).\n\n1. Why decision to show only one dataset and type of architecture in the main paper? I know that in the appendix there is a food dataset and different variation of resnet-l-h. However, why not going for imagenet (or subset)? \n\n1. Have you considered checking methods like SupCon [2] for a comparison? \n\n[1] Zhang, Chiyuan, et al. \"Understanding deep learning (still) requires rethinking generalization.\" Communications of the ACM 64.3 (2021): 107-115.\n[2] Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in neural information processing systems 33 (2020): 18661-18673.", "limitations": "Limitations are not discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. This paper tackles an important problem of understanding the representations built by SSL and asks important questions, e.g. how the clustering evolves during the training on multiple levels (instance, class semantic, high level semantic) and to what this phenomenon can be attributed. \n\nS2. The paper shows how the SSL methods gain clustering properties across training procedure (Fig. 2) and across layers (Fig. 5). While this property was intuitively known, this paper is the first to measure this property.", "weaknesses": "W1. The paper draws conclusions based on experiments that are made using only one SSL method, one dataset (CIFAR100 - which is small scale) and one type of neural network architecture (RES-L-H which is not widely used).\n\nW2. The contributions regarding Randomness impact do not sound solid. The randomness experiment is not clear (see questions sections) and it is hard to know what it tries to show. By this experiment, the authors want to show that “We argue that SSL-trained representation functions are highly correlated with semantic classes.”. Doesn't the S1 already show it? \nSimilarly, does not the fact that the classes are clustered would not imply that superclasses are clustered as well? I think it is a nice thing to show, but currently it is exposed as one of the main paper contributions. Presenting that some (super) emerges before the original - would be more insightful \n\nW3. This work is a series of empirical analysis, with some insightful information. However, the reading flow could be improved for that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We've taken your insights into account and have expanded our analyses and clarifications as detailed below.\n# Weaknesses\n## W1:\nWhile this work is an initial study into several geometrical phenomena in SSL, we've attempted to incorporate many networks and architectures. Most works in SSL [6-9] use a single backbone (e.g., RN50), and vary the loss. We've chosen the RES-L-H as it is a variant of the ResNet architecture with the same dim. in every block so that the intermediate performance can be compared. In the appendix, we've also shown results using CIFAR10 and FOOD101 datasets. Following the reviews, we strengthened the empirical results:\n- **Datasets** - Additional to CIFAR100, CIFAR10, and FOOD101, we included results on Tiny-ImageNet [1], Aircrafts [2], Oxford Pets [3], and Oxford Flowers [4]. These datasets are much more diverse and contain different features, different numbers of training examples, and different numbers of classes.\n- **SSL Methods** - To further enrich our study, we've added to VICReg and SimCLR, the clustering-based SSL method, SWaV, and Barlow Twins.\n- **Integration with Recent Architectures as Backbones** - Additional to RES-L-W of different widths and depths, we included Vision Transformer (VIT) [5] as an additional backbone.\n\nAs you can see in the attached page, these new models, SSL methods, and architectures exhibit similar behavior to the current experiments. These additions strengthen our paper by providing a more comprehensive and robust analysis.\n\n## W2:\nWhile we agree that the clustering phenomena point to agreement with semantic classes, the randomness experiment sheds light on how this clustering occurs. Namely, we claim that not only does the data cluster with respect to semantic labels, this clustering increasingly matches \"less random\" repr. In our work, \"less random\" means more aligned with SL on the same task. We're showing that not only does the clustering happen, but it becomes more aligned with the supervised training process on the same dataset. \n\nWe believe that clustering w.r.t superclasses is not trivial. The fact that classes are clustered does not imply that the superclasses are clustered. Consider two super classes with classes {1,2} and classes {3,4}. We can think of samples lying on a line. The samples in class 1 concentrate around x=1, the samples of class 2 concentrate around x=4, the samples of class 3 concentrate around  x=2, and the samples of class 4 concentrate around x=3. The samples themselves are clustered, but not with respect to their superclasses. \n\n## W3:\nWe thank the reviewer for this feedback and will attempt to improve the flow in the revised version.\n\n# Questions\n1. Thank you for this question. This experiment was shown in the Appendix (see Fig 10). We show that with a low regularization term, the linear accuracy greatly decreases. However, with a very large regularization term, the linear performance remains near optimal.\n2. We believe the gap is not due to suboptimal training but rather to applying SSL. When training the model with SSL, it does not have access to labeled data, and therefore, we cannot expect a linear probe to perform as well as SL without fine-tuning the pre-trained model.\n3. The Appendix provides this analysis (See Fig 6 in).\n4. We will attempt to further describe the process in the revised version. The randomness presented (unlike the reference) discusses the degree of agreement with a supervised classifier on the same dataset. This means that 1.0 randomness is the predictions made by a network at initialization, and 0.0 randomness is the predictions made by a trained supervised classifier.\n5. The randomness corresponds to 0.9 fraction of total training epochs. While the result is somewhat intuitive, it shows a strong connection between the representations learned in SSL and those of a supervised classifier on the same task. This shows that the SSL incrementally learns representations more inline with supervised training. Further, as is shown in Appendix B.5, different networks used for the supervised prediction behave differently with respect to SSL.\n6. We apologize if this sentence is not clear. It refers to Fig 2 (Left), yet it is not shown in the figure as it is normalized. Looking at Fig 6 (Middle) in the Appendix, we see that the sample-level clustering is near 1.0, and the original-label-level clustering is 0.41.\n7. Thank you for this comment. A more subtle statement may say that there is a general monotonic clustering trend in the intermediate layers, while some saturation (and even lower NCC) is in the final layers of the network.\n8. We decided to focus on a single architecture during the main paper to show the myriad of results in our work exhaustively. \n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n ## References \n[1] Y. Le et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015.\n\n[2] S. Maji et al., \"Fine-Grained Visual Classification of Aircraft\", 2013. \n\n[3] Zhang, H. et al. \"0/1 Deep Neural Networks via Block Coordinate Descent\", 2022\n\n[4] M.-E. Nilsback et al., \"Automated Flower Classification over a Large Number of Classes\", 2008\n\n[5] A. Dosovitskiy et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\",  2021\n\n[6] A. Bardes, et al., \"VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning\", 2022 \n\n[7] T. Chen, et al., \"A Simple Framework for Contrastive Learning of Visual Representations\", 2020\n\n[8] J.-B. Grill, et al., \"Bootstrap Your Own Latent a New Approach to Self-Supervised Learning\", 2020\n\n[9] M. Caron, et al., \"Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\", 2020"}, {"review_id": "EaRzFCRMKo", "reviewer": "Reviewer_MBnG", "summary": "This paper provides an empirical study on self-supervised learning (SSL) in terms of clustering. Specifically, the authors demonstrate 1) clustering process of SSL methods, 2) semantic learning of SSL methods based on random labels generated by various models, and 3) class hierarchies and intermediate layers of SSL methods.", "questions": "I understand that one of \"contrastive\" SSL methods works as clustering algorithm, but wonder whether it can be applied to \"cluestering-based\" SSL (e.g., [SWaV](https://arxiv.org/pdf/2006.09882.pdf)) and \"auto-regressive\" SSL (e.g., [MAE](https://arxiv.org/pdf/2111.06377.pdf)). I believe that this is not in the scope of this paper, but it would be better to discuss it for the future research direction.", "limitations": "The authors provide limitations of this paper in Appendix.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of this paper is really good: overally well-written, and easy-to-follow.\n\n- The message of this paper is clear, sound, and consistent to me: self-supervised learning works as a clustering algorithm of feature space, and various empirical observations support it well.\n\n- The experiments are conducted in depth with convincing (but seems narrow) setups.", "weaknesses": "The experiments seems not extensive in terms of dataset and self-supervised algorithms as follows:\n\n- Many self-supervised learning methods are evaluated using ImageNet dataset, and this paper lacks experimental results on ImageNet dataset. This is not critical for accepting this paper, but additional experimental results on ImageNet dataset would improve the message of this paper.\n\n- This paper mainly focuses on one self-supervised learning, VICReg, which may not represent all self-supervised learning methods. This is also not that critical. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer MBnG,\n\nThank you very much for your thorough review and positive feedback on our paper. We are pleased you found our work well-written, clear, and consistent. Your encouraging words on our study's presentation, soundness, and contribution are greatly appreciated.\n\nWe value your constructive comments and would like to address the weaknesses and questions you raised:\n\n## Weaknesses:\n\n1. **Additional Datasets, SSL Methods, and Architectures**: First, we would like to note that besides our results on VICReg with the ResNet backbone on CIFAR100, we have additional results in our appendix using the SimCLR SSL method and the Food dataset. We focused on these datasets and methods due to their simplicity on the one hand and their common use in the literature on the other hand. However, we agree that the analysis would be enriched by considering more datasets, SSL models, and architectures. Prompted by your feedback, we have added a few additional experiments. We ran our analysis on the following:\n- **Datasets** - Additional to CIFAR100, CIFAR10, and FOOD101, we included results on Tiny-ImageNet [1], Aircrafts [2], Oxford Pets [3], and Oxford Flowers [4]. These datasets are much more diverse and contain different features, different numbers of training examples, and different numbers of classes.\n- **SSL Methods** - To further enrich our study, we've added to VICReg and SimCLR, the clustering-based SSL method, SWaV, and Barlow Twins.\n- **Integration with Recent Architectures as Backbones** - Additional to RES-L-W of different widths and depths, we included Vision Transformer (VIT) [5] as an additional backbone\nAs you can see in the attached page, these new models, SSL methods, and architectures exhibit similar behavior to the current experiments. These additions strengthen our paper by providing a more comprehensive and robust analysis. \n\n\nAs you can see in the attached page, these new models, SSL methods, and architectures exhibit similar behavior to the current experiments. These additions strengthen our paper by providing a more comprehensive and robust analysis.\n\n## Questions:\n\n\n**Applicability to Other SSL Methods**: As mentioned above, based on your suggestion, we ran our analysis on SWaV (see Figure 3 in the attached page). As you can observe, the behavior is even clearer and more pronounced compared to contrastive (VICReg) and non-contrastive (SimCLR) methods. This behavior makes sense, as the SWaV loss is designed specifically for clustering. Therefore, even though clustering is not conducted explicitly with specific labels, it appears much more significant compared to methods that do not explicitly enforce clustering and only look at pairs of images.\n\nRegarding the \"auto-regressive\" SSL, since only recently, MAE methods succeeded in achieving state-of-the-art performance in vision datasets [6], we didn't investigate them. However, our intuition is that running our analysis on these methods will give results similar to what we obtain when we look at the hidden space of unsupervised methods [7]. Namely, the features are based on the directions with the highest variance in the input (similar to nonlinear PCA). It would be interesting to explore the differences in representation between SSL and unsupervised learning, as, in principle, we can compare features designed for generation (unsupervised learning) and those designed for the classification of downstream tasks (SSL).\n\n\n\nThank you again for your thoughtful review. We made an effort to address your feedback including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n\n\n\n\n## References \n[1] Y. Le et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015.\n\n[2] S. Maji et al., \"Fine-Grained Visual Classification of Aircraft\",  2013. \n\n[3] Zhang, H. et al. \"0/1 Deep Neural Networks via Block Coordinate Descent\", 2022.\n\n[4] M.-E. Nilsback et al., \"Automated Flower Classification over a Large Number of Classes\", 2008.\n\n[5] A. Dosovitskiy et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\" , 2021. \n\n[6] K. He et al., \"Masked Autoencoders Are Scalable Vision Learners\", 2022.\n\n[7] I. Higgins et al., \"beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework\",  2017.\n"}, {"review_id": "XePa3kpMsi", "reviewer": "Reviewer_b2bW", "summary": "This work presents a study of the structure of SSL representations with respect to sample labels (image identity), to class labels and to super-class labels. To that end, an analysis of the clustering properties of the embedding space according to such labels is performed. Additionally, probing linear layers are trained on top of the SSL representations to assess discriminative performance for the different label categories.\n\nVicReg on CIFAR-100 is evaluated in the experimental results section. The main results show that representations indeed cluster according to multiple hierarchies of labels (sample, class, super-class) and that such clustering is mostly related to the regularization term in VicReg.", "questions": "* The claim _\"Contrary to popular belief, the invariance loss does not significantly improve during the training process\"_ (L257) is not rigorously proven. Have the authors verified this property on a suite of SSL methods, to be able to state it as formulated in the manuscript? I would argue that this is indeed a property of VicReg in the setting of the paper. However, I am doubtful about this finding being a generic SSL property. For that, a much larger study of SSL methods, with different loss term weighings, etc. would be required.\n\n* I am curious about how much the super-class clustering is different from a random super-class assignment (in groups of 5 classes, similarly to the experimental setup already in the paper). Is this related to the label-noise experiment, or would this random grouping yield different results?\n\n-----\nMinor comments:\n* Fig 2 right is not referenced.\n\n-----\nMain reasons behind score:\n\n* Lack of comparison with other SSL papers or datasets. The representation properties analyzed are only based on VicReg, which strongly reduces the significance of this work and contradicts the main contributions (general SSL).\n* Some observations explored in this work are not original, and had been explored in previous works.\n\n-----\nUpdate after rebuttal:\n\nThe authors did a great work in the rebuttal, largely improving the experimental sections and thus addressing most of my concerns. In order to avoid ambiguity, I raise my score to 5.\n\nThe authors also included ImageNet results that support the paper claims, making the paper much stronger as large scale data is also part of the evaluation. Having 4+ methods on several datasets makes the claims of the paper more sound and provide interesting insights to the community. Therefore, after the rebuttal conversations, I raise my score to 6.", "limitations": "No concerns.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Clarity\n\n* The language used is clear.\n* The paper organization seems natural and easy to follow.\n\n### Significance\n\n* Investigating the intrinsic properties of SSL representations is of interest to the community. However, a ore throrough analysis would be needed (see Weaknesses).", "weaknesses": "### Originality\n\n* The analyses provided in this work have already been (at least partially) explored. This reduces the originality of the work. See some references hereafter, not being an exhaustive list:\n  * To the best of my knowledge, the fact that SSL representations cluster according to class labels has already been explored in the past, see (Vasequi, 2021) for example. Although such work focused on label noise resiliency, they show clear evidence of clustering in the t-SNE plots provided.\n    - _Zahra Vaseqi, Ibtihel Amara and Samrudhdhi Rangrej, Label Noise Resiliency with Self-supervised Representations, NeurIPS 2021_.\n\n  * (Grigg, 2021) also showed that linear probing performance increases with depth, in that case with SimCLR representations.\n    - _Tom George Grigg, Dan Busbridge, Jason Ramapuram and Russ Webb, Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?, NeurIPS 2021_.\n\n### Quality\n\n* The experimental results in the main paper only cover 1 SSL algorithm (VicReg) and 1 dataset (CIFAR-100). Such limited evaluation setup does not reflect the main claim of the paper (reverse-engineering SSL), being rather a study of VicReg properties on a specific dataset. It is yet to be seen whether the results obtained with VicReg transfer to:\n  * The myriad of SSL algorithms previously published (i.e. SimCLR, SimSiam, BYOL, SwAV, Barlow Twins, etc.), some of them much more widely used in the community than VicReg.\n  * Different datasets with different properties, specially different inter-class similarities which can strongly affect clustering. Such analysis would give the reader intuition about how different data sources can be expected to work in an SSL framework.\n\n### Clarity\n\n* The fact that the authors refer to SSL-wide conclusions while using conclusions on VicReg+CIFAR100 induces the reader to misunderstanding. I would suggest the authors to either (i) provide a thorough analysis over representative SSL methods and datasets or (ii) refer to VicReg properties instead of general SSL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer b2bW,\n\nThank you very much for your thorough review and feedback on our paper.\n\nWe value your constructive comments and would like to address the weaknesses and questions you raised:\n\n# Weaknesses:\n\n## Originality:\n\nWe appreciate the criticism that has been raised while also maintaining our confidence in the originality of our paper, which introduces several innovative concepts that have yet to be explored in previous research. Specifically, our study delves into SSL algorithms from the unique perspectives of Neural Collapse[6], Randomness, and Information Compression. We extend our gratitude to the reviewer for pointing out the two relevant references, which we will duly address in the revised version of our manuscript. Let us explicitly highlight the distinctions between their contributions and our own.\n\n* **(Vasequi, 2021)**  The focus of this paper is on investigating the robustness of SSL methods with respect to label noise. While a t-SNE visualization of representation from a single pair of networks is briefly presented, the paper does not quantify, study the intermediate relations, and view the SSL algorithms from a geometrical perspective. The clustering of SSL is briefly mentioned, using a visualization - while in our work, we aim to quantify and asses the intrinsic geometries, along with a visualization in Figure 1. \n\n* **(Grigg, 2021)** Thank you for the provided reference. Even though the paper analyzed the linear probing of SimCLR on ResNet50, it does not analyze the clustering aspect with respect to the dataset the SSL was trained on. Rather it studies the similarity between representations of SSL with those of Supervised Learning. This work does not study the geometry formed in different SSL algorithms alone. Rather it compares the representations with those of the supervised setting. Moreover, our paper stands out for its original analysis through the lenses of Neural Collapse, Randomness, and Information Compression, a perspective that sets our work apart.\n\n## Quality:\n\nWe're sorry for the possible confusion in our manuscript. We want to emphasize that even in our original manuscript, there are three datasets (CIFAR100, CIFAR10, and FOOD101) and two SSL algorithms (VICReg and SimCLR - see appendix for more details). Prompted by the reviewer's feedback, we've expanded our experiments and analysis. \n- **Datasets** - Additional to CIFAR100, CIFAR10, and FOOD101, we included results on Tiny-ImageNet [1], Aircrafts [2], Oxford Pets [3], and Oxford Flowers [4]. These datasets are much more diverse and contain different features, different numbers of training examples, and different numbers of classes.\n- **SSL Methods** - To further enrich our study, we've added to VICReg and SimCLR, the clustering-based SSL method, SWaV, and Barlow Twins.\n- **Integration with Recent Architectures as Backbones** - Additional to RES-L-W of different widths and depths, we included Vision Transformer (VIT) [5] as an additional backbone.\n\nAs you can see in the attached page, these new models, SSL methods, and architectures exhibit similar behavior to the current experiments. These additions strengthen our paper by providing a more comprehensive and robust analysis. \n\n## Clarity:\n\nAs mentioned above, we extensively extended our analysis for different datasets (CIFAR100, CIFAR10, FOOD101, Tiny-ImageNet, Aircrafts, Oxford Pets, and Oxford Flowers), SSL algorithms (VICReg, SimCLR, SwAV, BarlowTwins), and backbones (RES-5-250, RES-10-250, RES-5-1000, ViT). \n\n# Questions\n* **Invariance loss during training:** We thank the reviewer for highlighting the gap in our manuscript. Given the reviewer's guidance, we present the decomposition of the loss terms to invariance and regularization terms for SimCLR in the attached file. Figure 3 (left) shows that the observed behavior is similar to VICReg. In the revision of our paper, we will add this analysis to our existing SimCLR section.\n* We thank the reviewer for the interesting question! In Figure 4 (Left) of the attached pdf, we've added the superclass clustering in different layers and randomness levels. Figure 4 (Left) in the main text (of the semantic labels) is similar to that of the superclasses. It is additionally apparent that the superclass clustering improves with respect to decreasing randomness.\n\n# Minor Comments\nFigure 2 (right) is mentioned in L332 regarding the clustering with respect to different hierarchies. We will further reference it for brevity.\n\n\nThank you again for your thoughtful review. We made an effort to address your feedback, including multiple experiments and paper edits, and we would greatly appreciate it if you would consider raising your score in light of our response.  Please let us know if you have additional questions we can address.\n\n# References \n[1] Y. Le et al., \"Tiny ImageNet Visual Recognition Challenge\", 2015.\n\n[2] S. Maji et al., \"Fine-Grained Visual Classification of Aircraft\", 2013. \n\n[3] Zhang, H. et al. \"0/1 Deep Neural Networks via Block Coordinate Descent\", 2022.\n\n[4] M.-E. Nilsback et al., \"Automated Flower Classification over a Large Number of Classes\",  2008.\n\n[5] A. Dosovitskiy et al., \"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\" , 2021. \n\n[6] V. Papyan, et al., \"Prevalence of neural collapse during the terminal phase of deep learning training,\" 2020.\n\n"}]

Author rebuttal:
Dear Reviewers, 

Thank you for your thoughtful and detailed reviews of our work. We appreciate your time and the constructive feedback you have provided. We have carefully considered your comments and concerns and would like to address them in a unified response:

# 1. **Diversity of Architectures and Experiments**:
Several reviewers expressed concerns regarding the limited scope of our experimental studies, and some mistakenly believed that we only conducted experiments on CIFAR100 using the VICReg method. We recognize that we may need to be clearer in the paper about the diversity of our experiments, and we apologize for any confusion.

In the appendix of our original submission, we included additional results using CIFAR10 and FOOD101 datasets, as well as SimCLR as an SSL method. Additionally, we have included analysis using different random target functions, using both ResNet and ViT. We understand this information may not have been easily accessible, and we have taken steps to rectify this oversight in the revised manuscript. 

In response to the reviewers' feedback, we have further expanded our analysis to include:
- **Datasets** - Additional to CIFAR100, CIFAR10, and FOOD101, we included results on Tiny-ImageNet [1], Aircrafts [2], Oxford Pets [3], and Oxford Flowers [4]. These datasets are much more diverse and contain different features, different numbers of training examples, and different numbers of classes.
- **SSL Methods** - To further enrich our study, we've added to VICReg and SimCLR, the clustering-based SSL method, SWaV, and Barlow Twins.
- **Integration with Recent Architectures as Backbones** - Additional to RES-L-W of different widths and depths, we included Vision Transformer (VIT) [5] as an additional backbone.

As you can see on the attached page, these new models, SSL methods, and architectures exhibit similar behavior to the current experiments. These additions, along with clarifying our existing results in the appendix, strengthen our paper by providing a more comprehensive and robust analysis.

# 2. **Originality and Quality**:
We addressed concerns raised about the originality of our paper by emphasizing our unique approach, specifically our analysis through the lenses of Neural Collapse, Randomness, and Information Compression. We also clarified the distinctions between our contributions and related works cited by the reviewers. Our expanded experiments and more detailed explanations have enhanced our paper's quality.

# 3. **Clarity and Presentation**:
In response to feedback on the clarity and flow of our manuscript, we have revised sections to provide clearer explanations and more explicit connections between our analysis and figures. We've ensured that our results are presented in an accessible manner and that our key insights are communicated effectively.

# 4. **Addressing Concerns and Questions**:
We addressed each reviewer's individual weaknesses and questions in detail. Please refer to the individual responses below for detailed explanations of how we addressed each concern.

**In conclusion**, we sincerely believe that the revisions and clarifications made in response to the reviewers' feedback have significantly strengthened our manuscript. 

We have put forth a significant effort to address all your feedback, and your suggestions have improved our work.  We welcome further discussion, and we appreciate your valuable input. Again, Thank you for your time and insights that have greatly contributed to enhancing our work.

# References
[1] Y. Le et al., "Tiny ImageNet Visual Recognition Challenge", 2015.

[2] S. Maji et al., "Fine-Grained Visual Classification of Aircraft" , 2013. 

[3] Zhang, H. et al. "0/1 Deep Neural Networks via Block Coordinate Descent", 2022.

[4] M.-E. Nilsback et al., "Automated Flower Classification over a Large Number of Classes", 2008

[5] A. Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", 2021.

Meta-review: The paper provides an in-depth analysis of self-supervisedly learned visual representations from the perspective of latent semantic clustering. While the initial paper was still lacking larger-scale (imagenet) results, this was sufficiently addressed in the rebuttal to the satisfaction of the reviewers. As a result, the reviewers find this paper provides an important study with qualitative and quantitative results that is clearly written
Decision: Accept (poster)