Title: Reverse Engineering Self-Supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-Supervised Learning

Deep Learning

Representation Learning
Abstract: Understanding the learned representation and underlying mechanisms of Self-Supervised Learning (SSL) often poses a challenge. In this paper, we ‘reverse engineer’ SSL, conducting an in-depth empirical analysis of its learned internal representations, encompassing diverse models, architectures, and hyperparameters. Our study reveals an intriguing process within the SSL training: an inherent facilitation of semantic label-based clustering, which is surprisingly driven by the regularization component of the SSL objective. This clustering not only enhances downstream classification, but also compresses the information. We further illustrate that the alignment of the SSL-trained representation is more pronounced with semantic classes rather than random functions. Remarkably, the learned representations align with semantic classes across various hierarchical levels, with this alignment intensifying when going deeper into the network. This ‘reverse engineering’ approach provides valuable insights into the inner mechanism of SSL and their influences on the performance across different class sets.

Reverse Engineering Self-Supervised Learning

Abstract

Self-supervised learning (SSL) is a powerful tool in machine learning, but understanding the learned representations and their underlying mechanisms remains a challenge. This paper presents an in-depth empirical analysis of SSL-trained representations, encompassing diverse models, architectures, and hyperparameters. Our study reveals an intriguing aspect of the SSL training process: it inherently facilitates the clustering of samples with respect to semantic labels, which is surprisingly driven by the SSL objective's regularization term. This clustering process not only enhances downstream classification but also compresses the data information. Furthermore, we establish that SSL-trained representations align more closely with semantic classes rather than random classes. Remarkably, we show that learned representations align with semantic classes across various hierarchical levels, and this alignment increases during training and when moving deeper into the network. Our findings provide valuable insights into SSL's representation learning mechanisms and their impact on performance across different sets of classes.

1 Introduction

Self-supervised learning (SSL) [12, 5] has made significant progress over the last few years, almost reaching the performance of supervised baselines on many downstream tasks [42, 4, 31, 30, 47, 34, 15, 37, 66, 16]. However, understanding the learned representations and their underlying mechanisms remains a persistent challenge due to the complexity of the models and the lack of labeled training data. Moreover, the pretext tasks used in self-supervision frequently lack direct relevance to the specific downstream tasks, further complicating the interpretation of the learned representations.

In supervised classification, however, the structure of the learned representations is often very simple. For instance, a recent line of work [50, 36] has identified a universal phenomenon called neural collapse, which occurs at the terminal stages of training. Neural collapse reveals several properties of the top-layer embeddings, such as mapping samples from the same class to the same point, maximizing the margin between the classes, and certain dualities between activations and the toplayer weight matrix. Later studies [10, 28, 26, 56, 62] characterized related properties at intermediate layers of the network.

Initialization

Sample

Original

Super

End of train

Compared to traditional classification tasks that aim to accurately categorize samples into specific classes, modern SSL algorithms typically minimize loss functions that combine two essential components: one for clustering augmented samples (invariance constraint) and another for preventing representation collapse (regularization constraint). For example, contrastive learning methods [14, 48, 63, 46] train representations to be indistinguishable for different augmentations of the same sample and at the same time to distinguish augmentations of different samples. On the other hand, non-contrastive approaches [6, 66, 34] use regularizers to avoid representation collapse.

Contributions. In this paper, we provide an in-depth analysis of representation learning with SSL through a series of carefully designed experiments. Our investigation sheds light on the clustering process that occurs during training. Specifically, our findings reveal that augmented samples exhibit highly clustered behavior, forming centroids around the mean embedding of augmented samples sharing the same original image. More surprisingly, we observe that clustering also occurs with respect to semantic labels, despite the absence of explicit information about the target task. This demonstrates the capability of SSL to capture and group samples based on semantic similarities.

Our main contributions are summarized as follows:

Clustering: We investigate the clustering properties of SSL-trained representations. In Figure 2, we show that akin to supervised classification [50], SSL-trained representation functions exhibit a centroid-like geometric structure, where feature embeddings of augmented samples belonging to the same image tend to cluster around their respective means. Interestingly, at later stages of training, a similar trend appears with respect to semantic classes.

The role of regularization: In Figure 3 (left) we show that the accuracy of extracting (i.e., linear probing) semantic classes from SSL-trained representations continuously improves even after the model accurately clusters the augmented training samples based on their sample identity. As can be seen, the regularization constraint plays a key role in inducing the clustering of data into semantic attributes at the embedding space.

Impact of randomness: We argue that SSL-trained representation functions are highly correlated with semantic classes. To support this claim, we study how the degree of randomness in the targets affects the ability to learn them from a pretrained representation. As we show in Figure 4, the alignment between the targets and the learned representations substantially improves as the targets become less random and more semantically meaningful.

Learning hierarchic representations: We study how SSL algorithms learn representations that exhibit hierarchic classes. We demonstrate that throughout the training process, the ability to cluster and distinguish between samples with respect to different levels of semantic targets continually improves at all layers of the model. We consider several levels of hierarchy, such as the sample level (the sample identity), semantic classes (such as different types of animals and vehicles), and high-level classes (e.g., animals, and vehicles).

Furthermore, we analyze the ability of each layer to capture distinct semantic classes. As shown in Figure 5, we observe that the clustering and separation ability of each layer improves as we move deeper into the network. This phenomenon closely resembles the behavior observed in supervised learning [10, 26, 56, 62], despite the fact that SSL-trained models are trained without direct access to the semantic targets.

In Section 4, we investigate various clustering properties in the top layer of SSL-trained networks. In Section 5, we conduct several experiments to understand what kinds of functions are encoded in SSL-trained neural networks and can be extracted from their features. In Section 6, we study how SSL algorithms learn different hierarchies and how this occurs at intermediate layers.

2 Background and Related Work

2.1 Self-Supervised Learning

SSL is a family of techniques that leverages unlabeled data to learn representation functions that can be easily adapted to new downstream tasks. Unlike supervised learning, which relies on labeled data, SSL employs self-defined signals to establish a proxy objective. The model is pre-trained using this proxy objective and then fine-tuned on the supervised downstream task. However, a major challenge of this approach is to prevent 'representation collapse', where the model maps all inputs to the same output [39].

Several methods have been proposed to address this challenge. Contrastive learning methods such as SimCLR [14] and its InfoNCE criterion [64] learn representations by maximizing the agreement between positive pairs (augmentations of the same sample) and minimizing the agreement between negative pairs (augmentations of different samples). In contrast, non-contrastive learning methods [6, 66, 34] replace the dependence on contrasting positive and negative samples, by applying certain regularization techniques to prevent representation collapse. For instance, some methods use stop-gradients and extra predictors to avoid collapse [16, 34], while others use an additional clustering constraints [13].

Variance-Invariance-Covariance Regularization (VICReg). A widely used method for SSL training [6], which generates representations for both an input and its augmented counterpart. It aims to optimize two key aspects:

Invariance loss - The mean-squared Euclidean distance between pairs of embedding, serving to ensure consistency between the representation of the original and augmented inputs.

Regularization - Comprising two elements, the variance loss , which promotes increased variance across the batch dimension through a hinge loss restraining the standard deviation, and the covariance loss , which penalizes off-diagonal coefficients of the covariance matrix of the embeddings to foster decorrelation among features.

This leads to the VICReg loss function:

where L ( f ) is minimized over batches of samples I . s ( Z, Z ′ ) , v ( Z ) , and c ( Z ) denote the invariance, variance, and covariance losses respectively, and λ, µ, ν > 0 are hyperparameters controlling the balance between these loss components.

A Simple Framework for Contrastive Learning of Visual Representations (SimCLR). The SimCLR method [14], another popular approach, minimizes the contrastive loss function [35]. Given two batches of embeddings Z , Z ′ , we can also decompose the objective into an invariance term and a

regularization term:

̸

where sim ( a, b ) := a ⊤ b ∥ a ∥·∥ b ∥ is the cosine similarity between vectors a and b , and τ > 0 is a 'temperature' parameter that controls the sharpness of the distribution. Intuitively, minimizing the loss encourages the representations Z i and Z ′ i of the same input sample to be similar (i.e., have a high cosine similarity) while pushing away the representations Z i and Z ′ j of different samples (i.e., have a low cosine similarity) for all i, j ∈ [ B ] .

2.2 Neural Collapse and Clustering

In a recent paper [50], it was demonstrated that deep networks trained for classification tasks exhibit a notable behavior: the top-layer feature embeddings of training samples of the same class tend to concentrate around their respective class means, which are maximally distant from each other. This behavior is generally regarded as desirable since max-margin classifiers tend to exhibit better generalization guarantees (e.g., [2, 7, 33]), and clustered embedding spaces are useful for few-shot transfer learning (e.g., [32, 28, 27, 29]).

In this paper, we aim to explore whether similar clustering tendencies occur in SSL-trained representation functions at both the sample level (with respect to augmentations) and at the semantic level. To address this question, we first introduce some key aspects of neural collapse. Specifically, we will focus on class-feature variance collapse and nearest class-center (NCC) separability.

̸

To measure these properties, we use the following metrics. Suppose we have a representation function f : R d → R p and a collection of unlabeled datasets S 1 , . . . , S C ⊂ R d (each corresponding to a different class). The feature-variability collapse measures to what extent the samples from the same class are mapped to the same vector. To quantify this property we use the averaged classdistance normalized variance (CDNV) [28] Avg i = j [ V f ( S i , S j )] = Avg i = j [ Var f ( S i )+Var f ( S j ) 2 ∥ µ f ( S i ) -µ f ( S j ) ∥ 2 ] , where µ f ( S ) and Var f ( S ) are the mean and variance of the uniform distribution U [ { f ( x ) | x ∈ S } ] .

̸

A weaker notion of clustering is the nearest class-center (NCC) separability [26], which measures to what extent the feature embeddings of training samples form centroid-like geometry. The NCC separability asserts that, during training, the penultimate-layer's feature embeddings of training samples can be accurately classified with the 'nearest class-center (NCC) classifier', h ( x ) = arg min c ∈ [ C ] ∥ f ( x ) -µ f ( S c ) ∥ . To measure this property, we compute the NCC train and test accuracies, which are simply the accuracy rates of the NCC classifier.

A recent paper [22] argued that idealized SSL-trained representations simultaneously cluster data into multiple equivalence classes. This implies that there are various ways to categorize the data that result in high NCC accuracy. However, achieving a very small CDNV for multiple different categorizations is impossible since a zero CDNV would mean that feature embeddings of two samples from the same category are identical. Consequently, an intriguing question arises: with respect to which sets of classes do the learned feature embeddings cluster?

2.3 Reverse Engineering Neural Networks

Reverse engineering neural networks have recently garnered attention as an approach to explain how neural networks make predictions and decisions. While substantial work has been dedicated to understanding the functionalities of intermediate layers of neural network classifiers [23, 9, 17, 50, 51, 60, 56, 3], characterizing the functionalities of representation functions generated by SSL algorithms remains an open problem.

A major source of complexity is the reliance on a pretext task and image augmentations. For instance, to fully understand the success of SSL it is essential to understand the relationship between the pretext task and the downstream task [58, 65]. It is yet unclear how to properly train representations with SSL, and therefore, SSL algorithms are quite complicated and include many different types of losses [14, 66, 6] and optimization tricks [14, 16, 37]. Of note, it is not obvious what are the differences between representations that are learned with contrastive learning and non-contrastive

1

10

100

1000

1

1.4

1.8

2.6

Hierarchy

Original

Sample

Epoch

NCC Accuracy

1

10

100

1000

1

1.2

1.4

Hierarchy

Original

Sample

Epoch

Normalized Ratio

1

10

100

1000

1

1.2

1.4

1.6

Hierarchy

Sample

Original

Super

Epoch

Normalized Linear Accuracy

1

10

100

1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.3

0.4

0.5

Inv.

Acc.

Epoch

Loss

Linear Accuracy

1

10

100

1000

4.6

5

5.4

Epoch

Mutual Information

Supervised

Supervised + Aug.

SSL

0

0.2

0.4

0.6

0.8

1

Train

Test

NCC Accuracy

learning methods [52]. In this work, we make new strides to understand SSL-trained representations, by investigating their clustering properties with respect to various types of classes.

3 Problem Setup

Self-supervised learning (SSL) is frequently used to pretrain models, preparing them for adaptation to new downstream tasks. This raises an essential question: How does SSL training influence the learned representation? Specifically, what underlying mechanisms are at work during this training, and which classes can be learned from these representation functions? To investigate this, we train SSL networks across various settings and analyze their behavior using different techniques. All the training details can be found in Appendix A.

Data and augmentations. Throughout all of the experiments (in the main text) we used the CIFAR-100 [41] image classification dataset, which contains 100 classes of semantic objects (original classes), which are also divided into 20 superclasses, with each superclass containing 5 classes (for example, the 'fish' superclass contains the 'aquarium fish', 'flatfish', 'ray', 'shark' and 'trout' categories). In order to train our models we used the image augmentation protocol introduced in SimCLR [14]. Each SSL training session is carried out for 1000 epochs, using the SGD optimizer with momentum. Usually, SSL-trained models are tested on a variety of downstream datasets to assess their performance. To simplify our analysis and minimize the influence of potential distribution shifts, we specifically evaluate the models' performance on the CIFAR-100 dataset. Such evaluations are typically done in SSL research [6, 34, 66], usually training on and evaluating on ILSVRC (ImageNet). Additional evaluations on various datasets (CIFAR-10 [41], FOOD101 [11], Aircrafts [45], Tiny Imagnet [43], and Imagenet [19]) can be found in Appendix B.2.

0.99

0.95

0.75

0

0.35

0.45

0.55

Epoch

20

40

100

400

1000

Randomness Level

Linear Accuracy

0.99

0.95

0.75

0

0.25

0.35

0.45

0.55

Layer

1

3

5

7

9

Randomness Level

Linear Accuracy

20

100

1000

0.8

1

1.2

1.4

1.6

Metric

NCC

CDNV

Distance Sq.

Epoch

Ratio

1

2

3

4

5

6

7

8

9

10

0.4

0.5

0.6

Epoch

20

40

100

400

1000

Layer

Linear Accuracy

1

2

3

4

5

6

7

8

9

10

0.4

0.5

0.6

Epoch

20

40

100

400

1000

Layer

Linear Accuracy

1

2

3

4

5

6

7

8

9

10

1.12

1.16

1.2

1.24

Epoch

20

40

100

400

1000

Layer

Backbone architecture. In all our experiments, we utilized the RESL -H architecture as the backbone, coupled with a two-layer multi-layer perceptron (MLP) projection head. The RESL -H architecture is a variant of the ResNet architecture [38] where the width H remains constant across all L residual blocks as introduced in [26].

Linear probing. To evaluate the effectiveness of extracting a given discrete function (e.g., categories) from a representation function, we employ linear probing. In this approach, we train a linear classifier, also known as a linear probe, on top of the representation, using a set of training samples. To train the linear classifier, we used the Linear Support Vector Classification (LSVC) algorithm from the scikit-learn library [54]. The 'linear accuracy' is determined by measuring the performance of the trained linear classifier on a validation set.

Sample level classification. We aim to evaluate the capacity of a given representation function to correctly classify different augmentations originating from the same image and distinguish them from augmentations of different images. To facilitate this, we constructed a new dataset specifically designed to evaluate sample-level separability.

The training dataset comprises 500 random images sourced from the CIFAR-100 training set. Each image represents a particular class and undergoes 100 different augmentations. As such, the training dataset contains 500 classes with a total of 50000 samples. For the test set, we utilize the same 500 images but apply 20 different augmentations drawn from the same distribution. This results in a test set consisting of 10000 samples. To measure the linear or NCC accuracy rates of a given representation function at the sample level, we first compute a relevant classifier using the training data and subsequently evaluate its accuracy rates on the corresponding test set.

4 Unraveling the Clustering Process in Self-Supervised Learning

The clustering process has long played a significant role in aiding the analysis of deep learning models [1, 17, 50]. To gain intuition behind the SSL training, in Figure 1 we present a UMAP [57]

visualization of the network's embedding space for the training samples, both pre-and post-training, across different hierarchies.

As anticipated, the training process successfully clusters together samples at the per-sample level, mapping different augmentations of the same image (as depicted in the first row). This outcome is unsurprising given that the objective inherently encourages this behavior (through the invariance loss term). Yet, more remarkably, the training process also clusters the original 'semantic classes' from the standard CIFAR-100 dataset, even though the labels were absent during the training process. Interestingly, the high-level hierarchy (the superclasses) is also effectively clustered. This example demonstrates that while our training procedure directly encourages clustering at the sample level, the SSL-trained representations of data are additionally clustered with respect to semantic classes across different hierarchies.

To quantify this clustering process further, we train a RES-10-250 network using VICReg. We measure the NCC train accuracy, which is a clustering measure (as described in Section 2.2), both at the sample level and with respect to the original classes. Notably, the SSL-trained representation exhibits neural collapse at a sample level (with an NCC train accuracy near 1 . 0 ), yet the clustering with respect to the semantic classes is also significant ( ≈ 0 . 41 on the original targets).

As shown in Figure 2 (left), most of the clustering process with respect to the augmentations (on which the network was directly trained) occurs at the beginning of the training process and then plateaus, while the clustering with respect to semantic classes (which is not directly featured in the training objective) continues to improve throughout the training process.

A key property observed in [50] is that the top-layer embeddings of supervised training samples gradually converge towards a centroid-like structure. In our quest to better understand the clustering properties of SSL-trained representation functions, we investigated whether something similar occurs with SSL. The NCC classifier's performance, being a linear classifier, is bounded by the performance of the best-performing linear classifier. By evaluating the ratio between the accuracy of the NCC classifier and a linear classifier trained on the same data, we can investigate data clustering at various levels of granularity. In Figure 2 (middle), we monitor this ratio for the sample-level classes and for the original targets, normalized with respect to its value at initialization. As SSL training progresses, the gap between NCC accuracy and linear accuracy narrows, indicating that augmented samples progressively cluster more based on their sample identity and semantic attributes. Furthermore, the plot shows that the sample-level ratio is initially higher, suggesting that augmented samples cluster based on their identity until they converge to centroids (the ratio between the NCC accuracy and the linear accuracy is ≥ 0 . 9 at epoch 100 ). However, as training continues, the sample-level ratio saturates while the class-level ratio continues to grow and converges to ≈ 0 . 75 . This implies that the augmented samples are initially clustered with respect to their sample identities, and once achieved, they tend to cluster with respect to high-level semantic classes.

Implicit information compression in SSL training. Having examined the clustering process within SSL training, we next turn our attention to analyzing information compression during the learning process. As outlined in Section 2.2, effective compression often yields representations that are both beneficial and practical. Nevertheless, it is still largely uncharted territory whether such compression indeed occurs during the course of SSL training [59].

To shed light on this, we utilize the Mutual Information Neural Estimation (MINE) [8], a method designed to estimate the mutual information between the input and its corresponding embedded representation during the course of training. This metric serves as an effective gauge of the representation's complexity level, essentially demonstrating how much information (number of bits) it encodes.

In Figure 3 (middle) we report the average mutual information computed across 5 different MINE initialization seeds. As demonstrated, the training process manifests significant compression, culminating in highly compact trained representations. This observation aligns seamlessly with our previous findings related to clustering, where we noted that more densely clustered representations correspond to more compressed information.

The role of regularization loss. To gain a deeper understanding of the factors influencing this clustering process, we study the distinct components of the SSL objective function. As we discussed in section 2.1, the SSL objective function is composed of two terms: invariance and regularization. The invariance term's main function is to enforce similarity among the representations of augmentations of the same sample. In contrast, the regularization term helps to prevent representation collapse.

To investigate the impact of these components on the clustering process, we dissect the objective function into the invariance and regularization terms and observe their behavior during the course of training. As part of this exploration, we trained a RES-5-250 network using VICReg with µ = λ = 25 and ν = 1 , as proposed in [6]. Figure 3 (left) presents this comparison, charting the progression of loss terms and the linear test accuracy with respect to the original semantic targets. Contrary to popular belief, the invariance loss does not significantly improve during the training process. Instead, the improvement in loss (and downstream semantic accuracy) is achieved due to minimizing the regularization loss. This observation is consistent with our earlier finding that the per-sampling clustering process (which is driven by the invariance loss) saturates early in the training process.

We conclude that the majority of the SSL training process is geared towards improving the semantic accuracy and clustering of the learned representations, rather than the per-sample classification accuracy and clustering. This is consistent with the observed trend of the regularization loss improving over the course of training and the early saturation of the invariance loss.

In essence, our findings suggest that while self-supervised learning directly targets sample-level clustering, the majority of the training time is spent on orchestrating data clustering according to semantic classes across various hierarchies. This observation underscores the remarkable versatility of SSL methodologies in generating semantically meaningful representations through clustering and sheds light on its underlying mechanisms.

Comparing supervised learning and SSL clustering. As mentioned in Section 4, deep network classifiers tend to cluster training samples into centroids based on their classes. However, the learned function is truly clustering the classes only if this property holds for the test samples, which is expected to occur but to a lesser degree.

An interesting question is to what extent SSL implicitly clusters the samples according to their semantic classes, compared to the clustering induced by supervised learning. In Figure 3 (right), we report the NCC train and test accuracy rates at the end of training for different scenarios: supervised learning with and without augmentations and SSL. For all cases, we used the RES-10-250 architecture.

While the NCC train accuracy of the supervised classifier is 1 . 0 , which is significantly higher than the NCC train accuracy of the SSL-trained model, the NCC test accuracy of the SSL model is slightly higher than the NCC test accuracy of the supervised model. This implies that both models exhibit a similar degree of clustering behavior with respect to the semantic classes. Interestingly, when training the supervised model with augmentation slightly decreases the NCC train accuracy, yet significantly improves the NCC test accuracy.

5 Exploring Semantic Class Learning and the Impact of Randomness

Semantic classes define relationships between inputs and targets based on inherent patterns in the input. On the other hand, mappings from inputs to random targets lack discernible patterns, resulting in seemingly arbitrary connections between inputs and targets.

In this section, we study the influence of randomness on a model's proficiency in learning desired targets. To do so, we build a series of target systems characterized by varying degrees of randomness and examine their effect on the learned representations. We train a neural network classifier on the same dataset for classification and use its target predictions at different epochs as targets with varying degrees of randomness. At epoch 0, the network is random, generating deterministic yet seemingly arbitrary labels. As training advances, the functions become less random, culminating in targets that align with ground truth targets (considered non-random). We normalize the degree of randomness between 0 (non-random, end of training) and 1 (random, initialization). Utilizing this methodology, we investigate the classes SSL learns during training.

Initially, our emphasis is on exploring the impact of the SSL training process on the learned targets. We train a RES-5-250 network with VICReg and employ a ResNet-18 to generate targets with different levels of randomness. Figure 4 (left) showcases the linear test accuracy for varying degrees of random targets. Each line corresponds to the accuracy at a different stage of SSL training for different levels of randomness. As we can see, the model captures classes closer to 'semantic' ones (lower degrees of randomness) more effectively throughout training while not showing significant performance improvement on highly random targets. For results with additional random target generators, see Appendix B.5.

A key question in deep learning revolves around understanding the functionalities and the impact of intermediate layers on classifying different types of classes. For example, do different layers learn different kinds of classes? We explore this by evaluating the linear test accuracy of different layer representations for various degrees of target randomness at the end of the training. As shown in Figure 4 (middle), the linear test accuracy consistently improves at all layers as randomness decreases, with deeper layers outperforming across all class types and the performance gap broadening for classes closer to semantic ones.

Beyond classification accuracy, a desirable representation exhibits a high degree of clustering with respect to the targets [50, 24, 32, 28]. We assess the quality of clustering using several metrics: NCC accuracy, CDNV, average per-class variance, and average squared distance between class means [28]. To gauge the improvement in representation over training, we calculate the ratios of these metrics for semantic vs. random targets. Figure 4(right) displays these ratios, indicating that the representation increasingly clusters the data with respect to semantic targets compared to random ones, as the ratio of the NCC, CDNV, increases during training. Interestingly, we see that the decrease in the CDNV, which is the variance divided by the squared distance, is caused solely by the increase in squared distance. The variance ratio stays fairly constant during training. This phenomenon of encouraging large margins between clusters has been shown to improve the performance [24, 17].

6 Learning Class Hierarchies and Intermediate Layers

Previous studies have given evidence that in the context of supervised learning, intermediate layers gradually capture features at varying levels of abstraction [1, 17, 50, 26, 10]. The initial layers tend to focus on low-level features, while deeper layers capture more abstract features. In Section 4, we showed that SSL implicitly learns representations highly correlated with semantic attributes. Next, we investigate whether the network learns higher-level hierarchical attributes and which layers are better correlated with these attributes.

In order to measure what kinds of targets are associated with the learned representations, we trained a RES-5-250 network using VICReg. We compute the linear test accuracy (normalized by its value at initialization) with respect to three hierarchies: the sample-level, the original 100 classes, and the 20 superclasses. In Figure 2 (right), we plot these quantities computed for the three different sets of classes. We observed that, in contrast to the sample-level classes, the performance with respect to both the original classes and the superclasses significantly increased during training. The complete details are provided in Appendix A.

As a next step, we investigate the behavior of intermediate layers of SSL-trained models and their ability to capture targets of different hierarchies. To this end, we trained a RES-10-250 with VICReg. In Figure 5 (left and middle), we plot the linear test accuracy across all intermediate layers at various stages of training, measured for both the original targets and the superclasses. In Figure 5 (right) we plot the ratios between the superclasses and the original classes.

We draw several conclusions from these results. First, we observe a consistent improvement in clustering as we move from earlier to deeper layers, becoming more prominent during the course of training. Furthermore, similar to supervised learning scenarios [62, 26, 10], we find that the linear accuracy of the network improves in each layer during SSL training. Of note, we find that for the original classes, the final layers are not the optimal ones. Recent studies in SSL [49, 55] showed that the performance of the different algorithms are highly sensitive to the downstream task domains. Our study extends this observation and suggests that also different parts of the network may be more suitable for certain downstream tasks and class hierarchies. Figure 5(right), it is evident that relatively, the accuracy of the superclasses improves more than that of the original classes in the deeper layers of the network.

7 Conclusions

Representation functions that cluster data based on semantic classes are generally favored due to their ability to classify classes accurately with limited samples [32, 28, 27, 29]. In this paper, we conduct a comprehensive empirical exploration of SSL-trained representations and their clustering properties concerning semantic classes. Our findings reveal an intriguing impact of the regularization constraint in SSL algorithms. While regularization is primarily used to prevent representation collapse, it also enhances the alignment between learned representations and semantic classes, even after accurately clustering augmented training samples. We investigate the emergence of different class types during

training, including targets with various hierarchies and levels of randomness, and examine their alignment at different intermediate layers. Collectively, these results provide substantial evidence that SSL algorithms learn representations that effectively cluster based on semantic classes.

Despite the similarities observed between the supervised setting and the SSL setting in our paper, several questions remain unanswered. Foremost among them is the inquiry into why SSL algorithms learn semantic classes. Although we present compelling evidence linking this phenomenon to the regularization constraint, the explanation of how this term manages to cluster data with respect to semantic classes, and the specific types of classes being learned, remains unclear. A deeper understanding of the types of classes to which data clusters, coupled with the connection between neural collapse and transfer learning [28, 27, 29], may be helpful in understanding the transferability of SSL-trained representations to downstream tasks.

References

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017.

Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, USA, 1st edition, 2009.

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics , 6:483-495, 2018.

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A cookbook of self-supervised learning, 2023.

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations , 2022.

Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems , volume 30. Curran Associates Inc., 2017.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 , 2018.

Ido Ben-Shaul and Shai Dekel. Sparsity-probe: Analysis tool for deep learning models. ArXiv , abs/2105.06849, 2021.

Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022 , volume 196 of Proceedings of Machine Learning Research , pages 37-47. PMLR, 2022.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 , pages 446-461. Springer International Publishing, 2014.

Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard S¨ ackinger, and Roopak Shah. Signature verification using a 'siamese' time delay neural network. In J. Cowan, G. Tesauro, and J. Alspector, editors, Advances in Neural Information Processing Systems , volume 6. MorganKaufmann, 1993.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML) , 2020.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 22243-22255. Curran Associates, Inc., 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15750-15758, June 2021.

Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Dnn or k-nn: That is the generalize vs. memorize question, 2018.

Ekin Dogus Cubuk, Barret Zoph, Dandelion Man´ e, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 113-123, 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021 , abs/2010.11929, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving selfsupervised learning by characterizing idealized representations. In Advances in Neural Information Processing Systems , volume 35, pages 11279-11296, 2022.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021.

Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

William Falcon et al. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning , 3, 2019.

Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the Implicit Bias Towards Minimal Depth of Deep Neural Networks. arXiv , 2022.

Tomer Galanti, Andr´ as Gy¨ orgy, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. ICML Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward , 2022.

Tomer Galanti, Andr´ as Gy¨ orgy, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations , 2022.

Tomer Galanti, Andr´ as Gy¨ orgy, and Marcus Hutter. Generalization bounds for transfer learning with pretrained classifiers, 2022.

Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online bag-of-visual-words generation for self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6830-6840, June 2021.

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations , 2018.

Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriia Cherepanova, and Tom Goldstein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Hal Daum´ e III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 3607-3616. PMLR, 13-18 Jul 2020.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA , 9(2):473-504, 05 2020.

Jean-Bastien Grill, Florian Strub, Florent Altch´ e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´ emi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In Advances in Neural Information Processing Systems , Red Hook, NY, USA, 2020. Curran Associates Inc.

R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 1735-1742, 2006.

X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations , 2022.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations , 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical Report , 2009.

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for automatic colorization. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 , pages 577-593, Cham, 2016. Springer International Publishing.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017.

S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6706-6716, 2019.

Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noisecontrastive estimation. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Maxime Oquab, Timoth´ ee Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. , 18:83:1-83:52, 2017.

Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do selfsupervised vision transformers learn? In The Eleventh International Conference on Learning Representations , 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 8748-8763, 2021.

Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Proceedings of the 40th International Conference on Machine Learning , Proceedings of Machine Learning Research, 2023.

Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Computation , 33(11):2881-2907, 2021.

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 5628-5637, 2019.

Ravid Shwartz-Ziv and Yann LeCun. To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review. arXiv , 2023.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. ArXiv , abs/1703.00810, 2017.

Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et al. Lightly. GitHub. Note: https://github.com/lightly-ai/lightly , 2020.

Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 21478-21505. PMLR, 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv , 2018.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Julius von K¨ ugelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Sch¨ olkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 16451-16467. Curran Associates, Inc., 2021.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ ephane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , 2021.

A Training Details

A.1 Data augmentation

We follow the image augmentation protocol first introduced in SimCLR [14] and now commonly used by similar approaches based on siamese networks [13, 34, 14, 66]. Two random crops from the input image are cropped and resized to 32 × 32 , followed by random horizontal flip, color jittering of brightness, contrast, saturation and hue, Gaussian blur, and random grayscale. Each crop is normalized in each color channel using the ImageNet mean and standard deviation pixel values. The following operations are performed sequentially to produce each view:

Random cropping with an area uniformly sampled with size ratio between 0.08 to 1.0, followed by resizing to size 32 × 32. RandomResizedCrop(32, scale=(0.08, 1.0)) in PyTorch.

Random horizontal flip with probability 0.5.

Color jittering of brightness, contrast, saturation and hue, with probability 0.8. ColorJitter(0.4, 0.4, 0.2, 0.1) in PyTorch.

Grayscale with probability 0.2.

Gaussian blur with probability 0.5 and kernel size 23.

Solarization with probability 0.1.

Color normalization with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) (ImageNet standardization).

Supervised learning - Figure 3 (right). For the supervised learning setting, we used AutoAugment [18] with a policy created for CIFAR-10 [41].

A.2 Network Architectures

In this section, we describe the network architectures used in our experiments.

SSL backbone. The main architecture used as the SSL backbone is a convolutional residual network, denoted RESL -H . It consists of a stack of two 2 × 2 convolutional layers with stride 2 , batch normalization, and ReLU activation, followed by L residual blocks. The i th block computes g i ( x ) = σ ( x + B 2 i ( C 2 i ( σ ( B 1 i ( C 1 i ( x )))))) , where C j i is a 3 × 3 convolutional layer with H channels, stride 1 , and padding 1 , B j i is a batch normalization layer, for j ∈ { 1 , 2 } , and σ is the ReLU activation.

Random target functions. Throughout the paper, we use two different random target functions, a ResNet-18 [38] and a visual transformer (ViT) [21] (Appendix B.5). For the transformer architecture, we used 10 layers, with 8 attention heads and 384 hidden dimensions. The classification was done using a classification token.

A.3 Optimization

SSL. We trained all of our SSL models for 1000 epochs using a batch size of 256. We used the SGD optimizer with a learning rate of 0.002, a momentum value of 0.9, and a weight decay value of 1 e -6 . Additionally, we used a CosineAnnealing learning rate scheduler [44].

Random targets. The ResNet-18 targets were trained using the Cross-Entropy loss, with the Adam optimizer [40], using a learning rate of 0.05 and the Cosine Annealing learning rate scheduler. For the ViT training, we used the Adam optimizer with a 0.001 learning rate, weight decay of 5e-5, and a Cosine Annealing learning rate scheduler. We used the model's class predictions as the targets.

A.4 Implementation Details

Our experiments were implemented in PyTorch [53], utilizing the Lightly [61] library for SSL models and PyTorch Lightning [25] for training. All of our models were trained on V100 GPUs.

B Additional Experiments

Clustering in SSL. In Figure 2 in the main text, we reported behaviors of extracting classes of different hierarchies from SSL-trained models. In this experiment, we provide the un-normalized version of the results in Figure 2. Specifically, in Figure 6 (left), we show the un-normalized results, along with the normalized plots shown in the main text Figure 6 (right).

1

10

100

1000

0.3

0.5

0.7

0.9

Hierarchy

Sample

Original

Super

Epoch

Linear Accuracy

1

10

100

1000

1

1.2

1.4

1.6

1.8

Hierarchy

Sample

Original

Super

Epoch

Normalized Linear Accuracy

1

10

100

1000

0.2

0.4

0.8

Hierarchy

Sample

Original

Epoch

NCC Accuracy

1

10

100

1000

1

1.4

1.8

2.6

Hierarchy

Sample

Original

Epoch

Normalized NCC Accuracy

1

10

100

1000

0.5

0.7

0.9

Hierarchy

Sample

Original

Ratio

1

10

100

1000

1

1.2

1.4

Hierarchy

Sample

Original

Normalized Ratio

Epoch

Epoch

B.1 Network Architectures

SSL research primarily focuses on utilizing a single ResNet-50 backbone for experimental purposes. However, given the similarities between SSL and supervised learning, as demonstrated in the paper, it is worth exploring how the choice of backbone architecture affects the clustering of representations with respect to semantic targets. In Figure 7, we present the intermediate layer clustering for different network architectures for extracting the original classes (left) and the superclasses (right). In Figure 7 (top), we display the linear test accuracy of RES-5-50, RES-5-250, and RES-5-1000 networks at different epochs (20, 40, 100, 400, 1000). As can be seen, the network's width has a significant impact on the results for all intermediate layers and at all training epochs.

In Figure 7 (bottom), we present the linear test accuracy of networks RES-5-250, RES-10-250 throughout the training epochs (left) and intermediate layers (right). It is generally seen that in the later epochs of training, the additional layers benefit the linear test performance of the network. However, it is also interesting to note that for the initial layers of the network, the shallow RES-5-250 model gets more clustered representations both for the original classes and for the superclasses. This hints at the fact that deeper networks may need intermediate layer losses to encourage clustering in initial layers [24, 10].

Experiments with Vision Transformers. We extend our experimentation to different backbones to validate the results illustrated in the main paper. A Vision Transformer (ViT) is adopted as per the architecture detailed in [20], employing a 5-layer transformer with 4 heads and a hidden dimension of 384. In Figure 8, we examine the clustering degree produced by various models trained using different SSL algorithms on the Tiny Imagenet dataset. Specifically, we report the NCC train accuracy across intermediate layers for both the VICReg and SimCLR algorithms.

1

2

3

4

5

0.4

0.6

0.8

50

250

1000

Layer

Linear Accuracy

1

2

3

4

5

0.4

0.6

0.8

50

250

1000

Layer

Linear Accuracy

1

2

3

4

5

6

7

8

9

10

0.4

0.6

Depth

Shallow

Deep

Layer

Linear Accuracy

1

2

3

4

5

6

7

8

9

10

0.4

0.6

Depth

Shallow

Deep

Layer

Linear Accuracy

Original classes

Superclasses

16

32

64

128

256

512

1024

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Layer

1

2

3

4

5

Epoch

NCC Accuracy

16

32

64

128

256

512

1024

1

1.5

2

2.5

3

Layer

1

2

3

4

5

Epoch

NCC Accuracy

32

64

128

256

512

1024

0.05

0.06

0.07

0.08

0.09

0.1

0.11

0.12

0.13

Layer

1

2

3

4

5

Epoch

NCC Accuracy

32

64

128

256

512

1024

1

1.2

1.4

1.6

1.8

2

2.2

2.4

Layer

1

2

3

4

5

Epoch

NCC Accuracy

VICReg

VICReg Normalized

SimCLR

SimCLR Normalized

B.2 Experiments with Additional Datasets

CIFAR-10. We run similar experiments using RES-5-250 on the CIFAR-10 dataset. In Figure 10 (top), we report the NCC test accuracy for recovering both the sample level and the class labels. Similar to the results in the main text, the samples progressively cluster around their augmentation class means early in training. However, during the majoring of the training process, the augmented samples tend to cluster with respect to the semantic labels. In Figure 10 (bottom), we report the NCC and linear (from left to right) test accuracies of intermediate layers with respect to the semantic labels.

16

32

64

128

256

512

1024

1

1.2

1.4

1.6

1.8

2

Layer

1

2

3

4

5

Epoch

NCC Accuracy

16

32

64

128

256

512

1024

1

1.5

2

2.5

3

Layer

1

2

3

4

5

Epoch

Test NCC

16

32

64

128

256

512

1024

0.8

1

1.2

1.4

1.6

1.8

Layer

1

2

3

4

5

Epoch

Test NCC

SWaV, Tiny Imagenet

SWaV, Aircrafts

BarlowTwins, Aircrafts

20

100

800

0.5

0.6

0.7

0.8

Hierarchy

Sample

Original

Epoch

NCC Accuracy

20

100

400

800

1

1.1

1.2

Hierarchy

Sample

Original

Epoch

Normalized NCC Accuracy

20

100

800

0.45

0.5

0.55

0.6

0.65

Layer

1

2

3

4

5

Epoch

NCC Accuracy

1

2

3

4

5

0.5

0.6

0.7

Epoch

20

40

100

1000

Layer

Linear Accuracy

As can be seen, the degree of clustering monotonically improves at all layers during the course of training.

FOOD-101. Similar to CIFAR-10, we experiment on the FOOD101 dataset, consisting of 75,750 train samples and 25,250 test samples across 101 food classes (e.g. 'pizza', 'misso soup', 'waffles'), using VICReg and the RES-5-250 architecture. In Figure 13 (left), we plot the NCC test accuracy for both the sample level, and the semantic classes level, and in Figure 13 (right) we plot the NCC test accuracy of the intermediate layers with respect to the sample level labels. The results are similar to the ones shown in Section 4. Additionally, we present the sample level clustering in the intermediate layers. Here, we also see an improvement throughout the layers and the epochs.

Aircrafts. We applied the SWaV and Barlow Twins algorithms with the RES-5-250 architecture to the Aircrafts dataset, which consists of 10,000 images from 100 aircraft types. In Figure 9 we show the NCC train accuracy at the various intermediate layers.

Tiny Imagenet. We also experiment on more varied datasets, like the Tiny ImageNet dataset, consisting of 100,000 train samples and 10,000 test samples across 200 classes (e.g. 'nematode', 'harmonica', 'syringe'), using VICReg and the RES-10-250 architecture. In Figure 12 we plot the NCC test accuracy for both the sample level, and the semantic classes level, both unormalized and normalized resp. In Figure 9 we show the NCC train accuracy using the SWav SSL algorithm. The results are similar to the ones shown in the clustering section.

Imagenet. We extend our experimentation to the more complex ImageNet [19] dataset. In Figure 11, we show the linear test accuracy and NCC train accuracy of both SwA V and SimCLR on the ImageNet dataset, along training epochs, with respect to the semantic labels on the final layer of a ResNet-50. Similarly to the supervised setting, as shown in section Section 4, both the linear accuracy and the NCC accuracy dramatically improves during training.

B.3 Hyperparameters

In Figure 14, we present the training losses and linear accuracies for three different hyperparameter selections trained using VICReg and a RES-5-250 network. Specifically, we vary the µ (variance regularization) parameter and display the losses and linear accuracy for µ = 5 , 25 , 100 , from left to

0

100

200

0.01

0.1

0.8

SwAV NCC

SimCLR NCC

SwAV Linear

SimCLR Linear

Epochs

Linear and NCC Accuracy

16

32

64

128

256

512

1024

0.2

0.4

0.8

Hierarchy

Sample

Original

NCC Accuracy

16

32

64

128

256

512

1024

1

1.4

1.8

Hierarchy

Sample

Original

Normalized NCC Accuracy

Epoch

Epoch

right, respectively. Both loss terms are normalized by µ . In all cases, we used ν = 1 and λ = 25 by default.

Interestingly, the configuration with µ = 5 exhibits significantly lower performance compared to the default setting ( µ = 25 ). However, the configuration with µ = 100 achieves performance comparable to the default, despite the network having a considerably higher invariance loss term. This observation further supports our claim that the regularization term plays a crucial role in learning semantic features.

B.4 Experiments with Different SSL Algorithms

In our main text, we primarily focus on the VICReg SSL algorithm. However, to ensure the robustness of our findings across different algorithms, we also trained a network using the SimCLR algorithm [14] with our default RES-5-250 architecture. In Figure 15, we present several experiments that compare the network trained with VICReg to the one trained with SimCLR.

Firstly, Figure 15 (top) illustrates the linear test accuracies of intermediate layers for both algorithms throughout the training epochs with respect to the original classes (left) and the superclasses (right). The darkness of the lines indicates the progression of epochs, with lighter shades representing later epochs. Despite the significant differences between the algorithms, particularly in SimCLR's contrastive nature, we observe similar clustering behavior across different layers and epochs with VICReg consistently achieving better performance. This finding demonstrates that various SSL algorithms yield comparable clustering characteristics.

20

100

400

800

1

1.1

1.2

Hierarchy

Sample

Original

Epoch

Normalized NCC Accuracy

20

100

800

0.6

0.7

0.8

0.9

Layer

1

2

3

4

5

Epoch

Sample NCC Accuracy

20

100

1000

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.45

Inv.

Acc.

Epoch

Loss

Linear Accuracy

20

100

1000

0.1

0.2

0.3

0.4

0.5

0.6

0.45

0.5

0.55

Inv.

Acc.

Epoch

Loss

Linear Accuracy

20

100

1000

0.15

0.2

0.25

0.3

0.45

0.5

Inv.

Acc.

Epoch

Loss

Linear Accuracy

µ

= 5

µ

= 25

µ

= 100

Furthermore, in addition to clustering, we aim to assess the similarity of representations between the two algorithms. In Figure 15 (bottom left), we present the correspondence of the representations with random targets, similar to the approach used in Figure 4 (left), across different training epochs. The darkness of the lines represents the progression of epochs, with lighter shades indicating later epochs. Additionally, in Figure 15 (bottom right), we depict the same correspondence for different intermediate layers, similar to Figure 4 (middle), at the end of the SSL train. Across various layers and epochs, we observe similar behavior between the algorithms. However, it is worth noting that VICReg demonstrates a slightly higher performance in extracting the random targets.

B.5 Random Target Function Architectures

Neural network architectures possess inherent biases that aid in modeling complex functions. Convolutional networks, for example, implicitly encode biases through their structure, which includes locality and translation invariance. As a result, these networks tend to prioritize local patterns over global features. On the other hand, the ViT architecture takes a different approach by treating images as sequences of patches and utilizing self-attention mechanisms. This design introduces a bias that enables long-range dependencies and global context.

An interesting question arises regarding how the choice of the SSL backbone architecture influences the learned representations. To explore this, we examine the alignment of representations with different types of random targets. Specifically, we conduct experiments similar to those illustrated in Figure 4 (left and middle) but employ different random targets based on the ViT architecture [52].

In Figure 16, we monitor the linear test accuracy of VICReg-trained RES-5-250 in recovering both the ResNet-18 targets and the ViT targets. Figure 16 (left) presents the linear test accuracy at different training epochs (20, 40, 100, 400, 1000) from dark to light resp. and Figure 16 (right) shows the linear test accuracy at different intermediate layers (1-5) from dark to light resp. at the end of SSL

1

2

3

4

5

0.4

0.6

VICReg

SimCLR

Layer

Linear Accuracy

1

2

3

4

5

0.4

0.6

VICReg

SimCLR

Layer

Linear Accuracy

0.99

0.95

0.75

0

0.35

0.45

VICReg

SimCLR

Randomness Level

Linear Accuracy

0.99

0.95

0.75

0

0.25

0.35

0.45

VICReg

SimCLR

Randomness Level

Linear Accuracy

0.99

0.95

0.75

0

0.25

0.35

0.45

Target Generator

ResNet-18

ViT

Randomness Level

Linear Accuracy

0.99

0.95

0.75

0

0.25

0.35

0.45

0.55

Target Generator

ResNet-18

ViT

Randomness Level

Linear Accuracy

training. As can be seen, the results with ViT targets are consistent with the claims in Section 5. In other words, the SSL representations exhibit improved alignment during training epochs and across intermediate layers.

However, it is evident that the linear accuracy with respect to the ResNet-18 targets consistently outperforms the accuracy with respect to ViT targets at all training stages, except for the end of the supervised training phase ('0' randomness). This indicates that during the training process, the SSL-trained model acquires representations that are more aligned with those achieved by training convolutional network architectures. This behavior can be attributed to the implicit biases introduced by the backbone, which share similarities with the ones present in the ResNet-18 architecture.

C Limitations

While our study has yielded significant findings, it is not without its limitations. Firstly, our experiments were conducted on select datasets, which inherently possess their unique characteristics and biases. As such, applying our analysis to different datasets may potentially give different results. This reflects the inherent variability and diversity of real-world data and the possible influence of dataset-specific factors on our findings. Secondly, our analysis primarily focuses on the vision domain. While we believe that our findings have substantial implications in this area, the generalizability of our findings to other domains, such as natural language processing or audio processing, remains unverified.

D Broader Impact

In this paper, our primary objective is to characterize the different properties and aspects of SSL, with the aim of deepening our understanding of the learned representations. SSL has demonstrated its versatility in a wide range of practical downstream applications, including image classification, image segmentation, and various language tasks. While our main focus lies in unraveling the underlying mechanisms of SSL, the insights gained from this research hold the potential to enhance SSL algorithms. Consequently, these improved algorithms could have a significant impact on a diverse set of applications. However, it is crucial to acknowledge the potential risks associated with the misuse of these technologies.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a study of the structure of SSL representations with respect to sample labels (image identity), to class labels and to super-class labels. To that end, an analysis of the clustering properties of the embedding space according to such labels is performed. Additionally, probing linear layers are trained on top of the SSL representations to assess discriminative performance for the different label categories.\n\nVicReg on CIFAR-100 is evaluated in the experimental results section. The main results show that representations indeed cluster according to multiple hierarchies of labels (sample, class, super-class) and that such clustering is mostly related to the regularization term in VicReg.\n\nSTRENGTHS:\n### Clarity\n\n* The language used is clear.\n* The paper organization seems natural and easy to follow.\n\n### Significance\n\n* Investigating the intrinsic properties of SSL representations is of interest to the community. However, a ore throrough analysis would be needed (see Weaknesses).\n\nWEAKNESSES:\n### Originality\n\n* The analyses provided in this work have already been (at least partially) explored. This reduces the originality of the work. See some references hereafter, not being an exhaustive list:\n  * To the best of my knowledge, the fact that SSL representations cluster according to class labels has already been explored in the past, see (Vasequi, 2021) for example. Although such work focused on label noise resiliency, they show clear evidence of clustering in the t-SNE plots provided.\n    - _Zahra Vaseqi, Ibtihel Amara and Samrudhdhi Rangrej, Label Noise Resiliency with Self-supervised Representations, NeurIPS 2021_.\n\n  * (Grigg, 2021) also showed that linear probing performance increases with depth, in that case with SimCLR representations.\n    - _Tom George Grigg, Dan Busbridge, Jason Ramapuram and Russ Webb, Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?, NeurIPS 2021_.\n\n### Quality\n\n* The experimental results in the main paper only cover 1 SSL algorithm (VicReg) and 1 dataset (CIFAR-100). Such limited evaluation setup does not reflect the main claim of the paper (reverse-engineering SSL), being rather a study of VicReg properties on a specific dataset. It is yet to be seen whether the results obtained with VicReg transfer to:\n  * The myriad of SSL algorithms previously published (i.e. SimCLR, SimSiam, BYOL, SwAV, Barlow Twins, etc.), some of them much more widely used in the community than VicReg.\n  * Different datasets with different properties, specially different inter-class similarities which can strongly affect clustering. Such analysis would give the reader intuition about how different data sources can be expected to work in an SSL framework.\n\n### Clarity\n\n* The fact that the authors refer to SSL-wide conclusions while using conclusions on VicReg+CIFAR100 induces the reader to misunderstanding. I would suggest the authors to either (i) provide a thorough analysis over representative SSL methods and datasets or (ii) refer to VicReg properties instead of general SSL.\n\nQUESTIONS:\n* The claim _\"Contrary to popular belief, the invariance loss does not significantly improve during the training process\"_ (L257) is not rigorously proven. Have the authors verified this property on a suite of SSL methods, to be able to state it as formulated in the manuscript? I would argue that this is indeed a property of VicReg in the setting of the paper. However, I am doubtful about this finding being a generic SSL property. For that, a much larger study of SSL methods, with different loss term weighings, etc. would be required.\n\n* I am curious about how much the super-class clustering is different from a random super-class assignment (in groups of 5 classes, similarly to the experimental setup already in the paper). Is this related to the label-noise experiment, or would this random grouping yield different results?\n\n-----\nMinor comments:\n* Fig 2 right is not referenced.\n\n-----\nMain reasons behind score:\n\n* Lack of comparison with other SSL papers or datasets. The representation properties analyzed are only based on VicReg, which strongly reduces the significance of this work and contradicts the main contributions (general SSL).\n* Some observations explored in this work are not original, and had been explored in previous works.\n\n-----\nUpdate after rebuttal:\n\nThe authors did a great work in the rebuttal, largely improving the experimental sections and thus addressing most of my concerns. In order to avoid ambiguity, I raise my score to 5.\n\nThe authors also included ImageNet results that support the paper claims, making the paper much stronger as large scale data is also part of the evaluation. Having 4+ methods on several datasets makes the claims of the paper more sound and provide interesting insights to the community. Therefore, after the rebuttal conversations, I raise my score to 6.\n\nLIMITATIONS:\nNo concerns.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper provides an empirical study on self-supervised learning (SSL) in terms of clustering. Specifically, the authors demonstrate 1) clustering process of SSL methods, 2) semantic learning of SSL methods based on random labels generated by various models, and 3) class hierarchies and intermediate layers of SSL methods.

STRENGTHS:
- The presentation of this paper is really good: overally well-written, and easy-to-follow.

- The message of this paper is clear, sound, and consistent to me: self-supervised learning works as a clustering algorithm of feature space, and various empirical observations support it well.

- The experiments are conducted in depth with convincing (but seems narrow) setups.

WEAKNESSES:
The experiments seems not extensive in terms of dataset and self-supervised algorithms as follows:

- Many self-supervised learning methods are evaluated using ImageNet dataset, and this paper lacks experimental results on ImageNet dataset. This is not critical for accepting this paper, but additional experimental results on ImageNet dataset would improve the message of this paper.

- This paper mainly focuses on one self-supervised learning, VICReg, which may not represent all self-supervised learning methods. This is also not that critical.

QUESTIONS:
I understand that one of "contrastive" SSL methods works as clustering algorithm, but wonder whether it can be applied to "cluestering-based" SSL (e.g., [SWaV](https://arxiv.org/pdf/2006.09882.pdf)) and "auto-regressive" SSL (e.g., [MAE](https://arxiv.org/pdf/2111.06377.pdf)). I believe that this is not in the scope of this paper, but it would be better to discuss it for the future research direction.

LIMITATIONS:
The authors provide limitations of this paper in Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on an empirical analysis of how self-supervised learning (SSL) works. While training artificial neural networks for classification under supervision has been extensively explored, the authors propose to shed more light on SSL process using VicReg + ResNet-L-H and a series of experiments on the CIFAR-100 dataset. This work presents that during the training process, we can observe improvements in clustering for the original classes and in super-classes, that presents some emerging properties of hierarchical clustering during SSL training. What is important, this happens while the expected instance-level clustering stops improving. Although the authors do not provide an answer as to why this is the case, they attribute this phenomenon to the regularization term in the SSL loss function (what is presented in one of the experiments). In work we can find few more analysis.\n\nSTRENGTHS:\nS1. This paper tackles an important problem of understanding the representations built by SSL and asks important questions, e.g. how the clustering evolves during the training on multiple levels (instance, class semantic, high level semantic) and to what this phenomenon can be attributed. \n\nS2. The paper shows how the SSL methods gain clustering properties across training procedure (Fig. 2) and across layers (Fig. 5). While this property was intuitively known, this paper is the first to measure this property.\n\nWEAKNESSES:\nW1. The paper draws conclusions based on experiments that are made using only one SSL method, one dataset (CIFAR100 - which is small scale) and one type of neural network architecture (RES-L-H which is not widely used).\n\nW2. The contributions regarding Randomness impact do not sound solid. The randomness experiment is not clear (see questions sections) and it is hard to know what it tries to show. By this experiment, the authors want to show that “We argue that SSL-trained representation functions are highly correlated with semantic classes.”. Doesn't the S1 already show it? \nSimilarly, does not the fact that the classes are clustered would not imply that superclasses are clustered as well? I think it is a nice thing to show, but currently it is exposed as one of the main paper contributions. Presenting that some (super) emerges before the original - would be more insightful \n\nW3. This work is a series of empirical analysis, with some insightful information. However, the reading flow could be improved for that.\n\nQUESTIONS:\n1. One of the insightful outcome of this work is that the clustering property can be attributed to regularization term. But this was not analyzed. Why not increase the term for that component and check the behavior?  \n\n1. Looking at the figure 3 (right). The SSL method seems to work significantly worse than the SL method (41 vs 61% accuracy). This looks like a massive difference, maybe there is something wrong with the training run?\n\n1. Can we see figure 2 without normalized accuracies? I think the default option would be to show absolute values, so I wonder what we gain by showing normalized accuracies. One culprit is that in your “Sample” experiment you have 500 classes and in “original” 100 classes, but I would still like to see absolute values.\n\n1. The randomness experiment is not clear to me. The paper says: „We train a neural network classifier on the same dataset for classification and use its target predictions at different epochs as targets with varying degrees of randomness”, which I found hard to understand. Are the labels simply random at training time as in [1]?\n \nAlso: “We normalize the degree of randomness between 0 (non-random, end of training) and 1 (random, initialization)” - how do you do this? Does 0.9 degree of randomness correspond to 0.9 fraction of total training epochs? Or does it correspond to 0.9 of final performance of the network?\nFinally, looking at the fig. 4. Is there something surprising in the plot? (Increased randomness = reduced accuracy).\n\n1. I think there is something wrong with the following sentence: L.210: “Notably, the SSL-trained representation exhibits neural collapse at a sample level (with an NCC train accuracy near 1.0), yet the clustering with respect to the semantic classes is also significant (≈ 0.41 on the original targets).“ Maybe it is about Fig. 3 (right) but the NCC train accuracy for the SSL method is above 0.4 (not “near 1.0”)? \nAlso, it is not clear what is the message that this figure shows as i) SL and SSL have totally different accuracies so they are not comparable, ii) the fact that we get worse results on validation is also expected.\n\n1. “As shown in Figure 5, we observe that the clustering and separation ability of each layer improves as we move deeper into the network” - it is not true. For classification of original semantic classes the linear accuracy actually drops at the last layer (or several last layers in case of not fully trained networks).\n\n1. Why decision to show only one dataset and type of architecture in the main paper? I know that in the appendix there is a food dataset and different variation of resnet-l-h. However, why not going for imagenet (or subset)? \n\n1. Have you considered checking methods like SupCon [2] for a comparison? \n\n[1] Zhang, Chiyuan, et al. \"Understanding deep learning (still) requires rethinking generalization.\" Communications of the ACM 64.3 (2021): 107-115.\n[2] Khosla, Prannay, et al. \"Supervised contrastive learning.\" Advances in neural information processing systems 33 (2020): 18661-18673.\n\nLIMITATIONS:\nLimitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo reveal the underlying mechanism of self-supervised learning methods, this paper mainly investigate their clustering characteristics. Empirical analyses show that these methods cluster samples based on their semantic classes. Additionally, the use of regularization terms, introduced to prevent representation collapse, proves beneficial even after the clustering process.\n\nSTRENGTHS:\n1. Understanding the mechanism of self-supervised learning is one of the most important goals acknowledged by the machine learning community. \n2. This paper provides not only qualitative but also quantitative results. \n3. This paper is well-written and well-organized. Notably, the introduction section effectively summarizes the key takeaways\n\nWEAKNESSES:\n1. While the findings are interesting, I'm not fully convinced of their importance. A comparison with other pre-training methods or the proposal of a novel self-supervised learning method based on the conclusions drawn would emphasize the significance of the paper's findings.\n2. The scope of the setup is limited:\n    - The experiments solely utilize RES-L-H architectures.\n    - The primary downstream task focuses on image classification using CIFAR-100. This not only neglects dense prediction tasks but also relies on a small dataset compared with ImageNet.\n    - Many experiments mainly focuses on linear probing accuracy rather than fine-tuning performance. However, a high linear probing accuracy does not necessarily indicate a high fine-tuning accuracy [a]. In practice, fine-tuning accuracy is sometimes more crucial than linear probing accuracy.\n    - The title and the term “self-supervised learning” in this paper are potentially misleading, since this paper mainly focuses on contrastive and non-contrastive learning, rather than on masked image modeling. Moreover, it seems that some claims are only valid for contrastive or non-contrastive learning.\n\n[a] Park, Namuk, et al. \"What Do Self-Supervised Vision Transformers Learn?.\" ICLR (2023).\n\nQUESTIONS:\nOne of my major concerns is about the significance of the findings, since the paper primarily focuses on describing the properties of contrastive and non-contrastive learning methods.\n\nLIMITATIONS:\nI cannot find any ethical concerns. For technical limitations, see the Weaknesses section above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}