The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning
Authors: Romain Cosentino
Sarath Shekkizhar
Mahdi Soltanolkotabi
Salman Avestimehr
Antonio Ortega
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised learning
Transfer Learning
Graphs
Geometry
Embedding
Computer Vision

Abstract: Self-supervised learning~(SSL) has emerged as a desirable paradigm in computer vision due to the inability of supervised models to learn representations that can generalize in domains with limited labels. The recent popularity of SSL has led to the development of several models that make use of diverse training strategies, architectures, and data augmentation policies with no existing unified framework to study or assess their effectiveness in transfer learning.
We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each. Unlike existing approaches that consider mathematical approximations of the parameters, individual components, or optimization landscape, our work aims to explore the geometric properties of the representation manifolds learned by SSL models.
Our proposed manifold graph metrics~(MGMs) 
provide insights into the geometric similarities and differences between available SSL models, their invariances with respect to specific augmentations, and their performances on transfer learning tasks. Our key findings are two fold: $(i)$ contrary to popular belief, the geometry of SSL models is not tied to its training paradigm (contrastive, non-contrastive, and cluster-based); $(ii)$ we can predict the transfer learning capability for a specific model based on the geometric properties of its semantic and augmentation manifolds.

Under review as a conference paper at ICLR 2023
THE GEOMETRY OF SELF-SUPERVISED LEARNING MODELS AND ITS IMPACT ON TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision due to the inability of supervised models to learn representations that can generalize in domains with limited labels. The recent popularity of SSL has led to the development of several models that make use of diverse training strategies, architectures, and data augmentation policies with no existing unified framework to study or assess their effectiveness in transfer learning. We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each. Unlike existing approaches that consider mathe- matical approximations of the parameters, individual components, or optimization landscape, work aims explore the geometric properties of the representation manifolds learned by SSL models. Our proposed manifold graph metrics (MGMs) provide insights into the geometric similarities and differences between available SSL models, their invariances with respect to specific augmentations, and their performances on transfer learning tasks. Our key findings are two fold: (i) contrary to popular belief, the geometry of SSL models is not tied to its training paradigm (contrastive, non-contrastive, and cluster-based); (is) we can predict the transfer learning capability for a specific model based on the geometric properties of its semantic and augmentation manifolds.
INTRODUCTION
Self-supervised learning (SSL) for computer vision applications has empowered deep neural net- works (DNNs) to learn meaningful representations of images from unlabeled data (223 5 6
5 10 IT. These methods learn a feature space embedding that invariant to data augmentations (e.g., cropping, translation, color jitter) by maximizing the agreement between representations from different augmentations of the same image. The resulting models are then used as general-purpose feature extractors and have been shown to achieve better performance for transfer learning as com- pared to features obtained from a supervised model 12). Broadly, SSL models can be categorized
into contrastive (13 2), non-contrastive [14 15). prototype/clustering (6 17). There exist multiple differences in the resulting models, even among models belonging to the same category. For example, popular SSL models available as pre-trained networks (18) can differ in terms of training parameters (loss function, optimizer, learning rate), architecture (DNN backbone, projection head, momentum encoder), model initialization (weights, batch-normalization parameters, learning rate schedule), etc.
Recently, researchers have focused on developing an understanding of specific components of SSL models by studying the loss function used to train the models and its impact on the learned representations. For instance, (19) analyzes contrastive loss functions and the dimensional collapse problem. (20) also analyzes contrastive losses and describes the effect of augmentation strength as well as the importance of non-linear projection head. 2I quantifies the importance of data augmentations in a contrastive SSL model via distance-basee approach.
(22) explores a graph formulation of contrastive loss functions with generalization guarantees on the representations learned. In (23), the importance of the temperature parameter used in the SSL loss function and its impact on learning is examined.
(24) performs a spectral analysis of DNN's mapping induced by non-contrastive loss and the momentum encoder approach. However, these studies are unable to provide a unified analysis of the myriad of existing SSL models. Besides, these theoretical approaches only provide insights into the embedding obtained after the projection head, while in practice it is the mapping provided by the encoder that is actually used for transfer learning.

---

Under review as a conference paper at ICLR 2023
Closer to our approach, the general transfer performance of an SSL model has been predicted based on the performance it can achieve on the ImageNet dataset (25). This idea was shown to be effective for transfer datasets that are similar to ImageNet, but it cannot be generalized to all transfer learning problems (12). In fact, if ImageNet performance were highly correlated to general transfer learning performance, then it is unclear why SSL models would be needed, one could simply use supervised training with ImageNet to obtain image representations for transfer learning. Furthermore, existing empirical evaluations such as (25) only provide a somewhat coarse and partial understanding of SSL models. For example, they do not provide insights into how the level of invariance to specific augmentation in an SSL model relates to its performance on given downstream task. Since has been observed that invariance to some augmentations can be beneficial in some cases and harmful in others (26), our goal is to develop a more direct and quantitative understanding of augmentation invariance and how this invariance determines performance.
To achieve this goal, we propose a geometric perspective to understand SSL models and their transfer capabilities. Our approach analyzes the manifold properties of SSL models by using data-driven graph-based method to characterize the geometry of data and their augmentations, as illustrated on the left of Figure [ Specifically, we develop set of manifold graph metrics (MGMs) to quantify the geometric properties of existing SSL models. This allows us to provide insights about the similarities and differences between models (Figure [ right) and link their ability to transfer to specific characteristies of the target task. Because our approach can be applied directly to sample data points and their augmentations, has several important advantages. First, is agnostic to specific training procedures, architecture, and loss functions; Second, it can be applied to the data embeddings obtained at any layer of the SSL model, thus alleviating the challenge induced by the presence of projection heads; Third, it enables us to compare different feature representations of the same data point, even these representations have different dimensions.
We are interested in using our approach to answer the following questions about SSL models: (i) What are the geometric differences between the feature spaces of various SSL models?
(1i) What geometrie properties allow for better transfer learning capability for a specific task?
For our transfer study, we perform our MGM evaluation on 14 SSL models under 5 augmentation setting and show their impact in 8 downstream tasks comprising of 18 datasets in total.
Our contributions are summarized as follows:
We develop quantitative tools (i.e., MGMs) capable of capturing important geometrical aspects of SSL representations, such as the degree of aquivariance-invarianee the curvature, and the intrinsic dimensionality, Sec.
We leverage the proposed MGMs to explore the geometric differences and similarities between SSL models. As illustrated in the right part of Figure T we show that SSL models can be clustered using these geometric properties into three main groups that are not entirely aligned with the paradigm upon which they were trained, Sec.
We analyze the geometric differences between a Vision Transformer (ViT) and a convolutional network (ResNet). We show that while ResNet is biased towards a collapsed representation at initialization, ViTs are not. This initialization bias leads different geometrical behavior (atrraction/repulsion of representations) between the two architectures when training under an SSL regime detailed in Sec. 4
We demonstrate that observed MGMs are a strong indicator of the transfer learning capabilities of SSL models for most downstream tasks. Therefore showing that specifie geometrical properties crucial for a given transfer learning task, Sec.
2 BACKGROUND
Self-supervised Learning: SSL models are generated by producing multiple versions of the same image, via data augmentation, and training a DNN such that their embeddings coincide. Many SSL models are obtained with architectures that cascade two networks, the backbone encoder, from which the representation to be used for a downstream task is extracted, and the projection head, from which the output is fed into the SSL loss function. The main risk in such approach is the so-called the feature collapse phenomenon (19 24] (7)), where the learned representations are invariant to input samples that belong to different manifolds. To reduce the risk of feature collapse, multiple SSL
2

---

Under review as a conference paper at ICLR 2023
Data Augmentations
SSL Model
Encoder Space
Clustering SSL Models
2:0
1.75
Tangeat Plane
(Eneboldul) Manifold
1.50
0.73
0.50
Sample
Manifold Graph Metrics
Dep
Figure 1: (Left) Approach toward the analysis of SSL models For each model, we use as input of the DNN the validation set of ImageNet as well as their augmented versions. The output of the backbone encoder is used to quantify the properties of the manifold induced by the SSL algorithm. Specifically, we develop Manifold Graph Metrics (Sec. 3 that capture manifold properties known to be crucial for transfer learning. The MGMs allow us to capture the specificity of each SSL model (Sec. 4 and to characterize their transfer learning capability (Sec. (Right) We provide the dendrogram of the SSL models considered in this paper based on the manifold graph metrics propose. Although the underlying the manifolds induced by the SSL. algorithms can be categorized into three types. An important observation is that the resulting clusters are not necessarily aligned with the different classes of SSL algorithms. This result shows that although some training procedures appear to be more similar, one is required to provide a deeper analysis to understand in what aspect SSL models differ.
loss functions, and SSL paradigms are different,
algorithms have been proposed. Contrastive SSL. methods (e-g., simCLR-vl (3), simCLR-v2 2 MoCo-vl (28), MoCo-v2 (29), Infomin (I), PIRL (30), InsDis (ID) make use of negative pairs to force the embedding of data points in an unaugmented pairs to be pushed away during training. Non-contrastive SSL methods (BYOL 4 DINO 32 utilize teacher-student based approach where the weights of the students are the exponential moving average the teacher's 's weights. Protopype/clustering SSL methods (SeLA-v2 (33), DeepCluster-v2 (17), SwAV (17), PCL-v1 and PCL-v2 (16)) enforce consistency obtained from different transformations of the same image by comparing cluster assignments without negative pair comparisons.
We will analyze the geometrical differences of these SSL models and the impact of geometry on the transfer learning capability. To perform this analysis, we will use pre-trained SSL models based on ResNet50(1 Ix) backbone encoder architecture as well as supervised model available with the Pytorch library (34). We will also compare the geometrical differences between ResNet50 and ViT architecture both at initialization and after SSL training using DINO loss (32).
Graphs: Graph and local neighborhood methods have played a significant role machine learning tasks such as manifold learning (35), semi-supervised learning 37), and, more recently, graph- based analysis of deep learning 38 39 40). Typically their use in this context is motivated by their ability to represent data with irregular positions in space (æ, ) rather than directly modeling the distribution the data P(x). This type of DNN analysis starts by constructing good neighborhood representation (41 422. which is also crucial first step in our framework. The most common approaches to defining a neighhorhood are k-nearest neighbor (kNN) and 6-neighborhood. However, these approaches select points neighborhood based on only distance the query point, and do not consider their relative positions, while also relying on ad hoc procedures to select parameter values (e.g.. i or c). For this reason, we make use of non-negative kernel regression (NNK) (43) to define neighborhoods and graphs for our manifold analysis. Unlike kNN, which can be seen as thresholding approximation, NNK can be interpreted as form of basis pursuit (44), which leads to better neighborhood construction with robust local estimation performance in several machine learning tasks (45) (46). Of particular importance for our proposed SSL analysis framework the fact that NNK neighborhood have geometrie interpretation. While in kNN points x, and æg included in the neighborhood of data point 2,, denoted by N(x;) based their distances to æ; i.e., d(æ;, æz) and d(æ;, æx), in NNK this decision is made by also taking into account d(x,, 2g),, ), so that zy and Dg are both included in N(x;) only if they are not geometrically redundant. As result, an NNK neighborhood can be described convex polytope
{æ....... Li., }. solely
3

---

Under review as a conference paper at ICLR 2023
approximation of z,, denoted P(x:) the size and shape of which is determined by the local geometry of the data available around x; (43). This geometric property is particularly important for data that is not uniformly sampled and lies lower dimensional manifold in high dimensional vector space, which is typical for feature embeddings in DNN models. Note that NNK uses KNN as an initialization step, after which it has only modest additional runtime requirement 43). Thus,
scale our analysis to large datasets by speeding up the initialization using computational tools developed for KNN 47 48 NNK requires kernels with a range in [0, 1]. In this work, we use the cosine kernel since the encoder representations are obtained after ReLU and so the kernel satisfies the requirement.
MANIFOLD GRAPH METRICS
We use the NNK polytope P(x:)) to characterize the neigh- borhood of and the local and global geometry of the data manifolds. Our approach, summarized in Figure [ is motivated by the observation that while the DNNs used in SSL involve complex non-linear mappings, the induced transformations and the structure of the representation space can be inferred from the data samples by observ- ing their relative positions in that space. In particular, we propose a set of Manifold Graph Metrics (MGMs) that provide intuition and quantitative data characterize the geometry of an SSL model, Figure 2 We focus on three manifold properties that we consider important for the understanding of SSL-based feature embeddings and their transfer capabilities; (i) the level of invariance (or equivariance) with respect to a given transformation or augmentation, (ii) the curvature of the manifold, (iit) the local intrinsic dimension of the manifold.
P
0
P
Figure 2: MGMs in feature space are based on NNK polytopes. We dis- play data samples (red and blue dots) and two NNK polytopes (P(z:), P(xz)) in the encoder space that approximate the underlying manifold of the SSL model (gray surface). Our proposed MGMs capture invariance (polytope di- ameter), manifold curvature (angle be- tween neighboring polytopes), and local intrinsic dimension (number of vertices in polytope) of the output manifold for given SSL model.
serrriace-EEuivariian Metric: Given input images (xx } we apply the NNK neighborhood definition to the feature space representations (ft))) of the images to obtain N convex polytopes. We define the diameter of an NNK polytope as the maximum distance between the nodes forming the polytope, i.e.,
Diam(P(f(z:))
max (.EN((((.)
- f(a))), (1)
where f corresponds to the Iz-normalized feature embed- ding of a given input. These polytope diameters take values in the range [0, 2] and provide quantitative mea- sure of how much the input samples have been contracted or dilated by the DNN backbone of the SSL model. Thus, a constant or collapsed mapping, where multiple æ;; are mapped to the same f. would lead to a degenerate polytope with diameter equal to zero. Instead, a diameter of 2 corre- sponds to mapping where the neighbors are maximally scattered. Now, by considering as input only the augmented versions of an image, the diameter of the NNK polytope captures the level of invariance- equivvariiiveveeeeeeee of the DNN with respect to this specific augmentation. Alternatively, if we constrain ourselves to input images that have the same class label, the polytope diameter indicates the level of invariance-equivariinn In this setting, lower value of diameter corresponds to invariance of the DNN mapping to that transformation while higher values corresponding to equivariance.
of the representation to samples that belong to that specific class.
We study the curvature of a manifold by comparing the orientations of NNK polytopes corresponding to neighboring input samples. That is, given two samples æ; and x; that are NNK neighbors, and their respective polytopes Pff (x;)), P(f (f(()))) we evaluate the angle between the subspace spanned by the neighbors (f N (f(f))
Curvature Metric:
that make up the polytopes. Concretely,
4

---

Under review as a conference paper at ICLR 2023
use the concept of affinity between subspaces (49) to define the affinity between two polytopes as a quantity in the interval [0, 1] given by
(O1) + +... + cos? cos?(Ongn))
where Og. h 1..... Ri7ty are the principal angles between the two subspaces spanned by the vectors ((f(z))) and (f((x))) containing all the NNK neighbors of æj and æ;, respectively. Intuitively, this metric is equal to zero when the subspaces are orthogonal (polytopes oriented in perpendicular directions), while value of one corresponds to zero curvature or subspace alignment.
Intrinsic Dimension Metric: The local intrinsic dimension of a manifold can be estimated as the
number of neighbors selected by NNK. was shown in (43) 50), that the number neighbors per polytope, i.e., 72, correlates with the local dimension of the manifold around data point i. This observation is consistent with geometric intuitions; the number of NNK neighbors is decided based on the availability of data that span orthogonal directions, i.e., the local subspace of manifold:
(m:)) = Card(
where Card denotes the cardinal operator.
Experimental settings: For all SSL models, we analyze their equivariance-invaiianee, subspace curvature, and intrinsic dimension in a set of controlled and interpretable experiments. While in this work, we consider as inputs the validation set of ImageNet (51), the framework is applicable to any dataset (with or without labels). Our experimental setup is follows: for each input sample, (i) select an augmentation setting, (ii) sample T = 50 augmentations of the image, (iit) compute the similarity graph using NNK neighborhoods, and (iv) extract the proposed MGMs (Sec B). We limit ourselves to 5 augmentation types: (i) Semantic augmentations (Sem.). where we consider all the samples belonging to each class as augmented versions of each other in the semantic direction of the manifold, (ti) Augmentations (Augs.) which corresponds to the sequential application of various augmentations used during most SSL training process (random horizontal flip, colorjitter, random grayscale, Gaussian blur, and random cropping). (iii) Crop and (ir) Colorjitter (Colorjū.. specific augmentations that are part of the augmentation policies used to train the SSL., (v) Rotate, an augmentation that was not used for training SSL but is considered important for some transfer tasks.
Therefore, for each sample and for each MGM we obtain value (local manifold analysis), while the collection of these per-sample MGMs for the entire validation set gives distribution (such as the one displayed in Figure 3). order to extract differences between SSL models and highlight which geometric properties favor specific transfer learning tasks, extract two statistics from these distributions of MGMs (global manifold analysis): the mean (denoted by the metric name) and the spread (referred to metric spread). In total, for each SSL. model, obtain 26 geometric features, namely, {Sem., Augs., Crop, Colorjit., Rotate) x Equivariance, Equivariance spread, Affinity, Affinity spread, Nb. of neighbors J. as well the affinity and spread between Sem.-Augs, namely, Sem.-Augs. Affinity, Sem. -ugs Affinity spread J. Additional details about each metric and their variability across all models is given in Appendix
In Sec. 5 we evaluate the capability of the MGMs to characterize the transfer learning capability of SSL models. While we highlight here the overall setting of the transfer learning task, details regarding the datasets can be found in Appendix [ and regarding the transfer learning training settings in (12): FewShotKorablith and ManyShotl. Linear correspond to few/many-shot classification using SSL features extracted on 11 image classification datasets (25); ManyShooFFinetun is related to the classification performance as the previous setting, but with entire network along with the feature extractor updated; FewSShrCDFSL corresponds to few-shot transfer performance in cross-domain image datasets such as CropDiseases, EuroSAT, ISIC, and ChestX datasets (52):
DetectionFinetune and DetectionFrozen refer object detection task evaluated on PASCAL VOC dataset (53); Den corresponds to dense surface normal estimation evaluated on NYUv2 (54);
and DenseSeg refers to dense segmentation task evaluated ADE20K dataset (55).
5

---

Under review as a conference paper at ICLR 2023
2nd Princigel
2nd Princip. Comp.
Aam.
Rotate Squie.
Sem. Expat.
Crep Equlv.
Equiv. Spread
Equit. Spread
Auga Equly.
Spread
(0.77)
(0.52)
10.38)
(0.09)
(0.08)
[0.58]
[6.21]
0.2
Instle
0.22
0.53
0:36
McCo-vt
0.21
0.55
1.34
0.03
0.18
PIRL
0.22
0.53
1.22
0.45
0.17
0.08
0.17
0.0
BYOL
0.15
0.65
1.12
0.65
0.13
0.05
0.32
DC-22
0.16
0.41
0.06
0.10
0.37
0.48
0.98
0.11
SimCUR-v1
0.21
0.17
0.52
0.29
0.07
0.16
0.20
SimCLR-v2
PCL-1
0.50
0.75
1.25
0.51
0.23
0.17
0.27
PCL-22
0.17
0.62
0.23
-0.4
-0.5
0.0
0.5
1.0
McCow2
0.19
0.33
1.06
0.10
1st Princip. Comp.
InfoMin
0.20
0.51
1.02
0.45
0.17
0.20
0.23
Supervised
Table 1: (Left) Projection of SSL models MGMs onto principal components. We observe three distinct clusters based on the observed MGMs that distinguishes the geometric similarities and differences between the various models. Note that these clusters are not necessarily aligned with the underlying SSL training paradigm, i.e., contrastive, non-contrastive, prototype-clustrrig based. (Right) MGMs that make up the principal components and their values for each model, where we indicate by the two yellow boxes (a) and (b) the MGMs that capture the maximum variation of the models along the principal directions.
4 GEOMETRY OF SSL MODELS
In this section, we aim to characterize the geometric properties of the manifolds of various SSL models, as a way to highlight the differences and similarities between models. do so, we extract the MGMs proposed in Sec. 3ec.3for for 14 SSL models and use these MGMs to quantify model equivariance- invariance, curvature, and intrinsic dimension for each augmentation manifold.
Clustering of SSL models: The similarity between SSL models leads to the clustering illustrated by the dendrogram of Figure I In Table 4 (Appendix D) we provide the details of each SSL model and highlight the structural differences that lead to our MGMs- these clusters, we consider the sparse principal component analysis (PCA) of SSL models having as features the 26 MGMs. We project the MGMs of each SSL model onto the two main components and observe three clusters (see Table ! We also provide the MGMs that were selected by the sparse PCA and their associated importance in the principal components (see Figure 9 Appendix D). We note that both simCLR-vi and simCLR-v2 have a large variation with respect to the features selected by the first principal component. Interestingly, the geometrical property that characterizes both simCLR-vl and simCLR-v2 2 is the angle between the semantic and augmented manifold (as indicated by the yellow box (a) in Table T]. This implies that their ability project the augmented samples onto the data manifold varies significantly with respect to other SSL models. Specifically, while all the models have low-affinity value between these two directions (orthogonality), both simCLR versions have a high-affinity score (xe .8) meaning that the subspaces spanning the semantic direction and the augmented direction are more aligned. This shows that the dimensional collapse effect observed and analyzed in SimCLR's 's projection head (19 21, 20) appears to impact the backbone encoder representation as well. Specifically, simCLR-v 1 and simCLR-v2 are the only models projecting the augmentation manifold onto the data manifold such that they are hardly distinguishable. Another distinction of SimCLR models is the low intrinsic dimensionality (see Table again showing that, compared to other SSL models, the SimCLR backbone encoder tends to project the data onto lower dimensional subspace.
clustering. To further analyze
The two other clusters, composed of models having different paradigms are mainly characterized by their differences in terms of semantic equivariance spread. The models in the green cluster (bottom) have low semantic equivariance spread. This implies that InDis, MoCO-v1, PIRL, SeLa-v2, BYOL, DC-v2, SwAV have the same invariance across the input data manifold, i.c., same across all classes. For instance, InDis is highly equivariant to the semantic directions (semantic equivariance 1.23, as

---

Under review as a conference paper at ICLR 2023
Normal bed count
0.4
0.10
Resret50
0.3
VIT-S
0.08
ized
0.2
0.04
0.0
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0
1.5
2.0
0.00
0
2
Augs. Equivariance
Sem. Equivariance
Augs. Equivariance
Sem. Equivariance
Figure 3: Histogram of observed equivariance (Semantic, Augmentation) for convolutional encoder backbone (ResNet50) and a vision transformer backbone (ViT-S) (Left) initialization and (Right) after training with same SSL procedure (DINO (32)). We observe that at initialization the inductive bias in convolutional networks leads to invariant representation with respect both semantic and augmentation manifold. However, VIT leads to more scattered representations of input images belonging to the semantic as well as augmentation manifolds. After training both architectures converge to similar representation where the marginal variation in spread between the two models impact the performance in downstream tasks.
shown in the 4thh column), therefore, given its semantic equivariance spread, we know that it is highly equivariant with respect all ImageNet images.
Another observation from Figure iis that when considering v/ and 12 versions of PCL, SimCLR, and MoCO, only the two MoCo versions do not belong to the same cluster. The main difference between these two models is their semantic equivariance spread. While MoCo-vl has low spread in terms of semantic equivariance, MoCo-v2 has larger spread indicating the equivariance dependent on the data sample at hand. Note that the main difference between MoCo-vl and MoCo-v2 is the utilization of a MLP projection head for the latter. Therefore, this observation would suggest that adding projection head makes the equivariance property of the backbone encoder more dependent on the input data. This result seems intuitive given the fact that the projection head will absorb most of the invariance induced by the SSL loss function.
We also note that some models are strongly invariant to rotations, e-g.. SimCLR-v1, SimCLR-v2. while other are strongly equivariant, e.g., PCL-v1, PCL-v2. This is particularly surprising rotation typically not part of the augmentation policy used train SSL models.
Similarities in SSL models: From Table we also deduce the geometric properties that do not vary across SSL models. In particular, the semantic and colorjitter affinities, i.e., the curvature of the data manifold along the semantic (label) direction and the colorjitter augmentation direction, show similar behavior across all models studied. This observation shows that the linearization capability of different trained SSL models in these two manifold directions are very similar. We defer further analysis to Appendix and summarize the observations in Figure
VIT vs ResNet: Vision transformers (ViT) (56), built with an architecture inherited from natural language processing (57), have recently emerged desirable alternative convolutional neural networks (CNN) (17 in SSL (32). While ViTs are appealing, as they provide a more general- purpose architecture, the reasons that explain the benefits of the VIT representations over those obtained from CNNs remain unclear. We compare the representations learned by these architectures, focusing ViT-Small and ResNet50 architectures, these share similar model capacity (21M vs 23M) and throughput (1 (1007/s vs 1237/s), Figure depicts a geometric comparison of the two architectures using two of our proposed equivariance metrics at initialization as well as after SSL training using DINO (32). We observe that the two architectures lead to very different representations initialization on both the semantic and augmentation manifold directions. While ResNet50 is biased toward a collapsed representation for all inputs, VIT-S corresponds to a more spread out distribution at initialization. However, after training the two architectures converge to similar equivariant representation, with ViT-S showing better class separability This observation shows that convolutional networks learn by repulsion where representations are dispersed from collapsed starting point. On the contrary, ViT shows a different behaviour where starts from a scattered representation organizing representations by attraction. This could be an explanation for the observed robustness and generality of the representation learned by ViT (58. 32)
the semantic manifold.
7

---

Under review as a conference paper at ICLR 2023
5 TRANSFER PERFORMANCES OF SSL MODELS
In this section, we investigate the question of determining which geometrical properties are crucial for SSL models to perform better specific transfer learning tasks. First, we show that the clustering results of SSL models (Sec. 4 based on their geometry mainly coincides with the observed per-cluster transfer performances. We then show that our MGMs can provide intuition regarding the geometric prop- erties that are desirable in order to perform well on specific transfer learning tasks (c.g., classification tasks gain by hav- ing rotation invariant representations). Finally, we explore which manifold properties are the most informative for the transfer performance of SSL models in a given task.
30
Merging Distance
25
20
13
10
PCL-11
McCo-12
IntoMinn
McCo-v1
InaDis
PIRL
DeepClusee pppCuuttterz
Supervised
simCLR--1
SimCUR-12
BYOL
SWAV
PCL-92
Sela-2
Transfer accuracy based clustering of SSL models: We depict in Figure the hierarchical clustering of SSL models with respect to their transfer learning accuracy in various tasks. SSL models are clustered if they achieve similar ac- curacy, good or bad, on a set of tasks. When comparing the clusters obtained using (i) MGMs (displayed in Figure []. and (fi) transfer learning performances (in Figure 4), observe that PIRL. InsDis, and MoCo-vl belong to the same group in both cases. Similarly, SwAV and BYOL, and Sim- CLR models are highly similar for both clustering results. This confirms our hypothesis that there correlation be- tween the geometrical properties of the SSL models and their transfer learning accuracy.
Figure 4: Dendogram SSL models based on their transfer learning ac- curacy. We observe that the groupings obtained are highly correlated with the clustering performed using our MGMs, thus showing that there is an intrin- sic connection between the geometric properties of the SSL model and the transfer learning performances.
25
p-
35
20
-2
Accurco (Fcusheeromang
Aecur Recuraty [DemeOONC]]
scourcc
12
64
0.6 Equivariance
0.8
3.0
12
0.4
D.0
0.5
1.0
12
0,2 03 0.4
03
0,
0.7
34 33 52 0.1
Equieriaace
Equivarience
SmCLP
Selan2
BYOL
Deepclusterv22
MaCo
PCL-2
SimCL
Iniotin
SwiW
Figure 5: Correlation between equivariance and transfer learning performances. We display the pearson coefficient P at the top left of each subplot. We observe that the equivariance with respect to semantic direction and rotation of the SSL model is negatively correlated with its capability at performing well few-shot learning tasks (small domain distance). However, these quantities are positively correlated with the accuracy of the DNN observation confirms common intuitions regarding the properties that an embedding should have to transfer accurately these two tasks. The p values for all results are S 0.01.
dense surface normal estimation task. This
Recovering geometric properties for transfer learning:
Intuitively, known that to generalize well on image recognition, an SSL model should be invariant to augmentations such rotation, while to perform well on dense tasks, i.e., pixel-level prediction tasks, it should be equivariant to rotation (26). confirm this intuition through our proposed MGMs in Figure 5 We show that the semantic and rotation equivariance correlate expected with the transfer accuracy for two different tasks, namely, Few.hotkornblith and DenseSNE. In particular, find that for (i) FewSShttKrmblith (few-shot learning with small domain distance), the greater the model invariance to semantic direction and rotation, the better the accuracy. and (ii) DenseSNE (dense surface normal estimation) the higher the equivariance of the model the better.
8

---

Under review as a conference paper at ICLR 2023
Exploring geometric properties for transfer learning: We now propose explore some possibly counter-intuitive manifold properties of the backbone encoder that correlate with specific transfer learning task. Our method is based quantifying how well each MGM can explain the per-task transfer capability of SSL. models by using simple regression methods capable of performing feature selection. For each task regress the MGMs onto the transfer learning accuracy. Details and illustrations of the results are shown in Appendix
JUMPA Ph Phutkesee
InsDis
McCo-vl
PCL.VI
235
PIRL
PCL-v2
SimCLA-v3
Sem, Eor-srance
Kodde 2751R)
fug:: Afliity
Paga Exams Sproad
McCo-v2
SimCLR
499
Selaa-9
InfoMin
DeepCluster-2
SWiV
Dog Sproad
5em 500A) Spridd
Colgg: Spreat
Sem Eaily, Sprend
Figure Correlation between MGMs and transfer performances. We display the pearson coefficient p at the top left of each subplot. We observe that for most transfer learning tasks, there exists MGM that correlates with the per-task transfer performance. We recover some intuitive results such as for many shot learning linear, higher invariance in the SSL model corresponds to better transfer learning capability. We also note that dense segmentation appears not to be correlated to the considered geometrical metrics of SSL models. The values for each result was S 0.01 except for the last plot where the p-value= 0.34.
We show in Figure O the correlation between the selected MGMs and the per-task transfer learning accuracy. We observe that in the case of many shot (linear and fine-tune), few shot (small domain distance and and large domain distance), there exist negative correlation with intuitive geomet- ric property (Figure 6 Top): semantic equivariance, rotation affinity, augmentation affinity, and spread of augmentation equivariance. For the cases of ManyShotLimean ManyShotFinetuu Few.hotKornble we recover geometric properties that were previously considered important for classification (59) 49).. However, the case of FewShorCDFSL, where the datasets used differ largely from ImageNet, the transfer performances are correlated with second-order statistics, i.c., the spread, and less on the differences between the average the MGM distributions.
and
Detection Finettionnnn,,eee Detection Frzzen DenseSNE, and DenseSeg tasks, the geometric properties that are correlated with the performances are also second-order statistics (Figure Bottom). Specifically, there exists a positive correlation with the spread of crop affinity, semantic equivariance, and colorjitter affinity.
Therefore, we observe that there is an implicit relationship between the geometric properties of SSL models and their transfer learning capabilities. Specifically, for tasks with small transfer distance relative to ImageNet, we observe that the performances an SSL model are tied to well-known geometric properties, such invariance and linearization capabilities. While in the of large transfer distance, we note that the transfer learning capabilities relies on higher-order statistics of the geometry of the backbone encoder, i.e., capturing how the model maps different inputs.
CONCLUSION
We show that the geometry of SSL models can be efficiently captured by leveraging graph-based metrics. In particular, our data-driven approach provides a way to compare SSL models that can differ in architecture and training paradigms. Our analysis provides insights into the landscape of SSL algorithms, highlighting their geometrical similarities and differences. Further, the proposed geometrical metrics capture the transfer learning capability of SSL models. Our approach motivates the design of transfer learning specific SSL training paradigms that take into account the geometric requirement of the downstream task.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Caron, H. Touvron, L. Misra, H. Jégou, Mairal, Bojanowski, and A. Joulin, "Emerging properties in self-supervised vision transformers," CoRR, vol. abs/2 14294, 2021.
[2] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. E. Hinton, "Big supervied models are strong semi-supervised learners," Advances in neural information processing systems, vol. 33, pp. 22243-22255, 2020.
[3] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, "A simple framework for contrastive learning of visual representations"" CoRR, vol. abs/2002.05709, 2020.
[4] Grill, Strub, Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, A. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, "Bootstrap your own latent: A new approach to self-supervised learning," CoRR, vol. abs/2006.07733, 2020.
[5] I. Misra and L. van der Maaten, "Self-supervised learning of pretext- innearaant representations"" CoRR. vol. abs/1912.01991, 2019.
[6] D. Dwibedi, Y. Aytar, J. Tompson, Sermanet, and A. Zisserman. "With little help from my friends: neihhot- contrastive learning visual representations." CoRR, vol. abs/2104. 14548, 2021.
A. Bardes, J. Ponce, and Y. LeCun, "Viereg: Varfance-ivvarinncccovaarnnc regularization for self- supervised learning," CoRR, vol. aass/1055.049aa 2021.
[8] Zbontar, L. Jing, Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," CoRR, vol. abss21103.03230) 2021.
[9] M. Caron, P. Bojamowski, A. Joulin, and M. Douze, "Deep clustoring for unsupervised learning of visual features," CoRR, vol. abs/1807.05522 2018
[10] J. Li, P. Zhou, C. Xiong, R. Socher, and S. C. H. Hoi, "Prototypical contrastive learning of unsupervised representations." CoRR, vol. abs/2005 04966, 2020.
[11] Y. Tian, C. Sun, Poole, D. Krishnan, C. Schmid, and Isola, "What makes for good views for contrastive learning." CoRR, vol. abs/2005. 10243, 2020.
[12] L. Ericsson, Gouk, and T. M. Hospedales. "How well do self supervved models transfer?," Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5414 5423, 2021.
[13] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations." in International conference machine learning, pp. 1597-1607, PMLR, 2020.
[14] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, al., "Bootstrap your latent-a new approach to self-supervised learning"" Advances in Neural Information Processing Systems, vol. 33. pp. 11271-21286 2020.
[15] X. Chen and He, "Exploring simple siamese representation learning." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15755 2021.
[16] J. Li, P. Zhou, C. Xiong, and S. C. Hoi, "Prototypical contrastive learning of unsupervised representations." arXiv preprint arXiv:2005.0440 2020.
[17] M. Caron, Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments," Advonces in Neural Information Processing Systems, vol. 33, pp. 9912-9924, 2020.
[18] P. Goyal, Q. Duval, J. Reizenstein, M. Leavitt, M. Xu, B. Lefaudeux, M. Singh, V. Reis, M. Caron, P. Bojanowski, A. Joulin, and Misra, "Vissl." Ettps://githus: www.aaaw....ooaaaaaaeaeeeeeeaeae http///aaacooooeserrnla vissi 2021.
[19] L. Jing, P. Vincent, Y. LeCun, and Y. Tian, "Understanding dimensional collapse contrastive self- supervised learning," arXiv preprint arXiv:21 10,09348, 2021.
[20] R. Cosentino. A. Sengupta. S. Avestimehr, M. Soltanolkotabi, A. Ortega, T. Willke, and M. Tepper, "Toward geometrical understanding of self-supervised contrastive learning," arXiv preprint arXiv:200..06926, 2022.
10

---

Under review as a conference paper at ICLR 2023
[21] W. Huang. M. Yi, and X. Zhao, "Towards the generalization of contrastive self-supervised learning," " arXiv preprint arXiv:21 (t0::2111.0074.. 2021.
[22] J. Z. HaoChen, C. Wei, A. Gaidon, and T. Ma, "Provable guarantees for self-supervised deep learning with spectral contrastive loss." Advances in Neural Information Processing Systems, vol. 34, 2021.
[23] F. Wang and H. Liu, "Understanding the behaviour of contrastive loss." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495 2504, 2021.
[24] Y. Tian, X. Chen, and S. Ganguli, "Understanding self-supervised learning dynamics without contrastive pairs," in International Conference on Machine Learning, pp. 10268-102788 PMLR, 2021.
[25] S. Kornblith, Shlens, and Q. V. Le, "Do better imagenet models transfer better?," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2677 2019.
[26] T. Xiao, X. Wang, A. Efros, and T. Darrell, "What should not be contrastive in contrastive learning." " arXiv preprint arXiv:2008.0560 2020.
[27] T. Hua, W. Wang, Z. Xue, Y. Wang, Ren, and H. Zhao, "On feature decorrelation in self-supervised learning," CoRR, vol. has/2105.004777 2021.
[28] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning," 2020.
[29] X. Chen, H. Fan, R. Girshick, and K. He, "Improved basclines with momentum contrastive learning," " arXiv preprint or Xiv::20033304222 042977 2020.
[30] I. Misra and L. d. Maaten, "Self-supervised learning of pretext-invariant representations." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
[31] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non- non-parametric instance discrimination." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018. 2018.
[32] M. Caron, H. Touvron, I. Misra, H. Jégon, J. Mairal, P. Bojanowski, and A. Joulin, "Emerging properties in self-supervised vision transformers." in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
[33] Y. M. Asano, C. Rupprecht, and A. Vedaldi, "Self-lahelling via simultaneous clustering and representation learning," arXiv preprint arXiv: 1011v:91110053111 2019
2019.
[34] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," CoRR, vol. abs/1912.00
2019.
[35] M. Belkin, Problems of Learning on Manifolds. PhD thesis, The University of Chicago, 2003.
[36] M. Belkin, P. Niyogi, and V. Sindhwani, "Manifold regularization: A geometric framework for learning from labeled and unlabeled examples," The Journal of Machine Learning Research, vol. 7, pp. 2399-2434, 2006.
[37] A. Gadde, A. Anis, and Ortega, "Active ssmi-supervised learning using sampling theory for graph signals," in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014.
[38] N. Papernot and P. McDaniel, "Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning," CoRR, 2018
[39] M. Bontonou, C. Lassance, G. B. Hacene, V. Gripon, Tang, and A. Ortega, "Introducing graph smoothness loss for training deep learning architectures," in 2019 IEEE Data Science Workshop (DSW), pp. 160-164, IEEE, 2019.
[40] C. Lassance, M. Bontonou, G. B. Hacene, V. Gripon, J. Tang, and A. Ortega, "Deep geometric knowledge distillation with graphs," in International Conference Acoustics, Speech and Signal Processing, IEEB, 2020.
[41] M. Maiker, U. von Luxburg, and M. Hein, IIffuence of graph construction on graph-based clustering measures," Advances in Neural Information Processing Systems, 2009.
11

---

Under review as a conference paper at ICLR 2023
[42] C. A. R. De Sousa. S. O. Rezende, and G. Batista, "Influence of graph construction on semi-supervised learning," Springer, Berlin, Heidelberg, 2013.
[43] S. Shekkizhar and A. Ortega, "Graph construction from data by non-negative kemel regression." in ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Procesingg (ICASSP), 3892-3896, IEFF, 2020.
pp.
[44] J. A. Tropp, Topics in spurse uppreximation. The University of Texas at Austin, 2004.
[45] S. Shekkizhar and A. Ortega, "Model selection and explainability in neural networks using a polytope interpolation framework," in 2021 55th Asilomar Conference on Signals, Systems, and Computers, pp. 177- 181. IEEE, 2021.
[46] S. Shekkizhar and A. Ortega, "Revisiting local neighborhood methods in machine learning," in Data Science and Learning Workshop (DSLW), IEEE, 2021.
[47] P. Indyk and R. Motwani, "Approximate nearest neighbors: towards removing the curse of dimensionality, in Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 604-613, 1998.
[48] Johnson, M. Douze, and H. Jégou, "Billion-scale similarity search with GPUs," IEEE Transactions on Big Data, 2019.
[49] M. Soltanolkotabi, E. Elhamifar, and E. J. Candes, "Robust subspace clustering." The annals of Statistics. vol. 42, no. 2, pp. 669-699,, 2014.
[50] D. Bonet, A. Ortega, J. Ruiz-Hidalge and S. Shekkizhar, "Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs," in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022.
[51] J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," " 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
[52] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, broader study of cross-domain few-shot learning." in European Conference on Computer Vision, pp. 124- 141, Springer, 2020.
Smith. K. Saenko, T. Rosing, and Feris, "A
[53] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes (VOC) challenge." International journal of computer vision, vol. 88, no. 2, pp. 303-338, 2010.
[54] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, "Indoor segmentation and support inference from rghd images," in European conference computer vision, pp. 746-760, Springer, 2012.
[55] B. Zhou, H. Zhao, X. Puig. T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, "Semantic understanding of scenes through the ADE20k dataset," International Journal of Computer Vision, vol. 127, no. pp. 302-321, 2019.
[56] A. Dosovitskiy, L., Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010. 1999, 2020.
[57] A. Vaswani. N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances neural information processing systems, vol. 30, 2017.
[58] Y. Bai, J. Mei, A. L. Yuille, and C. Xie, "Are transformers robust than CNNs?" Advances in Neural Information Processing Systems, vol. 34, 2021.
[59] S. Gidaris, P. Singh, and N. Komodakis, "Unsuporvised representation learingg by predicting image 2018.
rotations." arXiv preprint
[60] P. H. Richemond, J.-B. Grill, F. Altché, C. Tallec, Strub, A. Brock, S. Smith, S. De, R. Pascanu, B. Piot, et al., "Byol works even without batch statistics," arXiv preprint arXiv:2010.10241 2020.
[61] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., "Pytorch: An imperative style, performafe deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[62] P. Pope, C. 7hu, A. Ahdelkader, M. Goldblum, and T. Goldstein, "The intrinsic dimension of images and its impact leamnin," " in International Conference on Learning Representations, 2020.
12

---

Under review as a conference paper at ICLR 2023
[63] S. Maji, E. Rahtu, Kannala, M. B. Blaschko, and A. Vedaldi, "Fine-grained visual classification of aireraft," CORR, vol. 30,1136.11,11
[64] L. Fei-Fei, R. Fergus, and Perona, "Learning generative visual models from few traningg examples: An incremental bayesian approach tested on 101 object categories," in 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004.
[65] Krause, J. Deng, M. Stark, and L. Fei-Fei, "Collecting large-scale dataset fine- ine-grained cars," 2013.
[66] A. Krizhevsky, G. Hinton, et al. "Learning multiple layers of features from tiny images." 2009.
[67]
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures in the wild," CoRR, vol. 0tn//311..3613 2013.
[68] M.-E. Nilsback and A. Zisserman, "Automated flower classification over large number of classes," in 2008 Sixth Indian Conference Computer Vision, Graphics Image Processing, pp. 722-729, 2008.
[69] L.. Bossard, M. Guillaumin, and L. V. Gool, "Food-101 100miin00 discriminative components with random forests," in European conference computer vision. pp. 446-461. Springer, 2014.
[70] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar, "Cats and dogs." in 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505, IEEE, 2012.
[71] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "Sun database: Large-scale scene recogni- tion from abbey 20o." in 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 3485-3492, 2010.
[72] S.P. Mohanty, D. P. Hughes, and M. Salathé, "Using deep learning for image-based plant discase detection," Frontiers in plant science, vol. 7, p. 1419, 2016.
[73] P. Helber, B. Bischke, Dengel, and D. Borth, "Eurosat: A novel dataset and deep learning benchmark land use and land cover classification," IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, vol. 12, no. 7, pp. 2217-2226, 2019.
[74] N. Codella, Rotemberg, P. Tschandl, M. Celebi, S. Dusza, D. Gutman, Helba, Kalloo, K. Liopyris. M. Marchetti, et al. "Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic)," arXiv preprint arXiv: 03368, 2019.
[75] P. Tschandl, C. Rosendahl, and H. Kittler, "The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions," Scientific data, vol. 5, no. 1. pp. 101.
2018.
[76] X. Wang, Y. Peng. L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, "Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases." in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2097-2106, 2017.
[77] S. Rifai. G. Mesnil, P. Vincent, X. Muller, Y. Bengio, Y. Dauphin, and X. Glorot, "Higher order contractive auto-encoder." in Joint European conference machine learning and knowledge discovery in databases, pp. 645-660, Springer, 2011.
[78] R. Cosentino, R. Balestriero, Baranuik, and B. Aazhang, "Deep autoencoders: From understanding generalization guarantees." " in Mathematical and Scientific Machine Learning, pp. 197-222, PMLR, 2022.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL DETAILS ON EXPERIMENTS
Algorithm 1 MGMs for SSL
Input: Dataset D = {a; 1= pre-trained SSL backbone g, data augmentation policy T
Output: MGMs for input SSL model and dataset
1: for = t N do
2: =
3: fort = to do
4: Perform random augmentations of x!!" Txx;)
Extract representation of augmented data v²!" & (g(x!")}
5:
6: for I. = 1 to 7 do
Compute NNK graph of augmented data in encoder space: g:")) = =NNK(g(=)"),
7:
Compute local MGM values: Diam(g) (g."), ID(G("), Subspace(G!"))
8:
Compute curvature MGM Aff (Sub(ppacee( Subspace(G) )), t= [1.... 77,t' E g
9:
10: Return (GM(g: D, using local MGM obtained { Diam, ID, Aff; i= (I,..NN
In Algorithm 1 we present the pscudo-code for geometric evaluation of an SSL model using proposed MGMs defined in See namely, invariance, affinity, and number of neighbors. We first obtain local MGM values computed using the NNK neighborhood g corresponding to each augmented input data forming the graph G. A summary of the MGM, its geometric association, and estimation is presented in Table The curvature MGM is evaluated for each augmented data and its neighbor and thus produces one value per edge the graph while invariance, dimension metrics result in one value per graph constructed. For cach metric (MGM), and each direction (7), we obtain distribution over the entire manifold MGM(g. D, T). We summarized this global manifold information spread) obtained across the dataset. Note that the type of augmentation considered reflects the manifold information corresponding to a specific direction in the feature space, i.e. using rotation augmentation allows us to characterize the embedding manifold induced by the augmentation. For each data, we obtain metric that provides information regarding the geometric property of the manifold in the considered direction. Cross- augmentation MGMs (e.g. Sem.-Augs. affinity) are obtained by comparing graphs corresponding different. augmentation policies in line of of the algorithm, i.e., Subspace(g.") (*) obtained with augmentation policy Ti
considering the first
moments of the metric distribution (mean and
Subspace ( obtained with augmentation policy Tz. Consequently, for evaluating aross-augmentttion
policies the loop over number of augmentations (lines 3 8 is done twice, one with Ti and other with Tz.
In our experiments, we evaluate 5 augmentation policies (T) with T = 50 augmented samples for each image in the validation set of ImageNet (N 50000 images). This corresponds to constructing 50000 graphs per augmentation policy with each graph containing 50 nodes except for Semantic setting where we obtain 1000 graphs with 50 nodes each. Note that each graph construction incorporates 50 NNK neighborhood optimization. one for each node.
B DATA AUGMENTATIONS
We follow the data augmentation policy used in (2) which is typical setting with most self sulf-super learn- ing models considered in this work. The augmentations performed using default settings associated with the augmentation function in PyTorch (61) with the following assigned parameters: random cropping (size=224, in- terpolation=hicubie).. Horizontal flip (p=0.5), coloriiter(hrightnssss
contrast=0.4, saturation=0.2 2, huc=0.1) 1)
randomly applied with p=0.8, grayscale (p=0.2) 2), Gaussian blur (p=1.0), rotation (degrees=90). The augmenta- tion composition for each setting used in our experiments are presented in Table
14

---

Under review as a conference paper at ICLR 2023
Setting
RandomCrop
HorizontalFlip
Colorjitter
GrayScale
GaussianBlur
Rotation
Sem.
Augs.
Crop
Colorjit.
Rotate
Table 2: Evaluation setting and composed augmentation functions (sequentially applied in the order listed from left to right). All images are resized to 224 (bicubic interpolation) if not randomly cropped the size and are mean-centered and standardized along each color channel based statistics obtained with the ImageNet training set.
Metric
Geometric property
Estimation
Polytope diameter
Invariance/Equiivrrance
The maximum distance between the neighbors of an NNK polytope
maxE.EEN(F(a..)) f((x)) - f(x)))
Affinity
Local curvature
The cosine similarity of principal components of two neighboring NNK polytopes
(01) +cos (Bngng)
n.
No. neighbors
Intrinsic dimension
The nonzero weighted neighbors identified by NNK (frd()((((z:)))
Table 3: Proposed Manifold Graph Metrics, their relationship to the geometric property of the manifold, and method of estimation using observed embeddings. Note that similar diameter evaluation with k-nearest neighbor will explicitly depend on the choice of & while using 1-nearest neighbor will reduce the local geometry to only one direction. Thus, the use of neighborhood definition that is adaptive to the local geometry of the data is crucial to successfully observe the properties of the manifold.
15

---

Under review as a conference paper at ICLR 2023
C SECTION 3 ADDITIONAL FIGURES
Interpreting the MGM mean and spread:
The local MGM values are calculated from distribution over the entire dataset. As noted earlier, we compute the mean the standard deviation (referred as the spread) of this distribution to capture the characteristics of an SSL model. The interpretation of the mean is straightforward and can be seen the (average) equivariance, curvature, dimension of the augmentation manifold embedded using the SSL model on dataset. The spread of the metric captures variation of the geometric property with respect to differences the input image. For example, SSL model can be equally invariant to all input images in the datasct (cquivariance spread small) while another model might show better invariance with respect to some classes while not with some other classes (equivariance spread large) but have the same average equivariance as the first model across the dataset. In this work, we do not perform per-class based analysis since the interest was study the impact of global characteristics with that of downstream performance which are in general coarse dataset-level measures. However, we believe that MGMs proposed can be extended to specific class/attribute representational heteroscedasticity in an SSL model or how the model adapts to domain shift during transfer (e-g., sketch images to real images of object), to name few.
analysis. Furthermore, we expect such studies can help capture
Variability in SSL models: In Figure for each of the MGMs studied, depict the observed variahility in these metrics across models. The affinity between semantic manifold and augmentations manifold (Sem.- Augs.) is the MGM capturing the main differences across models. This metric can be intuitively understood as the angle between the natural image manifold and the manifold produced by the data augmentation policy. This implies that the ability of an SSL. model to project the augmented samples onto the data manifold varies significantly. The second important feature capturing the dissimilarity between SSL models is the equivariance spread along semantic and colorjitter directions. It implics that while some SSL models have lot variation in their invariance, others show equal invariance across the entire manifold. Finally, another geometrical feature that highly contributes to the dissimilarity between SSL models is the equivariance rotation. Interestingly, models will be strongly invariant rolations while other will be strongly equivariant. Note that rolation not part the augmentation policy of the models we evaluate.
We also visualize in Figure 7 the geometrical properties that do not vary across SSL models. The main ones are the semantic and colorjiter affinities, i.e. the curvature of the data manifold along the semantic (label) direction as well as the colorjitter augmentation direction. This observation shows that the linearization capability of SSL models regarding these two manifold directions is highly similar.
Equivariance
Equiv. Spread
Affinity
Affinity Spread
#
Neighbors
0.30
0.25
0.20
0.15
0.10
0.05
0.00
Sem.
Augs
Crop
Rolate
Colorit.
Sem.
Augs.
Crop
Colorjit.
Sem
Augs.
Crop
Rotate
Colorit
Sem cuge.
Sem.
Augs.
Crop
Calorii,
Sem Augo.
Sem
Augs.
Crop
Rorate
Figure 7: Manifold graph metrics variability across SSL models. We depict the variation in evaluated MGMs across 14 pretrained SSL models considered in this paper. We note that the differences in the the different SSL. model embedding can be summarized using five manifold properties, namely, (i) Sem. equivariance, (éi) Rotate equivariance, (iil) Sem. equivariance spread, (iv) Colorjit. equivariance spread, (v) Sem.-Aug. affinity. These properties highlight the key manifold properties with largest variation across the different SSL models.
16

---

Under review as a conference paper at ICLR 2023
0.3
0.2
0.1
0.0
Sem. Sem.- Augs. Aff.
Sem. Equiv. Spread
Colorjit. Equiv. Spread
Rotate Equiv.
Sem. Equiv.
Rotate Equiv. Spread
Augs. Equiv. Spread
Crop Equiv.
Crop Equiv. Spread
Augs. Equiv.
Colorjit. Equiv.
Sem. Nb. Neigh.
Sem.-Augs. Aff. Spread
Rotate Nb. Neigh.
Rotate Aff. Spread
Augs. Nb. Neigh.
Crop Nb. Neig..
Sem. Aff. Spread
Rotate Aff.
Augs. Aff. Spread
Colorjit, Aff. Spread
Augs. Aff.
Crop Aff.
Crop Aff. Spread
Colorjit. Aff.
Sem. Aff.
Figure 8: (Sorted) Evaluation of manifold graph metries variability across SSL model. We compute here the standard deviation of normalized manifold graph metrics to highlight the varying manifold properties across SSL models. We observe that five manifold properties explain most of the differences in SSL models: (i) affinity between semantic and augmentations manifold, (ii) spread of semantic equivariance, (pre)s of colorjitter equivariance. (iv) rotation equivariance, (v) semantic equivariance
17

---

Under review as a conference paper at ICLR 2023
D SECTION 4 ADDITIONAL FIGURES
D.1 CLUSTERING ALGORITHM DETAILS
We used the python Scipy library's s linkage module to performing hicrarchical agglomerative clustering. The method used to compute the distance is based on the Vour Hees Algorithm. In Figure we use the transfer learning accuracy of each model with respect all the tasks defined in the experimental settings in section 3 That is, features the clustering algorithms are the per-model accuracy on the task. per-task performances are not mentioned here as it was already developed referenced paper (22). We propose this result to display the similarity between the clusters based on transfer learning performances and the clusters based on the manifold graph metries.
Merging
Inlohinn
Contrasthve
MoCo-12
Contrattvv
PIRL
200
1024
Contrastive
MoCo-dd
200
256
Contrastive
Supervised
120
256
BYOL
1000
4096
8
Seta-2
400
1016
Contrastive
Table 4: SSL model details and the associated dendrogram split learned by our geometric metrics. We highlight the structural differences between SSL models that would possibly correspond to the geometric differences and similarities we observe. It appears that the main difference between SimCLR models and all the other SSL. models could be explained by input normalization. Then, the batch size appears to also affect drastically the geometry. Finally, we observe momentum encoder, the presence projection head and memory bank could also lead geometric differences. We believe that the current classification of SSL approaches (contrastive, non-contrastive, cluster-based) not sufficient to capture their geometric differences. All the parameters described in here should be taken into account choices and the induced geometry of the trained SSL model.
finer scale that
believe there exists complex interplay between these
18

---

Under review as a conference paper at ICLR 2023
1.0
2nd PC
0.8
0.4
0.0
Sem, Equiv.
Augs. Equiv.
Crop Equiv.
Rotate Equiv.
Colorjit. Equiv.
Sem. Equiv. Spread
Augs. Equiv. Spread
Crop Equiv. Spread
Rotate Equiv. Spread
Colorjit. Equiv. Spread
Sem. Aff.
Augs. Aff.
Crop Aff.
Rotate Aff.
Colorjit. Aff.
Sem. Aff. Spread
Augs. Aff. Spread
Crop Aff. Spread
Rotate Aff. Spread
Colorjit. Aff. Spread
Sem. Augs. Aff.
Sem. Augs. Aff. Spread
Sem. Nb.Neigh.
Augs. Nb. Nelgh.
Crop No. Neigh.
Rotate Nb. Neigh.
Figure 9: Absolute value of principal components of per-feature manifold graph metrics matrix. We display here the absolute value the two principal components obtained after sparse PCA of the matrix of dimension number of models number of graph manifold metrics. This corresponds the principal components used to visualize the distribution of the SSL models in the two-dimensional plane in Figure I We observe that few manifold graph metrics encapsulate most of the variance contained across SSL models; namely: Semantic Equivariance, Augmentations Equivariance, Crop Equivariance, Rotation Equivariance, Semantic Equivariance Spread, Augmentations Equivariance Spread, Rotation Equivariance Spread, Colorjitter Equivariance Spread, Affinity.
19

---

Under review as a conference paper at ICLR 2023
E
SECTION 5 ADDITIONAL FIGURES
2.0
1.5
1.0
0.5
0.0
SimCLR-v1
SimCLR-v2
SeLa-v2
DeepCluster-v2
BYOL
SWAV
Supervised
PCL-v1
MoCo-v1
InsDis
PIRL
PCL-V2
MoCo-v2
InfoMin
Figure 10: Dendrogram of SSL models. We compute the dendrogram of the MGM of SSL models. This shows us that there are in fact three main classes of SSL models terms of manifold property as well as the proximity between different models. This result also confirms the clustering based on PCA visualized in Figure T In particular, SimCLR-v1 and SimCLR-v2 appear to be the most distant model from all the SSL paradigms tested here. Interestingly, the clustering obtained here does not correspond to the different classes of SSL algorithms: contrastive, non-contrastive, cluster-based, and memory-based.
20

---

Under review as a conference paper at ICLR 2023
F
FEATURE SELECTION
Equivariance
Equiv. Spread
Affinity
Affinity Spread
# Neighbors
ManyShott
#egeuan fow 540
ManyShotFiretune
Rrgession Erw.
0.0
1.0
0.5
Pepecsenree:00
FewGhoiCDFSL
Entectioniineture
0.0
1.0
0.5
Regeratio frow A00
DenseSag Regision Cree 100
Sem
Augs
Crop
Rotate
Coloritt
Sem
Augs
Crop
Rotate
Colorit.
Sem.
Augs
Crop
Rotato
Colorjtt
Sem.
Crop
Rotate
Colorit.
sem.
sem.
Augs
Crop
Rotate
Figure 11: Per transfer learning task MGM importance Decision Tree. For each transfer learning task, we exploit the feature selection of decision trees (depth= 5) to visualize which MGM are crucial characterize the transfer learning accuracy. To do so, fit the MGMs with the transfer learning accuracies. In this figure, we highlight the MGMs that explain the most the per-task transfer learning accuracy. Note that
interested by the regression error, but by the importance of each MGM for predicting each transfer learning task. While being intuitive, this result shows that mainly the invariance and curvature of the DNN characterize their transfer learning capability. The first observation is that the intrinsic dimension of the DNN (displayed in green as # Neighbors) does not allow one characterize any task-specific transfer learning accuracy. Depending on the task, different geometrical properties matter. For many shot linear, the equivariance to semantic direction of the data manifold is crucial. For few shot with small transfer domain distance, the lincarization capability of the DNN with respect to the augmentations captures most of the information regarding the transfer learning capability. While most tasks appear to be explained by single of few MGMs, in the case of frozen detection and dense segmentation, multiple MGMs required to explain the transfer learning capability of SSL models.
21

---

Under review as a conference paper at ICLR 2023
Equivariance
Equiv. Spread
Affinity
Affinity Spread
# Neighbors
ManyShotLinew
0.0
MaryShofFrntture
0.5
0.0
1.0
FewShotkoombit
0.5
0.0
1.0
FewShoiCDF3L
DetectionFinetury inntune
DetectionFrazen
0.0
DenseSNE 0.5
DenseSag
Eneraay
Sam
Crop
Rotate
Colorjt.
Sem.
Crop
Rocale
Colorjt.
Sem
Rotale
Colorjt.
Sen
Sem
Augs.
Crop
Robate
Colorjt.
Sem
Sem
Crop
Rotate
Figure 12: MGM Importance Per Transfer Learning Task Lasso. For each transfer learning task, we exploit the feature selection of LASSO method visualize which MGM are crucial characterize the transfer learning accuracy. As intuitively expected, we observe that depending of the transfer learning task specific manifold properties are more important. For instance, for dense detection, the equivariance spread of the semantic manifold highly correlates with the transfer learning accuracy. Indicating that the variation of equivariance/invariane induced by the DNN manifold w.r.t. the input data is critical to accomplish dense detection. Similarly, for dense segmentation, the curvature of the semantic manifold appears to the most important feature.
22

---

Under review as a conference paper at ICLR 2023
G MGMs VALUES FOR EACH SSL MODEL
Degferfence
Aftecty
Afbeity Sprcef
DYOL
1.12
0.66
0.12
0:3
0.77
0.2
0.72
6.73
031
0.8
0.11
0.1
0.36
0/8
12.45
8.2
648
DC-v0
0.82
0.34
0.10
0.6
0.00
8:2
0.44
0.86
0.88
052
0.30
0.38
005
048
0.04
0.0
0.00
11.00
0.35
7.24
0.57
Swar
0.5
080
011
007
0.1
0:3
0.55
0.5
0.18
0.4
0.2
0.57
008
0M
0.5
0/8
0.00
13.49
5.31
6.00
8:70
0.52
0.53
0.29
0.17
015
016
0.20
014
6:7
0.55
081
081
0.78
0.5
0.9
081
60K
604
646
0.66
0.7
0.6
5.2
5.40
5.69
1:33
SincLit22
0.35
0.33
0:10
0.08
011
015
0.0
6.1
8.66
0.14
0.81
0.2
0.30
0.76
090
0.3
041
6M0
046
0.66
0,7
0.01
0.39
4.1
4.0
333
POLLT
1.25
0.63
051
0.75
011
0.77
027
022
0.9
0.08
0.57
033
0.2
050
035
0.1
045
600
0.44
0.44
0.10
0.00
7330
0.40
531
5.00
POL-d
3.35
855
0.8
013
017
825
037
839
0.7
0.53
088
0.0
0.9
032
0.17
0.6
000
0.68
0.54
0.10
0.6
5.7
7.36
775
5.5
3.08
610
818
6.0
820
8:2
0.6
0.81
0.8
0.11
8.9
033
0.8
006
686
6.6
8.84
6.6
0.8
7.82
6.6
1.80
1.00
0.30
6.2
8:2
047
031
0.2
894
0.5
0.30
036
6AM
5.44
0.54
0.10
0.04
881
2.27
6.5
5.
Rupenioet
3.00
8.4
0.7
0.55
8.44
0.5
0.2
0.5
072
0.5
0.34
0.0
0M6
0.25
008
10:33
8.00
750
743
Table 5: Proposed MGMs observed for different SSL models. We display here the values each
MGM for each SSL model. These correspond to the 26 MGMs that we consider in all the analysis we provide in the paper.
H SIMULATED EXPERIMENTS
In this section, present study controlled manifold settings to demonstrate the ability of the proposed metric in capturing the geometry of the manifold. Using data points (N 1024) sampled from three 10- dimensional manifolds embedded in an 128 128-dimensional feature space (Figure 13 Further, we generate images corresponding to these embeddings using an ImageNet pretrained generative model BigGAN as in (62) to show the implication of the geometry and its relationship with respect to images. We restrict the generation to a single ImageNet class, namely, basenji. The images generated are of size 128 128 3 This setup is equivalent to knowing the augmentation manifold and then observing the images corresponding this manifold.
Random samples: Here, we sample 1024 embeddings from a 10-dimensional multivariate Gaussian simulate an embedding space where no explicit geometry exists.
Lincar subspace with uniformly spaced samples: We start with 8 seed embeddings (equivalent to unique image inputs) and then 128 embeddings (equivalent to augmentations of the images) obtained by uniformly spaced translation of the seed embeddings.
Spherical linear interpolated (SLERP) samples: In this setup, we start with pairs of seed embeddings and then 128 augmentation embeddings that are obtained via a symmetric weighted sum of each pair of endpoints.
Note that, the latter two settings correspond to setting where the local polytopes corresponding to neighbors will have high affinity owing the smooth curvature and equal invariance (diameter) by design. contrast, the geometry of the randomly sampled data will have low affinity and high diameter. This is because the neighboring polytopes will not be aligned with neighbors within each polytope possibly spanning the entire subspacd]
The image generation using BigGAN presents a unique opportunity to test and observe the relationship between the embedding manifold and the images. Of course, typically one is interested in the reverse direction where one has input images and would like observe the embeddings (as in our SSL study). But this simulation allows us explicitly study the relationship between different manifolds and their MGM metries relative to the image mapping of the embeddings.
1 TRANSFER LEARNING EXPERIMENTS DETAIL
1.1 MANY SHOT DATASETS
FGVC Aircraft 63) Caltech- 101 (64) Stanford Cars (65) CIFARIO (6).. DTD (67). Oxford 102 Flowers (68).
Food-101 (69). Oxford-IIIT Pets 70). SUN397 (7 and Pascal VOC2007 53
1.2 FEW SHOT DATASETS
Few
corresponds to the same datasets used for many shot learning except for VOC2007. For CDFSL, we the Few-Shot Learning benchmark introduced by (52). It consists of 4 datasets that exhibit
In the simulated experiment we present the angle between the polytopes instead of the affinity. These two metrics are directly related to each other in that one is the cosine of the other. This choice makes the local curvature property explicit for readers not familiar with affinity used subspace literature.
23

---

Under review as a conference paper at ICLR 2023
poysop Blemder
1B14 pa yrope cameeer (24,2
Figure 13: Sample BigGAN images, Observed NNK diameter (Invariance), and Angle between neighboring NNK polytopes (Affnity/Curvaaurr)) three embedding manifold settings: Random (Left), Linear (Middle), and Spherical (Right). As expected, the randomly sampled examples produce neighboring polytopes that are oriented almost orthogonal with respect to each other indicative the absence of smooth manifold. Further, the large diameter in this setting shows that the examples locally distinct i.e., there is no collapse in the representation the neighbors that form the polytope. In contrast, that the embeddings from linear and spherical subspaces have polytopes that in close relationship to each other and are locally invariant.
increasing dissimilarity to natural images, CropDiseases 72 FuroSAT 73 ISIC2018
14
75
and ChestX
(76).
1.3 DETECTION
For detection, we use PASCAL VOC dataset 58)
LIMITATIONS
In this work, we performed our geometric analysis using ImageNet validation dataset on SSL models with ResNet50 backbone and one VIT architecture with approximately same number of parameters. Our results based on the feature space embedding of the models and models and datasets, or even different modalities,
foresce any issues when scaling up to larger this remains to tested and is left open for future work. We also restricted our analysis to 5 augmentation settings, which we considered to be of practical relevance, but further exploration is required to better understand the properties of the embedding manifold. Although we show empirically the correlation between geometrical properties and several downstream tasks, did not explore the theoretical implication of having certain geometry and its relationship to transfer generalization. We emphasize that our formulation, unlike the accuracy metric previously studied (12 25), is amenable to theoretical study using spectral and graph theoretical concepts, We note that our work can be leveraged in conjunction with previously developed approaches, such as (77 78). to induce particular embedding geometry depending the transfer learning application hand. Our focus in this work
provide big picture tool for understanding features obtained with SSI models using geometry/graphs and hope that it leads to new ideas for understanding and training deep learning models.
24


Reviews:
[{"review_id": "foOlcBpgP0", "reviewer": "Reviewer_teRg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The core questions in the paper are \n1) What is the geometry of representations learned by contrastive learning with data augmentation? \n2) What are the determinants of good generalization in downstream tasks?", "summary_of_the_review": "An interesting meta-learning analysis with new intuitive descriptors, evidence is presented that they can inform choices in self-supervised learning", "strength_and_weaknesses": "Strengths:  \nA set of intuitive descriptors \"manifold graph metrics\" is proposed and evidence is presented that the can inform pre-training choice for given downstream tasks.  \nExperimental evidence is quite impressive across imagery and audio signals\n\nWeakness:   \nSome unclear points in the statistical analysis of experiments (figure 6., control for multiple comparisons?)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dbOL6sIS2yS", "reviewer": "Reviewer_HJu3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work attempts to establish general geometric statistics of deep learning representations (Manifold graph metrics---MGMs) that can be useful for quantifying various properties of deep learning models. According to the paper, these can be viewed as measurements of desirable qualitative features of the resulting models. The paper makes statements about the relationships between the self-supervised models (SSL) based on the statistics developed. Specifically, the authors claim that SSL models can be clustered usefully by these geometric properties, and that they are a strong indicator of transfer learning capabilities. They do so by analyzing clustering of the SSL models on both the MGM space as well as the accuracy space (across a range of tasks). \n\nTo create a set of clusterable features, the authors consider the space of MGMs applied to manifolds resulting from neighborhoods derived from 5 generating operations: semantic category, augmentations, crop, colorjitter, and rotations. For each of these they compute the mean and spread of: equivariance-invariance (diameter of the polytope resulting from NNK performed on the set generated by the mechanism) and affinity (the angle between the natural and augmented manifolds resulting from the NNK procedure) as well as the cardinality of the NNK neighborhood (which is a surrogate for dimension of the underlying manifold). \n\nFrom these sets of features cluster and correlation analysis is applied. The authors compare the clusters resulting from features comprised by accuracy across a variety of tasks to that of the MGM features. They claim that this shows that the there is an intrinsic connection between the geometric properties of the SSL model (MGMs in this context) and the transfer learning performances. The authors close with a claim that the MGMs could be used for improved SSL.", "summary_of_the_review": "This work proposes some novel descriptor of deep learning representations viewed through augmented inference. The authors then apply these descriptors to understanding the geometry of self-supervised deep learning representations as applied to computer vision. This aspect of the work is novel. The authors do not provide in depth analysis of their proposed descriptors, convincing the reader of their value and stated significance. By directly applying correlation and clustering analysis to these descriptors without providing compelling evidence or intuition of the meaning, the authors leave the reader uncertain of the actual result here. More in depth understanding of their proposed descriptors and whether they can actually provide the evidence of the claims made in the paper should be performed. Finally, the authors allude to the value of these descriptors for self-supervised learning. But they do not conduct any experiments towards this end. It is worth noting that conducting such experiments would go a ways towards resolving the complaint of the lack of evidence of the value of the descriptors noted earlier.", "strength_and_weaknesses": "The main strength of the work is that the objective of the paper seems novel and is clearly intelligible. \n\nThere are several weaknesses in my opinion. First of all, these choices of metrics are arbitrary: while the authors claim the diameter and affinity metrics have semantic meanings in terms of equivariant and intrinsic dimension, this is not corroborated by evidence. The minimal evidence provided is in the terms of post-hoc reasoning about the nature of the models on some datasets, in light of their MGM values. This does not hold up under scrutiny, since e.g. the effectiveness of the diameter metric at capturing equivariance will vary from model to model and from training strategy to training strategy. This is because these networks have high dimensional encodings and so the encodings can be far apart geometrically, but look similar to the self-supervised decision heads (e.g. the momentum encoder weights or the prototype weights, etc.).  There would need to be more analysis specifically on the MGMs looking at the uniformity of these metrics across a range of networks and training regimes to a establish the significance of them in the proposed analysis. Are the MGMs robust to hyperparameters or small changes in the SSL methods? How about other deep learning representations? There is not enough analysis dedicated to the value of metrics independent of the clustering and correlation analyses and that leads to the another weaknness. \nAll of the statements in this work rely on some kind of dimensionality reduction or summarization of the aggregated statistics of the MGMs across a large input set (Imagenet validation). For me, it is hard to take this kind of analysis seriously given that you are reducing very large and complex collections of weights and data (generally hundred of megabytes of networks, plus gigabytes of images) into dendrograms and x-y plots. Given the lack of analysis on the MGMs themselves, I do not feel I have any intuition on the significance of this kind of analysis or can infer whether it is \"right\". Slink is a nonparametric clustering approach, but the linkages are across different algorithms represented by MGMs. Again, this leaves the reader wondering what the significance of the linkages is---beyond a degree (distance in the MGM space) or a linkage (the closest neighbor under the slink algorithm used here). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6Aig3DiKBWt", "reviewer": "Reviewer_w8qW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work empirically showed that the geometry of SSL models can be efficiently captured by leveraging graph-based metrics. The work demonstrated that the proposed geometrical metrics are able to capture the transfer learning capability of many different SSL models. The analysis provides insights into the landscape of SSL algorithms, highlighting their geometrical similarities and differences. ", "summary_of_the_review": "Overall, this work is well written and well presented.\nIt provides new metrics for evaluating the representations learned vis SSL with geometrical insights on feature similarities. \n", "strength_and_weaknesses": "#strength: It provides new metrics for evaluating the representations learned vis SSL with geometrical insights on feature similarities. \nThe work empirically demonstrates the effectiveness of these metrics on various SSL models and encoder network architectures.\n\n#weakness: The current study focuses on existing network architectures and SSL paradigms. It is unclear how the metrics can lead to better SSL paradigms or representation designing methods, with better representation qualities.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject