Title: Harnessing small projectors and multiple views for efficient vision pretraining
Venue: NeurIPS 2024
Accepted: yes
Keywords: representation learning

self-supervised learning

data-augmentation

learning dynamics

sample efficient SSL

compute efficient SSL
Abstract: Recent progress in self-supervised (SSL) visual representation learning has led to the development of several different proposed frameworks that rely on augmentations of images but use different loss functions. 
However, there are few theoretically grounded principles to guide practice, so practical implementation of each SSL framework requires several heuristics to achieve competitive performance.
In this work, we build on recent analytical results to design practical recommendations for competitive and efficient SSL that are grounded in theory. 
Specifically, recent theory tells us that existing SSL frameworks are actually minimizing the same idealized loss, which is to learn features that best match the data similarity kernel defined by the augmentations used.
We show how this idealized loss can be reformulated to a functionally equivalent loss that is more efficient to compute.
We study the implicit bias of using gradient descent to minimize our reformulated loss function, and find that using a stronger orthogonalization constraint with a reduced projector dimensionality should yield good representations.
Furthermore, the theory tells us that approximating the reformulated loss should be improved by increasing the number of augmentations, and as such using multiple augmentations should lead to improved convergence.
We empirically verify our findings on CIFAR, STL and Imagenet datasets, wherein we demonstrate an improved linear readout performance when training a ResNet-backbone using our theoretically grounded recommendations.  
Remarkably, we also demonstrate that by leveraging these insights, we can reduce the pretraining dataset size by up to 2$\times$ while maintaining downstream accuracy simply by using more data augmentations. 
Taken together, our work provides theoretically grounded recommendations that can be used to improve SSL convergence and efficiency.

Harnessing small projectors and multiple views for efficient vision pretraining

Abstract

Recent progress in self-supervised (SSL) visual representation learning has led to the development of several different proposed frameworks that rely on augmentations of images but use different loss functions. However, there are few theoretically grounded principles to guide practice, so practical implementation of each SSL framework requires several heuristics to achieve competitive performance. In this work, we build on recent analytical results to design practical recommendations for competitive and efficient SSL that are grounded in theory. Specifically, recent theory tells us that existing SSL frameworks are actually minimizing the same idealized loss, which is to learn features that best match the data similarity kernel defined by the augmentations used. We show how this idealized loss can be reformulated to a functionally equivalent loss that is more efficient to compute. We study the implicit bias of using gradient descent to minimize our reformulated loss function, and find that using a stronger orthogonalization constraint with a reduced projector dimensionality should yield good representations. Furthermore, the theory tells us that approximating the reformulated loss should be improved by increasing the number of augmentations, and as such using multiple augmentations should lead to improved convergence. We empirically verify our findings on CIFAR, STL and Imagenet datasets, wherein we demonstrate an improved linear readout performance when training a ResNet-backbone using our theoretically grounded recommendations. Remarkably, we also demonstrate that by leveraging these insights, we can reduce the pretraining dataset size by up to 2 × while maintaining downstream accuracy simply by using more data augmentations. Taken together, our work provides theoretically grounded recommendations that can be used to improve SSL convergence and efficiency.

1 Introduction

Unsupervised representation learning, i.e., learning features without human-annotated labels, is critical for progress in computer vision. Modern approaches, grouped under the self-supervised learning (SSL) umbrella, build on the core insight that similar images should map to nearby points in the learned feature space - often termed as the invariance criterion . Current SSL methods can be broadly categorized into contrastive and non-contrastive algorithms, based on whether they formulate their loss functions using negative samples or not, respectively.

ideal from theory

x2

B

A

x1

E

augmentations (A)

x1

x2

AxA

features (D)

E

E

x0x2

NxDxA

x1

x2

our proposal

datapoints (N)

(f Tf)px = (TMf TMf)px

Despite this difference in their loss formulations, recent theoretical work has established an equivalence between the contrastive and non-contrastive SSL frameworks [19]. This work shows that these different SSL formulations are ultimately minimizing a loss that encourages the learning of features that best match the data similarity kernel defined by the augmentations used. However, this notion of theoretical equivalence holds only in the limit of ideal pretraining settings, i.e. with access to infinite data and compute budget, and the feature learning behavior of different SSL algorithms in practical scenarios is still not well understood. Therefore, researchers often use empirically driven heuristics that are theoretically ungrounded to design successful applications, such as (i) a high-dimensional projector head for non-contrastive SSL or (ii) the use of two augmentations per image [3]. Moreover, existing SSL algorithms are extremely data-hungry, relying on large-scale datasets [33] or data engines [30] to achieve good representations. While this strategy works exceptionally well in data-rich settings (like training on natural-images), it is not viable in data-constrained settings (like medical imaging), where samples are relatively scarce.

With these challenges in mind, the primary focus of this work is to develop theoretically grounded recommendations for improving the effectiveness and efficiency of feature learning, both with respect to the required compute budget as well as data points. Like any unsupervised representation learning algorithm, features learned through SSL depend on three factors: (i) implicit bias of the architecture, (ii) explicit invariance imposed by data augmentations, (iii) implicit bias of the learning rule. While previous works predominantly studied the role of the model architecture capacity and loss function, and their interplay with data augmentations [9, 44], our approach broadens this perspective by also considering the role of the learning rule (gradient descent) in optimizing these loss functions. Specifically, we extend the previous theoretical findings [44] that unified the desiderata of different SSL algorithms. We reformulate the idealized unifying loss to propose a functionally equivalent loss that is more compute-efficient (see Figure 1). Based on our loss formulation, we provide two practical recommendations that can help improve the efficiency of SSL pipelines while maintaining good performance. First, we show that optimizing the reformulated loss using gradient descent can often reduce the orthogonality among the learned embeddings, thereby leading to an inefficient use of the projector network's capacity. Consequently, we recommend using a stronger orthogonalization constraint to eliminate the requirement of high-dimensional projector heads, thereby significantly reducing the parameter overhead of good feature learning. Second, we show that increasing the number of augmentations leads to a better estimate of the data similarity kernel. Consequently, we recommend using more augmentations to improve optimization convergence and learn better features earlier in training.

We empirically verify our theoretically grounded recommendations using the popular ResNet backbone on benchmark datasets: CIFAR, STL and Imagenet. Strikingly, we show that our multiaugmentation approach can learn good features even with half of the samples in the pretraining dataset. Our recommendations provide a path towards making SSL pretraining more data and compute-efficient without harming performance and could unlock massive performance gains in data-constrained setups. In summary, our core contributions are as follows:

Efficient SSL loss formulation: We propose an functionally equivalent and computeefficient formulation of the SSL desiderata that yields the eigenfunctions of the augmentationdefined data similarity kernel.

Role of heuristics: Based on our loss formulation and the implicit bias of gradient descent in optimizing this loss, we provide a mechanistic explanation for the role of projector dimensionality and the number of data augmentations. Consequently, we empirically demonstrate that low-dimensional projector heads are sufficient and that using more augmentations leads to learning better representations.

Data efficient SSL: Leveraging the convergence benefits of the multi-augmentation SSL framework, we empirically demonstrate that we can learn good features with significantly smaller datasets (up to 2 × ) without harming downstream linear probe performance.

2 Preliminaries

Existing SSL approaches in computer vision In recent years machine learning researchers have developed a number of effective approaches for learning from data without labels. The most popular approaches use augmentations of data points as targets for themselves. One of the first was a Simple framework for Contrastive Learning (SimCLR), which relied on an infoNCE loss with augmentations of an image as positive targets and augmentations of other images as negative samples (to construct the contrastive loss) [12]. Other works have relied on non-contrastive approaches, notably BarlowTwins [43] and VICReg [5]. BarlowTwins, which was inspired by the ideas of the neuroscientist Horace Barlow (cite), also uses augmentations of images, but it instead aims to optimize the covariance structure of the representations in order to reduce redundancies in the feature space [43]. Variance Invariance Covariance Regularization (VICReg) was a modification of BarlowTwins that added a variance term in the loss in order to ensure that every feature dimension has a finite variance [5]. In this paper we will focus on non-contrastive methods like BarlowTwins and VICReg, but in line with previous work [44], we also consider how these approaches relate to contrastive methods like SimCLR.

Formalizing the self-supervised learning problem Now, we will formalize the unsupervised representation learning problem for computer vision. In particular, we assume access to a dataset D = { x 1 , x 2 , ..., x n } with x i ∈ R p consisting of unlabeled images. The objective is to learn a d -dimensional representation ( d < p ) that is useful across multiple downstream applications. We focus on learning the parameters of a deep neural network f θ ∈ F Θ , using the multi-augmentation SSL framework, wherein multiple views of an image are used to optimize the pretraining loss function, L pretrain ( f θ , D )

Non-Contrastive Self-Supervised Learning (NC-SSL) algorithms impose invariance to data augmentations, while imposing regularization on the geometry of the learned feature space. More generally, L pretrain can be formulated with two terms (i) L invariance : to learn invariance to data augmentations and (ii) L collapse : regularization to prevent collapsing the feature space to a trivial solution.

where β denotes a hyperparameter that controls the importance of the collapse-preventing term relative to the invariance term. This formulation separates features that are invariant to the augmentations from those that are sensitive to them. Intuitively, the ideal feature space is more sensitive to semantic attributes (e.g. 'that's a dog') and less sensitive to irrelevant attributes (e.g. 'direction the dog is facing'), facilitating generalization to new examples.

Data Augmentation graph was introduced by Haochen et al. [22] to analyze contrastive losses, like SimCLR [12]. Briefly, we define a graph G ( A , W ) that captures the relationship between images derived from all possible data augmentations. The vertex set ( A , ρ A ) is each augmented sample in a dataset, X , and the adjacency matrix, W , denotes the similarity between pairs of vertices. Let x 0 be an image in X , and let z = M ( x 0 ) ∈ A be a random data augmentation of the image, x 0 . We define the probability density of reaching z from x 0 via a choice of mapping M :

̸

Since the mapping is not generally invertible (e.g., cropping), we observe that p ( x 0 | z ) = p ( z | x 0 ) . Using this definition, we now formally define the strength of the edge between nodes x, z ∈ A of the

augmentation graph as the joint probability of generating augmentations x, z from the same image x 0 ∼ ρ X . Notably, the edge strength of the (degree-normalized) augmentation graph is equivalent to the data similarity kernel, defined in [44]. Formally,

The magnitude of w xz captures the augmentation-defined similarity between x and z . A higher value of w xz indicates that both patches are more likely to come from the same image and, thereby, are more similar.

The desiderata of different SSL algorithms can be understood as learning features F that best capture k DAF ( x, z ) , i.e. F ( x ) T F ( z ) ≈ k DAF ( x, z ) . Recent theoretical work has shows that different SSL losses can be formulated as special cases of the objective function that recovers the top-d eigenfunctions of k DAF ( x, z ) [44].

Note that all rotations of F that don't change its span define an equivalence class of solutions to Equation (4) and make no difference for the downstream generalization of a linear probe. Based on this insight, we define an equivalence among learned feature spaces:

Definition 2.1. Let F ( x ) = ( f 1 ( x ) , . . . f d ( x )) be a d -dimensional feature vector (a vector of functions). Define the subspace

to be the span of the components of F . Given an n -dimensional feature vector, G ( x ) = ( g 1 ( x ) , . . . , g n ( x )) we say the features G and F are equivalent, if V ( F ) = V ( G ) .

3 Implicit bias of non-contrastive SSL loss and optimization

We extend the recent theoretical results [44] to propose a compute-efficient reformulation of the loss function of the SSL desiderata that yields equivalent features, i.e. the functions spanning the eigenfunctions of the augmentation-defined data similarity kernel, k DAF . Furthermore, we study the role of gradient descent in optimizing this loss function and uncover a selection and primacy bias in feature learning. Specifically, we find that gradient descent tends to learn the dominant eigenfunctions (eigenfunctions corresponding to larger eigenvalues) earlier during training, and often over-represents these eigenfunctions under weak orthogonalization constraints.

Consequently, we propose employing a stronger orthogonalization constraint during optimization when using a low-dimensional projector to ensure that learned features are equivalent to those learned with a high-dimensional projector. Furthermore, we argue that using more augmentations improves our sample estimate of k DAF , thereby aiding the eigenfunction optimization problem. We dedicate the rest of this section to highlight our key theoretical insights, and practical recommendations that follow them.

3.1 Features in terms of data augmentation kernels

Let us define a kernel operator, T k , for a positive semi-definite data augmentation kernel, k DAF .

such that Equation (4) can be equivalently written as (Equation 5 of [44])

Wecan now use Mercer's theorem to factorize k DAF into corresponding spectral features G : X → ℓ 2 (where ℓ 2 represents square summable sequences) [15, 16, 31]. However, note that computing k DAF (or T k ) is expensive as it requires computing the overlap among all augmentations of every pair of data points. Instead of computing the eigenfunctions of T k directly, we propose using an alternative operator T M :

which averages the values of the function, f , over the augmented images x 0 = M ( x ) of the data, x . We show that T T M T M is equivalent to T k , and therefore T M and T k have shared eigenfunctions.

Theorem 3.1. Let G ( x ) be the infinite Mercer features of the backward data augmentation covariance kernels, k DAB . Let F ( x ) = ( f 1 ( x ) , . . . , f N k ( x )) be the features given by minimizing the following data augmentation invariance loss

which includes the orthogonality constraint. Then, V ( F ) ⊂ V ( G ) , lim N k →∞ V ( F ) = V ( G ) .

As shown in the Appendix B, L ( F ) is equivalent to a constrained optimization formulation of the BarlowTwins loss. Furthermore, L ( F ) with the additional constraint that ( f i , f i ) ≥ γ ∀ i ∈ { 1 , 2 . . . N k } is the constrained optimization formulation of the VICReg loss.

3.2 The implicit bias of gradient descent

Next, we investigate how the use of gradient descent for optimizing L ( F ) influences the characteristics of the learned feature space, V ( F ) . Given the similarity in its form with that of the BarlowTwins loss, we build on recent findings that demonstrate the sequential nature of learning eigenfunctions when optimizing the BarlowTwins loss under a strong orthogonalization regularization [36]. Since strong orthogonalization is seldom used in practice due to instabilities in training [5, 43], we believe studying the learning dynamics under weak orthogonalization regularization (i.e. low values of β in Equation (1)) is more relevant to provide recommendations for practitioners.

Theorem 3.2. (Informal) Let us denote the span of the feature space at initialization as V ( F 0 ) and after training as V ( F T ) . For small initialization of the network's weights, the alignment of V ( F T ) with the eigenfunctions of T k depend on two factors: (i) alignment of V ( F 0 ) with the eigenfunctions of T k ; (ii) singular values of T k .

Under weak orthogonalization constraints, the network tends to learn features that are strongly aligned with eigenfunctions corresponding to large singular values. We refer to this property as the 'selection' bias of gradient descent, wherein gradient descent selects certain eigenfunctions based on the corresponding singular values. This selection bias leads to redundancy among the learned feature space, thereby reducing the effective dimensionality of the network's output space compared to its ambient dimensionality. We will leverage this finding to improve the parameter overhead of good feature learning using BarlowTwins and VICReg loss frameworks.

3.3 Takeaway 1: Low-dimensional projectors can yield good representations

Given the proximity of the formulation of Equation (9) to that of BarlowTwins and VICReg losses, we will leverage existing heuristics that have been shown to work in practice. As such, BarlowTwins and VICReg frameworks call for high-dimensional projectors while using a weak orthogonalization regularization to facilitate good feature learning. We know, from Theorem 3.1, that the eventual goal of these frameworks is to learn the eigenfunctions of the underlying data similarity graph. For example, since the intrinsic dimensionality of Imagenet is estimated to be ∼ 40 [32], it is not unreasonable to expect that the span of desired features would be of similar dimensionality. It is, thus, intriguing that the current practice would suggest using an ∼ 8192 -dim projector head to capture the intricacies of the corresponding augmentation-defined data similarity kernel. This discrepancy can be explained by analyzing the learning dynamics, as in Theorem 3.2. Notably, a high-dimensional projector is likelier to have a greater initialization span than its low-dimensional counterpart, thereby increasing the alignment between V ( F 0 ) and relevant eigenfunctions of T k . We hypothesize that a stronger orthogonalization constraint for low-dimensional projectors can rectify this issue, reducing the redundancy in the network's output space and rendering it sufficient for good feature learning.

3.4 Takeaway 2: Multiple augmentations improve kernel approximation

By comparing the invariance criterion formulation in the standard BarlowTwins and VICReg losses to Equation (7), it can be inferred that current practices use a sample estimate of T k . Using only two augmentations per sample yields a noisy estimate of T k , yielding spurious eigenpairs [41] (see

Appendix C). These spurious eigenpairs add stochasticity in the learning dynamics, and coupled with Theorem 3.2, increase the redundancy in the learned feature space [11]. We hypothesize that improving this estimation error by increasing the number of augmentations could alleviate this issue and improve the speed and quality of feature learning.

Of course, increasing the number of augmentations ( m ) in the standard BarlowTwins and VICReg loss improves the estimate of T k but comes with added compute costs - a straightforward approach would involve calculating the invariance loss for every pair of augmentations, resulting in O ( m 2 ) operations. However, Theorem 3.1 proposes an alternative method that uses the sample estimate of T M , thereby requiring only O ( m ) operations, and hence is computationally more efficient while yielding functionally equivalent features (see Appendix B). In summary, Theorem 3.1 establishes a mechanistic role for the number of data augmentations, paving the way for a computationally efficient multi-augmentation framework:

where f i ( x ) = 1 m ∑ m j =1 f i ( x j ) is the sample estimate of T M f i ( x ) .

4 Experiments

In our experiments, we seek to (i) provide empirical support for our theoretical insights and (ii) present practical primitives for designing efficient SSL routines. Since our proposed loss function is closest to the formulation of BarlowTwins/VICReg, we present empirical evidence comparing our proposal to these baselines. In summary, with extensive experiments across learning algorithms (BarlowTwins & VICReg) and training datasets (CIFAR-10, STL-10 & Imagenet-100), we establish the following:

low-dimensional projectors can yield good representations .

multi-augmentation improves downstream accuracy, as well as convergence rate.

multi-augmentation improves sample efficiency in SSL pretraining, i.e., recovering similar performance with significantly fewer unique unlabelled samples.

Experiment Setup : We evaluate the effectiveness of different pretraining approaches using image classification as the downstream task. Across all experiments, we pretrain a Resnet feature encoder backbone for 100 epochs (see Appendix E.1 for longer pretraining results) and use linear probing on the learned representations 1 . All runs are averaged over 3 seeds; error bars indicate standard deviation. Other details related to optimizers, learning rate, etc., are presented in the Appendix D.

4.1 Low-dimensional projectors can yield good representations

Existing works recommend using high-dimensional MLPs as projectors (e.g., d=8192 for Imagenet in [5, 43]), and show significant degradation in performance when using lower-dimensional projectors for a fixed redundancy coefficient ( β ). To reproduce this result, we run a grid search to find the optimal coefficient ( β ∗ 8192 ) for d = 8192 and show that performance progressively degrades for lower d if the same coefficient β ∗ 8192 is reused for d ∈ { 64 , 128 , 256 , 512 , 1024 , 2048 , 4096 , 8192 } .

A

CIFAR-10BarlowTwins

B

STL-10 BarlowTwins

C Imagenet-100 BarlowTwins

86

84

62

Test accuracy

84

80

58

82

76

80

54

Optimal βd

Optimal βa

Optimal βd

Fixed β (0.0004)

78

Fixed β (0.0004)

Fixedβ (0.001)

72

50

D

1e2

1e3

1e4

E

1e2

1e3

1e4

F

1e3

1e4

β

1e-2

1e-2

1e-2

1e-3-

1e-3

1e-3-

1e2

1e3

1e4

1e2

1e3

1e4

1e3

1e4

Projector Dimensionality

Projector Dimensionality

Projector Dimensionality

Our insights in Section 3.3 suggest low-dimensional projectors should recover similar performance with appropriate orthogonalization. To test this, we find the best β by performing a grid search independently for each d ∈ { 64 , 128 , 256 , 512 , 1024 , 2048 , 4096 , 8192 } . As illustrated in Figure 2, using low-dimensional projectors yield features with similar downstream task performance, compared to the features obtained using high-dimensional projectors. Strikingly, we also observe that the optimal β d ∝ 1 /d , which aligns with our theoretical insights.

Recommendation: Start with low-dimensional projector, using β = O ( 1 d ) , and sweep over ( pdim = d, β = O ( 1 d ) ) if needed.

4.2 Multiple Augmentations Improve Performance and Convergence

Although some SSL pretraining approaches, like SWaV [10], incorporate more than two views, the most widely used heuristic in non-contrastive SSL algorithms involves using two views jointly encoded by a shared backbone. In line with this observation, our baselines for examining the role of multiple augmentations use two views for computing the cross-correlation matrix.

A

CIFAR-10BarlowTwins

B

STL-10 BarlowTwins

C Imagenet-100 BarlowTwins

90-

90-

65

o

60-

Test accuracy

80

80-

55

70

70-

50-

2 augs,pdim=256

60

50

2 augs, pdim=256

2 augs, pdim=8192

60

2 augs,pdim=8192

2 augs,pdim=256

45-

4

2

4augs,pdim=256

augs,pdim=8192

4 augs,pdim=256

50

4 augs,pdim=256

40-

8augs,pdim=256

20

40

60

80

100

20

40

60

80

100

20

40

60

80

100

Epochs

Epochs

Epochs

To demonstrate the role of multiple augmentations in pretraining, we adapt the invariance criterion of BarlowTwins/VICReg to be in line with Equation (10). In particular, for # augs ∈ { 2 , 4 , 8 } , we

pretrain a Resnet-50 encoder with our proposed loss. Building on the insight from the previous section, we use a 256-dimensional projector head for all multi-augmentation experiments.

In Figure 3, we track the downstream performance of the pretrained models across training epochs. For performance evaluation, we use the linear evaluation protocol as outlined by [13]. Figure 3(AC) shows that pretraining with multiple augmentations outperforms the 2-augmentation baseline. Furthermore, we observe that the four-augmentation pretrained models converge faster (both in terms of the number of epochs and wall-clock time) than their two-augmentation counterparts (see Figure 3(D-F)). Additionally, we show in Appendix E.2 that our framework can also be applied to multi-augmentation settings like SWaV, where not all augmentations are of the same resolution.

Recommendatation: Using multiple augmentations ( > 2 ) is likely to improve convergence as well as downstream accuracy.

4.3 Sample Efficient Multi-augmentation Learning

Data Augmentation can be viewed as a form of data inflation, where the number of training samples is increased by k (for k augmentations). In this section, we examine the role of multi-augmentation in improving sample efficiency. In particular, we are interested in understanding if the same performance can be achieved with a fraction of the pretraining dataset, simply by using more augmentations.

A

CIFAR-10BarlowTwins

B

STL-10BarlowTwins

Imagenet-100BarlowTwins

90t

06

o

60

80

80

Testaccuracy

55

70

70

50

60

60

45

50

40

50

20

40

60

80

100

20

40

60

80

100

20

40

60

80

100

Epochs

Epochs

Epochs

2 augs,pdim=256 (100% pretraining samples)

2 augs,pdim=8192 (100% pretraining samples)

4 augs, pdim=256 (50% pretraining samples)

8 augs,pdim=256 (25%pretraining samples)

To examine the relation between the number of augmentations and sample efficiency, we fixed the effective size of the inflated dataset. This is achieved by varying the fraction of the unique samples in the pretraining dataset depending on the number of augmentations k ∈ { 2 , 4 , 8 } , e.g., we use 50% of the dataset for 4 views. We then evaluate the performance of the pretrained models on the downstream task, where the linear classifier is trained on the same set of labeled samples. Strikingly, Figure 4 shows that using multiple augmentations can achieve similar (sometimes even better) performance with lesser pretraining samples, thereby indicating that more data augmentations can be used for feature learning to compensate for smaller pretraining datasets.

Recommendation: In a low-data regime, using diverse & multiple augmentations can be as effective as acquiring more unique samples.

5 Related Work

Self-Supervised Pretraining requires significant compute resources and most practitioners rely on empirical heuristics (see SSL cookbook [3] for a summary). While recent advances in SSL theory explore learning dynamics in linear (or shallow) models [39, 40], with a focus on understanding dimensionality collapse [20, 24], the theoretical underpinnings of most of the heuristics considered essential for good feature learning, are missing.

Contrastive SSL has received more theoretical attention, owing to its connection with metric learning and noise contrastive estimation [4, 25, 29]. In particular, HaoChen et al. [22] provide a theoretical framework for the SimCLR loss from an augmentation graph perspective, which leads to practical recommendations. Subsequently, Garrido et al. [19] establish a duality between contrastive and non-contrastive learning objectives, further bridging the gap between theory and practice.

Non-contrastive SSL algorithms' theoretical foundations have received more attention recently [9, 44]. Prior works [2, 18, 19] have demonstrated that with modified learning objectives, lowdimensional projectors yield representations with good downstream performance. Similarly, previous works have demonstrated notable performance boosts when using a multi-patch framework in contrastive [17] and non-contrastive SSL [10, 42]. However, the theoretical basis for the benefits and trade-offs of either low-dimensional projectors or multiple augmentations is largely unclear. It is worth noting that Schaeffer et al. [34] present an information-theoretic perspective of the recently proposed non-contrastive SSL loss that leverages multiple augmentations, namely MMCR [42], but the computational advantages of using multiple augmentations on the learning dynamics is an active area of research.

Deep Learning theory has made significant strides in understanding the optimization landscape and dynamics of supervised learning [1]. In concurrent works [9, 44], the interplay between the inductive bias of data augmentations, architectures, and generalization has been explored from a purely theoretical perspective, establishing an equivalence among different SSL losses [44]. Furthermore, Simon et al. [36] used a more straightforward formulation of the BarlowTwins loss and investigated the learning dynamics in linearized models for the case when the invariance and orthogonalization losses have equal penalties. Although such a setting rarely used in practice, their approach serves as an inspiration for our work in studying the learning dynamics of non-contrastive SSL losses.

6 Discussion

Summary : Our work builds on existing theoretical results that establish an equivalence among different SSL frameworks, and proposes a compute-efficient reformulation of the common SSL loss. Using this loss reformulation and a study of the optimization dynamics, we proposed practical recommendations to improve the sample and compute efficiency of SSL algorithms. Specifically, we recommended low-dimensional projectors with increased orthogonality constraints and multiaugmentation frameworks, and we verified the effectiveness of these recommendations empirically. It is worth noting that our multi-augmentation formulation improves the efficiency of learning without altering the desiderata of SSL, i.e. the network learns the same feature space using our proposed multi-augmentation framework as with the original SSL formulation in the limit of infinite pretraining budget. To demonstrate this equivalence between the original SSL loss and our proposed version, we show in Appendix E.1 that longer pretraining on the 2-augmentation loss leads to similar downstream performance as the multi-augmentation versions (4 and 8 augmentations).

We also showed that the multi-augmentation framework can be used to learn good features from fewer unique samples in the pretraining dataset simply by improving the estimation of the data augmentation kernel. This result has direct implications on improving the Pareto frontier of samples-vs-performance for SSL pretraining, wherein we can achieve better downstream performance when limited number of samples are available in the pretraining dataset.

Pareto Optimal SSL In the context of sample efficiency, training a model using two augmentations with different fractions of the dataset leads to a natural Pareto frontier, i.e., training on the full dataset achieves the best error but takes the most time ( Baseline (2-Aug) ). Our extensive experiments demonstrate that using more than two augmentations improves the overall Pareto frontier, i.e., achieves better convergence while maintaining accuracy ( Multi-Aug ). Strikingly, as shown in Figure 5, we observe that we can either use a larger pretraining dataset or more augmentations for a target error level. Therefore, the number of augmentations can be used as a knob to control the sample efficiency of the pretraining routine.

Multi-Aug

50

4-Aug

%

40

8-Aug

Error

30

20

10

0

25

50

75

100

Runtime (min)

Connections to Downstream performance : While our core theoretical results are aimed at accelerating convergence of the SSL loss itself, our empirical results highlight an improved downstream task performance earlier during pretraining. While this discrepancy might seem counter-intuitive at first, it is worth noting that the SSL loss inherent influences downstream performance as it encourages clustering of semantically similar images in the representation space. Such clustering properties in the representation space facilitates easier classification through methods k-nearest neighbors or linear decoding for a large number of tasks that rely on the semantic content of images. Previous works [2, 18, 20, 37] have discussed in detail how certain geometric properties of the learned representation space are connected to the linear classification performance for arbitrary decision boundaries, in expectation. However, an in-depth analysis of downstream tasks that are more amenable to linear decoding from the learned SSL representation space requires framing metrics of alignment between the pretraining objective (SSL desiderate) and the downstream task labels, and is an active area of research.

Open Questions : Looking ahead, it would be exciting to extend this analysis to other categories of SSL algorithms, such as Masked AutoEncoders (MAE). Furthermore, our insights provide opportunities to explore sample-efficient methods that rely on less data, which is particularly important in critical domains such as medical imaging, where data is often relatively scarce and expensive. On a different note, it is intriguing that animals often spend extended periods of time exploring novel objects, likely to gain multiple views of the object [6, 28]. Given the theoretical underpinnings of the computational benefits of multi-augmentation SSL outlined in our work, it would be exciting to develop models of biological learning that leverage these insights and enable sample-efficient continual learning in similar environments.

Limitations : Our algorithm relies on multiple augmentations of the same image to improve the estimation of the data-augmentation kernel. Though this approach speeds up the learning process, it also adds some extra computational overhead, which means that the impact of faster learning on wall-clock time is less than might be hoped for. One way to mitigate the effects of this limitation would be to scale up to a multi-GPU setting, since the computations for each augmentation can be run on a separate GPU in parallel. This could help ensure that the improved speed of learning directly translates to a significantly reduced wall-clock time for training.

Impact Statement : The goal of our work is to advance the general field of visual representation learning. Although there are potential downstream societal consequences of our work, we feel there are no direct consequences that must be specifically highlighted here.

References

Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks , 132:428-446, 2020.

Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. α -req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems , 35:17626-17638, 2022.

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 , 2023.

Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NeurIPS , 2022.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

Daniel E Berlyne. Novelty and curiosity as determinants of exploratory behaviour. British journal of psychology , 41(1):68, 1950.

Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards democratizing jointembedding self-supervised learning, 2023.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssl interplay: Augmentations, inductive bias, and generalization. In International Conference on Machine Learning , pages 3252-3298. PMLR, 2023.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. arXiv preprint arXiv:2306.04251 , 2023.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08954 , 2022.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, and Jun Zhu. Neural eigenfunctions are structured representation learners. arXiv preprint arXiv:2210.12637 , 2022.

Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef: Deconstructing kernels by deep neural networks. In International Conference on Machine Learning , pages 4976-4992. PMLR, 2022.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9588-9597, 2021.

Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning , pages 10929-10974. PMLR, 2023.

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations , 2023.

Arna Ghosh, Arnab Kumar Mondal, Kumar Krishna Agrawal, and Blake Richards. Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05808 , 2022.

I Gohberg. Classes of linear operator theory. Advances and Applications , 49, 1990.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Lajos Horváth and Piotr Kokoszka. Inference for functional data with applications , volume 200. Springer Science & Business Media, 2012.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 , 2021.

Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report , 2009.

Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffcv. https://github.com/libffcv/ffcv/, 2022. commit 3a12966.

Marianne Leger, Anne Quiedeville, Valentine Bouet, Benoît Haelewyn, Michel Boulouard, Pascale Schumann-Bard, and Thomas Freret. Object recognition test in mice. Nature protocols , 8(12):2531-2537, 2013.

Yazhe Li, Roman Pogodin, Danica J Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems , 34:15543-15556, 2021.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

David Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, and Kimberly L Stachenfeld. Spectral inference networks: Unifying deep and spectral learning. arXiv preprint arXiv:1806.02215 , 2018.

Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations , 2021.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, SueYeon Chung, et al. Towards an improved understanding and utilization of maximum manifold capacity representations. arXiv preprint arXiv:2406.09366 , 2024.

Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning , pages 19847-19878. PMLR, 2022.

James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. arXiv preprint arXiv:2303.15438 , 2023.

Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssl architectures. In The Twelfth International Conference on Learning Representations , 2024.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020.

Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning , pages 10268-10278. PMLR, 2021.

Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578 , 2020.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and SueYeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems , 36:24103-24128, 2023.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Runtian Zhai, Bingbin Liu, Andrej Risteski, Zico Kolter, and Pradeep Ravikumar. Understanding augmentation-based self-supervised representation learning via rkhs approximation. arXiv preprint arXiv:2306.00788 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims are backed by theoretical and empirical results. Theorem 3.1 presents our functionally-equivalent compute-efficient formulation of the SSL objective, and Theorem 3.2 demonstrates the implicit bias of gradient descent during optimizing the SSL loss. Our empirical results demonstrate the utility of our theoretical insights in improving the parameter overhead of good feature learning, optimization convergence and the sample efficiency.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a section on limitations in the discussion.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The formal statements alongside proofs are presented in the supplementary material (Appendices A to C).

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details in the supplementary material (Appendix D). We have also released our code base in the public github repo, FastSSL, to facilitate the implementation of our proposed framework.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: We use open-access datasets, like CIFAR, STL and Imagenet. Our code base can be found in the public github repo, FastSSL

Guidelines:

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We present details of the experiment setup and results in Section 4 of the main paper, and additional implementation details in Appendix D.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report standard error bars, computed over 3 seeds, for all result plots and tables.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All our CIFAR and STL experiments were done on a single 48-GB RTX8000 GPU and all Imagenet experiments were performed on 2 40-GB A100 GPUs. All experiments were performed on the Mila cluster, aided by compute resources, software and technical help provided by Mila (mila.quebec).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: We adhere to the NeurIPS Code of Ethics in our work, and research in general.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We add a statement on societal impact in the Discussion section. Since the goal of our work is to advance the general field of visual representation learning, we feel there are no direct consequences that must be specifically highlighted here. Although we recognize that there might be potential downstream consequences that warrant attention while building intelligent systems that leverage this work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any new data or state-of-the-art models.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We acknowledge and cite the datasets and model architectures used in this work. Our codebase is publicly available on our github repo, FastSSL. Moreover, our codebase relies on the Python packages of PyTorch, FFCV [27] and FFCV-SSL [7], which are referred to in the github repo README.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are released.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve research with human subjects.

A Hilbert Space of functions

A.1 Functions and inner product space

Definition A.1. Given X,ρ X , and f, g : X → R , define the L 2 ( ρ X ) inner product and norm, respectively,

Define

to be the (equivalence class) of functions with finite ρ X norm.

A.2 Spectral theory

In this section we quote the relevant (abstract) Hilbert Space theory.

Definition A.2 (Spectral Operator) . Given orthogonal functions, Φ = ( ϕ i ) i ∈ I in L 2 ( ρ X ) , and non-negative Λ = ( λ i ) i ∈ I , with ∥ Λ || 2 2 = ∑ i ∈ I λ 2 i < ∞ . Call (Φ , Λ) a spectral pair and define the corresponding spectral operator by

Theorem A.3 (Spectral Decomposition) . Suppose H is a Hilbert space. A symmetric positivedefinite Hilbert-Schmidt operator T : H → H admits the spectral decomposition equation 12 with orthonormal ϕ j which are the eigenfunctions of T , i.e. T ( ϕ j ) = λ j ϕ j . The ϕ j can be extended to a basis by adding a complete orthonormal system in the orthogonal complement of the subspace spanned by the original ϕ j .

Remark A.4 . The ϕ j in equation 12 can thus be assumed to form a basis, but some λ j may be zero.

We defer the reader to [21, 23] for an in-depth discussion and proof of Theorem A.3.

Denote by L the space of bounded (continuous) linear operators on H with the norm

Definition A.5 (Compact Operators) . An operator T ∈ L is said to be compact if there exist two orthonormal bases { g j } and { f j } , and a real sequence { λ j } converging to zero, such that

The λ j may be assumed positive. The existence of representation equation Compact is equivalent to the condition: T maps every bounded set into a compact set. Compact operators are also called completely continuous operators. Representation equation Compact is called the singular value decomposition.

Definition A.6 (Hilbert-Schmidt Operators) . A compact operator admitting representation equation Compact is said to be a Hilbert-Schmidt operator if ∑ ∞ j =1 λ 2 j < ∞ . The space S of HilbertSchmidt operators is a separable Hilbert space with the scalar product

where { f i } is an arbitrary orthonormal basis. Note the value of equation 13 is independent of the basis. The corresponding norm is

One can show that

Definition A.7. An operator T ∈ L is said to be symmetric if

(An operator with the last property is sometimes called positive semidefinite, and the term positivedefinite is used when the inequality is strict.)

B Data augmentation kernel perspective of non-contrastive SSL

Theorem B.1. Let G ( x ) be the infinite Mercer features of the backward data augmentation covariance kernels, k DAB . Let F ( x ) = ( f 1 ( x ) , f 2 ( x ) , . . . , f k ( x )) be the features given by minimizing the following data augmentation invariance loss

which includes the orthogonality constraint. Then, V ( F ) ⊂ V ( G ) , V ( F ) → V ( G ) as N k →∞ .

The idea of the proof uses the fact that, as linear operators, T k DAB = T ⊤ M T M and that T k DAF = T M T ⊤ M . Then we use spectral theory of compact operators, which is analogue of the Singular Value Decomposition in Hilbert Space, to show that eigenfunctions of T ⊤ M T M operator are the same as those obtained from optimizing L ( F ) . A similar result can be obtained using k DAF and T ⊤ M .

Note that L ( F ) is the constrained optimization formulation of the BarlowTwins loss. Furthermore, L ( F ) with the additional constraint that ( f i , f i ) ≥ γ ∀ i ∈ { 1 , 2 . . . N k } is the constrained optimization formulation of the VICReg loss.

B.1 Proof of theorem 3.1

We show we can factor the linear operator, leading to a practical algorithm. Here, we show that we can capture the backward data augmentation kernel with the forward data augmentation averaging operator

Lemma B.2. Using the definitions above, and with k in equation 6 given by k DAB ,

Proof. First, define the non-negative definite bilinear form

Given the backwards data augmentation covariance kernel, k DAB , define

We claim, that

This follows from the following calculation,

and positive-definite if

For implementations, it is more natural to consider invariance to data augmentations.

Theorem B.3 (equivalent eigenfunctions) . Assume that T M is a compact operator. Define the invariance bilinear form

Then B INV , B V AR share the same set of eigenfunctions. Moreover, these are the same as the eigenfunctions of B DA,B . In particular, for any eigenfunction f j of B V AR , with eigenvalue λ j , then f j is also and eigenfunction of B INV , with the corresponding eigenvalue given by ( √ λ j -1) 2 .

Proof. Define T MM by,

Define

Note, by the assumption of compactness, T M has the Singular Value Decomposition, (see the Hilbert Space section for equation SVD),

Let f j be any right eigenvector of T M , with eigenvalue µ j . Then f j is also a right eigenvector T M -I , with eigenvalue µ j -1 . So we see that T MM has f j as an eigenvector, with eigenvalue λ j = µ 2 j and T MS has f j as an eigenvector, with eigenvalue ( √ λ j -1) 2 . Finally, the fact that there are no other eigenfunctions also follows from equation SVD.

The final part follows from the previous lemma.

Equivalence of Barlow Twins loss to Equation (9). The BarlowTwins loss from [43] is as follows:

̸

where C is the cross-correlation matrix computed between the outputs of the network to two different augmentations. First, the BarlowTwins loss can be seen as the unconstrained optimization form of the following constrained optimization objective:

̸

where β is the Lagrangian multiplier [8]. In [43], the cross-correlation matrix C is computed by a dot product between normalized functions f i 's such that ( f i , f i ) ρ X = 1 ∀ i . The network output for one augmentation of x , a , can be thought of as a Monte-Carlo estimate (with one sample) of T M f i ( x ) , where f i is the i th dimension of the network's output. Therefore, the BarlowTwins loss can be written in its following equivalent form:

As shown by [44], the eigenvalues of T T M T M are always less than 1. Therefore, we do not need the square in Equation (28). Rewriting it, we get the following:

Using Theorem B.3, we show that the loss recovers the equivalent eigenfunctions for the following reason. We can rewrite the loss as

which recovers the loss Equation (9). Note that the VICReg loss [5], in addition to the constraints imposed by the BarlowTwins loss, ensures that the norm of f i 's are more than some threshold. This can be easily incorporated into the constraint with a constant along with δ ij . In conclusion, both BarlowTwins and VICReg losses can be seen as equivalent forms of the loss Equation (9).

Theorem B.4. (Informal) Let us denote the span of the feature space at initialization as V ( F 0 ) and after training as V ( F T ) . For small initialization of the network's weights, the alignment of V ( F T ) with the eigenfunctions of T depend on two factors: (i) alignment of V ( F 0 ) with the eigenfunctions of T ; (ii) singular values of T .

Theorem B.4. (Formal) Let Γ = V Λ V T represent the eigendecomposition of Γ , and define z as the projection of the weight vectors in W onto singular vectors of Γ , V . Formally, z = WV . Assuming small initialization (as in Simon et al. (2023), i.e. | z pi (0) | << 1 for all p, i , we can derive the following conclusions:

For all λ i , λ j > 0 , z pi ( t ) z pi (0) = ( z pj ( t ) z pj (0) ) λ i λ j where λ i denotes the i th singular value, i.e. i th element of diagonal matrix Λ .

Proof. We will first show that the above holds for a linear network, i.e. the output of the network with weights W ∈ R m × n is WX for some input X ∈ R n × b , where m is the output dimensionality, n is the input dimensionality and b is the batch size.

Let us first analytically compute the cross-correlation matrix C following [36].

where X and X ′ are matrices ∈ R n × b containing two augmentations of a each image in a batch of images. Also, we have defined T = XX ′ T , i.e. the augmentation-defined data correlation matrix. Rewriting the BarlowTwins loss function from [43]:

̸

To study the learning dynamics, we need to compute the gradient of L BT w.r.t. the parameters W .

̸

Let us now analytically compute the derivatives of C ii and C ij w.r.t W pq to simplify each of the terms in Equation (31).

Using similar algebra steps, we can simplify the second term:

̸

̸

̸

̸

Substituting Equations (32) and (33) into Equation (31), we get:

Note that setting β = 1 yields the dynamics equation presented by [36]. However, in practice, β is orders of magnitude less that 1. For sake of simplicity, we will analyze the extreme case of β = 0 , which will yield us insights into the weak-orthogonality constraint case. Therefore,

Let us denote the eigendecomposition of T be written as T = V Λ V T . Here, Λ is a diagonal matrix with singular values as the diagonal elements. Let us also denote the projection of the weight vectors onto the singular vectors of T , i.e. V as z . So, z = WV . Therefore, using these definitions, we can write the following:

Now, writing the update equations Equation (35) in terms of z pi :

Assuming small initialization of weights W , we can assume that | z pi (0) | << 1 , i.e. magnitude z pi at time 0 is very small. Let us define h p ( t ) = 1 -∑ j z pj ( t ) 2 λ j . For small initialization, h p ( t ) > 0 ∀ t . Therefore,

It is clear from Equation (37) that if λ i < 0 , lim t →∞ z pi ( t ) = 0 . Similarly, if λ i = 0 , then z pi ( t ) = z pi (0) ∀ t .

Therefore, akin to the conclusions of [36], the BarlowTwins loss recovers directions corresponding to positive singular values in the augmentation-defined covariance matrix, T and suppresses directions corresponding to negative singular values. Thus, the network outputs span the top singular vectors of T .

It is worth noting from Equation (36) that the following holds:

Without loss of generality, if λ i << λ j , then z pi ( t ) ≈ z pi (0) . Therefore, under small initialization, i.e. z pi (0) is small ∀ i , gradient descent biases the p th weight vector to be more strongly aligned to the eigenvector corresponding to the strongest eigenvalue, for all p 's. Hence, under weak orthogonalization constraints, the BarlowTwins loss will over "represent" the strong singular vectors of the augmentation-defined cross-correlation matrix.

When using high-dimensional projectors, specifically when m>> ∑ i 1 λ i > 0 , wherein 1 ζ is the indicator function that is 1 when condition ζ is true and 0 otherwise, this problem might be ameliorated because there are multiple weight vectors that might be aligned with the top singular vectors of T at initialization. However, when using low-dimensional projectors, we do not have such a luxury and therefore, using a weak orthogonalization constraint leads to dimensionality collapse in the representation space.

Extending to deep non-linear networks. Similar to the analysis in [36], we can repeat the above analysis by replace X and X ′ by the corresponding kernel versions, where the kernel corresponds to the Neural Tangent Kernel (NTK) of the network. Therefore, the implicit bias of gradient descent to yield dimensionality collapse in the representation space when using weak orthogonalization constraints still remains.

Dimensionality collapse under noisy optimization. From the rest of this section, we have seen that the BarlowTwins loss is a Monte-Carlo estimate of the true data-augmentation defined covariance matrix. Moreover, stochastic gradient descent adds noise due to mini-batch sampling to the optimization process. Note that there exist symmetries in our linear network, i.e. an orthogonal rotation of the weight matrix yields the same loss function. As explained in [11], such symmetry-invariant sets are potential candidates for stochastic collapse when performing noisy gradient-based optimization. Therefore, the presence of noise in the data-augmentation covariance matrix, T , as well as the batch noise can further worsen the dimensionality collapse problem where different weight vectors become parallel to each other due to noise in updates. One possible mitigation strategy is to obtain a better estimate of the true augmentation-defined covariance matrix (see Figure 7), which we discuss in the next section.

Empirical validation. We empirically validate our results on the learning dynamics on simplistic 2-dimensional settings. These results, demonstrating the difference in feature learning dynamics for weak vs strong orthogonalization, are presented as GIFs in the supplementary material, and can also be viewed at the project website.

C Multi-Augmentation Learning

C.1 Augmentation graph

We use the population augmentation graph formulation introduced in [22]. Briefly, we define a graph G ( X , W ) , where the vertex set X comprises of all augmentations from the dataset (could be infinite when continuous augmentation functions are used) and W denotes the adjacency matrix with edge weights as defined below:

, i.e. the joint probability of generating 'patches' x, x ′ from the same image ¯ x . Here A defines the set of augmentation functions used in the SSL pipeline. It is worth noting that the magnitude of w xx ′ captures the relative similarity between x and x ′ . A higher value of w xx ′ indicates that it is more

A

B

C

μ'

P

P

x1

x2

X1

μ

X2

X2

V

X3

μ

X3

X

p

p

likely that both patches came from the same image, and thereby are more similar. The marginal likelihood of each patch x can also be derived from this formulation:

C.2 Contrastive and non-contrastive losses suffer from the same issues

We will now show that the proposal of using multiple patches for the L invariance is pertinent to both the contrastive and non-contrastive SSL. Following [22], we use the spectral contrastive loss formulation and incorporate the augmentation graph relations:

where z := √ w x f ( x ) , D is a N × N diagonal matrix with entries { w x } and ¯ W = D -1 2 W D -1 2 .

We extend the duality results between contrastive and non-contrastive SSL loss, established by [19], to demonstrate how Equation (41) can be decomposed into the invariance and collapse-preventing loss terms.

where κ is some constant independent of Z . The first term in Equation (42) is the covariance regularization term in non-contrastive losses like BarlowTwins (implicit) or VIC-Reg (explicit), and the second term in Equation (43) is the variance regularization. Simplifying the third term in Equation (43) gives us:

This term encourages f ( x ) to be similar to f ( x ) , i.e. the mean representation across all augmentations of ¯ x , thereby requiring to 'sufficiently' sample A ( . | ¯ x ) . Given that both the contrastive and noncontrastive losses rely on learning invariance properties from data augmentations, we believe that our multi-patch proposal would improve the probability density estimation of A ( . | ¯ x ) and yield better performance with few training epochs.

C.3 Explaining training dynamics in low patch sampling regime

We now turn to a simple form of the augmentation graph to understand how using low number of augmentations affects the evolution of ZZ T . Minimizing Equation (41) implies that the spectral decomposition of Z would align with the top eigenvectors (and values) of W . Wewill demonstrate that in the low sampling regime (using few augmentations), the eigenvectors of the sampled augmentation graph ˜ W may not align with those of W .

Augmentation graph setup. We define an augmentation graph with only two instances from two different classes, similar to the one presented in [35]. Let us denote the four instances as ¯ x i for i ∈ 1 , 2 , 3 , 4 , where ¯ x 1 , ¯ x 2 belong to class 1 (i.e. y 1 , y 2 = 1 ) and ¯ x 3 , ¯ x 4 belong to class 2 (i.e. y 3 , y 4 = 4 ). Let us further assume that ¯ x 1 , ¯ x 3 have the highest pixel-level similarity among (¯ x 1 , ¯ x i ) ∀ i ∈ 2 , 3 , 4 , thereby making it more likely to have similar patches. We denote this relationship among input examples using G to indicate (pixel-wise) global similarity groups. So, G 1 , G 3 = 1 and G 2 , G 4 = 2 . We can use the following probabilistic formulation to model our augmentation functions (see Figure 6B):

̸

̸

̸

̸

̸

In our setting, ρ ′ + µ ′ + ν ′ + δ ′ = 1 . The adjacency matrix of our augmentation graph (as shown in Figure 6C) is as follows:

We defer the relations between ρ ′ , µ ′ , ν ′ δ ′ and ρ, µ, ν, δ to the appendix. The eigenvalues of this matrix are: ( ρ + µ + ν + δ , ρ + µ -ν -δ , ρ -µ + ν -δ , ρ -µ -ν + δ ). Corresponding eigenvectors are along [1 , 1 , 1 , 1] T , [1 , 1 , -1 , -1] T . [1 , -1 , 1 , -1] T , [1 , -1 , -1 , 1] T . Assuming that the augmentation functions induce semantically-relevant invariance properties that are relevant for identifying y i from f ( x i ) , we can say that ρ ′ > max { µ ′ , ν ′ } and min { ν ′ , µ ′ } > δ ′ . When we have sufficiently sampled the augmentations, any SSL loss will learn Z such that its singular values are span the top eigenvectors of the augmentation graph, and the eigenspectrum of ZZ T would simply be the above eigenvalues. In practical settings, the augmentation graph would have significantly higher dimension that the feature/embedding dimension 2 . Therefore, singular vectors of Z would span the top eigenvectors of W and the smaller eigenmodes are not learned. When we have accurately sampled the augmentation graph, µ > ν and therefore, the class-information preserving information is preferred over pixel-level preserving information during learning. But what happens when we do not sufficiently sample the augmentation space?

Ansatz. Based on our empirical experience, we define an ansatz pertaining to the eigenvalues of a sampled augmentation graph and validate it in tractable toy settings, such as the one described above. Specifically, we claim that when the augmentation space is not sufficiently sampled, {| µ -ν | , δ } → 0 . In other words, we claim that when only few augmentations per example are used, it is more likely to have an equal empirical likelihood for augmentations that preserve (pixel-level) global information and class/context information. Moreover, it is very unlikely to have augmentations that change both the class and global information. This is demonstrated in Figure 7.

Consequences of the Ansatz . When only a few augmentations are sampled, learning can suppress the class information at the cost of preserving the pixel-level information, thereby leading to an increased smoothness in the learned feature space.

̸

̸

eigenveco

0.4-

True eigenvec1

eigenvec1

True eigenvec2

eigenvec2

Eigenvalue

Trueeigenvec3

eigenvec3

0.3

0.2

0.1-

0.0-

10

100

Number of patches

D Implementation Details

Image Classification Datasets Across all experiments, our settings mainly follow [13]. In particular, Table 4a summarizes our pretraining settings on Cifar-10 [26], STL-10 [14] and Imagenet-100 [33]. The Imagenet-100 dataset was generated by sampling 100 classes from the original Imagenet-1k dataset, according to this list [38]. In Table 4b, we outline the corresponding linear evaluation settings for Resnet-50 (for CIFAR-10 and STL-10) and ResNet-18 (for Imagenet). Note that we add a linear classifier layer to the encoder's features and discard the projection layers for evaluation. Our code base is publicly available on github.

The key SSL loss functions that we use in this work are BarlowTwins [43] and VICReg [5]. Let us suppose that the embeddings of two augmentations of a batch of images are denoted as z and z ′ . The BarlowTwins loss function is as follows:

̸

C ij is the element of C at row i , column j and n is the batch size. For each projector dimensionality, d , we search for the hyperparameter, β , that yields the best downstream task performance.

The VICReg loss function is as follows:

̸

For each projector dimensionality, d , we search for the hyperparameter, µ , that yields the best downstream task performance.

D.1 Empirical results for low-dimensional projectors

A

CIFAR-10BarlowTwins

B

CIFAR-10 VICReg

841

84

Test accuracy (

80

80

76

76

72

Optimal βd

Optimal βd

72

Fixed β (0.0004)

68

Fixed β (0.0013)

1e2

1e3

1e4

1e2

1e3

1e4

C

D

1e-2

1e-4

Optimal

1e-5-

1e-3-

1e-6-

1e2

1e3

1e4

1e2

1e3

1e4

Projector Dimensionality

Projector Dimensionality

D.2 Empirical results with multi-augmentations along with Time

A

CIFAR-10 BarlowTwins

B

CIFAR-10 VICReg

C

STL-10 BarlowTwins

06

06

06

80

Test accuracy

80

80

70

等

70-

60

70

60

2 augs, pdim=256

50

2 augs, pdim=256

60

2 augs,pdim=256

50

2 augs, pdim=8192

40

2 augs,pdim=8192

2 augs, pdim=8192

4 augs, pdim=256

4 augs, pdim=256

50

.

4 augs, pdim=256

20

40

60

80

100

20

40

60

80

100

20

40

60

80

100

Epochs

Epochs

Epochs

D

E

F

106

06

Test accuracy (

80

80

80

70

70

林

60

70

60-

2 augs, pdim=256

50

2 augs,pdim=256

60

2 augs,pdim=256

50

2 augs,pdim=8192

4 augs, pdim=256

40

2 augs,pdim=8192

4 augs, pdim=256

50L

2 augs, pdim=8192

4 augs, pdim=256

0

20406080100120140

0

20

40

6080100120

0

200

4006008001000

Time (min)

Time (min)

Time (min)

+

A

CIFAR-10 BarlowTwins

B

CIFAR-10 VICReg

C

STL-10 BarlowTwins

901

90

90

80

Test accuracy

80-

80-

70

70

70-

60

V

60-

50

60

50

40

50

20

40

60

80

100

20

40

60

80

100

20

406080100

Epochs

Epochs

Epochs

D

E

F

g01

901

80-

80-

Test accuracy (

老·部

80

70

60

70

60

T

50

60

50

40

幕

50

0

20

406080

100

0

20

40

6080

0

100200300400500600

Time (min)

Time (min)

Time (min)

2 augs, pdim=256 (100% pretraining samples)

2 augs, pdim=8192 (100% pretraining samples)

4 augs, pdim=256 (50% pretraining samples)

8 augs, pdim=256 (25% pretraining samples)

Pretrain&evalonImagenet-100

65

65

60

60

Test accuracy (%)

Test accuracy (%)

55

55

50

50

45

augs_2_pdim_512

45

augs_2_pdim_512

augs_4_pdim_512

augs_4_pdim_512

augs_8_pdim_512

augs_8_pdim_512

40

40

20

40

09

80

100

0

200

400

600

800

Epochs

Time (min)

Pretrain&evalonImagenet-100

60

60

Test accuracy(%)

55

Test accuracy(%)

55

50

50

45

45

augs_2_pdim_512

augs_2_pdim_512

augs_4_pdim_512(0.50 trainset)

augs_4_pdim_512(0.50 trainset)

40

augs_8_pdim_512(0.25trainset)

40

augs_8_pdim_512(0.25trainset)

20

40

09

80

100

0

50

100

150

200

250

300

350

Epochs

Time (min)

D.3 Empirical results on transfer learning

In this section, we present extended version of results presented in Figure 3, Figure 4 but pretraining on CIFAR-10 (or STL-10) and evaluating on STL-10 (or CIFAR-10). These results, coupled with the ones in Figure 3 Figure 4, present a strong case for the advantage of using the proposed multiaugmentation loss for better convergence as well as downstream accuracy.

PretrainonCIFAR-10andevalonSTL-10

80

80-

Test accuracy (%)

70

Test accuracy (%)

70

60

60

50

50

augs_2_pdim_256

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

40

augs_4_pdim_256

40

augs_4_pdim_256

20

40

60

80

100

0

20

40

60

80

100

120

Epochs

Time (min)

PretrainonCIFAR-10&evalonSTL-10

80

80-

70

70

Test accuracy (%)

Test accuracy (%)

60

60

50

50

40

40

augs_2_pdim_256

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

30

augs_4_pdim_256

30

augs_4_pdim_256

20

40

60

80

100

0

20

40

60

80

100

120

Epochs

Time (min)

PretrainonSTL-10&evalonCIFAR-10

85

85

80

80

Test accuracy (%)

75

Test accuracy (%)

75

.0

70

65

65

60

60

augs_2_pdim_256

augs_2_pdim_256

55

augs_2_pdim_8192

55

augs_2_pdim_8192

augs_4_pdim_256

augs_4_pdim_256

10

20

30

40

50

100

200

300

400

Epochs

Time (min)

PretrainonCIFAR-10&evalonSTL-10

70

70

Test accuracy (%)

Test accuracy (%)

60

60

50

50

augs_2_pdim_256

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

augs_4_pdim_256(0.50 trainset)

augs_4_pdim_256(0.50 trainset)

40

40

20

40

60

80

100

0

20

40

60

80

100

T

Epochs

Time (min)

PretrainonCIFAR-10&evalonSTL-10

F08

F08

70

70

Test accuracy (%)

Test accuracy (%)

60

60

50

50

augs_2_pdim_256

augs_2_pdim_256

40

40

augs_2_pdim_8192

augs_2_pdim_8192

augs_4_pdim_256(0.50 trainset)

augs_4_pdim_256(0.50 trainset)

30

augs_8_pdim_256(0.25trainset)

30

augs_8_pdim_256(0.25trainset)

20

40

60

80

100

0

20

40

60

80

Epochs

Time (min)

PretrainonSTL-10&evalonCIFAR-10

80

80

75

75

Test accuracy (%)

Test accuracy (%)

70

70

65

65

60

augs_2_pdim_256

60

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

55

augs_4_pdim_256(0.50 trainset)

55

augs_4_pdim_256(0.50 trainset)

augs_8_pdim_256(0.25trainset)

50

50

10

20

30

40

50

50

100

150

200

250

Epochs

Time (min)

E Additional Experiments probing multi-augmentation learning

E.1 Longer Pretraining to determine early stopping

Pretrain&evalonCIFAR-10

16

16

augs_2_pdim_256

augs_2_pdim_256

augs_4_pdim_256

augs_4_pdim_256

14

augs_8_pdim_256

14

augs_8_pdim_256

Test error (%)

12

Test error (%)

12

10

10

8

8

6

6

100

200

300

400

0

100

200

300

400

500

Epochs

Time (min)

E.2 SwAV-like augmentations for compute efficient multi-augmentation framework

Pretrain&evalonSTL-10

06

F06

80

80

Test accuracy (%)

Test accuracy (%)

70

70

60

60

augs_2_pdim_256

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

50

augs_4_pdim_256

50

augs_4_pdim_256

augs_8_pdim_256

augs_8_pdim_256

20

40

60

80

100

0

200

400

600

800

1000

Epochs

Time (min)

E.3 Training with full dataset with 4/8 augmentations

Pretrain&evalonCIFAR-10

06

06

Test accuracy (%)

80

Test accuracy (%)

80

70

70

60

60

augs_2_pdim_256

augs_2_pdim_256

augs_2_pdim_8192

augs_2_pdim_8192

50

augs_4_pdim_256

50

augs_4_pdim_256

augs_8_pdim_256

augs_8_pdim_256

20

40

09

80

100

0

50

100

150

200

250

Epochs

Time (min)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a theoretical formulation of the contrastive loss used in self-supervised pretrainging of vision models. (One of the assertions of the paper is that all SSL (self-supervised learning) losses are variants of the same loss). The authors propose a more compute-efficient version of the loss objective, and use the theoretic derivations of their analysis of gradient descent dynamics to propose two changes to SSL protocols in practice: the use of a higher orthogonalization regularization coupled with a lower-dimensional projection, and using more augmentations per sample. These recommendations are supported by experiments showing that a lower-dimensional projection coupled with a higher orthogonalization constant leads to higher top-1 accuracy results, and that using more augmentations per sample leads to higher downstream classifier accuracy and faster SSL convergence.\n\nSTRENGTHS:\nUnfortunately the presentation in the paper was quite difficult to understand, which made it difficult to evaluate the paper's strong points (please see 'weaknesses' and 'questions' below. However, the following strengths still seem to be present.\n- There is a fairly strong coupling between the theoretical conclusions and the experimental results, making both more credible.\n- The use of a simplified (from a computational sense) loss objective to use a sample estimate (thm 3.1) is a helpful contribution that may help improving SSL training in general.\n- The experimental results are quite good in context of the baselines, showing higher accuracy at the same data size/similar accuracy at reduced data size, when the paper's recommendations are followed.\n- The authors make it clear where they are building on past results and where the novelty of their contribution lies.\n\nWEAKNESSES:\nThe paper is very difficult to read for several reasons:\n- Key terms are not defined inside the paper, making it necessary to reference multiple additional works in-depth just to get a basic understanding of this one. These include:\n    * SimCLR\n    * Barlow Twins\n    * VicREG\n    * backward data augmentation covariance kernels $k^{DAB}$\n    * Infinite Mercer Features and Mercer Theorem\n- Theorem 3.2 is only stated informally in the text, and is not restated formally in the appendix (though theorem B.2 seems like a restatement of the informal version of the theorem). Perhaps because of this, it is not clear at all from the statement of the theorem why SGD chooses redundant directions. \n- Typos around terminology, for instance in Figure 2, it seems that $\\lambda$ should actually be $\\beta$. Also, the scale for $\\lambda$ in Panel B seems very different to that in Panel E. \n- The experiments use the new, proposed loss objective (derived from Theorem 3.1), which should agree with the more computationally-expensive ones in the limit. However, the discrepancy between using the two is not ablated, and no baselines are used that use the original loss formula, even where it might be computationally feasible  (where $m$ is lower).\n\nQUESTIONS:\n* What is the impact of using the new, proposed loss objective, as opposed to using the traditional one on the accuracies of the downstream models?\n* Why does theorem 3.2 imply the loss of orthogonality when using SGD?\n* What are the specific data augmentations used for the experiments?\n\nLIMITATIONS:\nThe limitations were adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts a theoretical study the training dynamics of self-supervised learning (SSL) methods by using an idealized loss function that resembles the Barlow Twins/VICReg loss. The idealized loss function is built on the idea of forward data augmentation (\\emph{DAF}) graph kernel. The paper shows how this loss function maybe recast in an equivalent form that is computationally friendly.  The training dynamics behavior of this loss is then used to build a hypothesis that representation learning can be improved with stronger orthogonal constraint regularization for small dimensional projectors and  multiple augmentations improve performance. The paper provides empirical evidence to support the claims made in the paper.\n\nSTRENGTHS:\n- The analytical study conducted in the paper on training dynamics is very interesting to this reader and should be of interest to the community\n- The idea of using an equivalent loss to DAF that is simpler to compute may also be of interest to the community as it has connections to other recently proposed SSL methods\n- The practical takeaways may be relevant to SSL practitioners\n\nWEAKNESSES:\n- A main weakness that I see is that I am unable to relate the faster convergence claim/proof to improved downstream accuracy. This may be a matter of presentation or perhaps a gap in the reviewer's knowledge on how alignment argument in pretext task implies better downstream performance. Would the authors be able to clear this question up during rebuttal?\n\n- The experimental setup considered in the paper is on the smaller-scale side for understandable reasons. However, this may make the statements made in the paper less interesting to empirical scientists that work in SSL. Experiments using ImageNet-100 may help with showing the claims can hold with setups used in practice. However, I do not see any experiments with ImageNet-100 in Section 4.1 (low dimensional projection)\n\nQUESTIONS:\n- How would a reader understand claims related to downstream performance using your framework that argues for faster convergence with pretext task?\n\n- Do you have any results with Barlow Twins or VICReg using ImageNet-100 at 224x224 resolution images for Section 4.1?\n\n- The equivalent loss formulation calculates the mean of embeddings (in feature space). If my understanding is correct, the the following paper [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://openreview.net/forum?id=og9V7NgOrQ) could be of interest to the authors. Would the authors provide any insight on any possible connections in the paper?\n\n- The argument about lower dimensional projections is very interesting but appears to be opposite to empirical findings made by the following works:\n  - [RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank`](https://openreview.net/forum?id=uGEBxC8dnEh)\n  - [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://openreview.net/forum?id=f3g5XpL9Kb)\nLike above, are the arguments made in the paper consistent with the empirical findings made in prior works?\n\n- There appears to be a typo in Figure 2 where $\\lambda$ is used instead of $\\beta$. Please correct if applicable \n\nI look forward to getting clarifications during rebuttal for the questions above.\n\nLIMITATIONS:\nThe authors have clearly listed their limitations. In addition, the empirical setup may not be the one used by empirical scientists in SSL so making a note may be helpful (but not necessary)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies the implicit bias of non-contrastive SSL loss and optimization, and proposes two ingredients to improve SSL learning: 1) Low-dimensional projectors can yield good representations; 2) Multiple augmentations improve kernel approximation. Further, the authors propose that In a low-data regime, using diverse & multiple augmentations can be as effective as acquiring more unique samples. All of these insights get validated on experiments of ResNet-50-based models.\n\nSTRENGTHS:\n1. The paper is theoretically motivated well based on the formulation of NC-SSL. By casting the problem into the equivalent formulation of data augmentation invariance loss and the orthogonality constraint. \n\n2. The empirical evaluation fairly meets the expectations from the alternative formulation of NC-SSL, which validates the proposed insights.\n\nWEAKNESSES:\n1. I am not sure about the definition of weak/strong orthogonality constraints. How did the authors define what is a strong or weak constraint? In other words,  what value of beta is regarded as a strong constraint?\n\n2. All the experiments are conducted on ResNet-50. Can authors do experiments on other architectures like Vision Transformers (ViTs) or other CNNs, in order to show the conclusions also transfer to other models? \n\n3. The insights are empirically validated and reasonable. However, these are the things that a deep learning practitioner could expect or imagine, especially that data augmentation can help in the low-data regime and that multiple augmentation improves the performance. These insights are also consistent in the supervised learning setting. In other words, there is no big \"surprise\" from the paper; every conclusion can be predicted even without the knowledge of SSL.\n\n4. For the conclusion that Low-dimensional projectors can yield good representations, the experiments are only conducted on CIFAR10 and STL-10, where the dimension size 8192 is considerably larger than the usual setting and even 1024 can be considered as a large projector. Can author did experiments on ImageNet-100? I am also curious the see the results on the step-wise increasing dimensions: 256,512,1024,2048,4096 instead of some big jumps in Table 1.\n\nQUESTIONS:\nPlease refer to the questions.\n\nLIMITATIONS:\nThe paper is basically an investigation into the insights revealed by the alternative formulation of NC-SSL loss. The insights get empirically validated and the overall story makes sense. \n\nHowever, I do not see any \"surprise\" from the paper except for some predictable conclusions which generally hold in deep learning. I am also not really convinced by the results in Table 1, which has big jumps in dimension sizes and only validated on CIFAR10/STL-10. These issues prevent me from giving a higher score.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new loss for self-supervised retraining that is functionally equivalent to existing methods but is computationally more efficient. The method identifies a key equivalence between existing SSL objectives (VICReg and BarlowTwins) features when employing augmentation kernel. Furthermore, the authors demonstrate that under their new loss formulation far fewer projector dimensions are needed and in-turn the efficiency is improved (time to convergence). Additionally, the authors show that multiple augmentations improve the kernel approximation, improving accuracy and convergence speed.\n\nSTRENGTHS:\n⁃\tThe paper is very well written and presented, with all contributions clearly defined and rationale within the problem setting. Figures are simple yet informative and lead to a high-quality piece of work that is understandable to a wide audience.\n\n⁃\tThe proposed methodology is novel and highly original, addressing key and ongoing concerns in the field of self-supervised learning. Moreover, the method is theoretically and empirically supported with extensive proof and experimentation, thus addressing key concerns with correctness and confidence. \n\n⁃\tThe authors present a variety of recommendations that go above simple proposition of a new methodology, thus contributing to the impact and significance of the work.\n\n⁃\tThe empirical analysis is extensive and supports all claims made, the resulting performance is a faster convergence with fewer projector dimensions.\n\nWEAKNESSES:\nI have only two minor concerns/weaknesses:\n\n⁃\tYou enforce a orthogonalization constraint on the projector, yet how is collinearity of the features avoided in practice for earlier representations, specifically after the projector is dropped? Is this even a problem?\n\n⁃\tAlternative measures for efficiency could be presented alongside convergence speed. I would expect alternative measures in both iterations per second, and memory consumption to be included at the very least if making claims of improved efficiency. Additionally, how much of a speedup can be attributed to each component I.e. the reformulation vs orthogonality constraint vs reduced projector.\n\nQUESTIONS:\n⁃\tDoes the proposed formulation lead to better improved overall performance, by that I mean improved best case downstream performance compared to original formulations?\n\n⁃\tDoes number of layers or dimensions of intermediate layers of the projector have significant impact on the observed performance?\n\nLIMITATIONS:\nThe limitations are appropriately addressed", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}