Harnessing small projectors and multiple views for efficient vision pretraining
Authors: Arna Ghosh
Kumar Krishna Agrawal
Shagun Sodhani
Adam Oberman
Blake Aaron Richards
Venue: NeurIPS 2024 poster (2024)
Keywords: representation learning
self-supervised learning
data-augmentation
learning dynamics
sample efficient SSL
compute efficient SSL
Primary area: machine_vision
TL;DR: We characterize the implicit biases imposed by learning dynamics along with architecture and loss function for self-supervised representation learning, leading to practical recommendations for improving its sample and compute efficiency.

Abstract: Recent progress in self-supervised (SSL) visual representation learning has led to the development of several different proposed frameworks that rely on augmentations of images but use different loss functions. 
However, there are few theoretically grounded principles to guide practice, so practical implementation of each SSL framework requires several heuristics to achieve competitive performance.
In this work, we build on recent analytical results to design practical recommendations for competitive and efficient SSL that are grounded in theory. 
Specifically, recent theory tells us that existing SSL frameworks are actually minimizing the same idealized loss, which is to learn features that best match the data similarity kernel defined by the augmentations used.
We show how this idealized loss can be reformulated to a functionally equivalent loss that is more efficient to compute.
We study the implicit bias of using gradient descent to minimize our reformulated loss function, and find that using a stronger orthogonalization constraint with a reduced projector dimensionality should yield good representations.
Furthermore, the theory tells us that approximating the reformulated loss should be improved by increasing the number of augmentations, and as such using multiple augmentations should lead to improved convergence.
We empirically verify our findings on CIFAR, STL and Imagenet datasets, wherein we demonstrate an improved linear readout performance when training a ResNet-backbone using our theoretically grounded recommendations.  
Remarkably, we also demonstrate that by leveraging these insights, we can reduce the pretraining dataset size by up to 2$\times$ while maintaining downstream accuracy simply by using more data augmentations. 
Taken together, our work provides theoretically grounded recommendations that can be used to improve SSL convergence and efficiency.

Harnessing small projectors and multiple views for efficient vision pretraining
Arna Ghosh
Kumar Krishna Agrawal +2
Shagun Sodhani
Adam M. Oberman'
Blake A. Richards 1445
Abstract
Recent progress in self-supervised (SSL) visual representation learning has led the development several different proposed frameworks that rely on augmenta- tions of images but different loss functions. However, there few theoretically grounded principles to guide practice, so practical implementation of each SSL framework requires several heuristics to achieve competitive performance. In this work, we build on recent analytical results to design practical recommendations for competitive and efficient SSL that grounded in theory. Specifically, recent theory tells us that existing SSL frameworks are actually minimizing the same idealized loss, which is to learn features that best match the data similarity ker- nel defined by the augmentations used. We show how this idcalized loss can be reformulated functionally equivalent loss that is more efficient to compute. We study the implicit bias of using gradient descent to minimize our reformulated loss function, and find that using stronger orthogonalization constraint with a reduced projector dimensionality should yield good representations. Furthermore, the theory tells us that approximating the reformulated loss should be improved by increasing the number of augmentations, and as such using multiple augmenta- tions should lead to improved convergence. We empirically verify our findings on CIFAR, STL and Imagenet datasets, wherein we demonstrate an improved linear readout performance when training a ResNet-backbone using our theoretically grounded recommendations. Remarkably, we also demonstrate that by leveraging these insights, we can reduce the pretraining dataset size by up to 2x while main- taining downstream accuracy simply by using more data augmentations. Taken together, our work provides theoretically grounded recommendations that can be used to improve SSL convergence and efficiency.
Introduction
Unsupervised representation learning, i.c., learning features without human-annotated labels, is critical for progress in computer vision. Modern approaches, grouped under the self-supervised learning (SSL) umbrella, build on the core insight that similar images should map to nearby points in the learned feature space often termed as the invariance criterion. Current SSL methods can be broadly categorized into contrastive and non-contrastive algorithms, based on whether they formulate their loss functions using negative samples not, respectively.
"Equal Contribution, 1 Co-senior authorship. Correspondence: blake.richardse @ mcgill.ca
Mila Quebec AI Institute & Computer Science, McGill University. Montréal, QC, Canada
PUC Berkeley, CA, USA, Meta FAIR, Toronto, ON, Canada
3 Mila Quebec AI Institute & Mathematics and Statistics, McGill University. Montréal, QC, Canada
'Neurology & Neurosurgery and Montreal Neurological Institute, MeGill University, Montréal, QC, Canada
CIIAR Learning in Machines & Brains Program, Toronto, ON, Canada
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ideal from theory
augmentations (A)
A
E
Ag
A2
AxA
features (D)
E
O
E
NxDxA
proposal
datapoints (N)
(0 Toh,, = O
Figure 1:
Design of existing SSL algorithms relies heuristics. (A) Augmentation graphs common vision pretraining, providing generalizable features for downstream tasks. (B) We propose an equivalent loss function for SSL pretraining that recovers the same eigenfunctions more efficiently than existing approaches.
Despite this difference in their loss formulations, recent theoretical work has established an equiva- lence between the contrastive and non-contrastive SSL frameworks [19]. This work shows that these different SSL formulations are ultimately minimizing a loss that encourages the learning of features that best match the data similarity kernel defined by the augmentations used. However, this notion of theoretical equivalence holds only in the limit of ideal pretraining settings, i.e. with access to infinite data and compute budget, and the feature learning behavior of different SSL algorithms in practical scenarios is still not well understood. Therefore, researchers often use empirically driven heuristics that are theoretically ungrounded to design successful applications, such as (i) a high-dimensional
projector head for non-contrastive SSL or (ii) the use of two augmentations per image [3 Moreover, existing SSL algorithms are extremely data-hungry, relying large-scale datasets [33] data en-
gines [30] to achieve good representations. While this strategy works exceptionally well in data-rich settings (like training on natural-immn -mmuual,,,eess,..... it is not viable in data-constrained settings (like medical imaging), where samples are relatively scarce.
With these challenges in mind, the primary focus of this work is to develop theoretically grounded recommendations for improving the effectiveness and efficiency of feature learning, both with respect to the required compute budget well as data points. Like any unsupervised representation learning algorithm, features learned through SSL depend on three factors: (i) implicit bias of the architecture, (ii) explicit invariance imposed by data augmentations, (iii) implicit bias of the learning rule. While previous works predominantly studied the role of the model architecture capacity and loss function, and their interplay with data augmentations [ 44. our approach broadens this perspective by also considering the role of the learning rule (gradient descent) in optimizing these loss functions. Specifically, we extend the previous theoretical findings [4]] that unified the desiderata of different SSL algorithms. We reformulate the idcalized unifying loss to propose functionally equivalent loss that is more compute-efficien (see Figure TT. Based on our loss formulation, we provide two practical recommendations that can help improve the efficiency of SSL pipelines while maintaining good performance. First, we show that optimizing the reformulated loss using gradient descent can often reduce the orthogonality among the learned embeddings, thereby leading to an inefficient use of the projector network's capacity. Consequently, we recommend using stronger orthogonalization constraint to eliminate the requirement of high-dimensional projector heads, thereby significantly reducing the parameter overhead good feature learning. Second, we show that increasing the number of augmentations leads to a better estimate of the data similarity kernel. Consequently, recommend using more augmentations to improve optimization convergence and learn better features earlier in training.
We empirically verify our theoretically grounded recommendations using the popular ResNet back- bone on benchmark datasets: CIFAR, STL and Imagenet. Strikingly, we show that our multi- augmentation approach can learn good features even with hulf of the samples the pretraining dataset. Our recommendations provide path towards making SSL pretraining more data and compute-efficient without harming performance and could unlock massive performance gains in data-constrained setups. In summary, core contributions follows:

---

Efficient SSL loss formulation: We propose an functionally equivalent and compute- efficient formulation of the SSL desiderata that yields the eigenfunctions of the augmentation- defined data similarity kernel.
Based on our loss formulation and the implicit bias of gradient descent in optimizing this loss, provide a mechanistic explanation for the role of projector dimen- sionality and the number of data augmentations. Consequently, we empirically demonstrate that low-dimensional projector heads are sufficient and that using more augmentations leads to learning better representations.
Role of heuristics:
Data efficient SSL: Leveraging the convergence benefits of the auglii-uugmetttiion SSL framework, we empirically demonstrate that can learn good features with significantly smaller datasets (up to 2x) without harming downstream linear probe performance.
2 Preliminaries
Existing SSL approaches in computer vision recent years machine learning researchers have developed a number of effective approaches for learning from data without labels. The most popular approaches use augmentations of data points targets for themselves. One of the first was a Simple framework for Contrastive Learning (SimCLR), which relied on an infoNCE loss with augmentations of an image as positive targets and augmentations of other images as negative samples (to construct the contrastive loss) [1]. Other works have relied on non non-contrustive approaches, notably Barlow Twins and VICReg [5]. BarlowTwins, which was inspired by the ideas of the neuroscientist Horace Barlow (cite), also uses augmentations of images, but it instead aims to optimize the covariance structure of the representations in order to reduce redundancies in the feature space [43]. Variance Invariance Covariance Regularization (VICReg) was modification of Barlow Twins that added variance term the loss in order to ensure that every feature dimension has a finite variance [5]. In this paper will focus non-contrastive methods like BarlowTwins and VICReg, but in line with previous work 44 we also consider how these approaches relate to contrastive methods like SimCLR.
43
Formalizing the self-supervised learning problem Now, we will formalize the unsupervised representation learning problem for computer vision. In particular, we assume access to dataset D = {x1,722 "2,, N.) with 2; E RRP consisting of unlabeled images. The objective to learn a d-dimensional representation (d < p) that is useful across multiple downstream applications. We focus on learning the parameters of a deep neural network fo E Fe, using the nulii-auugmenatiion SSL framework, wherein multiple views of image are used to optimize the pretraining loss function, Epretrain (fa, D)
Non-Contrasio Self-Supervieed Learning (NC-SSL) algorithms impose invariance to data aug- mentations, while imposing regularization on the geometry of the learned feature space. More generally, Epretrain can be formulated with two terms (i) Linvariance: to learn invariance to data augmentations and (ii) Leollapse: regularization to prevent collapsing the feature space to a trivial solution.
Epretrain = Linvariance + 3Leclllasee
(1)
where 3 denotes hyperparameter that controls the importance of the collapse-preventing to the invariance term. This formulation separates features that are invariant to the augmentations from those that sensitive to them. Intuitively, the ideal feature space is more sensitive to semantic attributes (e.g. "that's a dog")) and less sensitive to irrelevant attributes (e.g. "direction the dog is facing"), facilitating generalization new examples.
relative
Data Augmentation graph was introduced by Haochen el al. 22 to analyze contrastive losses, like SimCLR [12]. Briefly, we define graph GAA, W) that captures the relationship between images derived from all possible data augmentations. The vertex set (A,, PA) is each augmented sample in a dataset, X. and the adjacency matrix. W. denotes the similarity between pairs of vertices. Let zo be an image in X, and let z = M(x)) E AA be a random data augmentation the image, Zg. We define the probability density of reaching from Zo via choice of mapping M:
p(2 zu) = P(z = (
(2)
Since the mapping is not generally invertible (e.g., cropping), we observe that p(ro | |) a p(2 mo). Using this definition, we now formally define the strength of the edge between nodes 2, 2 A of the
3

---

augmentation graph as the joint probability of generating augmentations from the same image To ~ px. Notably, the edge strength of the peggre-normalizeen augmentation graph is equivalent to the data similarity kernel, defined in [4]. Formally,
p(æ æo) p(z 20) p(z)
DAF (x,
Was
Exoxpx
p(æ)
The magnitude of Was captures the augmentation--deinet similarity between v and z. A higher value of waz indicates that both patches are more likely to come from the same image and, thereby, are more similar.
The desiderata of different SSL algorithms can be understood as learning features F that best capture A.D (I, i.c. F(x)" Fz)) z k (xx z).. Recent theoretical work has shows that different SSL losses can be formulated as special cases of the objective function that recovers the top-d eigenfunctions of DAP (x, 2) 44).
(F) Es [( (s, P(u)) F())²²]
Note that all rotations of that don't change its span define an equivalence class of solutions to Equation 4 and make no difference for the downstream generalization of linear probe. Based on this insight, we define an equivalence among learned feature spaces:
Definition 2.1. Let F(z) = (f.).....((()) be d-dimensional feature vector (a vector of functions). Define the subspace
V V(F) (h X R M(x) w F(x), w R
(5)
be the span of the components of F. Given an n-dimensional feature vector, G(x) (gr (x) (z)) we say the features G and F are equivalent, if = =VG),
3 Implicit bias of non-contrastive SSL loss and optimization
We extend the recent theoretical results 44 loss function of the SSL desiderata that yields equivalent features, i.c, the functions spanning the eigenfunctions of the sugmentation-definee data similarity kernel, D.F. Furthermore, we study the role of gradient descent in optimizing this loss function and uncover selection and primacy bias in feature learning. Specifically, we find that gradient descent tends learn the dominant eigenfunctions (eigenfunctions corresponding to larger eigenvalues) earlier during training, and often over-represents these eigenfunctions under weak orthogonalization constraints.
propose a compute- -fffcciiet reformulation of the
Consequently, we propose employing stronger orthogonalization constraint during optimization when using low-dimensional projector to ensure that learned features are equivalent those learned with high-di nigh-dimensional projector. Furthermore, we argue that using more augmentations improves our sample estimate of DAA thereby aiding the cigenfunction optimization problem. We dedicate the rest of this section to highlight our key theoretical insights, and practical recommendations that follow them.
3.1 Features in terms of data augmentation kernels
Let us define a kernel operator, Tk, for positive semi-definite data augmentation kernel, DAF fx)( = Es-px [k(2, (x)]
(6)
such that Equation can be equivalently written as (Equation 5 of 44
(.,(F) -F)
(7)
We can now use Mercer's theorem to factorize DAF into corresponding spectral features G: X + &
(where Eg represents square summable sequences) 15 16 31.. However, note that computing &DA (or T3) is expensive as it requires computing the overlap among all augmentations of every pair of data points. Instead of computing the eigenfunctions of T₃ directly, we propose using an alternative operator TM:
TM fx) Ex [f(x))]
Σ [p(zo fx(o)
(8)

---

which averages the values of the function, f. over the augmented images (a) of the data, We show that T Tas is equivalent to Tg, and therefore T'r and T5 have shared eigenfunctions.
Theorem 3.1. Let ((x) be the infinite M the backward data augmentation covariance
kernels, K Let F(x) = (fi fN. (2)) be the features given by minimizing the following data augmentation invariance loss
Nv.
I(F)=
Σ
ITusi filli((xx>> subject to (J+ = 6,j
(9)
1=1
which includes the orthogonality constraint. Then, V(F) C CVG), limn, V(F)= VG)).
As shown in the Appendix B L(F) is equivalent to a constrained optimization formulation of the MarlowTwins loss. Furthermore, L(P) with the additional constraint that (fi 2 y Vi E (1,2...N) 3 is the constrained optimization formulation of the VICReg loss.
3.2 The implicit bias of gradient descent
Next, we investigate how the use of gradient descent for optimizing L(F) influences the characteristics of the learned feature space, V(F), Given the similarity in its form with that of the BarlowTwins loss, we build on recent findings that demonstrate the sequential nature of lcarning eigenfunctions when optimizing the Barlow BarlowTwins loss under a strong orthogonalization regularization 36 Since
strong orthogonalization seldom used in practice due to instabilities in training [5) 43 we believe studying the learning dynamics under weak orthogonalizaaton regularization (i.e. low values of [3 in Equation (T) is more relevant to provide recommendations for practitioners.
Theorem 3.2. (Informal) Let us denote the span of the feature space at initialization as ((Fo) and
after training as ((Fr). For small initialization of the network's 's weights, the alignment of V (FF) with the eigenfunctions Th depend on two factors: (i) alignment of ((Fo) with the eigenfanctions of (ii) singular values of Th-
Under weak orthogonalization constraints, the network tends to learn features that are strongly aligned with eigenfunctions corresponding to large singular values. We refer to this property as the "selection" bias of gradient descent, wherein gradient descent selects certain eigenfunctions based on the corresponding singular values. This selection bias leads to redundancy among the learned feature space, thereby reducing the effective dimensionality of the network's output space compared to its ambient dimensionality. We will leverage this finding to improve the parameter overhead of good feature learning using Barlow Twins and VICReg loss frameworks.
3.3 Takeaway 1: Low-dimensional projectors can yield good representations
Given the proximity of the formulation of Equation (9) to that of BarlowTwinns and VICReg losses, we will leverage existing heuristics that have been shown to work in practice. As such, and VICReg frameworks call for high-dimensional projectors while using weak orthogonalization regularization to facilitate good feature learning. We know, from Theorem [3. that the eventual goal of these frameworks is to learn the eigenfunctions of the underlying data similarity graph. For example, since the intrinsic dimensionality of Imagenet is estimated to be ~ 40 [32], is not unreasonable to expect that the span of desired features would be of similar dimensionality. It is, thus, intriguing that the current practice would suggest using an 8192-dim projector head to capture the intricacies of the corresponding augmentation-define data similarity kernel. This discrepancy can be explained by analyzing the learning dynamics, as in Theorem 3.2 Notably, a high- nigh-dimensional projector is likelier to have greater initialization span than its low now-dimensional counterpart, thereby increasing the alignment between V ((F)) and relevant eigenfunctions of Tg. We hypothesize that a stronger orthogonalization constraint for -iwdenensional projectors rectify this issue, reducing the redundancy in the network's 's output space and rendering it sufficient for good feature learning.
3.4 Takeaway 2: Multiple augmentations improve kernel approximation
By comparing the invariance criterion formulation in the standard Barlow and VICReg losses to Equation 7 can be inferred that current practices use a sample estimate of Tü. Using only two augmentations per sample yields a noisy estimate of Ti. yielding spurious eigenpairs 41 (see
5

---

Appendix C. These spurious eigenpairs add stochasticity in the learning dynamics, and coupled with Theorem 3.2 increase the redundancy in the learned feature space [T]]. We hypothesize that improving this estimation error by increasing the number of augmentations could alleviate this issue and improve the speed and quality of feature learning.
Of course, increasing the number of augmentations (m) in the standard BarlowTwing and VICReg loss improves the estimate of T but comes with added compute costs straightforward approach would involve calculating the invariance loss for every pair of augmentations, resulting in ((m²) operations. However, Theorem 3.1 proposes an alternative method that uses the sample estimate of TM. thereby requiring only (m operations, and hence is computationally more efficient while yielding functionally equivalent features (see Appendix [B). In summary, Theorem [3. establishes mechanistic role for the number of data augmentations, paving the way for computationally efficient multi-augmentation framework:
m ΣΣ
No
T((F) E Rx
[a(x) [1(x))122((x))
subject to
(Si. f1)ex = 6,g (10)
where [1(x) 1 プラ fe(xz)) is the sample estimate of TM (z)).
4 Experiments
In our experiments, seek to (i) provide empirical support for our theoretical insights and (ii) present practical primitives for designing efficient SSL routines. Since our proposed loss function closest to the formulation of Barlow Tarlowwwins/VIRRgg our proposal to these baselines. In summary, with extensive experiments across learning algorithms (BarlowTwin & VICReg) and training datasets (CIFAR-10) 10, STL-10 & Imagenet-100), establish the following:
present empirical evidence comparing
low-dimensional projectors can yield good representations.
multi-augmentation improves downstream accuracy, well as convergence rate.
multi- multi-augmentationn improves sample efficiency in SSL pretraining, i.e., recovering similar performance with significantly fewer unique unlabelled samples.
We evaluate the effectiveness of different pretraining approaches using image classification the downstream task. Across all experiments, we pretrain Resnet feature encoder backbone for 100 epochs (see Appendix E. for longer pretraining results) and use linear probing on the learned representations]] All runs are averaged over 3 seeds; error bars indicate standard deviation. Other details related to optimizers, learning rate, etc., are presented in the Appendix D
Experiment Setup:
4.1 Low-dimensional projectors can yield good representations
Barlow Twins optimal g"
VICReg
pdim
fixed
fixed 8
optimal 3
64
73.6 + 0.9
82.1 + 0.2
68.9 + 0.2
81.9 + 0.1
256
75.9
83.4 0.4
75.3 0.2
81.9 + 0.3
1024
1.0
0.3
0.9
82.5 0.9
80.4 1.5 Table 1: Optimizing for orthogonality appropriately allows low-dimensional projectors match the performance (on CIFAR-100 10) of much higher- digber-dimensional projectors.
8192
82.2
0.4
1.5
Existing works recommend using high- nigh-dimensional MLPs as projectors (e.g.. d=8192 for Imagenet in 5 43 ), and show significant degradation in performance when using lower-dimensional projectors for a fixed redundancy coefficient (3). To reproduce this result, we run grid search to find the optimal coefficient (86192) for 8192 and show that performance progressively degrades for lower if the same coefficient 19 is reused for d (64, 128, 256, 512, 1024, 2048, 4096, 8192).
!Code: https///gitubb.oo/uuuuanrsnnmn/

---

A
CIFAR-10 Barlow Twins
B
STL-10 BarlowTwing
Imagenet- 100 Barlow Twins
Test accuracy (%)
84
86
62
80
58
82
76
80
54
Optimal B
Optimal Bd
Optimal B.
Fixed B (0.0004)
Fixed (0.0004)
Fixed (0.001)
72
50
D
1e3
1e4
E
1e2
163
1e4
F
1e3
1e4
Optimal B
1e-2
1e-2
1e-2
16-3
16-3
1e-3
1e2
1e3
1e4
1e2
1e3
1e4
1e3
1e4
Projector Dimensionality
Projector Dimensionality
Projector Dimensionality
Figure 2: Low-dimensional projectors can yield good representations. We demonstrate that using higher orthogonality constraint, B, for lower projector dimensionality can achieve similar performance over wide range of projector dimensions (d).
Our insights in Section 3.3 suggest now-dimensional projectors should recover similar performance with appropriate orthogonalization. To test this, find the best 3 by performing grid search independently for each d E {64, 128, 256, 512, 1024, 2048, 4096, 8192]. As illustrated in Figure using dimensioni projectors yield features with similar downstream task performance, compared to the features obtained using -igh-dimensional projectors. Strikingly, we also observe that the optimal Ba 1/d, which aligns with our theoretical insights.
Recommendation: Start with low-dimensional projector, using 3 O(2), and sweep (pdim d, B O (t)) if needed.
4.2 Multiple Augmentations Improve Performance and Convergence
Although some SSL pretraining approaches, like SWav 10 incorporate more than two views, the most widely used heuristic in non-contrastive SSL algorithms involves using two views jointly encoded by shared backbone. line with this observation, our baselines for examining the role of multiple augmentations use two views for computing the cross-correlation matrix.
A
CIFAR-10 BarlowTwins
B
STL-10 BarlowTwins
C Imagenet-100 BarlowTwins 65
90
90
Test accuracy (%)
60
80
55
70-
70
50-
2 augs, pdim=256 2dg
augs, pdim==26 augs, pdim==812
60
2 augs, pdim=256 augs, pdim=8192
45-
4 augs, pdim==25
2 augs, pdim=8192
augs, pdim=256
augs, pdim==256
8 __ags___pimeee pdim=256
50
40-
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Epochs
Epochs
Epochs
Figure 3: Using multiple augmentations improves representation learning performance and conver- gence. (A-C) Across Barlow Twins for CIFAR-10, STL-10 and Imagenet- 100 pretraining, using augmentations instead of 2 helps improve performance. Please see Appendix E.3 for results.
To demonstrate the role of multiple augmentations in pretraining, we adapt the invariance criterion of Barlow SellowTwinsVVICRe. to be in line with Equation I0 In particular, for #augs E {2, 4, 4),
7

---

augs
pdim
Barlow Twins
VICReg
Time (min)
Time (min)
8192
99.36 + 0.01
94.36 + 0.01
256
62.34 + 0.06
51.73 + 0.04
256
43.09 + 0.20
39.02 + 0.05
Table 2: Using multiple augmentations yields faster convergence, with reduced time to reach baseline performance on CIFAR- 10, i.c. performance feature encoder pretrained with an 8192-dim projector and 2 augmentations.
pretrain a Resnet-50 encoder with our proposed loss. Building on the insight from the previous section, we use 256-dimensional projector head for all mulli-augmentationn experiments.
In Figure B we track the downstream performance of the pretrained models across training epochs. For performance evaluation, we use the linear evaluation protocol as outlined by [13] Figure A C) shows that pretraining with multiple augmentations outperforms the 2- 2-augmentation basehin. Furthermore, we observe that the four-augmentation pretrained models converge faster (both in terms of the number of epochs and wall-clock time) than their two-augmentation counterparts (see Figure D)F)). Additionally, we show in Appendix F.2 that our framework can also be applied to multi-augmentation settings like SWaV, where not augmentations are of the same resolution.
Recommendatation: Using multiple augmentations 2) likely improve convergence as well downstream accuracy.
4.3 Sample Efficient Multi--uggmenaation Learning
Data Augmentation can be viewed as form data inflation, where number of training samples increased by & (for k augmentations). In this section, we examine the role of improving sample efficiency. In particular, we are interested in understanding if the same performance can be achieved with fraction of the pretraining dataset, simply by using more augmentations.
in
A 90
CIFAR-10 BarlowTwing
B 90
STL-10 BarlowTwill
C Imagenet-100 100 Barlow BarlowTwins
Test accuracy (%)
60
80
55
70-
70-
50
60
60
45
50
40
50
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Epochs
Epochs
Epochs
augs, pdim=256 (100% pretraining samples)
augs, pdim=8192 (100% pretraining samples) augs, pdim=256 (25% pretraining samples)
augs, pdim=256 (50% pretraining samples)
Figure Multi-augmentatee improves sample efficiency, recovering similar performance with signif- icantly fewer unique samples in the pretraining dataset. Across BarlowTwins pretraining on CIFAR- 10, STL-10 and Imagenet- 100 for the same effective dataset size (#augs X #unique sampppeeesmmpes))..... using more patches improves performance at the same epoch (A-C). However, tradeoff exists wherein more data augmentations fail to improve performance in the scarce data regime.
augs
pdim
Percentage of Dataset
BarlowTwins Time (min)
VICReg Time (min)
2
8192
100 %
63.43 + 0.02
66.05 + 0.01
2
256
100 %
39.52 0.04
40.64 + 0.04
256
%
28.25 0.01
32.39 0.01
8
256
25 %
27.74 0.01
34.76 1: 0.01
Table 3: Time required to pass 80% accuracy dataset, while using multiple augmentations. See Figure 5 for further discussion.
10
when pretraining fraction of the

---

To examine the relation between the number of augmentations and sample efficiency, we fixed the effective size of the inflated dataset. This is achieved by varying the fraction of the unique samples in the pretraining dataset depending on the number of augmentations & E {2, 4, 8). e.g., we use 50% of the dataset for views. We then evaluate the performance of the pretrained models on the downstream task, where the linear classifier is trained on the same set of labeled samples. Strikingly, Figure ! shows that using multiple augmentations can achieve similar (sometimes even better) performance with lesser pretraining samples, thereby indicating that more data augmentations can be used for feature learning to compensate for smaller pretraining datasets.
Recommendation: In a low-data regime, using diverse & multiple augmentations can be as effective as acquiring more unique samples.
5
Related Work
Self-Supervised Pretraining requires significant compute resources and most practitioners rely on empirical heuristics (see SSL cookbook [3] for summary). While recent advances in SSL. theory explore learning dynamics in linear (or shallow) models [39] 40), with a focus on understanding dimensionality collapse [20 24). the theoretical underpinnings of most of the heuristics considered essential for good feature learning, are missing.
Contrastive SSL has received more theoretical attention, owing to its connection with metric learning and noise contrastive estimation 4 25 (9]. In particular, HaoChen al. [22) provide a theoretical framework for the SimCLR loss from an augmentation graph perspective, which leads to practical recommendations, Subsequently, Garrido ef al. [99] establish a duality between contrastive and non-contrastive learning objectives, further bridging the gap between theory and practice.
Non-contrastive SSL algorithms' theoretical foundations have received more attention recently 19, 44). Prior works 12
18 19] have demonstrated that with modified learning objectives, low- dimensional projectors yield representations with good downstream performance. Similarly, previous works have demonstrated notable performance boosts when using a multi-patch framework in contrastive [17] and non-contrastive SSL [T0 42 However, the theoretical basis for the benefits and trade-offs of either low-dimensional projectors or multiple augmentations is largely unclear. It is worth noting that Schaeffer et al. [34] present an information-thhoretiic perspective of the recently proposed non-contrastive SSL. loss that leverages multiple augmentations, namely MMCR [42), but the computational advantages of using multiple augmentations on the learning dynamics is an active area of research.
Deep Learning theory has made significant strides in understanding the optimization landscape and dynamics of supervised learning [. In concurrent works [9] 44], the interplay between the inductive bias of data augmentations, architectures, and generalization has been explored from a purely theoretical perspective, establishing an equivalence among different SSL losses [4]. Furthermore, Simon et al. (36) used more straightforward formulation of the BarlowTwins loss and investigated the learning dynamics in linearized models for the case when the invariance and orthogonalization losses have equal penalties. Although such a setting rarely used in practice, their approach serves as an inspiration for our work in studying the learning dynamics of non-contrastive SSL losses.
6 Discussion
Summary: Our work builds existing theoretical results that establish equivalence among different SSL. frameworks, and proposes a compute-efficient reformulation of the common SSL loss. Using this loss reformulation and study of the optimization dynamics, we proposed practical recommendations to improve the sample and compute efficiency of SSL algorithmss Specifically, we recommended low-dimensional projectors with increased orthogonality constraints and multi- augmentation frameworks, and we verified the effectiveness of these recommendations empirically. It worth noting that our multi- aulttiaaugentttiion formulation improves the efficiency of learning without altering the desiderata of SSL, i.e. the network learns the same feature space using our proposed multi-augmeetationn framework as with the original SSL formulation in the limit of infinite pretraining budget. To demonstrate this equivalence between the original SSL loss and our proposed version, we show in Appendix [. that longer pretraining on the 2-augmentation loss leads similar downstream performance as the muuti-augmentttioon versions (4 and 8 augmentations).

---

We also showed that the augiliitagmetttiionn framework can be used to learn good features from fewer unique samples in the pretraining dataset simply by improving the estimation of the data augmentation kernel. This result has direct implications on improving the Pareto frontier of samples-vs perfforaseee for SSL. pretraining, wherein we can achieve hetter downstream performance when limited number of samples are available in the pretraining dataset.
Pareto Optimal SSL In the context of sample efficiency, training a model using two augmenta- tions with different fractions of the dataset leads to a natural Pareto frontier, i.e., training on the full dataset achieves the best error but takes the most time (Baseline (2-Aug)). Our extensive experiments demonstrate that using more than two augmentations improves the overall Pareto frontier, i.e., achieves better convergence while maintaining accuracy (Multi-Aug). Strikingly, as shown in Figure [5] we observe that we can either use a larger pretraining dataset or more augmentations for a target error level. Therefore, the number of augmentations can be used as a knob to control the sample efficiency of the pre- training routine.
Multi-Aug
50
Baseline (2-Aug)
4-A
Error %
40
8-Aug
30
20
10
25
50
75
100
Runtime (min)
Figure 5: Using > 2 augmentations with a frac- tion of the dataset improves overall Pareto fron- tier, speeding runtime up to ~ 2x.
Connections Downstream performance: While our core theoretical results are aimed at acceler- ating convergence of the SSL loss itself, our empirical results highlight an improved downstream task performance earlier during pretraining. While this discrepancy might seem counter-intuitive at first, it is worth noting that the SSL loss inherent influences downstream performance as it encourages clustering of semantically similar images in the representation space. Such clustering properties in the representation space facilitates easier classification through methods k-nearest neighbors or linear decoding for a large number of tasks that rely on the semantic content of images. Previous works 18 20 37 have discussed in detail how certain geometric properties of the learned representation space are connected to the linear classification performance for arbitrary decision boundaries, in expectation. However, in-depth analysis of downstream tasks that more amenable to linear decoding from the learned SSL representation space requires framing metrics of alignment between the pretraining objective (SSL desiderate) and the downstream task labels, and is active area of research.
Open Questions: Looking ahead, it would be exciting to extend this analysis to other categories of SSL algorithms, such as Masked AutoEncoders (MAE). Furthermore, our insights provide opportu- nities to explore sample-efficient methods that rely on less data, which is particularly important in critical domains such as medical imaging, where data is often relatively scarce and expensive. On different note, it is intriguing that animals often spend extended periods of time exploring novel objects, likely to gain multiple views of the object 6 28). Given the theoretical underpinnings of the computational benefits of multi-augmentateee multi-augment SSL outlined in work, it would be exciting to develop models of biological learning that leverage these insights and enable sample-efficient continual learning in similar environments.
Limitations: Our algorithm relies on multiple augmentations of the same image to improve the estimation of the aata-augmentation kernel. Though this approach speeds up the learning process, it also adds some extra computational overhead, which means that the impact of faster learning on wall-clock time is less than might be hoped for. One way to mitigate the effects of this limitation would be to scale up to a multi-GPP setting, since the computations for each augmentation can be on a separate GPU in parallel. This could help ensure that the improved speed of learning directly translates to a significantly reduced wall-clock time for training.
The goal of our work is to advance the general field of visual representation learning. Although there are potential downstream societal consequences of our work, we feel there no direct consequences that must be specifically highlighted here.
Impact Statement:
10

---

Acknowledgements
The authors would like thank Arnab Kumar Mondal for insightful discussions that helped shape the project's scope, Colleen Gillon for aesthetic contribution to the figures and Florian Bordes for helping setup the FFCV-SSL library. The authors are also grateful to Aidan Sirbu, Chen Sun, Jonathan Cornford, Roman Pogodin, Zahraa Chorghay and the anonymous reviewers whose comments and suggestions have significantly enhanced the quality and presentation of the results. This research was generously supported by Vanier Canada Graduate Scholarship (A.G.); NSERC (Discovery Grant: RGPIN-2020-05 105; RGPIN-2018 0220018--4211 Discovery Accelerator Supplement: KGPAS--020-0002
Arthur B. McDonald Fellowship: 566355-2022), Healthy Brains, Healthy Lives (New Investigator Award: 2b-NISU-8), and CIFAR Learning in Machines & Brains Program (B.A.R.): Canada CIFAR AI Chair program (A.O. & B.A.R.). The authors also acknowledge the material support of NVIDIA in the form of computational resources, as well as the compute resources, software and technical help provided by Mila (mila.quebec).
References
[1] Madhu Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-444 446, 2020.
[2] Kumar K Agrawal, Arnb Kumar Mondal, Amn Ghosh, and Blake Richards. a-req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. Advances Neural Information Processing Systems, 35:17626-1768 17638, 2022.
[3] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et A cookbook of self-supervised learning. arXiv preprini arXiv:2304. /2210, 2023.
[4] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NearlPS, 2022.
[5] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianceinnvarine--cvvare ization for self-supervised learning. arXiv preprint arXiv:2 2021.
regular-
[6] Daniel E Berlyne. Novelty and curiosity as determinants of exploratory behaviour. British journal of psychology, 41(1):68, 1950.
[7] Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards democratizing joint- embedding self-supervised learning, 2023.
[8] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[9] Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssl interplay: Augmentations, inductive bias, and generalization. In International Conference Machine Learning, pages 3252-3298. PMLR, 2023.
[10] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning visual features. Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[11] Feng Chen, Daniel Kunin, Atsushi Yamamura, and Surya Ganguli. Stochastic collapse: How gradient noise attracts sgd dynamics towards simpler subnetworks. arXiv preprint arXiv:2306.0422. 04251, 2023.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[13] Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance vicreg: Bag of self-supervised image patch embedding. arXiv preprint arXiv:2206.08954 08954, 2022.
11

---

[14] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[15] Zhijie Deng, Jiaxin Shi, Hao Zhang, Peng Cui, Cewu Lu, and Jun Zhu. Neural eigenfunctions are structured representation learners. arXiv preprint arXiv:2210.126377 2022.
[16] Zhijie Deng, Jiaxin Shi, and Jun Zhu. Neuralef: Deconstructing kernels by deep neural networks. In International Conference on Machine Learning, pages 4976-4992, PMLR, 2022.
[17] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representa- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
[18] Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning, pages 10929-10974. PMLR, 2023.
[19] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality hetween contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference Learning Representations, 2023.
[20] Arna Ghosh, Arna Kumar Mondal, Kumar Krishna Agrawal, and Blake Richards. Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05880, 2022.
[21] I GGobberg. Classes of linear operator theory. Advances and Applications, 49, 1990.
[22] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 14:5000-5501, 2021.
[23] Lajos Horváth and Piotr Kokoszka. Inference for functional data with applications, volume 200. Springer Science & Business Media, 2012.
[24] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2 10.09348, 2021.
[25] Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- view.net, 2023.
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[27] Guillaume Leclere, Andrew Ilyas. Logan Engstrom, Sung Min Park, Hadi Salman, and Alek- sander Madry. ffev.
2022. commit 3a12966
[28] Marianne Leger, Anne Quiedeville, Valentine Bouet, Benoît Haelewyn, Michel Boulouard, Pascale Schumann- -Brd, and Thomas Freret. Object recognition test in mice. Nature protocols, 8(12):2531-2577, 2013.
[29] Yazhe Li, Roman Pogodin, Danica Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. Advances in Neural Information Processing Systems, 34:15543-15556, 2021.
[30] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
[31] David Pfau, Stig Petersen, Ashish Agarwal, David GT Barrett, and Kimberly L. Stachen- feld. Spectral inference networks: Unifying deep and spectral learning. arXiv preprint arXiv:1806.02215, 2018.
12

---

[32] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations, 2021.
[33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2555 2015.
[34] Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, YueYeon Chung, et al. Towards an improved understanding and utilization of maximum manifold capacity representations. arXiv preprini (KKv::2406.033666 2024.
[35] Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning, pages 19847-19878. PMLR, 2022.
[36] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning, arXiv preprint arX1v.2303./5438 2023.
[37] Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint bedding ssl architectures. In The Twelfth International Conference on Learning Representations, 2024.
[38] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[39] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268- PMLR, 2021.
[40] Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578. 2020.
[41] Roman Vershynin. Introduction the non-asymptotic analysis of random matrices. arXiv preprint arXiv: 2010.
[42] Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and SueYeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations, Advances in Neural Information Processing Systems, 36:24103-24123 2023.
[43] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 1310--1322.. PMLR, 2021.
[44] Runtian Zhai, Binghin Liu, Andrej Risteski, Zico Kolter, and Pradeep Ravikumar. Understanding augmentation- based sulf-super representation learning via rkhs approximation. arXiv preprint arXiv:2306. 00788, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims are backed by theoretical and empirical results. Theorem 3.1 presents our pnnctiioally-eqiivalee compute-effciiet formulation of the SSL objective, and Theorem 3.2 demonstrates the implicit bias of gradient descent during optimizing the SSL loss. Our empirical results demonstrate the utility of our theoretical insights in improving the parameter overhead of good feature learning, optimization convergence and the sample efficiency.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a section on limitations in the discussion.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The formal statements alongside proofs presented in the supplementary material (Appendices to)
4. Experimental Result Reproducibility
Question: Does paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide implementation details the supplementary material (Ap- pendix D). We have also released our code base in the public github repo, FastSSL to facilitate the implementation of proposed framework.
5. Open access to data and code
Question: Does the paper provide open tions to faithfully reproduce the main experimental results, as described in supplemental material?
the data and code, with sufficient instruc-
Answer: [Yes]..
Justification: We open-access datasets, like CIFAR, STL and Imagenet. Our code base can be found the public github repo, FastSSL
Guidelines:
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present details of the experiment setup and results in Section 4 of the main paper, and additional implementation details in Appendix D.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
14

---

We report standard error bars, computed over 3 seeds, for all result plots and tables.
Justification:
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All our CIFAR and STL experiments were done on single 48-GB RTX8000 GPU and all Imagenet experiments were performed on 2 40-GB A100 GPUs. All exper- iments were performed on the Mila cluster, aided by compute resources, software and technical help provided by Mila (mila.quebee).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuiiisccccppiiieehhhhis
Answer: [Yes]
Justification: We adhere to the NeurIPS Code of Ethics in our work, and research in general.
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We add statement on societal impact in the Discussion section. Since the goal of our work is to advance the general field of visual representation learning, we feel there are no direct consequences that must be specifically highlighted here. Although recognize that there might be potential downstream consequences that warrant attention while building intelligent systems that leverage this work.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any new data or state-of-the-art models.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We acknowledge and cite the datasets and model architectures used this work. Our codebase is publicly available our github repo, FastSLL Moreover, our codebase relies on the Python packages Py FFCV [2 and FFCV-SSL 77] which are referred to in the github repo README.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets are released.
Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
15

---

Justification: Our work does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does involve research with human subjects.
16

---

A Hilbert Space of functions
A.1 Functions and inner product space
Definition A.I. Given X, Px. and f.g : X + R, define the (px) inner product and norm, respectively,
(f.9)xx = / f(x))())d)x(()) 11 = ffx
(11)
Define
(1: |1 < co}
be the (equivalence class) of functions with finite Px norm.
A.2
Spectral theory
In this section we quote the relevant (abstract) Hilbert Space theory.
Definition A.2 (Spectral Operator), Given orthogonal functions, & = (ot)ser in (px)), and non-negative A (Ax)sez, with ||A||2 Eier A < co. Call (Φ, A) spectral pair and define the corresponding spectral operator by
T Ah)
Σ X O1))
(12)
Theorem A.3 (Spectral Decomposition). Suppose is a Hilbert space. A symmetric positive- definite Hilbert-Schmice operator T H - H. admits the spectral decomposition equation T2 with orthonormal Φj which are the eigenfunctions Tf i.e. T (Φ,) X5oj. The oj can be extended to basis by adding complete orthonormal system in the orthogonal complement of the subspace spanned by the original dj-
Remark A.4. The Φj in equation 12 thus be assumed to form basis, but some X, may be zero
We defer the reader
21
23 for in-depth discussion and proof of Theorem A.3.
Denote by L the space of bounded (continuous) linear operators on H. with the norm
|T sup [(x) ||||| < 1}.
Definition A.5 (Compact Operators). An operator I' E L is said to be compact if there exist two orthonormal bases {9,} and {fz]. and real sequence {A₃} converging zero, such that
T(h) Σ Ashh, h hEH, 3=1
(Compact)
The AÚ may be assumed positive. The existence of representation equation Compact is equivalent to the condition: T maps every bounded set into a compact set. Compact operators are also called completely continuous operators. Representation equation [omppct called the singular value decomposition.
Definition A.6 (Hilbert-Schmin Operators). A compact operator admitting representation equa- tion Compact is said to be SiilbettSSchiitt operator if Σ X² < co. The space 8 of Hilbert- Schmidt operators is separable Hilbert space with the scalar product
T2)ss) = Σ (T (2)),
(13)
where {fi) is an arbitrary orthonormal basis. Note the value of equation 13 is independent of the basis. The corresponding norm is
I Σ X3
(HS)
121
One can show that
|T |s
17

---

Definition A.7. An operator T C is said to be symmetric if
(T(),g) - (),T())), Jgelt
and dositive-definite if
2) fEH.
(An operator with the last property is sometimes called positive semidefinite, and the term positive- definite is used when the inequality is strict.)
B Data augmentation kernel perspective of non-contrastive SSL
Theorem B.1. Let Gx)) be the infinite Mercer of the backward data augmentation covariance f2(x), fx(x)))))))))))))))))))))))))f be the features given by minimizing the
kernels, DAA Let F(x) following data augmentation invariance loss
No.
I(F)
[TMs - fill subject to
(Si, lilex = =āy
(14)
which includes the orthogonality constraint. Then, V(F) CV V(F) + V(G) as N + co.
The idea of the proof uses the fact that, linear operators, TRDAR TM and that TEDAP TarTir Then we spectral theory of compact operators, which is analogue of the Singular Value Decomposition in Hilbert Space, to show that cigenfunctions of TM operator are the same as those obtained from optimizing L(F). A similar result can be obtained using DAP and Th
Note that L(F) is the constrained optimization formulation of the BarlowTwins loss. Furthermore, L(F) with the additional constraint that (Js. ft) 2 7 Vi E {1,2... [ is the constrained optimiza- tion formulation of the VICReg loss.
B.1 Proof of theorem 3.1
We show we can factor the linear operator, leading to a practical algorithm. Here, we show that we can capture the backward data augmentation kernel with the forward data augmentation averaging operator
Lemma B.2. Using the definitions above, and with k in equation 6 given by DAB
T TT=TTT TM
First, define the non oen-tivaeeve definite bilinear form (f.g) AV Given the backwards data augmentation covariance kernel, DAB define
Proof.
(T
(15)
BDAB (f.g) (Tuf.g)px
We claim, that
BVAR_ = DAB
(16)
This follows from the following calculation,
BAB (f.g) = (Taf-g)px
(17)
(18)
pl x)) p(ru 12) [(=)g(s)
Extro
(19)
p(xr) p(ro)
((x))(z x)) p(xr)
((1)) Σ p(z)p(zo x) p(ar)
EEzo
s(z))
(20)
Ezz
(plæ) [x))(()) Σ (p(z m0)5(=1))
[Using Bayes' rule]
(21)
Ezz
(Zo = (Tas TM9)px = AV.
(22)
18

---

For implementations, it is more natural to consider invariance to data augmentations. Theorem B.3 (equivalent eigenfunctions). Assume that T'M is compact operator. Define the invariance bilinear form
BINY (Tuf)=((u
(23)
Then BINY BV AR share the same set of eigenfunctions. Moreover, these are the same as the AR, with eigenvalue }3. then
eigenfunctions of BDA.B In particular, for any eigenfunction f, of BVAR,
fg is also and eigenfunction of BINV, with the corresponding eigenvalue given by (V2j - 1)2.
Proof. Define TMM by,
TMMMJ = TMMTM
(24)
Define
Tais (TM - I) (TM I) Note, by the assumption compactness, Tas has the Singular Value Decomposition, (see the Hilbert Space section for equation SVD).
(25)
0C
(h) = Σ Ay 9;)fs j=1
(SVD)
Let fj be any right eigenvector of I M. with eigenvalue ps. Then fj is also a right eigenvector L'M I, with eigenvalue μ4 1. So we see that TAMM has fy as an eigenvector, with eigenvalue X = / and
TMS has f, eigenvector, with eigenvalue (v 1)2. Finally, the fact that there are other eigenfunctions also follows from equation SVD
The final part follows from the previous lemma.
Equivalence of Barlow Twins loss to Equation (9). The Barlow Twins loss from 43 is as follows:
LBT CCaa-1)2 + EE>>E
C
(26)
where C is the cross-correlation matrix computed between the outputs of the network to different augmentations. First, the Barlow loss can be seen as the unconstrained optimization form of the following constrained optimization objective:
LBT [(C,, - 1)2 subject to Ciz = 0 Vjj +
(27)
where 3 is the Lagrangian multiplier 8 In 43), the cross-correlation matrix C is computed by a dot product between normalized functions f, 's such that (fi. Vi. The network output for one augmentation æ, a, can be thought of as a Monte-Carlo estimate (with one sample) of Tas (z), where /a is the ith dimension of the network's output. Therefore, the Barlow loss can be written in its following equivalent form:
i (F) Σ (TMf..TM.)) - 1)² subject to
(frff)oxx 8ig
(28)
As shown by 44, the eigenvalues of TT are always less than Therefore, we do not need the square in Equation (28). Rewriting it, we get the following:
Nv.
(
Σ
(TM Tixx)
subject to (Ji,f)oxx = dij
(29)
Using Theorem B.3 we show that the loss recovers the equivalent eigenfunctions for the following reason. We can rewrite the loss
No.
i nT
i = Σ
((TM - -TJ6e, (TM T)fs)ex
subject to
(JssSs)ox = 5zz
1
2T Σ
Tasfe follip(0x)
subject to (fi 6ep
(30)
19

---

which recovers the loss Equation 9.. Note that the VICReg loss [S], in addition to the constraints imposed by the BarlowTwins loss, ensures that the norm of f. 's are more than some threshold. This can be easily incorporated into the constraint with constant along with 8.j. In conclusion, both Barlow Twins and VICReg losses can be seen as equivalent forms of the loss Equation (
Theorem B.4. (Informal) Let us denote the span of the feature space at initialization as ((Fo) and after training as V.FT). For small initialization of the network's 's weights, the alignment of V (FT) with the eigenfunctions T depend on two factors: (i) alignment of ((Fo) with the eigenfunctions Tf (ii) singular values T.
Theorem B.4. (Formal) Let I VAVT represent the eigendecomposition of and define z as the projection of the weight vectors in W onto singulur veclors ofT, V. Formally, WV. Assuming small initialization (as Simon al. (2023), i.e. (zpi(0) << for all p.í, we can derive the following conclusions:
1. sign( Afwil)) sign(A,)
(yi(1)
2. For all Xi >3 Spx(()
(p))()) =p)(0)
d where X, denotes the 8th singular value, iee. ith
[0)
element of diagonal matrix A.
Proof. We will first show that the above holds for linear network, i.e. the output of the network with weights W E mxx is X for some input X E Rxb,, where m the output dimensionality, is the input dimensionality and is the batch size.
Let us first analytically compute the cross-correlation matrix following 36
C = WX WTWT
Cpq Σ
W WT,WW
Opp Σ Wri i.
where and X' are matrices R R containing two augmentations each image batch of images. Also, we have defined T XX",, i.e. the augmentation-defined data correlation matrix. Rewriting the BarlowTwins loss function from 43
LBT [(aa- - 1)2 +
4Σ> C2
To study the learning dynamics, we need to compute the gradient of LBT w.r.t. the parameters W.
dW pq
dLAT = 2n E(c, 1) aw
aC:
aCi aw pq
= 11 ow... pd
2m
ΣΣ
C
(31)
dt
Let us now analytically compute the derivatives of On and Cij w.r.t Wo pq to simplify each of the terms in Equation
8C+
δ Σ WWiTT&W. = Ow
Σ 3
WiS O
OW pg
OW
pq
J.k
pq
Σ Tr + Σ
8pi
i.k
Σ Ta W pk + Σ W Tag
Spi
[W
Tm Spi
OC
Σ (CH -
2(Cpp 1) [WT]
(32)
aw kg
pg
20

---

Using similar algebra steps, we can simplify the second term:
ac.
[WT]] Jjg Ops +
[WT]ig 6pj
aw pq
ac
ΣΣ ##
Cjj
Cis (WTT),
jg Spi + [WT] big 8ps)
OW Pg
Csi [WT]]q
Cip [
j#q
2 [ ITWT
2(Cpp - 1) [WT]pM pg
(33)
Substituting Equations (32) and (33) into Equation B1p. we get:
dW,,
BL
pq
-1 ow pq
= -n4n(Cpp 1) [WT - 4n I)WT + 4n - 1) [WT]
dt
= 1-44 () - [W'] 4m3 [CC ( IIWT]
(34)
Note that setting 8 1 yields the dynamics equation presented by 36 However, in practice, is orders of magnitude less that 1. For sake of simplicity. we will analyze the extreme case of / = 0, which will yield insights into the veak-oothogonnality constraint case. Therefore,
dW WW Ann(ppp 1) [WT] dt
(35)
Let us denote the eigendecomposition of be written as T = VAVT. Here, A is diagonal matrix with singular values the diagonal elements. Let us also denote the projection of the weight vectors onto the singular vectors of ',, i.e. V as z. So, = WV. Therefore, using these definitions, we can write the following:
Cpp = [WTW'] pp [ZAZ"]
Σ Ap
Pp
WT = WVAV = ZAVT
Now, writing the update equations Equation 35 in terms of Api
džp:
dW
Σ
Vys
dt
dt
4
4n
2 Σ
Ajj -
Exeddal Σ Vyn Vis))
4n
Σ
2 A3j
1
Apida
(36)
Assuming small initialization of weights W. we can assume that (0) << 1, i.e. magnitude 2pa at time 0 is very small.
Let us define (1) = E3 Zpz (1) For small initialization, h1p(1) > VL. Therefore,
sign (zpp(1) 1 sign(As)
(37)
dt
Zpi
is clear from Equation (37 that if >; 0, lim4-soo 2pd (t) Similarly, if X; 0, then 2p)(t) = Zpi (0) Vt.
Therefore, akin to the conclusions (66], the BarlowTwins loss recovers directions corresponding to positive singular values in the augmentation-defime covariance matrix, T and suppresses directions corresponding to negative singular values. Thus, the network outputs span the top singular vectors of
T.
21

---

It is worth noting from Equation (36) that the following holds:
1
dlog(Zps)
1 dlog(Zpi)
X;
dl
dl
1
(p)
1
[tp(t)
log
log
A
2p)
A
Zpi(0) (0)
Zpi(t) (t)
3pp (t)
(38)
3po (0)
Zpg (0)
Without loss of generality, if X, << >y. then (t) = zp: Therefore, under small initialization, Le. Zp: is small Vi, gradient descent biases the phh weight vector to be more strongly aligned to the eigenvector corresponding to the strongest eigenvalue, for all p y's. Hence, under weak orthogo- nalization constraints, the Barlow/wins loss will over "represent" the strong singular vectors of the augmentation-deefned cross-correlation matrix.
When using nigh-dimensional projectors, specifically when >> Ei 1x,50- wherein Iç the indi- cator function that is when condition Ç is true and 0 otherwise, this problem might be ameliorated because there are multiple weight vectors that might be aligned with the top singular vectors of T at initialization. However, when using wwwdimensional projectors, we do not have such a luxury and therefore, using a weak orthogonalization constraint leads to dimensionality collapse in the representation space.
Extending to deep non-linear networks. Similar the analysis in [36], we can repeat the above analysis by replace X and X' by the corresponding kernel versions, where the kernel corresponds to the Neural Tangent Kernel (NTK) of the network. Therefore, the implicit bias of gradient descent to yield dimensionality collapse in the representation space when using weak orthogonalization constraints still remains.
Dimensionality collapse under noisy optimization. From the rest of this section, we have seen that the BarlowTwing loss is Monte-Carlo estimate of the true data- auggeetggmnttiionn defined covariance matrix. Moreover, stochastic gradient descent adds noise due to mini-bate batch sampling to the optimiza- tion process. Note that there exist symmetries in our linear network. i.e. orthogonal rotation of the weight matrix yields the same loss function. As explained in [T. such symmetry uummetry-invarian sets potential candidates for stochastic collapse when performing noisy gradient-based optimization. Therefore, the presence of noise the data-augmentation covariance matrix, T, as well as the hatch noise can further worsen the dimensionality collapse problem where different weight vectors become parallel to each other due to noise in updates. One possible mitigation strategy is to obtain better estimate of the true augmentation-definet covariance matrix (see Figure which discuss in the next section.
Empirical validation. We empirically validate results the learning dynamics simplistic 2-dimensional settings. These results, demonstrating the difference feature learning dynamics for weak strong orthogonalization, presented GIFs in the supplementary material, and can also be viewed at the project website
C Multi-Augmentation Learning
C.1 Augmentation graph
We use the population augmentation graph formulation introduced in (22). Briefly, we define graph G(X, W), where the vertex set X' comprises of all augmentations from the dataset (could be infinite when continuous augmentation functions are used) and W denotes the adjacency matrix with edge weights as defined below:
Waa' == EENPX (A(r)T)A(<(1E))
(39)
i.e. the joint probability of generating 'patches' x, x!' from the same image T. Here A defines the set of augmentation functions used in the SSL pipeline. It is worth noting that the magnitude of Wast captures the relative similarity between z and x'. A higher value of Way indicates that it is more
22

---

A
B
1
X
V
X3
X4
X
X
Figure 6: Schematic of augmentation graph. (A) Augmentations from each image span region in the image space which could overlap with the augmentation span of other images. (B) An augmentation graph schematic that uses probabilities to characterize the interactions among augmentation spans of different instances.
likely that both patches came from the same image, and thereby are more similar. The marginal likelihood of each patch x can also be derived from this formulation:
Exp' [Waxr]
(40)
C.2 Contrastive and non-contrastive losses suffer from the same issues
We will now show that the proposal of using multiple patches for the Linvariance is pertinent to both the contrastive and non-contrastive SSL. Following [22]. we use the spectral contrastive loss formulation and incorporate the augmentation graph relations:
Le E - [f(x) f(x*))x+)))))))"") + BE ((=)))
L. x 12 - D-D 11 122 - WIE
(41)
where 2 == /w)f(x), D is a x N diagonal matrix with entries {w*} and W = D W-D
3.
We extend the duality results between contrastive and non-contrastive SSL loss, established by [19] demonstrate how Equation (41 can be decomposed into the invariance and gallapse-rrvennting loss terms.
1Zz - WIP ||zT z - -all? + 2Tr [ZT((N (IN W) +
(42)
1 - Tall? + 2 >> -
Σ 20.00
Wzz 2;24 + K
(43)
where is constant independent of Z. The first term in Equation is the covariance regularization term in non-contrastive losses like Barlow (implicit) or VIC-Reg (explicit), and the second term in Equation (43) is the variance regularization. Simplifying the third term in Equation (43) gives us:
ΣV 2,2
Dast Zuth = ΣΣ
Wer f(o)/(())' ΣΣ x..
EawPx
Σ EEWPS
Σ A(CDMMAA f(x))
EzNPy EA(E|E) (()
(44)
This term encourages f (z) to be similar to fxx), i.e. the mean representation across all augmentations of ø. thereby requiring to "sufficiently" sample A(.()). Given that both the contrastive and non- contrastive losses rely on learning invariance properties from data augmentations, we believe that
our multi-patch proposal would improve the probability density estimation of A(-[z) and yield better performance with few training epochs.
23

---

C.3 Explaining training dynamics in low patch sampling regime
We now turn to a simple form of the augmentation graph to understand how using low number of augmentations affects the evolution of Zz Minimizing Equation (41) implies that the spectral decomposition of Z would align with the top eigenvectors (and values) of W. We will demonstrate that in the low sampling regime (using few augmentations), the eigenvectors of the sampled augmentation graph W may not align with those of W.
Augmentation graph setup. We define an augmentation graph with only two instances from two different classes, similar to the one presented in [35]. Let us denote the four instances as T for i E 1, 3, 4, where I1, belong to class 1 (i.e. Y/, 3/2 = 1) and ø3, 2 belong to class (i.e. Us: 14 = 4). Let us further assume that ē1, It have the highest pixel-level similarity among (01, Vvi E 2, 4, thereby making more likely to have similar patches. We denote this relationship among input examples using & to indicate (pixel-wise) global similarity groups. So, G G3 = 1 and the following probabilistic formulation model our augmentation functions
G2, G4 2. We
(see Figure 3):
p' j =
D¹ if +f and yj yi and G; + Gi
A(x,(ø,)
(45)
V if and yj ye and 9j G:
8' Hf i yj # y and 9, + 9;
+ 8' = 1. The adjacency matrix of our augmentation graph (as shown in Figure is follows:
In our setting, df + 4 +
P
1
V
6
8
V
W
(46)
8
P
8
V
P
P1
We defer the relations between p', ",,,'//8 v/8' and Ps p,,u, 1, to the appendix. The eigenvalues of this
matrix are: (p + μ1 + v + 8, + V 5,p /1
5,
- ō). Corresponding
eigenvectors are along 11,
-1, 11, 1, 1, -1 [1. 11, -1, Assuming that invariance properties that are relevant for
the augmentation functions induce comanticall--reeveaa identifying from /(x;), we can say that D' > (y', n/) and min(v', ,1') >> 8'. When we have sufficiently sampled the augmentations, any SSL. loss will learn such that its singular values are span the top eigenvectors of the augmentation graph, and the eigenspectrum of 7z" would simply be the above eigenvalues. In practical settings, the augmentation graph would have significantly higher dimension that the fcature/embedding dimension + Therefore, singular vectors of Z would span the top eigenvectors of W and the smaller eigenmodes are not learned. When we have accurately sampled the augmentation graph, μ > v and therefore, the flass-ifformation preserving information is preferred over pixel-level preserving information during learning. But what happens when we do not sufficiently sample the augmentation spuce?
Ansatz. Based on our empirical experience, we define an ansatz pertaining to the eigenvalues of a sampled augmentation graph and validate it in tractable toy settings, such the one described above. Specifically, we claim that when the augmentation space is not sufficiently sampled, {|u 1 v/). 8} + 0. In other words, we claim that when only few augmentations per example are used, it is more likely to have an equal empirical likelihood for augmentations that preserve (pixel-level) global information and class/context information. Moreover, it is very unlikely to have augmentations that change both the class and global information. This is demonstrated in Figure 7
Consequences of the Ansatz. When only few augmentations are sampled, learning can suppress the class information at the cost of preserving the pixel-level information, thereby leading to an increased smoothness in the learned feature space.
2 "Contrastive algorithms use large batch size, thereby optimizing high-dimensiona ZZT whereas non- contrastive algorithms large embedding dimension, thereby optimizing digeedimensional Z TT2 Z.
24

---

True eigenvec0
eigenvec0
True eigenvec1
eigenvec1
0.4-
True elgenvec2
eigenvec2
Eigenvalue
True eigenvec3
eigenvec3
0.3-
0.2
0.1-
0.0
10
100
Number of patches
Figure 7: Empirical verification of the subsampling Ansatz.
D Implementation Details
Image Classification Datasets Across all experiments, our settings mainly follow 13 In particular, Table |4 our pretraining settings on Cifar-10 (26). STL--10 [4 and Imagenet- 100 (33] The Imagenet- 100 dataset was generated by sampling 100 classes from the original Imagenet- 1k dataset, according to this list [88] In Table (4b) we outline the corresponding linear evaluation settings for Resnet-50 (for CIFAR- 10 and STL-10) and ResNet-18 18 (for Imagenet). Note that we add linear classifier layer to the encoder's features and discard the projection layers for evaluation. Our code
base is publicly available on github.
config optimizer
value
config optimizer learning rate batch size epochs weight-decay
value
Adam
Adam
learning rate
le-3
1e-3
batch size
512
128 (Imagnet), 256 (CIFAR, STL)
epochs
200
100
weight-decay
le-6
le-6
test-patches
16
Pretraining
(b) Linear Evaluation
Table 4: Experiment Protocol for comparing SSL algorithms
The key SSL loss functions that we use this work are Barlow 43 and VICReg (5). Let us suppose that the embeddings of two augmentations of a batch of images are denoted as and z'. The BarlowTwing loss function is as follows:
LBT = (Ca +3
=BΣ>>
(47)
where
C
Σ
(2t - 22(4 "1
71
k=1
1
and
2 =
2k
2
4
(48)
km
km
Ci is the element of C' at row column and the batch size. each projector dimensionality, d, search for the hyperparameter, B, that yields the best downstream task performance.
25

---

The VICReg loss function is as follows:
LVIC -1 Fb - =/1l + zul(((2) -(2))+ -2))) +
(49)
d
where
u(Z) =
maz Stdev(z..))
d iml
and dz) = ΣΣ
C(Z)
Σ 2k 2) - 2)²
n
For each projector dimensionality, d, we search for the hyperparameter, μ4, that yields the best downstream task performance.
D.1 Empirical results for low-dimensional projectors
A
CIFAR-10 BarlowTwins
CIFAR-10 10 VICReg
84
Test accuracy (%)
80
76
76
72
Optimal
Optimal A Fixed P (0.0013)
Fixed (0.0004)
68
72
C
1e2
1e3
1e4
1e2
1e3
1e4
Optimal B
1e-2
1e-4
1e-5
1e-3
1e-6
1e3
1e4
1e2
1e3
1e4
Projector Dimensionality
Projector Dimensionality
Figure 8: Low-dimensional projectors can yield good representations for both BarlowTwins and VICReg. We demonstrate that using higher orthogonality constraint, 3,, for lower projector dimensionality can achieve similar performance over a wide range of projector dimensions (d). Note that for VICReg, we plot the ratio of the coefficient of the covariance loss the coefficient of the invariance loss, i.e. 8 I_ where μ is the coefficient of the invariance loss. (See Equation (49 for dey
details of the loss formulation.)
Barlow Twins
VICReg
pdim
Projector params (approx)
fixed 8
optimal 8*
fixed 8
optimal 8*
64
135k
73.6 0.9
82.1 i 0.2
68.9 1 0.2
81.9 1 0.1
128
278k
74.7 + 1.4
83,0 + 1.1
70.6
0.3
82.3 1 0.4
256
589k
75.9 0.7
83.4 0.4
75.3
0.2
81.9 - 0.3
512
1.3M
0.8
82.8 - 0.5
0.4
82.1 0.6
1024
3.1M
81.3 1.0
82.9 0.3
0.9
82.5 = 0.9
2048
8.3M
81.0
82.3
8.0
81.9 = 1.2
4096
25.2M
82.3 1 0.4
82.3 1 0.4
80.5
+ 0.3
81.0 + 0.4
8192
83.9M
82.2 + 0.4
82.2
80.4
1.5
80.4 + 1.5
Table 5: Extended version of Table [ Optimizing for orthogonality appropriately allows low-
dimensional projectors to match the performance for MarlowTwing and VICReg (on CIFAR-10) of much higher- dibees-iimnnio projectors.
26

---

D.2
Empirical results with multi-augmeentations along with Time
A
CIFAR-10 BarlowTwins
B
CIFAR-10 VICReg
C
STL-10 Barlow BarlowTwins
90
90
90
Test Test accuracy (%)
80
80
80
70-
70-
60
70
60
augs, pdim=256
50
2 augs, pdim=256
60
augs, pdim=256
50
augs, pdim=8192
2 augis pdim=8192
2 augs, pdim=#892
augs, pdim=256
augs, pdim=256
50
augs, pdim=256
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Epochs
Epochs
Epochs
D
E
F
Test accuracy (%)
90
90
90
80
80
70-
70
60
augs, pdim=256
50
2 augs, pdim=256
60
2 augs, pdim=256 augs, pdim=8192
50
augs, 2dim=226
augs, pd92
40
2 augs, pdim=8192
augs, pdim=256
augs, pdim=256
50
0 20
40 60
100120140
20
40
60 80 100 120 Time (min)
0
200 400 Time (min)
800 1000
Time (min)
Figure 9: Using multiple augmentations improves representation learning performance and conver- gence. (A-C) Across BarlowTTwis and VICReg for CIFAR-10 and STL.100 pretraining, using augmentations instead of 2 helps improve performance. (D-F) Although the 4-augmentations take longer for each epoch, its performance still trumps the 2-augmentation version the algorithm at the same wall clock time. Please see Appendix E.3 for more results.
A
CIFAR-10 BarlowTwins
B
CIFAR-10 VICReg
C
STL-10 BarlowTwil
901
90
90
Test accuracy (%)
80
80
70
60
70
60
50
60
50
40
50
20
40
60
80
100
20
40
60 Epochs
80
100
20
40
60 Epochs
80
100
Epochs
E
F
Test accuracy (%)
80
80
70
70-
60
60
50
40
0
20
40
60
80
100
0
20
40
60
80
0 100 200 300 400 500 600
Time (min)
Time
(min)
Time (min)
augs, pdim=256 (100% pretraining samples)
2 augs, pdim=8192 (100% pretraining samples) augs, pdim=256 (25% pretraining samples)
augs, pdim=256 (50% pretraining samples)
Figure 10: Multi-augm augmentation improves sample efficiency, recovering similar performance with significantly fewer unique samples in the pretraining dataset. Across BarlowTwing and VICReg pre- training on CIFAR-10 and STL-10, for the same effective dataset size (#augs x # #unique ssrpue_samles) using patches improves performance the same epoch (A-C) wall clock time (D-F). How- ever, a tradeoff exists wherein more data augmentations fail to improve performance in the scarce data regime.
27

---

Pratrain Imagenet-10U
60
80
100
150
40
auge 2. pdin. 612
45
ougs 2, p.mm 512
auge, B_ __pdi__522
zugs 8_pdim_pdmmm__
40
8b
100
200
400
800
800
Epachs
Time (mi))
Figure 11: Barlow pretraining on full Imagenet- dataset with 2, 4 and augmentations.
Pretrain eval on Imagenet-100
Teat (1)
66
655
Text accuratty
60-
60
ouge pdim 512
adi B12
auge 1___ddd__55 toineol) sugs_ P_pdin__122222 5222 (0 25 transet))
ouge 1dddd___
40
40
auge L_thhrr f__xhm__122 (0 7t transett
100
200
250
Figure 12: BarlowTwins pretraining fraction of Imagenet-100 dataset with 2, and augmenta- tions.
28

---

D.3 Empirical results on transfer learning
In this section, we present extended version of results presented in Figure Figure but but pretraining on CIFAR- 10 (or STL-10) and evaluating STL-10 (or CIFAR-10), These results, coupled with the ones in Figure Figure present a strong case for the advantage of using the proposed multi- augmentation loss for better convergence well as downstream accuracy.
Pretrain CIFAR-10 10 and eval STL-10
(5).
70-
Tes: awwporuuuoooooo eay (56)
70
60-
80
65-
60
2. pdim. 258
ougs, 2, p...dmmmm 256 augs.
40
augs,
_dpddmmmmmmmm 256
40
augs, 4. p_pm 258
Epachs
(min)
Figure 13: Barlow Twins pretraining on CIFAR-10, linear evaluation on STL-10 labelled set.
Pretrain CIFAR-10
STL-10
80
80
70-
70
60
80
sugs_2_pdinn 256
2_odiin 258
suge_2_ 2twp___pdim____0 8162
auge 2_oddm__001111 0182
30
pdm__dm__2 258
30-
odim 258
100
Epochs
Time (min)
Figure 14: VICReg pretraining on CIFAR-10, linear evaluation on STL-10 labelled set.
STL-10 on
85-
85
80
80
aocurscy (8)
75-
Test eccuracy
86-
85
60-
60-
auga,
06-
auge 2_ 2_pdim 2_pdim__182 8182
55-
auga, 2 2___2_________111_0000
suge
suge, 2_poim_255
10
100
200
300
400
Epoche
Time (min)
Figure 15: BarlowTwin pretraining on STL-10, linear evaluation CIFAR-10 labelled
29

---

Pretrain on CIFAR-10 eval on STL-10
70-
70-
scouracy (56)
eccurasy
60-
80-
80-
2_pdii__25 auge pdim. 8.
50-
augs, 2. 2_pciimmmmmmmmmmmm 258
zugs 2 palm 0192
256 co traiset)
258
auge, R P_pdiim 258 (0 25 traisee))
augs, 8. p_lm 258 (0. trainset)
40
Time
Figure 16: BarlowTwins pretraining on fraction of CIFAR-10 trainset, linear evaluation on STL-10 labelled set.
Pretrain on CIFAR-10 eval STL-10
Test aozuraty (%)
80
80
150m
50-
augs pdim 256
Auga,2 p_pddmmmm 296
40-
40-
suga
sugs__ 1_-____pp __p_____mmm____m 226 (0.50 trainse))
auga 4. _____m___2560000 trainset)
258 :0.50 trainse)) augs
30-
sugs, pddmpdmmmmmm transet)
30
60
100
20
Epoche
Time (min)
Figure 17: VICReg loss pretraining on fraction of CIFAR-I0 trainset, linear evaluation on STL-10 labelled set.
Pretrain STL-10 & eval CIFAR-10
80
80
75-
75
70
Tep: SAOUU soy
70
lest
sugs, 2_ 2_pdim_256
auga, suga____oe augs, 2_pdim__0011111111 0182
augs, 3ugs_2_ 2_pddm_______11
8182
86
augs pdim
transet)
55
augs (0.50 trainset)
augs. pdim. 256 (0 (0.25 trainet))
augs. p.l.cim 256 10.25 trainset)
00-
50
30 Epachs
150
(mir)
Figure 18: BarlowTwins loss pretraining on fraction of STL-10 unlabelled set, linear evaluation on CIFAR- 10 train
30

---

E Additional Experiments probing multi-nngmentation learning
E.1 Longer Pretraining to determine early stopping
Protrain eval on CIFAR-10
16
suge 25E
siga
mage _egt___pdim____
auga 4 __p_im_250 258
14-
sage _pp___dim___
auga pdlm
Teal ener (90
200
400
200
300
Epochs
Time
Figure 19: BarlowTwing pretraining on full CIFAR-10 dataset for 400 epochs.
Algorithm
Best accuracy
Best accuracy @ epoch
Marlow-Twing (2-augs) w/ pdim=256
92.04 +/- 0.16
400
Barlow-Twins (4-augs) w/ pdim=256
92.39 +/- 0.17
340
Sarlow-Twing (8-augs) w/ pdim=256
92.64 +/- 0.10
140
Table 6: BarlowTwing pretraining on full CIFAR-10 dataset at 400 epochs (with early stopping)
E.2 SwAV-like augmentations for compute efficient multi-augmentation framework
Pretrain
20
20-
Toxt
62-
8
sugs_2 2_pdim________
auge_2 2o9_22_dim__220
aage_2_pding 2_pdi________1_ 152
auge___pdien
2_pdim__112
80
suga p___ddmmmmmmmmmmm
30-
auga
auge 8 pd__p______0 256
auge &. pdim 258
Figure 20: BarlowTwins pretraining on full STL-10 dataset for 100 epochs using SwAV-like augmen- tations. Specifically, the 2-augmentations setting uses two views that are 64 x 64, whereas the 4 (or 8) augmentation setting uses additional two (or six) augmentations that are 32 32.
31

---

E.3
Training with full dataset with 4/8 augmentations
Pretrain & eval CIFAR 10
80-
__pdiimmmm 256
Augs 2 2_pdim_mmmmm 256
aaga, 2. p_pddi,
ougs, 2. p_pddmm
50-
ouga 4. p..mdmmmmm 268
90-
ouge_ 4. p_lm 256
augs
augs
100
50
100
150
200
250
Epachs
Time (mir))
Figure 21: MarlowTwing pretraining on full CIFAR- 10 dataset with 2, 4 and 8 augmentations.
Algorithm
#augs=2
#augs=4
#augs=8
Barlow-Twine w/ pdim=256
86.43 +/- 0.72
91.73 +/- 0.16
92.71 +/- 0.19
Barlow-Twiee Twins w/ pdim=8192
85.44 +/- 0.54
91.40 +/- 0.32
92.40 02-40+/-0.11 0.13
Table 7: Barlow'Twins pretraining on full CIFAR- 10 dataset
100 epochs
32


Reviews:
[{"review_id": "7ZseyYcPf9", "reviewer": "Reviewer_EgLh", "summary": "This work introduces a new loss for self-supervised retraining that is functionally equivalent to existing methods but is computationally more efficient. The method identifies a key equivalence between existing SSL objectives (VICReg and BarlowTwins) features when employing augmentation kernel. Furthermore, the authors demonstrate that under their new loss formulation far fewer projector dimensions are needed and in-turn the efficiency is improved (time to convergence). Additionally, the authors show that multiple augmentations improve the kernel approximation, improving accuracy and convergence speed.", "questions": "⁃\tDoes the proposed formulation lead to better improved overall performance, by that I mean improved best case downstream performance compared to original formulations?\n\n⁃\tDoes number of layers or dimensions of intermediate layers of the projector have significant impact on the observed performance?", "limitations": "The limitations are appropriately addressed", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "⁃\tThe paper is very well written and presented, with all contributions clearly defined and rationale within the problem setting. Figures are simple yet informative and lead to a high-quality piece of work that is understandable to a wide audience.\n\n⁃\tThe proposed methodology is novel and highly original, addressing key and ongoing concerns in the field of self-supervised learning. Moreover, the method is theoretically and empirically supported with extensive proof and experimentation, thus addressing key concerns with correctness and confidence. \n\n⁃\tThe authors present a variety of recommendations that go above simple proposition of a new methodology, thus contributing to the impact and significance of the work.\n\n⁃\tThe empirical analysis is extensive and supports all claims made, the resulting performance is a faster convergence with fewer projector dimensions.", "weaknesses": "I have only two minor concerns/weaknesses:\n\n⁃\tYou enforce a orthogonalization constraint on the projector, yet how is collinearity of the features avoided in practice for earlier representations, specifically after the projector is dropped? Is this even a problem?\n\n⁃\tAlternative measures for efficiency could be presented alongside convergence speed. I would expect alternative measures in both iterations per second, and memory consumption to be included at the very least if making claims of improved efficiency. Additionally, how much of a speedup can be attributed to each component I.e. the reformulation vs orthogonality constraint vs reduced projector.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review of our work. We greatly appreciate your time and effort in assessing our paper and providing constructive feedback.\nWe are delighted you found the paper well-written, clearly presented, and understandable to a broad audience. It's delightful to hear that you find our approach new, original, and well-supported, theoretically and empirically. Your acknowledgment of the range and importance of our suggestions is very encouraging.\n\n1. **Avoiding collinearity of the encoder features**:\nYou raise an intriguing point about avoiding feature collinearity after dropping the projector. In practice, enforcing an orthogonality among the projector output features implicitly avoids collinearity in the feature space of the encoder. An intuitive explanation of this phenomenon is as follows. Since the projector output is a piecewise nonlinear function of the encoder output (owing to the fully connected layers with ReLU in the projector network), a collinearity in the encoder output space would imply a collinearity in the projector output space. Therefore, by enforcing an orthogonality constraint in the projector output space, we are able to avoid a collapse or collinearity in the feature space. An in-depth analysis of the learning dynamics is, however, an open research problem with some recent progress being made in simplified settings [1].\n\n2. **Metrics of efficiency**:\nThe suggestion to include additional efficiency measures such as iterations per second and memory consumption is well-taken. We will incorporate these metrics in a revision to provide a more comprehensive picture of the efficiency improvements. Disentangling the contributions of each component (reformulation, orthogonality constraint, reduced projector) to the overall speedup is also a valuable direction for further analysis. \n\n3. **Other questions:**\n- Regarding downstream performance, our proposed formulation achieves comparable best-case results to the original formulations while being more efficient. We believe there is potential for further performance gains with additional tuning and optimization.\n- The depth and width of the projector do impact the observed performance. We found that relatively shallow projectors (1-2 layers) with moderate width (e.g., 128-512 dimensions) worked well, but the optimal configuration likely depends on the specific dataset and backbone architecture. More systematic ablations on projector design would be informative.\n\nThank you again for your excellent suggestions and questions. We will carefully consider your points as we revise and extend this work. Please let us know if you have any other feedback or concerns.\n\nBest regards,\n\nThe Authors\n\n[1] Y. Xue, E. Gan, J. Ni, S. Joshi, and B. Mirzasoleiman, Investigating the Benefits of Projection Head for Representation Learning. ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JGMz0272tq", "reviewer": "Reviewer_rDTK", "summary": "This paper identifies the implicit bias of non-contrastive SSL loss and optimization, and proposes two ingredients to improve SSL learning: 1) Low-dimensional projectors can yield good representations; 2) Multiple augmentations improve kernel approximation. Further, the authors propose that In a low-data regime, using diverse & multiple augmentations can be as effective as acquiring more unique samples. All of these insights get validated on experiments of ResNet-50-based models.", "questions": "Please refer to the questions.", "limitations": "The paper is basically an investigation into the insights revealed by the alternative formulation of NC-SSL loss. The insights get empirically validated and the overall story makes sense. \n\nHowever, I do not see any \"surprise\" from the paper except for some predictable conclusions which generally hold in deep learning. I am also not really convinced by the results in Table 1, which has big jumps in dimension sizes and only validated on CIFAR10/STL-10. These issues prevent me from giving a higher score.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is theoretically motivated well based on the formulation of NC-SSL. By casting the problem into the equivalent formulation of data augmentation invariance loss and the orthogonality constraint. \n\n2. The empirical evaluation fairly meets the expectations from the alternative formulation of NC-SSL, which validates the proposed insights.", "weaknesses": "1. I am not sure about the definition of weak/strong orthogonality constraints. How did the authors define what is a strong or weak constraint? In other words,  what value of beta is regarded as a strong constraint?\n\n2. All the experiments are conducted on ResNet-50. Can authors do experiments on other architectures like Vision Transformers (ViTs) or other CNNs, in order to show the conclusions also transfer to other models? \n\n3. The insights are empirically validated and reasonable. However, these are the things that a deep learning practitioner could expect or imagine, especially that data augmentation can help in the low-data regime and that multiple augmentation improves the performance. These insights are also consistent in the supervised learning setting. In other words, there is no big \"surprise\" from the paper; every conclusion can be predicted even without the knowledge of SSL.\n\n4. For the conclusion that Low-dimensional projectors can yield good representations, the experiments are only conducted on CIFAR10 and STL-10, where the dimension size 8192 is considerably larger than the usual setting and even 1024 can be considered as a large projector. Can author did experiments on ImageNet-100? I am also curious the see the results on the step-wise increasing dimensions: 256,512,1024,2048,4096 instead of some big jumps in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough and insightful comments. We are also glad the reviewer found our proposal theoretically motivated and the empirical validation satisfactory. Below, we address the questions raised by the reviewer:\n\n1. **Definitions of weak/strong orthogonality constraints:**\nThe reviewer is correct in pointing out that we did not provide explicit definitions in the original manuscript as to what constitutes ”weak” vs “strong” orthogonality constraints. In the extremes, the weakest possible orthogonality constraint is a $\\beta = 0$, and the higher the value of $\\beta$, the stronger it is. Thus, we generally use these terms to describe the relative values of the hyperparameter, $\\beta$, i.e., we generally use these words to indicate differences between different models. But, we do not have a specific threshold that universally defines “weak” vs. “strong”. Based on reference [31], we generally consider $\\beta = 1$ a “strong” constraint. We will clarify this usage in the updated text and make it clear that there is no explicit threshold. \n\n2. **Experiments on other architectures like Vision Transformers (ViTs) or other CNNs:**\nWe thank the reviewer for this suggestion. In this work, we train Resnet-50 and Resnet-18 (Imagenet experiments) following previous literature in non-contrastive SSL. To our knowledge, self-supervised ViTs are generally trained using either the MAE or self-distillation objectives (JEPA-style). While the insight regarding projector dimensionality and orthogonalization constraint is not directly transferable to such settings, the loss formulation involving multiple augmentations can improve convergence in self-distillation settings. We can, however, test on other CNN architectures if the reviewer feels this would strengthen the empirical results. We will do this over the discussion period, but we are confident we will see similar results for other architectures.\n  \n\n3. **Core contributions of our work:**\nWe are glad that the reviewer found our work to be theoretically well motivated with sufficient empirical evaluation. The reviewer's point is well taken that it is not surprising that additional augmentations can help pretraining. However, we would like to highlight that while existing works and heuristics help with improving the generalization performance, they still require long pretraining, usually 800-1000 epochs for Imagenet. Instead, our work focusses on improving the convergence speed while maintaining performance. Moreover, we provide theoretical justification for how our recommendations impact the SSL pretraining, and such theoretical grounding helps to move machine learning towards more principled practice. \n\n4. **Low-dimensional projector experiments beyond CIFAR-10 and STL-10:**\nWe apologize for the oversight in presenting the results from Imagenet-100 in Table 1. In the rebuttal document, we have presented the results for Resnet-18 pretrained using BarlowTwins on Imagenet-100. We will also modify Table 1 to include all intermediate dimension sizes (see the rebuttal document for the updated Table 1 to be put in the revised manuscript). We hope these fleshed-out results will be more convincing for the reviewer.\n\nWe thank the reviewer again for their thoughtful comments and excellent questions. We will carefully consider your points as we revise and extend this work. Please let us know if you have any other feedback or concerns.\n\nBest regards,\n\nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LLZYa2DQmy", "reviewer": "Reviewer_RyLx", "summary": "The paper conducts a theoretical study the training dynamics of self-supervised learning (SSL) methods by using an idealized loss function that resembles the Barlow Twins/VICReg loss. The idealized loss function is built on the idea of forward data augmentation (\\emph{DAF}) graph kernel. The paper shows how this loss function maybe recast in an equivalent form that is computationally friendly.  The training dynamics behavior of this loss is then used to build a hypothesis that representation learning can be improved with stronger orthogonal constraint regularization for small dimensional projectors and  multiple augmentations improve performance. The paper provides empirical evidence to support the claims made in the paper.", "questions": "- How would a reader understand claims related to downstream performance using your framework that argues for faster convergence with pretext task?\n\n- Do you have any results with Barlow Twins or VICReg using ImageNet-100 at 224x224 resolution images for Section 4.1?\n\n- The equivalent loss formulation calculates the mean of embeddings (in feature space). If my understanding is correct, the the following paper [Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations](https://openreview.net/forum?id=og9V7NgOrQ) could be of interest to the authors. Would the authors provide any insight on any possible connections in the paper?\n\n- The argument about lower dimensional projections is very interesting but appears to be opposite to empirical findings made by the following works:\n  - [RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank`](https://openreview.net/forum?id=uGEBxC8dnEh)\n  - [LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures](https://openreview.net/forum?id=f3g5XpL9Kb)\nLike above, are the arguments made in the paper consistent with the empirical findings made in prior works?\n\n- There appears to be a typo in Figure 2 where $\\lambda$ is used instead of $\\beta$. Please correct if applicable \n\nI look forward to getting clarifications during rebuttal for the questions above.", "limitations": "The authors have clearly listed their limitations. In addition, the empirical setup may not be the one used by empirical scientists in SSL so making a note may be helpful (but not necessary)", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The analytical study conducted in the paper on training dynamics is very interesting to this reader and should be of interest to the community\n- The idea of using an equivalent loss to DAF that is simpler to compute may also be of interest to the community as it has connections to other recently proposed SSL methods\n- The practical takeaways may be relevant to SSL practitioners", "weaknesses": "- A main weakness that I see is that I am unable to relate the faster convergence claim/proof to improved downstream accuracy. This may be a matter of presentation or perhaps a gap in the reviewer's knowledge on how alignment argument in pretext task implies better downstream performance. Would the authors be able to clear this question up during rebuttal?\n\n- The experimental setup considered in the paper is on the smaller-scale side for understandable reasons. However, this may make the statements made in the paper less interesting to empirical scientists that work in SSL. Experiments using ImageNet-100 may help with showing the claims can hold with setups used in practice. However, I do not see any experiments with ImageNet-100 in Section 4.1 (low dimensional projection)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough and insightful comments. We are pleased that the reviewer found our analysis of the SSL loss formulation and the training dynamics exciting and relevant to the community. Below, we address the questions raised by the reviewer with enhanced clarity and additional insights:\n\n1. **Understanding Downstream Performance Claims:** The reviewer's question is important and touches on the core of representation learning in SSL. Our framework focuses on accelerating convergence of the SSL loss itself, which inherently influences downstream performance. By optimizing the SSL loss more efficiently, we ensure that semantically similar images are mapped closely in the representation space, facilitating easier classification through methods like k-nearest neighbors or linear decoding. This is why reducing the SSL loss more rapidly leads to more rapid improvements in downstream accuracy. Our empirical results confirm this (e.g. see Figure 3).We will emphasize this critical link in the revised manuscript to ensure clarity and robustness in our claims.\n\n2. **Results with Barlow Twins and VICReg on ImageNet-100:** We acknowledge the oversight in not including results from ImageNet-100. In the rebuttal, we have added plots for ResNet-18 pretrained with BarlowTwins on ImageNet-100. These results reinforce our recommendation in Section 4.1 that smaller projector dimensions can yield effective representations even in larger-scale pretraining scenarios. This addition aims to fully address the reviewer's concern regarding the scalability and applicability of our findings.\n\n\n3. **Connections to Maximum Manifold Capacity Representations (MMCR):** We thank the reviewer for bringing the MMCR paper by Yerxa et al. to our attention, which is highly relevant. Yerxa et al. introduce a non-contrastive SSL objective that leverages multiple augmentations to estimate the manifold of object centroids. By maximizing the nuclear norm of the covariance matrix of these centroid representations, they effectively enhance the representation space's suitability for linear decoding of semantic categories. Although this approach is rooted in a statistical mechanical perspective, Schaeffer et al. [3] have convincingly linked it to the information-theoretic viewpoints prevalent in the SSL literature. Consequently, the MMCR loss formulation can be viewed as a functionally equivalent form to the loss formulations discussed in our work.  \nWhile Yerxa et al. provide compelling empirical evidence, intuitive justifications, and posthoc analyses to underscore the benefits of multiple augmentations in improving semantic categorization, our work offers a complementary, theoretically grounded perspective. We elucidate how these augmentations not only contribute to a more efficient optimization process but also facilitate the early learning of superior features. This dual benefit—accelerated convergence and enhanced feature quality—underscores the efficacy of the MMCR framework in learning robust representations.\n\n4. **Contrasting results in Rankme and LiDAR:** This is an excellent point to clarify, thank you for highlighting the connection. Our focus differs from RankMe and LiDAR, which analyze the dimensionality of the learned representation space in SSL algorithms. On the other hand, our study investigates the design space configurations underlying these algorithms, precisely the number of units in the projector head and the orthogonalization constraint hyperparameter in the loss function. We believe there’s no inherent contradiction between our findings and RankMe/LiDAR, and we complement the previous studies by providing a theoretical basis for how these design choices impact SSL representation learning.  \nTo elaborate on the relationship between the design configurations and the dimensionality of the features, Agrawal et al. [4], demonstrate that the same number of units in the projector can yield different representation space dimensionalities depending on the orthogonalization constraint. One can reduce the number of neurons in the projector head while maintaining high dimensionality in the representation space as long as one does not reduce it beyond a threshold. Therefore, our results do not contradict the findings of RankMe, LiDAR, and $\\alpha$-ReQ. Instead, our work presents a complementary theoretically grounded explanation of how these design factors affect representation learning in SSL frameworks and under what conditions can good representations be learned using projectors with fewer units. But, in line with the reviewer’s question, RankMe, and LiDAR do indicate why one would not want to reduce the projector dimensionality down to extremely low values (e.g. 16 units). We will clarify this point in the updated manuscript.\n\nWe would like to thank the reviewer again for their thoughtful comments and excellent questions. Please let us know if you have any other feedback or concerns.\n\nBest regards,  \nThe Authors\n\n[1] Garrido et al. On the duality between contrastive and non-contrastive self-supervised learning. ICLR, 2023.  \n\n[2] Zhai et al. Understanding augmentation-based self-supervised representation learning via rkhs approximation and regression. ICLR, 2024.\n\n[3] Schaeffer et al. Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations. arXiv, 2024.\n\n[4] Agrawal et al. $\\alpha$-req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. NeurIPS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVN2rRlCPf", "reviewer": "Reviewer_VDXd", "summary": "This paper investigates a theoretical formulation of the contrastive loss used in self-supervised pretrainging of vision models. (One of the assertions of the paper is that all SSL (self-supervised learning) losses are variants of the same loss). The authors propose a more compute-efficient version of the loss objective, and use the theoretic derivations of their analysis of gradient descent dynamics to propose two changes to SSL protocols in practice: the use of a higher orthogonalization regularization coupled with a lower-dimensional projection, and using more augmentations per sample. These recommendations are supported by experiments showing that a lower-dimensional projection coupled with a higher orthogonalization constant leads to higher top-1 accuracy results, and that using more augmentations per sample leads to higher downstream classifier accuracy and faster SSL convergence.", "questions": "* What is the impact of using the new, proposed loss objective, as opposed to using the traditional one on the accuracies of the downstream models?\n* Why does theorem 3.2 imply the loss of orthogonality when using SGD?\n* What are the specific data augmentations used for the experiments?", "limitations": "The limitations were adequately addressed.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Unfortunately the presentation in the paper was quite difficult to understand, which made it difficult to evaluate the paper's strong points (please see 'weaknesses' and 'questions' below. However, the following strengths still seem to be present.\n- There is a fairly strong coupling between the theoretical conclusions and the experimental results, making both more credible.\n- The use of a simplified (from a computational sense) loss objective to use a sample estimate (thm 3.1) is a helpful contribution that may help improving SSL training in general.\n- The experimental results are quite good in context of the baselines, showing higher accuracy at the same data size/similar accuracy at reduced data size, when the paper's recommendations are followed.\n- The authors make it clear where they are building on past results and where the novelty of their contribution lies.", "weaknesses": "The paper is very difficult to read for several reasons:\n- Key terms are not defined inside the paper, making it necessary to reference multiple additional works in-depth just to get a basic understanding of this one. These include:\n    * SimCLR\n    * Barlow Twins\n    * VicREG\n    * backward data augmentation covariance kernels $k^{DAB}$\n    * Infinite Mercer Features and Mercer Theorem\n- Theorem 3.2 is only stated informally in the text, and is not restated formally in the appendix (though theorem B.2 seems like a restatement of the informal version of the theorem). Perhaps because of this, it is not clear at all from the statement of the theorem why SGD chooses redundant directions. \n- Typos around terminology, for instance in Figure 2, it seems that $\\lambda$ should actually be $\\beta$. Also, the scale for $\\lambda$ in Panel B seems very different to that in Panel E. \n- The experiments use the new, proposed loss objective (derived from Theorem 3.1), which should agree with the more computationally-expensive ones in the limit. However, the discrepancy between using the two is not ablated, and no baselines are used that use the original loss formula, even where it might be computationally feasible  (where $m$ is lower).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable suggestions for improving our paper. We appreciate the time and effort you've invested in providing this feedback. We acknowledge the shortcomings in our initial presentation, particularly the absence of a comprehensive overview of existing self-supervised learning (SSL) methods such as SimCLR, BarlowTwins, and VICReg. We are committed to rectifying these issues to ensure our paper is informative and comprehensible to a broad audience. Below, we address each of the reviewer's specific concerns in detail:\n\n1. **Lack of Defined Key Terms:** We apologize for the oversight in not defining some key terms within the text. To rectify this, we will incorporate a comprehensive preliminary section that outlines the foundational SSL methods referenced in our study. Additionally, we will append a glossary of formal terms, including $𝑘_{𝐷𝐴𝐵}$, in the Appendix and expand Section A to include a clear definition and intuitive explanation of Mercer features and Mercer’s Theorem.\n\n2. **Formal Restatement of Theorem 3.2:** We appreciate the reviewer's attention to the formal restatement of Theorem 3.2. We acknowledge the confusion caused by not linking Theorem 3.2 to its formal counterpart in the Appendix. We will amend the main text to explicitly state that Theorem 3.2 is formally restated and proven as Theorem B.4 in the Appendix. Furthermore, we will explain the theorem's implications regarding the selection of redundant directions by SGD in linear networks.\n\n> Theorem B.4: (Formal) Let  $\\Gamma = V\\Lambda V^T$ represent the eigendecomposition of $\\Gamma$, and define $z$ as the projection of the weight vectors in $W$ onto singular vectors of $\\Gamma$, V. Formally, $z = WV$. Assuming small initialization (as in Simon et al. (2023), i.e. $\\| z_{pi}(0) \\| << 1$ for all $p,i$, we can derive the following conclusions:\n> 1. $sign(\\frac{\\Delta z_{pi}(t)}{z_{pi}(t)}) = sign(\\lambda_i)$\n> 2. For all $\\lambda_i, \\lambda_j > 0$, $\\frac{z_{pi}(t)}{z_{pi}(0)} = (\\frac{z_{pj}(t)}{z_{pj}(0)})^{\\frac{\\lambda_i}{\\lambda_j}}$ \nwhere $\\lambda_i$ denotes the $i^{th}$ singular value, i.e. $i^{th}$ element of diagonal matrix $\\Lambda$.\n\nAs noted, the proof of the above theorem is presented in Appendix B (Pg 16-17).\nThe insights derived from this theorem are as follows:\n1. If $\\lambda_i = 0$, $z_{pi}(t) = z_{pi}(0)$ and if $\\lambda_i < 0$, then $lim_{t \\to \\infty} z_{pi}(t) = 0$.\n2. The alignment between a weight vector, say $W_p$ and a singular vector of $\\Gamma$, say $V_i$, i.e. $z_{pi}(t)$ depends on the corresponding singular values, i.e. $\\lambda_i$ and $z_{pi}(0)$.\n\nTheorem B.4 presents the implicit bias of gradient descent as an optimizer. We show that those eigenfunctions of $\\Gamma$ that are aligned with the feature space of the network at initialization and correspond to high singular values are more likely to be learned. Therefore, under weak orthogonalization constraints, the learned representation space will over \"represent\" the strong singular vectors, thereby losing orthogonality. We will further discuss these insights and implications in Appendix B (Pg 18).\n\n3. **Typographical Errors:** We thank the reviewer for identifying these errors and will diligently correct them to enhance the manuscript's clarity.\n\n\n4. **Impact of the Proposed Loss Objective:** \nThe reviewer is correct that the new loss formulation should agree with the less computationally efficient older formulations of the loss in the limit of long pretraining. The performance gap between the new loss formulation and original formulation, as identified by the reviewer, arises from the faster convergence of the efficient formulation in the limit of finite training budget (100 epochs in most of our experiments). To confirm this, we conducted extended experiments (400 epochs) to confirm that both formulations converge to similar levels of accuracy. This finding is detailed in the rebuttal document and Appendix E, Figure 18. We will update the main text to emphasize this equivalence.\n\nWe would like to thank the reviewer again for their thoughtful suggestions and excellent questions. We will carefully consider your points as we revise the presentation of our results and make the proposed changes. Please let us know if you have any other feedback or concerns.\n\nBest regards,  \nThe Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their thorough and insightful comments and suggestions. Below, we summarize the major points addressed in individual responses to the reviewers' comments.


1. **Inclusion of Results for ImageNet-100:**
We have added results for ResNet-18 pretrained using Barlow Twins on ImageNet-100 (please see attached rebuttal pdf). This addresses the reviewers’ request for additional empirical evidence for the utility of low-dimensional projectors. We are currently running experiments for Imagenet-1k, which we will add in the final version of the paper.

2. **Extended version of Table 1:**
Reviewer rDTK pointed out that there are major jumps in the projector dimensionality values presented in Table 1. Therefore, we present an extended version of Table 1 which indicates the performance for projector dimensionality of 64, 128, 256, 512, 1024, 2048, 4096 and 8192 (please see attached rebuttal pdf). We will update the Table in the main text for the final version of our paper.

2. **Clarification on Lower Dimensional Projections:**
We clarified the apparent discrepancy between our results and empirical findings from RankMe and LiDAR, as raised by reviewer RyLx. Specifically, we study the effect of design space characteristics of the SSL pipeline on the learning dynamics, whereas the aforementioned works characterize the dimensionality of the learned representation space of pretrained networks. We highlighted that the properties of the learned representation space and downstream performance depend on the interplay between projector dimensionality and the orthogonalization constraint, $\beta$. Our explanation demonstrates that our results are complementary to existing works like RankMe and LiDAR, providing a deeper understanding of representation learning in SSL frameworks.

3. **Explanation of Downstream Performance Claims:**
We clarified the relationship between faster convergence on the SSL pretraining loss and downstream performance. We detailed how the formal version of the SSL desiderata (invariance to semantic information-preserving augmentations) influences good feature learning. By explaining that a feature space with the desired properties facilitates easier kNN or linear readout, we highlighted how this leads to better downstream task performance. We believe that our work now more clearly shows the theoretical grounding of the framework's claims about improved convergence and representation learning.

4. **Acknowledgment of Typos and Miscellaneous Comments:**
We apologize for the confusion in notation of the orthogonalization constraint, $\beta$. We have used the updated notations in the results presented in the attached rebuttal pdf, and will update the figures in the final version of our paper. We have also addressed other presentation issues pointed out by reviewers, including missing definitions and formal versions of theorems, which will be updated in the main text of our paper. We hope that these corrections will improve the clarity and readability of the paper, making our work more accessible to a wider audience.

Best regards,  
The Authors

Meta-review: The paper introduces a novel, computationally efficient loss function for self-supervised learning (SSL) that outperforms existing methods. By requiring fewer projector dimensions, the new loss function accelerates convergence, making it highly valuable for large-scale SSL applications. The authors provide novel theoretical insights by identifying a key equivalence between SSL objectives when using an augmentation kernel, deepening the understanding of SSL methodologies. These insights lead to practical recommendations, such as employing lower-dimensional projectors and multiple augmentations, which have been empirically validated to improve both accuracy and convergence speed. The authors made a great effort in the clarity of presentation making the work accessible to a broad audience. This combination of novel theoretical insights, practical recommendations, and robust empirical evidence positions the paper as a significant contribution to the field of SSL.
Decision: Accept (poster)