Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning
Authors: Hugues Van Assel
Mark Ibrahim
Tommaso Biancalani
Aviv Regev
Randall Balestriero
Venue: NeurIPS 2025 spotlight ()
Keywords: Self-Supervised Learning
Representation Learning
Primary area: deep_learning
TL;DR: For datasets with high-magnitude noise features, joint-embedding is more robust than reconstruction for self-supervised learning.

Abstract: Reconstruction and joint-embedding have emerged as two leading paradigms in Self‑Supervised Learning (SSL). 
Reconstruction methods focus on recovering the original sample from a different view in input space. On the other hand, joint-embedding methods align the representations of different views in latent space.
Both approaches offer compelling advantages, yet practitioners lack clear guidelines for choosing between them.
In this work, we unveil the core mechanisms that distinguish each paradigm. 
By leveraging closed-form solutions for both approaches, we precisely characterize how the view generation process, e.g. data augmentation, impacts the learned representations.
We then demonstrate that, unlike supervised learning, both SSL paradigms require a minimal alignment between augmentations and irrelevant features to achieve asymptotic optimality with increasing sample size. 
Our findings indicate that in scenarios where these irrelevant features have a large magnitude, joint-embedding methods are preferable because they impose a strictly weaker alignment condition compared to reconstruction-based methods. 
These results not only clarify the trade-offs between the two paradigms but also substantiate the empirical success of joint-embedding approaches on real-world challenging datasets.

Joint- vs Reconstruction:
Provable Benefits of Latent Space Prediction for Self-Supervised Learning
Hugues Van 1,,, Mark Ibrahim² 3, Tommaso Biancalani, Aviv Regev Randall Balcstricro 2,3
Genentech, 2 Brown University, 3 'Meta AI, FAIR
Abstract
Reconstruction and joint-embedding have emerged as two leading paradigms in Self-Superriser Learning (SSL). Reconstruction methods focus recov- ering the original sample from a different view input space. On the other hand, joint-embedding methods align the representations of different views latent space. Both approaches offer compelling advantages, yet practi- tioners lack clear guidelines for choosing between them. In this work, we unveil the core mechanisms that distinguish each paradigm. By leveraging closed-form solutions for both approaches, preciscly characterize how the view generation ion process, e.g. data augmentation, impacts the learned representations. We then demonstrate that, unlike supervised learning, both SSL paradigms require a minimal aligmment between augmentations and irrelevant features achieve asymptotic optimality with increasing sample size. Our findings indicate that in scenarios where these irrelevant features have large magnitude, joint-embedding methods are prefera ble because they impose a strictly weaker alignment condition compared to reconstruction-base methods. These results not only clarify the trade-offs between the two paradigms but also substantiate the empirical suecess of eebtddenb approaches on real-worls challenging datasets.
Introduction
Training deep neural networks to extract informative data representations is central to AL. Numcrous familics of methods pursuc this goal 62 In supervised learning, one does so by prescribing labels that encode what is considered informative in the data. While this has been the dominant approach to representation learning over the past decades, it has become clear that. labels often overly specialized. Such specialization prevents learning representations that transfer across an ever-increasing diversity of downstream tasks 24 377. Self Supervised Learning (SSL) has emerged an alternative that moves away from labels A 50, [4] In SSL, one does not assume a priori what is informative; instead, one specifies which variations are uninformative and should be disregarded. Identifying, priori, the invariances representation should satisfy is broadly applicable principle. For instance, many downstream tasks involving natural images, such as recognition, counting, and segmentation, are inherently robust to minor changes in color or illumination. Consequently, these lasks benefit. from representations that exhibit such invariances, lypically encoded through a data-an anuggeetaanooo process. Two primary families of methods have emerged to learn representations using this principle: reconstructtonn-aaee and joint- eebedddnn approaches.
"Contact: van_assel. van_asssel.uguussgenne..
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Reconstruction
Joint Embedding
T
7
Sw
TI
T
x
T(X)
fE
fD
72 T
T2(x)
fw
Two self-supervised learning paradigms studied in this work. Left: Reconstruction problem of Equation [SST-RCC: random augmentation An encoder fE together with a decoder fD is trained to recover x from 7(x). Right: Joint embedding problem of Equation (SSL-JE): two independent augmentations T T of the same are mapped by fw to nearby representations, while embeddings of different inputs are pushed apart.
Figure 1:
T is applied to to form r(x).
1.1 The Reconstrruction-bssed approach
Resonstruclion-bbseed approaches train models by augmenting input signal, typically by adding noise or masking, and then training the model to restore the original input Gr (left side of Figure [ This process encourages the model to learn meaningful internal representations of the data's underlying structure and content to enable successful reconstruction. However, because the learning signal arises from minimizing reconstruction еrror in the input space, the model is naturally steered toward subspaces that explain the majority of the input's variance 65 g Whether such varianco-cxplaining fcatures are also the most scmantically discriminative or useful for downstream tasks depends strongly on the data modality.
41
In language, reconstruction--ert learning, as used in large language models, is highly effec- tive because textual tokens represent compact, semantically meaningful units that already abstract away most low-level variability. Although data quality may vary, language in itself highly compressed and rich modality where reconstruction can prove highly successful. Predicting missing token provides a learning signal that operates directly in semantic space: to succeed, the model must infer the contextual and syntactic relationships that determine mcaning, rather than replicate surface patterns. Consequently, minimizing reconstruction error encourages the emergence of abstract relational representations, capturing composi- tionality, long-range dependencies, and discourse coherence that align closely with human notions of meaning and reasoning
62
In contrast, in visual domains, variance-explaining features often emphasize aspects of the data that are statistically dominant but semantically shallow. Unlike language, visual data are essentially sensorial recordings of the physical world, capturing raw information without inherent semantic compression. As a result, pixel-level reconstruction objectives tend to drive models toward capturing local statistics and textures that account for most of the input's variance, rather than the higher-order structures and object-level relationships that underpin semantic understanding. This local bias can result in representations that are well-suited for low-level perceptual fidelity but suboptimal for recognition, categorization, or other tasks that depend on global context and semantic abstraction G 28]. Consequently, purely reconstrucctonn-hesd approaches in computer vision often struggle produce features that generalize well across tasks without additional supervision or adaptation. Fine-tuning thus frequently necessary to bridge the gap between variance-focused representations and those that encode meaningful, task-relevant semantics 3T.
1.2 The Joint-embedding approach
Joint-embeldinng methods, in contrast, operate entirely latent space (right side of Figure D) Their objective is to produce similar representations for different augmented views of the

---

same input while ensuring that representations of distinct samples remain dissimilar. This separation can be enforced explicitly through contrastive loss 14 or implicitly via architectural mechanisms such as self-distillation, stop-gradient operations, momentum encoders, or predictor heads that stabilize training and prevent representational collapse even without negative samples 13
26
66
B8
Unlike reconstruction--asse approaches, joint-embedding methods do not predict in the input space and are therefore less biased toward capturing high-variance components of the signal. Empirically, joint-embedding frameworks have shown strong performance across domains where the input signal is high-dimensional and semantically diffuse. Successful applications span histopathology [73, Earth observation [T], and video representation learning [ Despite this progress, the mechanisms through which latent consistency objectives outperform reconstruction- basod ones remain poorly understood, motivating the analysis presented in this work.
1.3
Contributions
The critical role of the prediction target in SSL, specifically whether to predict in the input space (reconstruction) the latent representation space (joint-embedding)) has been demonstrated numerous times 1. However, it remains unclear when to favor one approach over the other. This work clarifies when to prefer cach. Our key findings can be summarized as follows.
We derive closed-form solutions for both reconstruction-basee (Theorem 3.1 and joint- embedding (Theorem 3.2 SSL linear models. This enables a precise characterization of data augmentation impacts, analogous to well-known results in supervised learning [.
We then leverage these results to show that optimally aligning the augmentations with the irrelevant components of the input signal effectively eliminate these components and recover optimal performance for both families of methods (Propositions 4.3 and 4.4). However, in contrast the supervised learning scenario (Proposition 4.2) simply increasing the sample size cannot overcome any misalignment between the augmentation and the noise (Propositions 4.3 and 4.4).
By inspecting the alignment requirements for both reconstruction and joint-embeddinng methods, we show that in settings with low-magnilude irrelevant noise feat reconstruclion methods are preferable, as they require fewer tailored augmentations (Corollary 4.5 Con- versely, in scenarios with migh-mtagettdee irrelevant noise features, t.e, where such features significantly impact the input signal, eeiiit-imbeeddiig methods should be preferred, as they impose strictly weaker alignment condition than reconstruction methods (Corollary 4.5
In Section 5 we experimentally validate these findings on both vectorial and image data. We demonstrate that joint- crint-embedding methods such as DINO 13] and BYOL [27] are considerably more robust to severe data corruption than reconstruction- based methods like we further provide experimental validation for key results from our theoretical analysis. These experiments show that: (i) SSL methods exhibit significantly greater sensitivity to corruptions compared to supervised learning methods (Appendix D.2 and Figure 2); and (ii) SSL performance in noisy data settings is enhanced
MAE BI (Section 5.2h In Appendix D
by aligning augmentations with the underlying noise (Appendix D.3 and Figure 2
Background
Throughout, we consider samples X (x1.... x,,,,,,,)))) XT) Raxd and associated labels
Y (yI,...,,,,) R We consider data augmentation distribution T defined as distribution over transformations 5 Rd -
+ gd
Supervised Learning. For the regression task of predicting labels Y from observations X, the augmented empirical risk minimization problem is as follows:
min V
Σ ie
E [lly: - fv(T(xx))
(SL)
<E[n]
3

---

Interestingly, when using linear model fv x Vx with V E R the effect of data augmentation in Equation (SL) can be explicitly characterized as a Tikhonov regularization
problem as shown in Lemma B.1
46
11 which proof is provided in Appendix B
1 Σ T [ly, VT
I
Σ 1
ly: VE,-T [T(x:)] 12
(1)
n
ic|n!
where | = -TT(VEV] and
Σ
Σ E.-T (x(x))xx)))
ETOT (xxx)) ETAT [T(x,]]
(Cov)
denotes the covariance of the augmented samples. Therefore, the effect of data augmenta-- tion within supervised learning using linear models well understood from a theoretical standpoint.
Lack of Foundations in Self-Supervised Learning. Similar results are lacking for SSL,, where the explicit effect of data augmentation for linear models has not been rigorously studied.
Despite recent efforts clucidate the underlying principles 64 56
20
22
these methods remain only superficially understood 63 A robust statistical framework
still lacking to fully comprehend SSL methods and to position them relative to their
supervised learning counlerparts B]. Key open questions involve precisely characterizing the role of data augmentation in shaping final representation
within both reconstruction and joint- deitt-embedding frameworks. This work aims to lay the foundation for filling this gap.
3
Augmentation-Awan Closed-Form Solutions
In this section, we derive closed-form solutions for the two main families of SSL methods: reconstruction-bant based and joint-embedding approaches. To the best of our knowledge, the following results are the first. instances of closed-form solutions for SSL that are directly parameterized by the data augmentation structure. This stands in contrast to previous solutions highlighted in 5., which focused on the dependency graph between augmented samples and were unaware of augmentations. These results will then allow us to analyze how augmentations affect the learned representations in both families of SSL methods.
In line with previous theoretical works focused on analytical tractability 12 we study models that are lincar in their parameters. Note that these models can produce arbitrary nonlinear predictions via appropriate feature maps 3 and are known to describe regimes of wide neural networks 44
54 57
3.1 Reconstruction-Based Self-Supervised Learning
We first consider reconstruction-bont based SSL models. The problem can be framed as follows, where T is the data augmentation distribution:
1 Σ ENTT [[|x; fo((f(((xx)))11]
min
(SSL-RC)
iE|n]
In this formulation, each data sample is augmented and then passes through an encoder fE. followed by decoder fD. The objective is to minimize the reconstruction error between the original sample and the reconstructed one. This methodology is analogous to the Denoising Auto-Encoder 59 Masked
32 and similar frameworks. Interestingly, the reconstruction problem can be solved in closed form when considering linear models for both encoder and decoder. All proofs can be found in Appendix
A
Theorem 3.1. Let X; == BT-T(T(xx)) for each i E [M]] and define X = (x,,
Assume that n the singular value decomposition:
x X Σ is positive definite where Σ is defined in Equation Cov
Consider
X 1X x
-RPP
(2)

---

subject to where R E Radd and P E Raxd are orthogonal and +b = diag oa) with Φ1 2 ... Pd 2 O. Solutions of Equation [SLL-RC) for fE! x Ex and fD x + Dx take the form:
>) and D R.
= TP
[X X
where T is any invertible matriz in B PA and R₃ are the first &o columns of P and R respectively, and 4 diag(ó)....,
3.2 Joint-Embedding-Bane
Sclf-Supervised Learning
We now consider a joint-embedding SSL problem formulated as follows, where fw the SSL model and T is the data augmentation distribution:
min W
E T [l])((((x:))) -
n
iE
(SSL-JE)
E ETT (w((((x))))((((()))) = Ig.
In the above Equation SSL-JE
the objective represents the usual invariance term which ensures consistency between two augmented views of the same sample and is component of cront-embeddingg methods. The constraint enforces orthonormality in the learned representations, promoting diversity in the representation space 64 and thus preventing collapse. Most point-embeddie models incorporate similar repulsion term, either explicitly within the loss function, such as
770, Sim SimCLR 14, VICReg [0, and MoCo [3), or implicitly through architectural design choices, as demonstrated by BYOL [6]] and DINO B. In our case, we rely on the sum of the outer products of the representation vectors. This approach closely resembles VICReg [10), specifically its covariance regularization term. Interestingly, under the unifying formalism presented
2 most popular eebeddenb methods can be framed with this simple repulsive term.
The problem of Equation linear SSL model, as formalizer below.
JSL can also be solved in closed form when considering
Theorem 3.2. Let == E,,,, [xxx))((x))T] G G & [7xx)) EToT [x(x)) Assume that S is positive definile. Consider the eigendecomposiiton::
S GGS QUQT
where N = diag(w)..... "wolw with wi 2 ... 2 wd- Solutions of Equation SSL-JE for linear
model fw x + Wx take the form:
W UQ
(5)
where Q = (qi...., qk) and is any orthogonal matriz of size k.
Augmentation Alignment Requirements in Self-Supervised Learning
In this section, we build on the results of Section 3 to evaluate the ability of both families of SSL models to reach optimal performance. To define such notion of optimality, model our data as having important signal components and d k pure noise components. These noise components represent the variations that SSL methods are typically designed to be invariant to, e.g. background noise for image classification tasks. An idcal SSL encoder would rotain the k informative important dimensions while discarding the d k noíse components.
We formalize this scenario in Section 4.1 where a parameter a is introduced to control the alignment between the irrelevant features and the augmentations. In Section 4.2 we demonstrate that supervised learning models can effectively achieve optimal performance either when augmentations are well aligned with the irrelevant noise features at finite sample
5

---

Supervised
Supervised Corrupted Augmentation Unaligned
VICReg Augmentation Unaligned
VICReg Corrupted Augmentation Unaligned
VICReg - Corrupted
Augmentation Unaligned
Augmentation Aligned
Figure Injecting corruption ceruption-aligned noise into data augmentation improves SSL representa- tion quality on corrupted CIFAR-10. Thus aligning augmentations with the irrelevant components in the data is crucial in SSL. t-SNE [58] visualizations of (left right): (1) Supervised features (penultinate layer), clean data.
(2) Supervised features (penultimate layer), fog-corrupted (severity 5). (3) VICReg representations, clean data. (4) VICReg rep- resentations, fog-corrupted (severity 5). (5) VICReg representations, fog-corrupted (severity 5) with fog noise (severity 1) injection during augmentation. Unlike supervised features, VICReg represenlations degrade significantly under corruption (compare 3 and 4). Injecting
noise in the data augmentation (5) euhances class separability.
sizes or when the sample size large, regardless the augmentation employed. In Section 4.3 we show that, unlike supervised learning, SSL necessitates sufficiently good alignment to achieve optimal performance, even in the infinite sample limit. Finally Section [1. we compare the aligument requirements of joint-embedding and reconstruction based SSL methods, thus providing insights into the characteristics of both families of methods.
4.1 Data, Noise and Augmentation
We consider input dataset with two parts: important features and irrelevant noise. Optimal performance on downstream tasks achieved when using only the important features. Let X LKQ be the singular value decomposition of the important features, where K = diag(Ki..... fid) is the diagonal matrix of singular values. Each sample x, is corrupted by additive Gaussian noise constituting the vrrelevand features:
ViE [n]] =; 7 %t ~ (O,,T).
66)
with Y drawn independently across i E [n] and where Radd is positive cemi-definite. For simplicity, we assume that I is diagonalized by the same orthonormal matrix Q from the SVD above d.e. r = QArQ where Ar diag(Af..... .A, ..)). The matrix X (x),..., 8n) T E Raxd then forms the corrupled input data. We assume that the importanl features are concentrated in exactly k 2 components, meaning K; > for i [k] and K; = 0 for >> These components are referred to the imporland componenls. Additionally, we assume that the irrelevant noise are mull these h components, ie. X for all E [[]], and strictly positive otherwise, i.c. XT > for i [[k + d: We refer the [[ 1: d] components as the noise components.
Data augmentation. Let θ Rdxd be positive semi-definite and diagonalized by Q i.e. θ = QΛaQ where Ao = diag(A9 ......,,
..).. We consider the augmentation distribution,
V, T(a) = (T: (T: Rd -(x)= = x x a0, N), 7 ~(O. (7)
where 0 and are drawn independently for each transformation. Note that the term follows the same distribution as the noisy irrelevant features. Increasing the magnitude of thus aligns the data augmentation with these irrelevant features.
Remark 4.1. One can extend our result to augmentations beyond Gaussians by considering
any augmentation of covariance (defined in Cov O + rF 46

---

4.2 Supervised Learning Consistency Regardless of Augmentations
We first analyze the behavior of supervised learning models by identifying regimes in which the supervised model effectively disregards the noisy irrelevant features in X. This provides the foundation for pinpointing key differences between supervised and SSL models in Section 4.3 We rely on the data augmentation T(a), where a E IR+ controls the alignment between the data corruption and the augmentation process presented in Section 4.1
Proposition 4.2. [Superoised Leorning) Let V (resp. V*) be the linear model solving Equation SL with augmentation T(a) for X (resp. the corrupted X). The limit:
V
(8)
holds almost surely in either of the following regimes:
as a + too (perfect augmentation-nois alignment) for any fized sample size E N.
+ +oo (infinite samples) for any fized alignment a 2 0.
The above result shows that, when performing supervised learning with corrupted data, the model can achieve the same performance if it were trained only on the important features (thus achieving optimal performance) if either of the following conditions holds: i) The data augmentation process is well aligned with the noise corrupting the inputs (ar large). ii) A sufficiently large sample size is available to compensate for any misalignment between the augmentation and the input noise (n large).
4.3 Self-Supervised Learning Requires Aligned Augmentation and Noise
Building on the closed-form expressions for SSL provided in Theorems 3.1 and 3.2 we are now interested in studying the ability of SSL models to achieve optimal performance when trained on the corrupted dataset X, as defined in Section 4.1
Proposition 4.3. [Reconstruction] Let E" (resp. E* be the linear (encoder) model solving Equation SSL. for X (resp. the corrupted X). The limit:
E
E
hold almost surely in either of the following regimes:
as a + to0 (perfect angmentation-noise alignment) for any fioed sample size n. E N.
as n + too (infinite samples), if and only if the alignment a 2 0 salisfies:
X
1 -1
a aRc
max se[k-1:d
where
7
min KE[M]
(10)
l:ll
n
X
V + x
Proposition 4.4. [Joint- Soiinl-Emeeddnng/ Let W (resp.
W+) be the linear model solving
Equation JE for X (resp. the corrupted X). The limit:
W
W
(11)
hold 3
almost surely in either of the following regimes:
a + +00 (perfect augmentation-nois alignment) for any fized sample size N.
"Up to arbitrary invertible matrix (i.e., if E² solution, so TE* for any k x invertible matrix T).
Upp to arbitrary orthogonal rotation (i.e., if W is a solution, so is UW" for & x k orthogonal matrix U).

---

a3 nt + too (infinite samples), if and only if the alignment a 2 0 satisfies:
d
1 k
a² OJB
max x [k- l:d]
where 8 min ZE [
(12)
# + 1
The above Propositions 4.3 and 4.4 reveal that, when augmentations are well aligned with the noise i.e. when a is large enough, undesired noise components are removed from the obtained SSL representation for both families of models, even when combined with other augmentations. However, the alignment requirement persists even as the sample size n becomes arbitrarily large. Therefore, in SSL,, achieving optimal performance requires that the data augmentation process be sufficiently well aligned with the irrelevant noise features in the data. This marks key difference from supervised models. Unlike SSL, supervised models can overcome misalignment between augmentations and noise with enough samples, as it learns robustness by observing different noise realizations across identically labeled data. This underscores the critical role of augmentations in SSL. Figure 2 illustrates this by visualizing the embedding spaces of a supervised model and the VICReg [T] SSL model, revealing the latter's 's susceptibility to noise when augmentations lack proper alignment. This finding consistent with an empirical study by [TT, which concluded that improving augmentations is more impactful than altering architectural designs.
Interestingly, the above results reveal that reconstruction (Proposition 4.3 and joint- embedding (Proposition 4.4) methods exhibit different alignment requirements to achieve optimal performance. We next analyze these differences.
4.4 Comparison of Joint- Joint-Embedding and Reconstruction-Baseed Methods via Augmentation Alignment Requirement
Leveraging Propositions
4.3 and 4.4
we obtain the following key result.
Corollary 4.5. Lel OCE,, 5, ORC; and v be defined as in Proposition 4-4 and Proposition 4.3
If maxx Xo < (low noise), then aJe ane (reconstruction is preferable).
v/ (high noise), then OJE onc (joint-embeddingg is preferable).
If mingg]R++1 X
This result shows that when the spectral norm of the noise covariance T is small, recon- struction-baser methods impose a less stringent alignment compared to joint-embedding methods. Conversely, when the noise magnilude is large, joinl-embeedding methods exhibit lower sensitivity to the augmentation-nooi counterparts. Ultimately, the goal is to minimize the alignment requirement, the irrelevant components are typically unknown in real-world applications.
aligmment compared their reconstructions based
insonstructtion-based approaches are therefore well-suited for scenarios with weak, irrelevant noise features. Intuitively, the core important components in such scttings possess the greatest magnitude and are consequently prioritized by the model during reconstruction. Due to their reconstruction objective, these mothods depend less on data augmentations, which explains their superior performance and robustness over joinl-embeddingg techniques in this particular context.
However, when strong noise features obscure the important features within the raw input signal, joint-em eedntt-ebeddiigg methods demonstrate greater robustness. This is because joint- embedding techniques prioritize latent space prediction, thereby bypassing the need to reconstruct irrelevant noise components as model outputs. Consequently, in real-world scenarios where the extent of irrelevant features (typically image backgrounds or experimental batch offects in scRNA-soq data) cannot be preciscly quantified, joint-embedding approaches appear more reliable. This preference is further underscored by the principle that high- amplitude noise naturally exerts a more significant. impact on the final representation than low-amplitude noise, making robustness crucial when noise levels are uncertain. This practical advantage also explains the community's 's preference for joint-embedding approaches in challenging datasets
73 4 20
8

---

Supervised Learning
Joint-Embedding
Reconstruction
Distance to Optimalty
-
10
10
Weak Noise
10-3
10
10
10
10
10
10
10
10
10a
10
10
10a
10*
10a
10a
10
Sample Size
Sample Size
Sample Size
Distance to Optimallty
10
3x10
Strong Noise
10
10a4
x
10-3
10
10
10
6x10
10'
10a
10*
10a
10a
10*
10a
10
10*
Sample Size
Sample Size
Sample Size
Alignment Between Augmentation and Noise
Weak (a = 0.1)
Medium (a 1)
Strong (a= 10)
Figure Performance of linear supervised and SSL models (Sections 3.1 and 3.2 and Theo- rems 3.1 and 3.2 MNIST corrupted with synthetic Gaussian noise (Section 4.1 with various augmentation alignment a Section 4 Each subplot's y-axis is the absolute difterence of supervised linear probing loss (on clean vs. corrupted data) and its x-axis is the sample size n. This figure highlights that jpint-embeilding presence of strong irrelevant noise features. Conversely, reconstruction requires less tailored augmentation when dealing with weak irrelevant noise features. Weak noise corresponds to X 10 and strong noise to A 10 (details in Appendix Ch.
preferable to reconstruction in the
Key takeaway. When irvelevant features have low magnitude and there is limited prior information effective augmentations, reconstruction is preferable. In contrast, when these irrelevant features are non-negligible (as common with real-world data) or effective augmentations can be identified, ceint-embedding is preferable.
Experiments
This section validates the theoretical findings of Section 4 through experiments linear models (Section 5.1 and deep networks (Section 5.2). These experiments confirm that the results from the linear model are consistent with those observed in the nonlinear setting of deep networks.
5.1 Experiments With Linear Models
We first validate the theoretical results of Section 4 through experiments with linear models. Data features corrupted by adding synthetic Gaussian noise, allowing to precisely control the noise magnitude and its alignment with data augmentations. The details of our experimental design are provided in Appendix C Our primary results are illustrated in Figure 3 This figure effectively illustrates contrasting behaviors between the various types of methods as sample size and noise magnitude vary. On the left, one can notice that the supervised model achieves optimal performance with cither increasing sample size or increasing alignment, with any augmentation and regardless of the noise magnitude,
confirming the result of Proposition 4.2 In contrast, SSL models exhibit different sensitivities,
as predicted in Propositions 4.3 and 4.4 The middle panel shows that joinl- eebnlleebeddding indeed requires a minimal aligument between augmentation and noise to reach optimal performance Notably, joint-embeddinng maintains robust ness even with increasing noise magnitude, shown in Corollary 4.5 On the right panel, can see
(as predicted Proposition

---

Table 1: Linear probing top1 accuracy scores of MAE, DINO, and SimCLR on ImageNet with various corruptions 3 and relative performance drop from severity to
Pixelate
Corruption
Gaussian Noise Corruption
Zooublur Corrpttion
Method
Sev.
Sev.
Sev.
Drop (%%
Sev.
Sev. 3
Sev. 5
Drop (96)
Sev.
Sex. 3
Sev.
Drop (%)
BYOL
66.7
61.3
58.7
12.0
67.2
63.1
56.4
16.1
70.1
67.0
63.8
9.0
DINO
68.7
64.9
00.2
12-4
67.6
62.4
09.0
12.7
69.4
67.2
61.9
MAF:
64.9
52.3
46.8
27.9
61.6
46.7
44.8
27.3
64.1
58.4
51.3
20.0
that under weak noise conditions, reconstruction is robust to the choice of augmentation. However, as noise becomes stronger, reconstruction performance degrades and necessitates strong alignment between augmentation and noise. These obscrvations confirm the results
of Propositions 4.3 and 4.4 and Corollary 4.5 These findings are further supported by experiments on other datasets, including Fashion-MNIST. Kuzushij-MNISST and the single- cell RN A-seq data from [9,, presented in Figures 4 in Appendix C All experimental
outcomes align with and confirm the theoretical insights delailed in Section
5.2 Experiments with Deep Networks on Images with Various Corruptions
We conduct experiments using both VIT 21 and ResNet 8 architectures. Our evaluation focuses on top-1 linear probing accuracy on ImageNet-100 contain non-negligible features irrelevant to the classification task [ (e.g. background noise elements), which contributes the superior performance of foint-embedding methods over
ImageNet images inherently
reconstruction methods, as demonstrated in several benchmarks 17 [5 To further introduce and control the magnitude of irrelevant noise features, we utilize the ImageNet- C corruptions [5], which offers corruptions at various severity levels. The results are presented in Table [ The performance of MAE (ViT) BII, which uses reconstruction objective, much more affected by the increasing corruptions than DINO (ViT) [3] and BYOL (ResNet) 266 which use a joint-embedding objective. Indcod, there 25.1% average drop accuracy for MAE when the severity of the corruption increases from 1 to 5, while DINO and BYOL only experience a 10.5% drop and 12.4% drop, respectively. All experimental details are provided in Appendix [.1
We perform further ablation studies in the appendix (Appendix D highlighting this paper's key results. Specifically, our experiments confirm that: (i) SSL is considerably more sensitive to the alignment between augmentations and noise than supervised learning (see Appendix D.2h, and (ii) aligning augmentations with the underlying noise can enhance the performance of SSL models in noisy data scttings (sce Appendix D.3
6 Conclusion
A growing body of work demonstrates that joint-embedding methods often outperform reconstruction methods on real-world datasets, particularly whore extracting useful features for downstream tasks from raw signals is challenging [40] (2]. This is further supported by a consistent empirical finding: reconstruction methods typically necessitate fine-tuning to address the inherent misalignment between the feat ures they learn and those that are 6i In this work, we have established theoretical framework to explain this phenomenon. Our analysis provides clear guidelines for practitioners: opt for reconstruction when irrelevant components show low variance and prior information on effective augmentations is scarce. In contrast, prefer joint-embedding when those irrelevant components have high variance magnitude, as is common with real-world data, when effective augmentations are either readily available or can be found through cross-validation.
perceptually useful for downstream applications 72
This paper offers basis for future work. One could consider extending these results to finite sample size settings to precisely characterize the interplay between sample complexity and augmentation in SSL.
10

---

References
[4] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yamn LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE SEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15619-15629, 2023.
[2] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingan, Simon Kornblith, Ting Chen, et al. Big superssp models advance medical image classification. In Proceedings of the IEEE/CVE international conference on computer vision, pages 3478-3488, 2021.
3 Francis Bach. Learning theory from first principles. MIT press, 2024.
[4] Randall Balestriero, Mark Tbrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXic preprint orXiv:2304.1220 2023.
[5] Randall Balestriero and Yann LeCun. Contrastive and non non-contrastive self-supert supervised learning recover global and local spectral embedding methods. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Newrul Information Processing Systems, volume 35, pages 26671-2668
Curran Associates, Inc., 2022.
[6] Randall Balestriero and Yann Lecun. How learning by reconstruction produces unin- formative features for perception. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 1sst International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 2566 2585. PMLR, 21 27 Jul 2024.
[7] Randall Balestriero, Ishan Misra, and Yann LeCun. A data-angmentattin is worth thousand samples: Analytical moments and sampling-free training. Advances in Newral Information Processing Systems, 35:19631 19644, 2022.
[8 Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimization of nonlinear transform codes for perceptual quality. In 2016 Picture Coding Symposium (PCS), pages 1-5. IEEE, 2016.
19] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual reprosentation learning. 2023.
110] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianceiivariancccovvaaa regularization for self-supervised learning. arXio preprint arXiv:2105.04906, 2021.
[11] Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-11) 116, 1995.
[12] Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssl interplay: Angmentations, inductive bias, and generalization. In International Conference on Machine Learning, pages 3252 3298. PMLR, 2023.
[13] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (650-9660) 2021.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[15 Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. Tn Proceedings of the IEEE/CVP international conference on computer vision, pages
2021.
11

---

[16] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Ya- mamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint 2018.
[17] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of supervvid methods for visual representation learning. J. Mach. Learn. Res., 23:56-1 1, 2022.
[18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 288-255. leee, 2009.
[19 Jacob Devlin, Ming-Wir Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 conference of the North American chopter of the ossociation for computational tinguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[20] Michael Doron, Théo Moutakanni, Zitong S Chen, Nikita Moshkov, Mathilde Caron, Hugo Touvron, Piotr Bojanowski, Wolfgang M Pernice, and Juan C Caicedo. Unbiased single-cell morphology with self-supervised vision transformers. bioltvie 2023.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint orXiv:2010.11920 2020.
[22] Yann Dubois, Tatsunori Hashimoto, Stefano Ermon, and Percy Liang. Improving self-supervised learning by characterizing idealized representations. arXiv preprint ar wxXi:2209.0022. 2022.
[33] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211 218, 1936.
[24] Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-training for better downstream transferring. In International Conference on Learning Represenlations, 2022.
[25] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised lcarning. arXiv preprint ar XrX:::2206.022 02574, 2022.
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advonces in neural information processing systems,
2020.
[27] Thomas Grill and Jan Schlüter. Two convolutional neural networks for bird detection in audio signals. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 1764-1768. IEEE, 2017.
[28) Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[29] Jeff HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for sell-supervised deep learning with spectral castrastive loss. Advances in Neural Information Processing Systems, 145000050111 2021.
[30] David A Haslett and Zhenguang G Cai. How much semantic information is available large language model tokens? Transactions of the Association for Computational Linguistics, 13:408 423, 2025.
12

---

[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. ar preprint or arXiv:21 11.211111.677, 2021.
[32] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVL Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVI conference on computer vision and pattern recognition, pages 9729-9733 9738, 2020.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[36] Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of con- trastive self-supervised learning. arXiv preprint arXiv:2111.00743 2021.
[37] Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study the transferability of visual representations with contrastive learning. In 2021 TEEE/CVE International Conference on Computer Vision (ICCV), pages 8825-8835, IEFE.
[38] Abhishek Jha, Matthew B Blaschko, Yuki M Asano, and Tinne Tuytelaars. The common stability mechanism behind most self-supervised learning approaches. arXiv preprint arXiv:2402.1 14957, 2024.
[39] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arxiv preprint orXic:2100.93345 2021.
[40] Bidur Khanal, Binod Bhattarai, Bishesh Khanal, and Cristian Linte. How does self- supervised pretraining improve robustness against noisy labels across various medical image classification datasets? arXiv preprint arX.09,00..07990, 2024.
[41] Diederik P Kingma and Max Welling. Aulo-encoding variational bayes. ar preprinl ar xrXix133226614 14, 2013.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[43] Bipasha Kundu, Bidur Khanal, Richard Simon, and Cristian A Linte. Assessing the performance of the dinov2 self-supervieed learning vision transformer model for the segmentation of the left atrium from mri images. arXiv preprint or 2rXiv:241.11 11.09598, 2024.
[44] Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[45] Feng Liang, Yangguang Li, and Diana Marculescu. Supmae: Supervised masked autoencoders are elficient vision learners. arxiv preprint arXio:2205. 14540, 2022.
[46] Chi-Heng Lin, Chiraag Kaushik, Eva L Dyer, and Vidya Muthukumar. The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective. Journal of Machine Learning Research, (51):):1-85, 2024.
[47] Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang, and Joshua Susskind. How jepa avoids noisy features: The implicit bias of deep linear self distillation networks. orXiv preprint arXiv:2407.03/ 2024.
13

---

[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110:1711.05101, 2017.
[49] Evan Z Macosko, Anindita Basu, Rahul Satija, James Nemesh, Karthik Shekhar, Melissa Goldman, Itay Tirosh, Allison R. Bialas, Nolan Kamitaki, Emily M Martersteck, et al. Highly parallel genome genome-wide expression profiling of individual cells using nanoliter droplets. Cell, (2):11002-1121) 2015.
[50] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVE Conference on Computer Viston and Pattern Recognition, pages 6707-6717, 2020.
[51] Warren Morningstar, Alex Bijamov, Chris Duvarney, Luke Friedman, Neha Kalibhat, Luyang Liu, Philip Mansfield, Renan Rojas-Gord Gomez, Karan Singhal, Bradley Green, et al. Angmentations algorithms: What works in self-supervised learning. arXiv preprint ar 2rX:::200..55726, 2024.
[22] Amirreza Payandeh, Kourosh T. Baghaei, Pooya Payyazsanavi, Somayeh Bakhtiari Ramezani, Zhiqian Chen, and Shahram Rahimi. Deep representation learning: Funda- mentals, technologies, applications, and open challenges. IEEE Access, 11:137621-137652 2023.
[53] Patrik Reizinger, Randall Balestriero, David Klindt, and Wieland Brendel. An empir- ically grounded identifiability theory will accelerate self-supervised learning research. Xrviv preprint wrXic:2504.13100 2025.
[54] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pages 31852 31876. PMLR, 2023.
[55] Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, Malte Ebner, and et al. Lightly.
[56] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827 6839, 2020.
[57] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 11268-102777 PMLR, 2021.
[58] Hugues Van Assel, Nicolas Courty, Rémi Flamary, Aurélien Garivier, Mathurin Massias, Titouan Vayer, and Cédric Vincent-Cuazz TorchDR: Pytorch Dimensionality Reduction.
(59] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Ex- tracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096- 1103, 2008.
[60] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierro-Antoine Man- zagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, (12),), 2010.
[61] Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, and John Chuang, Panopticon: Advancing any-sensor foundation models for earth observation. arXiv preprinl ar Xix:22503.166 2025.
662] Chenxi Wang, Tianle Gu, Zhongyu Wei, Lang Gao, Zirui Song, and Xiuying Chen. Word form matters: Llms' semantic reconstruction under typoglycemia. Xiv preprial arXiv:203. 201100011,,, 01714,
2025.
14

---

[63] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2504, 2021.
[64] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages
PMLR, 2020.
[65] Zhou Wang, Alan C Bovik, Hamid R. Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, :3(:))600 612, 2004.
[66] Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive self-supervised learning. Advances in Newral Information Processing Systems, 35:24794- 24809, 2022.
[67] Johann Wenekstern, Eeshaan Jain, Kiril Vasilev, Matteo Pariset, Andreas Wicki, Gabriele Gut, and Charlotte Bunne. Ai-powered virtual tissues from spatial proteomics for clinical diagnostics and biomedical discovery. arXiv preprint (xXi::250100603 2025.
[68] Sang Michael Xie, Tengyu Ma, and Percy Liang. Composed funinnnnng: Freezing pre- trained denoising autoencoders for improved generalization. In International Conference on Machine Learning, pages 11424-11434 PMLR, 2021.
[69] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Xiv: 170. 03888, 2017.
[70] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. ar preprint arXiv:2103.0322 2021.
[71] Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation. arXiv preprint 2tte::244.062255 2021.
[72] Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, Yonggang Wen, and Dacheng Tao. Learning from models beyond fine-tuning. Nature Machine Intelligence, pages 12, 2025.
[73] Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, et al. Virchow2: Scaling self-supervised mixed magnification models pathology. arXiv preprint orXic:2488.00730 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide a clear overview of the paper's contributions in the introduction and abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or /r introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
It is fine to include aspirational goals as motivation as long as it clear that these goals are not attained by the paper.
2. Limitations
Question: Docs the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Addressed in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No mcans that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate 'Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images system might not be used reliably to provide closed captions for ouline lectures because fails to handle technical jargon.
taken in low lighting. Or speech-to-text
The authors should discuss the computational efficiency of the proposed algo- rithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

For each theoretical result, does the paper provide the full set of assump- tions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification: Proofs are provided in the appendix Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result. reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details about experiments are provided in the appendix Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dalaset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others to replicate the model with the same dalaset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) II the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g., with an open-source datasct or instructions for how to construct the datasct).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to
17

---

the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer:
[Yes]
Justification: Provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips for more details.
While encourage the release of code and data, we understand that this might not. be possible, so "No" is an acceplable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submis- sion guidelines for more details.
The authors should provide instructions on data access and preparation, in- cluding how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymily, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/ /dctaiss
Question:
Docs the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We refer to the appendix Appendix D for details on the experimental setup.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Justification:
Error bars are provided when relevant.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/tees split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumpliions made should be given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be carcful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negalive error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: Discussed in the appendix Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relovant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclosc whether the full rescarch project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make inlo the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and our work conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
19

---

Justification:
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicions or unintended uses (e.g., disinformation, generating fake profiles, surveillance) fairness consid- erations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational rescarch and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point, out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to atlacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MT.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [N]]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards by requiring that users adhere to usage guidelines or restrictions to access the model implementing safety filters.
allow for controlled
of the model, for example
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Not applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in thcode the package should be provided. For popular datasets, paperswi paperswithcode. com/
datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Not applicable.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communccate the details of the dataset/code //cdee/ddd/mee /oodel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose assel is used.
At submissioo time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and rescarch with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental matorial is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/reve /reviwww based on the requirements your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects rescarch. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between insti- tutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
J Justification: Not applicable.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.M policy for what should or should not be described.
22

---

Outline of the appendix:
Appendix A provide the proofs for the theoretical results.
Appendix A.1 provides the proof of Theorem 3.1 on the closed-form solution for the reconstrrutioon-aased SSL problem.
Appendix A.
provides the proof of Theorem 3.2 on the closed-form solution
for the joint- SSL problem.
Appendix A.3
provides the proof of Proposition
the asymptotic behavior
of supervised carning.
Appendix A.4 of rwt.onsttuttioo-baaaaaa SSL.
provides the proof of Proposition 4.3
on the asymptotic behavior
Appendix A.5 provides the proof of Proposition on the asymptotic behavior of joint-embeddingg SSL.
4.4
Appendix joint-embedding and reconstruction based SSL.
A.6 provides the proof of Corollary 4.5 on the comparison between
Appendix provides the proof of Lemma B.1 on the equivalence between regression with augmented samples and ridge regression.
Appendix C provides details the experiments using linear models.
Appendix D provides details on the experiments using deep networks.
A Proofs
A.1 Proof of Theorem 3.1
Theorem 3.1. Let & = BYT for each [m]], and define = [x . x,
Assume that 1 X is positive definite where E is defined in Equation Cov the singular value decomposition:
Consider
1X (XX X
R
(2)
where R Rddxd and E Rdxd are orthogonal and 4 = (lla(o1........ 0a) with di 2 Da 2 0. Solutions of Equation (SL-RC) for f xx:xx + Ex and JD :x: x Dx take the form:
E TP [x
and D' R.
(3)
where T is any invertible matria in Rexk Pa and Rs are the forst k columnss of P and R respectively, and 4x = diag(d)......
Proof. Relying on the result of Lemma B.1 can rewrite the objective
Σ ENT [||x, DE
=
Σ |x,
DE + |DE
(13)
EEIm]
where E = E, ENNT [x(x))(xx) E,,T [T E B [T [xx)) Define per-sample means
x; = E, [ (x,)) and stack them into X = (xI,..., x,, T. The objective can be rewritten
as:
min EERAXA DEROXA
||X XE D D
(14)
We consider the equivalent problem on M - DE E Rdxd,
min
1| - 13 [M] s.t. rank(M) < k
(15)
MeRdx
In the above, the rank constraint captures the fact that M has the form M DE with
D E d and E E Rkxd, The objective can be developed as
1 Tr X) + Tr(MX XM Tr(MX x) + T(MMM
(16)
23

---

Keeping only the terms that depend on M, we can rewrite the problem as
min
Tr (
2 Tr(MX X)
s.t. rank(M) S k. (17)
MERox
We now consider the change of variable M' - M X (x
The above problem then
becomes
Tr(M'MM)) - 2IF M' x x +))
-2 -xxxxxx x s.t. rank(M') S k. (18)
min M'ERixa
This problem is equivalent tO
min M' M
M M 1 X X X
X
s.t. rank(M S k.
(19)
Therefore the optimal M' is the Euclidean projection of X set of matrices of rank at most k. This projection has a well-known closed-form solution
X x
onto the
23
Consider the SVD of 1 x X n
x
1 x
_ x E
R&P
(20)
where R E Rdxd is orthogonal matrix, + Rddd is diagonal matrix with singular values 61 2 ... 2 Φs 2 0 on the diagonal, and P E idxdd is an orthogonal matrix. The optimal M'
then given by:
M = R.
(21)
where R₂ E d and Pk E Rd contain the first & columns R and P, respectively, and P R is diagonal matrix with the first & largest singular values $1,..., O& the diagonal.
Then the optimal M is recovered
M = M 1 = R2 P.P. (xxxx+) +)
-2
(22)
It follows that the optimal decoder D and encoder E are given by
D' = 2
(23)
E T x
(24)
where T is any invertible matrix in k
A.2 Proof of Theorem 3.2
Theorem 3.2. Let == ETaT [xxx)T((x)"' E (xxx) ET [T(x;)]
Assume that S is positive definite. Consider the eigendecomposition:
2GS QΩQ
where Ω -iiagww,.-.. wa) with wi 2 2 Wi- Solutions of Equatiion SSL.JE JE
for linear
model fw x + Wx take the form:
w SQ
where Q+ -(qi...., qx) and U is any orthogonal matria of size k k.
24

---

Proof. First, let us consider the constraint which ensures that the learned representations have orthonormal features. Using that fw + Wx and S = E [xxx))(xx)] we obtain:
Σ
WW. E (xx)) W wsw I
(25)
VE[x
Then, the invariance term measures the consistency between positive views and is given by:
B [I (2x)) (WEWT)
(26)
2n iE[n]
where
1
Σ
BrgTonT (xx) - T2(xx) - -22x)))) [xx
(27)
2n iE[n]
= E (T( - ENTT - [
[T(:)]
(88)
i=]n
= G.
(29)
Thus we have
Tr (WEWT) Tr (WSW Tr (WGWT)
(30)
= Tr (I+) Tr (WGWT)
(31)
= k - Tr (WGWT).
(32)
Therefore, the SSL problem simplifies to:
max WeRkxd
Tr (WWWW)
(33)
s.t. WSW = I
(34)
To solve this constrained non-convex optimization problem, we rely on the KKT conditions that are necessary conditions for optimality.
First-order condition.
We introduce the Lagrangian:
L= Tr (WGWT)- Tr (A (WSWT- - -.)).
(35)
where A = diag(Ns ,......,A...
.a, A&) is the diagonal matrix of Lagrange multipliers. Taking the gradient of the Lagrangian with respect to W and setting it to zero yields:
W*G AW*S.
(36)
S is positive definite thus invertible and we can write the above the following eigenvalue problem:
W -1 = AW"
(37)
Since S is positive definite, it admits a unique positive definite square root S1/2, and we can write:
GS = ( 1/2
(38)
The matrix S S-1/2GS-12 is symmetric thus admits an eigendecompossiiion:
8-1/2 = QΩQ T
(39)
where Q is orthogonal and Ω ding(w1, ... Wa) with wi ... wa. Substituting this into
the expression for GS-1, we obtain:
GS- S /2 Tot.nnnssss
(40)
Let P S1/2Q. Then GS-l admits the docomposition:
GS P
(41)
In order maximize the objective, the optimal choice for W is to pick the cigenvectors associated with the & largest cigenvalues of GS- Therefore, the rows of W lie in the span of .p......pp). Precisely, there exists a matrix U E Rixk such that
W UP = UQ_
(42)
where P3- (P....., P&) and Q2 (q) ....... gk). .k).
25

---

Primal feasibility. Finally, solutions must satisfy the primal constraint: W"SW*T = Ig.
Using the first-order condition and the fact that Q2 Q2= Ig. we obtain:
W'SW UQ 2ss U QQ UU
(43)
Therefore, to satisfy the condition W*SW* = Ik. it follows that is an orthogonal matrix.
A.3 Proof of Proposition 4.2
Proposition 4.2. (Supervised Learning) Let V (resp. V+) be the linear model solving Equation ST. with augmenlation T(x)) for X (resp. the corrupled X). The limil:
V
(8)
0.S.
holds almost surely in either of the following regimes:
+00 (perfect augmentation-noise alignment) for anyy fixed sample size N.
+00 (infinite samples) for anyy fiaed alignment o 0.
Proof. Let Q - Q2) where Q [g contains the first k columns of and Q2 Rdx(d-k) contains the remaining d - k columns spanning the null directions of X. All columns of X lie in the column space of Qi and have no component along Q2 i.e. K; >> for i [k] and K2 0 for > Formally, column rank k and XQ2 = 0.
QI where X₁ XQ1 gank has full
Recall that noise components orthogonal to the column span of Q1 Indeed, r QT with Ar = diag(AF , X i E [ d]
such that X! = for any E [A]] Finally, X! > for all
Uncorrupted data. Lemma B.1
We consider the problem regularized by data augmentation given by
min " 1 - XV 1 + IV @+
(44)
VERixa
Differentiating the objective with respect to V and setting the gradient to zero leads to the first-order optimality condition:
1 [XV+ - x =0
(45)
Given that the matrix T X n + 2T) is invertible, the closed form solution is given by:
V y x
(46)
n
Since X = X it gives:
V" = LY x, x + Q
(47)
where Λ0e d ..... 1 is the block of Ae corresponding the subspace spanned by Q
Corrupted data. We now consider the corrupted data x = x + N where N (n....,, TT The [niheen are independent variables such that for any i [n]] r where the (z]heen are independent (000 La) vectors.
Using Lemma B.I gives the following problem:
min
1 XV
IV 1 2
(48)
VeRixa
26

---

Splitting V into two parts: V1 VQi and V. = VQ2, one has that the optimal Vi is identical to the one in the uncorrupted case i.e.
Vi= Y x, X + Ao.I)
(49)
V N₂ YY N2 (2 + a²Ar 0. We define N2 = NQ2. For V2, canceling the gradient of the objective we get:
(50)
where Ae, = diag(1 9
Ad) and Ar diag(A 1 AT)
x+1
a) Asymptotic a + to0. this regime, the above condition equivalent to:
22
(51)
Since Ar invertible, the first-order condition implics that Vi = 0.
b) Asymptotic n + too. We obtain by the strong law of large numbers,
Y and N Ar,
(52)
Hence at the limit we have,
(A + (1 Ar,2) =0 =0
(53)
The matrix Ae,2 (1 a²) Ar, is invertible for any a 0, thus it follows that Vi 0.
Conclusion. Therefore, both the large-sample limit for fixed and in the large a limit for a fixed sample size, the optimal coefficients along the Qu-subspace vanish, t.e. V2 = Since Vi coincides with Vi, we recover the same solution as in the noiseless setting in both asymptotic regimes.
A.4 Proof of Proposition 4.3
Proposition 4.3. (Reconstruction) Let E' (resp. E*) be the lincar (encoder) model solving
Equalion SSL-RCC for X (resp. the corrupled X). The limil:
E
E
(9)
hold 4 almost surely in either of the following regimes:
too (perfect ougmentation-noiis alignment) for any fixed sample size N.
+ too (infinite samples), if and only if the alignment a 2 0 satisfies:
x 1 -1
1 1
a² > aRR = max se[k-I: L:d[
n x where
n min rE[M]
(10)
V + 1
Proof. Using Theorem 3.1 the closed-form solution for the encoder of the reconstruction SSL
problem of Equation (SSL-RC) applied takes the form E TTT ( +)
where T is any invertible matrix in kg and Ps the matrix containing the k columns of P associated with the & largest singular values of the matrix x Given the construction of T(a) (Section 4.1b, if follows that Σ θ .T
Σ) 2
Let Q = (Qi where Q1 E [ddk contains the & columns of Q corresponding to the important components and Qu E (ddx(d-k) contains the remaining d columns corresponding to the noise components. We denote A0,1 diag(1 ....., X Ag diag (1 ( and Ar, =
'Up to a arbitrary invertible matrix (i.e., if E* is solution, so is TE* for any l x l invertible matrix T).
27

---

Uncorrupted data. Decomposing
X X + E and X X in (Q1 gives:
x x + Σ = Q K + Ae.1) Q + Q2 (A + a² Ar,2) Q2
(54)
1 =Q
(55)
Then using the same reasoning in the proof of Proposition 4.4 (Appendix A.5 for the
uncorrupted data case, we have that the eigenvalues of _ noise components Q2 are all mill and they are strictly positive on the important components. Therefore the largest eigenvalues are found on the important components Q1 and we obtain Py = Q1-
x
Σj on the
n in
The solution of the reconstruction SSL problem is then given by
E T K2 + Ae.1) Q
(56)
where T is any invertible matrix of size k x k.
Corrupted data. Decomposing 1 1X X + E and 1 X X in (Q1(Q2) gives:
1 X Σ = Q1 ( Ae.1) Q + Q2 N N₂ + Λe, + a Ar.2) Q
(57)
X x : + N
(58)
a) Asymptotic a +o0. In this asymptolic regime the term a² Ar,2 dominates thus
Q T x Σ)
Q2 0
(59)
on the noise components Q2 all converge to 0 almost surely. On the important components Q1 the smallest singular value is positive. Hence at the limit - +o0, we have PA Q1-
Therefore the singular values of _X X x >
b) Asymptotic a +00. Again using the strong law of large numbers we obtain
1 I N2
Ar.
(60)
We denote by
1 x
min <E[
(61)
k + x V
In this asymptotic regime, ensuring that all eigenvalues of X - components Q2 are smaller than eigenvalues on important components Q1 boils down to
noise
x
Vi 11:
(62)
x + (1+
Rearranging this inequality gives
_
a²
max ie[++1ad]
1.
(63)
X.
Thus, this condition is satisfied, we obtain P Q
Conclusion. In the large a limit for a fixed sample size, and in the largo-sample limit under the condition of Equation 10), the SSL reconstruction problem solved by
- T + Ao.1) QI = E
(64)
where T is any invertible matrix of size k x k.
28

---

A.5 Proof of Proposition 4.4
Proposition 4.4.
Let W (resp. W be the linear model solving
Equation SSL-JE for X (resp. the corrupled X) The limil:
W
W
(11)
hold almost surely in either of the following regimes:
too (perfect augmentation-nooee alignment) for any fized sample size N.
as n t00 (infinite samples), if and only if the alignment a satisfies:
1-
2 aje
max sE[k-I:d
where ô min ie[R]
(12)
6
Proof. Let Q = (Q1)Q2) where Q1 E Rddk contains the & columns of Q corresponding to the important components and Q2 E [dx((dk) contains the remaining d columns corresponding to the noise components.
Using Theorem 3.2 the closed-form solution to the joint-embedding SSL problem of Equa-
tion SST.JF applied to X takes the form W = UQ_S _ where U any orthogonal matrix of size k x h. Recall that Q2 contains the h columns of Q associated with the k
eigenvectors with largest eigenvalues of the matrix S 2 GS 2. Given the construction of T(a) (Section 4.1h, follows that S = 1 XX
X x θ - T and G X
We denote A0.1 = diag(A diag(Af (44) ....
1 Ao = . 1 and Ar. =
Uncorrupted data.
Decomposing and G in (Q1 gives:
S Q1 KH Ae,1) Q Q2 (Ae.2 a² ArAr))) Qu
(65)
G K
(66)
It holds
Q s - GS Q2 = -
(67)
Hence on the noise components Q2, the eigenvalues of S GS are all null.
On the important components Q the smallest eigenvalue of s GS satisfies
max 1 1 (1 + A 19) > 0,
(68)
[E| a
since K; > for any i [k]. Therefore in the clean data setting, the largest eigenvalues of S GS are found on the important component thus it follows that Q = Q1-
The solution of the joint-embedding SSL problem is then given by
W = U ULK + Aee1) - QI
(69)
where U any orthogonal matrix of size k x k.
Upp to arbitrary orthogonal rotation (i.e., if W is a solution, so is UW" for any l x k orthogonal matrix U).
29

---

Corrupted data. Let us denote N2 - NQ2 with N being x d matrix whose rows are n. R where each n, is drawn independently from OO.
We define
1 ENTT [T())T(*))'
S
(70)
T1
iE|m]
G = n iEIn]
E [T(x;) BTNT [T(x;)
(71)
which is the equivalent of matrix S when replacing the clean dataset X by the noisy dataset X. Decomposing S and G in (Q1(Q2) gives:
$ Q ( Ae.) Q Q2 (NN, + Ae, + a² Ar.2 Q2 G Q NoO
(72)
(73)
a) Asymptotic a +o0. In this asymptotic regime the term a² Ar, dominates thus
2GS 2Q2 0.
(74)
Therefore the eigenvalues of $ GS the noise components Q2 all converge to 0 almost surely. On the important components Q1, using the same argument as in the above uncorrupted data case, the smallest eigenvalue is strictly greater than O. Therefore at the limit a + +oo, the largest cigenvalues are found on the important components and we obtain Qx Q1-
b) Asymptotic n + to0. By the strong law of large numbers,
1 N N₂
Ar.- a.s.
(75)
Let us denote
1 12 _ + X
= min
(76)
KE|k]
In this asymptotic regime, ensuring that all eigenvalues of S G--G on noise components
Q2 are smaller than eigenvalues on important components Q1 gives the condition:
X
Vi E [k- + 11:+1dd] 1: d],
(77)
ey + +1+
Rearranging this inequality, obtain
1- 5
max (E(A+1:d]
(78)
8
Thus, if this condition is satisfied, the eigenvalues corresponding to the noise components are strictly smaller than those on the important components. Consequently, the & largest
eigenvalues S --- come solely from the data subspace i.e. Q = =Q--
Conclusion. Both in the large a limit for a fixed sample size, and in the large-sample limit under the condition given by Equation [12 we obtain that the optimal solution of the SSL problem takes the form:
W U + Ae.1) Q W
(79)
where U is any ort hogonall matrix of size k x k. Therefore, in these two regimes the solution has the same form as in the uncorrupted data setting.
30

---

A.6 Proof of Corollary 4.5
Corollary 4.5. Let OJE, 6, arC, and n be defined as in Proposition 4.4 and Proposition 4.3
If [sxid X < d 8 (low noise), then aje arc (reconstruction is preferable).
r/² (high noise), then OJE < onc (joinl-embeldingg is preferable).
If minge]tt1d]] X
Proof. Recall the definition of OJE and aRC:
1
1 ! + X
max EE kE[K+I:d 1:d]
A
where
8 min 1
(80)
Ar
1 1
RC
2
I:d
max
172
X
where
min [E(A]
V 1 k/ + X,
(81)
A sufficient condition for aJu aRC is the following, for any i E [A + 1: d]
X
(82)
Rearranging this inequality gives:
7²
Vi E 1k:
(83)
Then, a sufficient condition for AJE < aRC the following inequality, for any E [k + 1 d]
1-8 8
(84)
17
It gives:
X > n
Vi 1k 11+1:d] d].
(85)
B Effect of Data Augmentation for Supervised Learning
We recall well-known result that establishes the equivalence between the effect of data augmentation and ridge regularization. We provide the proof for completeness.
Lemma B.1.
7 46
For any V E Rexd, it holds:
1
Σ E E [lly VT(x;)
IV +
Σ ly: tE|n]
VB (r(x:))
(86)
[E[x]
where
1 Σ E (xx)) E [r(x:)) E [r(x:)]
E
(87)
[<[T]
31

---

Proof.
E E
| [x(x))1
(88)
Σ
|1v:llē + BTNT [I) -ETTTT [r (y: VT(x:))]
(89)
11 i
Σ
1ly:l12 + Tr (VE (Txx) 2Tr (y, (E,~7(T(xx))
(90)
= Tr (VEVI) +
1y:112 - 2Tr (y. VI..T (x(x))) + VE,,T(T(x::) l1
(91)
I|VI + Σ
ly: VE [x
(92)
C Experiments with Linear Models
In this section, we experiment with linear models and synthetic noise in order to validate the theoretical results of Section g
To process the data, we apply a PCA of dimension 50. We then add 50 components of noise
to create corrupted version of the dataset.
The cigenvalues for and θ (as defined Soction 4.1 are randomly sampled from uni- form distributions. Specifically, eigenvalues for I are drawn from the range [0, Xmax] and eigenvalues for are drawn from [0, A
Across all experiments, we set a constant value for A = 10a To investigate the impact of input noise levels on model performance, we vary the value of Xmax. In the experiments presented in Figure we define a weak noise case with A = 10 and a strong noise case with X 10
Joint-embedding and reconstruction solutions are obtained using the closed forms provided in Theorems 3.1 and 3.2 For evaluation, we compute the supervised linear probing score:
min 1 Σ ly. 12, VEREX
Σ sE[n]
(93)
where z, the output of the SSL model on the i-th sample; i.e. z; W"x; for joint embedding where W* is the optimal joint-embedding model of Theorem 3.2 and z; Exx; for reconstruction where E* is the optimal encoder of the reconstruction model of Thoorem 3.1 We then compute the absolute difference between the score of the model trained on clean data (i.e. composed only of important features), and the score of the model trained on corrupted data (with the added irrelevand noisy feulures). As the model trained only the important features naturally selects these features as SSL represenlations, this absolute difference directly quantifies the model's ability to filter out the irrelevand noisy features.
Figures 5 and illustrate the experimental results, all of which support the intuitions outlined in Section 5.1 Notably, supervised models consistently discard noisy irrelevant components with increasing sample size or alignment strength, confirming the findings of Section A.3 However, SSI. models demonstrate varied success depending on the setting. The reconstruction SSL model fails retrieve important components from data with high noise magnitude, even with larger sample sizes and important alignment strength. Yet, with low noise, successfully identifies these components and remains robust. to aligument strength. In contrast, the joind-embesdding SSL model requires certain minimum alignment strength to filter out noisy irrelevant components, even low-noise settings, but it exhibits strong robustness to increasing noise magnitude (bottom figures).
32

---

Supervised Learning
Joint-Embedding
Reconstruction
Noise Magnitude: 103
Distance to Optimality
1
10
10
10
10-3
10"3
10
10a0
10
10'1
10
10"5
10.
10a8
10a
10a
10
10a
10a
10
102
10a
10*
Sample Size
Sample Size
Sample Size
10
Noise Magnitude: 10
Distance Optimalty
10"2
10"2
10
10
10-3
10
10
10"7
10
10.5
10a
10a
10
10
10a
10
102
10a
10
Sample Size
Sample Size
Sample Size
10a
Noise Magnitude: 105
10
Distance to Optimality
10
10-2
10
10-3
10.
10-4
10-7
10-2
10-
10
10a
10
10
10"
10a
10
10a
10
Sample Size
Sample Size
Sample Size
Noise Magnitude: 106
10-1
10
x
Distance to Optimalty
10
2 10
10-3
10-3
10-5
10-1
10
10-7
x 10-2
10.b
10'
10a
10
10
10"
10a
10a
10
10
Sample Size
Sample Size
Sample Size
Alignment Between Augmentation Noise
Weak (a = 0.1)
Medium (a 1)
Strong (a 10)
Figure 4: Performance of linear supervised and SSL models (Sections 3. and 3.2 and The- orems 3.1 and 3.2) with synthetic noise (Section 4.1 and Appendix on NIST. Each subplot's y-axis the absolute difference of supervised linear probing loss (on clean vs. corrupted data) and its x-axis is the sample size n.
33

---

Supervised Learning
Joint-Embedding
Reconstruction
Noise Magnitude: 103
10
Distance to Optimality
1
10
10-3
10
10.
10"
10'
10"
5
10.s
10a
10a
10
10a
10a
104
10a
10a
10*
Sample Size
Sample Size
Sample Size
10
Noise Magnitude: 10
10a2
Distance to Optimalty
10:2
10
10"2
10-a
10
10
10
10-5
10-3
10a
10a
10
10a
10a
10
102
10a
10
Sample Size
Sample Size
Sample Size
Noise Magnitude: 105
10-
10-1
Distance to Optimality
10-2
10
6 10
10-3
4 1
10-4
10.6
x
xx
10a5
10
10a
10
10
10a
104
10
10a
10
Sample Size
Sample Size
Sample Size
Noise Magnitude: 106
10
Distance to Optimalty
2 10
10-2
10
10:3
10
10
10
10a6
10-5
10
10'
10a
10
10
10"
10a
102
10
10
Sample Size
Sample Size
Sample Size
Alignment Between Augmentation Noise
Weak (a = 0.1)
Medium (a 1)
Strong (a 10)
Figure 5: Performance of linear supervised and SSL models (Sections 3.1 and 3.2 and Theo- rems 3.1 and 3.2) with synthetic noise (Section 4.1 and Appendix C on Rashion-MNE Each subplot's y-axis is the absolute difference of supervised lincar probing loss (on clean vs. corrupted data) and its x-axis is the sample size n.
34

---

Supervised Learning
Joint-Embedding
Reconstruction
Noise Magnitude: 103
10"
Distance to Optimality
10
10
10
10
10"3
10
10
10
10
10.5
10
10
10
10a
10a
10
10a
10a
10
10a
10a
10*
Sample
Sample Size
Sample Size
Noise Magnitude: 10
10
Distance to Optimalty
10
10
10
10-4
10.5
10
10"4
10-7
10-4
10:6
10a
10a
10
10
10a
10
102
10a
10
Sample Size
Sample Size
Sample Size
Noise Magnitude: 105
10-1
Distance to Optimality
10-2
10
10
10
10-
10-2
10-5
10
10a
10
10
10a
104
10
10a
10
Sample Size
Sample Size
Sample Size
2 2x-1
Noise Magnitude: 1
10-1
1
Distance to Optimally
10
10-3
10
10-8
10-5
10
10
6 10-2
10-5
10'
10a
10
10
10"
10
10a
10
10
Sample Size
Sample Size
Sample Size
Alignment Between Augmentation Noise
Weak (a = 0.1)
Medium (a 1)
Strong (a 10)
Figure 6: Performance of linear supervised and SSL models (Sections 3.1 and 3.2 and Theo- rems 3.1 and 3.2) with synthetic noise (Section 4.1 and Appendix C on KuzushijiMMa characters 16. Each subplot's y-axis is the absolute difference of supervised linear probing loss (on clean vs. corrupted data) and its x-axis is the sample size n.
35

---

Supervised Learning
Joint-Embedding
Reconstruction
10
Noise Magnitude: 100
Distance to Optimality
10
10
10
10
10
10
10
10
10
10-0
10-7
10
10a
10a
104
10a
10a
10a
102
10a
104
Sample
Sample
Sample
10-2
10a1
Noise Magnitude: 101
Distance to Optimalty
10"3
10-3
10"2
10-5
10
10-3
10a5
10
10"4
10a
10a
10
10
10a
10*
102
10
10
Sample Size
Sample Size
Sample Size
Noise Magnitude: 102
10-2
10a
10
Distance to Optimality
10-3
10"3
10-4
10.5
10-2
10.5
10-7
10
10a
10
10
10a
10*
10
10a
10a
Sample Size
Sample Size
Sample Size
x 10
Noise Magnitude: 103
10
10a1
Distance to Optimally
x
10-3
10-3
2 10
10-4
10
10
10
6 10-2
10
10"
10
10
10"
10"
10a
10
10a
Sample Size
Sample Size
Sample Size
Alignment Between Augmentation Noise
Weak (a = 0.1)
Medium (a 1)
Strong (a 10)
Figure 7: Performance of linear supervised and SSL models (Sections 3.1 and 3.2 and Theo- rems 3.1 and 3.2p with synthetic noise (Section 4.1 and Appendix C on single-ced cell RNA seq datasct from 49. Each subplot's y-axis is the absolute difference of supervised lincar probing loss (on clean vs. corrupted data) and its x-axis is the sample size n.
36

---

D Experiments with Deep Networks on Image Classification
D.1 Details about the Experiments
This section details the experimental setup and hyperparameters used in the experiments.
For the evaluation of SSL methods, we freeze the learned representations and then conduct linear probing, reporting the top-1 accuracy on the test set. For the data augmentation pipelines and SSL modules, we use default modules from the lightly library [5
To introduce irrelevant noisy features in the data, we use the ImageNett-C corruptions R5 This benchmark provides a comprehensive suite of corruptions designed to assess model robustness. ImageNet-C corruptions span categories such as noise, blur, weather, and digital distortions, each available at multiple levels of severity. For each experiment, we create a fixed corrupted dataset by assigning a deterministic corruption to each image, ensuring every time image is loaded it undergocs the same corruption. This effectively simulates corrupted dataset, consistent across epochs. For the CIFAR-10 dataset, these corruptions were adapted the specific image dimensions.
D.1.1 ImageNet Experiments
For the ImageNet 18 experiments, we use batchsize of 256 and train the model for 500 epochs. We use a ResNet-50 backbone 34 for BYOL 26 with a learning rate of 0.45 and a weight decay of 10 with the LARS optimizer [9]. We 1 for the student momentum parameter. For MAF and DINO, we use a ViT-B/10 backbone 21, and the AdamW optimizer 0.05.
cosine scheduler from 0.99 to
48 with a learning rate of 1.5 x 10-4 and weight decay of
For all methods, the hyperparameters, including architectural choice, were set to the values for ImageNet presented in their respective original papers. For augmentations, we use the default augmentations associated to BYOL, DINO and MAE from the lightly library 55 with default parameters.
D.1.2 CIFAR10 Experiments
In Appendices D.2 and D.3 we perform experiments on the CIFAR-10 42 dataset. For these experiments, use a ResNet-50 backbone [4 batch size of 256, and train the model for 1000 epochs. The LARS optimizer [+ is employed with a learning rate of and
weight decay of 10-6 Supervised training is conducted using the same architecture and optimization parameters. All experiments are run with 5 different random scods.
We conduct experiments using three SSL methods: SimCLR [4, BYOL 26 and VICReg [00 For SimCLR, we set the lemperature parameter to = 0.5. For VICReg, we use the following default hyperparameters:
scaling coefficient of 25 for the invariance term of the loss, 25 for the variance term, and for the covariance term For BYOL, we use a cosine scheduler from 0.99 to 1 for the student momentum parameter. For augmentations, we use the default augmentations associated SimCLR, BYOL and VICReg from the lightly library 55 adapted to the CIFAR-10 datasct as follows: the random resized crop is set to 32 and Gaussian blur removed.
D.2 Self-Supervised Learning is much more Sensitive to Corruptions than Supervised Learning
In this section, we compare the performances of SimCLR against a supervised model.
Scores for SimCLR and supervised learning with the same augmentations
corrupted datascts at various corruption strengths are shown in Figure g We observe that the performance of SimCLR tends to degrade rapidly as the level of noise in the data increases. similar trend is observed for supervised learning, but the decline siguificantly less steep. For instance, performance on corruptions such as fog and frost remains relatively stable across corruption strengths for supervised learning, whereas for SSL, performance can drop by factor of two (e.g., from approximately 0.8 to 0.4 in top-1 accuracy).
37

---

fog
frost
pixelate
zoomblur
Top1 Accuracy x 100
80
60
40
Supervised
SimCLR
2
3
1
2 3
1
5
5
Comuuption Strength
Corruption Strength
Corruption Strength
Corruption Strength
Figure 8: Top-1 accuracy for supervised learning and self-supervised learning methods on CIFAR-10 under various corruptions with severity levels ranging from to 5. SSL is performed using SimCLR with default augmentations while supervised learning uses the same augmentations as SimCLR. We observe that supervised learning exhibits greater robustness to data corruption compared to SSL. This confirms the theoretical results of Section
This phenomenon is confirmed by examining the learned representations of VICReg in Figure 2 without and with noisy corruptions, also on CIFARIO. While the supervised model maintains clear separation of clusters even under noisy conditions, the VICReg representations degrade significantly and fail to distinguish clusters effectively when data is strongly corrupted.
These observations corroborate the results proved in Section 4 Indeed, the lack of labels prevents the model from compensating for any misalignment between augmentation and noise. Consequently, SSL performance deteriorates rapidly in the presence of corruptions the data augmentations are not appropriately adapted.
D.3 Aligning Augmentation with Known Noise
In this section, we validate the findings of Propositions and 4.4 and demonstrate empirically the impact of aligning the data augmentation with the irrelevant noisy features in the data. To do so, we artificially create misalignment with usual augmentations by applying known corruption to the datasct. We then investigate whether augmenting the data with the same type of noise can help the model learn better representations.
Specifically, for each view, we append a transform of the same corruption type but not necessarily the same strength, at the end of the data augmentation pipeline. We refer the corruption applied to the dataset as the Corruption Strength and the additional corruption appended during augmentation as the Augmentation Strength. For each corruption tested, we evaluate the Corruption Strength over the set {1, 2, 3, 4, 5} and the Augmentation Strength over the set 1,2).
Experiments are conducted on CIFAR10 and results are displayed in Figure g For cach heatmap, we interested in verifying if the top rows corresponding to augmentation strength= = 1 and 2 yield better results than the boltom row without noise injection (augmen- lalion strength- 0). Looking SimCLR runs, we observe that noise injection generally boosts performance in most settings. This holds true for all 16 corruptions, except for spatter and brightness, where an augmentation strength of performs best. Some corruptions clearly demonstrate that noise injection improves performance regardless of the corruption strength. For instance, this is evident for fog, Gaussian blur, and glass blur. For other corruptions, the utility of noise injection depends on the strength of the noisc. For example, frost, saturate, and snow benefit from noise injection when the corruption in the input data is strong, whercas for impulse noise, JPEG compression, and defocus blur, noise injection in the augmentation pipeline is more effective when the corruption in the input data is not severe. In some cases, the improvements are substantial; for instance, for fog with corruption strength of 4, adding noise injection with a strength of increases the top-1 accuracy from 49.0 to 67.1. Across all configurations of noise and corruption strength, total
38

---

of 80 combinations are tested with SimCLR, comprising 16 corruption types at 5 strength levels. Noise injection leads an improvement in top-1 accuracy in 67.5% of these cases,
We observe that these trends are quite consistent across various SSL methods. For VICReg, noise injection improves top-1 accuracy in 85% of the configurations, while for BYOI, this improvement is observed in 60% of the tested configurations. For VICReg, noise injection results in significant improvements; for instance, in the case of fog with corruption strength 5, the top-1 accuracy increases dramatically from 43.5 to 70.9. This performance improvement evident in Figure 2 where the clusters corresponding to class labels are more clearly separated when noise injection is applied. For completeness, we provide in Figure 10 the top-1 linear probing accuracy scores evaluated on a clean test set (standard CIFAR- 10 test set). We observe that for SimCLR, noise injection improves top-1 accuracy in 68. 75% of the configurations, while for both BYOL and VICReg, enhances performance 85% of the configurations.
These outcomes confirm the results of Proposition 1.4 Aligning augmentations with noise directs the model's focus to important features, thereby enhancing SSL representations. Note that our theory does not cover cases where data and noise components are intertwined, as seen with e.g. spatter, where strong noise may discard important data features and render noise injection less effective. Finally, one key observation that, even under substantial data corruption, small injection of noise during augmentation can still yield benefits. Hence the augmentation noise strength does not need be precisely tuned match the severity of the corruption.
39

---

SimCLR glassblur
SimCLR frost
SimCLR saturate
SimCLR impulsenoise
Aasmenttinn Krreagh
-29.3 74.5 62.8 61.1 54.0
--55.447 79.9 73.6 68.4
58.4
80
-88.4
86.0
86.0
84.1
50.1
-87.0
85.0 $1.6 74,5 69.7
-25
1.3
80.4
74.6
62.7
59.6
53.6
25.6
77.5
70.4
65.9
58.8
-70
88.9
86.0
85.8
83.9
50.7
86.5
82.3 50.9 75.6
65.7
50
60
0.0
77.3 66,7
54.2
52.8
48.1
50
-8588
76.6
07.1
55.8
43.9
-9.11
86.1
81.5
78.2
72.6
75
90
-6.2
84.2
$1.0 80.3 76.5
1.0
2.0
3.5
4.0
5.0
1.0
2.0
3.0
4.0 Sreepth
5.0
1.0
2.0
3.6
4.0
5.0
1.0
2.0
3.0
5.0
Strongth
Streegth
Strenghh
SimCLR jpegcompression
SimCLR defocusblur
SimCLR
SimCLR gaussiannoise
2.
.8.44 75.6 74.9 68.5
59.1
80
-73.9
62.7 57.5 49.0 43.5
.8.5
77.5
76.2
69.5 58.7
.8.0
75.6
74.8 68.5 61.1
70
1.8
81.9
78.3
73.5
66.1
60.6
80.1
71.2
58.2
50.5
46.2
60
85.6
77.2
73.4
64.6
59.3
50
86.2 80.4
75.0
68.6
61.0
6.0
-79.1
74.3 73.8
62.3 63.4
65
77.3
63.9
56.0 48.3 46.5
50
.85.6
74.2
77.4
56.9
.66.5
80.4
74.9
60.9
65
100 2.0
3.0
4.0
5.0
1.0
2.0 3.0
4.0
5.0
1.0
2.0
3.6
4.0
5.0
1.0
2.0
3.0
4.0
5.0
Comuption Strangth
Comprtin Sceeph
Streegth
Corraptis Strengss
SimCLR gaussianblur
Sim spatter
SimCLR brightness
SimCLR pixelate
83.88
77.0 70.6 65.0 56.0
85.6
84.2 00.9
09.8 88.9
90
88
12 2.9
87.6
84.2 80.5 76.3
63.6
-50
86.0
82.1 76.9 69.6 61.6
1.5
.86.8
79.2 71.1
64.6
55.8
70
.88.5
65.1
63.0
89.8 88.8
86
.9.4
83.7
78.5 72.3
67.9
85.4
82.2
75.2
68.2
64.3
84
c.0
79.4 71.5 64.9
53.0
50
89.8
00.3
89.3
82
90.8
87.8
80.7
59.5
1.0
2.0
3.5
4.0
5.0
1.5
2.0
3.0
4.0
1.0
2.0
3.0
4.0
5.5
1.5
2.0
3.0
4.0
5.0
Cermutionn Mirogth
Compttion Mereagh
Corrastian Mirregt
Doraptis Strergt
SimCLR fog
SimCLR spacklenolse
SImCLR shotnoise
SImCLR zoomblur
85.9
83.4 77.8
50.6
80
Aigeense Streag
25.4
78.6
75.2
71.6
85
Auprentare smtweg
00 1. 2.0
83.5
75.4
61.7
82.5 78.2 72.0
-86.0
81.5
74.2
63.6 54.4
60
--86.4
84.0
75.0
75.8
71.4
800
-88.2
79.6
75.3
66.9
61.1
.88.4
85.1
$4.4
79.4
74.0
0.0
81.8
76.4
63.1 49.0
36.0
86.5
83.9
78.5
76.1
72.3
75
85.2
76.8
74.4
66.5
59.4
89.6
84.9
23.3
61.1
77.9
16
2.0
3.0
4.0
5.0
1.
2.0
3.0
4.0
5.0
1.0
2.0
3.0
4.0
5.0
1.0
2.0
3.0
5.0
BYOL fog
BYOL- specklenoise
BYOL shotnoise
BYOL zoomblur
Aspresttinen siroogh
86.0 84.9 80.5
74.5
63.0
82.6 80.4
76.3 71.9
67.0
80
79.7
75.8
70.8
63.5
57.4
75
85.3
83.2
83.0 02.4 79.0
1.0
-86.8
85.2
35.0
74.5 63.3
-.85.5 82.6
75.8 76.7
71.0
75
2..80
75.8
71.
63.7
59.2
70
1
.854
84.9
83.1
83
83.7
455
0.0
81.0
77.4
73.7
67.0
61.4
23.6
B1.4
75.4
72.1
68.2
70
82.2
77.1
74.8
68.2
58.5
87.2
85.5
13.0
84.1
B1.6
1.6
2.0
3.5
4.9
5.0
1.
2.0
3.0
4.0
5.0
1.0
2.0
3.0
4.0
5.0
1.0
2.0
3.0
5.3
VICReg fog
VICReg specklenoise
VICReg shotnoise
VICReg zoomblur
Aggremeng SPeegh
-89.8 89.5 36.6 80.6
56.9
50
28..5
88.8 82.0 79.3
64.5
.8.00
77.0
73.1
66.0
60.5
888.9 89.4 38.4 87.0
75.7
1.3
-90.5
89.2 86.0
81.4
70.9
70
.88.0
85.4 11.1
79.2
73.8
75
5. 1.3
12.2
77.8
72.9
65.3
59.8
-70
89.6
88.6 87.2
87.1
63.3
0.0
.8.22
83.2
86.9
71.3
43.5
.6.00
$4.5
82.9
77.3
70.4
18.22
77.9
72.6
64.7
59.5
88.2 88.1
90.1
96.5
61.9
1.0
2.0
1.3
4.0
5.0
1.0
2.0
3.0
4.0
5.0
1.0
2.0
3.6
4.5
5.0
1.0
2.0 3.0
5.5
Corrutiinn Strangth
Comprtion Strenpth
Corruption Streegth
Corragtion Strwnges
Figure 9: Top-1 linear probing accuracy of SSL methods evaluated on CIFAR10 test set with the same corruption type and severity as the training set. The experiments investigate varying levels of corruption severity in the input data (Corruption Strength) and different levels of noise injection severity in the augmentation pipeline (Augmentation Strength). The noise injection corruption type matches the data corruption type. Each reported value is an average over random seeds. The first four rows present SimCLR results across 16 distinct corruptions: glassblur, frost, saturate, impulscnoise, jpegcompression, defocusblur, snow, gaussiannoise, gaussianblur, spatter, brightness, pixelate, fog, specklenoise, shotnoise and zoomblur. These transformations are sourced from Imagenet-C [35.. The bottom two rows display results for BYOL and VTCReg on a subset of corruptions: fog, specklenoise, shotnoise and zoomblur.
40

---

glassblur
SimCLR
saturate
SimCLR Impulsenoise
2.0
46.6
38.7
45.0
52.8
55
71.0
77.4 69.6
-00
60
35.2
37.
36.6
40
47.0
55.0
2.0
1.0
4.0
3.0
4.0
5.0
Strongth
Strepth
Corruption Strength
Correption Strenpty
SimCLR jpegcompression,
"30
SimCLR defocusblur
SimCLR snow
SimCLR gaussiannoise
47.4
22.5
42.5
86
Aaterent 1.0
60
40.0
60.0 56.2
45.9
40.5
52.4
40
63.7
22.8
-00
2.0
3.0
2.0
Corrution Strength
Compation Stropth
Comuption
Corruption Strengts
SimCLR gaussianblur
SimCLR spatter
SimCLR brightness
SimCLR pixelate
2.3
Ausowenni Stwayg
Ragemsonce strngg
22
-50
80
3.C
10
-70
500
53.0
50
90.1
72.4
59.7
66.3
50
2.0
5.0
3.0
2.0
Stronghh
fog
SimCLR specklenoise
SimCLR shotnoise
SimCLR
2.0
70.3 58.9 56.7 54.5 44.5
67.2
53.4 43.8 43.1 31.9
-70
.9.7
73.2
59.0 37.9
26.5
85.2
81.8
78.0
74.6
65.5
3.C
81.6
65.0 50.6 48.0 46.4
79.5
39.2
35.4 32.5
-00
12
.8.7
563
37.2
.60
86.8
82.8 79.4
72.4
66.9
500
50
2.0
5.0
1.
4.0
500
3.0
4.0
2.0
5.0
Cermutionn Strwath
Comuation Streppth
Comuption Streegh
Comuptinn Strengts
fog
specklenoise
BYOL
zoomblur
2.3
86.0 85.3 80.4 69.6 58.2
78.8
46.3
39.3
23
75.0
45.7
34.5
27.5
86.4
79.5
80.9
79.5
69.9
-60
5.5
60
7775.8
-50
75
0.3
50
2.0
15
4.0
5.
3.0
4.5
2.0
5.0
70
Cemmul in Mtroagh
Compttin Mereogh
Correttin Mirregh
Cerpptiee Streeds
VICReg fog
VICReg specklenoise
VICReg shotnoise
VICReg - zoomblur
2.0
87.7
64.3
80
90
86.8
83.4
75.3
800
12 12
.7.11
423
700
88.6
67.4
85
1.3
70
70
400
0.0
60
50
330
2/0
1.
4.0
4.5
5.0
Cermult Strongth
Compation Stoenth
Corouatio Shreggh
Conreptive Stremat
Figure 10: Top-1 linear probing accuracy for SSL methods evaluated on the clean (uncor- rupted) CIFARIO test set. The experiments igate varying levels of corruption severity in the input data (Corruption Strength) and different levels of noise injection severity in the augmentation pipeline (Augmentation Strength). The noise injection corruption type matches the data corruption type. Each reported value is an average over 5 random seeds. The first four rows present SimCLR results across 16 distinct corruptions: glassblur, frost, saturate, impulsenoise, ipegcompression, defocusblur, snow, gaussiannoise, gaussianblur, spatter, brightness, pixelate, fog, specklenoise, shotnoise and zoomblur. These transforma- tions are sourced from Imagenet-C [5. The bottom two rows display results for BYOL and VICReg on a subset of 4 corruptions: fog, specklenoise, shotnoise and zoomblur.
41


Reviews:
[{"review_id": "CGhwmlwwWe", "reviewer": "Reviewer_CuYw", "summary": "This paper studies the trade-offs between two major self-supervised learning paradigms (reconstruction-based and joint-embedding-based methods) by analysing how these methods respond to noise and data augmentation. The authors derive closed-form solutions for linear models under both objectives, enabling a precise comparison of how augmentation alignment with irrelevant (noisy) input features affects learning. A central finding is that joint-embedding methods are more robust in high-noise scenarios, as they require weaker alignment conditions to achieve good performance. Conversely, reconstruction-based methods are shown to be preferable in low-noise settings, where they benefit from the ability to directly reconstruct relevant features without needing heavy augmentation tuning. These insights are validated across synthetic datasets, MNIST-style tasks, and ImageNet experiments. The paper provides useful theoretical grounding for an important design choice in SSL.", "questions": "1. In Appendix Figure 3, the probing error for the reconstruction-based method under high alignment ($\\alpha = 10$) is not consistently lower than that of $\\alpha = 1$ or even $\\alpha = 0.1$. This appears to deviate from Proposition 4.3, which suggests that performance should improve with stronger alignment. Could the authors elaborate on whether this is due to finite sample effects, model instability under large augmentations, or something else? A brief discussion of this discrepancy and its implications would be valuable.\n\n2. Could the authors clarify what latent dimensionality (i.e., value of $k$) was used in the synthetic experiments? While the theoretical derivations describe closed-form solutions in terms of the top‑$k$ components, I couldn't find an explicit mention of the actual value chosen. Since this dimensionality affects the probing performance and the comparison between methods, it would be helpful to know how $k$ was selected and whether it was consistent across settings.\n\n3. The experimental settings for the reconstruction-based method do not mention the use of masking, which is a common component in modern reconstruction-based SSL approaches (e.g., MAE). Could the authors clarify whether any form of input masking was used in their reconstruction experiments? If not, do they expect their conclusions about the comparative performance of reconstruction vs. joint-embedding methods to hold if masking were incorporated?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Quality:\nThe theoretical results are clearly stated, with assumptions that are reasonable in the linear setting. Proofs in the appendix are detailed and rigorous. On the experimental side, the setup is comprehensive, covering both synthetic and real-world data, and includes both linear and deep model evaluations. One area that could benefit from further discussion is the reconstruction-based method's behaviour in Figure 3. Its performance does not always improve with stronger augmentation alignment (higher $\\alpha$), which seems at odds with the theoretical predictions. This may reflect finite-sample effects or practical limitations, and would be worth addressing more directly.\n\nClarity:\nThe writing is clear, and the paper is logically organised. The theoretical development is presented in a way that is accessible to readers familiar with SSL, and the motivation for each component is well articulated. Figures are thoughtfully chosen and support the key messages. The appendix is well structured and easy to navigate.\n\nSignificance:\nThe paper offers a thoughtful theoretical perspective on when to prefer joint-embedding versus reconstruction-based SSL methods. It adds clarity to a design choice that is often made based on empirical benchmarks or intuition. The analysis of how noise and augmentation alignment influence performance is particularly insightful and contributes to a deeper understanding of SSL behaviour. That said, in real-world scenarios, it may not always be clear whether one is operating in a low- or high-noise regime, which could limit the immediate practical applicability of the guidance. Nonetheless, the theoretical framework is likely to inform future research and help shape more principled approaches to SSL model development.\n\nOriginality: A key strength of the paper is its analytical treatment of the impact of augmentations. By explicitly modelling the augmentation distribution, the authors derive closed-form solutions that make it possible to analyse how SSL objectives respond to alignment between irrelevant features and augmentation. This leads to actionable insights, such as when reconstruction-based SSL is preferable and when joint-embedding is more robust. This kind of augmentation-aware analysis is novel in the SSL literature and adds a useful new angle to our theoretical understanding of self-supervised learning.", "was_revised": true, "final_justification": "I maintain my score (5, accept). This paper provides a clear and rigorous theoretical comparison of reconstruction-based and joint-embedding SSL methods, offering valuable insights into the role of augmentation alignment and noise. My initial questions regarding the probing error behaviour in Figure 3, the choice of latent dimensionality in synthetic experiments, and the use of masking were all addressed in the rebuttal.\nOverall, the paper makes a meaningful theoretical contribution and offers insights that are likely to guide future SSL research and design.", "rebuttal": "We would like to thank the reviewer for the insightful review and very positive feedbacks.\n\n> 1. In Appendix Figure 3, the probing error for the reconstruction-based method under high alignment [...]\n\nThis is indeed a consistent and interesting point to discuss. \n\nWe believe this discrepancy stems from finite‑sample effects. By the multivariate Central Limit Theorem, the variance of the empirical noise‑covariance spectrum decays like $1/\\sqrt{n}$. In low‑sample regimes, this high variance causes the spectrum to deviate from the true noise covariance $\\mathbf{\\Gamma}$, thus worsening probing error. Because the excess variance scales with $\\alpha^2$, its impact is greatest at large $\\alpha$. As $n$ increases, these fluctuations vanish, and strong alignment consistently outperforms weaker settings, as predicted by Proposition 4.3.\n\nImportantly, we do not observe this behavior with joint embeddings: because both views undergo the same augmentation pipeline, they maintain identical variance levels for each $\\alpha$, even in small‑sample scenarios.\n\nWe will incorporate this clarification into the revised manuscript. Thank you for your feedback.\n\n\n> 2. Could the authors clarify what latent dimensionality (i.e., value of k) was used in the synthetic experiments? [...]\n\nWe apologize for the lack of clarity. In the synthetic experiments, we set the latent dimensionality to $k=50$ informative components (the top 50 principal components of the input data) plus 50 pure‐noise dimensions. Although this setup is described in Appendix C, we will state it explicitly in the main text.\n\nWe selected $k=50$ heuristically, and we observed that the qualitative patterns hold for other choices of $k$ as well. In practice, the relative magnitudes of the data components matter more than the exact number of dimensions.\n\n\n> 3. The experimental settings for the reconstruction-based method do not mention the use of masking, which is a common component [...]\n\nIn section 5.2, we indeed experimented with MAE that implements masking as data augmentation, as provided by the lightly SSL package : https://github.com/lightly-ai/lightly. We use the hyperparameters for ImageNet as provided in the original MAE paper [a]. Results are reported in Table 1 and show that MAE is much more sensible to data corruption than Joint-Embedding approaches, thus confirming the insights of Corollary 4.5.\n\nMoreover, if the reviewer is interested, we refer to our answer to reviewer ufsf for explanations on how the covariance operator of masking (as defined in [b]) can be leveraged to derive theoretical results similar to those obtained in the Gaussian case (section 4 in the paper). Exploring specific covariances related to known data augmentation functions, rather than generic ones, is a promising direction for future work.\n\n[a] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[b] Lin, Chi-Heng, et al. \"The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective.\" JMLR 25.91 (2024): 1-85.\n\nWe would be happy to address any remaining questions if needed."}, {"review_id": "7NmkM6VUV5", "reviewer": "Reviewer_FFZ4", "summary": "In this work, the authors theoretically analyze a linear implementation of two well-known self-supervised learning paradigms: reconstruction and joint-embedding based. The theoretical analysis is made under the assumption of existing data augmentations, i.e., the existence of a distribution of transformations. This idea has been previously explored in supervised learning, where the results conclude that the quadratic risk minimization problem is equivalent to a Tikhonov regularized problem. The authors derive, using similar principles, closed-form solutions for reconstruction and joint-embedding problems, where the results clearly indicate the tendency of the optimal solutions to align with the principal axes of variation in the data. Finally, under asymptotic regimes of infinite data and augmentation alignment, they derive critical criteria regarding the alignment of the models with the signal components that are important for solving the task. Experiments on synthetic and real datasets (with well-known SSL  implementations) validate the theoretical conclusions.", "questions": "Please consult the weaknesses section above", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The study presents the first attempt at characterizing the differences between joint-embedding and reconstruction/generative models in closed form under a distribution of augmentations. This is an interesting step ahead in a field that is becoming increasingly more relevant in practice.\n- The closed form results for both paradigms are derived cleanly, and they align with intuition, for example a reconstruction based method will have an optimal encoder that captures data covariance and the covariance due to augmentations, project it on the top-k directions and then freely transform it (linearly) to maintain this k-rank projection. This corresponds with the intuition practitioners have regarding generative models as \"closely fitting the details of the data distribution.\"\n- The derivation of the two regimes of $\\alpha$ and which type of model can perform better under different noise and augmentation regimes is an important result. \n- The experiments, although few, confirm the theoretical results both on ideal (synthetic) and real-world cases. This is an optimal result, especially considering that the analysis is done on linear models and linear optimization principles. \n\nWeaknesses and Questions:\n- One of the main assumptions of the theoretical analysis is that the labels are considered to be continuous in order to frame a least squares task. While this obviously allows for an in-depth theoretical exploration, how would it affect the performance of SSL models downstream on categorical matching tasks?\n- In Equation 7, where the augmentation distribution is formalized, is my understanding correct that the directions that are unimportant to the task are augmented more, i.e., there is more noise added, as alpha grows, in the directions with $\\lambda_i^\\Gamma = 0$. Is this correct? \n- As a minor element and more of a suggestion, I believe the proofs would benefit more people looking to understand theoretical research in SSL if some of the identities, e.g., regarding matrix traces and norms, were expanded more in the first few derivations. This greatly helps the readability of the more technical material.", "was_revised": true, "final_justification": "After reading the rebuttal, I am maintaining my initial score. I believe that the authors have provided clarifications to both myself and the other reviewers regarding the practical relevance of the results, especially regarding the use of a quadratic loss instead of the typical categorical cross-entropy. Finally, I would like to thank the authors for the timely and detailed rebuttal.", "rebuttal": "We would like to thank the reviewer for the constructive review and very positive feedback.\n\n> 1. One of the main assumptions of the theoretical analysis is that the labels are considered to be continuous in order to frame a least squares task. While this obviously allows for an in-depth theoretical exploration, how would it affect the performance of SSL models downstream on categorical matching tasks?\n\nIndeed, probing is often done through discrete labels hence this question is very relevant for practical scenarios. We agree that treating labels as continuous values via MSE is primarily a theoretical convenience. \n\nIn practice, MSE can be applied to classification by representing categorical labels via one-hot encoding, where each class corresponds to a unique basis vector. This approach enables the use of regression-based objectives even when the downstream task is inherently categorical.\nImportantly, recent empirical work [a] has demonstrated that training classifiers with squared loss on one-hot targets can match, and in some cases even exceed, the performance of models trained with cross-entropy loss. These findings suggest that squared loss is not only theoretically convenient but also practically viable for classification.\n\nThanks for the question. We will clarify our contributions by specifying that we focus on MSE loss in our study of supervised learning.\n\n[a] Hui, Like, and Mikhail Belkin. \"Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks.\" ICLR 2021\n\n> 2. In Equation 7, where the augmentation distribution is formalized, is my understanding correct that the directions that are unimportant to the task are augmented more, i.e., there is more noise added, as alpha grows, in the directions with $\\lambda_i^{\\mathbf{Γ}} = 0$. Is this correct?\n\nYes, your intuition is exactly right, the directions that are unimportant to the task (which we refer to as *irrelevant noise features*) are indeed augmented more as $\\alpha$ increases. Concretely, these correspond to components with $\\lambda_i^{\\mathbf{Γ}} > 0$ (rather than $\\lambda_i^{\\mathbf{Γ}} = 0$), since $\\mathbf{Γ}$ denotes the covariance of the noisy, task irrelevant signal components. By increasing $\\alpha$, our augmentation process injects proportionally more noise into these irrelevant directions, thereby encouraging the model to become invariant to them.\n\nThank you for pointing this out, we will clarify this intuition in the revised manuscript.\n\n> 3. As a minor element and more of a suggestion, I believe the proofs would benefit more people looking to understand theoretical research in SSL if some of the identities, e.g., regarding matrix traces and norms, were expanded more in the first few derivations. This greatly helps the readability of the more technical material.\n\nThank you very much for this helpful suggestion. We will expand the derivations to guide readers more smoothly through the proof. We’ll also include a comprehensive glossary of notation to further enhance clarity.\n\nWe would be happy to address any remaining questions if needed."}, {"review_id": "jkv5SVGNrx", "reviewer": "Reviewer_YR7n", "summary": "This paper investigates the differences in the learning dynamics and performance of reconstruction-based and joint embedding-based self-supervised approaches in an attempt to enable and guide practitioners to make an educated choice of one over the other.", "questions": "1) Are linear models used in section 3 a good proxy for the complex non-linear models that are used in practice? How do we ensure that the findings transfer to these more complex non-linear models?\n2) I understand how increasing alpha in Eq. 7 results in a more aligned augmentation with the noise added. However, how do the authors control for “unaligned” noise? It seems to me that the extremes of alpha either result in a dominant aligned augmentation with high alpha or just the theta term with alpha as 0. Shouldn’t we also control for the magnitude of theta?\n3) Propositions 4.2-4: How do we conclude that the limit holds (almost surely) as either alpha or n goes to positive infinity? It is unclear to me from the main text without any pointers to appendix.\n4) Figure 2, bottom right (Strong noise, reconstruction). Do the authors have insight why the model with strong alignment lags behind at a low sample size despite high alignment between the noise and the augmentation?\n5) For Table 1, I would highly suggest color-coding or adding a symbol to help readers separate joint embedding (BYOL, DINO) methods from reconstruction (MAE) methods at a glance.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n1) Well motivated and overall writing is good.\n2) A variety of experiments backup authors’ claims.\n3) Theoretical results consistent with and further explain results and patterns in previous work.\n4) The work would equip researchers and practitioners to better be able to decide on approaches\n\n**Weaknesses**:\n1) Unclear exactly what a good alignment of noise and augmentation means in practice. As far as I could tell, in real-world datasets when the high-frequency details are somewhat irrelevant, joint-embedding techniques are better. However, how do we judge this on other datasets? How can we judge if specific augmentations would be effective?\n2) Please refer to questions below.", "was_revised": true, "final_justification": "I thank the authors for addressing my points, I have no further questions. I will maintain my original score.", "rebuttal": "We would like to thank the reviewer for the constructive and insightful feedback.\n\n> 1. Are linear models used in section 3 a good proxy for the complex non-linear models that are used in practice? How do we ensure that the findings transfer to these more complex non-linear models?\n\nThanks a lot for this important question. We appreciate the opportunity to clarify why we chose a linear‑parameter framework for our analysis.\n\nModeling our system as linear in its parameters yields closed‑form solutions that can be derived and analyzed exactly. This approach is standard in theoretical studies of self‑supervised learning (SSL): for instance, prior work on the interplay of augmentations and inductive bias [a], on implicit bias in self‑distillation networks [b], and on SSL dynamics without contrastive pairs [c] all adopt linear‑parameter models to perform precise derivations.\n\nImportantly, “linear” in our work refers only to dependence on the parameters, not to the representation power of the model. We still allow arbitrary nonlinear feature maps, akin to kernel regression. Moreover, it is common to linearize around initialization via a first‑order Taylor expansion to study deep neural networks. In the infinite‑width limit, the Neural Tangent Kernel (NTK) framework [d] shows that gradient descent training behaves exactly like kernel regression with a fixed kernel, implying that the dynamics are linear in the parameters. Thus, our linear‑parameter analysis can be understood as capturing the same training regime as very wide deep networks.\n\nIn supervised learning, recent work has derived closed‑form regularization effects of data augmentation [e]. Our contribution is the analogous analysis for SSL, where augmentations play an even more central role. \n\nFinally, as emphasized in [f], many fundamental aspects of (implicit or explicit) regularization can be understood in the linear‑model formalism. We believe that the fact that we can explain the key distinction between reconstruction‑based and joint‑embedding objectives with such a simple standard formalism highlights the generality and clarity of our findings.\n\n[a] Cabannes, Vivien, et al. \"The ssl interplay: Augmentations, inductive bias, and generalization.\" ICML 2023\n\n[b] Etai Littwin, et al. \"How jepa avoids noisy features: The implicit bias of linear self distillation networks.\" NeurIPS 2024\n\n[c] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. \"Understanding self-supervised learning dynamics without contrastive pairs.\" ICML 2021\n\n[d] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural tangent kernel: Convergence and generalization in neural networks.\" NeurIPS 2018.\n\n[e] Balestriero, Randall, Ishan Misra, and Yann LeCun. \"A data-augmentation is worth a thousand samples: Analytical moments and sampling-free training.\" NeurIPS 2022.\n\n[f] Bach, Francis. Learning theory from first principles. MIT press, 2024.\n\n\n> 2. I understand how increasing alpha in Eq. 7 results in a more aligned augmentation with the noise added. However, how do the authors control for “unaligned” noise? It seems to me that the extremes of alpha either result in a dominant aligned augmentation with high alpha or just the theta term with alpha as 0. Shouldn’t we also control for the magnitude of theta?\n\nThanks for the question. The covariance $\\mathbf{\\Theta}$ can be any positive semidefinite matrix, potentially arbitrarily far from the true noise covariance $\\mathbf{\\Gamma}$. Our theoretical results hold in full generality for any choice of $\\mathbf{\\Theta}$. Concretely, if we choose $\\mathbf{\\Theta}$ to assign large eigenvalues to irrelevant noise components and small eigenvalues to informative data components, then we reproduce the case of a large $\\alpha$ where we match the data augmentation to the noise. However this is not the scenario we are interested in. This is why in the experiments we consider no correlation between the default augmentation structure ($\\mathbf{\\Theta}$) and the actual noise ($\\mathbf{\\Gamma}$).\n\nIn our simulated experiments, we independently and uniformly sample the eigenvalue spectra of both $\\mathbf{\\Theta}$ and $\\mathbf{\\Gamma}$. This ensures there is no correlation between augmentation and noise covariances. Setting $\\alpha = 0$ recovers the case where there is no correlation between actual noisy features and augmentation (because the augmentation follows the structure given by $\\mathbf{\\Theta}$), while larger $\\alpha$ interpolates towards aligning the augmentation with the noise (discarding the effect of $\\mathbf{\\Theta}$ as $\\alpha$ increases).\n\nThis protocol also mirrors our real‑data image experiments, where we inject corruption noise (from Imagenet-C) not present in the standard augmentation pipeline. Here, $\\mathbf{\\Theta}$ represents the default augmentation pipeline, unaware of the injected noise, thus creating a natural mismatch.\n\n\n> 3. Propositions 4.2-4: How do we conclude that the limit holds (almost surely) as either alpha or n goes to positive infinity? It is unclear to me from the main text without any pointers to appendix.\n\nWe apologize for omitting explicit pointers to the proofs in the main text. In the revised version, we will add direct links before Propositions 4.2–4 to guide readers to Appendix A.3–A.5.\n\nThe proofs in Appendix A.3–A.5 proceed by first invoking the closed‑form solutions from Theorems 3.1 and 3.2. We then decompose each solution into two parts: the components that cover important data features and the other components that cover the irrelevant noise features. We then analyze each part in the limits of infinite sample size ($n\\to\\infty$) and perfect alignment ($\\alpha\\to\\infty$). The infinite‑sample limit uses the law of large numbers to establish almost sure convergence of empirical covariances to their population counterparts, while the perfect‑alignment limit isolates the effect of the noise structure in the global augmentation covariance. We then compare the solutions obtained for clean and corrupted data settings.\n\nWe will also refine our notation and expand intermediate steps in the appendix to further improve readability. Please let us know if you would like any additional details.\n\n\n> 4. Figure 2, bottom right (Strong noise, reconstruction). Do the authors have insight why the model with strong alignment lags behind at a low sample size despite high alignment between the noise and the augmentation?\n\nThis is indeed a consistent observation and an interesting point also raised by reviewer CuYw. We provide the same answer below for consistency.\n\nWe believe this discrepancy stems from finite‑sample effects. By the multivariate Central Limit Theorem, the variance of the empirical noise‑covariance spectrum decays like $1/\\sqrt{n}$. In low‑sample regimes, this high variance causes the spectrum to deviate from the true noise covariance $\\mathbf{\\Gamma}$, thus worsening probing error. Because the excess variance scales with $\\alpha^2$, its impact is greatest at large $\\alpha$. As $n$ increases, these fluctuations vanish, and strong alignment consistently outperforms weaker settings, as predicted by Proposition 4.3.\nImportantly, we do not observe this behavior with joint embeddings: because both views undergo the same augmentation pipeline, they maintain identical variance levels for each $\\alpha$, even in small‑sample scenarios.\n\nWe will incorporate this discussion in the revised version, thanks a lot for pointing this out.\n\n\n> 5. For Table 1, I would highly suggest color-coding or adding a symbol to help readers separate joint embedding (BYOL, DINO) methods from reconstruction (MAE) methods at a glance.\n\nThank you for this excellent suggestion. We will do this in the revised version to make the results clearer for readers.\n\nWe’re grateful for the reviewers’ comments and are ready to provide additional clarification or expand on any point you wish."}, {"review_id": "BWbAYwLHlK", "reviewer": "Reviewer_ufsf", "summary": "The authors compare two popular self-supervised learning (SSL) families—reconstruction (e.g., MAE) and joint-embedding (e.g., BYOL, SimCLR, BarlowTwins).\nBy analysing a simplified linear setting they prove that which family works better depends on how strongly “unwanted” input variations (noise, corruptions, background) show up in the data and how well the chosen augmentations imitate those variations. They then confirm the theory with small synthetic tasks and with ImageNet-C / CIFAR-10-C experiments.", "questions": "How could one estimate alignment or noise magnitude from raw, unlabelled data in practice?\n\nCan the analysis be extended to non-Gaussian augmentations such as random crops, colour jitter or CutMix? In particular, how might you handle the patch-masking used in MAE?\n\nDo you have intuition or experiments for settings where signal and nuisance features overlap in the same subspace?", "limitations": "No.\nThe paper never talks about what it can’t do or any downsides. It should briefly add:\nSimplifying assumptions – It assumes linear models and simple Gaussian noise. Real data and augmentations are messier, so results may change.\nSmall-data limits – All proofs need huge sample sizes; no clue if things hold for typical datasets.\nHow to use it – Readers still don’t know how to measure “noise strength” or “alignment” in practice.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe research question is very practical and important: Both loss families are widely used, yet few papers ask which one to pick and when.\nThe theory is clean: A straightforward linearization yields transparent derivations and an easy-to-remember rule of thumb: if the unwanted variations are mild, reconstruction suffices; if they are strong, joint-embedding performs better.\nThe paper has decent empirical support:  Simple linear demos and modern vision models on corrupted ImageNet/CIFAR largely follow the predicted pattern.\n\nWeaknesses:\nThe promised guidance for practitioners is thin. The paper never demonstrates how to estimate augmentation–noise alignment or noise strength on real, unlabelled data.\nAll proofs rely on additive Gaussian noise and scaled augmentations. Real augmentations (crops, colour jitter, CutMix, masking) are more complex, so it is unclear how well the conclusions generalise.\nThe theory assumes that signal and noise occupy orthogonal directions, an assumption that seldom holds for natural images or text, where the two are usually entangled.", "was_revised": true, "final_justification": "The theoretical results are clearly stated, although there are some limitations. However, the authors acknowledged and addressed these limitations in the rebuttal phase.", "rebuttal": "We would like to thank the reviewer for the insightful feedback and comments.\n\n> Can the analysis be extended to non-Gaussian augmentations such as random crops, color jitter or CutMix? In particular, how might you handle the patch-masking used in MAE?\n\nWe appreciate the reviewer’s question regarding extending our Gaussian covariance-based perturbation analysis, giving us the opportunity to clarify an important point.\n\nLet us consider the first-order Taylor expansion around the mean augmentation. Defining $\\bar{\\tau}(\\mathbf{x}) := \\mathbb{E}_{\\tau \\sim \\mathcal{T}}[\\tau(\\mathbf{x})]$, for a given function $f$, we approximate:\n$$\nf(\\tau(\\mathbf{x})) \\approx f(\\bar{\\tau}(\\mathbf{x})) + Df(\\bar{\\tau}(\\mathbf{x}))[\\delta\\tau(\\mathbf{x})], \n\\quad\\text{with}\\quad\n\\delta\\tau(\\mathbf{x}) = \\tau(\\mathbf{x}) - \\bar{\\tau}(\\mathbf{x}).\n$$\nwhere $D$ is the Jacobian operator. From this linear approximation, the induced covariance of $f(\\tau(\\mathbf{x}))$ is given by:\n$$\n\\mathrm{Cov}(f(\\tau(\\mathbf{x}))) \n\\approx Df(\\bar{\\tau}(\\mathbf{x})) \\mathbf{\\Sigma} Df(\\bar{\\tau}(\\mathbf{x}))^\\top, \n\\quad\\text{where}\\quad\n\\mathbf{\\Sigma} = \\mathbb{E}[\\delta\\tau(\\mathbf{x})\\delta\\tau(\\mathbf{x})^\\top].\n$$\nas defined in equation (Cov) in the paper. When $f$ is linear, as considered in this work for analytical tractability (consistent with the SSL literature focused on tractability [a][b]), this approximation becomes exact as no higher-order terms remain.\nIn that case, we recover the same mean and covariance for $f(\\tau(\\mathbf{x}))$ as if we used the additive Gaussian noise model with covariance $\\mathbf{\\Sigma}$ to define $\\tau$.\n\nThus, modeling the augmentation distribution as Gaussian provides a natural second-order approximation, matching exactly the first two statistical moments. Notably, several practically relevant augmentations admit closed-form covariance expressions, as explicitly derived (see page 11 in [c]). These include Pepper noise injection, Random Cutout, and unbiased random masking. This shows that masking, in particular, can be captured through a tractable second-moment approximation, directly addressing the reviewer's question about its compatibility with our framework.\nConcretely, to incorporate such augmentations, one can simply substitute the corresponding closed-form covariance operator into our formalism (as Gaussian covariance), effectively mimicking the effect of the augmentation through its second-order moment.\n\nWhile finite-moment approximations of data augmentations and noise are established in supervised learning contexts [d], they remain under-explored in the SSL literature. Current analytical results in SSL [e] rarely exploit the structural properties of the data augmentation distribution explicitly, despite its central role. We believe leveraging finite-moment approximations is a promising direction for deriving meaningful theoretical insights for SSL methods.\n\nWe will insist on these aspects in the revised version to clarify our approach.\n\n[a] Cabannes, Vivien, et al. \"The ssl interplay: Augmentations, inductive bias, and generalization.\" ICML 2023\n\n[b] Etai Littwin, et al. \"How jepa avoids noisy features: The implicit bias of deep linear self distillation networks.\" NeurIPS 2024\n\n[c] Lin, Chi-Heng, et al. \"The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective.\" JMLR 25.91 (2024): 1-85.\n\n[d] Balestriero, R., I. Misra, and Y. LeCun. \"A Data-Augmentation Is Worth A Thousand Samples: Exact Quantification From Analytical Augmented Sample Moments. NeurIPS 2022\n\n[e] Balestriero, Randall, and Yann LeCun. \"Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods.\" NeurIPS 2022\n\n> How could one estimate alignment or noise magnitude from raw, unlabelled data in practice?\n\nWe thank the reviewer for this important question related to the point raised above.\n\nOur work provides precise analytical quantities in settings where second-order moment approximations hold and where noise statistics can be estimated from data (bounds of proposition 4.3 and 4.4). While the practical estimation of these quantities in real-world systems is indeed valuable, it falls beyond the scope of this study. Our focus is on establishing a controlled, theoretically grounded comparison between joint embedding and reconstruction-based SSL methods.\n\nMoreover, our experiments on real image datasets confirm the theoretical trend: reconstruction-based methods are significantly more sensitive to noise perturbations than joint embedding approaches, even beyond controlled or synthetic settings. This phenomenon has been observed in several prior studies, and our work provides a theoretical explanation for it. We believe this connection is valuable for the SSL community.\n\n> Do you have intuition or experiments for settings where signal and nuisance features overlap in the same subspace?\n\nYes, we do cover such cases in our experiments. In Section 5, we evaluate models on corrupted images from ImageNet-C, including transformations such as Pixelate, Gaussian noise, and Zoom Blur. These corruptions affect the same subspace as core signal features.\n\nDespite this overlap, we observe the same consistent trend: joint embedding methods remain more robust than reconstruction-based methods as noise severity increases. This suggests that the robustness of joint embedding extends even to more challenging settings where signal and nuisance features are entangled.\n\n> Limitations \n\nWe thank the reviewer for pointing out that the limitations of our work should be more clearly highlighted. We will address this in the revised version of the paper by more explicitly discussing the key assumptions and scope.\n\nWe appreciate the thoughtful discussion and remain at your disposal to elaborate on any part of our submission."}]

Author rebuttal:

Meta-review: All reviews of this work leaned toward acceptance, with two borderline accept recommendations [YR7n,ufsf], one accept recommendation [CuYw] and one strong accept recommendation [FFZ4].

The reviewers appreciated several aspects of the work:

+ The work was considered well motivated [YR7n] and the research question was considered practical and important [ufsf] 
+ The theoretical perspective on joint embedding versus reconstruction-based SSL was appreciated [CuYw,FFZ4]
+ The theoretical results were appreciated [YR7n], the theory was considered clean [ufsf] and the results were considered clearly stated [CuYw] and derived [FFZ4] and were considered to align with intuition [FFZ4]
+ The results on noise and augmentation regimes was considered important [FFZ4]
+ Analytical treatment of impact of augmentations was appreciated and considered novel [CuYw]
+ The assumptions were considered reasonable in the linear setting [CuYw]
+ The proofs were considered detailed and rigorous [CuYw]
+ The experiments were appreciated [YR7n,ufsf] and the experimental setup was considered comprehensive [CuYw]; another reviewer considered the experiments few but confirming the theoretical results [FFZ4]
+ The writing was considered clear [CuYw]


However, several concerns were also raised, such as concerns on practical guidance for practitioners, and reliance of the proofs on various assumptions. Authors addressed several of the concerns in the rebuttal stage. In detail: 

- More discussion of the reconstruction-based method's behaviour in Figure 3 was desired [CuYw] and similarly explanation for behaviour of the model with string alignment in Figure 2 bottom right [YR7n]; authors speculated this arises from finite sample effects.
- Practical applicability of the guidance was considered potentially limited due to uncertainty of whether one is in a low or high noise regime [CuYw]
- The guidance for practitioners was considered thin [ufsf] and lacking demonstration of how to estimate alignment or noise strength on real unlabelled data [ufsf]; authors claimed estimation is beyond the scope of the work.
- Proof reliance on additive Gaussian noise and scaled augmentations was criticised [ufsf] and generalisation to real augmentations was unclear [ufsf]; authors argued Gaussian modelling was a natural second-order approximation.
- Assumption or orthogonal directions for noise and signal was considered to rarely hold for natural images or text [ufsf] and intuition for settings where they overlap in the same subspace was desired [ufsf]; authors noted some of their experiments were in such settings.
- A question was raised about the continuous labels assumption and its impact on SSL model downstream performance on categorical matching tasks [FFZ4]; authors provided some discussion.
- A clarification on augmentation of unimportant directions was requested [FFZ4]; authors clarified it. 
- The meaning of good alignment of noise and augmentation was considered unclear [YR7n];
- Goodness of linear models as proxies for non-linear ones was questioned [YR7n]; authors argued linear refers only to dependence on parameters and not to representation power.
- Controlling for unaligned noise was questioned [YR7n]; authors provided some discussion.
- Some clarifications for proofs were requested [YR7n]; authors provided detail.
- Clarification of the latent dimensionality in the synthetic experiments was requested [CuYw]; authors clarified it.
- Clarification of masking was requested [CuYw]; authors discussed where it was used.
- Improvements on readability of some proofs and their identities was suggested [FFZ4]; authors agreed to expand the derivations.
- Lack of discussion of limitations was criticised [ufsf]; authors promised to more explicitly discuss key assumptions and scope.

After the rebuttal reviewers [CuYw,FFZ4,YR7n] had no further questions. It was unclear whether all concerns of [ufsf] were addressed but at least some seemed to be.

Overall, it seems even though some theoretical concerns mat remain, the work has clear contributions that were appreciated and can have at least theoretical impact even if not always practical applicability. Overall, it seems the contributions of the work are interesting enough that the work could be presented NeurIPS.
Decision: Accept (spotlight)