How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
Authors: Etai Littwin
Omid Saremi
Madhu Advani
Vimal Thilak
Preetum Nakkiran
Chen Huang
Joshua M. Susskind
Venue: NeurIPS 2024 poster (2024)
Keywords: SSL
JEPA
Primary area: learning_theory
TL;DR: We show that JEPA architectures are biased towards learning highly influential features

Abstract: Two competing paradigms exist for self-supervised learning of data representations. 
    Joint Embedding Predictive Architectures (JEPAs) is a class of architectures in which semantically similar inputs are encoded into representations that are predictive of each other. A recent successful approach that falls under the JEPA framework is self-distillation, where an online encoder is trained to predict the output of the target encoder, sometimes with a lightweight predictor network. This is contrasted with the Masked Auto Encoder (MAE) paradigm, where an encoder and decoder are trained to reconstruct missing parts of the input in ambient space rather than its latent representation. A common motivation for using the JEPA approach over MAE is that the JEPA objective prioritizes abstract features over fine-grained pixel information (which can be unpredictable and uninformative).
    In this work, we seek to understand the mechanism behind this empirical observation by analyzing deep linear models. We uncover a surprising mechanism: in a simplified linear setting where both approaches learn similar representations, JEPAs are biased to learn high influence features, or features characterized by having high regression coefficients. Our results point to a distinct implicit bias of predicting in latent space that may shed light on its success in practice.

How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks
Etai Littwin
Omid Saremi
Madhu Advani
Vimal Thilak Preetum Nakkiran
Chen Huang
Joshua Susskind
Apple
Abstract
Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) class of architectures in which semantically similar inputs are encoded into representations that are predictive of each other. recent successful approach that falls under the JEPA framework is self-distillation, where online encoder is trained to predict the output of the target encoder, sometimes using a lightweight predictor network. This is contrasted with the Masked AutoEncoder (MAE) paradigm, where an encoder and decoder are trained to reconstruct missing parts of the input in the data space rather, than its latent representation. A common motivation for using the JEPA approach over MAE is that the JEPA objective prioritizes abstract features over fine- ane-grained pixel information (which can be unpredictable and uninformative). In this work, seek to understand the mechanism behind this empirical observation by analyzing the training dynamics of deep linear models. We uncover surprising mechanism: in a simplified linear setting where both approaches learn similar representations, JEPAs biased to learn high-influence features, i.e., features characterized by having high regression coefficients. Our results point to a distinct implicit bias predicting in latent space that may shed light on its success in practice.
1 Introduction
Representation learning has arguably been one of the most promising prospects and longest-standing goals of deep learning. In simple terms, representation learning refers to the process of automatically discovering and extracting meaningful features or representations from raw data. The learned representations can then be used solve various downstream tasks, such as classification, and representations in robotics and embodied agents. In recent years,
regression, or as general-purport the practice of representation learning has witnessed remarkable strides, particularly in the domain of Self Supervised Learning (SSL) 1 3 5 7 g 9 [1] [2]. The SSL paradigm has garnered significant attention due its ability exploit vast amounts of unlabeled data, freeing practitioners from the burden of expensive annotation efforts. While many SSL approaches have been proposed in the literature, two paradigms have emerged particularly successful, driving the bulk of the recent breakthroughs in the field:
Masked AutoEncoders (MAE) The original MAE 8 and its successive variants (e.g. 13 14]
introduced a training objective that seeks to reconstruct missing pieces of data from its partially masked input. This approach, while simple, has proven efficient and scalable [15]. Formally, the MAE
objective uses an encoder function (X): Rd + Rd and a decoder function gv (f) Rd Rd to predict a target y given input
LMAE
E Eawwooaa (fw(2)) yl1?
(1)
2
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

where we define joint distribution over inputs and targets p p(x,, y) with x, Rd. Here assume tied dimensions for simplicity.
In general, we can think of x and y different inputs sharing the same underlying semantic information. One common practice is to let the input z correspond to a partially masked input, and the target y corresponds to the masked-out portions of the input. Note that a critical design choice of the MAE objective is that the loss penalizes the reconstruction error directly in the target's 's input space. This implies that, as perhaps some parts of the target y cannot be predicted given a, the quality of the encoding fw crucially depends on how z, are sampled.
Joint Embedding Predictive Architectures (JEPA) The JEPA family of models does not attempt to predict y directly and instead opts to predict the latent representation of y given by some encoding function. When the encoder function for x and is shared, simple JEPA objective is given by:
CEPAA | = (f - (w)ll)
Unlike the objective in ① the objective given by 2 is susceptible to collapse: it can be minimized trivially by employing an encoder and decoder that map all inputs to the same vector. Contrastive methods [16, 6 [7] avoid this issue by including an additional loss term that pushes apart repre- sentations for negative pairs (two different samples). The drawback to this approach is that it can require comparing sample to many negative examples to work effectively. This has led to a rise in the popularity of non-contrastive SSL. With non-contrastive methods, it becomes imperative to implicitly or explicitly bias the representation mapping fw (z) to avoid its collapse to trivial solu- tion (e.g.. -. fw (æ: = O, Vx). In this paper, we leverage the widely adopted [IT] StopGrad operator, preventing the flow of gradients through the fw (v) branch. Prominent examples of this foundational architecture include B YOL 19, data2vec [10), SimSiam [II] and the more recent I-JEPA 18] and V-JEPA [19] models. Consequently, the JEPA models under consideration fall under non-contrastive self-distillation methods. We will refer to such methods as JEPA for the remainder of this paper.
While both paradigms have found practical use, characterizing the implicit bias of each remains a major research question. Recently, it was shown in 120] that in the vision domain, perceptual features mostly reside in the data subspace which explains relatively small portion of the observed variance. This suggests that reconstruction losses are sub-optimal for perception tasks since they tend to prioritize learning subspaces that explain the bulk of the input variance. On the other hand, the JEPA approach appears to be more suitable for perception tasks: empirically, JEPA often achieves better downstream classification performance with fewer optimization steps K 19].
In this work, we seek to uncover the mechanisms behind these observations by analyzing tractable deep linear neural networks. To that end, we introduce fully solvable toy model admitting complete characterization of its training dynamics for both objectives. Our analysis is inspired by previous work on the greedy learning dynamics of gradient descent, and SSL in particular [21 22 (3 However, going beyond previous work, here we present a more nuanced analysis showing how the greedy nature of learning is affected by the choice of objective, model, and the properties of the data. Our analysis reveals a distinct implicit bias of JEPAs: the propensity of the JEPA objective towards learning influential features, defined features with large regression coefficient, a property for which the MAE objective is mostly agnostic. Moreover, we show that this distinct bias takes hold only when the encoder is deep and diminishes substantially when the encoder is shallow. This allows JEPAS to steer away from the top subspace of the data which explains its observed variance, and focus on the more semantically rich subspace. Our work therefore uncovers a fundamental implicit bias, shedding light on both the advantages and drawbacks of predicting in latent space.
To summarize, our specific contributions are:
1. We analytically solve the learning dynamics of JEPA and MAE, in the tractable theoretical setting of deep linear networks.
2. In this setting, we prove a significant difference between JEPA and MAE: informally, JEPA prioritizes "influential features" " (features which are most informative in prediction), whereas MAE only prioritizes highly covarying features (even if they are noisy and thus less informative).
3. We show that our theoretical setting is rich enough to capture two more realistic generative processes, based on static and dynamic linear models.

---

Our results help clarify the precise advantages of joint-mmeedding architectures, and complement the recent results of [20] on limitations of reconstruction-bet methods.
Preliminaries
In this paper, we consider linear over-pare parameterized encoders and a shallow linear decoder. This
tractable setting allows the non-linear training dynamics [24 23 26 [2]] of the deep linear encoder network to be solved exactly under certain set of assumptions. Let {W*)am Jamm..... Rdxd and
V E Rdxd denote the set of weights for a linear encoder f(x) of depth L, and that of a linear decoder g(x) respectively, and consider the linear parameterization
W II W f(x)= = Wxx (((())==V[]) = a=1
where I== W indicates the conventional right-to-left product of matrices. As for the input data distribution, we will take I, to be samples from a centered distribution with covariances:
E = DTE, Elyy """ E[ry" >.y
For the purpose of deriving learning dynamics, we will later assume that ΣNE DaV E E Rdxd diagonal. Let (G) denote the StopGrad operator applied on o. Thus, the JEPA and MAE objectives become
Cpp
-lllz.gppzz VW Wr SG(W) Sune = -a.-
yll
(5)
2
We will denote the diagonal elements of the diagonalized covariance and correlations matrix by of Mer and X; = D respectively, and let p; A denote the regression coefficients.
of
To motivate our theoretical analysis, we next show empirically that depending on the particular values of {A,] } and { optimizing the objectives in eq. (5) via gradient descent may learn different encoders entirely.
(8)
(b)
Distribution
MAE, Distribution
JEPA, Distribution 1
Encader Magnitude [Well
Iteration
(e) MAE, Distribution
Distribution 2
JEPA, Distribution
Encoder Magnitude [
Iteration
Festure Index
Feature Index
Feature Index (i)
Figure 1: Deep linear model trained using the MAE and JEPA objectives eq. 5 Features indices (æ-axis) are organized s.t. X, is monotonically increasing and p: is (a) (c) monotonically increasing (d) (f) monotonically decreasing. (b),()): Both objectives learn features in the same order given distribution 1. In (f).()) the MAE objective maintains the same learning order as (b) on distribution 2, however, the JEPA objective reverses the learning order, due sensitivity Pi-
2 The Implicit Bias of JEPA
Before presenting the theoretical results, we describe a simple experiment using the setup in section which will flesh out surprising key difference between the JEPA and MAE objectives in eq. (5)

---

First, note that each coordinate x; of the input data represents an independent feature characterized by the covariance and regression coefficient Ai, Pi, which we refer to as feature parameters. Let e; E IR" denote the i-th standard basis vector. We say that an encoder W has Iearned the i-th feature its projection on e;, namely |Wei]] is large. Intuitively, one should expect that if pi is small, |Welll should be small post training as well. Note that this should not be overly surprising since, least in the case of the MAE objective, the global minimizer is achieved with the regression coefficients VW Σ (Σ )- In this investigation, however, we seek an understanding of the dynamics of training. by studying how the MAE and JEPA objectives prioritize different features in training, based on their feature parameters.
To this end, we track how the projections |[ eill evolve during training for each component i by training models on Gaussian data with feature parameters Xi, Pi according the following two distributions (see Figure aTa and (d) for an illustration):
1. Distribution 1: Set both {Ar), {pt) to be monotonically increasing (i.e Vesjs X Aj. Pj).
2. Distribution 2: Set (Az} to be monotonically increasing, and {pi} to be monotonically decreasing.
For each distribution, we initialize all the weights with Gaussian initialization and train the models using both the MAE and JEPA objectives using gradient descent on batches of randomly sampled data. We illustrate the dynamics of training by measuring the magnitude of the encoder components || || eill during training in Figure I On the first distribution, we observe clear greedy learning dynamics for both objectives. where components with a higher A are learned earlier on in training. This behavior in greedy settings is in line with the findings in [21) which showed the same behavior in similar settings. However, on the second distribution, we observe a distinction: while the MAE objective retains order of learning as in distribution I, the JEPA objective exhibits a complete inversion of the learning order where features with higher Pi. rather than high X; are learned first. This implies that in practical scenarios with finite training budget, JEPA and MAE objectives can potentially learn entirely different features with little to no overlap, as illustrated in Figure [I To better understand these phenomena, we turn to a dynamical analysis of a simple model that fully recovers our observations.
3 Dynamical Analysis
In the following theoretical analysis, we train the corresponding models employing objectives eq. (5) by optimizing all weights simultaneously starting at some initial weights configuration [Wa (W)))))=m1.... and (0), using the gradient flow equations
Va. W W-wL セ=
(6)
where L is either JEPA or MAE objective. Generically, the two systems of ODEs are intractable without any simplifying assumptions. special class of initializations discussed below permits analytical treatment of the problem, allowing us to derive a depth separation result indicative of differences in the inductive bias of the two objectives. In the following, we describe our set of assumptions on the encoder and decoder weights at initialization, and data distribution formally: Assumption 3.1.
1. y y are diagonal and positive definite.
2. All weight matrices W* a = L...L, (0) are initialized as W = U", V()) = ei U where (Ua) JJa U are orthogonal matrices, and 0 < 6 < 1 is an initialization scale.
3. For JEPA, we set U" to be diagonal, and for MAE we set U such that U" T", U?) is diagonal.
few notes on assumption 3.1 before stating our results. The assumption on orthogonal weights implies that the weights are "balanced" at initialization, as is typically assumed in prior works on deep linear networks 28 22. The extra technical constraint on the uniformity of the initialization eigenvalues is necessary to deal with the non-uniformity of the data eigenvalues and approximately

---

holds in typical initializations with large Gaussian matrices. Likewise, initialize V differently for JEPA and MAE to achieve alignment of eigenvectors between the model and data at initialization. Although somewhat unrealistic, our special initialization schemes will prove informative to the general case. Applying assumption 3.1 to JEPA and MAE objectives in eq. (5) respectively, and using the gradient flow equations in eq. (). one arrives the following theorem:
Theorem 3.2 (ODE Equivalence), Suppose [W=)a=...1. and V are initialized according to as- sumption 3./ Let i; = -[Wee ll. where e48 are the standard basis. Furthermore, assume the JEPA objective in eq. (5) is optimized using gradient flow according to eq. (6. Then, we have
JEPA (l) (i,())=
-.
Similarly, the MAE objective eq.
5 optimized using gradient flow according to eq.
(6) yielding:
MAE: th (t) w. (t)²tt-t X; (ttp
(8)
Remarkably, the only difference between the JEPA and MAF dynamical equations in an exponent.
For derivation of these dynamical equations appendix B. Although the assumption 3.1 enables us to characterize the learning dynamics fully, we will show via numerical simulations (see section 4 that the observations made in this paper will not change qualitatively under the more general and realistic initialization. Since ODEs for each component i decouple, for the remainder of this paper, drop the subscript and consider single ODE for tē parameterized by the encoder and data feature parameters p). A direct corollary to theorem 2 that the training dynamics of MAE p}. for L >> described in eq. (8) approach those of JEPA in eq. for L 1. Additionally, depth-dependent difference is apparent in the fixed-point solutions between the two objectives. This formalized in the following corollary:
Corollary 3.3. Let WM L). (t. L) denote the solutions to eqs.
and 8 for depth L
encoders and at time t, given initial condition (0) = E, we have
@MARloc, L) = p=tr, (JEMA(DC, =
(9)
In addition, it holds that the dynamics of I-layer JEPA model match an infinite-depth MAE lim WMAe(t, L') Wtrra(t, 1).
(10)
When L > 1, JEPA will suppress the encoding of noisier directions with a lower regression coefficient and this suppression increases with the network depth. At large depths, the encoder becomes low-rank to the point a single dominating eigenvalue corresponding to the feature with the largest regression coefficient remains. Note that corollary B.3 also indicates that the JEPA solution for L > is not reachable by the MAE objective, and vice versa. While interesting in itself, the difference between the two objectives runs deeper than their fixed-point solutions and found by analyzing the training dynamics.
In the small initialization regime, the evolution of the weights during training for both objectives exhibits incremental learning dynamics, in which the encoder learns features progressively. We define the critical time, denoted t", to be the time takes for the encoder projection tē to reach positive finite- seteebbu-clooe-tto fraction of its final fixed point value. It is quantity we wish to track since it captures an important data-dependent difference between JEPA and MAE training dynamics, concerning the order in which feature learning proceeds in the encoder:
(11) The dependence of this definition of the critical time on a choice for p introduces a degree of arbitrariness in the actual value of t However, long p is not too close to 1 or 0, the leading order in the Laurent expansion for 1" = (c, X, p) in e, as we show in section is not affected by the specific values of p. The temporal ordering of t. (A, p)s across all features, allows observe which feature takes priority in learning according to each objective.
(((') ) (oc).
3.1 Critical Time Analysis
To derive the critical time we first solve the dynamical equations in eqs. 7 and (8) in the following. The JEPA dynamics given by eq. 77 can be solved implicitly a closed form as given by theorem B.7 with full proof in the supplementary material section.
5

---

(a)
(b)
(c)
JEPA,P
JEPA
3-845
Encoder Magnitudk
2-035
(e)
MAE,p=
MAE,X
MAE
Encoder Magnitude
Iterations
Iterations
Iterations
Figure 2: Simulations of the JEPA and MAE equivalent ODEs (eqs. and Each curve represents a numerical simulation of the corresponding ODE, for different values of p. A. (a), (d) darker curves correspond to higher À and p 1. (b), (e) darker curves correspond to higher p and == 1. As can be seen, both objectives exhibit greedy learning dynamics with respect to X, however, only JEPA exhibits greedy dynamics with respect p. (c), (f) darker curves correspond higher but lower In this case, the order learning is inverted between the objectives due to the different trends in p, A.
The closed form solution implicitly describes the full trajectory of ((t) for any time From this solution, it is clear that the encoder projection in the standard basis wū starting from its initial value at f. = 0, will reach its corresponding asymptotic value at f = co. The critical time t", defined in (11) the time scale for which tū exits the dynamical region near t and reaches finite fraction of its asymptotic value. Theorem 3.4 gives the critical time for JEPA with arbitrary depth L encoders:
Theorem 3.4 (JEPA critical time). The critical time the small initialization regime wit = 0) < for JEPA is given by
21-1
L
+ +
log(e)]
(12)
Jepa
np2h
n
n=1
as long as p is not too close, as defined in eq.
81
10 zero or one.
In the small initialization regime, the JEPA solution given in theorem B.7 then describes a wise learning process of features where each feature is learned in the time scale given by eq. (12). We also derive an analogous theorem B.10 applied to the MAE objective. For technical reasons, we assume >>> the following and provide the equivalent theorems for I in Appendix B.4 From the MAE dynamics summarized in eq. (12) we arrive at the following result for MAF critical time:
Theorem 3.5 (MAE critical time). The MAE critical time 1* in the small initialization regime of vi(1 0) 1 and I > is given by
T-T + +(1).
(13)
mae
ALL -L-1ec 1)e-
as long as p is not too close, as defined in eq.
109 to zero or one.
3.2 Comparing Learning Dynamics: JEPA vs. MAE
theorem 3.4 and theorem 3.5 reveal crucial distinction between the two objectives. To understand how each objective prioritizes features, we note the functional form of the leading orders in in the critical time 1*, For MAE, we observe that the leading order term depends principally on the inverse of X. Crucially, we note that the next to leading order term (NLO) is small in comparison for any encoder depth L.. This implies step-wise learning dynamics where the step ordering is predominantly set by the feature covariance X. Conversely, features with identical À will be learned in the same

---

timescale. In the case of JEPA, observe that the leading order term is again principally function of X, however, the NLO term, which depends inversely on the regression coefficient p. is increasingly large with the encoder depth L. That is, features with identical À but with different p will be learned in different timescales, where the feature with the highest regression coefficient is prioritized. Moreover, the priority towards large p increases with depth. This implies meaningful and distinct implicit bias of the JEPA objective towards learning features with a high regression coefficient, increasingly so with depth. This summarized in the following corollary:
Corollary 3.6. Let t jepalc, (C,,,, 7, p), Emae (c, X. P) denote the critical time for the JEPA and MAE objectives given feature parameters 6, A, p. WLOG assume pl >> pr let A
7. Then, holds that
1.
Siepa(e, =1+ Jepa p')
2L Apet + O(et),
C (c, = =1+0 -( 1 (c,
'
(14)
2L 2
We note the strong dependency on depth L and 4, in the JEPA case through the term A in eq. (14), indicating greedy learning dynamics with respect to p. property absent in the case of MAE. We next turn to empirical simulations to test the validity of our theory, as well as its generalizability
in less stringent settings.
(a)
(b)
(c)
(d)
(e)
AMA
Figure 3: Temporal model 15 (a) U1 U2, (b) temporal dynamics of flickering autocorrelation 71 = 0.99 and 72 0.95, (c) Eee empirical covariance with 500k samples, (d) Eey empirical correlation, (e) predictions versus simulations parameters and p varying log1o decreases from feature to 2 whereas p increase because noise added standard deviation decreases from 1 to 0.5. Note (c) show 2 standard deviation with 10 runs.
4 Numerical Experiments
We also conduct experiments directly verifying our theoretical findings. To that end, we numerically simulate the ODEs in eqs. ( and (8) for different distributions and depths starting from small initialization. In Figure 2 we fix the encoder depth L = 5 while varying the data parameters. As observe clear greedy learning process for both objectives. However,
predicted by the theory, only the JEPA objective exhibits greedy dynamics with respect to where components with larger learned first. Moreover, we observe that the JEPA objective reverse the feature learning order relative to MAE when and À have the opposite trends, qualitatively reproducing the observation in section In Figure 4 we run further simulations by varying the depth parameter L. As predicted, we observe that deeper encoders introduce a wider temporal spread between learning features with the same but different p. in contrast to MAE which learns all features at the same time independent of the depth. Moving beyond gradient flow and the assumptions on initialization. in Figure 5 we train randomly initialized deep linear neural networks using stochastic gradient descent on randomly sampled batches of Gaussian data with varying values for p, X, using both objectives. As is evident, our results carry over to this setup as well, where greedy learning of features can seen with an objective dependent ordering according to p, X.
4.1 Linear Generative Models
We consider generative models which satisfy our diaggoollllyyddaggaiizzblle assumption (in the large sample limit). This is a setting in which our theory holds exactly and datasets with this property allow both JEPA and MAE models learn the same modes. although not necessarily the same
7

---

order. We analyze the simplest linear generative models which still lead non-trivial learning differences between JEPAs and MAFs for two settings: random masking and a temporal model. The full details of these different settings can be found in Appendix [ In the first case, our views are random masks of the data, this corresponds to static setting and we derive closed-form expressions for À and p (see C.I. For the remainder of this section, we will focus in on the temporal model due to its simplicity and intuitive interpretation.
The temporal model considers the scenario where our views 2, y correspond to our data 2 ..... 2 two consecutive times: x = z' and y = _=+1_ The goal is to predict the next frame from the previous one (this setting consistent with comparing VideoMAE and V-JEPA video models). We now define a simple linear model corresponding to combining independently time-varying images {r0) to the model simultaneously with random noise. We can write this mathematically as:
2f Σ a (t)of + t
t= I.....
(15)
Note that we are allowing the amplitude of the images to vary with time according to a scalar function u(t); make this choice study the simplest non-trivial temporal variability. The full details of the setup and derivation for simultaneous diagonalizability can be found in Appendix C.2 where we
derive the correlation coefficient for each feature:
Ja 1/1
Aa = 11 Pa
(16)
0 + ||00|1
Here on represents the noise amplitude applied to mode a. See Figure 3 for a demonstration of our theory matched to simulations and an example of higher correlation modes with lower correlation coefficient in Figure 3(). Additionally, simulation demonstrating an approach to simultancously diagonalizable empirical covariance and correlation E and SaV can be found in F Combining (16) with our theory (14) demonstrates how JEPAs can be slower to learn noisy features in a way that MAEs are not. JEPA will also learn noisy features by converging to lower amplitude weights (9). making such features potentially less likely to be used downstream from the embedding.
5 Related Work
Self-supervised learning. One successful SSL paradigm is MAE S and its variants 13 14 which learn representations via input space reconstruction. However, recent works (e.g.. (20) show that such generative paradigms often learn uninformative features for perception, since MAE-like methods tend to learn the data subspace that explains the bulk of observed variance and can include unhelpful information for perception. By contrast, the JEPA paradigm the representations of similar image views in the latent space. Such objectives have been found to prioritize semantic features over detailed pixel information, leading to superior performance for perception tasks. To prevent the feature collapse of JEPA, sop-ggadients and other architectural tricks are widely used. More recent works propose techniques like spectrum modulation 22] and weight regularization 30). Another popular collapse prevention method is based contrastive loss against negative image pairs 2
g9
10
learns to predict
11 7. In this work, we focus on MAE and JEPA methods, seeking to understand their implicit bias that leads to different training dynamics. To measure representation quality in JEPAs, [31) devised a metric that effectively counts the number of directions with large regression coefficient in latent space. Our work can be seen as further motivation for this approach.
5
6.
Theoretical analysis of SSL. There have been several recent works attempting to understand the success of contratttattiee SSL (JEPA) from various perspectives. [32] studies how the self distillllatoon in JEPA avoids representation collapse, finding the key role of eigenspace alignment between the predictor and input correlation matrix. The authors of [33] further provide a theoretical bridge between contrastive and non-contrastive objectives towards global and local spectral embedding methods respectively. It is shown that all these methods can be deployed successfully if the pairwise relations during SSL are aligned with the downstream task. While for MAE-based methods, [20] shows they tend to learn uninformative features by pixel reconstruction, unlike the JEPA objective that prioritizes semantic features. In this paper, we seek to understand the mechanism behind this empirical observation by characterizing the implicit bias of both methods using deep linear models.
8

---

Learning dynamics in linear networks. Deep linear neural networks have been used in 24 25 26 |7|4 34 28, [35 36] to study the nonlinear dynamics of learning in various settings applying different assumptions, including the saddle point behavior as well as the step-wise behavior where distinct features are learned at different time scales. This is also in line with the observed greedy learning dynamics for modern architectures like Transformers [22 23]. In the context of SSL, the authors of [2I] observed a similar step-wise nature which corresponds to the learning of eigenmodes of the contrastive kernel ΣN + Σny However, we go beyond this observation by characterizing the distinctive implicit bias of JEPA relative to MAE, i.e., they learn the same features in step-wise fashion but not necessarily in the same order. Surprisingly, this difference in behavior only apparent when the encoder network is deep. Our findings about JEPA are related to empirical observations in 337 which show JEPA-based methods focus on "slow features" that vary slowly over time. This confirmed in our studies where JEPA-based methods will prioritize "slow features" for which the marginal distribution has the smallest variance.
Limitations
Our theory has several clear limitations. The theoretical results presented in this paper are restricted to conditions in assumption 3.1, which include a deep linear MLP with some restrictions on the initialization scheme, well simultaneously diagonalizable covariances E _ee, It is worth discussing the implications of these assumptions, and the generalizability of our results to more typical settings. On the model side, empirical simulations with deep linear models initialized using the default initialization (38) indicate our results qualitatively generalize to popular initialization schemes. However, a potentially stronger assumption we make is simultaneously diagonalizability of Eve and _", which allows JEPA and MAE to learn the same features. We provide generative frameworks that adhere to this property, however we expect data distributions of interest to significantly deviate from this assumption. Moreover, the generalizability of our claims to fully practical scenarios yet remains unexplored, and precise characterization of the implicit biases pertaining to more general data distributions and architectures is a direction for future work.
7 Discussion
We have introduced toy models of two of the most popular self-supervised algorithms, namely JEPA and MAE, where can completely characterize the learning process. We have uncovered novel learning principle that separates the two objectives: while MAE focuses on highly co-varying features early in training. the JEPA objective focuses on highly influential features (indicated by high values of the regression coefficient). This implicit bias of the JEPA objective allows it to focus on features that predictive, yet contain minimal noise, measured by their variance across the training set. This new understanding of the learning dynamics of JEPA sheds light on recent empirical observations and opens the door to new avenues of research strengthening these results. Certainly, one must first. data distributions, as well as the more
generalize these results to saiimuutnnnosslyydaannnaaa imposing challenge of non-linear models. Still, less than rigorous leap to practical settings may still provide valuable insights. For example, an implicit bias towards predictive yet low-variance features may explain the tendency of JEPA to learn more semantic features, which are inherently less noisy. Conversely, it may shed light on JEPAs vulnerability to spurious or slow features. Additionally. the feature prioritization distinction between the objectives may bear on the efficiency of JEPA in learning semantic features quickly, as these features tend to reside in the low variance subspace of the data, as shown recently (0). Finally, it is also worth investigating whether the mechanism uncovered here generalizes to other joint embedding architectures not relying on self-distillation.
8 Acknowledgments
We thank Dan Busbridge, Arwen Bradley, Stefano Cosentino, and Shuangfei Zhai for stimulating discussions and useful feedback on the research and writing of this paper.

---

References
[1] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Gold- stein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. cookbook of self-supervised learning, 2023.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton, A simple frame- work for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
[3] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Supe-supe learning from images with joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-16229 2023.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv's e egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sele-supe vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-96444 2021.
[5] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariacce-ovvair regular- ization for self-superviser learning. ArXiv, abs/2105.04906 2021.
[6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7729-9738, 2020.
[7] Chen Huang, Hanlin Goh, Jiatao Gu, and Josh Susskind. MAST: Masked augmentation subspace training for generalizable self-supervised priors. In ICLR, 2023.
[8] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena [9] Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
[10] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In International Conference Machine Learning, pages 1298-1312. PMLR, 2022.
[11] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758. 2021.
[12] R. Devon Hjelm, Alex Fedorov, Samuel and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ArXiv, 0bss/1808.066777 2018.
Karan Grewal, Adam Trischler,
[13] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervieed video pre-training. Advances in neural information processing systems, 35:10078-10093, 2022.
[14] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390-234000 2023.
[15] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14549-14560, 2023.
10

---

[16] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[17] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint (XXiv:1877.0377 2018.
[18] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Seff-supervised learning from images with joint- embedding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 15619-15629, 2023.
[19] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual repre- sentations from video. arXiv preprint arXiv:2404.08477 2024.
[20] Randall Balestriero and Yann LeCun. Learning by reconstruction produces uninformative features for perception. arXiv preprint arXiv:2402.11337, 2024.
[21] James Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pages 31852-31876. PMLR, 2023.
[22] Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. ArXiv, abs/2012.09839, 2020.
[23] Enric Boix-Adserà, Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua M. Susskind. Transformers learn through gradual rank increase. ArXiv, abs/2306.07042, 2023.
[24] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 1rXiv:1312.6120, 2013.
[25] Madhu Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020.
[26] Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1) 2018.
[27] Lukas Braun, Clémentine Dominé, James Fitzgerald, and Andrew Saxe. Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35:6615 5-2111666222 2022.
[28] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Neural Information Processing Systems, 2019.
[29] Xi Weng. Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fa- had Shahbaz Khan, and Lei Huang. Modulate your spectrum in self-supervised learning. International Conference Learning Representations (ICLR), 2024.
[30] Ali Saheb Pasand, Reza Moravej, Mahdi Biparva, and Ali Ghodsi WERank: Towards rank degradation prevention for self-supervised learning using weight regularization. ArXiv, abs/2402.09588 2024.
[31] Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Josh Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssl architectures. ArXiv, abs/2312,00000, 2023.
[32] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self- self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278 PMLR, 2021.
[33] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods, arXiv preprint arXiv:2205. 115008 2022.

---

[34] Andrew M Saxe, James McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, L16(23):1153--11546 2019.
[35] Raphael Berthier. Incremental learning in diagonal linear networks. J. Mach. Learn. Res., 24:171:1-171:26, 2022.
[36] Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. ArXiv, abs/2304.004888, 2023.
[37] Vlad Sobal, V JyothirS, Siddhartha Jalagam, Nicolas Carion, Kyunghyun Cho, and Yann LeCun. Joint embedding predictive architectures focus slow features. ArXiv, abs/2211. 2022.
[38] Kaiming He, Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. 2015 IEEE International Conference Computer Vision (ICCV), pages 1026-1034, 2015.
[39] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[40] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[41] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The central claims made the paper are stated in the Abstract and the Introduction. Additionally, we enumerate our contributions in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section in the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should refleet the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

The assumptions and main theoretical results included in the main text. Detailed proofs are available in the Appendix. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Experimental setup details included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [No]
Justification: The major contributions of the paper are theoretical in nature. The experiments included in the paper are used to support and illustrate analytical results. Consequently. the models used simple linear models trained with synthetic data. The paper describes how to generate data used in the experiments as well as other setup information needed to run and reproduce experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https::/nips.... cc/
bublic/guides// (woeeeauiiiiiiooooooe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hwwpolliigggidsseeeeeeeee guides/ http://oodaaaisssinniiiia oooioll for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all details used to construct datasets used for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiments are used to verify predictions made by the analytical model developed in the paper. The theory developed in this paper suggests strong effects that we illustrate via experiments. These effects are strong and do not require making fine-grained comparisons via statistical significance testing.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This information is included with experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We confirm that the work presented in this paper is performed in manner consistent with NeurlPS Ethics Guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impact of our research in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is theoretical in nature so this question is not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We include this information in experimental details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papaasts.papersvii paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

Contents
Introduction
2
The Implicit Bias of JEPA
3
Dynamical Analysis
3.1
Critical Time Analysis
3.2
Comparing Learning Dynamies: JEPA vs. MAE
4
Numerical Experiments
Linear Generative Models
5
Related Work
6
Limitations
Discussion
8
Acknowledgments
Additional Figures
A.I Experimental Details
21
Proofs
23
I
Proofs of theorem 3.2 and corollary 3.3
23
B.2
JEPA dynamics: proofs of theorem B.7 and theorem 3.4
28
B.3
MAE dynamies: proofs of theorem B. and theorem 3.5 for L
30
4
MAE: shallow encoder
33
B.5
Proof of corollary 3.6
33
Linear Generative Models
Random masking with isotropic noise
34
C.2
Temporal model
35
Imagenet Experiments
36
Broader Impact
36
20

---

Appendix A
Additional Figures
A.1
Experimental Details
For Figure D and Figure 5 train MLPs using stochastic gradient descent random Gaussian data. We use a constant d 100 width network and fix the encoder depth to be L 5. We sample I, y from a jointly Gaussian distribution with diagonal covariances, with different values of p, A per feature, as illustrated in the first column of each figure. At each iteration, we randomly sample minibatch of 1000 samples and train for 5000 iterations.
JERA,
JEN. L-5
JEPA
JERA.
Dacad Mapstce
terelees
L-2
Canade Mapitae
Canady Magrice
Tanade Magarace
Ioowien
Figure 4: Simulations of the JEPA and MAE equivalent ODEs for L = 2,5,6, (eqs. and (8). The covariance À fixed to 1 across all curves, and darker curves correspond to higher As evident, only in the case of the JEPA objective, deeper encoders induce a more pronounced incremental learning of features with respect to p.
21

---

Distribution
JEPA. Distribution
Eacoder Mageito de [[]]]
Excoo Magnns de
Iteradion
Distribution
MAE Distribution
JEPA. Distribution
Excoder Magnits de [Wo]
Excoder Magnit. de [Pa
Itoradoon
Fratuue Indor
Inder f
Distribution 3
MAE, Distrbbttion
Encoter Magnitule [WCll
JEPA. Distribution
Inrraiion
Encoder Magninde
Fecture indox (i)
Frature Index fi)
Foature Index (i)
Distribution
JEPA Distribution
Excoder Magnitude [10]]]
Excodr Magnindd de |
Itoration
Distibution 5
MAE. Diseriuttion
JEFA Distribution
Encoder Magninule [P2cl
Encoder Magninde [10]
Ieeraiio
Ieeration
Fecture Index
Feature Index ii)
Foature Index (i
Figure Deep linear networks trained on Gaussian data. The left most column represents the values of X, p used to generate the data. All networks were initialized using standard gaussian initialization with default scale, and the encoder depth is fixed to 5. The order feature learning is dictated by for the JEPA objective, and X for the MAE objective.
22

---

Appendix B Proofs
In the following sections, we derive dynamical equations for JEPA and MAE and characterize their training dynamics. We also write down closed-form implicit solutions for their dynamics, all presented in the form of proofs for the theorems/corollariis mentioned in the main paper.
B.1 Proofs of theorem 3.2 and corollary 3.3
Theorem 3.2 (ODE Equivalence). Suppose [Wo)a=-..1. and V are initialized according to as- sumption 3./ Let W; = -[eel), where e4s are the standard basis. Furthermore, assume the JEPA
objective in eq. (5) is optimized using gradient flow according to eq.
(6). Then, we have
JEPA (t) (0:(t)²=E E X; W. xt) XiPp"
Similarly, the MAE objective eq. is optimized using gradient flow according to eq.
6 yielding:
MAE (t) (t)
A 2
(8)
Proof. Let us begin with the derivation of the JEPA dynamical equation. The gradient flow equations given by
Ya, (1) V (t) jeppa V(t) vV) Cjepa-
(17)
Evaluating the gradient using the JEPA loss in eq. one obtains
Va: (1) = E(C II (*(0) (() W(Dy) x b=0+1
II wn
].
(18)
b=1
Veen = [((()t) - W(1))) T
(19)
Substituting for the expected values using L[xx 1 = -n and E[yz E one gets
II [x) V(t) V(O) (()) b=a+1
Var Veeco
(20)
+ w*[]] W())
II b=a+1
II wo()))
Ve (V(V())(())) (()))) (t)
V(WW
WW) W(I).
(21)
With the gradient flow equations eq. 17 in mind, we get the following system of ODEs for {w", V} to solve
Va (= [ II w() V(I) T W(O o0 w W()
(22)
b=a+1
L
[ II w'(0] v(t) (()) S b=a+1
w (((t)]
V(I) W(1) V(W
(23)
Similarly, using the MAE loss given by eq. (5). the gradient flow equations for MAE lead to the following system of ODEs
L
a-1
Var (((()= =
II b=o+1
w0)0]
T v(t) 00
II
(24)
[ II b=a+1
[II
W
(
((t) = T
(25)
23

---

For ease of presentation, in this appendix will V(2) (L) and denote W(L)
With this notation, the dynamical equations for JEPA can be rewritten for W ((1) and W W (t) as follows
L+1
a-1 w*(0] W(D) [*]
Vas W = II
(26)
b=a+1
II (*)t] "(()
w (t)
b=++1
For (tt) = 1Wr.++( (t) we obtain
(L+
(27)
From the product rule for derivative, we have
L+1
((1) = E (O
(t]
(28)
a=1 b=a+1
b=1
Substituting the expression for (t) in 28 gives
L+1
L+1
L+1
w(1
Σ
II
wh
[O][ II
(D]
W(D
w (0)
(*]
(29)
4=1
b=at1
b=at1
b=1
6=1
L+1
L+1
L+1
a-1
Σ
II
wx(t)]
w()
(x)
(t).
b=a+1
b=a+1
b=1
b=1
Similarly, the dynamical equations for MAE can be re-expressed for W = W(t)
Va,
(+ II (*)
E
w (0)
(30)
b=a+1
L+1
II [²]
W(t) -F
II w')
b=0+1
For V(t)= = (t) we get
WL+1(() (t) W(1)
(31)
For MAE, plugging the expression for Wa into 28 results
L+1
L+1
L+1
a-1
w((
Σ
II
[O] II
ga
I d=1
w'()]
(32)
a=l
b=at1
b=a+1
L+1
L+1
L+1
II bma+11
w"()]l
II bma+1
w*(+)]
(*).).
In the following, we study the relevant consequences of these dynamical equations and the assump- tion 3.1 Let us begin by noting that the above assumptions on initialization imply that the weights are "balanced" at initialization, i.e., W (0) (o)++ (0) W (0)W"" (O)) for a I..... In the following lemma, we show during training, the weights will remain balanced:
Lemma B.1. Wa+1T (t) (t) (*(t) (t) for a 1..., if (t) satisfies the ODE sets
266 and
27
or 30
and
BT

---

Proof. Multiplying both sides of the MAE dynamic equation (30) from the right by W (1) for 1... L 1 gives (after absorbing WaT in the relevant products)
L+1
W"(L)W"T
II (0)
E
(33)
bma+1
L+1
[*) W(
II w
(t)
bma+11
Rewriting 30 for + 1 S L layer index and multiplying by W from the left results in
I+1
W (1)1aa+ (1)
II
II W
(34)
b=a+1
L+1
(+) W())
II w
6ma+1
6
-t) (t).
Transposing 34
one obtains
W (1)Wa+(()) ('(DWVO (1).
(35)
Using and 31.. can also be shown that
V (T))(())) ( (1), V (T)((() = (TT
(36)
Therefore, for a 1...L Wa+1T (t)i + W w (tt)attt))))))) (((()) ( which can be written
(37)
Wa+17 (oo+l((t)aat)g (t) - (* (t) = 0,
(38)
dt
at all times. Consequently, W (t)Wa+1( (t) W* (t) C where C is matrix with
constant entries. On the other hand, according to assumption 3.1, W a = 1 1... L + are scaled
orthogonal matrices at initialization, therefore C 0. This implies
W (t)) (t) W (t) (t).
(39)
at all times for a = 1..- L, which proves the claim for MAE. The proof for the JEPA case is similar. This concludes the proof of the lemma.
Using the above lemma one can write the following alternative form for 229 and 32
Lemma B.2. The dynamical equations for JEPA and MAE share the same general form follows
W- =
(())
(1 A E mape W(t) W(I) #
(40)
where Ejepa V() and Emae
Proof. Let (I) = (a(waa (1) denote its SVD decomposition, where Qo, pa are the left and right eigenbasis of the corresponding SVD decomposition. Using lemma B.I know
W (tt+TT()**+1(0) w (t) (t),
(41)
25

---

for .... L. Substituting the corresponding SVD decomposition for S implies
(0==(0)== Qa (2
(42)
which leads to
(t)) Qt),), Va=1...L,
(43)
and the conclusion that ws are equal
(a(t) w(),, V==1..+11
(44)
It is easy to see that this implies
L++
L+1
πI
(0)
II
('] = Q
(45)
6ma+1
1
b=a+1
a-1
a-1
wh (t)
(*)) = P((()w2(a-))ppT (t).
(46)
On the other hand
W(t)W (t) Q1+1 (L) (t
(47)
W (1) = (()
(48)
Using the above relations, the equations concludes the proof.
29 and 32
result in the statement of the lemma. This
Now we are in a position state and prove the following thcorem:
Theorem B.3. For JEPA, suppose w(t) = w with a diagonal w(t). For any time t, the following ansatz
((I) (T(())
(49)
(t) = U++
(50)
where U is constant orthogonal matrix, solves dynamical equations for JEPA, if (1) obeys the following differential equation
+(1) = L [[00)) - +')E
(51)
with (0) = e1 at initialization.
Proof. First note that the ansatz satisfies conditions and of the assumption 3.1
initialization.
Plugging the ansatz into (40) give
u(1 "I" 1 (0)(())())) (T. 1y o (1) 33 (1)=
(52)
which is the claimed ODE after rearrangement of the terms and noting 4 is non-singular. Also, note that the equation for (t) is satisfied. In fact, from the ansatz one obtains
V(t) w (t) = uw (t)UT.
(53)
Substituting this in (27 we get
dis t (t)
u
u (o uwz+ (t)2T.
(54)
dt
Rearranging the terms and expanding the left-hand side gives
U T.24 [w (0) 53 (1)) u
(55)
which is satisfied if the claimed ODE in the theorem is satisfied. This concludes the proof.
26

---

Theorem B.4, For MAF, suppose @(1) w-(( (1) with a diagonal w()). The following ansatz
(()) = (t),
(56)
W w()U
where U is a constant orthogonal matrix. solves the dynamical equations 440 and 31). if
(tt) = L[ 2 (t) (t)=_=H)
(57)
with (0)) = e1 at initialization.
Proof. Let us start by noticing that the ansatz is consistent with conditions 2 and 3 of the assump- tion 3.1 at initialization. From the ansatz
(t))
WL+ (t).
(58)
Plugging this expression into the equation (40)
gives
L+1
L+1 _(1++1--))-++++++20-110101-
(L (1)(1))))))()) = Σ a))
Σ
(59)
where the fact that V and ye are diagonal was used. Simplifying the above expression, one obtains
(t) = (t)2 - 21+1 (t)2
(60)
One can rewrite this ODE in terms of ت = ((). Multiply both sides of 66 Iw--1(() gives
(tt) L[a 1 (t) (t)=ar)
(61)
Now let us turn 31 Using the ansatz eq. (56). we can write (31 as
(t)) = E Wn (1) w L-1 (1)=*=
(62)
which is the same (60) given y y0E and w are diagonal. This concludes the proof.
Keeping in mind (0) is proportional to identity at initialization and Σ and M are assumed diagonal, @ (tt) remains diagonal at all times during evolution according This also means the assumption w = w(t) diagonal used during derivation of eq. [5] and eq. 557 18 a self-consistent one. Let us denote (1) = (t).). Observe that
(51) and eq. (57).
T (t) = W. (t) = [[ (t)esll-
(63)
Using the above, one arrives at the following ODE for MAE
i (t) 2 w; (E**)); ).:.
(64)
where the constant L was absorbed into the definition the gradient flow time. At this point, we can drop index to avoid clutter
(i( -
(65)
Similarly, for JEPA one arrives at
(() = w() 3 - (t))
(66)
where the definition of the regression coefficient was used. This completes the proof.
We may now proceed to prove corollary 3.3
Corollary B.5. Let WMAE(t, L), WIEPA L) denote the solutions to eqs. encoders and at time t, given initial condition w() c, we have
and 8 for depth L
@MaEloo, L)) p WIEr L)) = p4_
(9)
In addition, it holds that the dynamics of 1-layer JEPA model match infinite-eppth MAE lim WMAN (t, L') WJEra (t, 1). 100000
(10)
27

---

Proof. The fixed-point solutions for finite I. can be easily derived by setting wi 0 for both the JEPA and MAE equivalent ODEs and solving for w. To show that the limit of the MAE fixed-point at equals the JEPA fixed-point
1, we note that the right-hand side of eq. (8 is smooth function of Ç = for L > 0. Hence, we have that @MAE(t, L = oo) is the solution to
i( oo) lim (t))²-A> m(t) 11 ( p
(67)
<-0
with the initial condition (0), which is identical to the JEPA dynamics in eq.
We will now proceed discuss the results JEPA dynamics next:
B.2 JEPA dynamics: proofs of theorem B.7 and theorem 3.4
In this section, we solve the JEPA training dynamies given by the ODE (66) for general depth Let us start by stating and proving the following lemma: Lemma B.6. Define
/
db
IL(P)=
(68)
U2L 21++11
then
2L-1 I
=
nabn
+ +log(1)) log(1 - 2)) + C,
(69)
n=1
where C is constant of integration.
Proof. We use induction to prove
69 For L = 1, the integral is elementary given by
I + log (p) log(1 x/) + C,
(70)
hence, the statement of the lemma holds. Next, we show if the statement of the lemma holds for an arbitrary L + 1, it will also hold for L + 1. Note that
(71)
22++
15)
2LL+
112++1
Integrating both sides with respect to ib gives the following recursion relation
I - In (2L 1) 2 + C.
(72)
Assuming the statement the lemma holds for some L + one will have
I++1 = I1 2L
(2L + 1) + C.
(73)
2L-1
+ log(u)) -log(1 - -1) 2L
(2L + 1) + +C,
(74)
nahn
n=1
2(L+1)-1
Σ
+ log(w) - log(1 1 - v/ +
(75)
Hence, the claimed relation (69) holds for L + well. An alternative proof strategy would have been based on integrating both sides of the following identity
Σ 22-1 16
1
(76)
2L (1 +))
2
This concludes the proof.
We are now in position to state and prove the main theorem:
28

---

Theorem B.7 (JEPA dynamies). Suppose w. The JEPA dynamics given by eq. the following implicit solution in
admits
カ-2L (1)m +c] 1 -
exp
(t) = 1 exp
(77)
A 1 E x-2t (t)n 21. +c
where (t==0) e and a constant of integration.
Theorem 3.4 (JEPA critical time). The critical time t² in the small initialization regime w(t = 0)
1 JEPP JEPA
given by
21-1
L
+ θ
log(e)
(12)
Jepa
X
n22b-n-1eh
ct
as long as p is not too close, as defined eq.
81
to zero or one.
Proof. A change for variable of the form it (pb)) in the JEPA dynamical equation given by leads to
(66
du
p²p21
t t C.
(78)
2/2h
2
L
To fix C, we need impose the initial condition at Using lemma B.6 B.6 and noting 4/
1ct
p
25. Σ
p"
1 L p
+ log(c) log(p)) log(1
(79)
To compute the critical time, take x// = (1*) to be the ib value at which we measure the critical time following the definition and arguments in (II). Putting (78) and (79) together leads to
0² p2
n(v*)" + log( log(1 x8) +
Σ
p"
1 log(() I
log(p) + log(1
1 t
L
ne
P
(80)
As long as ei not too close 0 or an c-dependent way, meaning 3 O(c2) where 1. or 1 xf* θ exp((---) where 3 ابي1 the first three terms on the right-hand side in the above
equation are sub-leading compared to the rest of the terms. Given that 1²* pł, these conditions for zi* translate to constraints on the scaling of p. defined in eq. (II). That is to say
p ( with L'
1-pt.
θ exp(-e -g] with 3
2L -
(81)
L
This implies the following Laurent expansion for the critical time holds
2L-1 Lo
log(e) + O(1).
(82)
no?et
02p2L
Using p 02. the above equation can be rewritten as
A
tpppa
25.-1
Σ
L mp2l-n-le
+ + log(o].
(83)
This concludes the proof.
Embedded in the steps of the theorem's proof is an implicit closed-form solution to the JEPA
dynamical equation in (66). which for completeness state theorem below:
29

---

Theorem B.7 (JEPA dynamics). Suppose 10 the following implicit solution in
w.
The JEPA dynamics given by eq.
admits
2 2 En=1
n- 11))-22 + +c]
I
exp
(t)
(77)
E n-2 (1) + c)
+ exp
where vi(t = t and is a constant of integration.
Proof. Substituting +- = (p in 666 one obtains
dv
0²
2
(84)
L
From lemma B.6
we have
21.-1
1
p²p21
log(1 1) + log 1b +
Σ
1---1
2L
t +C,
(85)
n1
2L
L
n=1
where
1= = _ r.
(86)
P
2L-1
C = - log(1 - wh==o) + log(w)==o) +
Σ
21
(87)
n 2L
11=0
n=1
which, using the definition of the regression coefficient, can be rewritten
} 2 - n=1 n n-21 (t)) 2T. + C
2L-1
cxp
(1)
(88)
3 1 DZL-1 - (1)m- + 1 + exp
This completes the proof.
Now move studying the MAE dynamics. It turns out, it makes sense to discuss MAE dynamics for L == and L 1 cases, separately:
B.3
MAE dynamics: proofs of theorem B.10 and theorem 3.5 for L 1
Now we turn to analyzing the MAE dynamics for an arbitrary encoder depth L. Let us start with the
following lemma:
Lemma B.8. Suppose Ω [6, u). where 8 a constant and u, 8 belong to (0, 1), then the following holds
I(u)= (
122 dip
20
u
(aa-)
+C,
(89)
vim
(a - 2)u
a-2
where ar >> 2 and C is an integration constant.
Proof. The integrand admits the following Laurent series expansion convergent within the integration domain ?
1
()
(90)
2/2
therefore, the series expansion can be integrated term by term
1
00 (
I(u) =
n(aa 2) 1 Cs
(91)
==1
8
am 21-1
niaa - 2) -1 + Css
(92)
1
u m(a-)
Cs.
(93)
(a 2)u n=0
a-2
30

---

where Ca is some delta-dependent constant. In passing, we also recognize that the last expression above can be written in terms of the Lerch transcendent function +b (2z, 8, a) for the special value of
1
a - --2-
I(u)
(94)
(a 2)
where Φ (22, can be represented as series the complex plane
= ((2,,,0)
(95)
m=0 + a)s
is convergent for |z| < 1 or on the unit circle |2| = when R(s) > 1. This concludes the proof of the lemma.
Next, we proceed to state the main theorem this section:
Theorem 3.5 (MAE critical time). The MAE critical time in the small initialization regime of (i( 0) = < 1 and I. > is given by
+ ((1)
(13)
mae
XL = -
as long as p is not too close, as defined in
109 to zero or one.
Proof. Let us make the following change of the dependent variable in (64)
(ft) = p u(t)(())))))) (())
(96)
Then it can be integrated to give the following
du
X(1 X(1-L) L) I+C,
I
(97)
x2
--
L
where C is an integration constant. The integral I can be performed using lemma (B.8)
L 1-1
( 1- 1
I where is the Lerch transcendent function. An initial condition needs to be imposed to fix the
+
(98)
(L + 1)u
1+
integration constant. Recall that
U|t=0 = p
(99)
Note that for L > 1
11r=0 - 0,
(100)
under the small initialization assumption where c - 0. From 97
C = I(up=o))
(101)
Together with (100), we deduce
Leading Order, - [(ui]
+0(e"),
(102)
1-
for some exponent r. Given
1
L+1 1-1
- L
1 1+
2
(103)
1
1 Lu
n + 1+L
This allows us to write
+
1+11
+ (
(104)
u
1-
31

---

This fixes C to be
C C=
+ +(c2).
(105)
p 1-t pitle" x
To compute the critical time, we start with (97) and TOS
1-1 L+i A(1 L) L)
4-1
1 L 1 L
(106)
pi+te
(L 1)u"
To further proceed, we make the following observation, stated lemma: Lemma B.9. The second term on the right-hand side of (106) is always sub-leading compared to the first unlexs u, is chosen so that u, ) or u. exp(-e where
Proof. Singular points of I= I(u)) near u = 0 is
Or The leading term in the Taylor expansion
I(u) u
O(u
(107)
If u ( with 3 < 1-1 I, then I(u.) would be sub-leading compared to the first term on the right-hand side of (I06). In addition, I(u) contains a logarithmically singular point at u 1. Taylor-exp eepadd-e around u 1 gives
L
log(1 u) O(1)
(108)
L +
If - u. ~ 0 (exxp(-e-0) then I(un) ~ - which is sub-leading compared to the first term of the right-hand side of (106) for 3 __ This completes the proof.
Therefore, for any choice of u", unless is too close to either of the two singular points at and u = (as quantified in the above lemma), the I term in (106) is sub-leading. Given u² = _ where p the fraction in the definition of the critical time, defined in eq. (II), the conditions on become constraints on not scaling too close to zero or one with small
p O(e²)
p o
exp(-c
}. with L-1
(109)
This means in the limit 0 the leading term in the Laurent expansion in of the critical time is given by
L
Ias
(110)
A(L
The sub-leading ((1) contribution to the critical that depends on the choice of u*
Emas
(L-11) (L 1)> + (
(111)
1-L where 7 = Lpitz I(u*), which u' -dddenenen This concludes the proof.
Let us state one of the intermediate results (closed-form solution to MAE dynamics) in proving the above theorem:
Theorem B.10 (MAE dynamics). Suppose L > and let
p t Lt. The MAE dynamics
given by eq. g admits the following implicit solution in ab
1- CL
p
2 (
(112)
X(-
XLL
1))())
I+L
where ④ = 1(2, q) is the Lerch transcendent function defined by
((2,s,0) Σ
2
|2| <
(113)
(a + n)a
and C is an integration constant.
32

---

From eq. 112 extract the critical time:
Proof. To prove this theorem, note that (97 gives
1(1
--1
l+t 4
(114)
L
u(L+1)
1+L
which reduces to the statement of the theorem after rearranging terms and constants, where C is given by (105). This proves the theorem.
B.4
MAE: shallow encoder L ==1
For a single-layer encoder we have (64)
(i)(t)
σ² pt](t - (t) = t + C.
(115)
which has a closed-form solution
In[#²(1)] In (l)2 pl) = 20 pt + C.
(116)
Solving for (t) and imposing the initial condition leads to
w² (t) (p v exp 2\t + c
(117)
(t) = Voexp Al + C/2
(118)
2\t + C / 1 exp
C = log
(119)
p- 22
Qualitatively, the dynamics of @(t) for e < can be described follows:
1. (t) remains indistinguishable from zero until critical time (c) reached.
2. At (e), w() grows rapidly until it converges to its asymptotic value given by ((x))
vp.
We can derive the critical time 5 defined in (T] The solution (16 can be written
2XL + log
log(w²) log (l - m)) O(1)
(120)
The leading contribution to the critical time in the limit - is
(e, log (4)
(121)
X
In summary, for L = 1 and small initialization, learning of projections of the encoder in the standard basis occurs in step-wise manner over the time scale t* which is controlled by À only.
B.5 Proof of corollary 3.6
Corollary B.11. Let Gepplee, dr Uime (e, >o p) denote the critical time for the JEPA and MAE ob- jectives given feature parameters e, N, P. WLOG assume p' > p, let Ap 1 1. that
7. Then, holds
2L 1 (e, = Eeoo x,,',p')
2L 2L-2 A + O f (c, (,,,,'')
(e,
1110 =
(14)
Proof. The statement of the corollary is easily seen to hold one forms JEPA/MAE critical time ratios using the results in theorem 3.4 and theorem 3.5) for some fixed X but two values of the regression coefficient, denoted as / and //. Taylor-expanding the resulting expression around
up to sufficient order proves the corollary.
33

---

Appendix C Linear Generative Models
C.1 Random masking with isotropic noise
The framework we will use is to assume data sample E Rd is generated as:
Sk + T)--
(122)
where 7/2 is noise and are model coefficients for important factors (rows of B). We will also find it useful to define E [s²²] Σ
[ and will assume that the variance of these coefficients and
E
noise is isotropic: E [m]]]]
We then sample z and from masked versions of as:
with probability f with probability f
Ij z,ms yi = 23 m;) m, =
(123)
Under assumptions that B& ~ (O, 2) and in the high dimensional limit of infinite d, infinite sample size and some assumptions the distribution of model coefficients (for instance that they y-sparse with finite 7), we can show:
Theorem C.I. Fee and E simultaneously diagonalizable in the basis B and have diagonal elements:
of = E ( 11) + 1
(124)
A = f(1 (E [s]] - [s02]
(125)
From this, we can see that
A
(LE
[s E [32] + [ + f(E E
Pt
E [s
(126)
We from this that the order in which components are learned does not depend the masking ratio, but it does impact the final solution learned and the precise timing in which components are learned. This derivation eventually be learned, but the weighting will depend on the sparsity and noise level of the data and which architecture and loss function we choose.
also be applied to show that all the parameters in sparse basis will
In order to derive these results, we compute the covariances and correlation terms of this model since results in the main paper are computed in terms of these:
E [maa,]] [m.
[x:2y] ((f 1 (2) E [z,,z)]
(127)
It follows that:
E [x>" -
E [2= + ( (2) [2
(128)
Diag(o) here is operator removing all non-diagonal elements in the matrix Similarly we can derive that (mn; - m2)] = (1 [1 - (11)))))))))) and IE [(1 m1,(1 m2)] = f(1 - 1) + (1 f)²
to show that:
Tyy" T = f( f)E [22 - f
ff Diag( [2=
(129)
E [uv" = (1 f [== + + - /²) [=="])-
(130)
Note that all of these matrices have the same form: weighted sum the full data covariance and the diagonal of this covariance. It is helpful therefore to compute this covariance directly:
E
22 lee.m
BT = BT (- +o?1) B
(131)
where Σ E Rdxd a diagonal with components Σi = E [ 8jk- Unfortunately, the diagonal of generally going be exactly simultaneously diagonalizable
the data covariance matrix above with the data covariance unless it is proportional to an identity matrix. We can however show that this will occur in a certain limit. To see this consider the diagonal elements of the full data covariance:
E 2 = b.. (Σ* + +o2) b, = Σ BL + +0?)
(132)
34

---

In the limit where Σx &T becomes self-averaged quantity meaning that it does not fluctuate we draw new instances of and from the dataset, we will have that these diagonal elements are also the same for all and is therefore proportional an identity matrix.
To keep our derivation simple we will assume that the coefficients are 7-sparse, meaning [ df are drawn from a distribution P. and the rest are zero ([+] rounds to the nearest integer). It follows that:
lim 7d-
E [
E [s*] + o
(133)
Thus, we have simultaneous diagonalizability.
C.2
Temporal model
2 = Σ u (1)
(134)
This corresponds to combining a set of images {v*) but flickering them with a tire-narying intensity (ut). We will assume the following properties in the large T limit and then construct appropriate functions:
E =1, (1)+)))+++))+++)) Ja: [(n(()) E [4 = o, (t = =.
(135)
(136)
This will hold for instance if we generate
" (1) no:
m + v 1 vi ~ N0,,1)
(137)
We must ensure T is large enough to allow for good mixing of values for simulations. Note that the mixing time will depend on %a-
We will also assume that the noise is uncorrelated in time so that: EE [4t [ 0. It follows that in the large limit:
City
Σ Tauf
(138)
For the purposes of this experiment, we assume the images take up different parts of the input space that between temporal correlation and noise. In this case, all vectors v are orthogonal by definition so immediately know that the above expression for Cry is diagonal. We then assume the noise is & ~ N0,, o2 (2))) where ai) denotes the image that pixel belongs to.
easily tune the level of noise that shows up in each to explore the trade-offs
000200
0.0017
0801225
080
0.0075
080000
Agp(7)
Figure 6: Simultaneous diagonalizability demonstration. We are in the same setting Fig 3 Simultaneous diagonalizability demonstrated by measured by mean of squared error in off-diagonal elements when attempting to diagonalize Er2 using the eigenbasis of Lææ The plot show 2 standard deviation with 10 runs.
We then have that
M
CZZ
a Uj +
(139)
a
35

---

Here diy refers to the Kroneker delta function which is when o j and one otherwise. It follows that r are all rescaled eigenvectors and these matrices are simultaneously diagonalizable with the corresponding parameters:
X Ya 1| a I,.., M, Aa = 0 otherwise.
(140)
Ya 1
Pa = a - 1 11
a = 1,,,, M, Pa otherwise.
(141)
Appendix D Imagenet Experiments
Our results in the paper predict that JEPA will tend focus on the lower subspace of data variance where most of the perceptual features reside as claimed in [20]. To test this prediction in realistic setting, we use the empirical setup described [20] to study the differences in the feature learning dynamics of I-JEPA 3 and MAE [8]. We first describe the empirical setup and then make our observations.
Setup We train a vision transformer (ViT)
139 encoder using I-JEPA and MAE [ on
ImageNet- 1K [40] data resized 64 64 pixels. We use MAE-style masking to ensure parity with the masking function and copy all other hyperparameters described in I-JEPA (3] and MAE (8]. We build additional datasets by removing certain principal components of the full data subspace as
described in (20 bottom refers to the dataset that preservers bottom 25% of explained variance while top denotes the dataset with top 75% of explained variance. We use the original (full) images as well the filtered images described above and extract representations using encoders trained with I-
JEPA and MAE, We then compare the representation between full-bottom and full-top via canonical correlation analysis (CCA) 41 for several checkpoints that are gathered during training.
Observe that I-JEPA's features obtained from the full and bottom datasets show higher similarity compared to MAE throughout training while the trend is reversed for full and top images. This implies that JEPA learns features from the bottom portion of the PCA space faster than MAE. Balestrieo and Lecun (0 suggest that the bottom portion of the spectrum contain features useful for discrimination tasks which what JEPA focuses on during optimization.
full bottom CCA
full full_top
0.40
JEPA
JEPA
MAE
MAE
0.16
0.35
CCA
0.14
0.25
0.13
0.20
0.12
0.15
0.11
200
400
epachs
epochs
Figure 7: Feature similarity dynamics for I-JEPA MAE Imagenet dataset.
Appendix E Broader Impact
This work a theoretical contribution to shed light on techniques self-supervised learning to try to understand their implicit biases. The analyses reveal differences in the kinds of features that get
36

---

learned, which potentially be helpful practitioners in selecting which technique to when. These benefits may be important in deriving better models for a variety of downstream tasks that people care about such as classification, planning, and pecision-making.
37


Reviews:
[{"review_id": "Eme4uKCcFs", "reviewer": "Reviewer_Mqg6", "summary": "The authors study a characteristic of two common approaches towards visual modeling in SSL--particularly MAE and JEPA. Previous works have demonstrated or identified empirically that JEPA architecture are more prone towards lower variance features whereas MAE optimize towards higher variance features. This work focuses on a theoretical understanding of why this occurs through deep linear encoders. The authors also verify their framework empirically, with both linear and nonlinear encoders (I believe they verify with nonlinear encoders, please see questions for more details).", "questions": "* There was no mention of the moving average commonly used in JEPA architectures to update the teacher encoder (the encoder with the stop grad operator). It would be nice to know how different approaches to train/get the teacher encoder affect the training dynamics.\n* I am a bit confused by the figures involving the encoder magnitudes, particularly about the values of encoder magnitudes and how they were gotten for each feature. For the first layer, or if the encoder had a single layer, this metric makes sense. However, as linear layers are fully connected, feature i in weight layer 1 does not necessarily correspond strongly with feature i in weight layer 3. Thus, if the average magnitude across each layer for feature i is being used that seems like an invalid metric.\n* Is the MLP used in line 478 nonlinear?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The theoretical contribution is very important, as the question as to whether raw pixel or feature reconstruction works better is widely debated. Thus, the theory can be used to justify specific architectures in different circumstances.\n* The theoretical results are insightful, and the experimental results strongly support the theoretical results. \n* The authors experimented with relaxed assumptions, and had the same or similar empirical results, providing evidence that the theoretical results, while derived under unrealistic assumptions, may still hold true.", "weaknesses": "* There are very strong assumptions that will almost never be realistic\n* I am not convinced that the magnitude of encoders is a proper metric for the training dynamics. There were not many details or justification for this. Please see the questions section for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q.** “There was no mention of the moving average commonly used in JEPA...”\n \n**A.**  Indeed EMA (exponential moving average) is often used in practice to boost performance in a variety of SSL methods that employ self distillation, however we argue that the stop gradient operator is the crucial design choice preventing encoder collapse, rather than EMA, which mainly contributes to increased training stability. Indeed, these claims were argued and verified in the SimSiam [1] paper, showing that EMA can be removed altogether without sacrificing performance. Having said that, we agree EMA should be mentioned, and will add a short discussion on it in the paper. \n\n**Q.** “I am a bit confused by the figures involving the encoder magnitudes...”\n\n**A.** Let us clarify. We do not compute the average magnitude across layers. Rather, for each feature i (feature i is the i’th component in the input), we measure the magnitude of the projection of the full encoder on e_i (i’th standard basis), which simply corresponds to the norm of the i’th column of the encoder (see line 127 in the paper). Since we analyze deep linear models, the full encoder is a matrix given by the product of all the layers belonging to the encoder (see \\bar{W} in eq 3 in the paper). This makes sense as a metric since the amplitude of the projection of e_i on the encoder is the amplitude of the response to a unit input in feature direction i, which corresponds exactly to how sensitive the encoder latent (output) is to feature i. A zero projection would indicate, for example, invariance to the feature, making it a useless feature for any downstream task that uses the encoder latent.\n\n**Q.** “Is the MLP used in line 478 nonlinear?”\n\n**A.** No, it is a deep liinear MLP\n\nWe thank the reviewer for their time and effort in reviewing our paper. If we have sufficiently addressed the reviewers concerns, we kindly ask them to consider raising their score.  \n\n[1] Chen et al: “Exploring Simple Siamese Representation Learning”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vz3efurIAp", "reviewer": "Reviewer_U4uU", "summary": "This paper aims to understand the implicit bias of on two paradigms of self-supervised learning, Joint Embedding Predictive Architectures (JEPAs) and Masked Auto Encoder (MAE). The authors introduce a tractable setting of deep diagonal linear networks and charaterize the learning dynamics of two objectives on the toy problem. Through theoretical analysis, the authors show different behavoirs of two objectives: JEPA prioritizes \"influences features\" while MAE prioritizes highly-covarying features. These observations are supported numerical experiments on the toy model and the Linear Generative Models.", "questions": "Please refer to the \"Weakness\".", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written the presentation is nice. The symbols and math formulations are clear. The demonstration on Section 3 is helpful on understanding the paper. \n2.  The analyis is comprehensive, including both the learning dynamics and the critical time. \n3. The difference between the behaviors of JEPA and MAE observed in this paper is interesting.\n4. The theoretical results are supported by the numerical experiments.", "weaknesses": "Presentation on the setting (Section 2) could be improved. More explanations on $\\rho_i, \\lambda_i, \\sigma_i$ would be helpful to understand the toy setting.\n\nOn Section 3, more explanation on the settings are needed, e.g, what's the motivation of choosing these two distributions? What senarios do the distributions represent?\n\nOn Section 4, line 165, the subscript $i$ is dropped. This should be said in the beginning of the section to avoid misunderstanding.\n\nThe conclusion needs to be clearly elaborated. The authors say that JEPA prioritizes \"influential features\", whereas MAE prioritizes \"highly-covarying features\". The connection between \"influential/covarying features\" and the parameter $\\rho_i, \\lambda_i, \\sigma_i$ is not clear to me.\n\nThe study is limited to diagonal linear networks. The analyis on the toy setting provides interesting insights about the learning dynamics of JEPA and MEA. But it's not sure whether the conclusions are applicable to more complex scenarios. I believe it'd be helpful to provide some examples on non-linear cases.\n\nFigure 3 is too small and not anotated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q.** “More explanations would be helpful to understand the toy setting.”\n\n**A.** Thank you for pointing this out, we plan to add more elaboration in the rebuttal to make it more accessible.\n\n**Q.** “what's the motivation of choosing these two distributions?”\n\n**A.** The motivation for choosing these distributions is to illustrate how changing the structure of a distribution affects the learning speed of different features, as predicted by our theory. Note that, for a linear model, the only relevant aspects of the training data population statistics are its first and second moments. Hence, when it comes to characterizing data features, \\lambda_i and \\rho_i are the only relevant quantities to consider (assuming a centered distribution with independent components). In other words, the time it takes to learn the i'th feature has to depend on the feature parameters \\lambda_i,\\rho_i. The distributions in section 3 are meant to illustrate how varying these feature parameters affects the order of learning for each objective. We will add a note to help clarify this in section 2.\n\n**Q.** “On Section 4, line 165, the subscript is dropped.”\n\n**A.** Thank you for noticing this.\n\n**Q.** “The connection between \"influential/covarying features\" and the parameter is not clear to me.”\n\n**A.** By “influential features” we simply mean features with a high regression coefficient \\rho. In the linear regression literature these features are sometimes referred to as influential/predictive/significant features. By highly covarying features we simply mean features with a high covariance parameter \\lambda. Is this explanation clear to the reviewer? We will make this clearer in the intro section, and will include references to this usage in prior work.\n\n**Q.** “...I believe it'd be helpful to provide some examples on non-linear cases.”\n\n**A.** We generally agree with this statement, hence we have conducted the following experiment: one prediction that directly follows from our theory is that the JEPA objective allows the model to focus on the bottom subspace of the observed data variance where, at least for natural images, most of the perceptual features tend to reside [1] (see lines 59 - 62 and 74 - 76 in the intro of the paper). We therefore design an experiment that directly tests this hypothesis (see figures in the added pdf). Additionally, we would like to highlight the fact that beyond analytical tractability, linear models are especially appealing as a testbed in our case for the reason that both the MAE and JEPA objectives eventually learn the same features (perhaps with different amplitudes, see Corollary 4.3 in the paper) if we train the models for sufficiently long. This allows us make fair comparisons on the timescale of learning these features in each objective. \n\n**Q.** “Figure 3 is too small and not annotated”\n\n**A.** Thanks for pointing this out, we will fix it for the final version. \n\nWe thank the reviewer for their time and effort in reviewing our paper. If we have sufficiently addressed the reviewers concerns, we kindly ask them to consider raising their score.  \n\n[1]: Randall et al: “Learning By reconstruction Produces Uninformative Features for Perception“", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHxRXnAgw6", "reviewer": "Reviewer_Nsp4", "summary": "Analyze learning in two self-supervised paradigms, JEPA and MAE, through the lens of learning dynamics in deep linear networks. Report on a qualitative difference in the order in which features are learned, thus demonstrating their different implicit bias.", "questions": "1.\tCan you offer qualitative prediction from the theory to real-world systems implementing JEMA or MAE? What would be such prediction if the same deep architecture was trained on both JEMA and MAE loss, what would you expect to see differently in terms of the learned features?\n 2.\tWhat is the spectrum of self-supervised algorithms through the lens of deep linear network dynamics?\n 3.\tWhat is the result presented in Figure 3? How does it demonstrate a correspondence between theory and the experimental results? Can you offer a similar plot for the random masking task?\n 4.\tCan you demonstrate the effect of the number of layers is different under JEPA and MAE, as the theory predicts?", "limitations": "The authors adequately addressed the limitations of their approach, namely the assumptions about a common diagonalization of the covariance matrices' diagonal dynamics in the low-variance initialization. It would be great if the authors provided experimental support for the applicability of their theory to real-world systems implementing JEPA and MAE or even just provided qualitative predictions derived from the theory.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "*\tOriginality:  the analysis of deep linear networks has been a prolific line of research in terms of deep learning theory, but the application to the self-supervised setting is novel and promising. Related work is adequately cited, as far as I can see.\n *\tQuality: the submission is very solid technically, with the assumptions of the theory clearly written and interesting results are derived, demonstrating a clear distinction between the two paradigms.\n *\tClarity: the manuscript is clearly written and well organized, introducing all the relevant background in a concise manner.\n *\tSignificance: theory provides here a unique view, where the implications of design choices made in development of different algorithms are made visible and thus can educate development of better algorithms. This premise is not fully fulfilled here, see weaknesses.", "weaknesses": "*\tQuality: this is a theory paper with very limited experimental support (beyond the numerical evaluation of the theory). The random masking and temporal model presented in section 5.1 are very briefly presented, and it is unclear how the simulation results relate to the theoretical predictions. I would expect a clear distillation of an experiment designed to highlight the difference between JEPA and MAE, the qualitative theoretical prediction and how the experimental results support it (or not).\n *\tSignificance: the implications of the results beyond the theory community are unclear. I would expect the authors to be able to offer (i) some characterization of JEPA and MAE, which is useful for practitioners, or (ii) some classification of possible behaviours of self-supervised algorithms, which can inspire the development of new algorithms with different properties than JEPA and MAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q.** “Can you offer qualitative prediction from the theory to real-world systems implementing JEPA or MAE? What would be such prediction if the same deep architecture was trained on both JEPA and MAE loss, what would you expect to see differently in terms of the learned features?”\n\n**A.** (Reproduced from our General Response to All Reviewers): The main qualitative insight from the toy model is that noisy (or high variance) features are learned more slowly and with lower amplitude when using the JEPA loss. A direct prediction that follows from our theory is that JEPA will tend to focus on the lower subspace of the data variance (PCA space) where most of the perceptual features reside in natural images (see lines 59 - 62 and 74 - 76 in the intro of the paper). We have conducted additional experiments on realistic models/data providing evidence for this claim (see attached pdf). As additional evidence for this in the literature we would like to point the reviewer to [1] which shows how reconstruction losses tend to focus on the upper part of the PCA space, and [2] which shows that JEPA tends to learn “slow features” (low variance). Our work can be seen as a first principled analysis of these claims in toy settings. Notably, the toy linear setting not only allows for tractable training dynamics but also has the added benefit that both setups learn the same features. This allows us to focus entirely on comparing the schedule according to which the features are learned for the two setups.\n \n**Q.** What is the spectrum of self-supervised algorithms through the lens of deep linear network dynamics?\n\n**A.** We are not sure we understood the question; could you please clarify what you meant by “spectrum of SSL algorithms”? (Do you mean the linear-algebraic notion of “spectrum”, or are you asking about how the variety of different SSL algorithms manifest in the deep linear setting, beyond MAE and JEPA?)\n\n**Q.** What is the result presented in Figure 3? How does it demonstrate a correspondence between theory and the experimental results? Can you offer a similar plot for the random masking task?\n\n**A.** In Figure 3, we demonstrate that the data distribution considered in our paper (along with its assumed constraints) can in principle be realized — that is, our assumptions are not vacuous. Moreover, it provides a simple data-generative model to help guide intuition about how JEPA and MAE will learn differently in deep linear networks.  We had included two candidates generative processes: one for the static data and the other for time-varying data. Fig 3. focuses on the time-varying (“video-like”) data generation variant. The generative process is given by eq. (17). To summarize, $v^{a}$s are static images, while $u^a=u^a(t)$ are stochastic processes characterized by the autocorrelation coefficients $\\gamma^a$s. The Figure 3.a depicts a sample set of $v$s. Fig 3.b shows two sample realizations of the stochastic functions $u^a$s for two different autocorrelation values. Fig 3.{c, d} show the resulting diagonal and off-diagnal pieces of the covariance matrix generated by the process eq.(17). Fig 3e demonstrates how rapidly one converges to the expected values of $\\rho$ and $\\lambda$ eq (18) derived in the Appendix (as a function of the log of the “mixing time”). The last plot on the right shows diagonal-ity condition is satisfied. These plots together illustrate that the process in eq.(17) generates a data distribution that satisfies our assumptions 4.1.1 and show us how the key parameters of correlation coefficient ($\\rho$) and correlation vary with noise and autocorrelation coefficient $\\gamma$ in each mode. We do include theory, but not a similar plot for the random masking task (Appendix C.1) . We put the discussion of this in the appendix because we wanted to keep the formulation simple in the main paper and avoid confusing readers, but it is possible to construct a similar formulation for the random masking case with some additional assumptions.\n\n**Q.** Can you demonstrate the effect of the number of layers is different under JEPA and MAE, as the theory predicts?\n\n**A.** Outside of figure 4 which verifies the effect in the linear setting, we have not done comprehensive experiments studying the effect of depth for non-linear networks. \n\nWe thank the reviewer for their time and effort in reviewing our paper. If we have sufficiently addressed the reviewers concerns, we kindly ask them to consider raising their score.  \n\n[1]: Balestriero et al: “Learning By reconstruction Produces Uninformative Features for Perception“\n\n[2] Sobal et al: “Joint Embedding Predictive Architectures Focus On Slow Features”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOxO4iWVNN", "reviewer": "Reviewer_5fn3", "summary": "The paper investigates the implicit bias of predictive self-supervised learning methods, specifically focusing on the Joint-Embedding Predictive Architecture (JEPA) and comparing it with the Masked Autoencoders (MAE). The study presents a theoretical analysis of the learning dynamics of these methods, revealing how different objectives lead to varied implicit biases in feature learning. It also includes numerical experiments with linear generative models to illustrate the theoretical findings.", "questions": "- How well do the theoretical results generalize to non-linear models and more complex data distributions? The paper's findings are based on linear generative models, which may not fully represent the behavior of JEPA and MAE in practical settings.\n- what are the practical implications of the implicit bias observed in JEPA and MAE? How should practitioners account for these biases when applying these methods to real-world tasks?\n-The paper suggests that JEPA may be more efficient in learning semantic features. Can this efficiency be quantified in practical scenarios, and how does it impact downstream tasks such as classification and object detection?\n-aree there strategies to mitigate the negative effects of the implicit bias observed in JEPA and MAE? For instance, can architectural modifications or additional regularization techniques help balance the feature learning dynamics?", "limitations": "The paper offers valuable theoretical insights into the implicit bias of JEPA and MAE in predictive self-supervised learning. However, it falls short in terms of related work, novelty,and experimental scope. Addressing these weaknesses and answering the important questions raised would significantly enhance the impact and applicability of the research.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a rigorous theoretical framework for understanding the implicit bias of JEPA and MAE. This is valuable for the community as it offers insights into why these methods might prioritize certain features over others during training.\n- By comparing JEPA and MAE, the paper helps delineate the strengths and weaknesses of each method. This comparative approach can guide practitioners in choosing the appropriate method for their specific tasks.", "weaknesses": "-The paper would benefit from experiments on more diverse datasets, including real-world data. This would demonstrate the practical implications of the theoretical findings and validate their robustness in more varied scenarios.\n- The numerical experiments are limited in scope. They primarily focus on linear generative models, which may not fully capture the complexities of real-world data. Expanding the experiments to include more diverse datasets and model architectures would strengthen the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q.** “How well do the theoretical results generalize to non-linear models and more complex data distributions?”\n\n**A.** We have conducted additional experiments on ImageNet (in attached pdf), which are consistent with aspects of our theoretical predictions. (Please see the pdf for details of the setup).\nRegarding our focus on linear models: The question of how to quantify the efficiency of learning semantic features in practical scenarios may require different metrics than the ones used in this paper. Beyond analytical tractability, concentrating on deep linear networks allows us to separate the question of which features are learned by MAE and JEPA models from the timing/order with which these features are learned, the later being the main focus of this work. Because there is a leap in going between the linear and non-linear setting, we focused on deriving exact results and theoretical guarantees in this paper. Given the long and prolific literature on deep linear models, we view this contribution as self contained, and leave for future work the construction of metrics (of which there may be many) that allow us to analyze the non-linear case where both the features and order are different between the two models. \n\n**Q.** what are the practical implications of the implicit bias observed in JEPA and MAE?\n\n**A.** The main qualitative insight from our results is that noisy (or high variance) features are learned more slowly and with lower amplitude when using the JEPA loss, since a large feature variance across the dataset (denoted as \\sigma_i in the paper) reduces the regression coefficient for a fixed cross covariance \\lambda_i. A direct prediction that follows from this is that JEPA will tend to focus on the lower subspace of the data variance (PCA space) where most of the perceptual features reside in natural images, as claimed in [1] (see lines 59 - 62 and 74 - 76 in the intro of the paper). \n\nWe have conducted additional experiments on ImageNet providing evidence for this claim (see attached pdf). As additional evidence for this in the literature we would like to point the reviewer to [1] which shows how unlike JEPA, reconstruction losses tend to focus on the upper part of the PCA space, and [2] which shows that JEPA tends to learn “slow features” (low variance). Our work can be seen as a first principled analysis of these claims in a toy settings. Additionally, our results provide an intuition to why JEPA objectives are perhaps inefficient for learning features suited for fine-grained pixel level tasks, as those features tend to be noisy (features that would correspond to a low regression coefficient in the linear setting). Finally, our results point to a fundamental limit of the efficiency of the MAE objective in learning semantic features since depth does not meaningfully change the feature learning dynamics (see theorem 4.7 and figure 4 in the paper), unlike the JEPA objective. Questions such as how practitioners should account for these insights and limitations in practice we consider out of scope and left to future work.\n\n[1]: Randall et al: “Learning By reconstruction Produces Uninformative Features for Perception“\n\n[2] Sobal et al: “Joint Embedding Predictive Architectures Focus On Slow Features”\n\n\nWe thank the reviewer for their time and effort in reviewing our paper. If we have sufficiently addressed the reviewers concerns, we kindly ask them to consider raising their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wJgzVd1jTo", "reviewer": "Reviewer_MHjv", "summary": "This paper presents a theoretical analysis of the JEPA and MAE SSL objectives for deep linear networks. Under a somewhat restrictive diagonal covariance assumption, the authors demonstrate that the critical time for learning a feature is dependent only on the input variance for MAE, while JEPA prioritizes learning features with high regression coefficients, which are predictive yet have minimal noise as measured by their variance across the training set (for large depth encoders).\n\nThe authors provide various numerical simulations to validate their theory and show that their findings are robust with respect to the network initialization scheme, even when using a more restrictive initialization necessary for the theory.\n\nOverall, this paper offers valuable insights into the behavior of JEPA and MAE SSL objectives in deep linear networks, and provides a solid foundation for future research in this area.", "questions": "- Would the main theoretical results hold for deep predictor/decoder? \n\n- Given that MAE predicts a corrupted input in the same input space, what does it mean for covariance_{x,y} to differ from covariance_{x,x}, and how does the corruption affect this difference?", "limitations": "Authors adequately addressed the limitations.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work is the first to provide theoretical insights into the empirical observation that JEPA-based approaches tend to learn 'abstract' features more efficiently than MAE. Although the theory is limited to a restrictive case of diagonal linear networks, it calls for further research to generalize these findings.\n\n- The numerical simulations and Section 3 effectively support the theory while also highlighting the qualitative differences between MAE and JEPA.", "weaknesses": "-While the paper is generally clear, the presentation of the theoretical results could be improved. Specifically, it would be helpful to expand on Theorems 4.4 and 4.6 in the main paper, as their validity is not immediately clear without referring to the appendix.\n\n-In addition to the diagonal assumption, the authors focus on the case where the predictor/decoder is linear. However, it is important to use a deep predictor/decoder for both JEPA and MAE methods in practice.\n\n-It would be useful to provide more intuition on what covariance_{x,x} and covariance_{x,y} represent for usual pretraining tasks. This would help readers better understand the underlying concepts and appreciate the significance of the theoretical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q.** Would the main theoretical results hold for deep predictor/decoder?\n\n**A.** Our analysis can be directly extended to deep linear decoders/predictors without changing the results in any qualitative way. Nonlinear predictors however are non-tractable, and therefore beyond the scope of this work. We opted to use a shallow rather than a deep linear predictor in our analyzed model due to 1) a deeper predictor would add to an already notation heavy presentation, without adding or changing the results in any meaningful way. 2) JEPA models typically use a lightweight predictor relative to the encoder, and even linear predictors (see [1]). And 3) Shallow linear predictors have been analyzed theoretically in [1] and [2] and have been shown to work even in practice (though not necessarily producing SOTA results).\n\n**Q.** Given that MAE predicts a corrupted input in the same input space, what does it mean for covariance_{x,y} to differ from covariance_{x,x}, and how does the corruption affect this difference?\n\n**A.** The difference comes from masking or mapping that is applied in the self-supervised model. In the case of random masking one place the difference occurs is on the diagonal elements: because the same element will always be zero in either x_i or y_i, the diagonal elements of C_{xy} will be zero, and this is not true of C_{xx}. For a full derivation and the full impact please see Appendix C.1. One intuitive impact of this difference is that increasing iid pixel noise will not impact diagonal elements of C_{xy} but will impact diagonal elements of C_{xx} leading to a suppression of regression coefficients.\n\nWe thank the reviewer for their time and effort in reviewing our paper. If we have sufficiently addressed the reviewers concerns, we kindly ask them to consider raising their score.  \n\n[1] Tian et al: Understanding Self-Supervised Learning Dynamics Without Contrastive Pairs“\n[2] Richemond et al: “The Edge of Orthogonality: A Simple View of What Makes BYOL Tick”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response to all Reviewers**

We would like to thank all reviewers for their time and dedication in reviewing our paper, and for their support. 
In response to several reviewers, we have conducted additional ImageNet experiments demonstrating phenomena consistent with our theory (described in the attached pdf).
We would also like to address a common question raised by the reviewers in the following general response.

**On the implication of the results to practice:**  The main qualitative insight from our results is that noisy (or high variance) features are learned more slowly and with lower amplitude when using the JEPA loss, since a large feature variance across the dataset (denoted as \sigma_i in the paper) reduces the regression coefficient for a fixed cross covariance \lambda_i. A direct prediction that follows from this is that JEPA will tend to focus on the lower subspace of the data variance (PCA space) where most of the perceptual features reside in natural images, as claimed in [1] (see lines 59 - 62 and 74 - 76 in the intro of the paper). We have conducted additional experiments on realistic models/data providing evidence for this claim (see attached pdf). As additional evidence for this in the literature we would like to point the reviewer to [1] which shows how unlike JEPA, reconstruction losses tend to focus on the upper part of the PCA space, and [2] which shows that JEPA tends to learn “slow features” (low variance). Our work can be seen as a first principled analysis of these claims in a toy settings. Additionally, our results provide an intuition to why JEPA objectives are perhaps inefficient for learning features suited for fine-grained pixel level tasks, as those features tend to be noisy (features that would correspond to a low regression coefficient in the linear setting). Finally, our results point to a fundamental limit of the efficiency of the MAE objective in learning semantic features since depth does not meaningfully change the feature learning dynamics (see theorem 4.7 and figure 4 in the paper), unlike the JEPA objective. Questions such as how practitioners should account for these insights and limitations in practice we consider out of scope and left to future work.

[1]: Balestriero and Lecunl: “Learning By reconstruction Produces Uninformative Features for Perception“

[2] Sobal et al: “Joint Embedding Predictive Architectures Focus On Slow Features”

Meta-review: This paper investigates the implicit biases in two self-supervised learning paradigms: JEPAs and MAE. Through theoretical analysis using deep linear networks, the authors demonstrate that JEPA prioritizes predictive features with minimal noise, while MAE emphasizes high-variance features. These insights are supported by numerical experiments, offering valuable understanding of the distinct feature learning behaviors of JEPA and MAE.

The reviewers (Mqg6, 5fn3, U4uU, MHjv, Nsp4) largely agree that the paper’s theoretical contributions are solid and represent a meaningful addition to the field. Although Mqg6 and MHjv expressed concerns about potentially unrealistic assumptions, the empirical verification under relaxed conditions, as noted by Mqg6, supports the paper’s claims. Additionally, concerns about the limited scope of the experimental results (raised by Nsp4 and 5fn3) were addressed satisfactorily during the rebuttal period with additional experiments.

I recommend accepting this paper for NeurIPS.
Decision: Accept (poster)