Self-Supervised Learning from Structural Invariance
Authors: Yipeng Zhang
Hafez Ghaemi
Jungyoon Lee
Shahab Bakhtiari
Eilif B. Muller
Laurent Charlin
Venue: ICLR 2026 Poster (2026)
Keywords: Self-supervised learning
representation learning
disentanglement
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs.
We study the one-to-many mapping problem in SSL,
where each datum may be mapped to multiple valid targets. 
This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames.
We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce 
a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings.
Our derivation yields a simple regularization term for standard SSL objectives.
The resulting method, which we call
AdaSSL, applies to both contrastive and 
distillation-based
SSL objectives, and we empirically show its versatility in
causal representation learning,
fine-grained image understanding, and world modeling on videos.

Published as a conference paper at ICLR 2026
SELF-SUPERVISED LEARNING FROM STRUCTURAL INVARIANCE
Yipeng Zhang 1.2f
Hafez Ghaemi 1223
Jungyoon Lee
Shahab Bakhtiari' 1.2
Eilif B. Muller 1,2,355
Laurent Charlin 1,2,4,5
Mila- - Québec AI Institute Uniiersitéé de Montréal
SCHU Sainte-Justine
HEC Montréal
*CIFAR Al Chair
ABSTRACT
Joint-embedding self-supervised learning (SSL), the key paradigm for unsuper- vised representation learning from visual data, learns from invariances between semantically-relatee data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., succes- sive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As remedy, we introduce a latent variable account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-baase SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding. and world modeling on videos.
INTRODUCTION
Over the last decade, joint-embedding supervidd learning (SSL) has become the dominant ap- proach in representation learning from unlabeled visual data (Chen et al., 2020a; Zbontar et al., 2021; Grill et al., 2020; Radford et al., 2021; Assran et al., 2023). The intuition behind SSL is to obtain semantically-related data pairs, often called positive pairs, and encourage their representations to be similar, with proper regularization to prevent the encoder collapsing constant function (Wang & Isola, 2020; Garrido et al., 2023a; Zhuo et al., 2023). The hope is that invariance to the differences between positive pairs leads generalizable representations.
However, positive pairs are typically built with handcrafted augmentations (e.g., cropping, color jittering). which perturb pixels while preserving semantics. Such augmentations cannot precisely mimic changes in natural factors of variation that drive real-world distribution shifts (Ibrahim et al., 2023). For example, brightness jitter across pixels does not reproduce how lighting varies across objects or environments (e.g., from indoor to outdoor). Consequently, augmentations may fail to in- duce the right invariances (Ibrahim et al., 2023; 2022; Bouchacourt et al., 2021), discard fine-grained information (Chen al., 2020a; Zhang et al., 2024), incur additional computation burden (Bordes et al., 2023), and require modality-specifff heuristics (Balestriero et al., 2023), ultimately harming downstream performance.
One alternative is to exploit naturally-paired data-e.g.. nearby video frames (Klindt et al., 2021; Bardes et al., 2024; Sermanet et al., 2018), image-caption pairs (Radford al., 2021), class la- bels (Khosla et al., 2020), or embeddings from other models (Sobal et al., 2025a; Feizi et al., (better) reflect real-world variations. From the lens of causal representation learning (CRL) (Yao et al., 2025; Reizinger et al., 2025), positive pairs (x, x+) are deterministically mapped from latent factors sampled according (z,z*) ) p(z)p(z* z),, and we aim recover z from x. Unlike augmentations that perturb the observations directly, natural positive pairs differ according to structured changes in latent factors of the data generating process (DGP). Modeling these latent
1 'Corresoondence to yinen.. zhang@mi 1a quebec. Code is available https: / /i thub. wtt. www.///rigww.//eee..o.oo

---

Published as a conference paper at ICLR 2026
changes can potentially improve generalization (Ibrahim al., 2022; Dittadi et al., 2021; Kaur et al., 2023) and visual understanding (Awal et al., 2024; Garrido et al., 2025; Lippe et al., 2023).
Despite the aforementioned benefits, modeling natural pairs for SSL remains challenging. When learning with augmented pairs, we typically assume the semantic latent factors are invariant, so the latent conditional distribution p(z* z) is highly concentrated near However, natural pairs induce complex p(z* z). For example, in world modeling (Ha & Schmidhuber, 2018b;a; Hafner al., 2025; Assran al., 2025), the present state may lead multiple plausible futures {zt} (e.g.. a car may turn left or right), making the conditional distribution inherently multimodal. For Iaage-captiinn pairs, caption details vary with image complexity, producing heteroscedastic noise. SSL methods that fail to capture this uncertainty often discard information not shared between the pair, leading to degraded performance (Chen et al., 2020a; Radford et al., 2021; Jing et al., 2022; Yuksekgonul et al., 2023; Trusca et al., 2024; Zhang et al., 2024). We argue that leveraging the structure of p(z+ z) enables SSL to learn more diverse and generalizable features- principle we call SSL from structural invariance.
Recent works enable contrastive SSL to learn p(z+ z) that has constant, anisotropic noise (Kügelgen et al., 2021; Zimmermann et al., 2021; Rusak et al., 2025). In this work, we con- tribute a solution to model more complex conditional distributions with unknown shapes. We are inspired by joint-embedding predictive architectures (JEPAs) (LeCun, 2022; Garrido et al., 2024; Assran et al., 2025), which use a latent variable to capture the prediction uncertainty. However, in contrast to prior work (Devillers & Lefort, 2023; Garrido al., 2024; Ghaemi al., 2024; Dan- govski et al., 2022), we do not assume access to this variable and infer it purely from (the structure hidden in) positive pairs. For contrastive learning, this latent variable allows us to derive new tractable bound on the mutual information (MI) between paired embeddings-2 core motivation for SSL (Linsker, 1988; Tschannen et al., 2020; Oord al., 2018) and we empirically show its compatibility with distillation-baan methods. We name our method Adaptive SSL (AdaSSL) to highlight that can adapt to different conditional distributions.
We evaluate AdaSSL in controlled settings with numerical data, natural images, and videos. On numerical data, we show that existing SSL methods lack the ability to model non-trivial condition- als, and AdaSSL achieves better performance both in- and out-of-distribution (OOD). On images, AdaSSL consistently recovers fine-grained features better than baselines and learns more disentan- gled representations. On videos, AdaSSL captures stochastic object accelerations that baselines discard without sacrificing class accuracy.
In summary, main contributions show that:
Naturally paired data that differ in their data-generating factors help SSL methods to recover these factors better and lead to improved generalization performance.
Existing SSL methods cannot capture the non-trivial conditional distributions induced by natural pairs, and we propose two variants AdaSSL to address this limitation.
AdaSSL consistetl outperforms baselines on variety of benchmarks, including weakly- supervised CRL, fine-grained image understanding, and world modeling.
2
BACKGROUND AND MOTIVATION
In this section, we lay out our problem ($2.1) and discuss the limitations of existing SSL methods in addressing it ($2.2, $2.3). We then present a theoretical result showing the importance of modeling heteroscedastic noise ($2.4), which motivates our method.
2.1 PROBLEM FORMULATION
In CRL, representation learning is viewed as learning to invert the DGP such that the latent factors responsible for generating the data are recovered (Zimmermann et al., 2021; Kügelgen et al., 2021; Rusak al., 2025; Reizinger al., 2025). We assume a data pair (x, x+) conforms the following generative process:
p(z), z i p(a* 2).
g(z), x = g(()))
where g Z + X is an unknown mixing function that produces the observations x, x E X based on the latent factors z, x² + 2.
2

---

Published as a conference paper at ICLR 2026
Under Eq. 1, the variability in x is entirely captured by the latent factors z since g is deterministic. A representation that encodes the full latent variability is therefore important for supporting a broad range of downstream tasks, each relying on some (a priori unknown) semantic structure in (Bengio et al., 3a). When SSL. models fail to accurately model p(z+ z), they may discard features that contribute the unmodeled variance, because doing removes them from the target and thereby reduces the prediction error.
Formally, our goal is to learn a function f X + Rd that encodes the data X' into an embedding space R4 such that we can predict, or identify, a subset of the latent factors that are useful for downstream tasks from f(x) with simple function, e.g.. an affine transformation. We denote this subset of latent factors as "content factors" с = žι for I S [dg]. and the other (less relevant) factors as "style" factors = Mdd,]xr following Kügelgen et al. (2021).
2.2 PRELIMINARIES: CONTRASTIVE SSL
Contrastive SSL methods assume the content factors be roughly unperturbed under the condi- tional law Pz- (2- and use an objective that encourage fxx and f(x+) to be similar. To prevent rep- resentation collapse where f becomes constant function, contrastive objectives use another term to encourage the representations to have high entropy (Chen et al., 2020a; Zbontar et al., 2021; Bardes et al., 2022; Wang & Isola, 2020). In this work, we focus on sample-contrastive methods based on InfoNCE (Oord et al., 2018; Chen et al., 2020a), and observe the duality between dimension- and sample-contrastive methods (Garrido et al., 2023a; Balestriero & LeCun, 2022). The InfoNCE loss has the form:
K
ea(x600 x +0)
1
LINONCE = E
log
(2)
((x(6), (
" (xxxx))
K
1=1 K E (=(x(0) xt Wiff
where is temperature parameter and s(,, -) similarity function over pairs. Intuitively, InfoNCE encourages the similarity function to assign a high score for positive pairs and a low score for pairs that does not come from the true joint. The similarity function often adopts simple form the
normalized embeddings, i.e., s(x. ) = (xx) T (y(y) where vb(-) = 2 The simplicity of the
f())
similarity function allows features to be easily extracted from the embedding space because they are used to discriminate between data points linearly during training (Tschannen et al., 2020).
has been shown that when the marginal p(z*) is uniform, the similarity function implicitly mod- els the log conditional: s* (xxxxt x+) log p(2" + z) (Zimmermann al., 2021). With dot-product similarity, the hypothesis class of Ps+|z reduces to von Mises-Fisher (vMF) distributions, where r controls the concentration strength. Since vMF distribution does not account for anisotropic noise, Rusak et al. (2025) introduces a diagonal matrix A that weighs the concentration along each di- mension: s(x,
-(1(x) vvy)) (+x(x) - vy()). Nevertheless, remains unclear how to flexibly model an arbitrary conditional distribution p(z" z) while keeping the similarity function simple enough allow efficient feature extraction.
2.3 PRELIMINARIES: DISTILLATION-BASED SSL
Distillation-based SSL methods do not use explicit regularization to prevent representation collapse. Typically, they use asymmetric encoders: an online branch predicts target representations, with a stop-gradient on the target (Grill al., 2020; Chen & He, 2021). We illustrate our findings with BYOL (Grill et al., 2020), the backbone of many recent successful distillation methods (Guo al., 2022; Assran al., 2025):
LBYOL [[n(((x)) VEMA (x+ (x+)
(3)
where d'EAA is the exponential moving average of the parameters defining zi over time and n7(-) an MLP predictor. Intuitively, the predictor accounts for cases where EE|z 2) * 2; but it remains unclear how it can capture complex noise structures in p(z* 2 which may be heteroscedastic or even multmooal--withou conditioning on additional information.
A Ithough full latent recovery is often the goal in theory, invariance to certain style factors can help gener- alization (Deng al., 2022) and prevent shortcut solutions SSL (Chen al., 2020a) under finite data.
3

---

Published as a conference paper at ICLR 2026
Seu
S3L
Jo+jj 1ix Σ (/o)))
sample
fixt
SOCC
qplel xxxx)
100
sparsity regularization
(a) Contrastive SSL
(b) Distillation-based SSL
(c) AdaSSL (Ours)
(d) AdaSSL-V
(c) AdaSSL-S
Figure 1: Visual comparison of models. Boxes denote vectors and arrows denote functions. We use denote both encoders although they may use different parameters in practice. (a) Contrastive SSL typically uses symmetric architecture. (b) iistillation-based SSL uses a predictor to predict the embeddings of one branch from the other, optionally with the help of some supervision r' related to the difference between the imputs. (c) Our method, AdaSSL, extends SSL by modeling the latent variable r. (d) AdaSSL-V learns a variational distribution, ( x, x ). and uses an MLP as predictor. (e) AdaSSL-S regularizes the sparsity of and uses modular predictor.
2.4 UNAVOIDABLE HETEROSCEDASTICITS
Before presenting our solution to these questions, we first provide theoretical result that under- scores the importance of modeling heteroscedasticity between paired embeddings.
Proposition 2.1. Let går idd++ denote the dy-dimensional unit sphere. Let g :Rd + gd be C² diffeomorphic to its image, and let f Rd. + gd be C² almost everywhere. Define h == fo oog Rd + g4r, Assume further that the random vectors z. E RRd= are sampled as z ~ Pz. zt E,E Ps. where Pz is not point mass and is independent of z, E[=] 0, and Cov(s) > O. Suppose that for Pz almost every z we have rank Dh(z) = dz. Write = h(z) and
H+ = h(z*). Then the conditional law Ph IH (n((("))) h(z)) necessarily heteroxcedastic: its conditional variance depends on h(z) for Pa Pa-almost every z.
Proposition 2.1 shows that heteroscedasticity between paired embeddings emerges from the geo- metric mismatch between the embedding space and the ground-truth latent space, regardless of the encoding function or embedding dimensionality (proof in SC.1). Intuitively, mapping the flat latent space Rd onto a curved manifold such as gúd distorts local neighborhoods differently depending on the location of hz),, causing input-dependent variance in p(h(2t) 1zz)). Here, explicitly show the case of projecting from unbounded latent space Rdd to normalized embedding space Sdj and discuss the reverse scenario $C.2.
In SSL, standard similarity functions and predictors implicitly assume that positive pairs exhibit the same noise scale ($2.2, $2.3), but Proposition 2.1 shows that this cannot hold when the (un- known) geometry of Z and the embedding space differ. Consequently, common similarity functions such the dot product fail capture this conditional variance, since they aggregate the variability uniformly across all embedding directions and data pairs. We show this empirically in $4.2.
METHOD
We now present our method, which addresses the aforementioned challenges by modeling uncer- tainty with latent-variable model. In $3.1, we introduce our overall objective. We then discuss two variants of AdaSSL in $3.2 and $3.3, which optimize this objective in distinct ways.
3.1 MODELING UNCERTAINTY WITH A LATENT VARIABLE.
Learning a representation that maximally preserves the mutual information (MI) between paired embeddings is useful for representation learning and motivates SSL (Linsker, 1988; Tschannen et al., 2020; Oord et al., 2018). Contrastive SSL optimizes lower bound on the mutual information [(f(x): f(x*)) (Oord et al., 2018) but, as discussed Sec. 2.2, relies on strong assumptions on the z).
latent conditional p(z
To relax these assumptions, we introduce an auxiliary latent variable r to parameterize the condi- tional uncertainty in p(2+ z). This induces latent-variable model p(r, 2 2) p(r z)p(z*
4

---

Published as a conference paper at ICLR 2026
Constant
Anisotropic
Heteroscedastic
Multimodal
Complex
Constant
condition on
p(x* x 1
piz"
p(x" |r -== pix" |r=r<) 1
Figure 2: Illustration of different types of noise structure in p(z+ 2). Here, "constant" refers isotropic, homoscedastic noise. Conditioning on latent variable can transform the noise into simpler form. For example, a car may turn left or right, producing a bimodal conditional distribution; conditioning on the driver's intention removes the irrelevant mode.
z), where p(r z) acts as conditional prior. In Fig. 1, we visually compare our method to exist- ing approaches. Intuitively, should contain information about zt that cannot be solely predicted from z. For example, r may represent camera movement, agent actions, or temporal gaps-factors that modify the underlying latents before they are mapped to observations through g. Consequently, modeling p(z* z, r) may require simpler model (c.g., s(..,) -) in Eq. 2 or 17()) in Eq. 3) compared modeling the full p(z" 2). as illustrated in Fig. 2.
With this in mind, we can rearrange fxx; f(x+)) into form that is easier to model. Specifically, by the chain rule of MI.
[ff(x); = (f(xx),r; - I(;
f(x)).
(4)
Intuitively, optimizing the first term on the RHS involves modeling p(z 2, r) instead of p(z* z)) to
This can be achieved through an SSL objective that encourages the model to use information reduce the uncertainty in predicting f(x+) from (x)). However, without constraints, r could encode f(x+) directly, increasing the second term and creating shortcut. This motivates the general form of our objective:
(5) where the SSL term is any standard SSL loss (e.g., LIIIONCE) and the regularizer Le (r) limits this degenerate behavior by discouraging overly informative r. The hyperparameter /3 controls the strength of regularization per standard practice (Higgins et al., 2017; Locatello al.. 2020). This objective matches the conceptual framework depicted in Fig. 13 of LeCun (2022).
LAdaSSL = (ss,((x) + (RReg(r),
3.2 ADASSL-V AND A LOWER BOUND ON I(((x),((xt))
We first estimate the posterior with a variational distribution (r(r x, x+) (Sohn et al., 2015). The joint then becomes p(x, x r) p(x, x x()((r)) x,x*). The informatioonal-thorretee proper- ties of contrastive learning allow us to optimize a lower bound on / xx,x+)², Specifically, the first term in Eq. is bounded by InfoNCE (Oord al., 2018) by treating (x, r) as single variable:
(x x B/5
1 K
1x(x,r:x*) -LafONCE E ((x()) xl K
log
K Eド=
Similarity function. As discussed in $2.2, our goal is to have similarity function that is flexible yet simple. With r as a latent variable, we still use the dro-prrduct similarity on embeddings:
t(f(x),r) [[t(f(x), +x2(x*)= f(x*)
s(x,x' xt .r) (x,, (2(x*)) where (xx,))
(7)
[[f(xt)llz
Specifically, we edit f(x) with the help of and editing function t(.,)) :) such that it lies in the vicinity f(xt). We parameterize / with linear projection or two-layer MLPs for AdaSSL-V.
We derive a bound for the second term of Eq. in $B:
(r, x) 2 -Reg =- -- [DKL (r x, xxxx* (1 [x)]]
(8)
For brevity, use the notation x(x; in this section, but in practice optimize T(f(x); f(x+)) T(x; x)) because method operates paired embeddings.
5

---

Published as a conference paper at ICLR 2026
Thus, by introducing a variational distribution on obtain tractable lower bound on I xx) In practice, we parameterize 4s and py using lightweight MLPs on top of the embeddings fxx and f(x") t,, modeling both factorized Gaussians. Plugging the terms into Eq. 5, we get
LAdaSSL-V = LSsl (B,, 1/1 (x. r), (x)))) + (D x, x+) (po xx)
(9)
We call this variant of our method AdaSSL -avaiia-vvaiiaiinnna) For InfoNCE, The first term explicilty stated in Eq. 6. For BYOL, we replace the input to the predictor 77-) in Eq. 3 with 4/1 (x,
Remark. Although AdaSSL-V is only theoretically justified for contrastive SSL, one can use a distillation-based SSL loss as well because they still encourage r to aid prediction of f(x+).
3.3 A S AND SPARSE MODULAR EDITS
Natural transitions usually correspond sparse changes in the latent factors, an inductive bias widely adopted in the CRL literature (Ahuja et al., 2022; Klindt et al., 2021; Lippe et al., 2023). Therefore, we hypothesize that we can implement Eq. 5 by predicting r and regularizing its sparsity. AdaSSL-SSparse) realizes this idea. Instead of variational approximation, we predict r determin- istically from f(x) and f(xt): ): r (xx),, f fxxx)(xx))))))) )), where is an MLP followed by tanh activation. We then regularize the sparsity of r:
LAaASSI- = CSSI. (vi (x, r), (x((xx)) )) + 3|rl|o,
(10)
where the To pe is made differentiable through the Gumbel- -Sigmoid estimator ($E.1). In- spired by Ibrahim et al. (2022); Hu et al. (2022), we use modular editing function for AdaSSL-S:
d (((()),)) -(x)) + Σ r.((((x))) = ((x) + Σ r. (B.A,,((x)) + bi),
d.
(11)
1=1
==1
where d, is the dimensionality of r. Each (.()) an affine transformation parameterized by rank-1 matrix A, and a scalar offset b; This design is motivated by the assumption that differences between the paired embeddings lie in a low-dimensional latent subspace, where edits are applied.
In $4.4, we will show that AdaSSL-S works well with contrastive learning. but requires additional care for distillation- methods in some setups.
4 EXPERIMENTS
We evaluate A in various settings to show it learns diverse and generalizable representations. We start with two benchmarks where we have access to the ground-truth data generating factors. First, we generate numerical data where we systematically increase the complexity of p(z* + | z), and show AdaSSL mitigates the limitations of existing SSL methods (64.2). Second, we use 3D- rendered images from 3DIdent (Zimmermann et al., 2021) and show AdaSSL identifies latent factors better than all baselines ($4.3).
For other benchmarks, we do not have access to the ground-truth data generating factors. Instead, we use proxy tasks such as fine-grained classification to evaluate the quality of the learned represen- tations. On natural images from CelebA (Liu et al., 2015), AdaSSL captures fine-grained features and generalizes to OOD data ($4.4). On large-scale image data from iNat-2021 (Van Horn et al., 2021), we show that AdaSSL is more robust to noisy data pairings than vanilla SSL (SD.2). Finally, in $4.4 and $D.5, we show that AdaSSL outperforms baselines on modeling the uncertainty in video transitions on an extended Moving-MNN -NIST dataset (Srivastava et al., 2015; Drozdov et al., 2024).
Throughout this section, we refer to data pairs that differ in the underlying latent factors as natural pairs and those constructed using augmented views of the same image as standard pairs. We show that natural pairs enable us to learn better representations in these settings.
4.1 OVERVIEW OF EXPERIMENTAL PROTOCOL
Baselines. Our experiments in $4.2, $4.3 focus on contrastive SSL. As discussed in $2.2, In- foNCE (Chen et al., 2020a; Oord et al., 2018) and AnInfoNCE (Rusak et al., 2025) are the con-
We slightly abuse the notation and use ||||| denote the number of non-zero elements of r.

---

Published as a conference paper at ICLR 2026
Table 1: Linear regression R² on unimodal p(z+ z). All experiments share the same E and the mixing function g for each trial. Although all models achieve good performance on the training set p(z), a flexible model is crucial to achieving good OOD performance. Values below 0.7 are dimmed.
Varjet [e)
Model
p(x)
MODEL SPACE: UNBOUNDED
-
[0055-
1Jooo
p(a)
MODEL SPACE: HYPERSPHERE
5
50055
-ooon
Idcatity
0.7410
0,5103 6004
0.1243 1.00
0,7410
0.5103 6:000
0,1243
InfoNCE
0,9912 è nxesI
0,9614 6000
0.8924
: w000
0.8657
03442
0.8004
a 00m0
0.2683
InfoNCE
0,9943
0.9731
0.9564
0,9785
0.9104
0.6944
H-InfoNCEAtion
0,9956 a worr
0.9736
6080
0.9592
a w0872
0.9953
2.00)
0,9645
0,9154
= came
InfoNCE
0.9968 1
0.9764
0.9668
0,9509
0,7755 11135
0,3523
Anisotropic
AnInfoNCE
0.9962 2000
0,9753
0.9627
0,9613 g.
0,8403 0000
0.4022
InloNCEAmi
0.9963 000ie
0.9685
60002
0.9510
0,9970
c.017
0.9537
8010
0,9018
0.025
InfoNCE
0.8553 40032
0.2664
-0.199
0.7851
4.000
0.0209
Heteroscedastic
AnInfoNCE
0.8447 2001
0.2745
-0,2277
0.7563
0,2563
0.0070
(altine-aciiaation)
I1 InfoNCEAMtio
0.9826
0.9482
0.8666
0,9426
0.6276
0.3106
H-InloNCExtr
0,9892
0.9610
0.9149
00044
0.9855
0,9288 8 80175
0.7633 m 68576
trastive baselines that account for isotropic and anisotropic noise in p(z* z), respectively. AnIn- foNCE learns directional weights of the similarity function, A. For a fair comparison, we also use a learnable scalar weight À for other methods in $4.2 and $4.4 and find it beneficial. On natural images in $4.4, we experiment with both InfoNCE and B BOL (Grill et al., 2020). We also compare with Ibrahim et al. (2022), which models the change between latent factors as Lie group transfor- mations; we denote this method as LieSSL. For the CRL benchmark in $4.3, we include classic disentanglement methods, including 3- 3-VAR (Higgins et al., 2017) and AdaGVAE (Locatello et al., 2020). For the video experiments in $4.4, we use BYOL as our base SSL method.
H-InfoNCE. In addition existing baselines. we introduce H-InfONCE, which extend AnIn- foNCE to account for heteroscedastic noise by predicting A₂ from f(x) with an affine function H-InfoNCCAAAted or an MLP (H-InfoNCExLLP; it replaces in AnInfoNCE's similarity function with this conditional Ax (see Table 8). Additionally, H-InfoNCE uses another MLP predictor to predict f(x+) from fxx., similar to distillation
SSL, except for in Table 1, where we ensure E|zt| x| z. Note that all InfoNCE, AnInfoNCE, and H-InfoNCE assume unimodal p(2+ 2) while AdaSSL relaxes this assumption by using a latent- variable model.
five-layer MLP as f for the numerical experiments in $4.2, ResNet-18 18 encoder followed by two-layer MLP projector as ʃ for the image experiments in $4.3 and $4.4, ResNet-50 encoder followed by two-layer MLP projector as f in $D.2, and five-layer 3D CNN followed by three-layer MLP projector as for videos. We train all models from random initializations. For evaluation in $4.2 and $4.3, we follow Zimmermann et al. (2021) by training an affine probe on top of the embeddings produced by the frozen f on the training data. For evaluation in $4.4, we train an affine probe on both the embeddings (output of the frozen projector) and the output of the frozen encoder, which we refer to representations. We then evaluate the probes' performance on the test set following standard practice (Chen et al., 2020a; Grill et al., 2020). All results are reported as the mean and standard deviation over at least three random seeds. Additional experimental details and data visualizations can be found in SE
Experimental setup. We
4.2 NUMERICAL DATA
In this section, we study the effect of complexity of the conditional variance in p(z* z). Specif- ically, we sample correlated latents с (0, Σ), where Σ is sampled from an Inverse-Wishar distribution (mean I, with typically nonzero correlations), and sample c+ from different condi- tional distributions p(c+ c). Style latents are sampled independently: s ~ (O, I), yielding z = [c, and z = |ct Denote the training latent distribution as p(z). A random invertible MLP parameterizing g (details in $E.2) maps these latents to observations x, x via Eq. 1. We then train ff on the (x, x*) pairs under the SSL algorithms and freeze it.
Next, we train linear regressor to predict from the frozen embeddings, f(x = (((((,,))) We perform three evaluations by varying the training and testing distributions of the regressor:
p(z): both train and test use z ~ p(z).
(0 5 I): both train and test use Z ~ (O, I). This tests embeddings under covariate shift.
(0,5 5 DOOD: train on z ~ p(z) but test on z ~ NN 5 I). This corresponds to a practical
scenario where distribution shifts happen after deployment when both the encoder and regressor
7

---

Published as a conference paper at ICLR 2026
Sample
I-NN of ((x)) r)
Table 2: Identifiability results on 3D1dent.
AdaSSL achieves the best disentanglement and R scores.
Position
denotes "same as above".
Model
Pairing disent. (1)
(1)
Spotlight
B.VAE
0,2076 gaen
0.6649 20000
3-VAEa-15
0,1883 sam
0.6672
0,3352 gease
0.6691 2
Hue
Natural
0,4098
0.6436
1d
0,3800 g:3
0.6511 1:00
AdGGVAE
0.4582 63355
0.6213
1000
All
InfoNCE AnlafoNCE
Standard
0.1447 2:000
4,000
0,1349 noxe?
0.3704
1-0033
InfoNCE AnIafaNCE
Natural
0,1178 tans
0.8184
Figure 3: AdaSSL-V performs controllable re- trieval. From the query image we sample F from different dimensions of the learned prior Po x) which correspond to interpretable changes in the edited image tf(()),
0.2772 12144
0.8243
4.0002
AdaSSL-V
0.4661
0,8857
AdaSSL- Viliea
0.2756 =: t:mo
0,9331 1:0007
AdaSSL-V VALL
0.1027 nows
0.89418 1-0307
AdaSSL-S
0.1777 2:00
0,9309
are frozen. This setting requires not only robust regressor but also that the representation be well aligned across the two distributions (Ruan et al., 2022).
Following prior works (Zimmermann et al., 2021; Kügelgen et al., 2021). vary the latent space assumptions (unbounded or hypersphere) and model flexibility (InfoNCE, AnInfoNCE, or H-InfoNCE) by changing the similarity function.
We first construct a unimodal conditional, where we expect H-InfoNCE to suffice. We sample c+ following of c NCT;; C22 (c)))), with (c) either O, isotropic, anisotropic, heteroscedastic, where (-) is random affine function followed by softplus activation.
Table leads to two main observations. First, models achieve high performance when both their embedding space and model flexibility match the true conditional p(ct c); otherwise we a decrease in performance, which corroborates the findings of Zimmermann et al. (2021). No- tably, we see clear performance drop with InfoNCE and AnInfoNCE with normalized embedding space. H-InfoNCE improves the performance by large margin by capturing heteroscedasticity. consistent with Proposition 2.1. Second, while latent correlations help all models perform well on in-distribution data p( z). flexible models' improved performance is more pronounced under OOD evaluation. Under heteroscedastic noise, the encoders learned with InfoNCE and AnInfoNCE fall short, even trailing the identity function. Interestingly, when Var(c+ c) O, generalization per- formance of InfoNCE is weaker than the best models each block, supporting our hypothesis that naturally varying pairs that have independent variation along latent factors can help generalization.
In $D.1, we show that H-ffNNCCE also improves upon baselines on more complex (heteroscedastic and multimodal) distributions of p(c+ c), and AdaSSL provides further improvements.
4.3 SYNTHETIC IMAGES
We next show AdaSSL can be used to recover all data-generating factors from natural pairs on 3DIdent (Zimmermann et al., 2021), dataset of realistically rendered images of teapot varying in ten data generating factors such as position, spotlight, and hue. Following Locatello et al. (2020), generate natural pairs by first drawing two samples from the marginal latent distribution. Then, each latent coordinate is replaced with some probability by the corresponding coordinate from the other sample (details in {E.3). We evaluate (a) disentanglement in the learned embeddings with the DCT disentanglement score (Eastwood & Williams, 2018), and (b) the recovery of latent factors, i.e., "empirical" identifiability, up to affine transformations with R².
Table 2 shows that V-VAE and AdaGVAE fail to identify the latent factors, though AdaGVAE achieves decent disentanglement. InfoNCE with augmentations performs worse, likely because augmentation invariance conflicts with the CRL objective. SSL baselines using natural pairs achieve good latent recovery but yield more entangled latent factors. We hypothesize that AdaSSL's regu- larization encourages efficient encodings of r, akin to 3-VAE (Higgins et al., 2017; Burgess al., 2018). Since r is modeled as factorized Gaussians, some disentanglement in r is expected. To verify this, we vary the complexity of the editing function (additive, linear, nonlinear) denoted by sub-
8

---

Published as a conference paper at ICLR 2026
Linear Fi scores on representations (en- coder output) and embeddings (projector output) trained on CelebA, under weak or strong augmentations. A soft performance upper bound, uses the ground-truth attribute difference as r.
Table 3:
94
Top-1 accuracy (%)
.3-
BYOI +GI
AdaSSL-V (d., B)
(2,
92
AdaSSUS
(2, 0.001)
(2, 0,005)
91
(2, 0.01)
Model
Patring
WEAK AUGNENTATION
Regc.
Emn
STRING ALCMENCATION
Repr.
Emb
90
BYOL
(5, 0,01)
(5, 0,001)
AstaSINCE
0.2534
(1 0.3098 wws
B1 111395 2r0n
0.1822
0,5922
0.5963
2400
1000
0,5742 2000
0.9064 101
(10, 0.001)
(10, 0.01)
CONTRASTIVE
InfoNCE
Natural
0.5473 Km
0.3747
0,5754 1000
0.444
BYOL+ Future
AnlafaNCE
0.5413
0.4240
0,5789
1100
0.495
LasSS
0.5331
(15079
014525
014550
0.5789
0.5935
0,513k 105
0 15585
0.88
0,90
0.92
AdiSSL V
0.5784
0.4014
043
Velocity decoding (R²)
S
0.5676
0.4581
0,5654
AdrSSE
06818
0.6540
0.677
2000
0,5822 M01
Velocity decoding and digit recognition from representations on stochastic Moving-MNIST. Marker ra- dius indicates standard deviation.
Figure 4:
DASTILL ACTION
BYOL
Standadd
0,2989 xmp
0.1832
0,5368
0.5143
200r
BYOL
Natuna
0.5455
0.4263
0.5605
1000
0
A
0,8816
0.5067
0,5702
0.532
0.6571
013014
0.354
0,477
0.9948
0.5730
0.5964 c00
0.5872
scripts. With simpler t, the dimensions of f(x) must align more directly with those of r, making any disentanglement in r more visible in fxx). Indeed, simpler leads more disentagled embed- dings while consistently outperforming baselines on regression. In particular, AdaSSL-V Linear and AdaSSL-S, which both use linear editing functions, achieve the highest R².
To better understand the learned r, we visualize its effect by retrieving nearest neighbor of query image x after editing it with samples r (Fig. 3). Given evidence of disentanglement, we expect sam- pling specific latent dimensions to induce meaningful changes in the edited embeddings (( (x), ) F). Concretely, we sample i; ~ (r; x) for i E TL S [d.]] for some set of latent indices IL., and fix all others to their expectations. Fig. shows results for three different L's. We find that we can retrieve objects that differ in position, spotlight, and color, while leaving most other factors unchanged, though orientation remains entangled with other factors. Finally, when sampling from the full prior, we retrieve images that differ sparsely in latent factors, consistent with the training DGP.
Together, these results highlight SSL promising path for CRL given its efficiency (no recon- struction) and demonstrated scalability to high-dimensional images.
4.4 NATURAL IMAGES AND VIDEOS
Natural images. Although we do not have access the ground-truth data generating factors of natural images, we perform experiments on CelebA (Liu et al., 2015) which contains celebrity im- ages with annotated facial attributes. We obtain real-world natural pairs by matching different photos of the same celebrity, which differ sparsely in their facial attributes ($E.4). We then train models on paired images and evaluate with affine probes on 40 facial attributes of unseen identities, inducing natural distribution shift. Results in Table 3 show that standard pairing rely strong augmen- tations to work well. However, using natural pairs largely reduces the gap, and AdaSSL-V is the only method that consistently improves upon the standard pairing baselines across all settings. This exposes the weakness of vanilla SSL objectives under complex conditionals in natural pairs. We still observe gap between AdaSSL and AdaSSL+GT, indicating room for improvement in future work.
Interestingly, while AdaSSL-S consistently improves InfoNCE on natural pairs, this benefit is not observed with BYOL. We hypothesize that explicitly regularizing the embeddings' information content (as in the denominator of Eq. 2) pressures the model to utilize r effectively. In contrast, distillation-based methods lack this direct tension and may require additional regularization on to achieve the same effect. We provide empirical support of this hypothesis in $D.4. We show AdaSSL scales to larger data and backbones in $D.2.
World modeling on videos. In sections above, we have shown AdaSSL models p(z+ z) well. Since modeling this transition distribution is central to world modeling on videos, we test AdaSSL on it. We hypothesize that inability to model uncertainty drives the model to discard variant fac- tors. We introduce uncertainty by injecting random changes in velocity between two segments of Moving MoVing-MNIST (Srivastava et al., 2015; Drozdov et al., 2024), which then used as positive pairs. We use BYOL (Grill et al., 2020) as the SSL method for this experiment, whose predictions can condition future segment (BYO++Futree similar to Liu et al. (2025) or the ground-truth
9

---

Published as a conference paper at ICLR 2026
change in velocity (BYOL+GT). Fig. 4 shows that AdaSSL captures both the invariant factor, digit, and the variant factor, velocity, better than baselines. Ablation on AdaSSL-V shows its robustness to the dimensionality of r under proper regularization. We include empirical analysis on AdaSSL's world modeling capabilities in $D.5 and additional experimental details in $E.5.
5 RELATED WORK
Self-supervised learning. SSL in the latent space has evolved from solving hand-crafted pretext lasks (Noroozi & Favaro, 2016; Doersch et al., 2015; Dosovitskiy et al., 2014; Gidaris et al., 20)8) to learning semantic-pin preaerriippne representations from invariance to augmentations (Oord et al., 2018; Wu et al., 2018; Gutmann & Hyvärinen, 2010; Chen et al., 2020b; Caron et al., 2020; Wu et al., 2018: He al., 2020: Radford et al.. 2021: Caron et al., 2021; Zbontar et al., 2021: Bardes al.,
2022: Ermolov et al., 2021; Chen & He, 2021; Grill et al., 2020; Assran et al., 2023; Baevski et al., 2022: Caron et al., 2020; He et al., 2016). Studies have also explored the relationship between in- variant representations and variational inference (Bizeul et al., 2024; Sinha & Dieng, 2021). Beyond invariance, equivariant representations preserve transformation information (Hinton et al., 2011) I). In SSL, this is achieved by providing augmentation parameters to the predictor (Garrido et al., 2023b; Ghaemi et al., 2024; Devillers & Lefort, 2023; Garrido et al., 2024; Park et al., 2022), or using subspaces for different invariances (Xiao et al., 2021; Eastwood et al., 2023). However, these ap- proaches are tied to chosen augmentations and break down when the sources of uncertainty are un- known. Alternatively, one can exploit the invariance between observation pairs that are transformed similarly (Shakerinava et al., 2022), or model transformation with Lie groups (Ibrahim et al., 2022). Unlike prior work, our method does not require transformation labels, handles multiple varying fac- tors, and provides a simple, theoretically justified objective that is compatible with standard SSL methods across diverse settings.
Causal representation learning. Much research examines recovering data-generating factors and their causal relations (Hyvarinen & Morioka, 2016; Schölkopf et al., 2021; von Kügelgen et al., 2023; Ahuja et al., 2023; Brehmer al., 2022; Locatello et al., 2020; Lachapelle et al., 2022; Lippe et al., 2023; Klindt et al., 2021; Ahuja et al., 2022; Lippe et al., 2022; Yao et al., 2025). While offering theoretical guarantees, these methods often rely on strong assumptions or probabilistic gen- erative models, limiting scalability. SSL has been connected to CRL (Zimmermann et al., 2021; Kügelgen et al., 2021; Rusak et al., 2025; Yao et al., 2024), where studies focus on identifying the content factors that follow simple conditionals ($2.2). This work relaxes these assumptions by allowing structured variation between paired latents and demonstrates strong performance on weakly-supervised CRL, a step towards understanding and advancing SSL. (Reizinger et al., 2025).
World modeling with SSL. Unlike image-based SSL that rely on augmentations, video world models with SSL learn the transition dynamics of videos, often by predicting target frames given some context (Sermanet et al., 2018; Feichtenhofer et al., 2021; Bardes et al., 2024; Assran et al., 2025; Schwarzer et al., 2021; Guo et al., 2022). Through the process, the model learns useful rep- resentations for downstream tasks such as video understanding. A key challenge is that uncertainty grows with the temporal gap between positive pairs, forcing models to fix temporal resolution (Fe- ichtenhofer et al., 2021; Bardes al., 2024), which may limit their ability learn features dif- ferent levels of abstractions (Zacks & Tversky, 2001) because the model can discard variant factors. Introducing latent variable r, as we do, can reduce the uncertainty and learn more diverse fea- tures ($4.4). Finally, although we focus on improving SSL in the latent space, we note there are successful approaches that predict in the observation space (Schmidt & Jiang, 2024; Tong et al., 2022; Feichtenhofer et al., 2022; Jang et al., 2024; Bruce et al., 2024; Yang et al., 2024).
CONCLUSION
In this work, we reveal the limitation of SSL methods when trained on naturally paired data and introduce AdaSSL, which learns a latent variable that captures the uncertainty in prediction targets. Our approach consistently outperforms existing methods across all benchmarks. We believe this is a promising step in expanding the capability of SSL methods, leading to potentially fruitful advance- ments in learning generalizable representations, identifying latent factors from nigh-dimensional images, and world modeling with uncertainty.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We appreciate the constructive feedback from the anonymous reviewers. We also thank Siddarth Venkatraman, Michael Chong Wang. Emiliano Penaloza, and Omar Salemohamed for insightful discussions, Anirudh Buvanesh and Lucas Maes for precise pointers literature, and Mehran Shak- erinava for proofreading. Additionally, YZ would like to thank Xiaofeng Zhang and Dhanya Sridhar for helpful feedback during the early development of the idea.
YZ. HG, EBM, and LC acknowledge the generous support of the CIFAR AI Chair program. Addi- tionally, YZ supported by the Al Scholarship from Université de Montréal. HG supported by the UNIQUE Centre
quebec). EBM is supported by NSERC Discovery Grants (RGPIN-2022- and the Canada Excel-
05033). SB is supported by NSERC Discovery Grants TRGPIN-2023-0330 lence Research Chairs (CERC) Program. This research was enabled in part by compute resources provided by Mila (mila.quebec)) and the Digital Research Alliance of Canada (alliancecan.ca)
ETHICS STATEMENT
This work uses the CelebA dataset (Liu et al., 2015), which consists of publicly available celebrity face images collected from the internet. We use it solely for non-commercial academic research in facial attribute learning, under research-only license. Our use does not involve face generation or manipulation, and all experiments are conducted strictly for evaluating algorithmic performance, in line with the intended use of the dataset.
REPRODUCIBILITY STATEMENT
We comprehensively detail our experimental setup in $E. Code to reproduce our results is available at https::t///gttt com
com wttpw//wttttc//////..oo.. AdaSS..
REFERENCES
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 15:15516-15528, 2022.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa- tion learning. In International conference on machine learning, pp. 372-407. PMLR, 2023.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 15619-15629, 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Am- mar Rizvi, Claire Roberts. Koustuv Sinha, Artem Zholus, al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09988, 2025.
Rabiul Awal, Saba Ahmadi, Le Zhang, and Aishwarya Agrawal. Vismin: Visual minimal- understanding. Advances in Neural Information Processing Systems, 37:107795-107829, 2024.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312 12. PMLR, 2022.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive sulf-super learning re- cover global and local spectral embedding methods. Advances in Neural Information Processing Systems, 15:26671-26668. 2022.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. cookbook of self- supervised learning. arXiv preprint arXiv:2304 122/0, 2023.

---

Published as a conference paper at ICLR 2026
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance- www-ininaaae-ovvviiiaee regulariza- tion for self-super learning. In International Conference on Learning Representations, 2022.
URL http://wwwweeeeeeteooeeeeeeeaeae
net forum?
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:
Featured Certification.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-188) 2013a.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1 2013b.
Alice Bizeul, Bernhard Schölkopf. and Carl Allen. A probabilistic model behind self- supervised learning. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL https:
/ //opnreeview. net/ fortm? id=iewz 74 7447tR.
Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards democratizing joint-embedding self-supervised learning. arXiv preprint arXiv:2303.01996, 2023.
Diane Bouchacourt, Mark Ibrahim, and Ari Morcos. Grounding inductive biases in natural images: invariance stems from variations data. Advances Neural Information Processing Systems, 34:19566- 19579, 2021.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38331 2022.
Philippe Brouillard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable causal discovery from interventional data. Advances in Neural Information 2020.
Processing Systems, 33:21
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In Forty-first International Conference on Machine Learning, 2024.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 8-vae. arXiv preprint 1.X1v:1804.03599 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PmLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). 2020b. URL https::/proceett
mlr. pewss/ v119 9 chen20j .html.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference on Learning Representations, 2022. URL https: /poeneeviee... net. teee:/forrmm/ddgggAAAfaaaaa
12

---

Published as a conference paper at ICLR 2026
Weijian Deng, Stephen Gould, and Liang Zheng. On the strong correlation between model invari- ance and generalization. Advances in Neural Information Processing Systems, 35:28052-28067, 2022.
Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In The Eleventh International Conference on Learning Representations, 2023. URL https:
wtt./eemmmnaadmeaaa...... YFt.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. International Conference on Learning Representations, 2021. URL https: wwwp:nrwwwww..eeeeeeeeeeeee www.mmdddd-iioo 1ttpt///oooroootooxoooo
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27, 2014.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson. Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Katrina Drozdov, Ravid Shwartz-Zii and Yann LeCun. Video representation learning with joint- embedding predictive architectures, arXiv preprint 1xXi::2412.10225: 2024.
Cian Eastwood and Christopher I. Williams. framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. URL https: /wwpreeieeeeeeeeeeeeeeeeeeeeeeee
Eoormmriddaaa 7dz--2 AZ2
Cian Eastwood, Julius Kügelgen, Linus Ericsson, Diane Bouchacourt, Pascal Vincent, Mark Ibrahim, and Bernhard Schölkopf. Self-supervised disentanglement by leveraging structure in data augmentations. In Causal Representation Learning Workshop NeurlPS 2023, 2023. URL https: wpwnrevieeeeeeeeeeeeeeeeeeeeeeee net
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learming Research, pp. 3015-3024. PMLR, 18 24 Jul 2021. htLhttps: http:///cccccc....//// wwt..prees//1333eemoooeo
html.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3299-3309, 2021.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958. 2022.
Aarash Feizi, Randall Balestriero, Adriana Romero-Soriana and Reihaneh Rabbany. Gps- ssl: Guided positive sampling to inject prior into self-supervised learning. arXiv preprint arXiv:2401.01999 2024.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the dual- ity between contrastive and non-contrastive self-supervised learning. In The Eleventh Interna- tional Conference Learning Representations, 2023a. URL https:://ppereeeeeeeeee /pppneevie.w.eeeeeeeeee forum? fruumidd=DEE.99 Duffpa.
/
Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. In International Conference on Machine Learning, pp. 10975-10996 PMLR, 2023b.
13

---

Published as a conference paper at ICLR 2026
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun, Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504, 2024.
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rab- bat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self- supervised pretraining on natural videos. arXiv preprint arXiv:2502. /1831, 2025.
Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, and Michael Rab- bat. Learning latent action world models in the wild. arXiv preprint arXiv:2601.05236, 2026.
Hafez Ghaemi, Eilif Benjamin Muller, and Shahab Bakhtiari. Seq-JEPA: Autoregressive predictive learning of invariant- -aruiaant-quivariantt world models. In NeurlPS 2024 Workshop: Self-Supervised Learning Theory and Practice, 2024. URL MO10LAKesJ.
net forum? id=
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL
bettps:/cpeerreet wtw.eeewwweeeeoeeniiteeeeeee S1144N220
net
Lean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraaroo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21233 2020.
Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, et al. Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35: 31855-31870, 2022.
Sharut Gupta, Chenyu Wang, Yifei Wang, Tommi Jaakkola, and Stefanie Jegelka. In-context sym- metries: Self-supervised learning through contextual world models. Advances Neural Informa- fion Processing Systems, 37: 17710550--102280 2024.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 297-304. JMLR Workshop and Conference Proceed- ings, 2010.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018a.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: /803. /0/22, 2018b.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pp. 1025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 9729-9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. In Interna- tional conference on artificial neural networks, pp. 44-51. Springer, 2011.
Christopher Hoang and Mengye Ren. Midway network: Learning representations for recognition and motion from latent dynamics. arXiv preprint arx05225/0.05558, 2025.
14

---

Published as a conference paper at ICLR 2026
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL https: http::://pppneevieeee net. forum? i.d=nZeVKeeFYf3
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Mark Ibrahim, Diane Bouchacourt, and Ari Morcos. Robust self-supervised learning with lie groups.
arXiv preprint urXiv:2210. /3356, 2022.
Mark Ibrahim, Quentin Garrido, Ari S. Morcos, and Diane Bouchacourt. The robustness limits of soTA vision models to natural variation. Transactions on Machine Leurning Research, 2023. ISSN 2835-8856. URL httpt:tt////o /ttpss///ppeevvieeeee ew net forum? wat.mm1ihdhHHhhhh00
Huiwon Jang, Dongyoung Kim, Junsu Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Vi- sual representation learning with stochastic frame prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21289-21305. PMLR, 21-27 Jul 2024. URL https: http::://ppoceeeingg.. mlr wttpssmlm..ppprr.e wwwpppessss333 jang24c html.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference Learning Representations, 2022. URL https: http//pppeeeeeeeeeeeee net/ htte://oooomm/deeeeeeveeese
Jivat Neet Kaur, Emre Kiciman, and Amit Sharma. Modeling the data-generating process is neces- sary for out-of-distribution generalization. In The Eleventh International Conference on Learning Representations, 2023. URL https: http///peeeeeeeeeettt.ee. forum? idduygkssssss LILZX.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18877 2020.
Thomas Kipf, Elise van der Pol, and Max Welling, Contrastive learning of structured world models. In International Conference on Learning Representations, 2020. URL https: //openzeview. watt/forumma deerummidddlggxxxtt.e.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with tempo- ral sparse coding. In International Conference on Learning Representations, 2021. URL
https: apwnreviewvveeeeeee net forum?
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations prov- ably isolates content from style. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: / //openreview net. forum? d=4pf_p0000s
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pp. 428-484. 484.
PMLR, 2022.
Samuel Lavoie. Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pre- training. In International Conference on Machine Learning, pp. 26070-26084 PMLR, 2024.
Yann LeCun. path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.
Ralph Linsker. Self-organization in perceptual network. Computer, 21(3):105-117, 1988.
15

---

Published as a conference paper at ICLR 2026
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning. pp. 13557-13603. PMLR, 2022.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence. pp. 1263-1273. PMLR, 2023.
Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, and Qingming Huang. When the future becomes the past: Taming temporal correspondence for self-supervised video representation learning. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 24033-24044, 2025.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International confer- ence on machine learning. pp. 6348-6359) PMLR, 2020.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL httpss://ppenrrrta www.reeeee.....c.o net forrmm/omm id= Bkg6RiCqY7.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous re- laxation of discrete random variables. In International Conference on Learning Representations, 2017. URL https: wwwpreveeeeeeeee////eee net forum?id= jE5L5g1.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint wrXiv:1807.03748. 2018
Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan- Willem Van De Meent, and Robin Walters. Learn- ing symmetric embeddings for equivariant world models. In International Conference on Machine Learning. pp. 17372-17389) PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Patrik Reizinger, Randall Balestriero, David Klindt, and Wieland Brendel. Position: An empirically grounded identifiability theory will accelerate self supervised learning research. In Forty-second International Conference on Machine Learning Position Paper Track, 2025. URL https:// openreview. net/ wtt.ffiddd-/oggpple111
Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal representations for covariate shift. In International Conference Learning Representations, 2022. URL https:://///ccc. pttttt::://peeeeviiii /opp/rveee..ee.eee httt/forrmiiidRRfSe.
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland Zimmermann, and Wieland Brendel. InfoNCE: Identifying the gap between theory and practice. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https://
openreview wet forum? Id==NQ2rxWhIs..
Dominik Schmidt and Mingi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024, URL uttsttps: http:///pppeeeeeeeeeeeeeeee/ www.eeew.ee...rreoeeeeeeee forum?? id=rvUg3cxpDF.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEF, 10965)6612-634 2021.
16

---

Published as a conference paper at ICLR 2026
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with prefiireeie representations. In International Conference Learning Representations, 2021. URL / http:///ppnneeeeeeneeeeeeeeeeeee norumu
(d=uCofPZwRaDli
Pierre Semmanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. contrcontrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Mchran Shakerinava, Amnb Kumar Mondal, and Siamak Ravanbakhsh. Structuring representations using group invariants. Advances in Neural Information Processing Systems, 35:34162-34174, 2022.
Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 14:12943-12254, 2021.
Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pietro As- tolfi, Kyunghyun Cho, and Yann LeCun. $1 m1 [X]S-sample contrastive loss: Improving contrastive learning with sample similarity graphs. In The Thirteenth International Confer- ence on Learning Representations, 2025a. URL https: http:///////eeeeee.eeeeee. net/forumiid c1Ng0f8ivn.
Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balestriero, Tim GJ Rudner, and Yann LeCun. Learning from reward-free free offline data: case for planning with latent dynamics models. arXiv preprint arXiv:2502. 14819, 2025b.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference on machine learning, pp. 843-852, PMLR, 2015.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances neural information process- ing systems, 15:10078-1009 2022.
Maria Mihaela Trusca, Wolf Nuyts, Jonathan Thomm, Robert Honig, Thomas Hofmann, Tinne Tuytelaars, and Marie-Francine Moens. Object-attribute binding in text-to-image generation: Evaluation and control. arXiv preprint arXiv:2404.13766 2024.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mu- tual information maximization for representation learning. International Conference on Learn- ing Representations, 2020. URL https: http::///peeeeeeeeeeeeee net/ forum? wawrrummidxxoo2244225
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12884-12893, 2021.
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. Advances in Neural Information Processing Systems, 36: 48603-48638. 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference Machine Learning, pp. 9929-9939. PMLR, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018
17

---

Published as a conference paper at ICLR 2026
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021. URL
attps::/cpenree/// ott.:///ew....//oouuo/////22222
net/ /ttp:ommmi---222222aa
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024. URL https:
Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. In The Twelfth International Conference Learning Representations, 2024.
URL https::///// wppp/ppeeeeiiieee / net forum? id=OGtnhKKJJs..
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying causal representation learning with the invariance principle. In The Thirteenth International Con- ference on Learning Representations, 2025. URL https: http///pppeeeeeeeeeee/ee/eeeeeee net forum?
id=1k20k5xjeu.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2023. URL https: /tppr//v/e//....w.. n /ertEoormm2 seiddddiin LdddKKKUUxuuuaa
Jeffrey M Zacks and Barbara Tversky. Event structure in perception and conception. Psychological bulletin, 127(1):3, 2001.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310- 12320. PMLR, 2021.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 13774 13784, 2024.
Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified thcoretical understanding of non-contrastive learning via rank differential mechanism. In The Eleventh International Con- ference on Learning Representations, 2023. URL https: http///ppneeeeeeeeeeeeeeeeoeeeem net forum?
[d=cIbjyd2Ver
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. International conference on machine learning, pp. 12990, PMLR, 2021.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
Limitations and future work
20
B Derivation of Eq. 8
20
Theory
20
Additional results and analysis
23
D.I Complex p(z z) on numerical data
23
D.2 Robustness noisy data pairings on iNat- IM
................................
24
D.3 Leveraging an additional view
25
D.4 A deeper look into AdaSSL-S's behavior
26
D.5 World modeling
27
D.6 Retrieval
28
E Data and implementation details
29
E.1
Implementation of Lo penalty
29
E.2 Numerical experiments in $4.2 and $D.1
29
E.3 3DIdent experiments in $4.3
31
E,4 CelebA experiments in $4.4
31
E.5 Video experiments in $4.4
33
E,6
M experiments in $D.2
34
19

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
In this section, we discuss the limitations of our work and suggest future directions.
First, while perform experiments on controlled settings with ResNet-18/50, it would be inter- esting to explore AdaSSL's behavior with VIT backbones and on uncurated data, such as web-scale image-caption pairs and natural videos. On such data, one can investigate the types of uncertainty (e.g., object-level versus global, kinematic versus intrinsic) that AdaSSL prioritizes to learn. Two recent works, while potentially incorporating stronger inductive bias, share similar conceptual motivation to AdaSSL on these specific settings (Lavoie et al., 2024; Hoang & Ren, 2025).
Second, the world model AdaSSL must reason about how the latent cause transforms the environment, i.e., p(z* r, task that naturally scales in difficulty with the complexity of the underlying transition. However, we hypothesize that operating the latent space makes this transition distribution considerably easier to model than in observation space. Future work could explore more structured transition models to better capture how latent factors drive world dynamics.
Third, natural extension of AdaSSL is action-five free representation learning vidcos, where the model discovers an implicit action space from naturally occurring changes. This learned action space can then support downstream tasks. For example, one may train decoder the frozen world model and prior over r for controllable video generation. We can also learn a projection between actions and the learned latent space enable planning (Sobal al., 2025b; Assran al., 2025). We perform a preliminary step of this idea where we evaluate the world model's 's capability predict future trajectories with given action $D.5.2.
Finally, we do not provide theoretical guarantees for identifiability and leave it as future work.
B DERIVATION OF EQ. 8
-1,; x)
-B, log q(1 X, x+)
p(r x)
= =EEp log q(1' x, x+)
+ log p(r x) - log p(r
p(r x)
E, log qI x, xx)
+ log p(r
x)
p(r x)
p(r
x)
p(r x)
= --E Ex(xxx)) (g(r( x, x [pp(r x)]] + E5
log p(r x)
= -Epxxx+) (qr( x, [1p((r [x)]] + $ p(x pxx)(xx x))((r x, log
5(r xx
drdxxt dx
p(r xx)
( x, x [pp(r [x)]] + / pos) (s p(xct xx)g(r xx x ddx
log p(r x)
= --E Ep(x,c))
drdx
p(r x)
( x, x [|p(r xx)]] + / p(x))(() xx) log
p(r x)
= --E Ep(x,ct)
drdx
p(r
x)
log p(r x)
Ex(x,x1)
(gr( x, x )|p(r| x))] + Ep(x)((rie)
p(r
x)
= Ep(xxt)) (r x, [pp(r xx)] + Ep(c)) DKk x)]lp(rr x))
> Ep(xxt)) (Kr( x, xxxx*)||p(X x)]]
C THEORY
Lemma C.1. Let A E Bmmx and let Σ E RR x be symmetric positive definite, Then
range( (EAT)) range(A).
20

---

Published as a conference paper at ICLR 2026
Proof. For any R",, have
" x (AEA (A (A E(A"x)) x:).
Since symmetric positive definite, the right-hand side is zero if and only if A x 0. Thus,
ker( (AEAT))A)))) ker(A)
Taking orthogonal complements yields
range (AEA)) range(A).
Remark. This is standard linear algebra fact; we include here for completeness.
Proposition C.1. Let S C Ig++ denote the k-dimensional unit sphere. Let Rd + R be C diffeomorphic to its image, and let f Rdd - sh be C almost everywhere. Define h == fog Rdd St Assume further that the random vectors 2, x E IRd are sampled as
z = 2 t, E Pe:
~ Pz.
where pz not a point mass and independent of E[ 0, and Cov(E) >
Suppose that for Pz az-almost every a we have h(z) E Sh and rank Dh(2) = d. Write H = h(2) and H+ h(2+). Then the conditional law
P [((2)),
is necessarily heteroscedastic: its conditional variance depends on l(z)) for pz-almost every z.
Proof. Fix 2 where h C¹ and rank Dh(2) For > small, define 2 2 with Pe- A first-order Taylor expansion and the delta method give
1(= + Øε) M(2) Dh(=) + o(())
which implies
[ov[h(=+or) =|| o² Dh(2)() Σ D= +
If the conditional covariance were homoscedastic leading order, there exists fixed positive semidefinite matrix C' such that
Dh(=) Σ Dh(=) C for pz-almost every z.
Let W == range(C). By Lemma C.1 and Σ > ( we have
range (Dh(z)) range (Dh())EDD(z)") range(C) W,
so range( D=())) = W the same d-dimensional subspace for pz- pz-almost every z. Because 1(2) E gk we have [|{()) 1 1, differentiating yields
h22) Dh(2) 0.
ie. rang)(DN(2)) C 4. Since range range(Dh(2)) = for almost every 2, we obtain W C h(2)* almost everywhere, hence h(2)) W for almost every z.
Pick any nonzero w E W. Then w h(2) = 0 for almost every 2, and differentiating gives w Dh(2) 0 for almost every i.e. 1 range (Dgge(DD())) W. Thus W C W... which forces W = {0}. This contradicts rank Dh(z) = d > O. Therefore the hypothesis that Dh(z)ED((z)) is constant is false, the leading-order conditional covariance must depend on for pz-almost every
Remark. The above argument establishes heteroscedasticity at leading order in the noise scale a,, which rigorously shows that the conditional covariance depends z for sufficiently small a. For larger a, oigher-order terms in the Taylor expansion of become significant and the exact condi- tional covariance may be more complicated; nevertheless, the local Jacobian Dh(z) still transforms the noise differently at different points, so the conditional variance remains intuitively location- dependent, even if simple closed-form expression exists.
21

---

Published as a conference paper at ICLR 2026
Proposition C.2 (Tangent-space variant of Proposition C.I). Let S R denote the k- dimensional wnit sphere, and U S open set. Let S + Sk be C¹ diffeomorphic to its image, and let f Ss' IR be C almost everywhere. Define h fo fog U - R.. We assume that is nondegenerate, i.e, h(U) is not contained in any proper affine subspace of its intrinsic dimension. Suppose that for almost every U, the derivative Dh(z) T2S* - R has full rank, Le. rank Dh(=)= = k. Assume further that the conditional distribution of = E S given z is locally Gaussian in the tangent space
p(= 12) x exp (- -) -(==+2)) -z))
with a constant positive definite diagonal matrix A.
Define H h(z) and H -1(2t). Then for generic nondegenerate C maps h, the conditional law
PH+In(h(277) | [h(2)),
is heterosccdasticc for almost every E U.
Proof. We construct = by a small Gaussian step in R++1 and normalization:
2 2+E
| ~~,((A
A first-order approximation for small gives
z+ - + O(1E))).
where P. = I- --22 22 is the projector to the tangent space, and the pushforward density on the sphere matches
p(=t 2) x exp (z+ - 2) A(=+ (+t=-))
up to higher-order terms.
Fix E U where h is C and rank Dh(z) has full rank. Euclidean Taylor expansion gives
h(2+) h(2) Dh(=)(=* - 2) + +(I++ =1133)
Substituting z+ - Rd P&e
h(=*)= = h(2) Dh(3));
where R(z) collects higher-order terms, and the leading-order conditional covariance is Cov(h(2+) 2) = Dh(2)E D + R(2), Σta = PPA-11A
with R(z) continuous and symmetric.
Suppose that Cov(h(=+) +)] 2) were constant across U. With Dtan > O, the range of the leading term Mang(DD(22) ttan () range (Dh(z)) would have to be the same subspace W C R for almost every 2 E U.
Eian
For any differentiable curve x(1) U through points where Dh(z(()) full rank, we can write
d
di h((((t) D)(((())((() w
Integrating along all such curves U gives
A(U) h((2)) + WW
for some base point 2p. This would imply that the image htU)) is contained in a fixed affine subspace
W Rd contradicting the nondegeneracy assumption on h. Therefore, constant pushforward covriance can only occur in the trivial case of no noise (Vtan = 0, or A 0) or in highly specific algebraic cancellation between Dh(2) and Stan For generic nondegenerate C maps h and almost every U, the conditional covariance is therefore heteroscedastic.
L
Remark. This is analogous to Proposition C.I, but with domain and codomain swapped; the argu- ment relies on the Jacobian of the map and the local Gaussian structure in the tangent space.
22

---

Published as a conference paper at ICLR 2026
Proposition C.3 (Extension of Proposition C.I). Letg: R. - R be C² with a local diffeomor--
phism and f Ed + M be C almost everywhere. Define h == f g R. + -M where M is a Riemannian manifold with strictly positive sectional curvature on a nonempty open set. Assume further that the random vectors 2, 2 Rd are sampled as
2 ~ P7, = 2 + g>pe: Pe:
where pz is not a point mass and s is independent of z, [E|s] = O, and Cov(e) > 0.
Suppose that for pz-almost every z have h(z) E .M and rank Dh(=) = d. Write H -b(z) and H+ = h(==). Then the conditional law
PH+)H(b(=7) [(2)),
is necessarily heteroscedastic: its conditional variance depends on h(2) for pz- pz-almost every z.
Proof. Following the same reasoning as in Theorem C.I, homoscedasticity at leading order would require a constant positive semidefinite matrix C' such that
Dh(2) E Dh
= C for pz plmattost every 2.
Since > 0, the above condition is equivalent to requiring that
(u, u Ev (DN(=)u, Dh(2)o) Vu, R for a.e. ie.. is a local Riemannian isometry from the flat space (R4, (s to the positively curved manifold (M, gM). However, local isometries preserve sectional curvature (Gauss' Theorema Egregium), so no such local isometry from an open subset of Rd to an open subset of M exists. Hence, the homoscedasticity condition cannot hold.
Therefore, for all sufficiently small a > 0, the conditional covariance
Cov/h(= + ar) =] o² D)()) E Dh(=)" + o(o²)
depends on 2, and the conditional distribution of h(2*t given h(z) necessarily heteroscedastic for pz-almost every
D ADDITIONAL RESULTS AND ANALYSIS
D.1 COMPLEX p(z
Z) ON NUMERICAL DATA
Table Linear regression R² on complex p(c* | c). All models normalize embed- dings and AdaSSL outperforms baselines.
In this experiment, we design DGP where p(c* c) both multimodal and heteroscedastic. We hy- pothesize that natural pairs usually differ sparsely in
c, and the differed factors are sometimes conditioned on latent variable. Therefore, we randomly select some dimensions of c and c to be shared, while the rest follow Gaussians conditioned on latent vari- able K. i.e., c;, ct k ~ N (()))) See $E.2 for details.
Model
piz)
A I)
A'0,5 -oon
InfoNCE
same
caae
AnlnfoNCE:
0.5446 20nss
0.5578
0.1652 2
H-InfNNEEaaa
0.8750
0.7781
0.5471
AdaSSL-V
AdaSSI
0.9187 2007
0,8472
0.6325
Table 4 shows that InfoNCE and AnInfoNCE struggle to recover the latent factors, especially on the hardest OOD evaluation, (O, I)oOD. H-InfoNCE improves performance, and AdaSSL variants improve further.
To understand why AdaSSL outperforms baselines in Table 4, visualize the aggregated marginal distribution of z implied by the learned predictor, Ez [z(pmodel (z+ z)]. We define one Monte- z) for each as follows. For InfoNCE, we first encode the input X = g(z) and then learn a projection from the embedding space to the ground-truth latent space by training linear regressor from [z)(z) to zt. For H-InfoNCE, we pass f(g(z) through the predictor and project the predicted representations. For AdaSSL-V, we sample from the learned prior f ~ Po(r (r x) and use i to edit the embeddings with (x)(xx, F) and project the edited embeddings. We repeat this process for random samples of z p(z) and visualize them in Fig. 5. InfoNCE embeddings produce overly concentrated densities, indicating their inability to accurately capture complex conditional uncertainties. H-InfoNCE partially corrects this, while AdaSSL best fits the ground- -ruth distribution, suggesting that its improvement arises from more accurate modeling of the conditional uncertainty.
Carlo sample of Pmodel(z"
23

---

Published as a conference paper at ICLR 2026
InfoNCE
H-InfoNCE
AdaSSL-V
(Ours)
Ground Truth
Pairs of latent dimensions
Figure 5: Aggregated marginal distributions Ey (Pmodel (z*
z)] across latent dimension pairs. In- foNCE produces collapsed densities and H-InfoNCE partially recovers variability, while AdaSSL-V aligns closely with the ground truth. The improvement most evident columns two and three, where AdaSSL-V captures both spread and orientation while baselines do not.
Figure 6: Visualization of images paired by class label from the iNat- MM dataset.
Figure 7: Visualization of images paired by superclass label from the iNat- 1M dataset.
D.2 ROBUSTNESS TO NOISY DATA PATRINGS ON INAT- 1 M
In this experiment, we train InfoNCE and AdaSSL-V with ResNet-50 backbone at 224x224 reso- lution on subset of iNat2021 (Van Horn et al., 2021), a large-scale image dataset that contains fine- grained (class) and boareegraiined (superclass) species labels. We call this subset iNat- which contains one million training and 250 000 validation images across 5000 fine- fine-grained classes and 11 superclasses. Fine-grained classification on iNat- 1M serves proxy task evaluate whether the model learns diverse features.
Since the goal of AdaSSL is more robust to noisy data pairings than vanilla SSL. The ideal setting is one where we have perfect class labels and we can pair up images within the same class (Fig. 6). To control the amount noise in the pairings, we corrupt subset of labels such that these corrupted data is paired with another image from the same super class instead (Fig. 7). This simulates real-world label noise, which is wolf
learn diverse features under target uncertainty, we hypothesize that
rarely random, but instead structured; it is much more likely for dog to be mislabeled than as an avocado. We gradually increase the percentage of data that are corrupted, and investigate how it affects the performance of online linear probes trained top of the representations.
24

---

Published as a conference paper at ICLR 2026
Top-1, 5000-Class
Top-5, 5000-Class
Top-1, 11 -1-Superclass
60
96
80
Accuracy [%]]
95
40-
60
94
InfoNCE
InfoNCE
InfoNCE
20
AdaSSL-V
AdaSSL-V
AdaSSL-V
25
50
75
100
0
25
50
75
100
0
25
50
75
100
Corrupted Labels [%]
Figure 8: Results MM. We gradually increase the percentage of pairings that corrupted. Performance of online linear probes on the validation set is reported. Left: top-1 accuracy across 5000 classes. Middle: top-5 accuracy across 5000 classes. Right: top-1 accuracy across 11 super- classes. AdaSSL-V's performance decays more gracefully than InfoNCE, showing its robustness to noisy pairings.
Table 5: Ablation of the additional view x+* on CelebA, trained with natural pairs under weak or strong augmentations. Linear F1 scores representations (encoder output) and embeddings (projector output) are reported. AdaSSL trained without ++ outperforms the baseline, and using
x x++ improves it further.
Model
Pairing
WEAK AUGMENTATION
STRONG AUGMENTATION
Repr.
Emb.
Repr.
Emb.
InfoNCE AdaSSL-S
Natural
0.5473
000007
0.3747
000551
0.5784
0.4941
á 0,0015
0,5676
4: 0,0009
0,4581
000016
0.5911
100114
0.5654
: 0,0007
Lwithout x++
0.5505
+ 0.0055
0.4466
0.0026
0.5716
0.0466
0.5149
0.005
AdaSSL-V
0.5784
1 0.0055
0.4794
0.0015
0.6014
0.006
0.5706
0,004
-without x++
0.5550
0,0057
0.4351
0.0042
0.5869
0.0011
0,5117
0.0014
Fig. 8 shows that both InfoNCE and A suffers decay in fine-grained classification accuracy when we increase the corruption rate. However, AdaSSL-VV's accuracy decays more gracefully. We also observe its advantage to be more visible when the corruption rate passes 50%. Interestingly, AdaSSL-V slightly improves InfoNCE even when we do not corrupt any labels. We hypothesize that this could be due to the potential label noise that already exist in the dataset as well as the unavoidable heteroscedastic noise (Proposition 2.1). We observe a slight drop in the superclass classification accuracy and not observe the gap between AdaSSL and InfoNCE to change much across corruption rates. This is expected because we always pair up data within the same superclass.
D.3 LEVERAGING AN ADDITIONAL VIEW
Chen et al. (2020a) show that we need strong augmentations create differences between views in their low-level image statistics. Otherwise, encoding these statistics alone is sufficient to satisfy the SSL objective shortcut solution (e.g., standard pairing with weak augmentations in Table 3).
For both AdaSSL-V and AdaSSL-S, the model is expected to learn the factors explaining the dif- ferences in the paired views. Allowing r to encode any differences facilitates this shortcut solution. Furthermore, applying pixel-level augmentations may exacerbate this issue by increasing the incen- tive to learn these discriminative, yet less semantic transformations.
One way to ensure learns the desired transformations while remaining invariant to others is to use a surrogate view x relationship with x in the underlying content factors c and c++ mimics that between and e+ but differ in low-level statistics-to replace x+. In other words, AdaSSL-V uses r sampled from (r f(x) f))++) and AdaSSL-S uses r predicted by m(f(x), f(x++((xxx)) These additional views usually easy to obtain, e.g., by applying the same random augmentations that created the differences between x and x to x We detail the + that we in each experiment below.
25

---

Published as a conference paper at ICLR 2026
B-0.1
6-0.5
Active Latents
0.6
-4
Entropy
0.4
0.2
0.0
0K
20K
40K Step
60K
80K
0K
20K
40K Step
60K
80K
0K
20K
40K
60K
80K
Step
Figure 9: Utilization of and performance on CelebA throughout training when AdaSSL-S is ap- plied to InfoNCE. Left: the number of active latent dimensions of r. Middle: the entropy of the probability vector with which the model picks which latent of r to activate. Right: training F1 score of online linear probe representations.
8-0.001
B-0.002
8-0.1
Active Latents
0.6
Entropy
0.4
0.2
#00
0.0
0K
20K
40K 60K Step
8OK
0K
20K
40K Step
60K
80K
0K
20K
40K
60K
BOK
Step
Figure 10: Utilization of and performance on CelebA throughout training when AdaSSL-S is applied to BYOL. Left: the number of active latent dimensions of r. Middle: the entropy of the probability vector with which the model picks which latent of to activate. Right: training Fj score of an online linear probe representations.
Notably, this additional view is not always required. When low-level statistics alone are insufficient for instance discrimination, the model is incentivized to learn other features. This is demonstrated in our CRL experiments in $4.3, where we learn to recover all data-generating factors from natural pairs without augmentations x++
We perform ablation studies of x +++ in Table 5 and observe consistent benefits.
D.4 A DEEPER LOOK INTO ADASSL-ST BEHAVIOR
Both AdaSSI-V and AdaSSL-S improves InfoNCE natural pairs, but only AdaSSI--V consis- tently improves BYOL. We hypothesize that explicitly regularizing the embeddings' information content (as in the denominator of Eq. pressures the model utilize effectively. In contrast, distillation-basee methods lack this direct tension. Since AdaSSL-S only regularizes the sparsity, it may require additional regularization on that encourages efficient utilization.
In Fig. and Fig. 10, we show statistics of r throughout training on CelebA with natural pairs and strong augmentations, for AdaSSL-S trained with InfoNCE and BYOL, respectively. For meaningful comparison, we fix d 20 and plot the curves for different values of B.. We show three metries. First, we show the sparsity of latent usage and plot average number of active dimensions of across the batch, i.c., ||]]. Second, show the spread of usage across dimensions. To do so, define latent variable p in the d,-dimensional probability simplex as the probability with which the model picks each dimension of activate within each batch. We then plot H Hp)), the Shannon entropy of p. Third, we plot the F; trace on the training set. We expect a successful model to activate few dimensions of each time, but utilize the entire space across different data pairs.
As shown in Fig. 9 and Fig. 10, the strength of regularization controls how many latents are ac- tive for both InfoNCE and BYOL. However, BYOL+ AdaSSL-S AdSS-S fails to maintain a high spread of
26

---

Published as a conference paper at ICLR 2026
Table Performance of linear probes trained on frozen representations and embeddings on stochas- tic Goving-MNIST Evaluation is performed on the online branch of BYOL.
SETTING A
SETTING B
Model
Represenfations
Embeddings
Representations
Embeddings
Acc. [S] Velocity [
Acc. [%:
Velocity [/t-)
Acc. [8]
Velocity [/(]]
Acc. [%]]
Velocity [129]
BYOL
6:3
a:4
wocs
91.00
00657
0.1486
BYOL+Future
88.31 134
0.9005
0.300
78.68
0.5890
woze
88.33 um
0.8996
78.99
0.6041
ca00
BYOL+GT
6:24
4.30
-0.0038
1.3c
93.55 4.30
00652
AdaSSL-V
94.18 om
0.8951
0:33
90.54
0.2867
soms
94,17 a.1.9
0.8961
10025
90.34
0.2875
18215
AdoSSL-V
0.9168
91.28
0.8695
94.31
0.9188
92.32
0.8594
AdaSSI -
91.89 2.11
0.9121
2:3000
86.00
a
0.8901
2000
91.95
0.9121
85.53
0.8750
information across r's latent dimensions, as reflected by a lower Hp)). We also observe higher instabilities when training AdaSSL-S with BYOL than with InfoNCE. the latter case, we observe that the model's performance is robust to different B.
We hypothesize that A works well with both contrastive and distillation-based SSL because minimizing the KL divergence enforces non-zero variance, which promotes utilization, while using a factorized prior discourages redundancy. Together, the KL regularization encourages learning an efficient representations of as we discussed in the disentanglement analysis in Sec. 4.3.
Our findings corroborates the motivation of designing additional regularization terms for sparsity regularization in concurrent work (Garrido et al., 2026), where the authors learn world model with latent variable regularization on frozen representations.
D.5 WORLD MODELING
D.5.1 NUMERICAL RESULTS FOR FIG. 4
We provide full evaluation results on stochastic Moving-MNIST in Table 6. These results further demonstrate AdaSSL's effectiveness in achieving strong performance in both digit recognition and velocity decoding. Our results and ablations in the main text in Fig. 4 uses Setting A because we do find significant difference between the results.
D.5.2 FUTURE TRAJECTORY PREDICTION
Given source (initial) trajectory and its ground-truth latent (change in velocity), we predict the emhedding of the corresponding target (future) trajectory with the learned predictors of B YOL and AdaSSL-V/S and retrieve the nearest neighbors from a pool of 4096 target trajectory embeddings. To generate this pool, we randomly sample 64 initial trajectories from the validation set and generate 64 future trajectories for each of them. To map the ground-trute -trth r space to the r space learned by AdaSSL-S/V, we train projection (an MLP with one hidden layer of dimension 128) by minimizing the prediction error on the training set using the frozen encoder and predictor 7).
Table 7 reports mean reciprocal rank (MRR) and hit rate at k (Hir@ k) which are standard metrics for assessing predictor quality (Kipf et al., 2020; Park et al., 2022; Garrido et al., 2023b; Gupta al., 2024). AdaSSI. consistently outperforms BYOL. Fig. 11 shows the retrieved trajectories and the rank of the ground- tround-truth target in the retrievals. We observe that all methods are able to retrieve the correct digit, while AdaSSL-V and AdaSSL-S predict the direction and velocity of the ground-truth target better than BYO..
D.5.3 FUTURE TRAJECTORY SAMPLING
In this section, we evaluate the diversity of the predicted future trajectories on Stochastic Moving- MNIST. We first sample random source trajectory from the validation set and 64 future tra- jectories from the ground-truth p(x* x). We encode the source and targets using BYOL and AdaSSL-V. Next,
retrieve the nearest neighbor of their predictions and visualize them in Fig. 12. For AdaSSL-V, V, we use the learned prior to sample multiple r's and condition the predictor t to predict 10 target embeddings. We then retrieve the nearest neighbors of these targets.
27

---

Published as a conference paper at ICLR 2026
Target
BYOL
AdaSSL-V AdaSSL-S
Context
Target
BYOL
AdaSSL-V
AdaSSL-S
ranled?
rank-1
runk-
rank-59
rank-4
rank-2
rank-13
ranke1
rank-3
rank-522
rank-1
rank-3
5
fif
10 11I
rank-41
rank-11
ranlens
ramk-19
rank-13
rank-5
rank-59
rank-1
runk-2
rank-16
rank
rank-3
Future prediction visualization. Given a source trajectory and its ground-truth latent (change in velocity), we predict the embedding of the corresponding future trajectory with the learned predictors of BYOL and AdaSSL-VE and retrieve the three nearest matches from a pool of 4096 future trajectory embeddings. All methods predict the correct digit, while AdaSSL-V V and AdaSSL-S predict the directional velocity of the ground-truth target better than B BOL.. Retrieval ranks (4) for the ground-truth target reported below each visualization.
Figure 11:
Table 7: Quantitative retrieval results on Stochastic Moving-MNIST. We encode the first three- frame segments of randomly sampled videos and use the learned predictors BYOL and AdaSSL- S/V to predict one sampled future in the embedding space. We then retrieve the best matched future trajectory from a pool of 4096 video segments (consisting of 64 randomly sampled future trajectories of 64 random initial segments), including the correct segment. We report the mean recipricol rank (MRR) and hit rate at & (Hit@k) k)) for the retrievals.
Model
MRR (1)
Hit@1 (t)
Hit@5 (1)
BYOL
0.1758
0.1182
0.1802
AdaSSL-V
0,4102
0.3071
0.5129
AdaSSL-S
0,5291
0.4338
0.6287
Fig. 12 shows the overlaid ground-truth future trajectories (left) and predictions by BYOL and AdaSSL-V. BYOL produces single deterministie prediction, whereas the AdaSSL-V predicts di- verse plausible futures.
D.6 RETRIEVAL
In Fig. 13 (left), we perform standard retrieval to accompany our analysis in $4.3. We retrieve the five nearest neighbors of the query image in the embedding space. We observe that both AdaSSL and the baselines are able to retrieve visually similar images. There are still some wrong retrievals in color and spotlight, and rotation is especially hard to learn for all methods.
28

---

Published as a conference paper at ICLR 2026
Context
GT
BYOL
AdaSSL-V
Context
GT
BYOL.
AdaSSI.-V
Context
GT
BYOL
AdaSSL-V
Figure 12: Sampling future trajectories. Given source trajectory, visualize the nearest neigh- bor (via cosine similarity of the embeddings) of the predictions of BYOL (middle) and AdaSSL-V (right). Samples from the ground-truth (GT) transition distribution is shown the left. AdaSSL-V samples multiple future trajectories from the learned prior while BYOL produces a deterministic prediction.
I-NN of (f(x), by AdaSSL-V
F~
5-NNs fx)
polr x, vury position)
InfoNCE
por x, vary spollight)
AdaSSL-V (Ours)
Por x, vary hue)
AdaSSL-S (Ours)
Pof x)
Figure 13: Image retrieval results on 3DIdent. Top left: query image. Bottom left: five nearest.
neighbors on the embeddings. Right: controllable retrieval by AdaSSL-V.
E DATA AND IMPLEMENTATION DETAILS
E. IMPLEMENTATION OF PENALTY
For AdaSSL-S, we implement the sparsity regularization following Lachapelle et al. (2022); Brouil- lard al. (2020). Specifically, we implement r as m i, where m; Bern(T;; is a binary mask and i is the value of the latent. Both the Bernoulli logits and the vector i are predicted by the MLP mx)(x); (x+)) or mx)(x), ((++)))
We encourage sparsity by penalizing the expected number of active gates:
[o]
Σ m;
Σ E [m:]
=
Σ ni
(12)
which is differentiable w.r.t. π.
Next, we sample the binary mask using the Gumbel-Sigmoid reparameterization of the Bernoulli distribution (Maddison et al., 2017). enable backpropagation through this discrete decision, we straight-through estimator, which backpropagates through the relaxed gates in the backward pass (Bengio et al., 2013b).
E.2 NUMERICAL EXPERIMENTS IN $4.2 AND $
In the numerical experiments, most of our setup follows prior work (Kügelgen et al., 2021; Zimmer- mann et al., 2021). We list the similarity functions used by the models in Table 8.
Complex p(c c), formally stated.
NO,EE,, e; N(r(t)),
(13)
29

---

Published as a conference paper at ICLR 2026
Table 8: Similarity functions used by different models, where v0-) normalized latent space, in which case InfoNCE and AdaSSL's similarity functions are equivalent to a dot product; otherwise (h(-) f(-). The same applies to Vi and eis, whose subscripts are used to
T if the model assumes
indicate the asymmetry of H-InfoNCE and AdaSSL. Note that in Table 1, H-InfoNCE has 4/1 = 1/2
because E/ct |c] c.
Model
s(x,y)
InfoNCE
-X(((x) (())) (x(x) (())
AnlnfoNCE
-(V(x) x(y)) ((e(x) (y))
H-InfoNCE
-(V1(x) +22())) Ax (x)(()) 22()))
AdaSSL
X(11(xx,F) +y())) (th (x, x22(8))
4i ~~ (e))( of 14,0 K lo(c [ = (N(u(()),
=i),
4 = 0
(14)
=
(ne + 2,1). For anisotropic noise, we sample (c)) InVGamma(2, 1). For heteroscedastic noise, we set ac)) softplus W, + softplus (1)). For complex p(ct c). we use μ(k) W + b, (xx)
Data. We set nee = = 5 and sample
softplus (W, softplus (1)). and TT(K)) = Sigmoid K. - -1) We sample each element of
W,, W.. and b from (O, 1). We parameterize gMLP as three-layer MIPP with LeakyReT.U ac- tivation (negative slope 0.2) with the same number of units in all layers. We ensure invertibility by using L normaizzee weight matrices that has the lowest condition number among 25 000 uniformly sampled candidates. We use x [MLP([CC s++]) where c is the same content factor in x and s ++ IO.
Architecture. For the encoder f, we use an MLP with four hidden layers of dimensionality 10m where n = n n, is the input dimension. For models that apply L normalization the outputs, we set the output dimensionality to n + to accommodate for the missing degree of freedom; otherwise we set to n. For H-InfoNCEAmtee we use an affine layer followed by so ftplus activation to predict Ax. For H-InfoNCEMILP we use MIP with three hidden layers of size 10n followed by softplus activation to predict Ax and an MLP of the same size to predict di (x) in Table For AdaSSL, we set d, 5. We use MLPs with two hidden layers of dimension 64 to parameterize 4á- Pg. and m and use a linear t for AdaSSL-V V. All MLPs except the encoder use a Bat chNo rm layer followed by LeakyReLU with the default negative slope (0.01) after each hidden layer.
Hyperparameters. We use the Adaml optimizer (Loshchilov & Hutter, 2019) with learning rate 5 x 10-4 and weight decay 10-4 on the parameters except biases. We use batch size of 2048. For the experiments on complex p(c+ c), we apply the loss symmetrically similar to Chen et al. (2020a) because the sampling process of and c+ is symmetric. We train the models for 200 000 steps and observe convergence. For AdaSSL-V, V, we linearly warmup 3 from 0 to 0.5 for 1000 steps to prevent early KL instabilities. We keep 3 1 fixed throughout training for AdaSSL-S. For the unimodal p(c c) experiments, we set = E o (c)] = 1 except when the variance is fixed to 0, in which case set 0.1. For the complex p(c* c) experiments, set 0.1.
Evaluation. We perform evaluation by training linear regressor on top of the frozen representa- tions on 000 unseen data samples and evaluate it on another 100 000 samples.
Hardware. Each trial of this experiment required approximately 15-20 hours to using eight CPU cores, 4 GB of system memory, and an MIG-partitioned slice of an NVIDIA H100 GPU providing roughly quarter of the GPU's compute capacity and 20 GB of GPU memory.
30

---

Published as a conference paper at ICLR 2026
E.3 3DIDENT EXPERIMENTS IN $4.3
Data. 3DIdent contains 000 training images in Derain and 25 000 test images in Drests which use for CRL experimenss. We sample latent pairs (z, 2 following
2 ~ p(z), 2 ~ p(2), 4 Bern(0.2), ,,- = 74 if 4 else ,,- = 7, for E [dz]-
(15)
Since 3DIdent is finite dataset, after obtaining latent pair, we find their nearest neighbor in the training set with FAISS (Douze al., 2024) and use the correspondingly rendered observations as inputs following the original authors (Zimmermann et al., 2021). AdaSSL does not use an additional view this experiment.
Data augmentations. For standard pairs, we use the same set of strong augmentations used for CelebA ($E.4). For natural pairs, we do not perform augmentations. We resize the images to 128 x 128 resolution.
Architecture. We use a ResNet-18 18 encoder followed by a two layer MLP projector with hidden size of 128 and output size of 16 as f. The hidden layer is followed by ReLU activation without Bat Barch, and the output layer does not have bias. For AdaSSL, we set d, = 16. We use MLPs with two hidden layers of dimension 128 to parameterize 9á. P8, and m. These MLPs use a BatchNore layer followed by ReLU activation after each hidden layer. As discussed in $4.3, ablate the parameterization of & for AdaSSL-V; the MLP parameterization has hidden layer of dimensionality 128 with BatchNo followed by ReLU activations. The VAE-based methods use R8 decoder that mirror the encoder.
Hyperparameters. We use the Adamw optimizer with learning rate 10-1, weight decay 10-5 non-bias parameters, and batch size of 256. For contrastive learning, calculate the loss symmetrically following standard practice (Chen et al., 2020a). We train all models for 150 steps and observe convergence on Durain All SSL methods use normalized embedding space, use = 0.05, and do not learn À in this experiment. For AdaSSL-V, we perform linear warmup of B from to 0.5 for 10000 steps to prevent early KL instabilities. For AdaSSL-S, we fix 0.5. For AdaGVAE, we search within the authors' recommended set of B's, [1, 2, 4, 8, 16], but find 3 = 100 to give the best disentanglement.
Evaluation. We perform evaluation on Drest with the frozen embeddings and ground-truth latent factors with linear regression and the DCI disentanglement score. We normalize the embeddings for the SSL based models such that they align with the training objective, similar to Zimmermann et al. (2021). We use the posterior mean as the embeddings for VAE-based models and do not normalize them. For the DCI disentanglement score, importance matrix.
the weights of Lasso regressors the relative
Hardware. Each trial of this experiment required approximately 15-20 hours to run, using eight CPU cores, 32 GB of system memory, and an MIG-partitioned slice of an NVIDIA H100 GPU providing roughly three-eighths of the GPU's compute capacity and 40 GB of GPU memory.
E.4 CELEBA EXPERIMENTS IN $4.4
Data. We split the CelebA dataset into Dirain- Dval. and Drest following an 8-1-1 ratio; this gives us 161 908 training images, 20 346 images in the validation set and 20 345 images in the test set. To create a natural distribution shift, we sample celebrity identity such that the people in Demai does not appear in vaal U D This gives 8 8142 celebrities in Dursin and 2035 celebrities in Dval U Drest- To construct a structured positive pair, we randomly sample two images of the same person. This results in 850 possible positive pairs. Data pairs examples are visualized in Fig. 14 and the distribution of the number of differed attributes between pairs are shown in Fig. 15, confirming that attributes differ sparsely between positive pairs. During training. we augment the sampled pair using data augmentations and obtain x and x We use another augmented view of x as ++ This is helpful because our goal is not to learn the low-level style factors, but instead the semantic content factors that differ structurally between x and x. The standard pairing process still use augmented versions of the same image as positive pairs.
31

---

Published as a conference paper at ICLR 2026
Figure 14: Visualization of images paired by identity from the CelebA dataset.
Data augmentations. We investigate the effect of both strong and weak augmentations. For strong augmenta- tions, we apply the standard set of augmentations used in SSL studies (Chen et al., 2020a; Grill et al., 2020). We use RandomHori zontalFli
6.38
0.12
Density
0.08
with 0.5 probability, then RandomRes zedCrop with crops of size within (8%, 100%] of the original image and aspect ratio within (0.75, 1.33]. which are then resized to 64 x 64. Next, with probability 0.8, we randomly apply ColorJitter where the brightness, contrast, saturation and hue of the image are shifted by a uniformly random offset. We parameters 0.4, 0.4, 0.1, respectively. Fi- nally, we apply RandomGrayScal ity 0.2, GaussianBlur with probability 0.5, and colarization with probability 0.2. For weak aug- mentations, we only apply Randomfforizontall with probability of 0.5 and RandomRes LandoRResizedropp zedCrop with crops of size within [80%, 100%] of the original image and aspect ratio within [0.9, 1.1]. Notice that this cropping operation significantly weaker than the one used for strong augmentations.
0.04
0.00
5
10
15
20
# of differed attributes between pairs
with probabil-
Figure 15: Distribution of the number of differed attributes between pairs of im- ages of the same identity.
Architecture. We use ResNet-18 encoder (He et al., 2016) followed by a two layer MLP projec- tor with hidden size of 1024 and output size of 128 as f. For InfoNCE, the hidden layer is followed by ReLJ activation without Bat chNorm, and the output layer does not have bias following Chen et al. (2020a). For BYOL, the hidden layer uses Bat chNorm followed by ReLU activation, and the output layer does not have bias following Grill et al. (2020). BYOL's predictor design is the same its projector. For AdaSSL, 1024 to parameterize q6, p9, and m. We use an MLP with one hidden layer of dimension 512 to parameterize for AdaSSL-V; this MLP does not have bias term in the output layer, similar the predictor in BYOL. These MLPs use a Bat chNorm layer followed by ReLU activation after each hidden layer.
set d, 20. We use MLPs with one hidden layer of dimension
Hyperparameters. We use the AdanN optimizer with learning rate 2 x 10 and weight decay 10-4 on the parameters except biases. We use a batch size of 512. We calculate the SSL. loss sym- metrically following standard practice (Chen et al., 2020a). We train the models for 80 000 steps and observe convergence on D For contrastive learning, all models use a normalized embedding space and use 7 = 0.1. The following are the hyperparameters of different methods used in combi- nation with InfoNCE. For AdaSSL-V, V, we perform linear warmup of 3 from to 0.1 for 10 000 steps fix B 0.5. For LieSSL, best- performing coefficient of the last regularization term from [0.01, 0.1, 1, 10] because we do not assume access transformation labels, and choose the berf-pero number of basis functions
to prevent early KL instabilities. For AdaSSL-S,
search for the
32

---

Published as a conference paper at ICLR 2026
from [1, 10, 20, 40]. For BYOL, anneal the EMA momentum from 0.996 following cosine schedule. With BYOL, AdaSSL-V uses a linear warmup of 3 from 0 to 0.005 for 10000 steps for weak augmentations and to 0.001 for strong augmentations. For AdaSSL-S, we fix 0.002 for weak augmentations and B = 0.0005 for strong augmentations.
Evaluation. Following standard practice, we train a linear classifier with the BinaryCrossEntrop loss for each attribute on top of the frozen representations and embeddings on Dirain until convergence and evaluate on Drest- We use the Fi score of the minority class the evaluation metric because the attributes are highly imbalanced. To do that, compute the F1 score for each attribute then report the mean score over attributes.
Hardware. Each trial of this experiment required approximately 15-20 hours to run, using 12 CPU cores, 24 GB of system memory, and NVIDIA L40S GPU with 48 GB of GPU memory.
E.5 VIDEO EXPERIMENTS $4.4
Data. We construct custom dataset similar to Moving- MNIIST (Srivastava al., 2015; Drozdov et al., 2024), where nine-frame videos are generated stochastically on the fly from sample images in MNIST. For given image, first create black 64 x 64 canvas. Afterwards, we resize the original 28 x 28 image to 16 x 16 and place it on the canvas after uniformly sampling its initial center coordinates from [8, 16]. In frames 1-3, the digit moves from this center based on velocity in the horizontal direction, denoted by Vx.1:3, and in the vertical direction, denoted by y We sample these initial velocities uniformly from [0, uyl where uh = 3. Then, with an equal probability, we sample one direction and change its velocity by adding a Gaussian noise proportional to the initial velocity (i.e., heteroscedastic):
U2,4:0 ~ (V.,,,13 (a,,1a) U1,4:9 1:, 0 {
(055,
(16)
U2, N( (1, (2,11))) Un,4:9
U2,1:3 1:3
1
This makes the new velocity in frame 4-9 within (-vo, 300) with high probability. Generated video samples are shown in Figure 16. We refer to this as Setting A.
In Setting B, we let depend on the digit input. Concretely, we use equally spaced bins between 0. and 0.9 for the ten digits:
0.9 - 0.1
4 Bern(ps), where Pk 0.1
0.......
(17)
10
This means the distribution of the direction of acceleration varies for different digits.
We partition each sampled video into frree-frame segments and use them x, x xnd (SD.3). The model predicts f(x+) from f(x) (and optionally f(x++) by AdaSSL and BYOL+Future). The goal is to capture both the digit class and the velocity in the three-frame video representations. We partition the 60 000 MNIST images into 50 000 training images and 10000 validation images and use each set for generating training and validation videos on the fly. Note that we always sample the velocities online, and the model observes different videos in every epoch.
Architecture. The encoder f consists of a 3D convolutional encoder, followed by an MLP projec- The 3D convolutional encoder consists of five convolutional layers with [32, 64, 1 128, 128, 256] channels with Batchhorm and ReLU activations after each layer. The first two and the last layer have spatial-only kernels of dimensions [1, 3, 3] and the third and fourth layers have temporal con- volutions with kernels of dimensions [3, 1, 1]. The encoder outputs are average-pooke on the spatial dimensions and then flattened across the temporal dimension resulting in d68-nimnal repre- sentation. The representations are passed to an MLP projector with two hidden layers of size 1024, each followed by Bat rm: and ReLU activations. The output embeddings have a dimension- ality of 128, and are batch-normalized. The projector is followed by an MLP predictor /t with two hidden layers of dimensionality 1024 with chNorm and ReLU activations after each hidden layer. The predictor output does not BatchNore dimensional r, which is concatenated to fxx as the predictor input. We use MLPs with one hidden layer of dimensionality 1024 to parameterize qs. Po. and m. These ML.ss Bat Patchhormm rm layer
ReLU. For AdaSSL-V,
two-
33

---

Published as a conference paper at ICLR 2026
333333
coll
Random samples (nine-frame video sequences) from the stochastic Moving-MNIST dataset. For each example, the first three frames (context) are shown on the left. Then, three different future trajectories of the next six frames (targets) are randomly sampled according to Eq. 16 and visualized the right of the initial three-frame segment. The third frame is overlaid on all canvases for reference. The motion uncertainty arises from random velocity changes along spatial directions.
Figure 16:
followed by ReLU activation after each hidden layer. For BYOL+Future, we concatenate the pro- jector embeddings f(x) and f(x=+) and it the predictor input. BYOL+GT predicts f(xt) from (x) and r*, the ground-truth difference between the velocities of x and x We experiment with concatenating r* directly with fxx passing it through learnable linear embedding before concatenation, and find that using an embedding layer slightly improves performance.
Hyperparameters. For methods, we train the model for 75 000 steps with the AdamW optimizer using a batch size of 128. We use an initial learning rate of 10-4 and decay it following cosine schedule, following Grill et al. (2020). We use a constant weight decay of 10-4 For the EMA momentum, we use a constant decay rate of 0.996. In BYOL+GT, we learn an affine projection to create an embedding for r* of dimensionality 32. For all AdaSSL models, we use a constant regularization coefficient 8, and in our default setting, d, = 2 and B 0.001.
Evaluation. To perform evaluation, we train linear probes with CrossEnt ropy (for digit clas- sification) and MSE (for velocity regression) losses top of the frozen video representations and embeddings of the online branch on Derain until convergence. We then report the digit prediction accuracy and velocity decoding R² scores on fixed video test set generated from the 10000 test images of MNIST.
Hardware. Each trial of this experiment required approximately 6-8 hours run, using six CPU cores, 32 GB of system memory, and an NVIDIA H100 GPU with 80 GB of GPU memory.
E.6
INAT- 1M EXPERIMENTS IN $D.2
Data. iNat2021 (Van Horn et al., 2021) is a large-scale image dataset collected and annotated by community scientists that contains fine-grained (class) and coarse-grained (superclass) species labels. We create a balanced subset from iNat2021 about half of its size. We first randomly sample 5000 classes that have at least 250 images from iNat2021's training set. We then randomly pick 200 images as D Derain and 50 as D This results a one-million millionnimmaee training set and validation set of 250 000 images. We name this subset of iNat2021 as iNat- M. During training.
34

---

Published as a conference paper at ICLR 2026
investigate the model's behavior under different natural pairings. We start with setting where we pair up each image with an image from the same class, randomly sampled at each step. Since evaluate the model's performance contrastive learning (Khosla et al., 2020) because the model is encouraged to cluster images within the same class together. For the rest of the settings, randomly corrupt the pairing process of each image with probability w E {0.25, 0.5, 0.75, 1}. Our corruption changes the pairing method for selected images pairing within the same superclass. We expect AdaSSL to adapt different corruption ratios better than baselines because it's less restricted on the uncertainty in features that could change under coarse/corrupted pairing. We augment paired data using data augmentations and obtain and x for all methods. We use another augmented view of x x
fine-gratn fin-grainedd classes, this setting is similar to supervised
Data augmentations. We resize the images to 224 224 resolution and use the same set of strong augmentations used for CelebA ($E.4).
Architecture. We use ResNet-50 encoder (He et al., 2016) followed by a two layer MLP projec- tor with hidden size of 1024 and output size of 128 The hidden layer is followed by ReLU acti- vation without Ba tottchh chho rm, and the output layer does not have bias following Chen et al. (2020a). For AdaSSL, we set d,- 8. We MLPs with hidden layer of dimension 1024 to parameterize Pe. and t. t does not have bias term in the output layer. These MLPs use a BatchNoo chrorm layer followed by ReLU activation after each hidden layer.
Hyperparameters. We use the AdamN optimizer with learning rate 2 x 10 and weight decay 10 on the parameters except biases. We calculate the SSL loss symmetrically. We use batch size of 256 and train the models for 200 000 steps. For contrastive learning, all models use normalized embedding space and use т 0.1. For AdaSSL. we perform linear warmup of / from Pänal for 10 000 steps. We search 8rnal from {0.2, 0.4, 0.6, 0.8) and find Bnnal 0.8 performs the best for all settings except on the noisiest setting when w 1, where we need weaker regularization, Sinal 0.4, accommodate the higher uncertainty.
Evaluation. We train two online linear probes with CrossEnt ropy loss on top of the represen- tations, one for predicting the superclass and one for the fine-grate one-grained species classes. The gradient of these losses do not propagate back to the encoder. We then report their performance on D val-
Hardware. Each trial of this experiment required approximately 40 hours run, using 12 CPU cores, 60 GB of system memory, and an NVIDIA H100 GPU with 80 GB of GPU memory.
35


Reviews:
[{"review_id": "C3zbWjhEmk", "reviewer": "Reviewer_TJcB", "summary": "The paper proposes a model for self supervised learning that extends previous approaches to include an auxiliary variable that capture the variation between representation of related data, e.g. augmentations or naturally occurring variations. The results suggest indicate that the model is better able to adapt to data generated under more flexible assumptions, such as heteroskedasticity, relative to baselines.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is relatively clear and well explained (see weaknesses for exceptions). The motivation seems sound and the proposed solutions are rationally justified. Experimental results suggest the proposed models are effective.", "weaknesses": "A main weakness of the paper is its occasional lack of clarity. For the most part the paper is easy to follow, which makes the following very noticeable. The paper would improve greatly if these areas were addressed.\n* 33 - this does not seem to be about \"distribution shift\", i.e. a change to the distribution, but rather that artificial augmentations do not span the full variation in the distribution of natural images\n* Prop 2.1 - this is unclear, e.g. in line 156 we already know h maps to the unit sphere so why restate? It is unclear what this proposition adds since it appears to be specific to Gaussian distributions under a mapping between different topologies. This is a contrived scenario so it is unclear how it is necessarily relevant or \"unavoidable\" in practical scenarios. If the point is that the distributions of $z^+|z$ may vary over $z$ for real world datasets, which sounds highly plausible, Prop 2.1 doesn't appear to prove that and seems redundant.\n* 199: Eq 4 does not appear to support the \"intuition\" that follows, e.g. $r$ could convey no information and Eq 4 holds, so \"*should* help\" is untrue.\n* 209: The use of $p$ and $q$ suggest that $p$ is a ground truth posterior distribution over $r$ that $q$ learns to approximate similarly to a VAE, but a VAE is quite different since a ground truth posterior $p(z|x)$ is defined by the model's likelihood and prior, which $q$ provably learns to approximate. Here there is no defined prior over $r$ or ground truth posterior $p$, so the notation seems spurious (it would seem more accurate to refer to $r$ as an \"auxiliary\" variable (as in Khemakhem et al. (2020)) and \"conditional\" $q$.\n* 234: it is unclear how $r$ is included in InfoNCE or how the projector compares to the MLP used in BYOL. This is a key part of describing the model and should be very clear. \n* 308: the distinction between embeddings and representations is unclear from the text.\n* 4.2: this section is very difficult to read, e.g. \n    - \"$f$ is the frozen encoder trained on $p(z)$\" presumably means $f$ was trained under the SSL algorithm given $x, x^+$ pairs sampled under the described generative process? If so, that is hard to parse and should be made clearer. \n    - (a) appears to describe generating data under the described method and training a regressor to predict the true generative $c$ component of $z$ from the representation $f(x)$. It is much less clear what (b) and (c) describe (c appears to be about robustness of the regressor?)\n    - since $\\Sigma$ is unstated, the significance of $5I$ in the \"OOD\" cases has no context (presumably 5 is higher variance than $p(z)$)?\n* \"OOD experiments\" - the theoretical background does not appear to suggest robustness under distribution shift, so there is no clear explanation for the improved results and \"only flexible models generalize OOD\" seems unjustified. It is fair to note the improved OOD results, but it seems unexplained.\n    - 366 - as above, the emphasis on OOD sees out of keeping with the rest of the paper. The model is designed to learn more flexible latent conditionals, this is shown in column 1 of Table 2, which seems the main result justifying the approach. No explanation has been given why this model would be expected to perform better OOD.\n* 377 - \"identifiability\" - what does this refer to?\n\nCelebA results: while the paper considers an adaptation to the InfoNCE loss, I believe that InfoNCE does not achieve state of art performance, so the results are not well contextualised in terms of current model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VH8slijS1V", "reviewer": "Reviewer_gRec", "summary": "This paper presents novel self-supervised learning methods that model the uncertainty in data pairs generated from natural generative processes, using regularized latent variables. For example the uncertainty between consecutive frames in a video. Two variants are presented, one based on variational inference on the other on enforcing sparsity on the latent variable. Experiments on artificial and reel data are conducted to demonstrate the effectiveness of the approach in identifying latent factors of variations and modelling uncertainty.", "questions": "- Line 160: Then the solution is just to project and do prediction in the same space ?\n\n- In AdaSSL-V, how could you make more explicit the mechanism that regularizes the latent variable regularized, basically what is L_reg ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a key problem in self-supervised learning: modelling conditional uncertainty, which arises in many other problems related to causal prediction. The potential applications are therefore numerous: video prediction, video generation, world modelling and latent action prediction, efficient self-supervised learning ect. The paper could actually do a better job at motivating these applications.\n\n- The ideas presented in the paper are interesting, described in depth, well-motivated, and seem to be good candidate solutions, at least on the toy problems explored in the experimental section.", "weaknesses": "- The paper is complex to understand, with lots of formalism (the probabilist framework, Proposition 2.1), complex vocabulary (heteroscedasticity, modular editing, DCI) that is not necessarily introduced, or introduces lots of new vocabulary (CRL, DGP, SSL from structural invariance, Adaptive SSL, natural pairs), all for ideas that are actually fairly simple. I feel like this over–complixification hinders the reading flow and makes it harder to deliver the message it intends to deliver.\n\n- Also, the paper put a lot of emphasis on particular SSL losses such as contrastive vs non-contrastive, as well as several variants of InfoNCE, which does not seem very relevant for this study, and adds too many factors of variation that make the conclusions of the paper less clear and convincing. Section 2.2 and 2.3 are probably unnecessary, and the new variants H-InfoNCE, AnInfoNCE, ect are not well motivated.\n\n- Finally, all this formalism is derived by the JEPA framework and the authors mention that they only take inspiration from JEPA, whereas I see these contributions as instanciations of JEPAs, just with various ways of regularizing the latent variables. In Figure 1, b) and c) are JEPAs.\n\n- The experiments are conducted on toy problems which limits the credibility of the approach. How does it behave on more concrete problems ? I don’t think people care about the artificial numerical problems of section 4.2, these should be more of a tool for you to debug the approach. Section 4.2 is interesting but very artificial, and 4.3 is Moving MNIST which is good again for debugging but nowhere near close to the actual interesting problems.  Finally, the focus is on velocity decoding, which is good for debugging but not as interesting as the problem of prediction. It would be more interesting to show experiments where a model predicts the future trajectory in moving MNIST, and being able to sample several possible future trajectories by sampling from the latent.\n\n- Related to this, the claims made at the beginning of the paper need to be toned down, for example Line 20 “and we empirically show its superiority on identifiability, generalization, fine-grained image understanding, and world modeling on videos”. Superiority against which concurrent method ? And on benchmarks that are too toy.\n\n- The paper ignores the vast literature existing on uncertainty modelling and latent variables. All the work in generative modes, video generative models, video prediction, latent action models.\n\n- In conclusion, the paper is tackling an interesting problem and presents interesting ideas but it is hard to be convinced by the toy experiments. These points would make it much stronger:\n\n- Remove the studies on InfoNCE variants, along with sections 2.2 and 2.3, and focus on AdaSSL. Maybe rename using the JEPA terminology and just name the latent variable regularization methods.\n\n- Remove section 4.2 and focus more on real data experiments.\n\n- Add more motivations in terms of potential applications\n- Acknowledge other literature in uncertainty modelling and world modelling.\n\n- Focus the experiments more on video world modelling, and the prediction capability, rather than training probes to recover properties such as velocity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLk8p79MxH", "reviewer": "Reviewer_NVc3", "summary": "This paper addresses the problem of self-supervised representation learning when the two views of the data used to learn the representation are so-called {\\em natural pairs}  instead of handcrafted augmentations, such that the two views are generated from (unknown) latent factors with the dependency between the latent variable encoded by some unknown conditional probability distribution.", "questions": "I understand how, from their probabilistic definition in terms of latent factors, natural data pairs may be different from the \"augmented\" pairs typically used in SSL. From an intuitive point of view, however, I do not really see how nearby video frames are qualitatively different from image crops, say. Both can be seen as crops, temporal or spatial, of the data. I would appreciate that the authors comment on this point.\n\nPlease explain the significance of Prop. 2.1.\n\nAs noted by the authors, AdaSSL-V is only justified for contrastive SSL, but it is used for non-contrastive SSL as well. Could you please justifiy this?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "SSL methods are widely used, and their theoretical study is welcome. The dea of modeling the dependency of the views on some latent variables is interesting.", "weaknesses": "Although the topic of the paper is interesting, the presentation is hard to follow without well identified objectives and experiments mostly limited to toy examples. \n\nThe presentation should be clarified. For example:\n- Section 2 defines the data generation process in terms of latent variables z and z+, but in Section 3 these disappear to be replaced by a latent variable r presumably there to parameterize the predictor, simillar to predictive SSL methods.\n- I did not understand the difference between the AdaSSL loss and that typically used in predictive SSL, in part because the dependency of the model on the latent variable r is never defined before giving Eqs. (4) and (5) so I didn't understand how the terms in these equations were computed.\n-psi_1 and psi_2  are used in Eq. (8) before they are defined in Eq. (9).\n-The function t, which was an arbitraty MLP until then, is defined explicitly in Eq. (11) as a modular editing function.\n\nI could not find any justification as to why it should be possible to recover the latent variables since they are never used, as far as I know, in the actual loss of the AdaSSL variants. This is problematic since the experimental evaluation is for the most part dedicated to this recovery.\n\nRemark: although it is frequently used in non-contrastiva approches to SSL, the EMA formulation in Eq. (3) is, as far as I know, ill defined since the exponential moving average is normally taken over the parameters defining psi over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UFTq1MkVk", "reviewer": "Reviewer_y8NR", "summary": "The paper studies the Self-Supervised Learning problems. It argues that existing methods fail to model the heteroscedasticity in $p(z^+|z)$  for positive pairs. The paper proposes AdaSSL by introducing a latent variable $r$ that captures the stochastic transformation from one view to another. By jointly learning the encoder f(x), the latent transformation r, and an edit function that reconstructs the target embedding, the method adapts to the heteroscedasticity in the above conditional. The paper performs extensive experiments on synthetic and real datasets and shows improved disentanglement and generalization compared to InfoNCE and BYOL variants.", "questions": "- Why is the proposed approach better than H-InfoNCE? How does it encourage disentanglement?\n- How learning an efficient representation of r (line 397) leads to a more disentangled feature f(x) in Table 4?\n- Why is another view $x^{++}$ introduced in CelebA experiment? What if only using $x$ and $x^+$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- I find the paper interesting and enjoyable to read.\n- Learning from pairwise images and the heteroscedasticity in $p(z^+|z)$ is a promising and under-explored area.\n- Strong motivations backed by theoretical and empirical analysis.", "weaknesses": "- While proposition 2.1 shows heteroscedasticity exists, and 2.2 explains how existing methods fail to account for it. It remains unclear to me (at least intuitively) how modeling the complex conditional $p(z^+|z)$ contributes towards the ultimate SSL objective, i.e., generalization on a wide range of downstream tasks that predict a subset of factors in z.\n- It is even harder for me to understand how the proposed approach should be better in downstream tasks compared to baselines with similar motivation (e.g., H-InfoNCE).\n- As acknowledged in Section 3.2, the proposed approach is only theoretically justified for contrastive SSL. This introduces a gap due to the popularity and performance of non-contrastive SSL.\n- $q_\\phi,p_\\theta$ are both modeled as factorized Gaussians. This is slightly against the idea that the conditional can be quite complex, as it is up to the edit function $t$ to model the complexity, which can complex model design and reduces learning efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)