seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models
Authors: Hafez Ghaemi
Eilif Benjamin Muller
Shahab Bakhtiari
Venue: NeurIPS 2025 poster ()
Keywords: Self-supervised learning
world models
computer vision
equivariance
Primary area: deep_learning
TL;DR: A self-supervised world model that jointly learns two architecturally distinct representations for equivariance- and invariance-demanding downstream tasks

Abstract: Joint-embedding self-supervised learning (SSL) commonly relies on transformations such as data augmentation and masking to learn visual representations, a task achieved by enforcing invariance or equivariance with respect to these transformations applied to two views of an image. This dominant two-view paradigm in SSL often limits the flexibility of learned representations for downstream adaptation by creating performance trade-offs between high-level invariance-demanding tasks such as image classification and more fine-grained equivariance-related tasks. In this work, we propose \emph{seq-JEPA}, a world modeling framework that introduces architectural inductive biases into joint-embedding predictive architectures to resolve this trade-off. Without relying on dual equivariance predictors or loss terms, seq-JEPA simultaneously learns two architecturally separate representations for equivariance- and invariance-demanding tasks. To do so, our model processes short sequences of different views (observations) of inputs. Each encoded view is concatenated with an embedding of the relative transformation (action) that produces the next observation in the sequence. These view-action pairs are passed through a transformer encoder that outputs an aggregate representation. A predictor head then conditions this aggregate representation on the upcoming action to predict the representation of the next observation. Empirically, seq-JEPA demonstrates strong performance on both equivariance- and invariance-demanding downstream tasks without sacrificing one for the other. Furthermore, it excels at tasks that inherently require aggregating a sequence of observations, such as path integration across actions and predictive learning across eye movements.

seq-JEPA: Autoregressive Predictive Learning of World Models
Hafez Ghaemi Ahhaemi.2.. 1,2,3
Eilif Muller1222 2.33
Shahab Bakhtiari 1:21
Université de Montreal, Mila Quebec Al Institute, ²CHU Sainte-Justine
Abstract
Joint-emhedding self-supervised learning (SSL) commonly relies on transforma- tions such as data augmentation and masking learn visual representations, task achieved by enforcing invariance or equivariance with respect to these transforma- tions applied two views of image. This dominant two-view paradigm in SSL often limits the flexibility of learned representations for downstream adaptation by creating performance trade-offs between high-level eevariance-demande such as image classification and more fine-gre tineggrined equivariance-relatee tasks. In this work, we propose seq-JEPA, a world modeling framework that introduces architectural inductive biases into joint-embedding predictive architectures to re- solve this trade-off. Without relying on dual equivariance predictors or loss terms, seq-JEPA simultaneously learns two architecturally separate representations for equivariance- and invariance-demandingg tasks. sequences of different views (observations) of inputs. Each encoded view is con- catenated with an embedding of the relative transformation (action) that produces the next observation in the sequence. These view-action pairs passed through a transformer encoder that outputs an aggregate representation. A predictor head then conditions this aggregate representation on the upcoming action to predict the representation of the next observation. Empirically, scq-JEPA demonstrates strong performance both equivariance- and invariance-demandinn downstream tasks without sacrificing one for the other. Furthermore, it excels at tasks that inherently require aggregating sequence of observations, such path integration across actions and predictive learning across eye movements.
tasks
so, our model processes short
Project Page
Code
Models
STL Saliency
ImageNet- lk Saliency
3DIEBench-OOD
1 Introduction
Self-supervised learning (SSL) in latent space has made significant progress in visual representation
learning, closing the gap with supervised methods across many tasks. Most SSL methods rely on
comparing two transformed views of an image and enforcing invariance the transformations MMssaa
and van der Maaten 2020)
Chen et al., 2020) He et al. 2020)] Dwwbeddi et al., 2021 Hao Che
al. 2021 Yeh
2022 Caron
2020 2021 Ermoloo al. 2021 Assran al. 2022
Zbontar et al. 202 Bardes et al. 2022]. Another group of methods employ techniques to preserve
transformation-speee speciic information, thereby learning equivariant representations [Lee al.
202
Xiao et al. 2021 Park et al. 2022 Dangovski et al. 2022 Gupta et al. 2023 Garrido et al.
2023
2024 Gupta
2023
2024
Yerxa al. 2024..
*Correspondence hatez. gnaemi @umontreal.co
EEuul Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Equivariance is a crucial representational property for downstream tasks that require fine-graine grained distinctions. For example, given representations that are invariant to color, it is not possible to distinguish between certain species of flowers or birds [Lee et al. 2021 Xiao et al., 2021] Moreover, recent work has shown that equivariant representations are better aligned with neural responses in primate visual cortex and could be important for building more accurate models thereof (Yerxa
cct al. 2024]. While some equivariant SSL approaches have reported minor gains on tasks typically associated with invariance (c.g., classification) [Devillers and Lefort 2022 Park et al. 2202 Gupta et al. 2023], a growing body of work highlights a fundamental trade-off between learning invariance and equivariance, i.e., models that capture equivariance-
style latents do not fare well in
classification and vice versa [Garrido et al. 2023 2024 Gupta et al. 2024 Yerxa et al. 2024
Rusak et al., 2025] This trade-off has recently received theoretical support [Wang et al. [2024b],
goals. underscoring the need for new architectural and objective designs that can reconcile these competing
In contrast to the two-view paradigm in SSL., humans and other animals rely on a sequence of actions and consequent observations (views) for developing appropriate visual representation during novel object learning [Harman et al. 1999 Vuilleumier et al. [002]. For example, they recognize 3-D object by changing their viewpoint and examining difterent sides of the object [Tarr et al., 1998]. Inspired by this, we introduce seq-JEPA, a supervseer world modeling framework that combines boint-embedding predictive architectures [LeCun 2022] with inductive biases for sequential processing. seq-JEPA simultancously learns two architecturally distinct representations: one that is equivariant to specified set of transformations, and another that is suited for invariance-demanding tasks, such as image classification.
Specifically, our framework [ processes a short sequence of transformed views (observations) of an image. Each view is encoded and concatenated with an embedding corresponding to the relative transformation (action) that produces the next observation in the sequence. These view-action pairs are passed through a transformer encoder, a form of learned working memory, that outputs an aggregate representation of them. A predictor head then conditions this aggregate representation on the upcoming action to predict the representation of the next observation.
Our results demonstrate that individual encoded views in seq-JEPA become transformation//ccionn equivariant. Through ablations, we show that action conditioning plays key role in promoting equivariant representation learning in the encoder network. In contrast, the aggregate representation of views, produced at the output of the transformer proves highly effective for invariance-demanding downstream tasks. This emergent architectural disentanglement of invariance and equivariance is central to seq-JEPA's competitive performance compared to invariant and equivariant SSL methods
on both categories of tasks (Figure 3. Unlike most prior equivariant SSL methods [Lee et al.
2021
Park et al.] 2022 Dangovski et al., 2022, Gupta et al., 2023 Garrido et al., 2023] Gupta et al. 2024
Yerxa et al. [224]. J. our model does not rely on explicitly crafted loss terms objectives achieve
equivariance, nor is it instructed to learn the decomposition of two representations. Instead, the dual
representation structure arises naturally from the model architecture and action-conditional predictive
learning.
Beyond resolving the srarrincce-euivariinnee trade-off, seq-JEPA further benefits from processing a sequence of observation views; we show that it performs well on tasks requiring integration over sequences of observations. In one scenario, inspired by embodied vision in primates, our model learns image representations without augmentations or masking, solely by predicting across simulated eye movements (saccades). In another setting, it performs path integration over sequences of actions-such as eye movements or 3D object rotations in 3DIEBench [Garrido et al. 2023). Our key contributions are as follows:
We introduce -EE-, a self-supervised world model that learns architecturally distinct representations for invariance- and gquivariance-demaddii downstream tasks through sequential prediction over action-observation pairs, without requiring explicit equivariance losses or dual predictors.
We empirically validate that seq-JEPA matches or outperforms existing invariant and equiv- ariant SSL methods across tasks requiring either representational property.
We demonstrate that -EP- naturally supports tasks that involve sequential integration of observations, such as predictive learning across saccades and path integration over action sequences.

---

2 Method
2.1 Invariant and equivariant representations
Before presenting our architecture and training procedure, we briefly define invariance and equiv- ariance in the context of SSL (Dangovski et al. 2022 Devillers and Lefort 2022]. Let T denote distribution over transformations, parameterized by a vector These transformations such as augmentations or masking can be used to generate multiple views from a single image z. Let 2i and 22 two such views, produced by applying transformations 4 and l2 sampled from T. Additionally, let a denote the relative transformation that maps æ1 to z2. Additionally, we denote as transformation that transforms #1 to 222. We distinguish between (an individual transformation) and (an action), where the latter reflects the change from one view to another. Let f be an encoder that maps inputs to latent space. We say that is equivariant to if:
E T. 3. s.s.t.
f(t(z)) = ue((x)))
(1)
where u, transformation in latent space corresponding to t. Equivariance can similarly be defined in terms of relative transformations (actions):
Va E T, 3, 8.j
f(x2) ua((((r))
(2)
As a special case, if us and un are identity functions, then f' is invariant to the transformation: [(((x)) (x) or (((22) f(x)).
Architecture
Figure 1 presents the overall architecture of seq-JEPA. Let A be sequence of views generated from sample æ via transformations { M+1 The relative transformations (actions) { are L.
defined as a; A A4i,+1, i.e., the transformation mapping x; to x141 our default setting, we use learnable linear projector to encode these actions.
backbone encoder, f encodes the first M views, producing representations {2. 1= Except for 2M. each is concatenated with its corresponding action embedding and passed to transformer encoder (no MLP projector is used after the encoder), which aggregates the sequence of action- observation pairs. The transformer uses learnable [AGG] token (analogous to the [CLS] ] token in ViT [Dosovitskiy al., 2020)) to generate the aggregate representation:
AGG g((21, 01), (22, 02).... (M-1,4M-11) 2M1)
This aggregate representation ZAGG is then concatenated with the final action embedding aM (corresponding to the transformation from TM to M++1), and passed to an MLP predictor h to predict the representation of M
2 M h(ZAGG, aM).
The ground truth #M++1 is computed using target encoder-an exponential moving average (EMA) of f. This target representation is passed through stop-gradient operator (sg) to avoid representational collapse (Grill al. 2020 Chen and He 2021 Assran al. 2023). The training objective is to maximize the cosine similarity between 2 and 2M+1 with the loss function:
Leeq -EPA = 1 2M+1 sg(2M+1)
(5)
| l2
[|g(2MM1)]l
No additional loss terms equivariancc-sseciffe predictors used during training.

---

a)
(G,,2))
3D dbject renolecing REDLED ench)
Hand arshed aigmentalions
6x
Predictive lesminn saccodis
Enc
Enc
Foeegalsedd cemundrngtasin
[A66]
b)
IE map
Inhitition of
Sequence Aggregator
Caunce
Predictor
Zuren
posilions
Figure I: seq-JEPA sule-suee world model that leverages sequence of action-observation pairs learn architecturally distinct representations for downstream tasks requiring transformation invariance or cquivariance.
Figure 2: a. Transformations and observations used for training b. In predictive learning across saccades. image saliencies and inhibition of return help create more informative and less redundant patch sequences.
2.3 Action and observation sets
To evaluate generalization across transformation types, we consider three sets action-observation pairs (Figure 2). See Appendix A. for details of each setup.
3D Invariant Equivariant Benchmark (3DIEBench). The 3DIEBench dataset [Garidoo al. 2023 is designed to evaluate representational invariance and equivariance. It includes 3D object renderings with variations rotation, floor hue, and lighting. In this benchmark, the action between two views is the relative difference of these three factors of variation. We primarily study equivariance to SO(3) rotations and secondarily to appearance factors of floor and light hue.
Hand-Crafte Augmentations. In this setting, we use transformed views generated via common SSL augmentations (e.g., crop, color jitter, blur), and actions correspond relative augmentation parameters. We use CIFAR 100 and Tiny ImageNet for experiments in this setup, and follow
EquiMod's 's augmentation protocol [Devillers and Lefort 2022
Predictive Learning Across Saccades (PLS). Going beyond conventional transformations such augmentations 3D rotations, show that seq-JEPA can learn visual representations from sequence of partial observations thanks to architectural inductive hand-crafted augmentations. Our PI.S has similar flavor to I-JEPA [Assran al. 2023 but does not require engineered masking strategies. In PLS, we train seq-JEPA on sequences of patches extracted from full-resolution images. For instance, with STL-10 dataset, patches to form the observation sequence. In this setting, actions correspond to the relative positions between patch centers, simulating saccadic eye movements and inducing 2-D positional equivariance to representations. To select fixation points, we adopt two biologically inspired techniques that increase informativeness, reduce redundancy, and improve the downstream utility of the aggregate representation (Figure 2
relying on any
small 32 32
Saliency- Based Fixation Sampling. Using DeepGaze IE [Linardos al.) 2021
extract saliency maps for each image and use them to probabilistically sample fixation points [Itti et al. 1998. Li 2002 Zhaoping 2014). The maps are pre-computed and introduce training overhead. Inhibition of Return (IoR). To reduce spatial overlap between patches and emulate natural exploration [Posner et al. 1985), we implement loR by zeroing out the sampling probability of areas surrounding previously sampled fixations.

---

3 Experimental Setup
3.1 Compared methods and baselines
We compare seq JEq- against both invariant and equivariant SSL baselines. Invariant methods include SimCLR [Chen et al. 2020). BYL [Grill et al. 2020]. and VICReg [ardes et al. 2022] and ViCReg with trajectory regularization (Wang et al. 2024a] Equivariant methods include SEN [Pakk et al.
2022 EquiMod [Devillers and Lefort 2022], SIE [arrido et al. 2023]. J. and ContextSSL [Gupta
al. 2024 For all baselines, architectural details are given in Appendix A.3 We also evaluate two
hybri baselines based on our architecture:
Conditional BYOL, A two-view version of seq-JEPA with sequence aggregator, where BYOL's predictor is conditioned on the relative transformation between target and online views. This encourages representations encode transformation information.
Conv-JEPA. baseline for the saccades setting. It uses the same sequence of salieney- sampled patches as seq-JEPA and predicts the final patch's 'ss representation from each earlier patch individually. These losses are summed across the pairs before backpropagation.
3.2 Training protocol
All models use R8 (He et al. 2016 as the backbone encoder. For action conditioning, use a learnable linear projection to learn action embeddings (default action embedding is 128-d). The sequence aggregator in seq-JEPA is lightweight transformer encoder [Vaswani al. 2017 with three layers and four attention heads. The predictor is a 2-layer MLP with 1024 hidden units and ReLU activation. In order to control for and eliminate any performance gain resulting from using a transformer encoder in seq-JEPA instead of an MLP projection head, we trained baselines that typically use MLP projectors two variants: (1) with original MLP projector; and (2) with the MLP replaced by transformer encoder and sequence length of one. We did not see any benefit from switching to transformer projectors in any of the baselines, and include the transformer-pron projeccoo results in Appendix B.2] All models are trained from scratch with batch size of 512. We use 1000 epochs for 3DIEBench and 2000 epochs for other datasets to obtain asymptotic performance. We use Adam W for models with transformer projectors (including seq-JEPA) due to its stability and improved regularization in transformer training. For ConvNet-only models with MLP heads, we use the Adam optimizer. Full hyperparameters are detailed in Appendix A
3.3 Evaluation metrics and protocol
To assess equivariance, we follow the protocol of Garrido et al. 2023 and train a regressor on frozen encoder representations to predict the relative transformation (action) between views. We report the R² score on the test set. In addition to action decoding R², we also report retrieval-based metrics including Mean Reciprocal Rank (MRR), Hit and Hit@5 (Kipf et al. 2020 Park al. 2022 Garrido et al., 2023) to evaluate the quality of the predictor. As a proxy measure of invariance, we use top-1 classification accuracy of linear probe on top of frozen representations. For all baselines, probes are trained on encoder outputs. For seq-JEPA, we measure accuracies on top of the aggregate representation (ZAGG in Figure [ and report the number of observation views used during training and inference. For completeness, we also report classification performance on top of encoder representation (2ę in Figure for seq-JEPA models in Appendix [B77 Training details of evaluation heads are given in Appendix A.2
Results
4.1
Quantitative evaluation 3DIEBench
We use the 3DIEBench benchmark to quantitatively compare performance on equivariance- and invariance-demandiin downstream tasks in seq-JEPA with baseline methods. This benchmark allows us to measure equivariance through decoding 3D object rotations while enabling invariance measurement through object classification. Table [ provides a summary of our evaluation on the 3DIEBench where equivariant methods have been conditioned on rotation (a 4-D quaternion representing the relative rotation between two views). addition to the relative rotation between two views, in the last column provide the R² score for predicting individual transformation parameters
5

---

Table 1: Evaluation on 3DIEBench for linear probe clas- sification derrianncdddmmdiinn)g and rotation prediction cetivarrianceddmmnndinn Equivariant models and seq-JEPA are conditioncd on rotation. For seq-JEPA, training and infer- ence sequence lengths are denoted by Me and Myat-
ContextS81
0.7
SeqUJEPA1
Relatinn Predicion (R2)
0.6
Inverant
Equinarinnt
Method
Top-1 Acc. (96
Rel. Rot. (T²)
Indiv, Rot. (TP)
0.4
8 SmmCLR
Invoriant
0.3
ViCRgg Tra)
SEN
EquMod
BYOL
SinCLR
81.13
0.35
0.25
0.54
VICRop
VICReg
80.48
0.20
0.36
VICRogTrai
81,26
0.27
0.43
BYOL
Eguivariont
0.1
SEN
83.43
0.35
0.57
84
Equi
84.29
0..3
0.55
Top-1 Linear Accuracy (%)
SIE
77.49
0.62
Conditional BYOL.
82.61
0,47
ContexiSSL (c 126)
80.10
0.74
0.78
Figure 3: Top-1 linear classification (invari- ance) vs. rotation prediction (cquivariance) per- formance on 3DIEBench. seq-JEPA learns good representations for both tasks. Subscripts indicate training and inference sequence lengths.
Invriiant Equivariont
seq-JEPA
seq SEqJEPA (1.
85.31
0.65
0.69
seqJEPPA (3,
N6. 14
(1.1
0.74
JEPA (3,5)
87.41
0.71
0.74
soq-JHPA (80 act cond)
86.05
0.37
from representations of single view. For seq-JEPA, we trained models with varying training sequence lengths (denoted by Mrr in the table) and measure the linear classification performance on top of aggregate representations with different inference lengths (denoted by Moat).
Among invariant methods, BYO achieves the highest classification accuracy, yet does not offer high level of equivariance. Adding linear trajectory regularization loss to VICReg Wang et al. [2024a] without using ground-truth transformations improves over VICReg, which shows that im- posing geometric priors can improve both invariant and equivariant performance even when these priors do not fully materialize in the environment, e.g., when we have non-smooth angle changes as in 3DIEBench. Among equivariant baselines, SIE and ContextSSL yield strong rotation prediction performance due to their specialized equivariance predictors and loss functions, but underperform in classification. EquiMod and SEN offer better classification performance, yet compromise equivari- ance. In contrast, seq-JEPA achieves strong performance in both, matching the best rotation It2 scores while exceeding baselines in classification, with gains increasing with inference sequence length Myat- Ablating action conditioning leads to sharp drop in equivariance but retains classification accuracy, confirming our hypothesis that condiniondi sequential aggregation enables distinct tasks. For additional results
representational structure for invariance- and equivariance-demanne on 3DIEBench including MRR, Hit@ 1, and @5 metrics, performance of models with varying training and inference sequence lengths and models conditioned on both rotation and color, and evaluation on an disti--ditt (OOD) set of 3DIEBench see Appendices B. to B.6
4.2 Qualitative evaluation 3DIEBench
To visualize equivariance in representational space, retrieve the three nearest representations of query image from the validation set of 3DIEBench (Figure 4. While all models retrieve the correct object category, only seq-JEPA and SIE consistently preserve rotation across all retrieved views, consistent with their high R²2 scores. Next, we projected encoder and aggregate representations using 2D UMAP (Figure 5). The left panel shows encoder representations colored by class label, while the middle panel displays the same encoder representations colored by rotation angle. The smooth color gradation across the map within each class cluster the middle panel suggests that the encoder captures rotation angle as continuous factor, implying equivariance to rotation (e.g., the red class in the bottom-right corner of the right panel and the corresponding part in the middle panel). The right panel shows aggregate representations colored by class label. Comparing the class-colored plots (left and right panels), we observe that both encoder and aggregate representations contain class information. However, when we aggregate multiple views of sample, some of the intra-class variability (resulting from transformations such as rotation) eliminated, causing each class' representational cluster to become more homogeneous. This aggregation procedure likely reduces variation due to rotation and makes the representations more invariant, resulting in decreased intra-class spread and increased inter-class distance. We create similar UMAP visualization for seq-

---

SinCin
EquiMted
SEN
JER
Quury
Can Line
2080
Figure 5: 2-D UMAP projections of seq-JEPA's encoder and aggregate representations trained 3DIEBench and conditioned on rotation with M each view
Figure 4: Retrieval of nearest representations; given query image, we extract the three nearest encoder representations in the validation of 3DIEBench. The retrieved
and Meat = 5. Encoder representations observation, color-coded hy class (left) and rotation an- gle (middle). Aggregate representation for Meat color-coded class (right).
models with highest quantitative rotation equivariance performance maintain the rotation of the query image across all retrieved views.
Table Evaluation with hand-crafted augmentations on CIFAR 100 and Tiny ImageNet; equivariance is mea- sured by predicting relative transformation parameters associated with crop, color jitter, or blur augmentations. For all seq-JEPA models, M 5.
CIFARIO
The InaareNet
Conditioinng Method
Clasectiatton top
Crop (TC')
Inter Imer(())
Blar (IC)
Clasification (top
Crop UP')
Jiter (RP))
Blur Uit)
Issariasr
SanCLK
61.72
11.55
0.18
0.04
34.29
0.30
0.08
0.15
BYOL.
52.17
0.30
0.25
-0.01
35.70
0.17
0.01
0.16
VICRey
61.35
0.49
0.11
-0.06
35.20
0.31
0.05
0.19
VICKogTig
M1.07
0146
0.34
0.00
008
0.16
SEN
61.93
0.65
0.51
0.85
36.01
0.24
0.88
0,87
EquiMed
61.30
1159
0.49
0.24
36.75
0.40
0:86
STE
Cunditionl
0.46
0.73
0:87
M0.56
0.1%
01100
Crop
FggiMaal
0.01
EquiMad
jiter
SIR
0.35
Cundrionod
Bler
SIE
0.07
0.71
0.85
Cundeional NYCH.
50.34
0.11
0.69
0.00
0.55
JapJEPA
0.52
0.64
0.55
0.95
saq JaPJEPA (M,,-
58.33
0.79
0.63
0.02
34.85
0.67
0.64
0.56
Crup
soq JEPA (M4
59.32
0.78
0.01
0.10
35.74
0,70
0.12
0.46
Caler liter
say- JKPA (34
S8.82
0.68
BA6
0.20
35.91
0.60
0.66
0.6
Blar
way-JRIAA (36.-
$6.82
0.71
0.15
0.74
15.90
0.58
0.21
0.97
seq JEPA (ide
59.37
0.64
0.14
0.16
35.97
0.52
0.18
0.47
JEPA with ablated rotation conditioning in Appendix B.8 to highlight the role action conditioning in achieving equivariance rotation.
4.3 Evaluation with Hand-Crafted Augmentations
We assess downstream performance under hand-crafted augmentations by training CIFAR100 and Tiny ImageNet (Table 2). Models with action conditioning are trained conditioned on crop, color jitter, blur, or all three (indicated in the first column of the table). seq-JEPA consistently achieves higher equivariance than both invariant and equivariant baselines across all transformations. Notably, except for the model trained on CIFAR-100 and conditioned on blur, the best equivariance performance for given augmentation is achieved when the model is specialized and conditioned only on that augmentation. Furthermore, ablating actions (last row in the table) causes seq-JEPA to lose its equivariance across transformations compared to action-condition action-conditioned models. Overall, our model outperforms both invariant and equivariant families in terms of equivariance, while being competitive in terms of classification performance. For additional results with varying training and inference sequence lengths, see Appendix B.6
4.4 Predictive Learning across Saccades and Path Integration
In our third action-onservation setting, we consider predictive learning across simulated eye move- ments to exhibit JEPA's ability in leveraging sequence of partial observations to learn visual
7

---

Table 3: Evaluation of predictive learning across sac- cades on STL-10. Equivariance is measured with re-
a)
b)
Saccado Patn Frod 000 80
spect to fixation coordinates. Unless stated otherwise,
Mir- Meal 4.
Vedeact cord
44
(ile cand.
Dondisonin
Tip-l Auc Posttion (89)
SinCLR (oupecttaioo))
Cun-JEPA (M
Otsenvtiin Ohances (K
No. RotaRot Oa
poctiun
JFPA
JEPA
81.44
70.45
0.30
Figure 6: a) Visual path integration movements, b) Angular path integration across ob- ject rotations (results over three random seeds)
eye
pretiun
oap-IPA (M...)
84.
0.50
postio
JEFA twln sallency & 168)
79.85
0.88
prolous
=a-JEPA (wih IcR)
77.90
0.55
representations. In Table 3 scq-JEPA reaches 83.44% top-1 accuracy on STL-10, comparable to SimCLR (85.33%) trained with full-resolution images and strong augmentations. This gap narrows further when increasing the inference sequence length from Meal = 4 to Myal 6. Ablating action conditioning causes the accuracy on top of the aggregate representations to drop sharply, indicating that 2-D positional awareness is essential to forming semantic representations across simulated eye movements. Compared to Conv-JEPA wwhihhhhchhhhh accumulates prediction losses pairwise- Pairwise-seq-JEPP seq-JEPA performs better in classification, highlighting the importance of sequence aggregation when dealing with partial observations in SSL. Further ablations show that saliency-driven sampling and IoR are critical for forming informative, non-overlapping patch sequences and subsequently a high-quality aggregate representation. Interestingly, while random uniform patch sampling negatively impacts classification accuracy due lower semantic content, results in the highest positional equivariance as the model samples patches and corresponding saccade actions from more diverse set of positions across the entire image, not just the salient regions. The UMAP projections for PLS with and without action conditioning in Appendix B.8 further underscore the role of action conditioning in enabling positional equivariance.
Path Integration. In the context of eye movement-driven or any sequential observations, an ability that naturally arises from predictive learning is path integration (McNaughton et al.) [006]. i.e., predicting the cumulative transformation/action from a sequence of actions. We evaluate this task in both eye movements in PLS (visual path integration) and object rotations in 3DIEBench (angular path integration). As shown in Figure 6 scq-JEPA demonstrates strong performance in both settings, with performance degrading gracefully as sequence length increases. Ablating action conditioning causes path integration to fail, whereas ablating the visual stream has only a minor impact highhighing that action information is the dominant signal for this task. Full details of the path integration setup are provided in Appendix B.9
4.5 Action Conditioning Ablations
To better understand the mechanisms underlying seq-JEPA's invariant-cquivariant representation learning and role of action conditioning, we perform a set of ablation experiments on 3DIEBench. Specifically, study: (i) the role of action conditioning in the transformer and predictor; (ii) the im- pact of action embedding dimensionality. Table summarizes our ablation results. Removing action conditioning entirely causes significant drop in equivariance (R² from 0.71 to 0.29), although classi- fication accuracy remains high thanks to sequence aggregation and segregated wwariance-equivarinn in our model. Conditioning only the transformer only the predictor leads intermediate results, with predictor conditioning proving more critical for equivariance. We also vary the dimensionality of the learnable action embeddings: performance saturates around the default size of 128, with smaller sizes (e.g., 16 or 64) already sufficient to capture the rotation structure.
4.6 Scaling Properties: Role of Training and Inference Sequence Lengths
We study the effect of both training and inference sequence lengths on performance across tasks, i.e., scalability seq-JEPA terms context length (Fi We draw inspiration from recent findings in foundation models, where increased training and inference context- whethhr in text [Brown et al.
2020 Touvron et al. 2023). vision [Zellers et al. 2021 Chen et al.) 202T] or video [Bain et al., 202T consistently leads to stronger representations.
Arnab et al.
2021
8

---

Table 4: Ablation results for action condition- ing (3DIEBench). All models use Mr. = 3. Meal (results over three random seeds).
0.
2 S
200t
Variant
Top-1 Ace. (%)
Rotasion (fP))
Aet. conditioning
0.4
5
Nowe
87.36 0.7
0.29 0.04
Leugth
Length
0.37 0.06
transfomerr cond.
86.33
0.53 0.05
Figure 7: Effect training and inference sequence length on seq-JEPA's performance; left: Equivariant performance (R²) versus training sequence length; middle: Classification accuracy versus training sequence length; right: Classifica- tion accuracy versus inference sequence length.
Act. embeddiig dim.
Oze 18
$6.29 0.4
0.70 000
Gan 64
87.11 0.2
0.20 = 0.02
128 idefault)
$7.41 0.5
0.71 0002
256
87.
+
0.72 102-000 0.00
We observe that equivariance generally improves with longer training sequences (left panel). A possible explanation for this observation is the presence of more transitions (zi, a,, 2i+1) ) in working memory, which means the predictor has access to a richer context Zage- This enables the predictor to more accurately approximate the transition p( | the model to preserve and utilize information about dg, the encoder is implicitly encouraged to learn structured, more equivariant representations.
ia az). Because accurate prediction requires
Classification performance 3DIEBench and STL-10 (middle panel) benefits from longer training sequence. In contrast, on CIFAR 100 and Tiny ImageNet with synthetic augmentations, longer training sequences slightly decrease classification performance. We hypothesize that leveraging and aggregating sequence of action-observation pairs, i.e. seq-JEPA's architectural inductive bias, is most effective in settings where the downstream task benefits from sequential observations. In the case of object rotations in 3DIEBench, seeing an object from multiple angles is indeed beneficial in recognizing the object's category, which explains the improved classification accuracy with increased training sequence length. Similarly, in the case of predictive learning across saccades, each eye movement and its subsequent glance provides additional information that can be leveraged for leamingg richer aggregate representation.
At inference time, all datasets benefit from longer context lengths (Muat), confirming that richer aggregate representations yield stronger performance (right plot). This scalability via sequence lengths opens avenues for efficient representation learning with small foveated patches in lieu of full-frame inputs, mirroring how foundation models scale with input tokens at test time. Together with our transfer learning results on ImageNet-IK (Appendix IO), these findings suggest that seq-JEPA's architectural inductive bias enables graceful scaling via longer sequence lengths.
5 Related Work
Non-Generative World Models and Joint-Embedding Predictive Architectures. Non-generative world models predict the effect of transformations or actions directly in latent space, avoiding reconstruction in pixel space. This includes contrastive SSL methods that model transformed views from context representations [van den Oord et al. 2019 Gupta et al. 2024), as well as approaches in model-based reinforcement learning (RL) to improve sample efficiency, generate intrinsic rewards, or capture environment transitions [Schwarzer al., 2021, Khetarpal et al., 2025 Ni al., 2024
Tang et al. 2023 Guo et al. (202). Joint-embedding predictive architectures LeCun 2022 form subclass non-generative world models. They introduce an asymmetric predictor conditioned on transformation parameters to infer the outcome of an action applied to latent view. Examples include I-JEPA[Assran et al.] 2023), which predicts masked regions from positional cues, and IWM [Garrido] augmentation parameters. JEPAs have been recently extended to physical reasoning [Garrido et al. 2025) and offline planning (Sobal et al., 2025), illustrating the framework's 's versatility in representation learning and world modeling.
al. 2024]. which conditions
information in the latent space, typically by augmenting invariant objectives with an additional equivariance term. Some approaches directly predict transformation parameters [Lee et al.) 2021 Scherr et al. 2022
Equivariant SSL. Equivariant SSL methods aim retain transformation-speete
Gidaris et al. 2018 Gupta et al. 2024 Dangovski et al. 2022). Methods such as EquiMod [Devillers) and and Lefort] 2022) and SIE [Garrido al. 2023) predict the effect of a transformation latent space
via predictor in addition to their invariant objective. SEN (Park al. 2022 similarly predicts

---

transformed representations but omits the invariance term. Xiao et al. [2021) use contrastive learning with separate projection heads for each augmentation, treating same-aug aame-augmentation pairs as negatives.
ContextSSI. [Gupta et al. 2024] conditions representations on both current actions and recent context and employs dual predictor for transformation prediction to avoid collapsing to invariance. Other
approaches (Shakerinava et al., 2022] Gupta et al. 2023 Yerxa et al., [024] do not require explicit transformation parameters but instead enforce equivariance by applying the same transformation to multiple view pairs and minimizing a distance-based loss. Action-condition
JEPAs incorporate augmentation parameters and mask positions by conditioning the predictor to induce equivariance
without additional objectives [Garrido et al. 2024]. Chavhan al. [023 use ensemble of heads trained on top of a pre-trained SSL encoder to span a diverse spectrum of transformation sensitivities across the latent spaces of each head. Downstream probes then learn to linearly combine these feature
heads depending on the desired anvariance-euuvarrinne trade-off.
Positioning of Our Work. seq-JEPA belongs to the family of joint-embedding predictive architectures and is -onggenerat world model. Unlike most equivariant SSL methods, seq-JEPA does not rely on an equivariance loss or transformation prediction objective, nor does it require view pairs with matched transformations. Instead, it leverages action conditioning and architectural inductive biases to learn two separate invariant and equivariant representations. In contrast to ContextSSL, which extends the two-view contrastive setting of contrastive predictive coding van den Oord et al. [2019] using transformer decoder projector conditioned previous views via key-value caching, seq JEPA operates on sequences of action-observation pairs in an online end-to-end manner by incorporating sequence model (e.g. transformer encoder) as learned working memory during both training and inference. Moreover, while ContextSSL. aims to adapt equivariance to recent transformations by dynamically modifying the training distribution, seq-JEPA is designed to explicitly learn both invariant and equivariant representations with respect to specified set of transformations, and is also well-suited for downstream tasks that require multi-step observation aggregation.
Limitations and Future Perspectives
We have validated viability of JEQ- across a range of transformations in the image domain. Here, we discuss limitations and possible future directions. First, seq-JEPA is capable of autoregressive prediction in time, and therefore, can be leveraged for autoregressive latent planning in control tasks. Second, the transformer-baseee aggregator in seq-JEPA could support multi-moda fusion across language, audio, or proprioceptive
multi-modal world modeling and generalization. Third, while our results show clear benefits from longer sequences, we have experimented with relatively short training and inference sequence lengths as the backbone is also trained end-to- end. 'The sequence scaling trends observed in the paper suggest that seq-JEPA can benefit from longer context windows over representations of pre-trained backbones as in [Pang et al., 2023 Lin ett al. 2024]. Fourth, our method assumes access for 3D rotations). Designing group-agnostic or learned transformation models [Finzi et al. 202T without access to transformation parameters or pairs of same transformation is open challenge in equivariant SSL that future work may tackle. Finally, our preliminary ImageNet- 1k transfer results (Appendix [0) point to potential for broader generalization. Scaling seP-JEPA to larger foveated image settings or video and the development of lightweight, saliency-driven agents capable of learning efficiently from partial observations in embodied settings with a limited field of vision.
known transformation group (c.g.. SO(3)
datasets such as Ego4D Song et al. [2023] could support
Acknowledgments and Disclosure of Funding
This project was supported by funding from NSERC (Discovery Grants RGPIN-2022-05033 to E.B.M., and RGPIN--023-03875 to S.B.), Canada CIFAR AI Chairs Program and Google to E.B.M.. Canada Excellence Research Chairs (CERC) Program. Mila Quebec Al Institute, Institute for Data Valorization (IVADO), CHU Sainte-Justine Research Centre, Fonds de Recherche du Québec-Santé (FRQS), and Canada Foundation for Innovation John R. Evans Leaders Fund grant to E.B.M. This research was also supported in part by Digital Research Alliance of Canada (DRAC) and Calcul Québec.
10

---

References
Anurag Arnab, Mostafa Dehghani, Georg Hcigold, Chen Sun, Mario Lucic, and Cordelia Schmid. ViVIT: A Video Vision Transformer. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6816-6826, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-6654 org
2812-5. doi: 10.1 R0.11090CCV44922.00000 .00676. URL
document/ / 9713415/
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-Efficient Learning. In Proceedings of the 17th European Conference Computer Vision, ECCV 2022, April 2022. URL http:// /ttp::///axiiv.a// wtwpi.ivivv....../aaaa.aaaaaal.a wtt.///vorr//ooooooo /204. 07141 arXiv:220.007141 [cs, eess].
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Seff-Superviee Learning from Images with a Noint-Emmeddiig Predictive Architecture. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recogní- fion (CVPR), pages 15619- Vancouver, BC, Canada, June 2023. IEEE. ISBN 979-8-3503- 0129-8. doi: 10. 10.11009/ #ocument/102054
URL httph: / /tttps://ieeepplorre.eeee org/
Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. Frozen in Time: A Joint Video and Image Encoder for End-to-Ein tod Retrieval. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pages 1708-1718 1718, Montreal, QC, Canada, October 2021. IEEE ISBN 200.4448922.021.00777 URL httttsss::/eeeepppppp leee.org/documeee 97111155
978-1-6655-2812-5. doi: 10. 11///CCCV8922.2011......
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg:
Regular- ization for Self-Supervised Learning. International Conference on Learning Representations, 2022. /tppLttt//ppeeeeii..... hwtp::/wwwe..toottteooeee..e X
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Asso- ciates, Inc., 2020. URL /RRLLitppss/pprreensss...t.t B020//as/11557700666644444181883
wttpspppccpppppppppeeeeee..eee.e files/paper
Abst cobbstraac.htmm html
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc., 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self- Superviee Vision Transformers. In Proceedings Computer Vision, pages 9650-9660, 2021. httpeeeessssstteceeeeee thecef. com/ wttt:ot//teettccceocoooooo
of the IEFF/CVF International Conference http:////cootttoooot//////////e/ URL
Emerging Properties SateissnnnnnnSseeeeeeeeeeee
Se Stweerr-Superriseeeiieeeee oore
2021 paper
Ruchika Chavhan, Henry Gouk, Da Li, and Timothy Hospedales. Quality diversity for visual pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5384-53440 5394, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, pages 1597-1607. PMLR, November 2020. htttst ISSN: 2490-3498.
/ppooeediigg.co mlr wwt..mms...p..sseeeemem chen20 .htll
11

---

Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758.
2021. URL https: openaccess. / thecvf cww cctmm//conteeet/////2211111111 Siamese _ Represseiiattoooollammnnnnnoonn
Chen_ Exploring_simeer
paper html
Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Self-Supervised
Vision Transformers, August 2021. URI. httper//axxii..or////////1111
arXiv:2104.02057 [es].
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant Self Supervised Learning: Encouraging Equivariance in Representations. In International Conference Learning Representations, 2022. URL
net/ httpss://ppenr: http://caaaaaaaaaee..o......aaaa wat.te/ooooum.ddggLLLEEE..
Alexandre Devillers and Mathieu Lefort. EquiMod: An Equivariance Module to Improve Visual Instance Discrimination. In The Eleventh International Conference Learning Representations, September 2022. http:tts:::/// http///wnneeeeeeeeeeeeeeee / net wwwp//oororddddeiseeeeeer YFt
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, October 2020. URL https //openreview. wwtp:/oetttedmdiiiiiio..oo dNNTT
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With Little Help From My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for Self- Supervised Representation Learning. In Proceedings of the 38th International Conference on Machine Learning, pages 3015-3024, PMLR, July 2021. URL http/tttts//poceeetteeee ISSN: 2640-3498.
mll
press/ v139//3///e eroolo221 html
Marc Finzi, Max Welling, and Andrew Gordon Wilson. A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups. In Proceedings of the 38th
International Conference on Machine Learning, pages 3318-3328 PMLR, July 2021. URL
wmw.eeddiiiimmm.m.....mmmm.rmmmm mlr wtt..r..rees/v/1///iiee
html
ISSN: 2640-3498.
Quentin Garrido, Laurent Najman, and Yann Lecun. Self-supervised learning of Split Invariant Equiv- ariant representations. In Proceedings of the 40th International Conference on Machine Learning, pages 10975-10996. PMLR, July 2023. URL wtt/tttts://ppcceeeiiii ISSN: 2640-3498.
mlr. wwwpppsssvvOs.
garrido23b. html
Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, and Yann LeCun, Learning and Leveraging World Models in Visual Representation Learning, March 2024.
URL htth::/t/axxxi/// org/ abs//403..../// 00504 arXiv:2403.0055: [cs].
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos, February 2025.
abs/
1862111831 arXiv:2502. 11831 [cs].
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. February 2018 URL https:://penzeeveew www.:e...eeeeee..eee.ee.eeeeeeee hwt./frr.tfffoo
id=S1v4N210
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284. Curran Associates, Inc., 2020.
12

---

Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, lean-Basties Grill, Yunhao Tang. Michal Valko, Remi Munos, Mohammad Gheshlaghi Azar, and Bilal Piot. BYOL.Exppore:: Exploration by Bootstrapped Prediction. In Advances in Neural Information Processing Systems, volume 35, pages 31855- 31870, December 2022.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-Bastir Grill, Florent Altché, Remi Munos, and Mohammad Gheshlaghi Azar. Bootstrap PreecitPree Representations for Multi- task Reinforcement Learning. In Proceedings of the 37th International Conference Machine Learning, pages 3886. PMLR, November 2020. URL BRLhhtps:: URL /ptpp//ewoccc.g...e//o// mlr ISSN: 2640-3498.
sress/vi19/ gu0 html
Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structur- ing Representation Geometry with Rotationally Equivariant Contrastive Learning. In The
Twelfth International Conference on Learning Representations, October 2023. URL https 7 //openreview. net/ 1o
Sharut Gupta, Chenyu Wang, Yifei Wang, Tommi Jaakkola, and Stefanie Jegelka, In-Context Symmetries: Self-Supervised Learning through Contextual World Models. In The ehirty-eighth Annual Conference on Neural Information Processing Systems, November 2024. URL https
//openreview.oo net/ For.miidd-tttPPia-SS35
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable Guarantees for Self- Supervised Deep Learning with Spectral Contrastive Loss. In Advances in Neural Information Processing Systems, volume 34, pages 5000-5011. Curran Associates, Inc., 2021.
Karin L. Harman, G.Keith Humphrey, and Melvyn A. Goodale. Active manual control of object views facilitates visual recogmition. Current Biology. (322):1315-1318 November 1999. ISSN 09609822. doi: 10..001//000000-2222000000000000
URL
wwtgghhb..eseeeee. elsevier
com/ hwwp://eereeeee//1////6666222000
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 770-778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851 doi: 10.1 1009/ CVPR.2016.90. URL httt:::ttteeeppp//lla http:///w.......eoooooooooee///o
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9733 Seattle, WA, USA, June 2020. IEEE. ISBN 978-1-72817-168-33 doi: 1001009//V124220022000002. leee
URL htttps::/ieeexppooo/
L. Itti, C. Koch, and E. Niebur. A model of saliency- based visual attention for rapid scene analysis. IEEE Transactions Pattern Analysis and Machine Intelligence, 20(11):1254 1259, November 1998. ISSN 1939-3539, doi: 10.1109/34,730558.
URL/httss: URt/httss://eeeexp /ttp::/eeeeeeppeeeeee ieee. / Conference Name: IBEE Transactions on Pattern
document / 1230558/aarmmbbeee htt.:///eee/..../////mmmeeeeee 2araammmmeer:30555
Analysis and Machine Intelligence.
Khimya Khetarpal, Zhaohan Daniel Guo, Bernardo Avila Pires, Yunhao Tang, Clare Lyle, Mark Rowland, Nicolas Heess, Diana L. Borsa, Arthur Guez, and Will Dabney. Unifying Frame- work for Action-Conditiona Self-Predictive Reinforcement Learning. In The 28th Interna- tional Conference on Artificial Intelligence and Statistics, February 2025. URL https //openreview.rr net/ www.:ooonddmddcoioooo
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive Learning of Structured World Models, January 2020. UtLptttp://axxx wwtp:/wwwwwwww.ww///xxeweee 12247 arXiv:1911.122 12244 [cs, stat].
Yann LeCun. A Path Towards Autonomous Machine Intelligence Version 0.9.2, 2022-06-27, 2022. 2022 OpenReview.
Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving Transferability of Representations via Augmentation-Aware sule-vuuerviiion. In Advances in Neural Information Processing Systems, volume 34, pages 17710- 17722. Curran Associates, Inc., 2021.
13

---

Zhaoping Li. A saliency map in primary visual cortex. Trends in Cognitive Sciences, 6(1):
9-16. January 2002. ISSN 1364-6613, 1879-307X.
doi: 10. 10.0066/$1364-6611100011177777
URL https: / www. ce 11 com/ awrenddscogman www.nttiiessceennssssaattttttete
S1364-6613 018779 Publisher: Elsevier.
Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, and Koustuv Sinha. VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning, October 2024. / [es].
03478 arXiv:2410.03478
Akis Linardos, Matthias Kummerer, Ori Press, and Matthias Bethge. DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-an saliency modeling. In 2021 IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 12899-129088 Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-6654-2812-5 doi: 00.100/CCC44992.220.00000
URL
httts:://eeeepplee
eee org. wwt.or/g/oooooooo http:///dcmmmet//ceeo 97 9711473/
Bruce L. McNaughton, Francesco P. Battaglia, Ole Jensen, Edvard I. Moser, and May- May-Britt Moser. Path integration and the neural basis of the "cognitive map'. Nature Reviews Neuroscience, 7
(8):6636678, August 2006. ISSN 1471-0048. doi: 10. 10.10038nn1999 1932. URL https:/ /www nat nature. cww.commcaa....... a aaraiiiie es/ nrn 932
Publisher: Nature Publishing Group.
Ishan Misra and Laurens van der Maaten. Self-Supervised Learning of Pretext-Invariant Representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6707-6717 2020. URL https: httpp:://pppeaceeeeoeoo
thevvf com/ chtp///otteetttcccce CVPR 2020/htm1/Miee Self- Superiseee Learning of Pretext-Invariant Representations CVPR 2020 paper.html
Tianwei Ni, Benjamin Eysenbach, Erfan SeyedSalehi. Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Lue Bacon. Bridging State and History Representations: Understanding Ceff-Predictive
RI. In The Twelfth International Conference Learning Representations, 2024. URL [RL / //openreviee lew net/ www.eeeeeeeeeeeeeeeeeoee www.ooommdmmmssspaeoess
Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. Frozen Transformers in Language Models Are Effective Visual Encoder Layers. October 2023. URL https::/openreviieee
forum?1d=t0FT30665
Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan-Willem Van De Meent, and Robin Walters. Learning Symmetric Embeddings for Equivariant World Models. In Proceedings of the 39th International
Conference on Machine Learning, pages 17372-17389 17399. PMLR, June 2022. URL https://
proceedings mlr pwwps/eesss... 62 http//aak22aa.a html
ISSN: 2640-3498.
Michael I. Posner, Robert D. Rafal, Lisa S. Choate, and Jonathan Vaughan. Inhibition of return: Neural basis and function. Cognitive Newropsychollgy,, 2(3):211-228 August 1985. ISSN 0264- 3294, 1464-0627. doi: 10. 10.10800026431985502222266 URL http::/www. wttp://wwwwww.onn/ooooooooooo
com com//oo////// /ttp////6....1000/22222222222222
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S. Zimmermann, and Wieland Brendel. InfoNCE: Identifying the Gap Between Theory and Practice, April 2025, URL http:://axxv/ www.:iiv.rrgaeoo hbtpg//bb//4400 2411101177701303333 00143 arXiv:2407.00143 [cs].
Franz Scherr, Qinghai Guo, and Timoleon Moraitis. Self- Self-Supervised Learning Through Efference
Copies. Advances in Neural Information Processing Systems, October 2022. URL https
//openreview.rr net forum "oorumidddDttegtt//
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-Efficient Reinforcement Learning with Self- Celf-Predictive Representationss In International Conference Learning Representations, 2021. URL www./vvieeeeeeeeeeeeeeeee.eee.ee net. forum?
mowxr19thLabee 1ooVer119hhiaaDvt72mmmixx
Mehran Shakerinava, Arna Kumar Mondal, and Siamak Ravanbakhsh. Structuring Representations
Using Group Invariants. Advances Neural Information Processing Systems, volume 35, pages 34174, December 2022.
14

---

Vlad Sobal, Wancong Zhang, Kynghyun Cho, Randall Balestriero. Tim G. J. Rudner, and Yann LeCun. Learning from Reward-Free Offline Data: Case for Planning with Latent Dynamics Models, February 2025. BRL [es].
14819 1X1v:2502.14819
Yale Song, Gene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4D Goal-Step: Toward Hicrarchical Understanding of Procedural Activities. November 2023. URL
net/
/ Eorum? id
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Avila Pires, Yash Chandak, Remi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, András György, Shantanu Thakoor, Will Dabney, Bilal Piot, Daniele Calandriello, and Michal Valko. Understanding Self-Predictive Learning for Reinforcement Learning. In Proceedings of the 40th International Conference on Machine Learning, pages 33632-33656. PMLR, July 2023. URL ISSN: 2640-3498.
Sttts::/proceest wwt.ssmm.r..oom mlr wtwppppwssss/////22
html
Michael J. Tarr, Pepper Williams, William G. Hayward, and Isabel Gauthier. Three-dimensional object recognition is viewpoint dependent. Nature Neuroscience, 1(4):275-277, August 1998. ISSN 1546- 1726. doi: 10. 1038/1089. URL htt.::httts/w.... httt::::/www.nattre.om nwwure..r............ wwwo/aatti. httm:/arrtcccccnnneeee.. 275 Publisher: Nature Publishing Group.
es
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023. URL httttp::/arxiv..og//a/////////// 13971 arXiv:2302.1391 [cs].
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Con- trastive Predictive Coding, January 2019. URL hatoo//araav..orr//b//BB... arXiv:1807.03748 [es, stat].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
P. Vuilleumier, R. N. Henson, J. Driver, and R. J. Dolan. Multiple levels of visual object con- stancy revealed by event-related fMRI of repetition priming. Nature Neuroscience, 5(5):491-499, May 2002. ISSN 1546-1726. doi: 10. 10.1038/nn839. URL httpt::t///// hwwps:///wwww.. www.ww.waaa...... com/
a articles/nn88n nn/9
Publisher: Nature Publishing Group.
Jayun Wang, Yubei Chen, and Stella X Yu. Pose-aware self-supervised learning with viewpoint trajectory regularization. In European Conference on Computer Vision, pages 19-37. Springer, 2024a.
Yifei Wang, Kaiwen Hu, Sharut Gupta, Ziyu Ye, Yisen Wang, and Stefanie Jegelka. Understanding the Role of Equivariance in supersupe Learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024b. URL httptttss//ppeeeeeeeeee net/ heetfooze htt.:/oouummid-dLaddcdo
Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What Should Not Be Contrastive in Contrastive Learning. In International Conference on Learning Representations, 2021. URL
https::/
.et/ www.moorroooooooo forum=ii=C22BYNNauV
Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. De- coupled Contrastive Learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, pages 668 684, Cham, 2022. Springer Nature Switzerland. ISBN 978 77783-001--18009-0
doi: 10. 10.1107/978-3-000 0-188----3----00- 19809-0 38.
Thomas Edward Yerxa, Jenelle Feather, Fero P. Simoncelli, and Yeon Chung. Contrastive- Equivariant Self-Supervised Learning Improves Alignment with Primate Visual Area IT. November 2024. UtU.ttsss//eeeeeeeeeeeee/eerrree
15

---

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, pages 12310-12320. PMIR, July 2021. URL wtt.hhtts::/ htttpss://proceeedenee
m1 r.lr. wtt.pres//vl3// (btr:/99//ootraaaa .html
ISSN: 2640-3498.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. MERLOT: Multimodal Neural Script Knowledge Models, October 2021. URL
http:://arxta org httgg//b///ee wttab///15. 2106 02636 arXiv:2166.02636 [cs].
Li Zhaoping. The V1 hypothesis -reattiisseceattnngg bottom-up saliency map for preattentive selection and segmentation. In Understanding Vision, pages 189-314. Oxford University PressOxford, doi: 10. /00pp///oooo
I edition, May 2014. ISBN 9780199564668. 000.0000..00 URL BtRl:tttss///cammicccmmm oup. wtw. wwtp:/cc/coocccccceeeeeeoeee 9ttp/////cchhhccpe 154784147
978-0-19-1772500
A Implementation Details
A.1 Data preparation
3DIEBench. The original 256 x 256 images are resized to a 128 x 128 resolution for all experiments. Normalization is done using the means and standard deviations in Garrido et al. [2023), i.c., μ = [0.5016, 0.5037, 0.5060) and a [0.1030, 0.0999, 0.0969) for the three RGB channels, respectively,
CIFAR100, We use 32 32 images with normalization parameters typically used in the literature, i.e. /L = (0.4914, 0.4822, 0.4465] and a [0.247, 0.243, 0.261]. For data augmentation, we follow EquiMod's 's augmentation strategy.
Tiny ImageNet. The training set consists of 100000 ImageNet- Ik images from 200 classes (500 for each class) downsized to 64 x 64. The validation set has 50 images per class. We use normalization parameters μ = [0.4914, 0.4822, 0.4465) and / = [0.247, 0.243, 0.261]. ]. for the three RGB channels, respectively. For data augmentation, the kernel size of Gaussian blur adapted to the 64 x 64 images.
the same augmentation parameters CIFAR 100 with
STL10. In order to extract the saliencies, we resize images to 512 x 512, feed them to the pre- trained DeepGaze IIE [Linardos et al. 2021]. resize the output saliencies back to 96 x 96, and store them alongside original images. We use normalization parameters μ [0.4467, 0.4398, 0.4066), o = [0. 22,12 0.2215, 0.2239] for the three RGB channels, respectively. After sampling fixations from saliencies, the patches that extracted from the image to simulate foveation are 32 x 32 (compared to the full image size of 96 x 96). For loR, we zero-out a circular area with radius of 16 around each previous fixation.
Transformation parameters. For 3DIEBench, we use the rotation and color parameters provided with images for action conditioning as done in Garrido et al. 2023 For the augmentation setting, we use the parameters corresponding to each of the three augmentations and form the action as the relative augmentation vector between two images. For crop, and horizontal coordinate, and height and width. For color jitter, we use four variables: brightness, contrast, saturation, and hue. For blur, kernel. In predictive learning across saccades, the action is a 2-d vector, i.c., the normalized relative (3r, y) coordinate between two patches.
four variables, i.e., vertical
one variable: the standard deviation of the blurring
A.2 Training and evaluation details
Additional Training Details. We used the PyTorch framework for training all models. For experi- ments that use CIFAR100 and low-resolution S0 patches in predictive learning across saccades, we use the CIFAR variant of R8sNet-18. For models trained with AdamW, we used with default 31 and 82. a weight decay of 0.001, and a learning rate of x 10-4 with linear warmup for 20 epochs starting from 10-5 followed by a cosine decay back to 10-5 For models trained with Adam, we use the Adam optimizer with a learning rate of 10 default 31 and 3g. and no weight decay.
Protocols for training evaluation heads. For linear probing, we follow common SSL protocol and train linear classifier on top of frozen representations with batch size of 256 for 300 epochs
16

---

Method using the Adam optimizer with default hyperparameters. For action prediction, we follow a similar protocol as SIE [Garrido et al. 2023). Specifically, for rotation, color jitter, and crop. we train an MLP regressor with a hidden dimension of 1024 and ReLU activation for 300 epochs. For color (in 3DIEBench), blur (in the augmentation setting), and position (in predictive learning across saccades), we use linear regressor and train it for 50 epochs. For path integration experiments, the same regressor architectures as the equivariance evaluation heads are used, i.c., the MLP for angular path integration and the linear regressor for saccade path integration. All regression heads are trained using the Adam optimizer with default hyperparameters.
Hardware. Each experiment was run on single NVIDIA A100 GPU with 40GB of accelerator RAM.
Compute cost and FLOP analysis. We report pretraining compute time across methods (see Table All experiments are run under the same configuration: single A100 GPU, 128x 128 resolution, batch size 512, for 1000 epochs on 3DIEBench. seq-JEPA with sequence length of one has similar runtime other baselines (e.g., BYOL, SimCLR), while seq-JEPA with sequence length of three incurs moderate increase in wall-clock time (15,1 GPU-hours) due to encoding additional action-view pairs.
Table 5: GPU-Hour comparison across methods
A 100 pretraining GPU hours
SimCLR
11.4
BYOL
11.1
SIE
12.0
VICReg
10.9
EquiMod
11.8
seq-JEPA (train seq len 1)
12.3
seq-JEPP (train seq len
15.1
seq-JEPA's inference cost is primarily governed by two factors: sequence length and input resolution. Below, we report detailed FLOP counts (in Gigaflops) per forward pass of single datapoint across range of configurations (Note: these reflect inference-time cost).
Table 6: FLOPs for different sEP-JEPA configurations (in gigaflops)
Resolation Config
Eacoder only
Post-igeregater
seg
sey
soy len
seq
seg
Each view 178 128
0 1.82G (224) 0.15 (64)
0
1.24G
1.85
2,46G
4.9G
Views of 61 full image (224X224)
0.18G
0.34G
0.51G
0,67G
1.32G
Views of 84 vs. full image (234x(224)
1.82G (234) 0.30G (84)
0.33G
0.64G
0.95G
1,26G
2.510
We observe that for single-view inference at 128x 128 resolution, the encoder requires only 0.60G FLOPSs which matches the compute cost of standard SSL baselines such as SimCLR and BYOL. As sequence length increases, the additional compute cost grows sub-linearly as additional views should be encoded and aggregated. For example, lengths 2 and 3-where invariant performance improves substantially- -ncurr-nnur--- ~ 1.2G and ~ ~1,85G FLOPs, respectively. The cost can be offset by reducing input resolution. In saccade-based setup, we can feed low-resolution foveal glimpses (c.g., 64x64 or 84x84 84 crops sampled via saliency maps), which require only 0.15G-0.35 This enables the use of longer sequences at overall compute cost comparable to full-resolution, single-view pipelines (e.g., compare the FLOPs of aggregating eight 64 crops with encoding a single 224 x 224 image in the second row of Table 6).
FLOPs per view.
A.3 Architectural details
Below, we describe the architectural details and hyperparameters specific to each baseline.
We use a projection head with 2048-2048-242 2048 intermediate dimensions. The predictor has a hidden dimension of 512-d with ReLU activation. We use the same EMA setup outlined in the original paper [Guo et al. 2020), i.e., the EMA parameter starts from Thase 0.996 and is increased following cosine schedule.
BYOL.
17

---

SimCLR. We use temperature parameter of r = 0.5 with projection MLP with 2048-2048-200 intermediate dimensions.
VICReg. We use Anov Av 10, Ac and projection head with 2048-2048-2045 intermedi-
ate dimensions.
VrCRegTraa. We use the same architecture as VICReg and add a trajectory loss (Equation Wang et al. (2024al) with a coefficient Atraj = 0.01.
The architecture is the same BYOL, except that the predictor also receives the normalized relative transformation parameters. We use a linear action projector of 128-d and the same EMA setup BYOL.
Conditional BYOL.
SIE. For both invariant and equivariant projection heads, use intermediate dimensions of 1024- 1024-1024. For the loss coefficients, we use Ana = Xv = 10, Asqui = 4.5, and Ac 1. We use the hypernetwork architecture for all experiments.
SEN. We use temperature parameter of 0.5 with projection MLP with 2048-2048-206
intermediate dimensions.
EquiMod. We use the version based on SimCLR (both invariant and equivariant losses are con- trastive with r == 0.1 and have equal weights). The projection head has 1024- 1024-0022-22 128 intermediate dimensions. We use a linear action projector of 128-d.
ContextSSL. We use the pre-trained weights provided by the authors (trained for 1000 epochs) and follow their evaluation protocol on 3DIEBench.
seq-JEPA. For the sequence aggregator, we use a transformer encoder with three layers, four attention heads, and post-normalization. For the predictor, we use an MLP with a hidden layer of 1024-d and ReLU activation. The linear action projector in our default setting is 128-d. We use the same EMA setup as BYOL.
B Additional Experimental Results
B.1 Evaluation results on 3DIEBench for models conditioned on rotation and color
Table reports performance of seq-JEPA and several equivariant baselines when conditioned on both rotation and color in 3DIEBench. All methods suffer drop in classification performance and become highly sensitive to color. Similar performance degradations have been previously observed with 3DIEBench [Garrido et al.) 2023 Gupta et al. 2024]. though without clear explanation, One plausible explanation aligned with Principle II in Wang et al. [2024b], is that color 3DIEBench, ie., floor and light hue, weakly correlated with class labels in 3DIEBench (low class relevance). Therefore, foreing the encoder to encode color information would cause class information be lost, resulting in degradation of classification accuracy.
Table 7: Evaluation on 3DIEBench for rotation and color prediction (equivariance) and linear probe classification (invariance). All models are conditioned on both rotation and color.
Method
Classification (top-1)
Rotation pred. (R²)
Color pred. (R²)
SEN
82,17
0.29
0.96
EquiMod
82.58
0.27
0.95
Conditional BYOI.
81.95
0.38
0.94
75.34
0.46
0..7
seq-JEPA (Mr 4. M 5)
79.31
0.52
0.97
B.2 Results with a transformer projector instead of an MIP projector for baselines
By using transformer projector for baselines that normally use an MLP projector, we tested whether wrasformer-based projectors alone account for any performance gains. We did not see any benefit
18

---

from switching to transformer projectors in any of the baselines. 8 shows the transformer projector results for the baselines on 3DIEBench (plus sign indicates transformer projector).
Table 8: Comparison of baselines trained with transformer projectors 3DIEBench.
MettinglMetrie
Top-1 Acc
R2 (rel rot)
R (ahs rot)
SimCLR+
77.92
0,32
0,48
BYOL+
81.05
0.09
0.16
VICReg+
75.10
0.15
0.28
VICRegTraj+
77.82
0.24
0.36
SEN+
81.74
0.31
0.44
Cond. BYOL+
73.62
0.27
0.36
B.3
Out-of-distribution results
created an OOD set for 3DIEBench enable evaluation on unseen transformations. The original dataset samples rotations from (- m/2). Our OOD test set instead uses the disjoint range (- - U (T/2,T]] r), ensuring no angular overlap with the training set. Rendering this dataset single A6000 GPU. From Table 9 OOD rotation decoding (with R even reaching near -.0).. The sharp drop in R² values may be
that all methods fail at attributed domain shift in transformation geometry: while representations
took around five hours
well-aligned with in- distribution rotation trajectories (inside the range /2 they are not geometrically structured to extrapolate beyond this range to (- -1/2) U (T) Despite the failure in OOD equivariance generalization, Seq-JEPA exhibits a graceful degradation in classification accuracy compared to other baselines. We hypothesize that this robustness stems from its sequence-level aggregation mechanism: even when encoder representations become less equivariant under OOD transformations, aggregation across multiple views still filters out transformation-speeiff variability and recovers shared semantic content. Thus, seq-JEPA maintains object identity better under distribution shift, indicating that aggregation over input views supports invariance-demanding downstream tasks, even when equivariance is imperfect.
Table 9: Unseen Transformation Generalization (OOD Rotations)
Setting\Metric
Top-1 Ace (drop)
OOD R (rel rot)
OOD R (abs rot)
SimCLR
63.86 (-17.27)
-0.41
-0.201
BYOL
55.68 (-27.22)
-0.25
-0.198
VICReg
61.28 (-19.20)
-0.31
-0.206
SEN
60.03 (-23.40)
-0.41
-0.201
SIE
60.19 (-17.30)
-0.63
-0.200
EquiMoD
58.54 (-25.75)
-0.45
-0,206
Cond. BYOL
57.91 (-24.70)
-0.42
-0.202
VICRegTraj
62.94 (-18.32)
-0.36
-0.214
seq-JEPA (1,1)
61.53 (-22.55)
-0.69
-0.199
seq-JEPA (3,3)
65.03 (-21.11)
-0.71
-0.211
B.4 Equivariance predictor evaluation metrics
Following the protocol Garrido et al. [202]], we compute and report MRR, Hit@ 1, and Hit@5 metrics to evaluate predictor quality on the 3DIEBench validation set for seq-JEPA and two predictor- based baselines (Table [O). These results confirm that seq-JEPA achieves strong equivariance performance not only in terms of R², but also in top-rank retrieval metrics.
B.5 Training with more compute
To examine convergence under high-compute regime, we trained five models using 2000 epochs, 256 x 256 resolution, and batch size 1024. Each of these experiments on A100 GPUs (our
19

---

Table 10: Predictor evaluation metrics on 3DIEBench validation set.
Setting
MRR
H@1
H@5
SIE
0.319
0.215
0.404
EquiMod
0.136
0.037
0.186
seq-JEPA (1.1)
0.340
0.241
0.442
scq-JEPA (3.3)
0.388
0.273
0.468
main experiments were run a single A100), The evaluation results (Table confirm that seq- JEPA achieves a strong performance the high-compute regime without suffering from trade-off between invariance and equivariance. Importantly, our method achieves near-saturated performance already in the low-compute regime, indicating that it requires fewer steps for convergence and is less sensitive to input resolution and batch size than competing methods.
Table 11: Evaluation results under the high-compute regime.
Setting
Top-1 Ace
R² (rel rot)
R² (abs rot)
MRR
H@1
H@5
SIE
82,652
0.721
0.764
0.411
0.287
0.490
SimCLR
85.961
0.473
0.609
EquiMod
86.833
0.492
0.625
0.154
0.048
0.201
seq-JEPA (1,1)
85,370
0.661
0.713
0.365
0,263
0.447
seq-JEPA (3,3)
87,581
0.736
0.781
0.419
0.282
0.483
B.6
Complete evaluation results for linear probing on top of aggregate representations
We provide our complete evaluation results for linear probing on top of scq-JEPA's aggregate repre- sentations for our three transformation settings with different training and inference sequence lengths. Figure shows the top-1 accuracy 3DIEBench models conditioned on rotation. Figure 9 shows top-1 accuracy on STL-10 for models trained via predictive learning across saccades. Figures TO] and I1 show top-1 accuracy CIFAR 100 and Tiny ImageNet, respectively, with different types of action conditioning (crop, color jitter, blur, or all three). These heatmaps reflect the same trends observed in Figure and discussed in Section 4.6 illustrating the consistent effect of sequence length on representation quality.
84.1
81.2
82.0
82.0
85.0
84.9
85.3
85.2
85
85 4
85.3
86
86.3
86
86.1
(10)
85.5
87: 0
87.4
87.5
87.3
85 2
86 8
87
87.5
87.66
85.6
86
87.5
87.8
87.7
Trah Seg Lergth
Figure 8: JEq-JEPA's performance 3DIEBench with rotation conditioning; the heatmap shows linear probe accuracy on top of aggregate representations for different training and inference sequence lengths.
B.7 Comparison of evaluation results encoder representations and aggregate representations
For completeness, we provide linear probe classification on encoder representations for different transformation settings in for different inference evaluation lengths. The aggregate representation generally achieves much higher classification performance thanks to the architectural inductive bias in JEq-JEPA.
compare them with accuracy aggregate representations
20

---

64.7
72.6
72.7
71.5
70.7
86
78. 0
79.7
79.0
79.4
Vall Smy Lengh
68
80:
82.7
82.9
68
81.1
8:3.0
83
83.6
70
68
81.3
8:3.6
84.7
68
81.3
833.9
85.2
Figure 9: seq-JEPA's performance on STI-10 with predictive learning across saccades; the heatmap shows linear probe accuracy top of aggregate representations for different training and inference sequence lengths.
Crepr.atertttarr
Chop
10
07.5
67.8
58.5
15.0g
38.0
56.1
56.3
58.9
62.8
57.0
59.5
58.6
64
58.2
62.8
80
88.9
57.1
59.5
07.4
52
60.1
58.0
58.7
56.7
53.1
60.8
502
57
50.0
50 8
57.7
59.8
50.8
62
58.7
56.8
62.7
60.5
69.6
07.0
59.5
07.6
60.2
59.2
68.0
56.9
Leogth
Figure 10: seq-JEPA's performance on CIFAR100 with different types of action conditioning (crop, color jitter, blur, or all three); the heatmap shows linear probe accuracy top of aggregate represen- tations for different training and inference sequence lengths.
Cop--JleesTle
59
36
148 Seat Sesteru
33.5
35.2
M
34.9
37.1
34.7
349
36.7
34.2
35.8
$6.2
36.6
35.8
37.1
36.7
35.0
36.5
36.1
34.9
35.5
35.2
35.6
35.9
372
38
15.3
3
36.4
35.0
36.8
26:
36.3
36.
57.2
35.1
36.5
36.5
35.4
35.3
35.2
35.3
36.2
Leoggh
Leigh
Figure 11: SEq-JEPA's performance on Tiny ImageNet with different types of action conditioning (crop, color jitter, blur, all three); the heatmap shows linear probe accuracy top of aggregate representations for different training and inference sequence lengths.
B.8 Additional UMAP Visualizations
In Figure we visualize the UMAP projections of seq-JEPA representations trained on 3DIEBench without action conditioning. Similarly, Figure shows projections for models trained STL-10 via predictive learning across saccades, also without action conditioning. Compared to the action- conditioned counterparts (Figures and [3), these projections exhibit weaker smooth color gradients in their corresponding transformation-colon UMAP--ndicatinn reduced equivariance to transformation parameters.
B.9 Details of Path Integration Experiments.
While agent executes sequence of actions in environment, transitioning from initial state to final state, should capable of tracking its position by integrating its own actions. This is also crucial cognitive ability that enables animals estimate their current state in their habitat McNaughton et al. 2006). Here, we evaluate whether seq-JEPA is capable path integration. Given the sequence of observations {a,] 1M+1 generated from transformations {4+)++ and the corresponding relative action embeddings ( we define the task of path integration over the sequence of actions predicting the relative action that would directly transform zj to M+1 given
21

---

Table 12: Comparison of seq-JEPA classification performance across datasets and conditioning. Top-1 classification accuracy is reported for Zres and Zagg. with varying inference lengths Meent.
Dataset
Conditioning
Mi
Zres
Zagg
Mesutt
Monat
Mornl
3DIEBench
Nome
3
80.91
81.61
86.05
87.36
3DIEBench
Rotation
1
84.88
84.08
85.34
85.31
3DIEBench
Rotation
3
82.49
81.72
85.32
87.41
3DIEBBench
Rotation Color
74.88 4
71.14
75.97
79.31
CIFAR 100
None
2
53.00
51.60
57.05
58.37
CIFARION 100
Crup + Jitter Blur
1
56.23
54.24
52.90
52.92
CIFARION 100
Crop Jitter Blur
2
52.07
53.07
59.34
60.35
CIFARI00
Crop Jitter Blur
3
46.31
49.48
57.60
58.33
COF
Crop
52.62
52.06
58.07
59.32
COFARTO0
Color Jitter
3
54.92
50.21
57.20
58.62
CIFAR100
Blur
51.41
48.69
55.54
56.72
Tiny ImageNel
Nome
2
32.84
30.48
35.03
35,97
Tiny ImageNet
Crop Jitter Blur
1
33.03
32.34
36.14
37.07
Tiny ImageNet
Crop Jitter Blur
2
27.20
30.57
34.99
35.56
Tiny ImageNet
Crop Jitter Blur
3
24.74
28.84
33.78
34.68
Tiny ImageNet
Crop
31.13
30.89
35.07
35.69
Tiny ImageNet
Color Jitter
3
31.85
29.05
34.27
35.21
Tiny ImageNet
Blur
3
27.20
28.83
34.82
35.79
STL-10
None
4
61.38
62.21
69.06
70.45
STL-10
Position
4
81.20
71.45
81.53
83.44
STL-10
Position (no saliency)
79.29
63.14
76.93
79.85
STL.110
Position (no loR)
4
72.49
68.95
76.84
77.97
Rotation Angle (radians)
Class Labe
Class Label
Figure 12: 2-D UMAP projections of seq-JEPA representations on 3DIEBench without action conditioning
(M, 3, Meai 5). Encoder outputs color-coded hy class (left) and rotation angle (middle); aggregate
token representations color-coded by class (right).
ZAGG and aM-- In other words, given the aggregate representation of sequence of action-observation pairs and the action, we would like to predict the overall position change from the starting point (xi) to the end point (1M+1). We consider path integration for rotation angles in 3DIEBench and across eye movements with STL-10. For rotations, the task is integrating series of object rotations from the first view to the last, i.e. angular path integration. For eye movements, the task is integrating the eye movements from the first saccade to the last, i.c. visual path integration. To measure path integration performance for inference sequence length M, we train a regression head on top of the concatenation of AGG and as predict the transformation from xy M++1 Figure 6 shows that seq-JEPA performs well in both angular and visual path integration. The red curve corresponds the performance the original seq seq-JEPA. The blue curve corresponds to experiments in which the action embeddings are ablated (zeroed-out during inference for all views). The green curve corresponds experiments in which the encoder (visual) representations are ablated during inference. As expected, path integration becomes more difficult as the number of observations increases (red curves). Ablating action conditioning (blue curves) results failure of path integration. the other hand, ablating the visual representations (green curves) results only in small performance drop
22

---

Césa Like
Coontrate
Figure 13: 2-D UMAP projections of seq-JEPA representations STL-10 with action conditioning (Mr.
Ment 4), Encoder outputs are color-coded by class (top-left) by X/Y fixation coordinates (bottom); aggregate token is color-coded by class (top-right).
Figure 14: UMAP projections of seq-JEPA representations on STL-10 without action conditioning
(M. = M = 4). Encoder outputs are color-coded by class (Ieft) and fixation coordinates (middle); the aggregate token is color-coded by class (right).
compared to the original model, indicating that action conditioning is the key factor that enables path integration.
B.10 Transfer learning results on ImageNet-Ik
To evaluate generalization our model beyond STL-10, we assess transfer performance of the model trained on STL-10 patches via predictive learning across saccades ImageNet-1k. kk. We follow the same linear probing protocol as in-distribution evaluations: we freeze the ResNet and transformer encoder and train linear classifier on aggregate representations generated from foveated patches.
We extract saliency maps for ImageNet-1k images. Saliencies were extracted using DeepGaze IIE with the MIT 1003 centerbias prior [Linardos et al.) 2021]. Maps are saved at native resolution (matching the original ImageNet image dimensions) and normalized probability distributions, For our transfer learning experiments we resize the images and saliencies to 224 224, and sample sequences of patches sized 32 32 84 84 for this evaluation setting. Figure shows top- linear probe accuracy on ImageNet- 1k validation set across varying inference sequence lengths (Meat). For both patch sizes, performance improves with longer inference sequences, validating seq-JEPA's ability to benefit from extended context even in this difficult OOD ImageNet-1k setting. These results echo findings the main experiments for different training and inference sequence lengths and confirming the possibility of model's scalability in terms of data, parameter count, and compute.
23

---

20
Top-1 Acc (96)
Patch
Val Seg Length (Mox)
Figure 15: Linear probe transfer learning accuracy on ImageNet- Ik for two different patch sizes; the model is pre-trained on STL-10 via predictive learning across saccades.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims in the abstract and introduction (summarized by the items at the end of introduction) are supported by controlled experimental results on different benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The main limitations are discussed in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The work does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All details required to reproduce our results without access to the code provided or are included Sections 3.2 and 3.3 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code, date, and checkpoints have been made publicly available. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
des publ / guides/ CCodeSuom ss iisooolllic cy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. http:ssccccppccccccccccccccccc /ttliiccuiiiie gwts////eeeeeeeee////e//ee/ / CeeeeeoodSSSaee Submis ss cesssonPoliiei oly) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The most important experimental details necessary to understand the results
are discussed in the main text (Section 3) Other details are provided Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The main experimental results in Tables I and 3 are averaged over three
independent random seeds. For additional experiments, we pre-trained one model per setting due to compute limitations. Standard deviations are reported in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The hardware setup and approximate wall-clock time are provided in Ap- pendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /ttpa///aaiii///////aaoooooooooa http://wwwpppppoooooooo/ccctttta
Answer: [Yes]
Justification: The paper complies with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper proposes a method and architecture that is not closely tied to any societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper is not releasing such data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data used in our experiments open-source. We credit and cite the datasets and methods in the paper when necessary.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We use STL-10 and ImageNet- -K saliency maps and distr--distiiuution set of 3DIEBench. We have made all the three publicly available
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pettp:://eerrips.cc//oooott LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "RZnMynR7ht", "reviewer": "Reviewer_Drps", "summary": "The submission introduces a novel self-supervised learning approach, referred to as seq-JEPA, that is able to learn a combination of invariant and equivariant representations of vision data. This is accomplished via architectural constraints, rather than through the use of a loss function that promotes these properties. The experimental evaluation demonstrates that the representations learned via this approach are useful for both determining the relative pose between images of the same object (presumably via equivariant features) while also being effective for classification (presumably via invariant features). There is also some investigation into the role of sequence lengths used in the training and inference process as well as the role of action conditioning.", "questions": "Can you provide more detail about how the actions are represented? This was not very clear in the manuscript.", "limitations": "Limitations are discussed in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is quite well-written. I found it quite easy to follow what was going on, for the most part, but I do have one clarification question below.\n\nThe proposed approach is quite simple; different transformations of the same image are embedded into a feature space, paired with a transformation description, serialised into a sequence, and fed into a transformer. The goal is to predict the transformation description of the final image in the sequence. This avoids much of the complexity associated with some SSL pipelines.\n\nThe tasks used to evaluate the proposed approach are reasonably sensible; there is some evidence from prior work to suggest that invariance and equivariance are important for object recognition and camera pose estimation, respectively. I am less familiar with the experiment related to human vision, but from a non-specialist point of view this seems fine. The ablations answer the main questions I had about designs of the method, so these are also appreciated.\n\nI would have liked to see more discussion and quantitative comparison with other work that has aimed to develop representation learning approaches for similar settings. E.g., [A] develop a method for training a single feature extractor that is able to encode both invariance and sensitivities to various transformations. It would be good to understand to what extent the current work is trying to solve a similar problem, and how the solutions match up.\n\n[A] Chavhan et al. Quality Diversity for Visual Pre-Training. In ICCV, 2023.", "was_revised": true, "final_justification": "My overall assessment of the paper (somewhat positive) is unchanged by the authors response.", "rebuttal": "# Response to Reviewer 1:\n\nWe thank the reviewer for their constructive comments and for highlighting the clarity and simplicity of our approach. We are pleased that you found the method easy to follow and appreciated the ablations and task selection. Below, we address your main concerns and clarify your question about the representation of actions.\n\n---\n\n## **Connection to [A]**:\n\nWe thank the reviewer for pointing out the relevant work of Chavhan et al. [A], which indeed bears a conceptual connection to ours. In [A], the authors use an ensemble of heads trained on top of a base encoder (MoCo) to span a diverse spectrum of transformation sensitivities across the latent spaces of each head (encouraged via a diversity loss). Downstream probes then learn to linearly combine these feature heads depending on the desired invariance-equivariance trade-off.\n\nWhile the motivation is related, seq‑JEPA offers a fundamentally different and more architectural approach. In our framework, equivariant and invariant representations emerge within a single model, with no need for additional heads or specialized loss functions. Specifically, the encoder produces equivariant representations, while the sequence aggregator (transformer) forms an invariant representation by aggregating views. This dual structure arises purely from our architectural inductive bias and predictive learning objective, which enables seq-JEPA to navigate the invariance-equivariance trade-off through its core architectural design, without requiring explicit regularization or auxiliary heads.\n\nWe will add a discussion of this connection in the final version to better position our work in the landscape of hybrid invariant-equivariant representation learning.\n\n---\n\n## **Clarification on Action Representation**:\n\nAs noted in Section 2.2 of the paper, the relative transformations (actions) \\$a_i$ are defined as the transformation that maps \\$x_i$ to \\$x_{i+1}$, i.e.,\n\\$a_i := t_{i → i+1}$.\n\nThe specific instantiation of the action vector depends on the transformation setting. We have provided these details at the end of Appendix A, but we will clarify this more explicitly in the main text in the final version of the paper:\n\n**3DIEBench (Rotations):**\nWe use the 4D quaternion vector representing the relative rotation between two views, consistent with the original benchmark [B].\n\n**Predictive Learning Across Saccades:**\nThe action is a 2D vector indicating the normalized relative (x, y) position between the centers of two patches, simulating a saccadic eye movement.\n\n**Hand-Crafted Augmentations (Crop, Jitter, Blur):**\nEach augmentation is parameterized as follows, and the corresponding action vector a_i is computed as the difference in parameters across views:\n\nCrop: [x, y, height, width] (4D)\n\nColor jitter: [brightness, contrast, saturation, hue] (4D)\n\nBlur: [kernel standard deviation] (1D)\n\nIn all settings, the action vector is passed through a learnable linear projector (default: 128D), then concatenated to the visual representation before being passed to the transformer aggregator.\n\n---\n\nWe again thank the reviewer for their thoughtful feedback and hope the above clarifications, along with the related responses to other reviewers, help further contextualize our design choices and contributions.\n\n---\n\n**References:**\n\n[A] Chavhan, Ruchika, et al. Quality Diversity for Visual Pre-Training. ICCV 2023.\n\n[B] Garrido, Quentin, Laurent Najman, and Yann Lecun. Self-supervised learning of split invariant equivariant representations. arXiv preprint arXiv:2302.10283 (2023)."}, {"review_id": "2DZ3DJ5nqY", "reviewer": "Reviewer_hDF5", "summary": "The authors propose a method to create representations that are both equivariant or invariant using a novel self-supervised architecture. They achieve this by augmenting input images with a sequence of transformed views paired with the corresponding transform taken to obtain them. By using both the image and the corresponding action taken in a JEPA-like approach, they manage to train a model that is both equivariant and invariant without the need to use any auxiliary losses. They validate that this is indeed the case using linear probes and they compare to standard baselines in both equivariant and invariant settings.", "questions": "As mentioned above is it not clear to me why does the model keep the action information when aggregating. I assume this is because it makes predicting the next representation easier during training. Can the authors provide confirmation of this or whatever reason they hypothesized is behind this?", "limitations": "The authors appear to have addressed or acknowledge any limitation I can think of.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# \n\nStrengths:\n\n- Problem is well defined and relevance clearly established.\n- Architecture idea is intuitive and easy to understand, with a clear explanation of the different components.\n- Experiments are well described and the models being compared to relevant to the hypothesis.\n- Evaluation protocols are also clear.\n\nWeakness:\n\n- Reporting just the evaluation scores (e.g as in Table 1) clearly establishes that the model retains much more information about the transformations (i.e. becomes equivariant), but doesn't establish exactly why this is. Presumably this is because of the self-supervised training, but some more data would be nice.", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 2:\n\nWe thank the reviewer for their thoughtful and encouraging feedback. We are glad you found the problem setup well-defined, the architecture intuitive, and the evaluations thorough. Below, we address your main question and provide our intuition for how equivariant and invariant representations emerge in seq-JEPA—namely, why the encoder retains action information (equivariance) and how the aggregator yields invariant representations.\n\n---\n\n## Why does seq-JEPA retain action information and develop equivariance?\n\nWe begin with some background. In contrastive SSL using InfoNCE, it has been shown that, under certain assumptions, the learned representations $z$ and $z'$ can recover the data-generating factors of observations $x$ and $x'$ [A]. However, this recovery only holds for factors *shared* across the pair, meaning factors that are invariant (e.g., object category). Transformation- or action-specific latents (e.g., pose or viewpoint) are, by design, not captured.\n\nWhile non-contrastive predictive methods (e.g., BYOL, JEPA variants—including seq-JEPA) do not offer formal identifiability guarantees, they implicitly solve a latent-space world modeling task, aiming to model transitions of the form $p(z' \\mid z, a)$. If the model does not have access to the action $a$, it must implicitly marginalize over it—leading to greater ambiguity and a broader solution space. In contrast, when $a$ (or a reliable estimate; see our response to Reviewer 4) is available, the encoder is incentivized to preserve action-relevant information. This makes the conditional prediction $p(z' \\mid z, a)$ more tractable and structured (action-conditioning makes it easier to predict the next state).\n\n---\n\n## Why does aggregation induce invariance?\n\nGiven the above, seq-JEPA exhibits two other emergent behaviors:\n\n**First**, the aggregator naturally promotes invariance. It processes multiple transformed views (e.g., rotated object images) whose representations orbit around the category center in latent space (Figure 5, left panel). By attending to and averaging these, the aggregator filters out transformation-dependent components while reinforcing shared content—i.e., invariant factors. As a result, the aggregated representation gravitates toward the center of the class cluster, reducing intra-class variance and increasing inter-class separability in the aggregated latent space (see Figure 5, right panel).\n\n**Second**, equivariance improves by observing longer sequences of action-observation pairs during training. Feeding additional tuples $(z_i, a_i, z_{i+1})$ into the transformer-based aggregator creates a richer context. This aggregated context, denoted $z_{\\text{agg}, t}$, is passed to the predictor, which learns to jointly encode $z_{\\text{agg}, t}$ and the next action $a_t$ in order to predict the future state $z_{t+1}$. With access to a longer sequence—i.e., more transitions in working memory and a richer context—the predictor becomes better at capturing how specific actions influence latent dynamics. This enables it to more accurately approximate the transition distribution $p(z_{t+1} \\mid z_{\\text{agg}, t}, a_t)$. Because accurate prediction requires the model to preserve and utilize information about $a_t$, the encoder is implicitly encouraged to learn structured, more equivariant representations. Empirically, we indeed observe that increasing the training sequence length leads to improved equivariance performance (see Figure 7, left panel).\n\n---\n\n## Emergence vs. enforcement\n\nWe emphasize that the emergence of both invariant and equivariant representations in seq-JEPA is not enforced explicitly, but rather emerges from its predictive world modeling objective and inductive biases. Our discussion above clarifies how components like action conditioning, sequential aggregation, and predictive modeling each contribute to this separation.\n\nNotably, the model is not *constrained* to learn this decomposition. In principle, it could converge to a trivial solution by marginalizing over actions and learning a fully invariant world model. However, this does not occur in practice. Instead, the design of seq-JEPA naturally encourages the encoder to retain transformation-sensitive information (to predict transitions), while the aggregator averages over such views (to form invariant summaries). We view this emergent disentanglement as a key strength of the method and a promising direction for further investigation.\n\n---\n\n**Reference:**\n\n[A] Zimmermann, Roland S., et al. *Contrastive learning inverts the data generating process*. ICML 2021."}, {"review_id": "1SvvF7Btsp", "reviewer": "Reviewer_eSzt", "summary": "The authors present a self-supervised representation learning approach that utilises a sequence of inputs each encoding different transformation state. Representations of each state are then enforced through a sequential transformer architecture that represents sequential action states by their relative changes to transformation parameters. The authors demonstrate superior invariant (top-1 classification) performance on a variety of benchmark datasets, as well as reporting transformation (relative and individual) prediction performance. The authors also present different sampling schemes for the sequential transformations, which present interesting findings, notably in regard to predictive learning across saccades, which arguably will motivate future work.", "questions": "- The rotation performance of SIE in table 1 is much lower than reported in their work. Can you explain this drop in performance?\n- Can you clarify in the tables which results are obtained with the MLP or Transformer projector networks for the benchmarks?\n- Have the authors explored the performance on unseen rotations / transformations?\n- What is the additional compute overhead compared to prior works, this would help demonstrate the performance-efficiency trade-off.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n+ Taking advantage of sequential transformation is an interesting direction, and the proposed method is simple yet effective. The proposed methodology aligns with speculative work, and makes good contributions to the direction of this research area.\n+ The sequence length investigation presents interesting results that are highly impactful in the community, notably when capturing representations that represent rotation and translations. This confirms and supports conjectures of other works that advocate for sequence learning for world-models and representation learning.\n+ Results demonstrate strong invariant and transformation prediction performance, reducing the trade-off demonstrated by previous works.\n+ I particularly like the honest and in-depth limitations and broader impact, it is nice to see you address the concerns around the use of known transformation groups.\n+ The predictive learning across saccades is a neat method to enforce such equivariance, while the performance is not entirely competitive, these findings motivate future work and studies. \n\n**Weaknesses**:\n- While the authors convincingly demonstrate the invariance of the learned representations. They do not fully substantiate the claim of equivariance, however, experiments should go beyond predicting the transformation parameters. While predicting the rotation is a useful task, it does not on its own guarantee that the representations are truly equivariant. A more direct analysis, which confirms that the entire representation vector transforms predictably with the input transformation, would provide stronger evidence for the models equivariance.\n- Some of the implementation details are not entirely clear, notably are the transformation sequences a natural progression of actions, for example a step-by-step iterations of a fixed rotation amount or random transformations structured as a sequence? \n- There is no direct analysis of representation equivariance. While the R2 rotation prediction acts as a proxy this is not a measure of equivariant \n- Some benchmark results (notably, SIE) seem surprisingly low compared to their original reporting, this raises concerns about the implementation and comparisons, however, if the authors address this in the rebuttal this would add clarity.\n- Other works such as [1] utilise a multi-view loss, and while not sequential like the method in question, is a good analogue for comparison, and arguably more comparable.\n\n[1] Wang, J., Chen, Y. and Yu, S.X., 2024, September. Pose-aware self-supervised learning with viewpoint trajectory regularization. In European Conference on Computer Vision (pp. 19-37). Cham: Springer Nature Switzerland.", "was_revised": true, "final_justification": "I have maintained my already positive score given the additional evaluations have been presented during the rebuttal.", "rebuttal": "We thank the reviewer for their thoughtful and detailed review. Below, we address your questions and concerns.\n\n---\n\n### **On Substantiating Equivariance Beyond Transformation Prediction**\n\nWe agree that demonstrating full functional equivariance—where the representation transforms in a predictable manner under input transformations—is stronger than transformation regression alone. Regression is often used since functional equivariance is hard to quantify in high dimensions. Therefore, our evaluation follows the established practice in equivariant SSL, where transformation prediction is used as a practical proxy for equivariance [B–D].\n\nTo go beyond this, we also provided **qualitative evidence in Fig 5 (left and mid panels)**, which shows that UMAP projections of encoder representations form smooth, structured orbits around object category centers, with a continuous gradient across the rotation angle (see the visible alignment between the red band in class-based coloring and its smooth rotation-based coloring in Fig 5). This behavior is consistent with a structured, transformation-aware latent geometry and offers indirect evidence of equivariance.\n\nThat said, we agree that **direct testing** (e.g., using linear equivariant maps or commutativity checks with group actions) would provide more rigorous validation. We consider this a valuable direction for follow-up work, especially in the context of latent world modeling where group structures may be implicit, partial, or noisy.\n\n---\n\n### **Clarifying Transformation Sequence Generation**\n\nIn our current setup, transformation sequences are externally defined and fixed during training—there is no learned policy over actions. Specifically:\n\n- **3DIEBench**: Each object is rendered with a random sequence of 3D rotations, sampled uniformly from $(-\\pi/2, \\pi/2)$ across axes. These sequences are independent and diverse.\n\n- **PLS**: Saccade trajectories are generated using saliency maps with inhibition-of-return (IoR), producing structured, non-repetitive sequences over natural STL-10 images. This simulates foveated active vision and enables real-world applicability.\n\n- **CIFAR-100 / TinyImageNet**: Sequences are composed from standard data augmentations (crop, jitter, blur) with known parameters. “Actions” are defined as the difference between consecutive augmentation parameters.\n\nThus, our framework handles both randomized and structured transformation regimes.\n\n---\n\n### **On the SIE Performance Drop**\n\nThank you for flagging this. The reported SIE performance drop is due to differences in hyperparameter regimes. The original SIE paper trained with: 2000 epochs, 256×256 input resolution, and batch size 1024.\n\nIn contrast, as noted in Section 3 of our paper, we standardized training across all baselines (including SIE) to: 1000 epochs, 128×128 resolution, and batch size 512. This was necessary to enable a fair comparison across baselines within our compute budget.\n\n---\n\n### **Comparison to [A]**\n\nWe thank the reviewer for suggesting this comparison. [A] proposes a regularization that constrains adjacent views (e.g., with small pose deltas) to lie on smooth trajectories in latent space. This reflects a “local linearity” prior, which is well-motivated for passive or smooth environments (e.g., robotic arms, camera pans).\n\nIn contrast, **seq-JEPA is designed for active perception settings**—where transitions may be sparse, discontinuous, or highly non-local (e.g., long-range saccades, or abrupt object rotations). It assumes no trajectory smoothness or small-angle assumptions.\n\nWe empirically compare [A] by adding it as a baseline across  three datasets. We implemented their best-performing model variant (VICReg + trajectory regularization) and tuned the regularization strength $\\alpha$ on 3DIEBench across values {0.001, 0.01, 0.1, 1.0}. We found $\\alpha = 0.01$ to yield the highest top-1 accuracy and rotation prediction and used it for the other two datasets. Results are given below:\n\n| Setting\\Metric | Top-1 acc | R² (relative rot) | R² (abs rot) |\n|----------------|------------|-------------------|----------------|\n| 3DIEBench      | 81.26      | 0.27              | 0.43           |\n\n| Setting\\Metric | Top-1 acc | R² (crop) | R² (jitter) | R² (blur) |\n|----------------|------------|-----------|-------------|------------|\n| CIFAR100       | 61.07      | 0.46      | 0.14        | 0.02       |\n| Tiny ImageNet  | 34.95      | 0.28      | 0.09        | 0.16       |\n\nWe see that trajectory regularization indeed improves over VICReg (compare with tables 1 and 2 in the paper), which shows that imposing geometric priors can improve both invariant and equivariant performance (even when the priors do not fully materialize in the environment, e.g., when we have non-smooth angle changes as in 3DIEBench).\n\n---\n\n### **Unseen Transformation Generalization (OOD Rotations)**\n\nSince 3DIEBench does not include an OOD test set, we created one ourselves to enable evaluation on unseen transformations. The original dataset samples rotations from $(-\\pi/2, \\pi/2)$. Our OOD test set instead uses the **disjoint range** $(-\\pi, -\\pi/2) \\cup (\\pi/2, \\pi)$, ensuring no angular overlap with the training set.\n\nThis makes the task **strictly extrapolative**, unlike [A], which evaluates models on denser but still interpolative pose samples from within the same angular support.\n\nRendering followed SIE’s protocol and took ~5 hours on a single A6000 GPU. We will release this benchmark publicly.\n\nFrom the table below, we see that all methods fail at OOD rotation decoding (with R²s even reaching near −1.0). The sharp drop in R² values may be attributed to a domain shift in transformation geometry: while representations are well-aligned with ID rotation trajectories (inside the range \\$(-\\pi/2, \\pi/2)$), they are not geometrically structured to extrapolate beyond this range to \\$(- \\pi, - \\pi/2) \\cup (\\pi/2, \\pi)$. The near −1.0 R² values likely result from a reversal in regression sign: the learned probe fails to interpret latent geometry correctly in the unseen region and predicts anti-correlated values. This supports our interpretation that the latent space does not generalize to novel angular regions, despite appearing structured within the ID range.\n\nDespite the failure in OOD equivariance generalization, seq-JEPA exhibits a graceful degradation in invariant classification accuracy compared to other baselines. We hypothesize that this robustness stems from its sequence-level aggregation mechanism: even when latent representations become less equivariant under OOD transformations, aggregation across multiple views still filters out transformation-specific variability and recovers shared semantic content. Thus, seq-JEPA maintains object identity better under distribution shift, indicating that aggregation over input views promotes invariance, even when equivariance is imperfect.\n\n| Setting\\Metric | Top-1 Acc (drop) | OOD R² (rel rot) | OOD R² (abs rot) |\n|----------------|-------------------|-------------------|-------------------|\n| SimCLR         | 63.86 (-17.27)    | -0.41             | -0.201            |\n| BYOL           | 55.68 (-27.22)    | -0.25             | -0.198            |\n| VICReg         | 61.28 (-19.20)    | -0.31             | -0.206            |\n| SEN            | 60.03 (-23.40)    | -0.41             | -0.201            |\n| SIE            | 60.19 (-17.30)    | -0.63             | -0.200            |\n| EquiMoD        | 58.54 (-25.75)    | -0.45             | -0.206            |\n| Cond. BYOL     | 57.91 (-24.70)    | -0.42             | -0.202            |\n| VICRegTraj     | 62.94 (-18.32)    | -0.36             | -0.214            |\n| seq-JEPA (1,1) | 61.53 (-22.55)    | -0.69             | -0.199            |\n| seq-JEPA (3,3) | 65.03 (-21.11)    | -0.71             | -0.211            |\n\n---\n\n### **Clarifying MLP vs. Transformer Projectors**\n\nTraining with a transformer projector (for models originally with an MLP projector) was designed to isolate the contribution of model capacity by controlling for architectural differences in the projection head. Specifically, we tested whether transformer-based projectors alone account for any performance gains. We did not see any benefit from switching to transformer projectors in any of the baselines and the results in Tables 1 and 2 reflect original MLP-based models. We will clarify this in the paper and include the full transformer-projector results in the appendix. Below, you can see these results for the baselines on 3DIEBench (plus sign indicates a transformer projector).\n\n| Setting\\Metric | Top-1 Acc | R² (rel rot) | R² (abs rot) |\n|----------------|------------|----------------|----------------|\n| SimCLR+        | 77.92      | 0.32           | 0.48           |\n| BYOL+          | 81.05      | 0.09           | 0.16           |\n| VICReg+        | 75.10      | 0.15           | 0.28           |\n| VICRegTraj+    | 77.82      | 0.24           | 0.36           |\n| SEN+           | 81.74      | 0.31           | 0.44           |\n| Cond. BYOL+    | 73.62      | 0.27           | 0.36           |\n\n---\n\n### **Time and Memory Comparison**\n\nWe kindly refer the reviewer to our FLOP and GPU-hour comparisons in the response to Reviewer 4. In summary, seq-JEPA maintains reasonable compute overhead (e.g., 12.3G–15.1G GPU-hours vs. 11–12.5G for baselines), and can amortize sequence cost by using low-resolution glimpses (e.g., a sequence of small foveated crops from a large image), which are viable in embodied vision (as exemplified in our PLS setup, where low-resolution glimpses support effective sequential aggregation).\n\n---\n\n**Refs:**\n\n[A] Pose-aware Self-supervised Learning with Viewpoint Trajectory Regularization\n[B] Self-supervised Learning of Split Invariant Equivariant Representations\n[C] Learning and Leveraging World Models in Visual Representation Learning\n[D] Contrastive-equivariant SSL improves alignment with primate visual area it"}, {"review_id": "Xo6auX8Xc7", "reviewer": "Reviewer_WMFR", "summary": "This paper proposes seq-JEPA, a network architecture that simultaneously learns two representations, for equivariance and invariance respectively. This is achieved by encoding the relative transformation between two views and concatenate with the original encoding. Numerical experiments on benchmark datasets show the proposed architecture outperforms other methods designed only for either equivariance of invariance.", "questions": "1. Can you clarify on your architecture's reliance on having ground truth transformations, and whether such information is provided in all experiments? How realistic is this in practical applications?\n2. It would be interesting to closely examine the two learned features for invariance/equivariance separately, to understand how well each feature is indeed trained to perform the corresponding task.\n3. Please provide the time and memory cost comparison between seq-JEPA and other baseline methods.", "limitations": "The usage of ground truth transformations need to be clarified.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n1. The paper proposes an interesting perspective of unifying invariance and equivariance in network architecture, and proposal a novel architecture achieving both goals.\n2. Experiments show decent performance on both invariance and equivariance tasks.\n\nWeakness\n1. The relative transformations $a_i$ are fed as input to the model, but in practice this is unknown, which is a major limitation of the architecture.\n2. The experiments are conducted on synthetic or artificial datasets, and do not show the general applicability of the proposed architecture. This is related to the requirement of ground truth transformation info mentioned above.\n3. The paper does not provide new theoretical insights. While this is not a requirement for every paper, the limited practical applicability of the proposed architecture undermines the contribution of this paper.", "was_revised": true, "final_justification": "I raised my review score to 4, as the authors' response addresses the majority of my previous concerns, especially regarding the assumption of known transformations and synthetic experiments.", "rebuttal": "# Response to Reviewer 4\n\nWe thank the reviewer for their thoughtful comments, and for highlighting the importance of grounding our assumptions, clarifying applicability, and assessing computational overhead. We address each of your concerns below.\n\n---\n\n### **On Access to Relative Transformations**\n\nWe respectfully disagree with the claim that requiring access to relative transformations (actions) is a “major limitation” of our method. Our design is both consistent with prior work and motivated by embodied learning principles.\n\nFirst, the assumption of known or controlled transformations is standard in the equivariant SSL literature. Many works in this space assume access to transformation/action parameters [A–C] or to at least two pairs of views where the same transformation/action is applied [D, E]. Our setting is consistent with this standard practice in the literature and follows this precedent by modeling the transition \\$ p(z_{t+1} \\\\mid z_t, a_t)$, where \\$a_t$ is either known or can be approximated.\n\nSecond, as noted in the Introduction (see paragraph 3 in Intro), our design is grounded in the embodied learning perspective, where perception is inherently active. In such cases (e.g., robotic manipulation, foveated vision, embodied agents in simulation) the agent performs the actions itself and therefore has access to them. From a neuroscience perspective, this setup aligns with the concept of efference copy (or corollary discharge), i.e., an internal copy of motor commands sent to both the motor system and sensory areas to anticipate the consequences of self-generated actions. This mechanism enables animals to predict sensory inputs conditioned on their own actions, and we adopt the same inductive bias in seq-JEPA.\n\nIn domains where the agent does not execute the actions itself (e.g., passive observation or multi-agent settings), estimating the actions becomes necessary. In this case, the model can either be given approximate action estimates (e.g., via egomotion sensors or inverse modeling), or extended to jointly infer latent actions along with representations by observing other agents. This opens a natural direction for future work; building theory-of-mind-capable agents that model others’ actions and policies from observational data.\n\n---\n\n### **On Dataset Scope and Practical Applicability**\n\nWhile 3DIEBench involves synthetic renderings, our experiments are not limited to artificial domains. We also evaluate seq-JEPA on CIFAR-100, TinyImageNet, and PLS (Predictive Learning across Saccades on STL-10), all of which use natural images. In these settings, the “action” is either:\n\n- The relative augmentation parameters (e.g., crop coordinates, jitter levels, blur kernel size), which are available in any augmentation pipeline.\n\n- The relative positions between foveated glances, sampled with biologically plausible saliency and inhibition-of-return (IoR) heuristics (in PLS), which simulate human-like visual behavior.\n\nOur results on these settings confirm that our approach generalizes beyond synthetic 3D datasets, and operates effectively in low-resolution, weakly-structured, or biologically grounded regimes.\n\n---\n\n### **On Separability and Interpretation of Learned Representations**\n\nWe appreciate the request for a closer analysis of the learned equivariant and invariant features. Our probe-based results in Table 1 already reveal this division: the encoder (pre-aggregation) preserves action-relevant information (high R² in transformation prediction), while the aggregator (post-aggregation) supports invariant object classification (high top-1 accuracy).\n\nAdditionally, we include UMAP visualizations in Figure 5, which show this duality in latent space. The left and middle panels display structured orbits of transformed views around class centers in the encoder space—indicating equivariance. The right panel, which shows post-aggregation representations, demonstrates class-wise collapse and increased inter-class separability, confirming that the aggregator learns to filter transformation-specific variation. Finally, the middle panel, colored by rotation angle, confirms that encoder orbits are transformation-aware: latent trajectories evolve smoothly with rotation, and are consistent across object instances (e.g., compare the red cluster in the left panel with the corresponding cluster in the middle panel).\n\n---\n\n### **On Compute and Memory Overhead**\n\nWe report pretraining compute time across methods (see table below). All experiments are run under the same configuration: a single A100 GPU, 128×128 resolution, batch size 512, for 1000 epochs on 3DIEBench. seq-JEPA with a sequence length of one has similar runtime to other baselines (e.g., BYOL, SimCLR), while seq-JEPA with sequence length of 3 incurs a moderate increase in wall-clock time (15.1 GPU-hours) due to encoding additional action-view pairs.\n\n#### **GPU-Hour Comparison Table**\n\n| Method                        | A100 pretraining GPU hours |\n|------------------------------|-----------------------------|\n| SimCLR                        | 11.4                        |\n| BYOL                          | 11.1                        |\n| SIE                           | 12.5                        |\n| VICReg                        | 10.9                        |\n| EquiMod                       | 11.8                        |\n| seq-JEPA (train seq len = 1) | 12.3                        |\n| seq-JEPA (train seq len = 3) | 15.1                        |\n\n---\n\nseq-JEPA’s inference cost is primarily governed by two factors: sequence length and input resolution. Below, we report detailed FLOP counts (in Gigaflops) per forward pass of a single datapoint across a range of configurations (Note: these reflect inference-time cost):\n\n#### **FLOPs by Configuration**\n\n| Resolution / Config                                   | Encoder (pre-aggregator) | Post-aggregator (seq len = 1) | Post-aggregator (seq len = 2) | Post-aggregator (seq len = 3) | Post-aggregator (seq len = 4) | Post-aggregator (seq len = 8) |\n|--------------------------------------------------------|---------------------------|-------------------------------|-------------------------------|-------------------------------|-------------------------------|-------------------------------|\n| Each view is 128×128                                   | 0.60G                     | 0.63G                         | 1.24G                         | 1.85G                         | 2.46G                         | 4.9G                          |\n| Partial obs (fovea 64×64, full image 224×224)          | 1.82G (224) / 0.15G (64)  | 0.18G                         | 0.34G                         | 0.51G                         | 0.67G                         | 1.32G                         |\n| Partial obs (fovea 84×84, full image 224×224)          | 1.82G (224) / 0.30G (84)  | 0.33G                         | 0.64G                         | 0.95G                         | 1.26G                         | 2.51G                         |\n\n---\n\nWe can observe that:\n\n- For single-view inference at 128×128 resolution, the encoder requires only 0.60G FLOPs, which matches the compute cost of standard SSL baselines such as SimCLR and BYOL.\n\n- As sequence length increases, the additional compute cost grows sublinearly as additional views should be encoded and aggregated. For example, lengths 2 and 3—where invariant performance improves substantially—incur ~1.2G and ~1.85G FLOPs, respectively, remaining within practical limits for most modern applications.\n\n- Crucially, this cost can be **offset by reducing input resolution**. In our saccade-based setup, we can feed low-resolution foveal glimpses (e.g., 64×64 or 84×84 crops sampled via saliency maps), which require only 0.15G–0.30G FLOPs per view. This enables the use of longer sequences at an overall compute cost comparable to full-resolution, single-view pipelines (e.g. compare the flops of aggregating eight 64 by 46 crops with flops of encoding a single 224 by 224 image in the second row of the table). This approach is particularly appealing in real-time robotic systems, where sensors frequently provide partial, asynchronous, or spatially limited observations (e.g., narrow field-of-view cameras, event-based sensors, or sequential glances across a scene). In such contexts, seq-JEPA can accumulate and integrate these glimpses over time to form a compact, invariant summary for downstream reasoning, while maintaining low computational cost.\n\n---\n\nWe thank the reviewer again for their helpful suggestions. We hope that these clarifications, particularly around access to actions, applicability to natural images, and compute-performance trade-offs, address your concerns. We also invite you to consult our responses to the other reviewers.\n\n---\n\n**References**\n\n[A] Garrido, Quentin, Laurent Najman, and Yann LeCun. *Self-supervised learning of split invariant equivariant representations.* Proceedings of the 40th International Conference on Machine Learning. 2023.  \n[B] Garrido, Quentin, et al. *Learning and leveraging world models in visual representation learning.* arXiv preprint arXiv:2403.00504 (2024).  \n[C] Devillers, Alexandre, and Mathieu Lefort. *EquiMod: An Equivariance Module to Improve Visual Instance Discrimination.* The Eleventh International Conference on Learning Representations, 2023.  \n[D] Yerxa, Thomas, et al. *Contrastive-equivariant self-supervised learning improves alignment with primate visual area IT.* NeurIPS 2024.  \n[E] Shakerinava, Mehran, Arnab Kumar Mondal, and Siamak Ravanbakhsh. *Structuring representations using group invariants.* NeurIPS 2022."}]

Author rebuttal:

Meta-review: **Summary of paper:**

This paper introduces seq-JEPA, a self-supervised framework that learns representations which are simultaneously invariant and equivariant to transformations. By embedding transformation sequences and predicting their outcomes, the method avoids auxiliary losses and achieves strong performance across invariant, equivariant, and transformation prediction tasks.

**Summary of Reviews:**

The reviewers agree that the paper is well-written, the problem well-motivated, and the architecture intuitive. Strengths highlighted include the clarity of experimental protocols, the novelty of jointly modeling invariance and equivariance, and the broad applicability across benchmarks. Concerns focused on the level of comparison with prior work, the role of ground-truth transformations, and the computational overhead. Reviewers gave borderline-to-positive ratings, but engaged constructively with the rebuttal.

**Decision Rationale:**

The authors’ response directly addressed reviewer concerns: they clarified design choices, provided additional comparisons (including time/memory trade-offs), and explained the role of learned representations in capturing invariance/equivariance. The discussion further demonstrated the robustness and novelty of the approach. Taken together, the paper advances representation learning with a clear, well-executed idea, strong results, and thoughtful positioning relative to existing methods. While some limitations remain, they do not outweigh the contributions.
Decision: Accept (poster)