Rethinking JEPA: Compute‑Efficient Video Self-Supervised Learning with Frozen Teachers
Authors: Xianhang Li
Chen Huang
Chun-Liang Li
Eran Malach
Joshua M. Susskind
Vimal Thilak
Etai Littwin
Venue: ICLR 2026 Poster (2026)
Keywords: SALT
video
SSL
video_representation_learning
masked_video_modeling
MAE
JEPA
latent_space_ prediction
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: SALT: A simple, scalable, and compute‑efficient alternative to EMA‑based self‑distillation for video representation learning.

Abstract: Video Joint Embedding Predictive Architectures (V‑JEPA) learn generalizable off-the-shelf video representations by predicting masked regions in latent space with an exponential moving average (EMA)‑updated teacher.
While EMA prevents representation collapse, it complicates scalable model selection and couples teacher and student architectures. 
We revisit masked‑latent prediction and show that a frozen teacher suffices. Concretely, we (i) train a target encoder with a simple pixel‑reconstruction objective under V‑JEPA masking, then (ii) freeze it and train a student to predict the teacher’s latents on masked regions. 
This leads to a two‑stage, unregularized scheme, that we refer to as SALT (Static-teacher Asymmetric Latent Training). SALT decouples optimization into pixel reconstruction (teacher) and masked latent prediction (student), increasing transparency, efficiency, and scalability while preserving the ability of representations to generalize under frozen evaluation. Empirically, our student models outperform recently proposed V-JEPA 2 encoders under frozen backbone evaluation across diverse benchmarks. They are also more compute‑optimal: at matched pretraining FLOPs, our method achieves higher probing accuracy, and its scaling curves dominate V‑JEPA’s accuracy–FLOPs Pareto frontier. 
Finally, we find that student quality is remarkably robust to teacher quality: high-performing students emerge even with small, sub-optimal teachers. This points to a compute budget allocation that should overwhelmingly favor the student.
These results position SALT as a simple, scalable, and compute‑efficient alternative to EMA‑based self‑distillation for video representation learning.

Published as a conference paper at ICLR 2026
RETHINKING JEPA: COMPUTE-EFRICIENT VIDEO SSL WITH FROZEN TEACHERS
Xianhang Li Apple
Chen Huang Apple
Chun-Liang Li
Eran Malach Apple
Josh Susskind Apple
Vimal Thilak : Apple
Etai Littwin Apple
ABSTRACT
Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable off-the-shelf video representation by predicting masked regions in latent space with an exponential moving average (EMA)-updated teacher. While EMA pre- vents representation collapse, it complicates scalable model selection and cou- ples teacher and student architectures. We revisit masked- latent prediction and show that a frozen teacher suffices. Concretely, we (i) train target encoder with a simple rexossrecontttuttion objective under V-JEPA masking, then (ii) freeze it and train a student to predict the teacher's latents on masked regions. This leads to two-stage, unregularized scheme that we refer to SALT (Static- teacher Asymmetric Latent Training). SALT decouples optimization into pixel reconstruction (teacher) and masked latent prediction (student), increasing trans- parency, efficiency, and scalability while preserving the ability of representation to generalize under frozen evaluation. Empirically, recently proposed V-JEPA encoders under frozen backbone evaluation across diverse benchmarks. They also more compute-optimal: matched pretrain- ing FLOPs, our method achieves higher probing accuracy, and its scaling curves dominate V-JEPA's accuracy-FLOPs Pareto frontier. Finally, we find that student quality is remarkably robust to teacher quality: high-performing students emerge even with small, sub-optimal teachers. This points compute budget allocation that should overwhelmingly favor the student. These results position SALT as a simple, scalable, and compute-efficient alternative to EMA-based self-distillation for video representation learning.
student models outperform
INTRODUCTION
self-supervised learning (SSL)-based methods have emerged as standard approach for represen- tation learning in computer vision. These methods pretrain neural networks that use vast amounts
of image (He et al.) 2021 Assran et al. 2023 Caron et al. 2021 Oquab et al., 2024 El Nouby et al. 2024) or video (Tong et al. 2022 Wang al.) 2023b Bardes et al. 2024 Assran et al. 2025)
data to learn backbones that have been shown to work well on many downstream tasks. Among these methods, Joint Embedding Predictive Architecture (JEPA)-based methods (LeCun) 2022 have demonstrated a strong ability to learn powerful semantic features that perform well on downstream
image (IJEEPA) (Assran al. 2023) and video (V-JEPA) (Bardes et al. 2024 Assran et al. 2025) tasks.
As concrete instantiations of the Joint Embedding Predictive Architecture (JEPA), I-JEPA (Assran jet al. (023) and V-JEPA (Bardes et al.] [2024] Assran et al.) 2025) are masking-based pretraining methods that learn powerful semantie representation by predicting masked-out portions of the input in a learned embedding space. Specifically, these methods consist of a context (student)
'Wor done during internship Apple.
Work done while at Apple. I Equual last author contribution.

---

Published as a conference paper at ICLR 2026
encoder and predictor that trained to make predictions that match the embeddings provided by a target (teacher) encoder. While powerful, the JEPA family of models are often complex, byperparametter-riitee and use uninformative loss metric that is a poor proxy for representation quality, requiring practitioners to rely on other more downstream-predictiv metrics (Agrawal al. 2022; Garrido al., 2023] Thilak al., 2024). These issues stem from the core JHPA design: because student and teacher representation co-evolve, trivial collapsed solutions with near-zero loss exist, and must be avoided. To prevent representation collapse. these models are implicitly regularized using the self-distillation approach pioneered by BYOL (Grill et al.) (020). Namely, the stop-gratd graddentt operation is applied on the target encoder, and its weights are updated by an exponential moving average (EMA) copy of the student weights, according to some EMA scheduler. It is worth mentioning that other variants of joint-embedding SSL models utilize more explicit regularizers to prevent collapse (Zbontar et al. 2021: Bardes et al. 2021).
In this paper, we present comprehensive empirical study that challenges the common assumption that involved collapse prevention mechanisms are required for learning high-quality semantic fea- tures. Specifically, targets needed to optimize the student model can be obtained in a more efficient manner with a frozen encoder. This design obviates the need for both the EMA update and the stop-gradient, streamlining the self-distillation process and reducing implementation complexity. We start with a simple two-stage pretraining scheme: (i) train a target encoder with pixcl-reconstructton objective under V-JEPA-style masking; (ii) freeze this encoder and train a student with the JEPA objective to predict the teacher's latents on masked regions (Bardes al. (224). Prior work has explored using pretrained frozen encoders as masked- prediction targets (Wang et al. 2023c: Li et al. 2023), but typically assumes access strong teachers and often relies on tinettunig the student to realize the benefits. In contrast, our study provides fair and direct comparison against strong baselines, includ- ing V-JEPA 2, on larger datasets and models; we dub our method SALT (Static-teacher Asymmetric Latent Training) and show that:
show that dynamic teacher is unnecessary, and that stable, high-quality
Small, "sub-optimal" teachers suffice. High-quality semantic features competitive with state-of-the-arn under frozen evaluation protocols can be learned from much smaller and cheaper teachers. Using the strongest available pretrained encoders unnecessary and yields at most marginal gains for the student.
2. Compute efficiency. Our two-stage design is more compute-efficient than 2MA-based self-distillation (e.g.. V-JEPA): at matched FLoating Point Operations (FLOPs) and wall-clock, and even when accounting for the cost of training the teacher, our method achieves better accuracy-FL.c
trade-of
3. Interpretable model selection. Our design yields student loss that provides infor- mative, training-tima metric that correlates strongly with downstream accuracy under the frozen-backhone protocol, in contrast to EMA-based methods that require proxy heuristics for model selection.
Taken together, results suggest that elaborate online student-teache dynamics and EMA-based collapse prevention machinery may be unnecessary for learning high-quality representation.
METHOD OVERVIEW
We first review video-based JEPA models that include both V-JEPA and V-JEPA 2, and then describe our simple approach named SALT for representation learning from videos. Note that V-JEPA 2 uses the same pretraining method described V-JEPA but employs updated hyperparameters so our method review applies to both models.
2.1 V-JEPA
V-JEPA employs a masked prediction objective: the context encoder preedictr reconstructs masked regions from visible frames in learned representation space, while an EMA-updated target encoder
FL.OPs and total number of training steps used interchangeably to refer to compute. includes an explanation for this choice
Appendix G
2

---

Published as a conference paper at ICLR 2026
La loss
74
Apply
Predicor
73
VIT
72
VITH
VT--g
SSv2 16x1x1 Accuracy
71
Teacher Eacoder
Student Encoder
70
vir-I1
VEL
69
68
VJEPA-2
67
SeLT-teacher
66
SALT-studen
0.2
2
Compute (x102))
Frozen-teacher, learnable student and predictor. The frozen teacher encoder is obtained via Stage 1 (not pictured above) by training using pixel reconstruction objective. The student and predictor are jointly optimized to learn representation from video in Stage 2 using latent space prediction objective. (Right): SALT's compute campute-accuracy curve dominates V. JEPA
Figure (Left) SALT Stage 2:
supplies the supervision. Following the notation used by Bardes et al. (2024), the latent space prediction objective can be written:
min E., Be.gloo(fo(t)) stop grad(fo(v))
8,0
where x and y denote two disjoint regions of the input, f. and g denote the encoder, target encoder and predictor respectively, stop-grad denotes the slop-gradien operation and dy denotes the spatio- temporal positions missing regions in the input that act as context for the predictor.
2.2 STATIC-TEACHER ASYMMETRIC LATENT TRAINING (SALT) A SIMPLIFIED VIDEO REPRESENTATION LEARNING METHOD
The V-JEPA method uses self-distillation approach incorporating stop-gradient and exponential moving average (EMA). This approach requires properly setting the associated hyperparameters to prevent representation collapse. In this work, we advocate for an alternative solution, which refer to as SALT, that does not require the use of EMA operation. Specifically, we simplify the architecture by breaking down video representation learning into two steps:
Stage 1 The teacher target encoder is trained optimize pixel reconstruction objec- tive in Stage 1. The objective is identical to the objective used in VideoMAE (Tong et al.) 2022). However, our Stage method differs from VideoMAE we use more efficient masking scheme, the details of which are described in Section 5.2
The weights of the teacher from Stage I are frozen and used to train student and predictor network as shown Figure I The JEPA objective, described by Equation (T) is used to optimize the student and the predictor.
Stage 2
The simplification described above results in two loss objectives that are proper loss functions which are easier interpret in practice, and are immune to representation collapse by design. This stands in contrast to V-JEPA's objective in Equation (T) which is difficult to interpret due to its self self-distillation nature that, turn, necessitates the use of surrogate metrics (Garrido al. 2023 Thilak et al. 2024). Moreover, our two-stage approach completely decouples the teacher and student architectures, un- locking considerable compute efficiency gains by utilizing small teachers to train larger students, as shown in Figure I and Table We observe from Figure [ that SALT shows remarkable improve- ment over V-JEPA 2 on Samnething-Something..
(SSv2) (Goyal et al. (017). which a temporal understanding task. Furthermore, Table I shows that smaller but noticeable improvement is ob- served on Kinetics-400 (Kay et al. 2017). ). an appearance understanding benchmark. We describe the experimental setup in Section and discuss results in detail in Section

---

Published as a conference paper at ICLR 2026
2.2.1 SALT DESIGN PRINCIPLES
SALT follows the contemporary trend toward simple, principled architectures and objectives, avoid- ing elaborate engineering. We provide a simple recipe to train the teacher in Stage I with method that we call V-Pixel that uses pixel reconstruction objective along with the multi-block masking method described in V-JEPA The decoupled design of SALT allows us to study the role of archi- tecture and dataset choices for training the teacher and student in granular manner. We uncover a surprising finding that a high performing teacher, as measured by its downstream performance, is not necessary to train a high-quality student. As we show in Section 5.1] Section 5.3 and Sec- tion 5.4 the student's 's ultimate quality is surprisingly robust to suboptimal data mixture, teacher size and compute budget. Overall, our simplified design demonstrates superior efficiency, scalability, and interpretability over the baseline V-JEPA.
EXPERIMENTAL SETUP
Training Our training data includes Kinetics-710 (K710) Kay et al. (2017), constructed by
merging Kinetics-400/60000 and removing all validation samples, Something-Somettiee
V2
(SSV2) Goyal al. (2017), and 2.8 million subset of the Panda70M [Cen et al. (024) result-
ing in approximately 3.6 million (3.6M) video dataset that we refer to as the V-3.6M dataset in our work. Note that our training dataset differs from the training datasets used in V-JEPA and V-JEPA
2 as the latter methods use Howto I 100 (Miech et al. 2019) and YT--emporal--1 IB datasets (Zellers) et al. 2022) while we use a subset of 7aM in V-3.6M. Our models are standard Vision Trans- formers (ViT) (Dosovitskiy et al.) 2020) with rotary positional embeddings (RoPE) (Su et al.) 2024) which is identical to the architecture described in V-JEPA 2. Specifically, we use ViT-Large (ViT- L). ViT- Huge (ViT-H) and ViT-giant (ViT-g) in experiments, the details of which are described in Appendix B All baseline models (V-JEPA and V-Pixel) are trained with the same batch size of 3072 using the AdamW optimizer with hyperparameters described in detail in Appendix C To en- sure fair comparisons, we keep the number of optimization steps fixed for both our baseline methods and SALT. In other words, the total number of steps for Stage and Stage 2 is identical to the num- ber of steps used by baseline methods. The optimal number of steps to train a teacher and student is obtained via an ablation described in Section 5.4
Evaluation We evaluate our models a variety of video and image tasks. For video classifica- tion, following, we use Kinetics-400 (K400), Something-Somcttingg Something-v2. COIN classification (Tang et al.) 2019). Jester (Materzynska et al., 2019) and Diving-48 (Li et al., 2018 freezing the back- bone and training an attentive classifier to assess performance. For image classification, we adopt the same protocol on ImageNet- 1K (Russakovsky al., 2015), replicating each image 16 times to form the input sequence. Furthermore, we evaluate our models on intuitive physics understanding benchmarks, which measure performance by comparing the model's surprise scores for possible versus impossible videos. Following (Garrido et al.] 2025), we use the predictor to forecast future representation. We assess performance on the IntPhys (Riochet et al. [0018. GRASP (Jassim al., 2023), and InfLevel (Weihs et al., (022) datasets. All setup information and hyperparameters used for our evaluations are described detail in Appendix D
EXPERIMENTAL RESULTS
Systematic comparison of SALT with existing baselines Table I lists the performance of SALT and existing work that serve as strong baselines including V-JEPA 2, VideoPrism (Zhao et al.] 2024). Intern Video2 (Wang et al. 2024). VideoMABv2 (Wang et al. 2023b), Perception Encoder (Bolya et al.
2025) and image encoders that include DINOv2 (quub et al. 2024 and SigLIP2 (Tschannen et al. 2025). We use the Kinetics-400 (K400) and Something Soaenn--Smettngg---. v2 (SSv2) as benchmark datasets and evaluate SALT following the same multiclip. multiview setting used in existing bascline.
We observe from Table that our largest models, ViT-g and ViT-G, trained with SALT outperforms all of the baseline methods SSv2, which tests the motion understanding ability of video models. On K400, which is an appearance understanding benchmark, the encoders trained with our method
This method is implicitly described in Table 1 by Bardes al. 2024). We name the method V-Pixel for clarity in presentation.
4

---

Published as a conference paper at ICLR 2026
Table I: Systematic comparison of state-of-the-art video encoders under frozen-backbong eval- uation, using SSv2 (16x2x3) and K400 (16x2x3). The comparison includes several baselines including encoders trained with V-JEPA 2 method on our V-3.6M dataset. The V-JEPA 2 encoders trained on V3.6M and SALT encoders are evaluated using the protocol in Section and Appendix D The results for other models are duplicated from Table V-JEPA (Assran et al. 2025). A detailed description of FLOPs calculation is available in Appendix
Method
Param
Prefraining Dotaset
Teather (Params)
Todal Camgute
#f been Somples
5812
K,01
Pooza
IR
1.68t
$2.6
MVD
303M
INIK
79.4
PR.
19B
MasCCLPP-58 Auccial (004
NIA
568
55.4
$6.5
Inter
2024
1B
IV 2:25
BnenVLL4 VideoMBAEN2 1.003
67.3
$7.9
SilesPiiun Zaplrst Nilee
18
VT--HM
Tuge---ViT-g- 1.06)
2.08
58.5
$7.6
DINOx2
L.B
LVD--12M
lzaclan (1.1B)
1.88
$3.6
VJERA
300M
VXF
85.1
NJFPA NIHH
6BM
VM--2MM
FMA nocher (503M)
0.78
74.0
$5.5
2
18
VM 22M
EMA Jealher (1B)
533
0.78
75.3
$6.5
VURRA MILL
10M
V-3.0M
EMA teater (MKM)
144
0.78
06.3
83.8
2 MTHH
600M
V-3.MM
leacher
2.6
0.78
84.6
SALT
103
V.3.MM
1.2
0.78
$5.4
SALT VIE
SALT WTFg
600M
18
V-3.MM
V-1.M
(2000)
199
1.5
0.78
0.71
75.4
76.2
86.8
$6.0
SALT VIEG
28
V.3.M
2.6
0.78
76.1
$7.2
V-JEPA-2 (VITAL, 240K)
Stago-1 [MTT-L, 80K)
SALT Stage-2 (VITAL, 160K)
VIT-H
VIT
Avg Top-3 Accuracy
60-
VIT-B
Accuracy (hg
78-
08
8533
X1.9
26
26-
733
SALT
VJEN
100
.00
500
700
56x2
K4000
CDIN
1814
JESTER
Params, (in million)
(a) Per-benchmaak accuracy of ViT-I.
(b) Scaling model-size.
Figure 2: V-JEPA vs. SALT at matched total steps on V-3.6M. Both methods are trained on the same V-3.6M dataset for an identical number of pretraining steps. SALT uses an 80k-step teacher and 160k-step student. We evaluate all models under the same frozen-backbone protocol across standard video/image benchmarks: K400 with 16x1x1 input; SSv2 with 16x 1 x Diving48 and Jester with 16x4x3; and COIN 16x8x3. Table 12 provides a breakdown of downstream perfor- mance for each dataset used in this evaluation.
exceeds the performance of V-JEPA 2 across all scales and remains highly competitive with other state-of-the-art methods including the recently proposed Perception Encoder.
Static teacher improves representation quality A key design choice of SALT the use of a static teacher which differs from the dynamic momentum-encodde teacher used in V-JEPA 2. In order to ascertain the differences between these two approaches. we use the same V-3.6M dataset and same input resolution of 224 x 224 to train SALT and V-JEPA 2. We train both methods for total of 240k steps in this study. Figure 2a shows the downstream performance results of this study for ViT-L-based teacher and student model setup while Figure 2b shows the scaling behavior of the two methods as we scale up the student encoder while using a teacher encoder that is the same size or smaller than the student. We observe from Figure 2a that SALT improves the average accuracy over the V-JEPA 2-based encoder by 2.3% where the average is calculated over six benchmarks. Furthermore, we observe from Figure 2b that SALT displays improved performance as we scale up the student. Note that we use the same-sized teacher student for ViT-B and ViT-L while we use smaller VTT-I. teacher model for training ViT-H student encoder. We refer the reader to Appendix B for detailed model size and other architecture information. Together, Figure 2 suggests that the static teacher-based SALT learns higher quality features when compared to V-JEPA 2 that uses an EMA-based teacher.
Small teachers unlock compute efficiency Table and fig. 2 show that strong students can be trained from a frozen teacher, which is considerably cheaper: fixed VIT-L teacher success-
5

---

Published as a conference paper at ICLR 2026
Stadentoos Lose
(a) SALT-tcacher 80k. R
(b) SALT-teacher 40k.
(c) SALT-tcacher 20k.
0.951
R 0.972
R² 0.984
Figure 3: Correlation between student loss and downstream accuracy. Observe that the student model's training loss is predictive of downstream accuracy.
fully trains same-size ViT-L students, and much larger ViT-H/g/G students. Consequently, SALT achieves lower total pretraining FLOPs than the EMA-based baseline across model sizes, when accounting for the teacher pretraining stage. The savings stem from the simple, efficient teacher pretraining (e.g.. ViT-L V-3.6M) and grow with both model size and spatial resolution. FLOPs computation details appear in Appendix
SALT enables interpretable model selection key challenge with using joint-embedding meth- ods including JEPA is that the training loss is typically uninformative of representation quality. Fig- ure 3a| shows student training loss versus student downstream accuracy plot for various student checkpoints that the same SALT Stage 1 checkpoint the teacher. This checkpoint is obtained by training a teacher for 80K steps. The plot shows that the student loss is highly predictive of the downstream accuracy with R²2 value of 0.951, suggesting an almost linear relationship. This sult implies that SALT significantly simplifies tracking the quality of representation during student pretraining, and provides clear signal for improvement via simple loss minimization. Similar ob- servations can be made by teacher trained for 40K and 20K iterations in Figure 3 Lastly, we study whether teacher-related metries such teacher loss or RankMe (Garrido et al., 2023) are predictive
of downstream performance in Figure 9 We find that neither the teacher's ' loss nor embedding rank are predictive of the student encoder's downstream performance.
Intuitive physies evaluation
Garrido al. (2025) have shown that video models trained with the
JEPA objective show an emergent understanding of intuitive physics. We follow the setup described in (Garrido et al. 2025) measure the intuitive understanding ability of video models trained with
SALT. The evaluation setup and results are discussed in detail in Appendix E.1 due to space limi- tations. The main finding that intuitive physics understanding observed on models trained via SALT as well as V-JEPA 2.
VJEPA-2 150K)
Stage
SALT Stage (MIT-L S0K)
78
72.5
72
62
56
52
IN1K
K710
Panda2 BM
5542
K710-55V2
K710455V2+ Panda2 BM
Figure 4: Training data of static-teacher. We ablate the impact training data of teacher, thus fixed the student's training data as the whole data-mix by default. Table [3 provides a detailed breakdown of the results show above.

---

Published as a conference paper at ICLR 2026
5 TEACHER DESIGN CHOICE ABLATION
The empirical analysis in Section 4 shows that SALT provides high-quality representation that out- performs several existing methods. A key aspect of SALT is the static (frozen) teacher that provides high-quality prediction targets. We study the design choices involved in training the teacher model and student model that lead optimal representation via series of ablations described in the fol- lowing.
5.1 TRAINING DATASET
In this section, we study the role of pretraining data distribution on a teacher model. Specifically, train ViT-L teacher model with six training datasets: (i) Kinetics-710 (K710), (ii) Something-
Something-V2 V2 (SSv2), (iii) a 2.8 million subset of Panda70M, (iv) ImageNet- Ik, (v) data aggregated
from K710 and SSv2, and (vi) V-3.6M which is data aggregated from K710, SSv2 and Panda70M subset. The exact details of the datasets are described in Section 3 The teacher model is trained using the Stage 1 approach (V-Pixel) described in Section 2.2 For each teacher model from Stage 1, we train a ViT-L-based student model on the combined V-3.6M dataset using Stage 2 approach
described in Section 2.2
F shows the result of benchmarking the teacher and student models trained with the datasets described above. We observe that the performance each student model described above improves over that of its corresponding teacher. Additionally, each student model's downstream performance exceeds the performance of a V-JEPA 2-based encoder with the exception of the student model trained on ImagetNet-IKK IK-bedd teacher model. Among the teacher models trained on video datasets considered in this study, we observe comparable performance with the notable exception of the teacher model trained effective teacher may be trained with a relatively small amount of data to build strong foundation models.
V2. Taken together, these results suggest that
5.2 TEACHER MASKING STRATEGY
We study the role of masking strategy used to train the teacher that provides targets to opti- mize the student model. To this end, train a ViT-L using random masking used in Video- MAE, multi-block masking used in V-JEPA and a modified method that we call multi-random tube where adapt the short-range and long- range masking idea from V-JEPA to random masking. We refer the reader Table 13 for setup details used in this ablation. Figure shows the results for this ablation. We observe that the multi-block masking approach works best for V-Pixel model achieving an accu- racy of 72.5%. This result in and of itself is a new empirical finding as VideoMAE models typically random-tube masking. Further- more, we observe from Figure that the student trained with multi-block teacher achieves the highest accuracy while the other student models also show a big improvement over their correspond- ing teachers. We conclude that multi-block masking strategy is effective with training our teacher and name the pixel reconstruction method with multi-bles block masking as V-Pixel.
TED
128
1++.
Chubd PIKE
Figure 5: Masking Strategy of static-teacher. We study the impact of random multi-block masking strategy influences a student's perfor- mance. Table T5]incluees hyperparameters and re- sults information.
5.3 TEACHER MODEL SIZE
Next, we study the impact of a teacher model's size on a student model's performance. We train ViT-B, ViT-L, ViT-H and ViT-G based V-Pixel models and use these models to supervise ViT-L and ViT-G based student models. Figure o shows the results of this ablation. We observe that the best performing VIT-L student has average accuracy of 77.4% and is obtained by training with
7

---

Published as a conference paper at ICLR 2026
ViT-L teacher. This result is remarkable this accuracy is better than the accuracy obtained with ViT-H and ViT-G based teachers that are larger than the student. A similar observation can be made about the VIT-G student where the highest average accuracy of 78% is obtained with ViT-L teacher. Additionally we observe that all student models show improvement over their teachers which are of the same smaller size. These observations suggest that the multi-stage training proposed in SALT allow the student to bootstrap from a weaker teacher to learn high-quality representation.
SALT Stage-1 (V-Pwal,, BOid
SALT Stage (BOK)
(a) VIT-L
(b) VIT-G
782
25
Topk Accoreyy
12.
724
Avg Tos1
1399
Sigpeti
Rade
Lang
Hage
Figure 6: Teacher model size ablation. We train ViT-B, ViT-L, VIT-H and ViT-G based teacher and use the teacher to train a ViT-L and ViT-G student. Observe that the best performing student is obtained from ViT-L teacher with modest performance. Table 14 provides a detailed breakdown of results on downstream benchmarks.
5.4 IMPACT OF TEACHER- MRACHER-STUDENT COMPUTE ALLOCATION
Due to frozen teacher approach used in SALT, we are confronted with the problem of allocat- ing compute between the teacher and the student. Training compute is a function of the model size, the number of optimization steps well as the number of FLOPs per step. In this abla- tion, we hold the model size, the total number of optimization steps, and the training dataset (V- 3.6M) constant as that allows us to conduct fair comparison of SALT and V-JEPA 2 bascline. We use ViT-L based model for both teacher and student in this ablation where we vary the num- ber of steps used to train the teacher and stu- dent for a fixed total number of optimization steps. We observe from Figure / that SALT outperforms V-JEPA 2 baseline over a range of optimization steps considered in our experi- ments. Furthermore, we observe that SALT ex- ceeds V-JEPA 2's performance R the same FLOPs level which in turn suggests that SALT more compute efficient than V-JEPA The best performing student is obtained by training on 240k total steps, as can be seen from Fig- 7 that is supervised by teacher that is only trained for 40k steps. This finding underscores the effectiveness of our multi-stage training ap- proach and demonstrates that we should focus on students in SALT, and that aiming for a high- performing teacher may be wasteful. Addi- tional visualization and analysis that supports these claims are provided in Appendix E.2
78.5
4 200
78.0
004-163k
O
77.5
226
27.0
Arg. Tope Axcmnay
24e-443
26.5
WyEW22
5AG
147
26.0
75.5
75.0
1MA
14.3
76.0
4%.10
6xx10
10
Total Compute Ex1071 FLOPs)
Figure 7: Comparison of compute allocation in SALT. We show average Top-1 accuracy across benchmarks against total training FLOPs. Our SALT curves dominate V-JEPA-2 at matched bud- gets. See Table T6 for additional details.
6 RELATED WORK
Video foundation models: Masking- based self-supervised learning (SSL) (Tong et al. 2022; Fe-
ichtenhofer et al. 2022 Wang al. 2023a Bardes al. 2024 Assran et al. 2025 Wang al. 2023c Li et al. 2023 Zhao et al. 2024; Wang et al.] (0226) is a prominent approach used to
leam representation from large-scale video datasets for building video foundation models. Several
8

---

Published as a conference paper at ICLR 2026
works (Tong al. 2022 Feichtenhofer et al. 2022 Wang et al. 2023b) have extended image-based masked autoencoders (He et al. (021) to video data by using random masking to learn representa- tion via pixcl-space reconstruction. Recent works incorporate motion aware masking and temporal correspondence to better capture temporal dynamics (Thoker et al. 2025 Sun et al. 2023 Salehi et al. 2024 Huang et al., (023). An alternate approach for representation learning to learn via latent-space predictions. These methods are known to learn features that differ in quality from those obtained via reconstruction-basee
methods (Littwin et al. 2024 Balestriero & Lecun (022)) Promi- nent among latent-space prediction methods for video are V-JEPA (Bardes et al. (0224) and V-JEPA 2 (Assran et al., (025) that use an online/momentum encoder to learn the teacher that provides pre- diction targets. SALT simplifies the JEPA pipeline by using a frozen teacher that is reliable and efficient that leads to higher quality representation as shown in our analysis.
Distillation from frozen teacher encoder: While many studies utilize frozen pretrained mod- els as teachers for student encoder supervision, we limit our review to prior work that is directly relevant to our core method and refer the interested reader (Balestriero et al. 2023) for a com- prehensive survey on SSL literature. MVD (Wang et al. 2023c) and Intern Video (Wang et al. 20226) use VideoMAE (Tong et al.) (022) while other works such as UnMasked Teacher (Li al. 2023]. VideoPrism Zhao et al. (2024). Intern Video2 Wang et al. 2024 and many more use a vision-
language model (Radford et al. (02T). as a frozen teacher. Perception (Bolya et al. 2025) is recent vision foundation model that uses features from predefined layer from within the model as well as features from an external teacher SAM (Kirillov et al. 2023) to encourage fea- ture locality. AM-RADIO (Ranzinger et al. 2024) proposes to learn a student from multiple teacher models. Beyond prior work, which require access to powerful pretrained encoders, we uncover a weak-teacher, prrong-student effect: students supervised by much weaker frozen teachers consis- tently outperform those trained with EMA-based teachers. Our method is purely self-supervised and unregularized, unlike self-training Xie al. (2020), which relies labeled and unlabeled data and explicit noise regularization. Related world- approaches (Karypidis et al. 2024:] [aal- dassarre et al. 2025b Zhou al., 2024) also fix encoders for stability but optimize for future-state prediction, while our aim is representation learning. Nevertheless, SALT 's strong video features make a promising backbone for world models. Lastly, recent advancements in image-to-vine -ividoo distillation (Li & Liu, [023 Hu et al.) 2022; Liu al. 2025) incorporate temporal information into strong representation models during distillation while we learn representations directly from video.
Wang et al. (2023c) propose a two-stage method called masked video distillation (MVD) that first trains two separate encoders one each for image and video input using an MAE (He et al. 2021 Tong et al. 2022] like approach. These teacher encoders then provide targets (latent features) used to optimize smaller student encoder. SALT resembles the approach taken by MVD but has several critical differences. The first difference is that we provide an improved method to train the video teacher encoder as result of careful empirical analysis in Figure S Additionally, we do not use a separate encoder for image data but instead focus on learning representation from large-scale video datasets. The most critical difference is that we use teacher model whose size is the same or smaller than that of the student model. Furthermore, we conduct detailed ablations in Section 5 to show how choose teacher model (checkpoint). Finally, SALT learns superior features as our benchmark results are based on frozen backbone evaluations while MVD uses fine- tuning for downstream evaluation.
Masked video distillation:
LIMITATIONS AND CONCLUSION
We present SALT, a simple, compute-efficient and scalable framework for video representation learning. Across standard benchmarks, SALT consistently outperforms strong baselines, including V-JEPA-2 in -ozen-evaluation protocols. Strikingly, we find that sub-optimal, often smaller teach- ers can yield much stronger students, raising questions as to how the quality of the teacher should be assessed, and whether EMA-based machinery is necessary to learn highly-semantic representa- tion. Prior works including SimSiam (Chen & He| 2021) and Asym-Siam (Wang et al.) (202aa) have explored the hypothesis high quality student encoders can be trained using targets from a teacher encoder that provide lower variance targets. Exploring the applicability of this hypothesis to setup as well as building& a principled characterization of teacher quality is left as future theoretical work. On the empirical front, fuller study of SALT 's scaling behavior with respect to data and
9

---

Published as a conference paper at ICLR 2026
model size is left future work.
While SALT improves compute efficiency and downstream performance over self-distillation, has limitations. Our ablations (Section()) suggest that simple V-Pixel recipe usually suffices to train an effective teacher and that compute is best allocated to the student; however, they do not fully explain what makes "good" teacher. We also observe modest gains from additional student-training stage, but the mechanism remains unclear. Given the experiment volume, we focused compute on simple scalar diagnostics of teacher quality (Section 5 and Appendix [.3). Finally, performance plateaus as model size grows in our setting, likely reflecting data limits, and that larger pretraining sets may extend the scaling trend.
ACKNOWLEDGMENTS
We thank Eeshan Gunesh Dhekane and Preetum Nakkiran for their insightful feedback and dis- cussions provided during the process of preparing this manuscript. VT would like to thank Meng Cao for advice on video datasets. We thank the entire Apple Machine Learning Research team for providing resources and inspiration that allowed us to complete this project.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform 2025.
for physical ai. arXiv preprint arXiv:2501.033:
Kumar Krishna Agrawal, Arnab Kumar Mondal, Arma Ghosh, and Blake Aaron Richards. a-ReQ Assessing Representation Quality in self-supervised learning by measuring eigenspectrum decay. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /ttttp::///pperevieww net forum?
1d=119X4vt2G12
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15619-15629, 2023. URL https://api..emmmiiicchie. www.nticscnocool...o.o.occcoc.
brg/Corpus ID:
Mahmoud Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Ar- naud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Mare Szafraniec, Kapil Krishnakumar, Yong Li, Xi- aodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.099 09985, 2025.
Federico Baldassarre, Mare Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann Le- Cun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: Dino as foundation for video world models. arXiv preprint arXiv:2507. 19468, 2025a.
Federico Baldassarre, Mare Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann Le- Cun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: Dino as a foundation for video world models. arXiv preprint arXiv:2507. 17...9946 19468, 2025b.
Randall Balestriero and Yann Lecun. How learning by reconstruction produces uninformative fea- tures for perception. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 2566-2588. PMLR, 21-27 Jul 2024. URL. http::ttt///tt httt::://prrceed ngs. www..r.res/235.. balestriero245 htm11
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. cookbook of self- supervised learning. arXiv preprint arXiv:2304 122/0, 2023.
10

---

Published as a conference paper at ICLR 2026
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: haapeeee-iiaaainne-oovraannn regular-
ization for supervied learning. ArXiv, has/2105.04906, 2021. URL httpss//apil
semanticscholar: :ttg://orrps:::233335535
Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint ar wxXv:2404.088771 2024.
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint 2504::2500.13188. 2025.
Florian Bordes, Quentin Garrido, Justine T Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux. Intphys Benchmarking intuitive physies understanding complex synthetic environ- ments. arXiv preprint 250::0004599999 2025.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e e J' e'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. 2021 IEEF/CVF International Conference Computer Vision (ICCV), pp. 9630-9640, 2021. URL https:
//api. semant www.ccccc.aaccoooo wrrachollar.or//ooopsseese http///oooput0::344433333 233444273,
João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, Joseph Heyward, Skanda Koppula, Etienne Pot, et al. Scaling 4d represen- tations. arXiv preprint arXiv:2412. 15212, 2024.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wet Chao, Byung Eun Jeon, Yuwei Fang. Hsin- Ying Lee, Jian Ren, Ming- Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross cross-modality teachers. arXiv preprint
24Xi::2402.19499, 2024. URLhttps::/aarxt //thtttss////axiiii wwtp:/orggggga///gggeo.e 19.99
Xi Chen, Xiao Wang. Soravit Changpinyo, Anthony Piergiovanni, Piotr Padlewski, Daniel Salz, Sehastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209. 06794, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750- 15758, June 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rX1v:2010.11929, 2020.
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai Miguel Ángel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M. Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In Proceedings of the 41st International Conference on Machine Learning. pp. 12371-12384, 2024.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958, 2022.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. RankMe: Assessing the
downstream performance of pretrained self-supervised representations ings of the 40th International Conference on Machine Learning, 2023.
reppeveeside by their rank. In Proceed-
Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rab- bat, Emmanuel Dupoux, and Yann LeCun. Intuitive physics understanding emerges from self- supervised pretraining on natural videos. arXiv preprint arXiv:2502. 1831, 2025.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitte Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 5842-5850 2017.
11

---

Published as a conference paper at ICLR 2026
Jean-Base Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- new approach to self-supervvsed learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284 Curran Associates, Inc., 2020. URL |ttps::/ neur neur ips wttpppssccccccccccceoo cc/paper Files/paper/2022 2020/f1lee /ttp://aaaac0ee01011111111111110 (twpbb/8922228988 Paper pdf
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Doll'ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVF Conference on Computer URL https://api.
Vision and Puitern Recognition (CVPR), pp- 15979-15988, 2021.
2439985988
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Yingdong Hu, Renhao Wang, Kaifeng Zhang, and Yang Gao. Semantic-aware ine-grained corre- spondence. European Conference on Computer Vision, pp. 97-115. Springer, 2022.
Bingkun Huang, Zhiyu Zhao, Guozhen Zhang, Yu Qiao, and Limin Wang. Mgmae: Motion guided masking for video masked autoencoding. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 13493-13504 2023.
Serwan Jassim, Mario Holubar, Annika Richter, Comelius Wolff, Xenia Ohmer, and Elia Bruni Grasp: A novel benchmark for evaluating language grounding and situated physies understanding in multimodal language models. arXiv preprint arXiv:231
2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Komodakis. fono-foresight: Looking into the future with dino. arXiv preprint 2tti::2//2.11673: 2024.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.00650, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference computer vision, pp- 4015 4026, 2023.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62 2022.
Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficien video foundation models. Proceedings of the IEEE/CVF international conference computer vision, pp. 19948-19966 19960, 2023.
Rui Li and Dong Liu. Spatial-then-temponn temporal sulf-super learning for video correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2279-2288 2023.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without represen- tation bias. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang. and Joshua Susskind. How jepa avoids noisy features: The implicit bias of deep linear self distillation networks. grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems. volume 37. pp. 91300-91336. Curran Associates, Inc., 2024. URL.hEtrsss//prrreeiing..e.eeeee
In A. Globerson, L. Mackey, D. Bel-
http:ppppewr.illeeppppeeeeeeeeee files/paper fi 21126000077000001135511000000000
12

---

Published as a conference paper at ICLR 2026
Yang Liu, Qianqian Xu, Peisong Wen, Siran Dai, and Qingming Huang. When the future becomes the past: Taming temporal correspondence for self-supervised video representation learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24033-24044, 2025.
Ilya Loshchilov and Frank Hutter. arXiv:1711.5101, 2017.
Decoupled weight decay regularization. arXiv preprint
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset:
A large-scale video dataset of human gestures. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pp. 0-0, 2019.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp- 2630-2640, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khali- dov. Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Featured Certification.
lnttps::/openzeviiee httpriewwweeteee wwwpst/fooumiddddeeeee a68SUt 62Ft
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12490-12500, June 2024.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint 18Xiv:1803.cm 07611,, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225 2015.
Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees GM Snoek, and Yuki M Asano. Sigma: Sinkhom-guidde masked video modeling. In European Conference on Computer Vision, pp. 293-312. Springer, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Xinyu Sun, Peihao Chen, Liangwei Chen, Changhao Li, Thomas H Li, Mingkui Tan, and Chuang Gan. Masked motion encoding for self-supervised video representation learning. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. LiDAR: Sensing linear probing performance in joint embedding SSL architectures. In The Tweifth International Conference on Learning Representations, 2024. URL https://openrevieee net httt://orrmmml.l
13

---

Published as a conference paper at ICLR 2026
Fida Mohammad Thoker, Letian Jiang. Chen Zhao, and Bernard Ghanem. Smile: Infusing spatial and motion semantics in masked video learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8438 8488,449,448 2025.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances neural information process- ing systems, 10:10078-10099 2022.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. 14786, 2025.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking, 2023a.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14549 14560, June 2023b.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self- supervised video representation learning. In CVPR, 2023c.
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning, 2022a. URL
2204.00613
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.0319) 2022b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pp. 396-416. Springer, 2024.
Luca Weihs, Amanda Yuile, Renée Baillargeon, Cynthia Fisher, Gary Marcus, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in AI. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:
/ //ppenrvveew. fort/forum? (eterrmmmmiiiiin d= 9 4
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings the IEEE/CVF conference on computer vision and patlern recognition, pp. 10687-10698. 2020.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In ICLR, 2024.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ArXiv, abs/2103.03230. 2021. URL https:
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16375-16333 16387, 2022.
Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. Videoprism: A foundational visual encoder for video understanding. arXiv preprint arXiv:2402.1 13217, 2024
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. arXiv preprint 1.Xvv2241..04983 2024.
14

---

Published as a conference paper at ICLR 2026
A TRAINING DATASET
We describe the datasets used to train vision transformer (ViT) models with SALT. We form the Kinetics-710 (K710) dataset by combining training samples from Kinetics-400/66007000 (Kay al. 2017] and removing duplicated samples as well as samples that are in the validation sets of the above datasets. We then add training samples from the Something-Something Something-v2 (SSv2) (Goyal et al., 2017) dataset. Finally, we add an approximately 2.8 million video clips subset of Panda70M (Chen al. 2024) to form our dataset that we refer to as V-3.6M train our models. We apply stratified sampling to select the subset of clips from Panda70M that enables us to have clips whose duration ranges from seconds to seconds. We not apply any other form filtering or curation to form our training dataset. Table 2 lists the sample count information for our V-3.6M dataset.
Table 2: V-3.6M Training dataset details.
Dataset
Sample Count
Kinetics-710
657,257
Something Something-v2
168,913
Panda70M
2,799.959
V-3.6M
3,626,129
B ARCHITECTURE DETAILS
We use Vision Transformers (ViTs) (Dosovitskiy et al. 2020) to implement our video encoders and predictors. We use a spatial patch size of 16 16 and temporal patch size of all of our models. Bable[]and Table lists the model architecture in detail for our encoders and predictors re- spectively. We follow V-JEPA (Assran et al. 2025) and use rotary position embedding (RoPE) (Su et al. 2024) to encode position information in all of our models. Note that our predictor's last layer projects the embedding dimension to compatible with that the teacher encoder. This informa- tion is captured in the input and output dimension columns in Table 4 We ViT-L teacher to train all encoders except the ViT-B model which uses a ViT-B teacher. This information captured in the output dimension column in Table 4
Table Encoder model architecture details. M indicates a million and B a billion.
Model
Parameter Count
Width
Depth
Heads
ViT-B
86M
768
12
12
VIT-L
303M
1024
24
16
ViT-H
632M
1280
32
16
ViT-g
1.012B
1408
40
16
VIT-G
1.843B
1664
48
16
Table 4: Predictor model architecture details. M indicates million.
Predictor & (Encoder)
Input Dimension
Output Dimension
Parameter Count
Width
Depth
Heads
ViT-Predictor (ViT-B)
768
768
21.88M
384
12
16
ViT-Predictor (ViT-L)
1024
1024
22.08M
384
12
16
ViT-Predictor (ViT-H)
1280
1024
22.18M
384
12
16
ViT-Predictor (ViT-g)
1408
1024
22.23M
384
12
16
ViT-Predictor (ViT-G)
1664
1024
22.32M
384
12
16
15

---

Published as a conference paper at ICLR 2026
Table Hyperparameter details used to train models with SALT. Note that "indicates that Stage uses the same hyperparameter value as listed in Stage 1.
Parameter
Stage 1
Stage 2
Input spatial resolution
224 x 224
Tubelet size
2
Patch size
16 x 16 x 2
Number of frames
16
Frame step
4
Random resize aspect ratio
[0.75, 1.35]
Random resize scale
[0.3, 1]
Short-range Spatial mask scale
0.15
Long-range Spatial mask scale
0.7
Temporal mask scale
1
Mask aspect ratio
[0.75, 1.5]
Batch size
3072
Number of Steps
Variable
Variable
Steps per epoch scale
Start learning rate
0.0002
learning rate
0.000625
Final learning rate
Ic-6
Start Weight decay
0.04
End Weight decay
0.4
Clip grad
0.02
Learning rate schedule
Cosine
Warmup steps
10000
AdamW W 31
0.9
Adam W 82
0.95
"
C TRAINING DETAILS
Recall from Section 2.2 SALT is multi-stage training approach in which the teacher is trained at first via V-Pixel method followed by student training using a frozen or static teacher in the last stage. Table lists hyperparameter information in detail that are used train video encoders with SALT. Observe that we use multi-block masking method (Bardes et al. (024)) for V-Pixel. Note that the hyperparameters for setting up multi-bloc' are copied over from those used in V-JEPA (Bardes et al. 2024).
Table also lists optimization-relate hyperparameters that we used train video encoders. We use value of 240, 000 steps in total to show results in Table [ We conduct ablations with the number of steps set to 120, 000, 160, 000, or 240, 000 for results shown in Figure 2 and discussed in Section 5 Observe that use the standard cosine weight-decay strategy during training (Bardes)
ct al. 2024). We use value of 0.95 for 3g in Adam (Loshchilov & Hutter, 2017) as this value is used by Carreira et al. (2024) train VideoMAE-Hid like models both at scale but most importantly large scale. We also opt not to use virtual early stopping approach adopted in V-JEPA (Bardes et al. 2024] and V-JEPA 2 (Assran et al. 2025) that scales the training steps by 25% to avoid training instabilities the latter part of training. Empirically, we observe that frozen teacher provides a stable representation that allows SALT be stable throughout training.
D EVALUATION DETAILS
We adopt the evaluation protocol used in V-JEPA 2 (Assran al. 2025) that uses attentive probing to ensure fair comparison between SALT, V-JEPA 2 and several baselines reported by Assran et al. (025). We Kinetics-400 (Kay al. 2017), Something Something-Something
(SSv2) (Goyal et al.) 2017) to systematically compare against otate-of-the-eee baselines the results of which are reported T
16

---

Published as a conference paper at ICLR 2026
Systematic evaluation setup for K400 and SSv2 We use imputs with 16 frames, 8 segments clips per input and spatial views per segment which identical to the setting used in V-JEPA 2 (Assran et al. 2025 for this dataset. The probe consists of attentive pooling which is implemented via four Transformer blocks where the first three blocks are self-attention based blocks while the last layer uses coss-attention with learnable query token. This pooling operation is followed by standard linear layer where the number of outputs is set to the number of classes for a classification dataset. This value 400 for Kinetics-400 dataset and 174 for SSv2 dataset.
The attentive probe is trained with AdamW for 20 epochs using a learning and weight decay hy- perparameters that are determined via a grid search. Table 6 reports the hyperparameters that are common to SALT and V-JEPA 2.
The key difference between SALT and V-JEPA that we use a spatial crop of 224 224 while V-JEPA 2 uses 256 x 256. This difference makes the results obtained with SALT even more remark- able as we spend much less compute during probing compared V-JEPA 2 due using smaller resolution.
Table 6: Kinetics-400 and Something-Sometting Something-v2 evaluation hyperparameters that are common to SALT and V-JEPA 2. The results of this evaluation are shown Table Note that "denotes the value is the same the used in K400 evaluation.
Parameter
K400
SSv2
Number frames
16
Segments / Clip
8
2
Views / Segment
3
Frame step
4
Epochs
20
Batch size (global)
256
Classifier heads
20
Classifier learning rates
[5e-3, 3e-3, 1c-3, 3e-4, lc-4]
Classifier weight decay
[.8, 4. .1, .01]
Fast evaluation setup for K400 and SSv2 Due to the sheer volume of compute involved with range of downstream datasets, we
training and probing methods evaluation protocol for many ablations and results shown in the main paper. The main difference is the use of clip and view per frame while keeping the 16 frames per input clip as described above. The evaluation hyperparameters identical to the values used in V-JEPA (Bardes et al.) (202)) and are described in Table 7 for completeness. The results of these evaluations are described in Figures to 8 and
efficient
Table 7: Kinetics-400 and Samething-Someehing--y evaluation hyperparameters that are common to SALT and V-JEPA 2.
Parameter
K400 & SSv2
Number of frames Segments/ / Clip
16
Views / Segment
Frame step
Epochs
20
Batch size (global)
256
Classifier heads
Classifier learning rates
1e-3
Classifier weight decay
.01
COIN, Diving-48 and Jester Evaluations We report results using COIN classification TTag ct al. 2019). Diving-48 (Li et al., (018)), Jester (Materzynska et al. (019) and ImageNet-IK (Rus- Sakovsky al. 2015) benchmarks in addition to Kinetics-400) and Something-Something bench-
17

---

Published as a conference paper at ICLR 2026
marks. The number of classes in COIN, Diving-48, Jester, and ImageNet-1K are 180, 48, 27 and 1000 respectively. Table reports the hyperparameters used to evaluate frozen backbones with these benchmarks. The results of these evaluations reported in Figures I to 7 8
and
Table 8: COIN Tang et al. 2019). Jester (Materzynska et al. 2019). Diving-48 (Li et al. ImageNet-1K 1K (Russakovsky et al. 2013) evaluation hyperparameters that are common to SALT and V-JEPA 2. Note that "denotes the value the same the one used in COIN evaluation.
201 8 and
Parameter
COIN
Jester/Diving-455
ImageNet-II 1K
Number of frames
16
Segments/ Clip
8
4
1
Views Segment
3
Frame step
4
2
NA
Epochs
20
" 1024
Batch size (global)
256
128
Classifier heads
Classifier learning rates
le-3
Classifier weight decay
.01
Intuitive physics We follow the protocol established by Garrido al. (2025) and use the surprise score in our evaluations with MnPhys-2019 or IntPhys (Riochet et al. 2018), GRASP (Jassim et al., 2023) and InfLevel (Weihs et al., 2022) datasets. In the following, by Garrido et al. (2025) to quantify surprise. We let f be the context encoder, be the predictor or the decoder and h be the target encoder. V denotes a frame of video clip, C' denotes the context frame count and M denotes the number of future frames. The surprise at time t is given by:
reproduce the equations used
S. |l9 (fe (Vus-c)) ho (Va++++M) 111-
The surprise above can then be calculated over all windows to obtain the following global surprise score:
Average Surprise-
Σ
Se or Maximum Surprise
max (6(1,14.....T (C+
S..
1E(1.18..... T(CCA..)....
(3)
where we set to and use the average surprise score to quantify the surprise between a pair of videos following the methodology used by Garrido al. (2025). The scores are then converted to relative accuracy using label information for video pairs to obtain the relative accuracy values discussed in Table 9
E ADDITIONAL RESULTS
In this section, we include additional tables and results support figures and tables in the main paper.
E. INTUITIVE PHYSICS BENCHMARKS
In this section, we evaluate the intuitive physics understanding of video models. We follow the protocol and datasets described in Garrido et al. (2025) to test a video model's understanding of intuitive physics zero-shot setting. Following the protocol Garrido et al. (2055) we calculate surprise metric that measures the deviations from expected physical behavior. The benchmark probes the predictor or the decoder to test for physical attributes such as object permanence, spatio- temporal continuity, shape and color constancy, gravity, support, solidity, inertia and collision. We refer the interested reader to Garrido et al. (2025) for additional details on datasets and definition of the attributes mentioned above.
18

---

Published as a conference paper at ICLR 2026
Table 9: Comparison on Intuitive physics benchmarks (IniPhys, GRASP, InfLevel).
Method
Encoder
Predictor
IntPhys
GRASP
Inflevel
Rezults reported in Buldozsarze cl.
2025c
COSMOS 4B (Agaee & p00c
VAL
4B
99.5
60.1
44,8
68.
V-JEPA Burdes
VIT-L
22M
92.2
67.0
58.9
72.7
VJJEPA Badds
WITH
73.0
Rexalts reported in Bonler et al. 2025
V-JEPA Aassra ar
VIF-
22M
87.2
Eval
ViderMAF V2 Wang et al. 20236
ViT-g
12M
59.4
61.3
$4.4
98.4
V-JEPA 2 (V-3,683)
VIT-B
22M
76.6
56.1
52.4
61.7
V-JEPA (V-3.6M)
VIT-L
53.0
57.4
V-JEPA (V-3.6M)
VI-H
22M
88.5
65.1
60.0
71.2
SALT
VIEB
22M
72.4
56.4
$4.9
61.2
SALT SALT
VIT-L
22M
53.5
54.2
66.1
WITH
22M
95.8
58
58.2
70.7
Table 9 shows the results of this benchmark for SALT as well as several baseline video models. Ta- ble 9 shows that SALT's average accuracy scales with model size. SALT compares favorably with
published baselines including COSMOS-4B (Agarwal et al. 2025), V-JEPA (Bardes et al. 2024)
and V-JEPA 2 (Assran et al. 2025) and VideoMAEv2 (Wang et al. 2023b). Finally, we consider setup where we train V-JEPA 2 and SALT models using the same dataset and optimization budget. We observe that V-JEPA 2 models trained under the conditions stated above compare favorably with SALT. These results suggests that emergent intuitive physics understanding behavior observed in video models trained with V-JEPA objective (Garrido et al., 2025) is seen with SALT as well.
E.2
IMPACT OF SACCHER-STUDEN COMPUTE ALLOCATION
(a) Total steps = 120k
(b) Total steps = 160k
(c) Total steps 240k
10.0
26.
31.0
21.0
11.6
253
Aeg. 1op-1 Acouracy P86
22.5
70.4
21.4
70.0-
67.5
00.1
55.0
12.5
21k
419 50%
41A Teacher training steps
BUK
20k 41h 806
Figure 8: SALT trained with compute budget of (a) 120K steps (b) 160K steps and (c) 240K steps. The X-axis shows the number of steps allocated to the teacher with the rest used to optimize the student. Observe that the optimal allocation favors training the student longer than the teacher.
Figure 8 provides alternative view of the plot shown in Figure 7 We train a ViT-L model in this ablation. It is clear from Figure g that the teacher encoder's downstream performance increases with the number of training steps across all values of total number of training steps. Remarkably, the student encoders improve over the teachers that they use to obtain predictions targets. The best performing model is obtained by training teacher for 40,000 steps and using the remaining steps on the student. This observation suggests that the optimal compute allocation should favor the student.
E.3 How TO CHOOSE A TEACHER CHECKPOINT?
A question that arises naturally with SALT is whether there is principled way to choose an optimal teacher checkpoint. By optimal, we here mean choosing a checkpoint that maximizes the student's 'ss performance. We study this question empirically by looking the correlation between the student's ss benchmark accuracy and teacher's 's embedding rank, training loss and teacher model's 's downstream accuracy. Figure A shows plot of embeddings rank where the embeddings are extracted from teacher model versus student accuracy. We use RankMe (Garrido et al. 2023) to estimate the em- bedding rank. Garrido al. 2023) have shown that high embedding rank is necessary condition
19

---

Published as a conference paper at ICLR 2026
18.3
77.
Student Accuracy (80
26
29.5-
25.0-
26.5
73.0
32
100
1500
300
300
320
400
2.00
0.5
Toacher RankMe
Teacher Loss
Figure 9: Teacher quality student performance. We take all the teachers trained in SALT and measure the MankME(Garridde et al. 2023) of the embedding and pretraining loss and analyze the correlation between them and student's downstream performance. Each point represents single SALT run. We control the total training budgets of 160k for both stages to the be same for all models in this comparison.
for good downstream performance in joint-embedding self-supervised learning (JE-SSL) models. We observe from Figure that the the teacher's ' embedding rank is not predictive of student's down- stream performance. A similar trend can be observed from Figure 99 with the teacher's pixel recon- struction or training loss. We see that neither the teacher loss nor its embedding rank are predictive of downstream student's performance.
F FLOATING POINT OPERATIONS (FLOPS) ESTIMATION
A common approach to estimating the total training compute for Transformers, including ViTs, by using the well-known 6ND formula (Kaplan al.) 2020; Hoffmann al. 2022). Here N stands for the model parameter count while D represents the total number of tokens used to train the model. This simple approximation assumes that the backward pass during training costs twice as much as the forward pass, Consequently, we use 2.N D to approximate the training compute of teacher model that provides targets in distillation- based methods. The total number of input tokens observed by model during training is a function of input resolution, spatial for image models and spatio-temporal for video models, the patch size, the batch size per step and the total number of optimization steps. Note that we include the embedding layer in our parameter count. With these preliminaries place, we present the total compute estimate for models presented in Table I
VAde (Wang et al., 2023b) We use a masking ratio of 0.9 and 0.5 for Video- MAEv2 (Wang al. (2023)) encoder and decoder respectively. The model considered here is ViT-g model with an input patch size of 14 that operates on spatial input of size 16 x 16. Our analysis uses 1200 epochs for calculating the number tokens in the encoder and decoder. The other details used to estimate FLOPs is shown in To
(Assran et al. 2025 The spatial resolution is 256 x 256 with patch size of 16. We use a masking ratio of 0.9 for the encoder on average following the recommendation made in V- token count that is half of that by the encoder due to temporal stride being set to 2. In other words, the predictor sees the union of mask tokens used for missing regions and context tokens used for visible regions. We assume that the model is trained for 240,000 steps using a batch size of 3072. Note that we need to account for the teacher forward call that do our analysis.
V-JEPA 2
JEPA (Bardes al. 2024h. The predictor operates
The spatial resolution is 224 x 224 with patch size of 16. We use a masking ratio of 0.9 for all stages in our training we use the same multi-block masking for training teacher and student encoders. The rest of the details are identical to those described above for V-JEPA 2. The main difference between as well using a smaller resolution for the inputs. Together, these significantly lower the training compute requirements for SALT over V-JEPA 2.
SALT
method and V-JEPA 2 is the of same-sized smaller teacher encoder
20

---

Published as a conference paper at ICLR 2026
Finally, we report the GPU-hours estimated by running training for a 20 steps single NVIDIA A100 GPU in Table IT] We observe a strong correlation between the ordering models provided by GPU-hours versus that obtained from "6ND" FLOPs estimate.
Table 10: FLOPS estimate for VideoMAEv2, V-JEPA and SALT models.
Model
Input Resolution
N₂
N,
D, (x10")
Dr
Samples (B)
Total FLOPs
(B)
(B)
(x10%)
(x10")
(x10"))
VideoMAEE2-g/14
16 224 224
1.1
0.012
331.8
165.9
1.6
2.2
V-JEPA 1/16
256
0.303
0.303
0.022
3019,9
1510.2
0.7
V-JEPA 2 H/16
16 x 256 256
0.632
0.632
0.022
302.0
3019.9
1509.9
0.7
3.5
V-JEPA 2 g/16
16 x 256 x 256
1.012
1.012
0.022
302.0
3019.9
1509.9
0.7
5,3
SALT-L/16
16 x 224 x 224
0.3
0.303
0,022
154.1
1541.4
770.7
0.7
1.2
SALT-H/16
16 x 224 x 224
0.6
0.303
0.022
154.1
1541.4
770.7
0.7
SALT-g/16
0.022
1541.4
770.7
SALT-G/16
16
224 224
1.8
0.303
0.022
154.1
1541.4
770.7
0.7
2.6
Table 11: Training compute and GPU hours. We evaluate our models on SSv2 with input of 16x2x3 (*V-JEPA 2 uses spatial resolution of 256 x 256, and SALT utilizes 224 224.). We compute TFLOPs under the same batch size and masking strategy, and measure on one single A A100 GPU for all methods ensure fairness and we exclude data-loading overhead and GPU-communication load from all measurements to ensure they are CPU-agnostic. The results in this table are used
Figure T
Methol
Teacher Parama
Student Params
A Seen Samples (<:1P))
Total Compute (x:(()) FLOP))
GPU-brx
S5v2 Top-1 (%)
16:2x3
16x16]
VJEEN WTT LAaouu at
302M 631 18
302M 531M
2.4
1.9
9800
73.7
09.6
JUPA Mastmmi Ca
7.4
3.5
14377
09.6
VURUN WT Aeral &
18
7.4
5,3
18708
75.3
72.2
SALT--Sugo-1 VIT-I.
744
VIT-I.
SALT
SALT
SALT
12509
G COMPUTE BUDGET SPECIFIED VIA FLOPS AND OPTIMIZATION STEPS
While we use FLOPSs in our accuracy-complns trade-off analysis, we specify compute via the total number of optimization steps in our experiments. The use of the latter quantity is natural in our setup as the teacher's EMA update in V-JEPA 2 (Assran et al. (025) depends on the student getting updated first which is similar to the nature of the update in two-stage training needs to be trained first followed by the student. The advantage with SALT is due from the fact that the teacher training is light-weight, and crucially, once teacher model is trained, it may be used to train multiple student models.
the teacher
21

---

Published as a conference paper at ICLR 2026
H ADDITIONAL TABLES AND FIGURES
Table 12:
V-JEPA 2 vs. SALT on same pretraining set. Kinetics-400 uses 16 X 1 (number of
frames in clip by temporal crops by spatial crops), Something-Something v2 (SSv2) uses 16 x x
while COIN is run with 16 x 8 3. All models are evaluated using spatial resolution of 224 x 224 pixels. The results in this table are used in Figure
Method
Teacher
Student
IN-1K
K400
SSv2
COIN
Diving-48
Jester
Avg
VIT-B
ViT-B
66.9
66.9
61.4
68.4
71.0
95.9
71.8
VJJEPA (w/ our dataset)
VIT-L
VIT-I.
73.7
73.3
68.4
83.1
82.1
97.0
79.6
VIT-H
VIT-H
76.7
73.6
68.9
84,9
84,5
97.1
81.0
N/A
VIT-B
70.3
65.2
60.9
73.3
72.9
95.7
73.1
SALT Stage (V-Pixel)
N/A
VIT-I.
75.5
70.4
66.2
77.9
76.8
96.9
77.3
VIT-B
VIT-B
70.9
96.7
78.0
VIT-I
76.0
71.3
85.3
82.5
97.2
81.9
ViT-L
VIT-H 79.6
77.2
72.6
87.0
86.4
97.3
83.4
SALT Stage 2
VIT-g 79.7
78.0
72.9
87.0
85.5
97.4
83.4
VIT-G
80.3
78.9
73.2
87.5
85.3
97.4
83.8
Table 13: Ablation on teacher pretraining datasets, We test teacher and student model using frozen- backbone evaluation %). We report Top-1 accuracy on K400, SSv2, and ImageNet- 1k, and COIN. This ablation is used in Figure
Dataset
# Samples
K400
SSv2
IN1k
COIN
Avg
SAI.T-teacher
V-3.6M (default)
3,626,089
70.4
66.2
75.5
77.9
72.5
K710
657,217
71.0
65.2
74.2
78.9
72.3
Panda2.8M
2,799,959
69.2
64.9
75.3
79.5
72.2
SSv2
168,913
56.8
63.6
61.7
69.0
62.8
K710 + SSv2
826,
70.0
67.8
74.0
79.2
72.8
ImageNet-Ik
1,281,167
51.9
39.3
80.6
67.4
59.8
SALT-student
V-3.6M (default)
75.5
70.9
78.4
84.9
77.4
K710
75.7
70.6
78.4
85.0
77.4
SSv2
72.9
69.8
76.2
83.1
75.5
V-3.6M (default)
Panda2.8M
75.3
70.5
78.4
84.4
77.2
K710 + SSv2
75.1
71.1
78.0
84.9
77.3
ImageNet-Ik
72.1
66.5
79.1
82.0
74.9
I LLM USAGE STATEMENT
During manuscript preparation, LLMs were used to help with editing and polishing the draft.
22

---

Published as a conference paper at ICLR 2026
Table 14: Teacher model size ablation. We report frozen-backbone Top-1 on K400, SSv2, INIK, and COIN. The top block shows teachers evaluated directly. The lower blocks show students distilled from different teacher sizes (two student sizes: ViT-L and ViT-G). The data in this table is used Figure
Model size
K400
SSv2
INlk
COIN
Avg
Teacher
Student
VIT-B
63.2
60.9
70.3
73.3
67.4
VIT-L (default)
70.4
66.2
75.5
77.9
72.5
ViT-H
71.1
67.4
76.7
81.5
74.2
VIT-G
73.6
68.5
77.4
81.4
75.2
SALT-student
ViT-B
74.4
69.5
78.0
84.0
76.5
VIT-L (default)
75.5
70.9
78.4
84.9
77.4
VIT-I.
ViT-H
75.6
70.7
78.3
84.4
77.3
VIT-G
75.5
71.5
78.5
84.7
77.6
ViT-B
76.9
71.8
79.0
85.4
78.3
VIT-L (default) VIT-G
77.6
71.9
79.3
87.0
79.0
ViT-H
77.5
72.4
79.6
85.3
78.7
Table 15: Masking strategy ablation. We report COIN; Top block: teachers evaluated directly. Bottom block: students (fixed student recipe) trained from different teachers. The data in this table used in Figure
Top-1 on K400, SSv2, INIK, and
masking strategy
Masking ratio
Student Masking strafegy
K400
Avg
SALT-saacer
Long-short block mask (default) Random sube
x2
0.9
70.4
66.2
75.5
77.9
72.5
[0.9, 0.91
69.2
64.7
74.1
74,9
70.7
[0:9]
Causal mask
[0.9. 0.9]
47.4
34.3
57.3
58.8
19.5
SAl.T-studen:
short block mask (default) Random
0.9
75.5
70.9
78.4
54.9
77.4
[0.9, 0.91
Long-short block musk
Rondom tube
x1
[0.9]
75.0
70.4
78.0
84.8
77.1
Causal mask
x2
[0.9.0.9)
71.7
66.3
75.0
81.7
73.7
Table 16: Compute-accuracy tradeoffs at matched total steps. Each block fixes the total pretrain- ing steps (budget) and compares V-JEPA,2 two-stage schedule (teacher+student), FLOPs reported as x1221 Metrics are frozen- backbone Top-1 on K400, SSv2, INIk, and COIN. This table includes data presented Figure
Teacher Steps
Teacher FLOPs
Student Steps
Student Pl.OPs
Total FLOPs
Total Steps
K400
S8y2
INIK
COIN
Budget: steps
V-JEPA (baseline)
1.900
[.900
240%
73.3
68.4
73.7
83.1
74.6
306+80R
0.717
80K
0.476
1.193
2406
76.5
71.8
79.0
86.8
78.5
120k.
806
0.241
1601.
0.951
1.192
240%
76.0
71.3
79.0
85.3
77.9
40&
0.121
200k
1.190
1.311
240%:
76.3
71.7
79.1
85.6
78.2
20k
0.061
2201
1.309
L,370
210%
75.8
71.2
78.7
$4.9
77.7
Budget: steps
V-JRPA (huseline)
1.260
1.260
160k
73.5
68.3
74.%
82.9
74.9
S0k
0.241
S0k
0.476
0.717
160k
75.5
70.9
78.4
$4.9
77.4
120k
0.714
75.5
7009
74.9
78.0
Budget: 120k total steps
V-JEPA (baseline)
0.951
0.951
120%
73.3
67.9
74.7
81.5
74.1
77.7
40%:
0.121
0.476 0.595
0.597
120%
75.0
70.2
78.0
$4.9
77.0
200c
0.061
100k
0.656
120k
74.6
70.1
77.5
$4.5
76.7
23

---

Published as a conference paper at ICLR 2026
64
67
250
300
350
600
450
500
550
600
Studont RankMttc
Figure 10: Correlation between RankME and downstream accuracy. We use the same SALT-Stage- 1-80k teacher checkpoint.
24


Reviews:
[{"review_id": "oiYqxuBhHm", "reviewer": "Reviewer_3hkH", "summary": "This paper introduces SALT (Static-teacher Asymmetric Latent Training), a two-stage framework for compute-efficient self-supervised video representation learning. The method challenges the necessity of the complex, online, EMA-updated teacher used in state-of-the-art models like Video Joint Embedding Predictive Architectures (V-JEPA). The core design of this work is borrowed from Masked Video Distillation, using two-stage training: (1) At stage-1, a teacher encoder is trained using a simple pixel-reconstruction objective, similar to VideoMAE; (2) At stage-2, the teacher encoder is frozen, and a student encoder, along with a predictor, is trained to predict the teacher's latent representations for masked regions of the input video. The results of experiments show that high-performing student models can be trained using smaller teachers—models. The main results show that SALT outperforms the strong V-JEPA 2 baseline on multiple video understanding benchmarks (e.g., Something-Something-v2, Kinetics-400), especially when matched for computational cost.", "questions": "1. The V-JEPA paper reported frozen evaluation results using a pretrained model from MVD. Given that this represents a highly relevant baseline for a frozen-teacher approach, the authors should include the results of this baseline in the main results table.\n\n2. Have the authors considered the possibility of iterating the training process? Specifically, could the student model trained in Stage 2 be frozen and used as a new teacher to train a subsequent, potentially stronger, student model? This appears to be a natural extension of the method.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental validation is extensive and rigorous. The systematic comparison against strong baselines, including a retrained V-JEPA 2, convincingly demonstrates SALT's superiority in both accuracy and efficiency. The paper includes a comprehensive set of ablations that analyze the impact of key design choices, such as the teacher's training data, masking strategy, model size, and the compute allocation between teacher and student.\n\n2. The demonstration that the student's training loss is highly predictive of downstream performance (Figure 3) is a major practical advantage.\n\n3. This paper is very well-written, clearly structured, and easy to follow.", "weaknesses": "1. **Limited Novelty in the Self-Supervised Learning Framework**: The proposed two-stage distillation framework, which uses a frozen teacher to provide target features, shares a significant conceptual overlap with prior work. Specifically, the core idea was proposed and studied in Masked Video Distillation (MVD)[1] two years ago and has been scaled and improved in subsequent works like VideoPrism[2] and InternVideo2[3]. Therefore, the method's design is not an entirely novel concept. The paper's primary contributions are better framed as a large-scale empirical study that (a) provides a fair and direct comparison against a strong momentum-encoder baseline (V-JEPA 2) on larger datasets and models, and (b) uncovers some valuable insights like smaller teachers can produce strong students. Key conclusions related to the method's design (such as the student outperforming the teacher or a frozen teacher being superior to an EMA teacher) are consistent with findings in these earlier studies. It would be beneficial for the authors to more precisely position their contributions in the introduction, emphasizing the scientific insights and empirical discoveries rather than the originality of the algorithmic framework itself.\n2. **Lack of Deeper Analysis on Teacher Model Size Saturation**: The experiments in Figure 6 suggest that, under the paper's settings, increasing the teacher's model size does not yield significant performance gains for the student. However, the paper does not offer a deep theoretical or experimental analysis to explain this phenomenon. For instance, an analysis of the target features provided by teachers of different sizes, or an investigation into how this finding correlates with the student's model size, would provide much greater insight into the dynamics of the teacher-student relationship in this framework.\n3. **Unexplored Performance Bottleneck at Larger Scales**: Table 1 reveals a slight performance degradation on the SSv2 benchmark as the SALT model is scaled from 1B to 2B parameters. The paper does not address or analyze this observation. This trend suggests that SALT may face a performance bottleneck at very large scales, and an investigation into the potential causes would significantly strengthen the paper's contribution to understanding the scalability of this approach.\n\n[1] Wang, et al. \"Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning.\" CVPR 2023.\n\n[2] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024).\n\n[3] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MjfS46P9zA", "reviewer": "Reviewer_ZRSW", "summary": "This paper introduces SALT (Static-teacher Asymmetric Latent Training), a two-stage, unregularized framework for video representation pretraining. The authors challenge the conventional approach of using dynamic teachers in self-distillation architectures, such as V-JEPA, and propose that a less optimal, pre-trained teacher is sufficient for distilling a student model. Specifically, they pretrain a teacher model using a pixel reconstruction objective, freeze the teacher model, and then use it to distill the student, demonstrating performance improvements over V-JEPA under fair settings. Extensive experiments validate the effectiveness of SALT and provide justification for its design choices.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an alternative to traditional V-JEPA, which relies on EMA and stop-gradient operations.\n2. The proposed SALT method demonstrates superior performance and greater efficiency compared to V-JEPA.\n3. The experiments are thorough and well-executed, including ablation studies that effectively validate the contributions of each design element in the proposed method.", "weaknesses": "1. The VJEPA framework utilizes EMA-based self-distillation, where performance improvement is driven by the joint optimization of both the teacher and student models. However, it seems counterintuitive that SALT employs a smaller-scale or less capable teacher to distill a more performant student. This discrepancy warrants a more detailed analysis and clarification.\n2. The authors claim that \"a high-performing teacher is not necessary to train a high-quality student.\" This assertion appears to hold under the assumption of fixed total training steps across two stages. However, in an unconstrained setting, does the performance of the teacher model positively correlate with that of the student? Specifically, if teachers are trained for 20k, 40k, and 80k steps, respectively, and students are trained for 80k steps in all cases, would the performance of the models improve sequentially?\n3. The proposed method primarily relies on pretraining and distillation, while it seems that not explicitly present temporal optimization. Recent advancements in image-to-video distillation [1-3] have incorporated temporal relationships into strong representation models during distillation. The authors are encouraged to compare their approach with these recent methods in the related work section, in order to investigate how temporal information can be leveraged in representation distillation.\n\n[1] Liu Y, Xu Q, Wen P, et al. When the future becomes the past: Taming temporal correspondence for self-supervised video representation learning. In CVPR, 2025.\n[2] Li R, Liu D. Spatial-then-temporal self-supervised learning for video correspondence. In CVPR, 2023.\n[3] Hu Y, Wang R, Zhang K, et al. Semantic-aware fine-grained correspondence. In ECCV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ogifavZUn", "reviewer": "Reviewer_R6kh", "summary": "The paper revisits video self-supervised learning under the JEPA framework and introduces SALT , which replaces the EMA-updated teacher with a frozen one.\nBy decoupling teacher and student training into pixel reconstruction and masked latent prediction, the method improves efficiency, transparency, and scalability. Experiments show that SALT outperforms V-JEPA2 at matched compute while remaining robust to teacher quality.", "questions": "1.Although SALT claims lower computational cost than the EMA baseline, it still requires training the teacher before the student. Compared to end-to-end EMA methods, does it truly save overall training time?\n2.Experiments show that SALT achieves significant improvements on SSv2, but only limited gains on K400. Is SALT more suited for temporal modeling? If the task primarily involves appearance or scene recognition, does the method still maintain a clear advantage?\n3.SALT relies on a two-stage teacher-student training, which adds some training complexity. Additionally, its performance gains on the K400 dataset are modest and fall short compared to the latest methods (PEcoreG, InternVideo2-1B).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The method consists of two stages: pixel reconstruction and masked latent prediction, allowing small teacher models to train large-scale student models. Each training stage has a clear loss function with physical meaning, enabling SALT to avoid representation collapse.\n2.On multiple mainstream video understanding benchmarks, SALT consistently outperforms V-JEPA2 with the same model architecture, particularly excelling on motion understanding tasks (SSv2), demonstrating stronger temporal modeling capabilities.\n3.SALT can successfully train larger student models using smaller teacher models, improving computational efficiency and scalability.", "weaknesses": "1.Although the teacher is frozen, a pixel reconstruction training stage is still required first, introducing an additional pretraining step and making the process slightly longer compared to fully end-to-end methods.\n2.While SALT performs exceptionally on temporal tasks (SSv2), its improvements on appearance-based tasks (such as K400) are modest, and it still lags slightly behind the latest methods (PEcoreG, InternVideo2-1B).\n3.Although the overall computational cost is lower, SALT’s two-stage process still requires training the teacher model first, increasing the overall implementation complexity and making the workflow slightly longer compared to end-to-end EMA-based methods.\n4.Missing code reduces reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWMfC8JfzM", "reviewer": "Reviewer_k4fL", "summary": "The paper proposes a modification to the V-JEPA framework, which consists of two stages. In the first stage, a teacher network is pretrained on a dataset—VMAE-v2 in this case. In the second stage, a significantly larger student network is trained to predict the teacher’s features using the V-JEPA loss function. Since the teacher is considerably smaller than the student, the total computational cost is dominated by the student. Nevertheless, this approach achieves higher performance and converges faster than the original V-JEPA framework, which trains the teacher and student jointly via EMA, under the same computational budget.", "questions": "Please refer to weaknesses. I tried to provide clear questions there.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of employing a smaller teacher than the student is both interesting and computationally efficient.  \n2. The method’s two-stage design simplifies training and is likely to offer greater stability compared to EMA-based teacher frameworks such as V-JEPA.  \n3. The approach is supported by an extensive set of experiments ensuring fair comparisons in terms of total compute and performance.  \n4. The method is evaluated across diverse settings, including image and video datasets as well as intuitive physics understanding, providing a comprehensive assessment of its effectiveness.", "weaknesses": "1. The main motivation behind the proposed framework is to address the instability of V-JEPA, which reportedly requires meticulous hyperparameter tuning. However, the paper does not provide empirical evidence to support this claim. Is your model more stable with respect to the training hyperparameters listed in Table 5? It would strengthen the argument to include a comparison between V-JEPA and SALT in terms of stability across those hyperparameters.\n\n2. The paper does not include an ablation study on the impact of teacher pretraining. How sensitive is the proposed method to the choice of the pretrained teacher? What is the rationale for using VMAE-v2 specifically? I encourage the authors to include an ablation for this factor, at least for a ViT-B teacher. For example, a teacher could be selected from one of the following works: [1, 2, 3, 4].\n\n3. The concept of training from a frozen teacher network has been previously explored in [2, 4], although in those works the teacher is an image model. It would be fair to discuss these approaches in the related work section, as the underlying idea is conceptually related to the current paper.\n\n4. Is there a particular reason why the proposed method is only evaluated on video datasets? Was it also tested on images, and if so, did it perform comparably? Including such an analysis could clarify whether the idea of using a smaller teacher than the student is specific to video settings.\n\n---\n\n**References**\n\n1. *Masked Motion Encoding for Self-Supervised Video Representation Learning*, CVPR 2023  \n2. *SIGMA: Sinkhorn-Guided Masked Video Modeling*, ECCV 2024  \n3. *MGMAE: Motion Guided Masking for Video Masked Autoencoding*, ICCV 2023  \n4. *SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning*, CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)