Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
Authors: name: Xianhang Li; name: Chen Huang; name: Chun-Liang Li; name: Eran Malach; name: Josh Susskind; name: Vimal Thilak; name: Etai Littwin
Published: 2025-09-29 05:55:17

Abstract: Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable
off-the-shelf video representation by predicting masked regions in latent space
with an exponential moving average (EMA)-updated teacher. While EMA prevents
representation collapse, it complicates scalable model selection and couples
teacher and student architectures. We revisit masked-latent prediction and show
that a frozen teacher suffices. Concretely, we (i) train a target encoder with
a simple pixel-reconstruction objective under V-JEPA masking, then (ii) freeze
it and train a student to predict the teacher's latents on masked regions. This
leads to a two-stage, unregularized scheme that we refer to as SALT
(Static-teacher Asymmetric Latent Training). SALT decouples optimization into
pixel reconstruction (teacher) and masked latent prediction (student),
increasing transparency, efficiency, and scalability while preserving the
ability of representation to generalize under frozen evaluation. Empirically,
our student models outperform recently proposed V-JEPA 2 encoders under frozen
backbone evaluation across diverse benchmarks. They are also more
compute-optimal: at matched pretraining FLOPs, our method achieves higher
probing accuracy, and its scaling curves dominate V-JEPA's accuracy-FLOPs
Pareto frontier. Finally, we find that student quality is remarkably robust to
teacher quality: high-performing students emerge even with small, sub-optimal
teachers. This points to a compute budget allocation that should overwhelmingly
favor the student. These results position SALT as a simple, scalable, and
compute-efficient alternative to EMA-based self-distillation for video
representation learning.