Title: VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
Primary area: foundation or frontier models, including LLMs
Keywords: JEPA, VLM, video-language, efficiency

Abstract: We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model can focus on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that the model can naturally support selective decoding and reduce the number of decoding operations by ~2.85x while maintaining similar performance compared to dense non-adaptive uniform decoding. Beyond generation, the embedding-space formulation naturally supports open-vocabulary classification and retrieval without any architecture modification. VL-JEPA achieves leading SoTA zero-shot results on diverse real-world video-language understanding tasks on COIN, CrossTask, EgoExo4D, SSv2, and WorldPrediction-WM, substantially outperforming larger generative VLMs trained with more data.

TL;DR: We introduce a vision-language model based on JEPA, that achieves competitive socres while being more efficient during training and inference.