VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
Authors: Delong Chen
Mustafa Shukor
Théo Moutakanni
Willy Chung
Lei Yu
Tejaswi Kasarla
Allen Bolourchi
Yann LeCun
Pascale Fung
Venue: ICLR 2026 Poster (2026)
Keywords: JEPA
VLM
video-language
efficiency
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a vision-language model based on JEPA, that achieves competitive socres while being more efficient during training and inference.

Abstract: We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by ~2.85× while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance of VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance to classical VLMs (InstructBLIP, QwenVL) on four VQA datasets—GQA, TallyQA, POPE, and POPEv2—despite having only 1.6B parameters.

Published as a conference paper at ICLR 2026
VL-JEPA: JOINT EMBEDDING PREDICTIVE ARCHITECTURE FOR VISION-LANGUAGE
Delong Chen! Mustafa Shukor Théo Moutakanni', Willy Chung! Jade Yu', Tejaswi Allen
Allen Bolourchi'o, Yann LeCun Pascale Fung!2 12 Meta FAIR, 2KKUST, SSoboonne Université, ANYU, University of Amsterdam, USC 'Equal contribution
delong. wwwn@connect.....cccccmccmcocc.c ust. hk
ABSTRACT
We introduce VL-JEPA, a vision-language model built on Joint Embedding Pre- dictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in abstract representation space, the model focuses task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VI.M training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text de- coder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that re- duces the number of decoding operations by ~2.85 x while maintaining simi- lar performance compared to non-adaptive uniform decoding. Beyond genera- tion, the VL-JEPA's embedding space naturally supports open-vocabulary classi- fication, text-to-video retrieval, and discriminative VQA without any architecture modification. On cight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance classi- cal VLMs (InstructBLIP, Qwen on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.
INTRODUCTION
One of the most important aspects of advanced machine intelligence the ability to understand the physical world that surrounds us. This ability enables Al systems to learn, reason, plan and act in the real world in order to assist humans (LeCun) (002)). Intelligent systems that need to act in the real world includes wearable devices and robots (Fung et al., 2025). Machine learning tasks that make up for this ability include captioning. retrieval, visual question answering, action tracking,
reasoning and planning etc (Bordes et al.) 2224 Chen et al. 2025b). Systems for such real-world applications must have real-time response with low latency and inference cost.
Currently, the common approach to achieve these tasks is use large token-generative Vision Lan- guage Models (VLMs) (Liu et al. 2023 Dai et al., 2023 Alayrac et al., 2022] Chen et al. 2024b) Cho et al. 2025; Chen et al. 2022). which takes visual input Xy. textual query XQ to generate desired textual response Y autoregressively in token space, i.e., (Xv, XQ) + Y. This is straight- forward but inadequate for two main reasons. First, VLMs are expensive to develop, because they are trained to generate responses Y to queries by capturing both task-relevan semantics with task- irrelevant surface linguistic features such as words choice, style or paraphrasing. During training. VLMs must model both aspects, which results in unnecessary computing effort spent producing di- verse token sequences that ultimately do not impact the correctness of the output. Second, taal-tmme tasks involving live streaming video (e.g., live action tracking) require sparse and selective decoding (e.g.. emitting a description only when a new event occurs) (Zhou et al.) 2024). However, VLMs rely autoregressive token-by-toke decoding, which must be completed before revealing the un-

---

Published as a conference paper at ICLR 2026
derlying semantics of Y. This process introduces unnecessary latency and hampers the ability to update semantics dynamically in real time.
This paper introduces the Joint Embedding Predictive Architecture for Vision-Languadn (VL-JEPA), turning expensive learning of data-space token generation into more efficient latent-space semantic prediction. As illustrated in Fig. | the model employs x-encoder map vision inputs Xy into embedding Sy, a y-encoder to map the textual target Y into an embedding Sy, and a predictor that learns the mapping XQ) + Sy where XQ is textual query (i.e, the prompt). The training objective is defined in the embedding space Lyz CEPA = D(Sy, Sy) instead of the data space LVTM Y), During inference, y-decoder reads out the predicted embedding Sy to text space Y when needed.
Thanks to its non-generative nature, VL-JEPA is not forced to reconstruct every surface detail of Y in the token space. Instead, it only needs to predict the abstract representation Sy in the em- bedding space. In the raw one-hot token space, different plausi- ble Y outputs for the same input often appear nearly orthogonal if they don't share overlapping tokens. However, in the embed- ding space, these diverse targets can be mapped to nearby points that share similar semantics. This simplifies the target distribu- tion thus makes the learning process more efficient. In addition, unlike VLMs, this approach eliminates the need for learning lan- guage generation with a heavy decoder during training, resulting in significant efficiency gains.
Y-Decoder
5,
Predictor
5,
Textual Query
X-Encoder
Y-Encoder
Visuel
Textual Target
Input
Figure I: VL-JEPA.
Thanks to its non-autoregreesive nature, VL-JEPA can produce continuous streams of target se- mantic embeddings within sliding windows with minimal latency as only require single forward pass without autoregressive decoding. This is particularly advantageous for real-time online appli- cations such as live action tracking, scene recognition, or planning. where the embedding stream can be selectively decoded by a lightweight y-decoder, enabling efficient and prompt updates.
In this work, we empirically validate the advantages of VL-JEPA. We conduct a strictly controlled comparison against classical token-generative VLM (Liu et al. [023 Cho et al. 2025): both setups use the same vision encoder, spatial resolution, frame rate, training data, batch size, and number of iterations, etc., with the only difference being the objective in token space or embedding space. Under this matched training condition, VL-JEPA delivers consistently higher performance on zero- shot captioning and classification while using roughly half the trainable parameters, indicating that embedding- space supervision improves learning efficiency.
Beyond the training phase, VL-JEPA also delivers substantial inference-time efficiency improve- ment through selective decoding, where decoding happens only due to significant change in the predicted embedding stream. Empirically, this strategy reduces the number of decoding operations by- ~2.55 x while preserving overall output quality measured by average CIDEr scores.
pretraining stage using caption data tablish robust vision-language alignment, and 2) a supervised finetuning (SFT) stage that equips the model with VQA capabilities. The model resulting from the first stage, denoted VL-JEPABASE. is evaluated on zero-shot classification and text-to-videe retrieval. VL-JEPABAsE outperforms CLIP (Radford al.] 2021). SigLIP2 (Tschannen et al.) 2025). and Perception Encoder (Bolya et al.) 2025) models in terms of average classification accuracy (across 8 datasets) and retrieval recall @ (across datasets). Following the second stage, the resulting VI.-JEPA demonstrates signifi- cantly improved classification performance due to its exposure to in-domain training data. As uni- fied generalist model, VL-JEPAsrT approaches the performance of specialist models optimized for individual benchmarks. Simultaneously, VL-JEPAsET exhibits effective VQA capabilities, achiev- ing performance on par with established VLM families, such as InstructBLIP (Dai et al.) (002)) and Owen-VL. (Bai et al., 2003),, across four datasets covering compositional visual reasoning (Hudson & Manning 2019), complex object counting (Acharya et al.) 2019). and object hallucination al. 2023b 2025b).
Our final VL-JEPA models are trained in two stages:
In summary, the contributions of this paper are as follows:
2

---

Published as a conference paper at ICLR 2026
We introduce VL-JEPA, the first non-generative model that perform ceneral-domann Assionllaguaae tasks in real-time, built on joint embedding predictive architecture.
We demonstrate in controlled experiments that VL-JEPA, trained with latent space embed- ding prediction, outperforms VLMs that rely on data space token prediction.
We show that VL-JEPA delivers significant efficiency gains over VLMs for online video streaming applications, thanks to non-autoregressive design and native support for se- lective decoding.
We highlight that our VI.-JEPAsPT model, with an unified model architecture, can effec- tively handle a wide range of classification, retrieval, and VQA tasks the same time.
METHODOLOGY
We propose VL-JEPA (Fig. D. a model with the joint embedding predictive architecture (JEPA) for vision-language tasks. VL-JEPA is trained with triplets XXvxXe, Y), where Xy denotes the visual input (a single image or a sequence of video frames), XQ is a textual query (i.e., a question) and Y the textual target (i.e., the answer) be predicted. The VL-JEPA comprises of four components:
1. X-Encoder (Xv + Sy) compresses high-volume visual inputs to compact visual embeddings-a sequence of continuous vectors analogous to "visual tokens" in classical VI.Ms.
2. Predictor ((Sy.Xa) XQ) + Sy) is the core component of VL-JEPA. It maps visual embed- dings to a prediction of target embedding, with a textual query as conditioning.
3. Y-Encoder (Y + Sy) embeds the textual target into a continuous latent space as the pre- diction target. The target embedding expected abstract away task irrelevant information.
4. Y-Decoder (Sy + Y) is not involved during the main training phrase of VL-JEPA. At inference time, it translates the predicted embedding as human-read readable text when necessary.
Selective Decoding
Project
Vislon-textt-o--eet Generation
Predictor
Regularizatio
"Dhverbe video aestennca""
Projection
Projection
Discriminative VQA
Visual Embedeinggs
Target Embbedding S,
& Classification
VL-JEPA
ViEareder
"Watt sa the actioon showadd"
X-Eneoder VuirAA
Tokenization Embedciing
V-Encodar Embaddd recCerme
Text-to-vision
XX
VL-JEPA
Y-Encoder
Input
Testual X
Retrieual Query
Figure 2: Left: VL-JEPA Architecture. It learns to predict the target embedding Sy, instead of reconstructing the raw target Y in token space as in classical VLMs. Right: VI-JEPA Ap- plications: It handles vision-text-to-tex generation tasks (e.g., captioning) with selective decoding mechanism natively supported. Furthermore, VL-JEPA's embedding space facilitates discriminative VQA, open-vocabular classification and text-to-vides video retrieval tasks using a single unified model architecture.
Fig. 2 illlsstrttes how we instantiate the VL-JEPA architecture in this paper. For the X-Encoder, we chose V-JEPA (Assran al., 2025), a Vision Transformer that outputs sequence of visual tokens, which are then projected and fed into the P Predi ctor initialized using Llama 3 Transformer layers. Query conditioning achieved by tokenizing and embedding the textual query and feeding the resulting textual token embeddings into the Predictor along with the visual embeddings. The outputs of the Llama Transformer layers are pooled and projected into the target embedding space produced by the Y Y-Encoder, which is initialized by EmbeddingGeemma-300 (Vera al. 2025
We provide more technical details in $
Training Objective. JEPA models typically optimize two objectives jointly: 1) prediction error in the embedding space, and 2) additional regularization that avoids representation collapse (Bardes) al. 2021 Balestriero & LeCun 2025 Any loss that implements these two properties can be
3

---

Published as a conference paper at ICLR 2026
applied to VL-JEPA. Alternatvely,, the regularization term be replaced by other anti-collapse strategies, such as using an exponential moving average (EMA) for the Y-Encoder (Assran et al., 2025 freezing the Y-Encoder (Zhou et al. 2025).
In this work, we adopt the InfoNCE loss (Radford et al., 202I due to its maturity in the vision- language domain. More advanced non-sample-contraat
regularization, such VICReg (Bardes) et al. 2021) and SIGReg (Balestriero & LeCun, (025) can also be applied but we leave the explo- ration to future works. InfoNCE loss can be mathematically divided (Wang & Isola 2020) into: 1) representation alignment term that minimizes the distance between normalized prediction and target embeddings. and 2) a uniformity regularization term that pushes embeddings in a batch apart from each other. thus avoiding representation collapse. We train the P Predi ctor and the Y Y-Encoder jointly with bi-directional InfoNCE loss, enabling them to mutually learn from each other.
Compared to the token-space loss used by generative VLMs, calculating the training loss in the embedding space is beneficial due to the simplified target distribution. Specifically, many real- world prediction tasks are inherently ill-posed: for the same input X, there may exist multiple plausible targets Y that are all acceptable. For example, given the query "What will happen here if I flip this light switch down?", , both "the lamp is turned off" and "room will go durk" are valid answers. In the raw one-hot token space, however, the two sequences are orthogonal since they share no overlapping tokens. But when VL-JEPA's Y-En.oder r embeds them into nearby points (ideally yielding a compact unimodal distribution), the learning task becomes much easier: the model no longer needs fit multiple disjoint high-density regions in sparse token space, but only single coherent mode in a continuous embedding space.
Multi-tasking, VL-JEPA supports diverse tasks using a single, unified architecture (Fig. 2 For vision-text-to-teex generation tasks, such as captioning or open-ended VQA, the query Xe is a captioning prompt or question, and the predictor learns to predict the embedding of the target output, Sy, which then decoded into text. VL-JEPA also supports CLIPP-ttle open-vocabulary classification and discriminative VQA, where candidate label texts are encoded into embeddings and compared with prediction Sy to select the nearest match. For text-to-video retrieval, candidate videos are mapped to their predicted embeddings Sy using retrieval a captioning prompt, and then ranked by similarity to the encoded textual retrieval query.
Selective Decoding. Real-world video applications often require online streaming inference, such as tracking user actions in smart glasses for procedural assistance (Chen et al. 2024c), monitoring world states for online planning, navigation and robotics (Shukor al., 2025 Black et al. 2025 Song et al. 2025). central challenge is balancing two competing needs: the model must continu- ously update semantics new frames arrive, but computational efficiency and latency are critical.
Existing VLMs typically rely on explicit memory mechanisms (Zhou et al. [024]] Qian et al. 2024)
to decide when to decode complex KV-cache optimizations (Di et al.) 2025) for efficiency, since autoregressive language models are expensive to run continuously. VL-JEPA, in contrast, natively supports selective decoding. Since it predicts semantic answer embedding aon-auorrgrrssivelyy the model provides continuous semantic stream of Sy that can be monitored in real time. This stream can be stabilized with simple smoothing (e.g., average pooling) and decoded only when significant semantic shift is detected, such as when the local window variance exceeds threshold. In this way, VL-JEPA maintains always-on semantic monitoring while avoiding unnecessary decoding, achieving both responsiveness and efficiency.
3 IMPLEMENTATION OF VL-JEPA
3.1 MODEL. ARCHITECTURE
X-Encoder. Unless otherwise specified, we use a frozen V-JEPA 2 ViT-L (Assran et al. 2025) with 304M parameters, self-supervised vision model that excels at both image and video tasks. Each video input is uniformly sampled into frames at 2562 resolution. For image inputs, the same image is duplicated match the input shape.
Predictor. The predictor is initialized with the last 8 Transformer layers of L1ama-3, 3.2-1B, resulting in 490M trainable parameters. The text tokenizer and token embedding also from
4

---

Published as a conference paper at ICLR 2026
Llama-3. 2-11B.. We allow maximum 512 query tokens, and put [PAD] tokens for short queries. We disable the causal attention mask so that both vision and query embeddings can be jointly at- tended. Linear projections connect the predictor with the vision and text embeddings, and average pooling on non- [PAD] tokens is applied to obtain the predicted target embedding.
Y-Encoder. We use Embeddi EmbeddingGemmar inggemaaaaammmmm 30000 (Vera et al. 2025) as the initialization of the Y-Encoder. We set maximum context length of 512 to handle detailed captions. We found that setting a learning rate multíplier of x0.05 all text encoder parameters improves performance, since the quality of embedding prediction would be suboptimal in the beginning of training. Linear projection head is applied both Predictor and Y-Encoder, obtaining a shared embedding space with 1,536 dimensions, where the loss is calculated.
3.2 TWO-STAGE TRAINING
Large-scale Pretraining. VL-JEPA is trained with two stages. The first query-free pretraining stage aims to establish robust vision-language alignment using massive caption data. We use Datacomp (Gadre et al. 2023) and YFCC-100M (Thomee et al. (016) for image-text data. For video-text data, use ACTION 1OOM (Chen al., 2026), which consists action description and video captions generated on HowTo To100M videos (Chen et al. 2023b).
We first do image-only training Datacomp and YFCC-100M with only frame per visual input, which allows us to use a large batch size of 24k. After 100k iterations, the model has seen 2B samples and achieved 61.6% ImageNet zero-shot accuracy (without prompt ensembling). Then, we continue with video pretraining with 8 frames per input for 60k iterations and 32 frames for 10k iterations the end. The pretraining takes 4 weeks using 24 nodes with XXNVIDIA H200 GPUs each. We adopt constant learning rate of 5x 1xx00= to facilitate extended training. We refer the readers to the Action 100M paper (Chen et al.) 2026) for more details. We call the resulting model VL-JEPABsSE and measure zero-shot classification and retrieval performance with this model.
Supervised Finetuning. The second qunry-conditioned supervised finetuning (SFT) stage empow- ers VL-JEPA VQA capabilities while maintaining the pretrained vision-language alignment for clas- sification and retrieval. The training data is selected from the PLM data mixture (Cho et al. 2025), including 25M VQA samples, 2.8M captioning samples, 1.8M classification samples, and down- sampled pretraining stage data to avoid catastrophic forgetting.
We train the model for 83k steps with batch size of 3,072 (~2.55 days with 24 nodes), with cosine learning rate annealing applied to improve convergence. Since excessive human labeled data is included in this SFT data mixture, we longer emphasize zero-shot evaluation for the resulting VL-JEPAsPT from this stage. Instead, we evaluate VQA capabilities and compare it with state-of- the-art specialist models.
4 EXPERIMENTS
We begin by evaluating VL-JEPA's classification and retrieval performance in $4.1 and benchmark VL-JEPA on VQA datasets in $4.2 We demonstrate application of VL-JEPA for understanding the relationship between world state changes and action concepts (i.e., inverse dynamics) in $4.3 In
$4.4 we demonstrate the advantage of embedding prediction by comparing with a loken-predictte VLM baseline under a strictly controlled setting. In $4.5) we evaluate the effectiveness of VL-JEPA's selective decoding, and show that it reduces decoding cost while maintaining the performance. Next,
we analyze VI-JEPPA' Y -Encoderr r in $4.6 Finally, we present ablation studies in $4.7
4.1 CLASSIFICATION AND RETRIEVAL
Evaluation Setup. We evaluate VL-JEPA following the CLIP-style evaluation protocol (see Fig 2 and $2 "Multi-tasking"). We assess VL-JEPA on broad suite of benchmarks, including 8 classi- fication datasets and 8 retrieval datasets. For zero-shot evaluation, we compare against generalist foundation models CLIP (Radford et al. 2021), SigLIP2 (Tschannen al.) (2025,, and Perception Encoder TEE-Cor)(Bolyy et al. (025). We additionally report reference numbers from specialist models that are individually optimized for each benchmark.
5

---

Published as a conference paper at ICLR 2026
Table 1: Video classification and text-to-video retrieval. Best zero-shot performance in each dataset are highlighted. Samples seen = training step x effective batch size.
Viduo Chacilration (lip-l Accuraay)
Trat-lontide Retrieral (Rrzal)e)
Genrrali Mindel
Purammerr
Zero-shet
Avaragge
76M
3.1
124M
12.88
25.4
3.1
2.8
09.5
11.2
47.3
71.5
29.3
31.0
19.5
25.7
34.0
6.1
21.0
43.5
42.9
WIF.
36MM
12.88
90.7
3.8
3.7
7.6
19..
14.7
63.5
701.8
78..5
15.5
35.
25.4
10.7
41.9
7V
36.7
56.8
69.3
53.1
SILUIP
WIRL
852M
4189
38A
4.5
6.4
63300
24.2
71..5
35.1
50.
15.0
41.6
32.7
35.1
53.5
19.0
59.2
71.6
50.9
ViT.
1,98
406
39.8
6.1
6.1
5.6
66.00
26,0
90.4
35.1
90.8
47.5
43.4
33.9
38.9
56.0
22.2
66.4
13,0
52.5
WHH.
671M
158
43.
9.3
6.0
11.6
7344
27.1
K33
52.5
95.5
90.2
68.9
41.7
40.8
56.3
21.5
64.7
75.9
51.0
VITG
2.38
118
44.
9.0
6.4
13.6
76.4
20.0
56,0
40.3
97.2
58.1
51.6
49.1
44.5
58.7
25,0
77.0
89.2
68.5
1.48
53.2
83.1
96.2
46.2
Pariies SUTM (Sacluding swwcialtn wwwees
22.
56.4
478
00.7
07.5
95.1
64.
96.0
62.8
76.1
74.2
01.4
30.9
770
Table 2: VQA benchmarks. We report accuracy on GQA (Hudson & Manning 2019), TallyQA
(Acharya al. 2019) POPE (Li al. 2023b), and POPEv2 (Li et al. (025b). Scores lower than our model are marked in red. Scores from SmoIVLM are obtained by our evaluation, while other baselines are reported in the literature.
601 compositional vicual rassonting
TaltyA4 compler object countiing
POPE: odject Ballicmaato
POPE12: object kulfucination
Model
Accuracy
Medel
Accuracy
BLIP 2 2.70
33.9
SmolVLM 256M
32,5
Sma1YMM2 250M
26.4
SmolVLM 256M
62.3
(Faat15X0.)
Sme M-256M
57.9
5-133
InsrrcDLLP (TlaTTSXL)
48.4
PuLl 700M
62.5
LLIVA 7B
72.9
Iee 8B
74,5
Insrschii
SmalVi
(Viam--133))
InmV
QwanVI-CCle-78
57.5
P.L1-3
65.8
Vilas-La (7B)
53.
Qwen6-V1-728
70.4
VL-7B
59.3
IsttuBLLI (Vicuss-3BB)
SmdIVLM-5000
55.
SmatVLM 500M
IeenVL-Char (Vioms-78))
59.5
Pu
71.9
LL 5-78
$5.9
Qwent-V1-78
87.0
LLAVA-15 (Vicuna 7B)
LLAVA-15 (Vicuno-138)
LLLNA-L.--1B-BE
56.3
SmulVLM-22
IntenYL Chat (Vicuna 13B)
06.6
PaliGemma (5B)
76.8
SmdVLM 26
57.5
QwenZ VL 2B
91.3
YL-JEPAgr
VL-JERAgr: (1,6B)
69.9
VL.JEPPArr (1.68)
55.7
VL-JEPAgr
86.3
Results. Table [ summarizes the results. In the strict zero-shot setting, VL-JEPAsasz achieves higher average accuracy (52.5 44.7) across the 8 classification datasets and higher average call@1 1 (63.7 vs 58.1) across the retrieval datasets than the best baseline PE-Core-G. Per-dataset scores show that VL-JEPApasE is particularly strong on motion-centric benchmarks (SSv2, EK- 100, EgoExo4D, and step recognition on COIN and CrossTaskk, while relatively weaker on appearance- centric benchmarks (Kinetics-400 and task recognition
COIN and CrossTassk.. This is due to VL-JEPA has seen substantially fewer vision-language pairs (only 3.6B in comparison with PE- After supervised finetuning, VL-JEPAseT improves significantly upon VL-JEPA since the model has seen in-domain training data. As single generalist model, the performance of VL-JEPA is approaching specialist models optimized individually for each dataset.
Core-G's After
4.2 VISUAL QUESTION ANSWERING
Evaluation Setup. We evaluate VL-JEPAsPT on discriminative VQA tasks. The inference process involves encode candidate answers using the Er.coerr and selecting the answer that minimizes the distance to the predicted embedding (see Fig. 2). We select four benchmarks that prioritize visual perception rather than knowledge and reasoning. We evaluate on GQA (Hudson & Manning 2019), dataset for real-world visual reasoning and compositional QA, reporting accuracy on the testdev- balanced split. For TallyQA (Acharya et al.) 2019), which targets complex counting, we follow Chen al. (2022) and report the weighted average accuracy across the "simple" and "complex" splits. Finally, to assess object hallucination, we utilize POPE (Li et al. (2023b) and POPEv2 (Li al. 2025b). For POPE, we report the average accuracy across the "random", "popular", and "adversarial" settings on MS-COCO.
Results. Table 2 compares VL-JEPAspT against established VLM families, including BLIP-2 (Li ct al. 2023a), InstructBLIP (Dai et al. 2023), Qwen-VL (Bai et al., 2023), InternVL (Chen et al. 2024d), Llava-1.5 (Vallaeys al. 2024), SmoLVLM (Marafioti al. 2025), Pal.I (Chen et al.

---

Published as a conference paper at ICLR 2026
Table WORLDPREDICTION- WM benchmark results. We compare the accuracy between large VLMs, socratic LLMs, and VL-JEPA. VL-JEPAsrT achieves new SoTA at 65.7%.
Vision Language Models
Socratic LLMs (wi Ques2.5. S-L-722 captions)
VI-JKPA
IntereVi2.5
Qwen2 5-VI
Siana-3.1 .1
Liana-1
0xen22.5
CPI-Co
Cloude-3.5
Conini-2
BASB:
SF7
100m
1.60
20.0
29.5
45.5
52.7
2022).
PaliGemma (Beyer al. 2024), and Video-LLaVA (Lin et al. 2024). VL-JEPASFT outper- Torms many of these baselines despite requiring significantly less computational resources-classice VLMs rely on extensively pretrained CLIP backbones combined with multi- -tage visual instruction tuning. In comparison, VL-JEPAgET employs unified architecture and single embedding space to seamlessly handle VQA, classification, and retrieval (Tab. D
4.3 WORLDPREDICTION WM
Evaluation Setup. We evaluate VL-JEPA on the "world modeling" task in the WORLDPREDIC- TION (Chen al., 2025a) benchmark, where the model is provided with two images representing the initial and final world states and must identify, among four candidate video clips, the action that explains the observed transition. To adapt VL-JEPA, we duplicate and concatenate the initial and final state images to extract a state embedding, and encode each action candidate into action embed- dings. The model then selects the candidate whose embedding is closest to state embedding.
Results. Table shows accuracy comparisons. VL-JEPA attains 63.9 % and VL-JEPASeT at- tains 65.7% % top-1 accuracy on WORLDPREDICTION WM, establishing a new state of the art. Our VL-JEPA model not only substantially surpasses existing VLMs of comparable larger scale but also exceeds the performance of frontier LLMs such as GPT-4o, Claude-3.5-sonnc. and Gemini-2..
4.4 EMBEDDING PREDICTION VS. TOKEN PREDICTION: A CONTROLLED COMPARISON
Evaluation Setup. In this section, we compare VL-JEPA to token--eneratie generative VLM baseline under strictly aligned training conditions. Both models use the same Perception Encoder (Bolya et al. 2025) (frozen ViT-L-14 with 3362 resolution, no tiling. 16 frames per video) for vision inputs. We use the same training iterations with the same effective batch size of 128, same learning rate scheduler on the same pretraining data mixture described above ($3) The only difference is the prediction task: VL-JEPA predicts target embeddings (Duquenne et al. 2023) using a 0.5B predictor, whereas the VLM baseline performs next-token prediction with cross-entropy using IB LLM. For VLM, we use the standard training recipe and codebase of PerceptionL.M. (Cho et al. 2025), aligning frozen vision encoder and text-only LLM Llama-3. 2-BB. For VL-JEPA, we initialize the predictor from the 8-16 layers of Ll ama - 3. 3-2-BB.
We evaluate both models regular checkpoints throughout training spanning from 500K to 15M samples seen. At each checkpoint, we measure the performance on video captioning and video clas- sification. For video captioning, we report CIDEr scores averaged across YouCook2 (Zhou et al.] 2018), MSR-VTT (Xu et al. 2016) and PVD-Bench (Bolya et al. 2025). VL-JEPA decodes the pre- dicted embeddings while VLM generates the tokens directly. For video classification, we report top- 5 accuracy averaged across CrossTask-Ster
CrossTaske Task (Zhukov et al. 2019) and EgoExo4D (Grauman al.) 2024). For VL-JEPA we choose the candidate with lowest cosine distance the predicted embedding, while for VLM we pick the class with lowest perplexity.
Results. shown Fig. both models yield comparable performance after 500K samples seen in both tasks, with respectively 1.23 and 1.35 CIDEr in video captioning and 14.9% and 14.0% top- accuracy for VL-JEPA and VLM. After a few iterations, we show that VL-JEPA's performance increase is much sharper compared to VLM, reaching 14.7 CIDEr and 35.3% top-5 accuracy after 5M samples seen. This gap remains constant as training scales at 15M samples with 14.8 CIDEr and 41.0% top-5 accuracy for VL-JEPA, while the VLM baseline yield respectively 7.1 CIDEr and 27.2% top-5 accuracy. This controlled comparison highlights the benefit of predicting embeddings rather than tokens, showing both higher sample efficiency and stronger absolute performance.
7

---

Published as a conference paper at ICLR 2026
Captioning
Classfication
Parameters
Average
0.58 Predictor
203ma
000
Acumcy 2
Uaree 22:10
16 LLM
Freeder Prodett
125ma
Sumples
VL-JEPA
VLM VL-JEPA
Figure 3: Comparison of embedding prediction (VI-JEPA) and token prediction (VLM). We conduct a fair comparison of under strictly aligned training settings (encoder, data, batchsize, etc.). Left: Zero-shot video captioning CIDEr score averaged over datasets and zero-shot classification accuracy (top-5) averaged over 3 benchmarks. Right: Comparing the trainable parameters and average inference time cost.
Annotations
9:
93
9+
CIDER
Selective Decoding
D
D
D
D
Symp pwwporrionaece decoding
Sceete Devoring be por for Dect
Embedding Stream
Input Video Stream
VL-JEPA Encoder Predictor
Average Decoding Interval (seconds)
Figure 4: Evaluation of selective decoding. Left: We compare uniform sampling of decoding points at fixed intervals (red) and embedding-guiee measured by the average CIDEr score between each annotation and its closest decoded output ÿ. Right: Results on EgoExo4D show that selective decoding achieves a Pareto improvement over uniform sampling: for the same performance level, it requires fewer decoding operations.
selective decoding (blue). Performance
We compare the inference cost of the above VL-JEPA and the VLM by pre-loading 64 video frames into memory and repeatedly decoding text 100 times with the same prompt, measuring the average time per sample. As shown in Fig. (right most), both models exhibit comparable latency when generating text. What differentiates our model from classical VLM is the decoupling between the prompt processing ("Query Embedding") and the video encoder ("Encoder + Predictor") from the text generation module ("Decoder"). This allows us only use the first part of the model to perform retrieval and decode text only when needed (see Section beloww, making our model more scalable for online video inference.
4.5 EFFECTIVENESS OF SELECTIVE DECODING
Evaluation Setup. We evaluate the effectiveness of VL-JEPA's embedding-guidd selective de- coding on long-form video streams. To this end, design a benchmark task where the goal is to recover a temporal sequence of annotations while minimizing the number of text decoding op- erations, which dominate inference cost. As shown in Fig. 4 (left), decoding is performed only at selected points along the VL.JEPA embedding stream, yielding a sequence of N decoded out- puts [(ti, 91), (t2, .,), (Ex, )N)]. Each ground-trute annotation [(t1, 1/1,, (t2, M2)...., (tr, yr)) is then aligned to its nearest decoded output in time (illustrated as o -- in Fig. 4 and CIDEr is computed between matched pairs. We use the EgoEXo4 (Grauman et al., 2024) validation set in procedural activity domains, which consists of 218 videos with an average duration of 6 minutes and about T 143 atomic action annotations per video.
As a baseline, we consider uniform sampling, where decoding points are placed at fixed intervals regardless of the underlying video content. Standard streaming VLMs are limited to this strat- egy, whereas VL-JEPA supports more effective alternative: adaptive selection of decoding points guided by its predicted embeddings. We apply agglomerative clustering with temporal connectiv- ity constraints (Murtagh & Contreras) 2012) to partition the embedding sequence into N segments of high intra-segment monosemanticity (Chen et al. 2024a), measured by variance (i.e. Ward dis- tance). The intuition is that within semantically coherent segment, decoded outputs are highly similar, so decoding once per segment captures the essential information while greatly reducing overall decoding cost. The midpoint of each segment is then chosen the decoding point, and de-
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of text-encoders performance. We report triplet-based accuracy (%) on
SugarCrepe++ and VISLA datasets.
SugnrCrepes+
Dampala et
2024g
VISLA
Dumpala et
2024h
Model
Purams. (lotal)
é Params, (text encoder)
Replace Attribute
Replize Object
Kesttee Relation
SWig
Swap Object
Average
Amtiibue
Averuge
Genetic
Spatial
CLIP
VIT-
389M
85M
44.5
56.7
K3.0
42.5
27.0
13.5
34,5
37.6
31.3
Sigt
WT-g
66.9
52.1
30.6
40.4
48.7
PE-Core
31.4
VL-JEPSaaa
VITL
166
300M
63.9
22.2
90.1
52.2
62.9
42.0
42.9
49.8
35.9
VL-JEPAere
VTL
1.68
300M
58.4
68.5
90.9
474
55.4
29.8
39.5
44.8
34.2
coding is performed either from the exact embedding or from the average-pooled embedding within the segment.
Results. As shown in Fig- (right), we sweep the average decoding frequency from 2.0 down to 0.01 Hz (i.e., average intervals between consecutive decoding operations from 0.5s to 100s) by ad- justing either the stride of uniform sampling or the number clusters adaptive selection. Across the entire range, adaptive selection consistently Pareto-dominates uniform sampling. In particular, selective decoding at 0.35 Hz (i.e. ~2.85s interval) matches the performance of uniform decoding Hz, reducing decoding cost by ~2.55 We further observe that average pooling provides con- sistent gains for both strategies, since provides denoising and stabilization on embeddings prior feeding into the decoder.
4.6 EVALUATION OF Y-ENCODER
Evaluation Setup. We evaluate whether the JEPA architecture improves the Y-Encoder by fol- lowing the uni-modal text-only (TOT) evaluation setup. We use the hard-negative benchmarks SugarCrepe++ (Dumpala et al. 2024a) and VISLA (Dumpala et al. 2024b). These datasets test sensitivity to semantic and lexical changes in image descriptions. Each dataset contains triplets: two semantically similar descriptions of the same image (pl and p2), and one negative description (ni created by altering attributes, relations, or objects. We compare Y -Encoderr from different models by computing the cosine similarity for all description pairs. We check that the similarity be- tween positives spm, p p2) is higher than both the similarity between each positive and the negative sim sim(pl, n)) and sim(p2, nt). We report accuracy (%) across all samples.
the performance of different models on text hard-negative benchmarks. VL- JEPA achieves a micro average accuracy of 63.9% on SugarCrepe++ and 42.9% VISLA. This is higher than the best other models: PE-Core scores 58.6% on SugarCrepe++ and SigLIP2 scores 40.4% on VISLA. The finetuned VL-JEPAgrt model also achieves competitive results, with 58.4% on SugarCrepe*+ and 39.5% on VISLA. These results indicate that VL-JEPARASe has a Y-Encoder that is resilient to text hard-negatives.
Results.
4.7 ABLATION STUDY
Evaluation Setup. We study different design choices for VL-JEPA. Here we train all ablation models on the SFT stage data for 10K steps with batch size of 512 (5M samples seen) and constant learning rate. We report average classification top-1 accuracy of 8 datasets (Tab. [], average text-to- video retrieval recall I of 8 datasets (Tab. D]. and average VQA accuracy of 4 datasets (CLEVR, GQA, TallyQA simple and complex). We report the results in Tab. S
Results. (a) Pretraining. Dropping the first query-free pretraining stage on image and video cap- tions significantly hurt performance, especially on classification (-21.7) and retrieval (-17.3). (b) LR Multiplier. The sweet point of learning rate multiplier to the Y-Encoder around 0.05 to 0. Either faster or slower learning degrades the performance. (c) Loss Function. InfoNCE generally give superior performance compared to cosine, LI, and L2 losses, with the only exception being cosine loss outperform InfoNCE on VQA. However, only InfoNCE has the anti-collapse regular- ization and can be applied with unfrozen Y-Encoder. (d) Predictor. In terms of predictor size, more layers yield better performance, especially VQA performance. We also that if using the original causal attention instead of updating to bi-direction attention hurt VQA performance (-1.9), since query tokens are appended after visual tokens, and visual tokens longer able attend to

---

Published as a conference paper at ICLR 2026
Table 5: Ablation studies results. The default setting adopted by VL-JEPA is marked in blue We calculate dellta within each group of ablations in comparison with the default setting.
Classification
Retrieval
VQA
Classification (Accurccy) inttiolization
Retrieval (Recalló
VQA
(Aceurcc)
(Rocalle 1)
(Accurcc))
(Accuracy)
VL-JEPPArr 75.4
63.8
74.2
Predictor architecture
(0) Effectveenss pretruining stoge ow cuption
Layer 8-16
27.3
30.2
42.5
w/ Pretruiming
49.0
47.5
46.1
Layer 0-2
24.3
27.8
40.1
w/o Pretruining
27.3
30,2
6179
42.5
Layer 0-4
25.1
28.9
43.6
Layer 0-8
27.2
(4.1)
29.3
(40)
43.4
(9..)
(b) Leaming rate multipller for V. Encoder
Layer O-16
27.4
(60.0)
31.0
(00.)
45.5
(+1.)
mulrpplirr= = 0.05
27.3
30.2
42.5
w/o Bi-direction Attention
26.7
31.2
(+..)
40.6
619)
multiplier= = 1.00
23.7
13.0
28.8
(1.4)
40.7
(4.8)
w/o Llama-3 Initialization
28.
(908)
30.4
(00.)
40.6
(41)
mulriplier 0.10
26.9
(04)
30.2
(02)
42,9
(994)
Y-Encoder (trainable Inner projection top frogen encoder)
multiplier= = 0.01
25.6
64.7
27.7
(2.5
41.0
(4.5)
Embeddinggeemmaa000
19.5
24.1
42.5
multiplier- - 0.00
20.0
(3.
25.9
(4.)
41.4
64.3)
Owen3 Embeedingg-..k 0.60
24.5
(4300
24.5
(+0.)
41.5
(440)
(c) (witt no projection top froten encader)
Bwend-Banokllnng-4
27.7
(+b.)
26.6
(+2.5)
38.1
(4.4)
InfoNCE
23.3
Been--Emmedding--o SB
29.6
(+101)
29.5
(+5.)
41.9
(00)
Cosine
16.5
(63)
(-111)
(+2.3)
PR (356M)
29.4
(49%)
(+101)
35.9
(69
L1 14.8
614B
PB (356M)
(423)
(+003)
42.9
(904)
L2 135 (48)
(188
PB
(47)
query tokens. Finally, we also see that LLama-3 initialization is beneficial to VQA performance, al- though vision-language alignment (classification and retrieval) is bit worse compared to randomly initialized Transformer layers. (e) Y-Encoder. We tried different text encoder as the Y-Encoder, and confirmed that VL-JEPA works well with other embedding models than EmbeddingGemmaa- 300M. Generally, larger encoder leads to better performance, with visually aligned text encoders (PE models) has sigmificant advantage classification and retrieval.
CONCLUSION
We have present VL-JEPA, a new vision- vision-language model built upon the joint embedding predictive architecture. By shifting supervision from discrete token space to continuous semantic embedding space, VL-JEPA simplifies the learning target, avoids redundant modeling of surface linguistic vari- ability, and enables non-autoregressive prediction. Through controlled experiments, we show that VL-JEPA outperforms generative VLMs trained with cross-entropy loss under matched training data budget, while achieving superior training efficiency and significantly lower inference latency. Beyond generation tasks, the embedding- vocabulary classification and cross-modal retrieval within single unified architecture. Its ability to emit continuous semantic embeddings also makes it particularly well suited for real-time video applications, where selective decoding can improve both responsiveness and efficiency.
design further allows VI.JJEPA to handle open-
LIMITATIONS
In this work, we demonstrated the advantages of VL-JEPA over standard VLMs, particularly in efficiency, streaming, and video-language tasks. Our goal at this stage, is not to propose a universal alternative to VLMs, this would require broader evaluation and agentic behaviors where current token generative VLMs excel. Finally, although our results show clear benefits from scaling parameters and dataset size, we did not fully explore this direction, leaving it for future work.
tasks such as reasoning, tool use,
LLM USAGE
We used large language models (LLMs) solely as writing assistants for this paper. Specifically, they were employed help rephrase sentences for clarity and readability. No content, ideas, or experimental results were generated by LLMs. The authors take full responsibility for the scientific contributions and all written content.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8076- 8084, 2019.
Jean-Baptia BapttBBptttteee Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Bart, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803-5812 2017.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Am- mar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint wrxv:25506009955, 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprini arXiv:2309. 16609, 2023.
Federico Baldassarre, Marc Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann Le- Cun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: Dino as a foundation for video world models. arXiv preprint arXhv:2507.19465 2025.
Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning with- out the heuristics. arXiv preprint arXiv:2511.08544 2025.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-invarianc--ovvvaann waw.araaraanneovviiiionnnrrrrraa for self-supervised learning. arXiv preprint TrXiv:2105.04906 2021.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. 2023.
Loic Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, et al. Large concept models: Language modeling in sentence representation space. arXiv preprint arXiv:2412.088211 2024.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprini 20X1::2407.07726, 2024.
Kevin Black, Manuel Y Galliker, and Sergey Levine. Real-time execution of action chunking flow policies, arXiv preprint arXiv:2506.077 07339, 2025.
Daniel Bolya, Po-Yao Huang, Peize Sun. Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. /3181, 2025.
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay. Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al. An introduction to vision-language modeling. arXiv preprint 2rX5v:2405.17241 2024.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.

---

Published as a conference paper at ICLR 2026
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Hajishirzi. Pumer: Pruning and merging tokens for efficient vision language models. arXiv preprint 3.XV::2305.153300 2023.
Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and new benchmark. arXiv preprint arXiv:2410.03051, 2024.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pp. 100-200, 2011.
Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Shaoqiu Zheng, Ying Tan, and Erjin Zhou. Proto- clip: Prototypical contrastive language image pretraining. IEEE Transactions Neural Networks and Learning Systems, 2023.
Delong Chen, Samuel Cahyawijaya, Jianfeng Liu, Baoyuan Wang. and Pascale Fung. Subobject- level image tokenization. arXiv preprint arXiv:2402. 14327, 2024a.
Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual instruction tuning with polite flamingo. In Proceedings of the aaai conference on artificial intelligence, volume 38, pp. 17745- 17753, 2024b.
Delong Chen, Willy Chung, Yejin Bang, Ziwei Ji, and Pascale Fung. Worldprediction: A bench- mark for high-level world modeling and long-horizon procedural planning. arXiv preprint arXiv:2506.0m 043366,,,, 2025a.
Delong Chen, Theo Moutakanni, Willy Chung, Yejin Bang, Ziwei Ji, Allen Bolourchi, and Pas- cale Fung. Planning with reasoning using vision language world model. arXiv preprint arXiv:2509.02722, 2025b.
Delong Chen, Tejaswi Kasarla, Yejin Bang, Mustafa Shukor, Willy Chung, Jade Yu, Allen Bolourchi, Theo Moutakanni, and Pascale Fung. Action100m: A large-scale video action dataset. arXiv preprint arXiv:2601. 10592, 2026.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song. Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large lan- guage model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18407-18418, 2024c.
Xi Chen, Xiao Wang, Soravit Changpinyo, Anthony Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0611 067944 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 24185-24199 2024d.
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open- access data and models for detailed visual understanding. arXiv preprint 2rXiv:2504.13188. 2025.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning. Advances in neural information processing systems, 01.49250-49266 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- trie vision: Collection, pipeline and challenges for epic-kitchens- 100. International Journal of Computer Vision 130(1 (1131)333555.. 2022.
12

---

Published as a conference paper at ICLR 2026
Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-ana answering with in-context video kv- cache retrieval. arXiv preprint arXiv:2503.00540, 2025.
Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Has- san Sajjad. Sugarcrepe++ dataset: vision-language model sensitivity to semantic and lexical al- terations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024a. Curran Associates Inc. ISBN 9798331314385,
Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. Visla benchmark: Evaluating embedding sensitivity semantic and lexical alter- ations. arXiv preprint 24.iv:2404.163665 2024b.
Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: sentence-level multimodal and language-agnostic representations. arXiv preprint 17Xi::3388.11666 2023.
Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A-jepa: Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311. 15830, 2023.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregres- sive pre-training of large vision encoders. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9641-9645555 2025.
Pascale Fung. Yoram Bachrach, Asli Celikyilmaz, Kamalika Chaudhuri, Delong Chen, Willy Chung, Emmanuel Dupoux, Hongyu Gong, Hervé Jégou, Alessandro Lazaric, et al. Embodied ai agents: Modeling the world. arXiv preprint arXiv: 22355, 2025.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092-22 222112, 2023.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phat, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904 4916. PMLR, 2021.
Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, al. Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24905-24916, 2025.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola. Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 777700000900 2017.
13

---

Published as a conference paper at ICLR 2026
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. International Conference on Machine Learning, Pp. 17283- 17300. PMLR, 2023.
Yann LeCun. A path towards autonomous machine intelligence. Open Review, 62(1):1-62, 2022
Hongyang Lei, Xiaolong Cheng, Qi Qin, Dan Wang, Huazhen Huang, Qingqing Gu, Yetao Wu, and Luo Ji. M3-jepa: Multimodal alignment via multi-gate moe based on the joint-embedding predictive architecture. In Forty-second International Conference on Machine Learning, 2025.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning space: Multimodal visualization-of- -f-thhughtt arXiv preprint arXiv:2501.0754. 2025a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- with frozen image encoders and large language models. In International conference on machine learning, pp. 19742.19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Yifan Li, Kun Zhou, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. Analyzing and mitigating object hallucination: training bias perspective. arXiv preprint arXiv:2508.00 04567, 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pp. 5971-5984, 2024.
Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing. 1EEE Transactions on Geoscience and Remote Sensing, 62:1-16, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Za- kka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvim: Redefining small and efficient multimodal models. arXiv preprint wxXiv:2504.05299 2025.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXiv:2209.1510 2022.
Fionn Murtagh and Pedro Contreras. Algorithmss for hierarchical clustering: an overview. Wiley interdisciplinary reviews: data mining and knowledge discovery, 2(1):86-97) 2012.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang- Streaming long video understanding with large language models. Advances Neural Information Processing Systems, 19:119336-11 1119933366116600.111111 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763. PmLR, 2021.
Mustafa Shukor and Mathieu Cord. Skipping computations in multimodal lms.. arXiv preprint arXiv:2410.09454, 2024.
Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual augmentation of language models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 22056-22063 2023.
Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine. Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision-languae language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.0018 01444 2025.
14

---

Published as a conference paper at ICLR 2026
Wenxuan Song, Jiayi Chen, Pengxiang Ding, Han Zhao, Wei Zhao, Zhide Zhong, Zongyuan Ge. Jun Ma, and Haoang Li. Accelerating laan--laaugggattiinn model integrated with action chunking via parallel decoding, arXiv preprint arXhv:2503..23110 2025.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec 100m: The new data multimedia research. Communications of the ACM, 59(2):44-3,, 2016.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.1 1.X7::2502.14786, 2025.
Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of llms. In European Conference on Computer Vision, pp. 369-387. Springer, 2024.
Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokula Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, al. Fastvlm: Efficient vision encoding for vision language models. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 19769-19780, 2025.
Henrique Schechter Vera, Sahil Dua, Biao Zhang, Daniel Salz, Ryan Mullins, Sindhu Raghuram Panyam, Sara Smoot, Iftekhar Naim, Joe Zou, Feiyang Chen, et al. Embeddinggemmaa Powerful and lightweight text representations. arXiv preprint urXiv:2509.2035:: 2025.
Jiawei Wang. Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634 2024.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp- 9929-9933, PMLR, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 -42327, 2022.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang. Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone, arXiv preprint arXiv:2408.00 01000,0000000000 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image preiinnin In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre- trained visual features enable zero-shot planning. In Forty- second International Conference on Machine Learming, 2025.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
15

---

Published as a conference paper at ICLR 2026
Xingyi Zhou, Anurag Arnb, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Na- grani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18243-18252, 2024.
Dimitri Zhukov, Jean-Baaaaaaeeaeeee Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3537-3545, 2019.
16

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
JEPA Models. JEPA model learns by predicting the representation of a target input Y from the representation of context input X. Early instantiations include I-JEPA for image encoding (Assran et al. 2023) and V-JEPA for video encoding (Bardes et al. 2023), which demonstrated the effec- tiveness of this objective over pixel reconstruction approach in their respective modality. Recent JEPA work falls into two categories. One category of work emphasizes better unimodal represen-
tation learning (Assran et al. 2023 Bardes al.] 2023: Fei et al. 2023) or cross-modal alignment
(Lei et al. 2025 Jose et al. 2025). The other direction targets world modeling, where pretrained
encoders are frozen and cettooncooniiioneel predictors are trained for conditional prediction of state
representations (Zhou et al. 2025 Baldassarre et al. 2025 Assran et al. 2025). This has shown good
results but remains limited to narrow domains like mazes or robotic pick-and-place Our proposed VL-JEPA is the first designed for general-purpon vision-languaee tasks. It performs conditional latent prediction over vision and text, and preserves efficiency while enabling flexible, multitask architecture.
Vision Language Models. Existing vision-language models largely fall into two families: (1) CLIP- style models with non-predictive joint-embedding architecture (JEA) (Radford et al. 2021 Zhai itt al. [023] Bolya et al. 2025 Liu et al. [2224] Chen et al., (023) encode images and texts indepen- dently into a common latent space, Xv + Sy and Y Sy, By minimizing CCLIP = (Sv, Sy)
with a contrastive loss (e.g., InfoNCE), CLIP learns aligned representations that support zero-shot
classification and vision- Languag retrieval; (2) Generative VLMs (Liu et al. 2023 Chen et al., 2022 Dai et al., 2023) Alayrac et al., [202]] Chen et al. [024bb Cho et al., [025 Beyer et al. 2024 connect a vision encoder (Radford et al. 202 Fini et al. 2025) with a language model (e.g., LLM).
They are typically trained with LVIN = D(Y, Y). i.e. next token prediction with cross-entropy loss, and can learn to handle various vision-text-to-text generation tasks such as visual question answering (VQA).
Our proposed VL-JEPA integrates the architectural advan- tages and task coverage of both CLIPs and VLMs (Tablelo).
CLIP
VLM
VL-JEPA
Since VL-JEPA learns in embedding space, it can leverage web-scale noisy image-text pairs (Jia et al. 2021), ) yield- ing strong open-domain features. On the other hand, VL-- JEPA supports conditional generation tasks with a readout text decoder. Meanwhile, compared to generative VLMs that optimize directly in data space, VL-JEPA is more efficient at learning in the latent space. In addition, is also more efficient for online inference, as it allows naturally selective decoding.
Generation
Retrieval
X
Table 6: Task coverage comparison.
Efficient Vision Language Models. The growing size and training cost of VLMs has motivated efforts to improve efficiency. On the training side, strong performance can be achieved by updat- ing only a subset of parameters, such as the vision-language connector (T'simpoueell et al. 2021
Alayrac et al. 2022 Vallaeys et al., 2024 Shukor et al. 2023 Koh et al.] Merulo
Merullo et al. 2022
Dai et al. (002)) At inference, efficiency is pursued through pruning parameters or visual tokens
(Cao et al.) 2023, Shukor & Cord. 2024 Vasu et al. 2025). For real-time use cases, recent work ex-
plores small VLMs (Yao et al. [2024] |024| Marafioti et al. 2025 and heuristics to reduce query frequency
in asynchronous inference (Shukor et al.) 2025).
Latent-space Language Modeling. Current state-of-the-an LLMs are trained to decode and rea- son in text space using autoregressive generation and thain-of-thoughh prompting (Wei et al. 2022). Text-space L.MMs have rapidly improved and now achieve strong results on wide range of bench- marks. However, the discrete nature of their reasoning trace may limit both speed and performance in the long term. Several works have explored latent-space LLMs that process or reason in latent space, such as Large Concept Models (Barrault et al. 2024 and COCONUT (Hao et al.) 2024). These models focus on unimodal sstce reasoning. With VL-JEPA, our goal is to align vision and text representations in shared multi-modal latent space. This approach aims to enable better abstractions and improve both the performance and speed of vision-language models (VL.Ms). We hope VI-JEPAA will serve as foundation for future work on multi-modal latent space reasoning, including visual chain-of ffain-of-thougee thought methods (Li ct al. 2025a).
17


Reviews:
[{"review_id": "5szVvU7gzg", "reviewer": "Reviewer_k4aJ", "summary": "This paper presents VL-JEPA, a vision-language model formulated on the Joint Embedding Predictive Architecture (JEPA). Instead of traditional autoregressive token-level generation, VL-JEPA learns to predict target text embeddings directly in continuous space, thereby abstracting away surface linguistic variability and focusing on semantic representation. The model demonstrates improvements in efficiency and sample complexity compared to classical token-generative VLMs, particularly in zero-shot video understanding, retrieval, and real-time streaming scenarios. Extensive experiments benchmark VL-JEPA against leading models, with ablation studies and scalability analyses provided.", "questions": "1.How does the L2 loss framework handle inherently multi-modal or ambiguous targets where multiple, semantically distinct ground truths exist? Does this not lead to regression towards a semantically blurry \"average\" embedding?\n\n2.Can you provide quantitative evidence (e.g., t-SNE, cluster variance) that the embedding space actually simplifies the target distribution (e.g., maps \"light went out\" and \"room is dark\" to nearby points) compared to a standard token space?\n\n3.How robust is VL-JEPA to the choice of the frozen y-encoder? What is the performance impact of replacing the SONAR encoder with a standard CLIP or Sentence-BERT encoder?\n\n4.Can you clarify the 2.85x saving calculation (is it 1Hz / 0.35Hz)? More critically, can you provide any statistical validation (e.g., mean and std. dev. over 3+ runs) for your key SOTA claims, or at least for the comparison in Fig 4?\n\n5.Loss Function: Why was L2 loss chosen over Cosine Similarity loss? Cosine loss would ignore magnitude and only focus on direction, which might be a more robust objective. Was this tested?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper's core contribution—applying a predictive JEPA-style objective to the cross-modal VL problem—is highly novel. Moving VL learning from the discrete token space to a continuous semantic space is a well-motivated and promising direction to address the known efficiency and latency bottlenecks of standard generative VLMs.\n\n2.The \"selective decoding\" mechanism (Sec 4.3) is a standout contribution. The idea of monitoring the latent embedding stream for semantic variance and only triggering the expensive text decoder when a significant shift is detected is an elegant and practical solution for real-world, low-latency video streaming applications. \n\n3.The model achieves state-of-the-art (SOTA) results across a wide and diverse range of video-language benchmarks, demonstrating the effectiveness and generalizability of the learned representations for both zero-shot and finetuned tasks.", "weaknesses": "1. While the paper emphasizes that the shift from token-space to embedding-space simplifies the target distribution, it provides no rigorous analysis of the measurability and discriminability of the resulting semantic embedding space.\n\nTo substantiate this claim, the authors should provide supplementary analysis, such as:\n\n(i) Visualization or quantitative studies on the embedding space's structure (e.g., its clustering properties, separability) to demonstrate this claimed simplification.\n\n(ii) A theoretical elucidation of the target distribution's compressibility, perhaps through the lens of the Information Bottleneck principle.\n\n(iii) A crucial ablation study comparing the performance impact of using different embedding spaces (e.g., from CLIP, SONAR, BERT-base) as the prediction target.\n\n2. L2 loss implicitly assumes a unimodal, deterministic target distribution.Real-world VL tasks are full of \"one-to-many\" ambiguities (e.g., \"the light went out\" vs. \"the room became dark\"). Both are valid but semantically distinct answers. The L2 loss will penalize the model for predicting either correct answer, forcing the predictor to regress towards a non-existent \"average\" embedding located somewhere between the two valid target points. This regression to the mean will likely result in semantically \"blurry,\" generic, or even nonsensical decoded outputs. The paper completely fails to address this fundamental limitation.\n\n3. Unfair and Misleading Efficiency Comparison: The core comparison in §4.2, which pits a 0.5B VL-JEPA predictor against a 1B VLM baseline, is fundamentally biased. The authors claim superior parameter efficiency, but their 0.5B predictor is not a neutral model; it is \"cherry-picked\" from the top-most, most semantically potent layers (L8-16) of the 1B Llama model. The paper's own ablation study (Table 5) confirms this bias, showing that these top layers (45.20% accuracy) are vastly superior to the bottom layers (35.86%). This is not an \"apples-to-apples\" comparison and does not prove the framework's efficiency, but rather the known fact that top-level LLM layers handle more complex semantics.\n\n4. Lack of Statistical Rigor: The paper suffers from a critical lack of statistical validation. All reported results—including all SOTA claims in the tables and the key efficiency curves in Figures 3 and 4—appear to be point estimates from a single training run. No error bars, standard deviations, or significance tests are provided. This makes it impossible to determine if the reported gains are statistically significant or merely the artifact of a single, fortunate random seed, which undermines the scientific validity of all conclusions.\n\n5. Missing Ablation on the Critical Y-Encoder Component: The entire methodology is critically dependent on the properties of the frozen y-encoder (SONAR). The paper fails to provide the most crucial ablation study: testing the VL-JEPA framework with different frozen text encoders (e.g., CLIP's text encoder, or a standard Sentence-BERT). Without this, the paper's claims are not generalizable. It is impossible to know if the authors have discovered a robust, general-purpose framework or simply a special-case solution that is uniquely and luckily compatible with the SONAR embedding space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BoqSFJSytQ", "reviewer": "Reviewer_BnrP", "summary": "This paper presents VL-JEPA a vision language model built on top of a JEPA architecture. The model is evaluated on video understanding and world modeling.", "questions": "1 - Why the benchmarking of this model does not follow standard VLM benchmarking suites?  \n2 - Is there a justificiation for focusing on video understanding?  \n3 - Do VL-JEPA need a pre-training phase? How does the model size scale with the data used for training? In the paper, it is said 64M samples are seen, how much is this in number of tokens?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 - The paper is clearly written and easy to follow and understand.  \n2 - A new vision-language model leveraging JEPA architecture instead of regular transformer decoders.  \n3 - Comparable performance to existing transformer-based VLMs, with less parameters.  \n4 - Extensive details are given about the training setup and resources.", "weaknesses": "1 - The model seem to be focused on video understanding as most of the training data are related to this task. This raises questions about the comparision to other VLMs that are trained and designed to be more generalist.  \n2 - Experiments focus on only a subset of use cases of a vision-language model (video understanding). More experiements on other types of tasks wuold have been appreciated (e.g., MMMU, OCRBench, DocVQA, etc.). If the JEPA architecture is intended to replace transformer-based VLMs then more generalization experiemnts are required.  \n3 -  The choice of evaluation benchmarks is not well justified. For example, WORLDPREDICTION-WM is not known by the vision-language modeling community. If I'm not mistaken the paper introducing this benchmark [1] was cited once.  \n\n\n[1] Chen, D., Chung, W., Bang, Y., Ji, Z., & Fung, P. (2025). WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning. arXiv preprint arXiv:2506.04363.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XvxtoT53pt", "reviewer": "Reviewer_sGvm", "summary": "This paper introduced VL-JEPA, a visual language JEPA model that is trained by predicting the embedding of target texts. The authors demonstrate that this results in faster and more efficient training, and for selected tasks yield state-of-the-art results. The model also obtains impressive zero-shot retrieval scores despite its training paradigm. In addition, a promising approach for selective decoding was also presented.\n\nOverall, the paper is well-written, and it is easy to find the relevant pieces of information.", "questions": "I would love to get a more substantial justification for the choice of benchmarks/evaluation datasets.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper introduces (or successfully reapplies) the JEPA architecture to the VL setting.\n\n- shows notable gains in both training speed and performance on zero-shot video captioning and classification (fig 3) while using a well-argued training procedure (JEPA).\n- shows non-trivial adaptation to retrieval and open-label classification. E.g. seen on youcook2, MSR-VTT (table 6).\n- The paper explores underexplored areas in the field, by exploring alternatives to generative token decoding, resulting in promising decoding strategies (selective decoding) and a reduced number of parameters compared to alternative models (fig 3, table 2, 4)", "weaknesses": "- The relevant benchmarks used for evaluation are only briefly introduced. I would have loved to see a more substantial justification for choosing these specifically.\n- While it is stated that the model and code will be open source, it could be shared through existing anonymous platforms\n- VL-JEPA is in Table 6 compared to contrastively-trained models. It is implicitly argued that this is the reason for the subpar performance on some of the tasks. I would have loved to see a contrastive adaptation to see if this assumption is correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ep6qJYuJ2Z", "reviewer": "Reviewer_NXvc", "summary": "This paper introduces VL-JEPA, a kind of non-autoregressive vision-language model that predicts target text beddings from visual tokens and a query. The inference is conducted in the embedding spaces. This non-autoregressive nature allows for selective and low latency decoding at inference, while still exhibits strong zero-shot capability.", "questions": "1. Most datasets used in this paper do not contain question-style queries. What are the queries used for training and evaluation? \n2. Are you using any queries when you perform CLIP-like zero-shot evaluation? \n3. Since your training data only entails limited number of tasks, have you tested the instructions following capability of the model?\n4. It seems that the current VL-JEPA can only perform well on some standard video understanding tasks such as action recognition and step recognition. For such tasks, what is the advantage of Encoder-decoder model like VL-JEPA, compared with CLIP-style encoder-only models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The exploration of a novel architecture deviated from the current mainstream (autoregressive VLM) is meaningful and encouraged.\n3. The low-latency and selective decoding properties of VL-JEPA makes it adaptable for many practical applications such as robots and wearable devices.", "weaknesses": "1. The core idea of VL-JEPA, to predict the target response in the embedding space, is very similar to LCM [1]. The training objective is also the same. However, the authors did not discuss the similarity and relationship with LCM.\n2. Encoder-decoder architecture may impair the capability of the model to understand long query and generate long responses. The model can neither perform multi-round interaction. \n3. The evaluations use accuracy and CIDEr as the main metric. It does not consider the readability of the generated responses. User study or LLM-based evaluation is needed.\n4. In Table 2, the bold metrics are not the best ones.\n\n[1] Barrault, Loïc, et al. \"Large concept models: Language modeling in a sentence representation space.\" arXiv preprint arXiv:2412.08821 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)