LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures
Authors: Hai Huang
Yann LeCun
Randall Balestriero
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
JEPA
fine-tuning
pretraining
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior to their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: {\em can language training methods learn a few tricks from the vision ones?} The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step in that direction where we develop LLM-JEPA, a JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA is able to outperform the standard LLM training objectives by a significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSM8K, Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: \url{https://github.com/galilai-group/llm-jepa}.

Published as a conference paper at ICLR 2026
LLM-JEPA: LARGE LANGUAGE MODELS MEET JOINT EMBEDDING PREDICTIVE ARCHITECTURES
Hai Huang
Yann LeCun NYU
Randall Balestriero
Atlassian
Brown University rbalest wwwrowrbrrrweor edu
hhuang3@atlassiian
yann lecun@nyu. edu
ABSTRACT
Large Language Model (LLM) pretraining, finetuning, and evaluation rely on input-space reconstruction and generative capabilities. Yet, it has been observed in vision that embedding-space training objectives, e.g., with Joint Embedding Predictive Architectures (JEPAs), are far superior their input-space counterpart. That mismatch in how training is achieved between language and vision opens up a natural question: can language training methods learn a few tricks from the vision ones? The lack of JEPA-style LLM is a testimony of the challenge in designing such objectives for language. In this work, we propose a first step that direction where develop LLM-JEPA, JEPA based solution for LLMs applicable both to finetuning and pretraining. Thus far, LLM-JEPA able to outperform the standard LLM training objectives by significant margin across models, all while being robust to overfiting. Those findings are observed across numerous datasets (NL-RX, GSMSK. Spider, RottenTomatoes) and various models from the Llama3, OpenELM, Gemma2 and Olmo families. Code: https:// github. com/ galglllllllllao waalllaiggruup//a jepa
80
Baseline
71.0
LLM.JEPA (Ours)
70-
66.6
56,0
55.2
51.5
Accuracy [%
50
40-
Acturacy
40
32.0
30
27.2
30,9
4
31,8
4
22.5
22.5
21.6
10.4
20
SYNTH
U SYNTH
gemma2 SYNTH
OpenLLM SYNTH
Uama3 Spidde
Liama TURK
GSMBK
Epoch
Figure 1: LLM-JEPA produces strong fine-tuned models across datascts and models.
INTRODUCTION
The research landscape around representation learning has been increasingly divided into two camps:
(i) generative or reconstruction-basse methods (Brown et al. 2020bb Chowdhery al.) [2023] He et 2022 LeCun 2022) and (ii) reconstruction- free Joint Embedding Predictive Architectures (JEPAs)
(Assran al. 2023 Baevski al. 2022 Bardes al., 2024]. While the former self-explanatory, the latter learns representation by ensuring that different views, e.g., pictures of same building at different time of day, can be predicted from each other, while preventing a collapse of the embeddings. By moving away from input-space objectives, JEPAs training benefits from less biases (Littwin et al. (2024), at the cost of potential dimensional collapse of their representation (Jing et al. 2021 Kenneweg et al.) 2025). That divide has been well studied in vision, where it found that JEPAs offer multiple provable benefits when comes to knowledge discovery for perception tasks. In the realm of Natural Language Processing however, reconstruction- based methods remain

---

Published as a conference paper at ICLR 2026
Natural Language to Regular Expression
(Teex))((
source: "lines containing the string 'dog' 2 or more times"
Text
Enn(ex)) Enc(Code))
target: ".*(dog) ).
Code
Natural Language to SQL
Enc(Tex))
Enc(Code)
db_id:
"department" "ameent_annagmmnt"
Text
question: "How many heads of the departments are older than
Text
Code
query:
"SELECT count(*) FROM head WHERE age
Code
56"
Figure 2: Left: JEPA applied NLP tasks that has Text and Code. where Tezt and Code are naturally two views of the same thing. Right: (top): An illustration of the NL-RX-SYNTH dataset, where each sample consists of a description of the regular expression in natural language (T (Teat)) and the regular expression itself (Code). (bottomm) The Spider dataset, where Tent is the datahase ID and description of the SQL. query and Code is the SQL query itself.
predominant. In fact, today's Large Language Models are mostly judged from their ability to generate samples and answers in input space in text form-making it challenging to leverage JEPA objectives.
Yet, LLMs task also involve perception and reasoning where JEPA is known to be preferable. It thus seems crucial to adapt JEPA solutions to LLMs in the hope to showcase the same benefits as witnessed in vision. This first step is exactly what present in this study. We propose improve the representation quality of LLMs by leveraging a novel objective combining both the original reconstruction based loss-with an additional JEPA objective. To do so, focus first tasks and datasets that are inherently suited for JEPA objectives: the ones providing multiple views of the same underlying knowledge. One typical example is git issue and the corresponding code diff (fig. 2 (Jimenez et al.] 2024). The two samples are two views-one being plain English and one being in code-of the same underlying functionality. Let's use that particular example to highlight our core contribution:
Viewing the (text, code) pairs as views of the same underlying knowledge enables JEPA objec- tives to be utilized with LLMs, complementing the standard text + code generative task.
We strongly emphasize that being able to obtain non-trivial views, such as described above, is crucial to the success of JEPA objectives. While we restrict ourselves to datasets offering those non-trivial views, developing a mechanism akin to data-auumentttion vision would enable JEPA objectives to be used any dataset. Nonetheless, we believe that our proposed solution-coines LLM-JEPA-amn empirical study will serve as first step towards more JEPA-centric L.LM pretraining and finetuning. We summarize our contributions below:
Novel JEPA-based training objective: We present the first JEPA-based training objective for LLMs operating in embedding space and with different views-perfectly following vision-based JEPAs without sacrificing the generative capabilities of LLMs
Improved SOTA: We empirically validate our formulation in various finetuning settings, where we obtain improvements over standard LLM finetuning solutions. We also explore pretraining scenarios showing encouraging results of LLM-JEPA
Extensive empirical validation: on various model family (Ilama, gemma, apple/openelm, al- lenai/olmo), dataset (NL-RX, GSM8K, Spider, Rotten Tomatoes), and size.
BACKGROUND
Large Language Models. Contemporary LLMs are mostly built from the same core principles: stacking numerous layers of nonlinear operations and kkip-comeection--knovn as Transformers. While subtleties may differ, e.g., about positional embeddings, initialization, normalization, the main driver of performance remains the availability of high quality dataset during the pretraining stage. The training objective in itself has also been standardize throughout methods: autoregressive token-space reconstruction. Let's first denote by LLLM the typical L.M objective used for the specific task and
2

---

Published as a conference paper at ICLR 2026
dataset hand. In most cases, this will be cross-entropy loss between the predicted tokens and the ground truth token to reconstruction. We note that our LLM-JEPA construction is agnostic of LLLM hence making method general to numerous scenarios.
LLLM ( (Texti.L-1, Textz) CrossEntropy (Classifier (Enc([Txxt.. Textz), (I)
where Classifier(-) predicts the logits of the next token Text, given the past tokens Text1:L-1- Computation of eq. (T) is done at once over through causal autoregression. Different stages and tasks may vary the input and output of the loss.
Alternative training objectives. Next token prediction (NTP) is the prevalent pretraining solution for today's 's latest LLMs. There exists few alternatives, e.g., SimCSE leverages contrastive loss in the latent space by treating different dropout-induess views of the same sentence as positive pairs, resulting in state-of-the-art sentence embeddings quality (Gao et al. (02TT. In a similar spirit, (Wang et al. 2022) uses weak supervision from text pairs to learn a joint embedding architecture. An alternative relying on pretrained models instead of the supervised text pairs was explore in (Ni et al., 2021). While those approaches are powerful, they are all concern with producing text embeddings without generative capabilities which greatly limits the applicability of those models since numerous evaluations and use-cases require generation-which is core to our proposed method. Another solution employs BERT pretraining coupled with a latent space semantic loss ensure that representations of semantically similar sentences are nearby in embedding space. This additional term led to improved performance on semantic tasks compared to masked language modeling alone (Reimers Gurevych 2019) yet again by building atop BERT this solution prevents generative evaluation and use.
3 JEPA-LLM: IMPROVING LLMS' REASONING AND GENERATIVE CAPABILITIES
We propose the LLM-JEPA loss in section 3.1 along with extensive empirical validations in section demonstrating clear finetuning and pretraining benefits.
3.1 THE LLM-JEPA OBJECTIVE
Throughout this section, we will use Text and Code as concrete examples of having different views of the same underlying knowledge. It should be clear to the reader that our proposed LLM-JEPA objective handles different types of views similarly.
The construction of our LLM-JEPA objective relies on two principles. First, we must preserve the generative capabilities of LLMS and we therefore start with the LLLM from eq. [ Second, we aim to improve the abstraction capabilities of LLMs using the joint embedding prediction task. On top of LLLM. we then propose to add the well-established JEPA objective leading to the complete loss L defined as
CLLM-JEPA C Texte) +A d(Pred(Enc (Text)), (hoc)))d)) (2)
abstraction capubilities (JBPA)
generative capahilities (1.LM)
where X 2 0 an hyperparameter balancing the contribution of the two terms, Pred and Enc are the predictor and encoder networks respectively, and d is metric of choice, e.g., the Ez distance. Let's now precisely describe each of those components.
The encoder. We use the h dden_state of the last token from the last layer as the embedding of an input sequence-as as commonly done for L.M probing. We pack both Text. and Code into single context window, applying attention mask to ensure they do not reference each other. Implementation-wise
most HuggingFace sransformerz support an additive attention mask, where setting entry (i. () -oo prevents token j from attending token i (for i j). Using this mechanism, we implement LLM-JEPA with only one additional forward pass. This introduces extra cost during training. but not during inference
section for further discussion.
The metric. When comes comparing embeddings, it now widely accepted vision to leverage the cosine similarity. We thus propose do the same for I.M-JEPA.
3

---

Published as a conference paper at ICLR 2026
The predictor. We leverage the a nature of LLM and their internal self-attention to define a tied-weights predictor. By introducing a special token [PRED] at the end of a given Text, we allow for further nonlinear processing of the input hereby producing Pred(-) at the final embedding (the hidden state of the last [PRED] token) of the last layer. By reusing the internal weights of the LLM for the prediction task, we greatly reduce the training overhead and architectural design choices. Practically, we append & E .O.....fu K} predictor tokens to an input prompt and use the embedding of the last predictor token to be Pred( (red)(Enc()), When k O, the predictor simply becomes an identity mapping, i.e., (r) I.
3.2 IMPLEMENTATION WITH CUSTOM ATTENTION MASK
The most important challenge in implementing our proposed LLM-JEPA objective lies in obtaining the embeddings of the different views, e.g., Text and Code in eq. (2). A priori. it is not possible to get them one forward pass because the current self-attention-albess being causal. Even if we were to concatenate the two views which is already done for the next token prediction part of the loss, it would make the representation of the second view rely on the first view.
result, we propose the following custom self-attention mask that is causal per block, with number of blocks set to 2:
def acaitive mask (k: int):
"*"Returns & by & triangle mask. ... mask = torch, zeros ( (k, k), dtype-torch. float32)
3
mask (ask(torc..tte (tterch.triuttoeee ones ones (k, k), diagonal-1) -- 1] -torch. inf return mask
Initialize all elemets to -inf. mask = surch.fulll bhatch_siize 2, 1, length, seq_ ).to (device)
-torch.inf
Text starts from t_start and is of size t_size, and Code starts size c_size. set for the 1-th batch. mask [i, :, t_start: t_start + t_size, t_start: t_start + t_size]
from c_start and
11
additi tieetiieetttstsiiiiie
mask [i, :, c_start: e_start + c_size, c_start: c_start + c_size] additive (askkc_ssassa size)
By leveraging the above implementation, we able obtain the LLM-JEPA loss value in one extra forward passes instead two. While this is still substantial slowdown, we will explore later the manuscript a dropout version where the JEPA term is only applied some % of the mini batch-the end result will be that are comparable FLOPS the proposed LLM-JEPA is still able to outperform the baseline.
Relation to Previous Work. Because loss functions such as CLLM (input space reconstruction since tokens are lossless compression of the original prompts) have been shown be sub-optimal in vision, a few LLM variations have started to employ embedding space regularizers and training objectives (Barrault al. 2024 Wang & Sun 2025). Current solution however rely on intricate structural constraints of the embedding space, e.g., e. hierarchical organization and cluster, and thus fall out of the JEPA scope. We also note that our interpretation of views when it comes to LLM datasets, e.g., (text issue, code diff), is something that has been leveraged as part of the LLM finetuning solutions-bi by learning to generate one from the other-win without other-without 1EPA-style loss. This includes natural language
to regular expression translation (Locascio et al. 2016 Ye et al. 2020 Zhong et al., 2018). natural language to SQL. parsing (Guo et 2019
Iyer et |0099] al. 2017 Li et al. 2023] Wang et al. [0199] Yu
2018) and the more recent issue descriptions to code difis (Cabrera Lozoya al. [021] Hoang al.
ct al.
2020 Tian et al. 20200 Zhou et al. 2023). More intricate examples involve text-basee problem
solving and their counterpart program induction (Amini et al. 2019
Cobbe al. 2021 Hendrycks
et al. 2021 I Ling et al. 2017).
3.3 A GOOD NEXT TOKEN PREDICTOR NOT A GOOD JEPA
Before validating the proposed LLM-JEPA to real task and models, we ask ourselves simple question. Is it really necessary to have an additional JEPA term? Is that term already implicitly minimized by the original next token prediction objective?
4

---

Published as a conference paper at ICLR 2026
Base Model
Regular Fine-tuning
baseline (NTP) baseline (pred)
LLM-JEPA (ours)
LLM-JEPA (NTP)
LLM-JEPA (pred)
J-M-EEPAARA k=0 (ours)
Skquarkebe
0.1
0.01
12a
0.001
100
Figure 3: left: The top 100 singular values of Enc(Tees a few magnitudes lower than that of base model and regular fine-tuning, meaning the mapping from Text to Code are confined within narrow subspace, fostering the nice structure we see in Figure|4] Right: Losses in fine-tuning with LLLM loss (CLLM) and GLLM-JEPA loss (CLLM-JEPA, our method). We the cross-entropy loss for next token prediction (Loss LLM. LLLM in chart) and JEPA prediction loss (D(:,) :); pred in chart), although the latter does not contribute in the baseline case. The accuracy is 51.95% for LLAM and 71. 11.10% for LLI.M-RPP
Enc(Code) The curves of LLM-JEPA (ours) are
both
Since LLAM and LLAM- JJPAA share similar LLAM loss, the LLAM loss cannot explain the gap between the accuracy. pred stays constant in CLLMM while is minimized in CLIM-JFRE hence pred should be main reason behind the accuracy gap.
To answer that, we compare two controlled settings. We will be using Llama-3.2- 1-,.-11B-Inntruu and NL-RX-SYNTH and have two training setup. In the first, we do the usual next-token prediction loss but monitor the JEPA objective, i.e., no gradient comes from the JEPA loss. In the second, we will use the proposed LLM-JED cases. We obtain the following finding in fig. 3 minimizing LILMM does not implicitly minimize LarPA-indicating that it is required to add that term during training. This can be seen by comparing the red and green lines. A natural follow-up question is are we trading off next token prediction for JEPA?. In the same fig. we obtain that the next token prediction capability is not hindered by the presence of the JEPA term (compare the blue and yellow lines that are overlapping). This is a very important observation echoing what was empirically observed in (Balestriero & LeCun 2024 where it was observed that autoencoders in image space could become much stronger classifiers without sacrificing the reconstruction and generative objective. All in all we obtain that employed LLM-JEPA only brings additional structure of the LLM latent space without altering its generative capabilities. As we will see the follows sections, we indeed obtain much better performances that the baseline-ipg in the generative evaluation setup.
for gradient computation. We also monitor the prediction loss in both
EMPIRICAL VALIDATION: LLM-JEPAS OUTPERFORM LLMS
In this section, we first validate the proposed LLM-JEPA across four model families and four datasets with natural two-view structures (section 4.1 The consistent improvements observed across the board motivate us to further examine the internal representations of LLM-JEPA to better understand the source of its strength (section 4.2). We conclude this section with rigorous ablation study on key design choices (section 4.3)
We adopt universal protocol across all experiments by using five fixed random seeds, (82, 23, 37, 84, 4), for training. Each experiment is repeated five times, once with each seed, This setup enables assess the stability of LLM-JEPA and to conduct paired one-tailed f-tests for statistical significance.
4.1 FINE-TUNING AND PRETRAINING STRONGER GENERATIVE MODELS VIA JEPA
LLM-JEPA Improves Finetuning. We run experiments across multiple pretrained LLMs (Llama- 3.2- 12-1B-Insee Instruct (Grttafiioi et al. 2024), gemma-2-2b- (Team et al. 2024). OpenELM-1 OpenEELM-I_IBInsttruue IB-Instruct (Mehta et al. 2024), and
(OLMo al. 2024)) with various datasets
5

---

Published as a conference paper at ICLR 2026
RNL-RX-SYNTH, NL-RX-TURK (occascio et al. 2016). GSM8K (Cobbe et al. 2021 Spider Yu ct al. 2018
To demonstrate that LLM-JEPA improves from the strongest possible baseline, first search for the best learning rate ir {le - 2e - 5, 4e 8e - 5] by selecting the value that yields the highest accuracy of LLLM after 4 epochs for given (model,dataset) pair. Then we tune the hyperparameter specific to -LIM-IEPA & and À in a two dimensional grid defined by (k, X) E {0, 1, 43 {0.5, 1, 2, 43 (fig. 7 and table 12 in the Appendix, also appendix AA.8 presents an empirical approach to efficiently identify the optimal (A, For both NL-RX-SYNTH and NL- RX-TURK, accuracy is exact match of the generated regular expression; for GSM8K, accuracy is exact match of the final result; and for Spider, accuracy is exact match of the execution result of the generated query.
We provide results demonstrating that LLM-JEPA improves performances across
four model families, see fig. left and table 14 in the Appendix.
four datasets, see table 15 in the Appendix.
6 training epochs (fig. right). We also observe that LLM-JEPA resists overfitting, whereas
standard fine-tuning does not.
four different sizes: IB, 3B, 7B, and 8B, see table 17 in the Appendix
Examples of inputs, targets, model predictions, and error analyses are provided in table [ (more examples can be found in table 13 in the Appendix). For LoRA fine-tuning, the performance gains of LLM-JEPA hold consistently across different LoRA ranks (table 10 in the Appendix). We also observe the same resistance to overfitting in the LoRA setting (fig. the Appendix).
Table 1:
Regular expressions generated by Llama-3.2-1 3.--st-nntt after fine-tuning with CLLM loss and
CLLM- JEPA loss (ours). Color code: wrong
extra
missing
Ground Truth
CLLM
LLLM JEPA (ours)
lines not having the string 'dog' followed by number, more times (0oog:"[[-9).)EE)))
((dog. (33,))
((dog ((0-9).)))..)
lines containing ending with a vowel, zero or more times
*(*M((AAEOOsccioo))))
(. ((((AEIQUaciiu))
(+) (.9 (IABIOUacioul)
lines with number character before vowel
((Oo9I)()))((AAEHUUCaa))))))) ((0--9)..)). ((AOOUUCiI,)
lines with words with the string 'dog'. letter, and number
(((09))){rddo2)MM[AZ==)))))) (((09))){(ddo))))((AZ7)))))))
(C [0-9))&(dog))( ( (AA-Za-))
LLM-JEPA Improves Pretraining, A natural extension of our fine-tuning results is to examine pretraining. We pretrain L.ama-3.2-1BB- B--Isruutttt from randomly initialized weights on the NI-RX- SYNTH dataset. Owing the limited size of the dataset, the pretrained model fails reliably learn how to terminate generation. To address this, we adjust the evaluation criterion, deeming a generated solution valid long begins with the ground-truth sequence. We obtain that LLM-JEPA also improves the quality of the learned representation, as shown in table
Table 2: Pretraining accuracy on dataset NL-RX SYNTH by Next Token Prediction (CLLM) loss vs. CLLM- JEPA loss (our method). We inherit the best configuration from fine-tuning. Each case runs five times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed t-Test.
Model
Method
Accuracy (%) t
p-value 4
Config
LLLM
54.38 + 1.70
Ir Se
Llama-3.2- 1..21B-Innee In-struuctt
2.94c -
JILM-JRPA (ours)
60.59 + 1.01
A
2, 3, same b
We then conduct a more advanced pretraining experiment on dataset cestwc/paraphrase containing groups of 5 paraphrases. We leverage the five paraphrases construct the JEPA loss by having the

---

Published as a conference paper at ICLR 2026
Text
Code
(a) Baseline: Fine-tuned by NTP loss
(b) LLM-JEPPA (Ours) & =
Figure 4: 1-SNE plot of Texx and Code representations (a) Baseline that is fine-tuned with Next Token Prediction (NTP) loss. (b) LLM-JEPA (ours) with & = 0. Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model. A full version of this figure is in appendix A.2
i-th version of a paraphrase predict the (i + 1)-th version. The goal to encourage the JEPA loss to tie the embeddings of all versions into a compact subspace, providing geometric foundation to align their representations. We pretrain the model for four epochs and then evaluate it Rotten Tomatoes and Yelp after one epoch of fine-tuning. Although there no direct link between the pretraining and evaluation datasets, we show that LLM-JEPA pretraining yields statistically significant improvements in downstream performance (see table in the Appendix). Note that fine-tuning does not employ the JEPA loss highliging that the benefits arise specifically from the pretraining stage.
For Rotten Tomatoes and Yelp, we fine-tune the model to generate discrete sentiment labels: Good and Bad for Rotten Tomatoes, and Very Good, Good, Medi ocre, Bad, and Very Bad for Yelp. A prediction is deemed correct the generated output begins with the ground-truth label. This evaluation approach-mann mapping free-form generation to categorical labels and applying prefix matching- follows common practice in prior work on text classification with generative models (Mc Cann et al. 2018 Raffel et al. 20200 Wei et al. (022).
Lastly, we provide in table (in the Appendix) generated samples demonstrating that JEPA pretraining does maintain the generative capabilities of the model when prompted with the first few tokens in the cestwc/paraphrase dataset.
4.2 STRUCTURED REPRESENTATIONS INDUCED BY LLM-JEPA
We also examine the representation space better understand how LLM-JEPA regularizes learned fea- tures. Specifically, we plot 1-SNE embeddings for both Text and Code across three settings: the base model. model fine-tuned with LLLM, and a model fine-tuned with LLLM-JEPA. As shown in fig. 4 clear structure emerges after fine-tuning with LLLM-JEPA. We hypothesize that CLLM-JBPA enforces structure in the representation space by constraining the mapping from Eno(Tex) within narrow subspace. If this is the case, the SVD decomposition of Enc Text) Enc Enc(Code) should yield significantly smaller singular values, which confirmed in fig. 3 Furthermore, we hypothesize that the mapping is approximately linear. To test this, we compute the least-squares regression error, and table in the Appendix supports this hypothesis. Together, these results sug- gest that LLM-JEPA promotes near-linear transformation between Text and Code representations, which may underlie its accuracy improvements.
Enc(Code)
4.3 ABLATION STUDY ON DESIGN CHOICES
In this section, examine several alternative design choices. Specifically, compare the of cosine similarity against fn-mrr and mean squared error (MSE); appending a [PRED] token to the end of Text versus prepending it; and using Text. predict Code versus the reverse direction
7

---

Published as a conference paper at ICLR 2026
(Code Text). As shown in table B none of these alternatives perform as well LLM-JEPA, although some outperform standard fine-tuning.
Table 3:
Fine-tuning accuracy on NL-RX-SYNTH under different design choices. Reported are average accuracy and standard deviation across runs. Learning rate ir = 2e-5, X = 1.0, and = 1. The 2.22 for f2-norm is not typo. Sec appendix A.11 for discussion.
Accuracy (%)
Baseline
LLM-JEPA
22-norm
MSE
Prepend
Code - Text
InfoNCE loss
57.29 5.32 71.46 + 1.34 2.22
0.07
70.64 1 2.05 68.07 + 2.57
65.70 + 2.63
34.40 + 6.10
Additionally, we replace our cosine similarity loss with the InfoNCE loss (van den Oord et al. 2018), which results lower accuracy compared to the baseline. Moreover, its standard deviation is substantially higher than that of other alternatives. We use the commonly adopted temperature of r = 0.07 for the InfoNCE loss (Chen et al., 2020) Radford et al., 2021). We performed an additional ablation in which replaced the hidden state of the last token with the average of all hidden states over all tokens. This modification degraded performance to 65.46 + 3.51.
We hypothesize that the accuracy improvements stem from representation alignment-i.ee the model learns to collapse representations of semantically similar text into a tightly aligned region (often nearly line), which facilitates extrapolation and improves generalization. Under this view, InfoNCE can be detrimental because its contrastive objective explicitly pushes representations apart, counteracting the alignment needed for extrapolative behavior. We validated this hypothesis through additional experiments; please see appendix A.10 for details.
For O, we append multiple predictor tokens: [PRED_] 1], .... [PRED_K k], to the text and use the
hidden state of [PRED k] as the predicted representation. To better understand the mechanism, we examined whether performance gains come primarily from (i) the number of predictor tokens or (ii) the variety of distinct predictor tokens. We conducted ablations using identical predictor tokens, and the results (table 4 show only minor differences. This suggests that the dominant factor is simply increasing the number of prediction steps (FLOPs), rather than requiring distinct token embeddings.
Table 4: Comparison of distinct vs. identical [PRED] tokens. Both settings achieve comparable performance.
Config
Distinct [PRED]
Identical [PRED]
Llama 3.2/ GSM8K
36.36 + 0.20
36.74 0.70
BRenELM/SYNIE
25.40 + 2.40
25.01 = 1.60
An alternative design is to replace the [PRED] tokens with standalone linear predictor network. A linear predictor is sufficient because the mapping from Text to Code representations nearly linear (table 15 and fig. 3 left). However, in the fine-tuning regime this approach suffers from cold-start problem the linear head must be trained entirely from scratch. Empirically, a straightforward linear predictor yields consistently subpar performance compared to using [PRED] tokens (table 5).
Table 5: Trainable linear predictor yields consistently subpar performance compared using [PRED]tokens.
Linear À 0.5
Linear À
Linear À
[PRED] = 1
70.16 1.87 67.99 + 2.70
64.54 + 4.99
71.46 1.34
TOWARDS FASTER AND MORE GENERAL LLM-JEPAS
The structured representations induced by LLM-JEPA have the potential to provide universal benefits across diverse LLM applications. In this section, we explore its limits by evaluating datasets without natural two-view structures and models with richer capabilities (section 5.1). The promising results further motivate us to investigate methods for accelerating LLM-JEPA (section 5.2), as computational overhead remains key obstacle to broad adoption.
8

---

Published as a conference paper at ICLR 2026
Table 6: Fine-tuning accuracy Llama-3.2-11 3.2-1B-Instruee with Next Token Prediction (CLLM) loss vs. LLLM loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed TeTst.
Dataset
Method
Accuracy (%) f
p-value 4
Config
LLAM
20.12 + 0.41
Ir 2e - -
NQ-Open
2.44c - 3
-IPA (ours)
21.59 0.40
À 1024, k: 0. same
LLAM
27.93 + 0.46
= 4e 5
HellaSwag
8.12c
LELM (ours)
35.22 2.09
A = 1, 4, same Ir
Table 7: Fine-tuning accuracy of GSM8K with Next Token Prediction (CLLM) loss vs. LLLM- loss (our method). Each case five times. Average accuracy and standard deviation reported. We also report p-value of paired, hinge-tailed Tesst.
Model
Method
Accuracy (%)
p-value +
Config
Qwen3-1.7B
LLLM
44.32 + 0.39
0.0115
=
LLLM-JEPA (ours)
45.00 + 0.40
À 1, k = 0, same Ir
CLLM
13.87 + 1.01
Ir = 8e - 5
RI-Distil--wen--1.5
0.0396
-LAM-JEPA (ours)
15.04 + 0.15
0.5, k 1, same Ir
5.1 BEYOND CODE: APPLYING LLM-JEPA TO Q& DATASETS AND REASONING MODELS
We evaluate Llama-3.2- 1 1-- Instruct on two QA benchmarks: NQ-Open (Lee et al. 2019a) and
HellaSwag Zellers et al., 2019). Our results show that LLM-JEPA achieves statistically significant improvements on both datasets, demonstrating its capability beyond the canonical setup where Text and Code are treated as two complementary views of the same object.
For NQ-Open, we regard Text as the question and Code the answer span, typically consisting of only a few tokens, which contrasts with the more balanced sequence lengths found other datasets. HellaSwag, by contrast, is context-completion multiple-choice task. Rather than defining Code as the answer label (a single token from A, B, C, D). we instead let Text denote the context and Code represent the correct continuation. This formulation differs from prior setups in two important ways: (i) Both Text and Code are now integral components of the question, and (ii) The relationship between context and completion is more diverse than the near-equivalence seen in NL-RRegex or NL-SQL mappings.
Despite these differences, LLM-JEPA consistently improves accuracy on both benchmarks table
For NQ-Open, generated answers may include additional syntactic supporting tokens. Following prior work, we deem a prediction correct if any ground-truth answer appears as substring of the generated output (Lee et al. 2019b Guu et al., 2020 Izacard & Grave 2021 For HellaSwag, we compute the relative probabilities of the four candidate options A, B, C, D and select the answer with the highest probability, consistent with standard practice in multiple-choice language understanding
benchmarks (Zellers et al. 2019 Brown et al. 2020a OpenAl 2023).
scale À up to 1024, without encountering a plateau. While table 12 in the Appendix suggests that extreme values of can degrade accuracy, in certain cases it remains beneficial to push X further, yielding extra gains.
An additional observation is that performance continues to improve
We then evaluate two strong reasoning models -wen3-wennn (Yang et al. 2025) and DeepSeek- R1 RI-Distill- RI-Distill-Qwen-Il Qwen-1.5B (DeepSeek-Al et al. 2025 on GSM8K. As shown in table 7 both models achieve statistically significant improvements when augmented with LLM-JEPA. These results offer promising evidence that LLM-JEPA extends its benefits to large reasoning models (LRMs).
5.2 FASTER LLM-JEPAS VIA Loss DROPOUT
To further reduce compute, we introduce random JEPA-loss dropout (LD). During training or fine-tuning, randomly drop the JEPA loss at specified LDD. rate. Loss dropout is applied the

---

Published as a conference paper at ICLR 2026
70
Figure 5: LLM-JEPA converges faster than regular fine-tuning at the same PFLOPs. Furthermore, random JEPA-loss dropout (LD) helps save PFLOPs and boost accuracy the same amount of compute. LD 0 regular LLM-JEPA. Learning rate Ir 2 and 1. varies.
60
50
LD-0,5, A-2.0
LD-0,5, A-1.0
30
LD=0
20
4.83
9.66
14.49
10:32
24.15
28.98
PFLOPS
Table 8: Random JEPA-loss dropout (LD) help save PFLOPs and the same time, boost accuracy. LD is the regular LLM-JEPA. Reported average accuracy and standard deviation across Each row is 4.83 PFLOPs apart. Learning rate Ir 2e-5 5 and l 1. varies.
Accuracy (%) f
LD=0, x=1
LD=0.5, x=1
LD=055, X=2
LD=0.75, x=1
LD=0.75,
LD=0.75, X=4
19.85 + 2.44
25.00 3.73
24.50 4.40
32.46 + 3.32
32.10
31.45 3.34
40.70 + 2.67
48.96 + 6.03
50.71 6.52
53.77 5.53
17.03
56.75 + 4.63
55.60 + 5.16
64.08 2.75
63.79 5.96
64.51 - 7.28
67.03
65.32 + 3.54
60.43 + 3.21
89.87
70.11
67.80 4.94
06.80
68.93 + 4.59
63.57 + 1.01
69.74 2.99
71.20
70.00 4.74
70.77 4.08
72.42 + 1.28
63.96 + 1.07
70.60 + 3.05
72.11 + 2.18
70.31 1.64
70.92 + 4.62
73.08 + 1.28
batch level. When active, it eliminates the need for an extra forward pass to compute Enc(Text) and Enc(Code) thereby saving compute. LD = a, the per-epoch cost becomes - a times that of standard fine-tuning, since each batch saves forward passes. As shown in Figure 5nd and Table g LLM-JEPA tolerates aggressive loss dropout rates (e.g., 0.5 or 0.75), which leads to higher accuracy under the same compute budget. Moreover, increasing À in proportion to the dropout rate further improve performance. Empirically, we observe that keeping provides useful guideline for co-tuning À and balance compute efficiency and accuracy. The use of the loss dropout coupled with our custom attention mask offers some positive perspectives further scale LLM-JEPA full scale pretraining with minimal computational overhead. See appendix A.9 for additional experiments at extremely high dropout 0.0625.
(1 - a) approximately constant
CONCLUSION AND FUTURE WORK
We introduced an alternative training objective for LLMs leveraging JEPAs. Our formulation is an exact replicate the JEPA objective extensively used in vision-but that hadn't been adapted to language yet. Crucially, our proposed LLM-JEPA maintains the generative capabilities of LLMs while improving their abstract prompt representation as empirically validated across datasets and models. While our experiments mostly focus on finetuning, preliminary pretraining experiment are promising which we plan to scale and more thoroughly test future work. Regarding the limitations of LLM-JEPA, the primary bottleneck at present is the 2-fold increase in compute cost during training, which is mitigated by random loss dropout.
Limitations Despite its strong accuracy gains, LLM-JEPA introduces two additional hyperparameters. As shown in fig. 7 the optimal configuration may occur at any point in grid (X, k)), which imposes significant cost for hyperparameter tuning. While we have not identified an efficient method to explore this space, we empirically observe that adjacent grid points often yield similar accuracy, suggesting the potential for more efficient tuning algorithm.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorsk, Yejin Choi, and Hannanch Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- formalisms. arXiv preprint arXiv: 13319, 2019.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, pp. 1298-1312. PMLR, 2022.
Randall Balestriero and Yann LeCun. Learning by reconstruction produces uninformative features for perception. arXiv preprint arXiv:2402.1133/ 2024.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404. 08471, 2024.
Loic Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussă, David Dale, al. Large concept models: Language modeling in sentence representation space. arXiv preprins (0X00:2412.08821, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of NeurlPS, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b.
Rocío Cabrera Lozoya, Arnaud Baumann, Antonino Sabetta, and Michele Bezzi. Commit2vec: Learning distributed representations of code changes. SN Computer Science, 2(3):150, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020. URL https: /arxivivivv.////// orrziv..rg/// abs /2002
05709
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 241240):1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, John Schulman, Jacob Hilton, Melanie Knight, Adrian Weller, Dario Amodei, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang,
11

---

Published as a conference paper at ICLR 2026
Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shamhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. I.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He. Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https://arxii org abs//5l1..21111. 12948
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.00 004208 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.2 2024.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. Towards complex text-to-sql in cross-domain database with intermediate representation. arXiv preprint arXiv:1905. 100.000055,,, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Retrieval augmented language model pre-training. Proceedings of ICMI., 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 2Xiv:2103.03874 2021.
Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall. Cc2vec: Distributed representations of code changes. In Proceedings of the ACM/IEEE 42nd international conference software engineering, pp. 518 518-529, 2020.
Srinivasan Iyer, loannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. arXiv preprint arXiv: S00iv:170.0087770 2017.
Gautier Izacard and Edouard Grave Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint wrxiv:200.001282 2021.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Leurning Representations (ICLR), 2024. URL httptttpps///eeeeie.iitieo n Forum? Cerum?idee (ideddddivFFFF 8YNMM66 Oral presentation.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supe learning. arXiv preprint arXiv:2110.09348, 2021.
Tristan Kenneweg, Philip Kenneweg, and Barbara Hammer. Jepa for rl: Investigating joint-emb predictive architectures for reinforcement learning. arXiv preprint arXiv:2504. 16591, 2025.
12

---

Published as a conference paper at ICLR 2026
Yann LeCun. A path towards autonomous machine intelligence (version 0.9.2). OpenReview, 62(1): 1-62. jun 2022. URL 0.9.2, released June 27, 2022.
Version
Kenton Lec, Ming- Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. (096-6096, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 18655/ v1/P19-1612. StRphttp::
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings ACL, 2019b.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13067-13075, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. 2017.
Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang, and Joshua Susskind. How jepa avoids noisy features: The implicit bias of deep linear self distillation networks. Advances in Neural Information Processing Systems, 37:91 300-91336, 2024.
Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay. Neural generation of regular expressions from natural language with minimal domain knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1918-1923, 2016.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. In Proceedings of ICLR, 2018.
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al. Openelm: An efficient language model family with open training and inference framework. arXiv preprint arXiv:2404. 14619, 2024.
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfci Yang. Sentence-15: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2/08.08877 2021.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo furious. arXiv preprint 000606501.00666, 2024.
OpenAl. Gpt-4 technical report. arXiv preprint
2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021. URL https
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In Proceedings of JMLR, 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019.
13

---

Published as a conference paper at ICLR 2026
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint a:X1n:2408.00118, 2024.
Haoye Tian, Kui Liu, Abdoul Kader Kaboré, Anil Koyuncu, Li Li, Jacques Klein, and Tegawendé F Bissyandé. Evaluating representation learning of code changes for predicting patch correctness in program repair. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering, pp. 981 981-992, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. Rat- sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942. 2019.
Boshi Wang and Huan Sun. Is the reversal curse a binding problem? uncovering limitations of transformers from basic generalization failure. arXiv preprint arXiv:2504.01928, 2025.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei Text embeddings by weakly- suukey-upeeriieed contrastive pre-training. arXiv preprint 2eXiv:2212.03333 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. In Proceedings of ICLR, 2022.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https::///// //s:://aae http//aaiivv/.. org/ abs 2505. 09388
Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, and Greg Durrett. Sketch-driven regular expression generation from natural language and examples. Transactions of the Association for Computational Linguistics, 8:679-694, 2020.
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale numan-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, 2018.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, Tao Xie, Jian Jian-Guang Lou, Ting Liu, and Dongmei Zhang. Semregex: A semantics-based approach for generating regular expressions from natural language specifications. In Proceedings of the 2018 conference empirical methods in natural language processing, 2018
Xin Zhou, Bowen Xu, GyngGyun Han, Zhou Yang. Junda He, and David Lo. Cobert: Self-supervieed code change representation learning. In 2023 IEEE International Conference on Software Mainte- nance and Evolution (ICSME), pp. 182-193. IEEE, 2023.
14

---

Published as a conference paper at ICLR 2026
Table 9: Generated samples by model pretrained by cestwo/paraphrase dataset. The pretrained model is not good at terminating sentence. prompt and generation
Ground Truth vs. Generation
Ground Truth
A garden of flowers and bench stating "City of London."
Generation
A garden of flowers and vase with a flower in it..............
Ground Truth
A person that riding horse in grass field.
Generation
A person that riding in field..................
Ground Truth
man riding horse in field.
Generation
A man riding horse in field......
Ground Truth
There are two birds standing on top of a building
Generation
There are two birds standing on a rock............................
Ground Truth
Two hawks sit on top of roof spire.
Generation
Two hawks sit on top of a wooden bench
Ground Truth
A young woman serving herself at cookout.
Generation
A young woman serving herself in kitchen.....
Ground Truth
2 bowls of fruit sit table.
Generation
2 bowls of fruit sit table..........
Ground Truth
A wooden bench written "CITY OF LONDON" at the park
Generation
A wooden bench written 'CTTY and tree............................
Table 10: Fine-tuning accuracy on dataset NL-RX-SYNTH, LoRA vs. full fine-tuning, both by CLLM loss and LLLM-JEPA loss (our method). Configuration is (r 2e 5, À 1, 1. Each cell runs five times. Average accuracy and standard deviation are reported. At every LoRA rank, LLLM-JEPA (ours) has better accuracy. At LoRA rank 512 (22.59% trainable parameters), LLLM (ours) achieves same accuracy as full fine-tuning, but CLLM still has a significant gap from full fine-tuning.
LoRA Rank
Method
Accuracy (%) f
LLLM
6.09 + 0.55
32
LLLMM JEPA (ours)
7.45 1.87
LLIM
21.09 + 1.90
64
-LLM-JEPA (ours)
32.46 + 1.26
LLAMM
34.21 1 2.82
128
CLLM (ours)
48.45 + 3.66
LLLM
45.57 + 4.52
256
LLLM-JETA (ours)
60.80 + 2.31
LLLM
50.18 + 5.15
512
LLIM-JEPA (ours)
72.41 +: 2.94
LLLM
57.29 + 5.32
Full
LLIMMJEPAA (ours)
70.42 1 2.36
A APPENDIX
A.1 FASTER LORA CONVERGENCE
Table TO demonstrates that LoRA fine-tuning with LLIM-JEPA loss not only achieves substantially higher accuracy than using LLLM alone, but also converges more quickly. Notably, LoRA rank of 512, our method already reaches accuracy comparable to full fine-tuning, whereas LoRA with only LLLM still exhibits clear performance gap.
15

---

Published as a conference paper at ICLR 2026
Table 11: Pretraining + fine-tuning Llama-3. 2ama-3.2-1B 1--InBtructtttt accuracy on pretraining dataset cestwc/paraphrase and fine tine-tuiin dataset Rotten Tomatoes and Yelp by Next Token Prediction (CLLM) loss loss vs. LLLM- loss (our method). Note that LLLM-JEPA is applied only at pretraining. We tune (rpre and (r by LLLM. and stick them in L.LMJJPPA pretraining. We run pretraining 5 times, and for each pretrained model, we run fine-tuning 5 times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed T-Tsst.
FT Dataset
Method
Accuracy (%) f
p-value
Config
LLLM
56.37 + 1.66
Irpre 8e - 5, br 4e
Rotten Tomatocs
7.38e
LLLM-JER JEPA (ours)
57.76 + 1.33
A= 0.5, k 2, same pr lrse
LLLM
26.46 0.92
Irpre Se 55,1rs 8e
Yelp
1.00e
LLIM-JRR (ours)
27.15 + 0.93
A 0.5, k = 2, same pre br
Code
Text Code
(a) Base model: No fine-tuning
(b) Baseline: Fine-tuned by NTP loss
Code
Code
(c) LLM-JEPA (Ours) k: =
(d) LLM-JEPA (Ours) & =
Figure 1-SNE plot of fTert and Code representations (a) Base mode without fine-tuning, (b) Baseline that
fine-tuned with NTP loss, (c) LLM-JEPA (ours) with = 0, and (d) LLM-JEPA (ours) with k = 1. Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model.
A.2 LLM-JEPA INDUCES STRUCTURED REPRESENTATION
We present additional t-SNE plots of Text and Code representations in fig. 6 which show that different values of yield similar structural patterns. In contrast, standard fine-tuning appears to further disrupt the representation structure compared the baseline model.
16

---

Published as a conference paper at ICLR 2026
Table 12: Fine-tuming accuracy on dataset SARRT-TTTTHHT -LLM-JEPA JEPA loss (ours) over various 7/X. Configuration is = 2e 5, À 1, O. We maintain max(";, A) 1.0 to use fixed (r. Each cell runs five times. Average accuracy and standard deviation are reported. When 7 = 0.0, it generate only empty output.
7/A
Config
Accuracy (%)
0.0
0.0, À 1.0
0.00 + 0.00
0.01
7 0.01, 1.0
1.38 0.06
0.1
7 0.1, 1.0
45.80 + 5.04
1.0
7 1.0, 1.0
70.42 2.36
10.0
7 1.0, À 0.1
67.52 + 1.45
100.0
7 = 1.0, À = 0.01
66.83 + 3.89
00
7 L.0, 0.0
57.29 1 5.32
k=
2=
k=
0
k-
k-2
k-3
k k-4 4
0.5
35.68%
36.12%
36.32%
36%13%
36.36%
À. - 0.5
49.43%
49.52%
49.55%
49.61%
49.18%
x= 1.0
36.03%
33.83%
33.01%
33.86%
36.15%
à. 1.0
48.13%
48.54%
49.00%
50.55%
50.13%
2.
31.11%
33.54%
24.64%
17.27%
34.13%
46.95%
47.20%
48.79%
47.41%
48.50%
(a) Llama on GSM8K, Ir
2e
(b) Ilama on Spider,
1e -
4 0
k 1
k 2
k 3
A-
k= 0
k= 1
k= 2
k= 3
k = 4
7. - 0.5
34.87%
34.87%
10%
38.57%
35.25%
À 1.0
14.18%
10.52%
13.53%
12.96%
17.92%
2.
34.87%
35.60%
36.32%
37.28%
38.36%
A 2.0
12.87%
12.00%
19.70%
19.33%
13.72%
À. = 2.0
32.69%
34.49%
37.50%
41.70%
43.12%
A 4.0
15.63%
13.11%
18.67%
25. 25.40%
16.63%
(c) Gemma SYNTH, Ir
le 5
(d) OpenELM on SYNTH, br
8e
==0 0
k=
k= 2
k= 3
A= 4
k-0 0
k
k 2
k - 3
k 4
7- 1.0
87.43%
87.38
87.30%
83.40%
83.53%
2. 0.5
57.69%
57.62%
60.26%
60.15%
60.47%
87.52%
87.14%
87.33%
82.64%
78.17%
2 1.0
57.65%
58.43%
99.92%
59.78%
59.74%
A= 4.0
87.12%
87.10%
87.30%
87.27%
87.36%
2 = 2.0
57.48%
56.64%
60.03%
60.59%
60.14%
(e) OLMo SYNTH, Ir 8c
(I) Llama SYNTH, Pretrain, Ir Sc
Figure 7: general we didn't find any pattern on where the best accuracy could appear. It could be at either high-end or low-end of cither À or k. Furthermore, there can be dips and spikes in random locations. Nonetheless, adjacent cells have close accuracy most of times, and sweeping (k. X) {0. 1, 3, 1) x {0.5, 1, 4] normally yield satisfialle results. Each cell is average of five runs, epoch
A.3 ABLATION STUDY ON THE ROLE OF LLLM
One limitation of eq. 2 is that the contribution of LLLM cannot be effectively reduced to 0. To address this, we introduce an additional hyperparameter r to explicitly control its relative strength:
LIIM-JEPA =
LLAM (Textree,,, Text:) +A x (encc(ex)))T abstraction capabilities
2=2
generative capabilities
We vary the ratio 7/A within [0, 1] while enforcing max(7, X) 1 to maintain constant learning rate. Table [12 shows that LLLM remains essential for generative performance: when 7 O, the fine-tuned model produces only empty outputs. This indicates that the JEPA component primarily serves regularization term, complementing the generative loss.
A.4 ADDITIONAL GENERATION EXAMPLES
Table 13 presents additional examples generated by fine-tuning -lamm--3.2-13 1.2--II-InnItt RX-SYNTH dataset using LLLM and -ELA-JEPA. respectively.
Instruct on the NL-
17

---

Published as a conference paper at ICLR 2026
Table 13:
regular expressions generated by 1iama-3.2-BB-Insttee after fine-tuning with CLLM loss and LLLM-JEPA loss (ours). Color code:
wrong
extra
missing
Ground Truth
CLLM
LIAM (ours)
lines ending with vowel or starting with character
([AEIOUaciou] "|A-7a-z].*>
([AEIOUaciou), "[A-7a-z].") ++++
([AEIOUaciou) (AANOUuaiou)MAAA7aa/a. "|AA-7a-7)))))
ines containing either lower-case letter, vowel, letter
(.9) ([AEIOUaciou)) (()()))) (.*) ([AEIOUaeiou) ((.)(.)))
lines starting with the string 'dog' before vowel
((AA-2a-1)7.).Mdddo)) lines not containing a letter and the string 'dog'
((A--a--))7,)). "(dog).*
((IA-za-2)).))
(([A->))+
lines with a character before vowel and the string 'dog', zero or more times "())&((--)))<ddog..
(()&(((-))){(ddg)).
lines with vowel at least once before not character
((A-72>>2)))<<<<000)1)..>
([AA-7--)))))+<(((00))))))
A.5 OVERFITTING BEHAVIOR LORA TUN--G
We also conducted experiments examine whether LoRA fine-tuning with LLLM-EPP exhibits similar resistance to overfitting. As shown in fig. 8 accuracy under LLIM-JEPA generally continues improve with additional epochs, whereas fine-tuning with LLLM shows clear signs of overfitting. Notably, the standard deviation is much higher than in full fine-tuning, likely reflecting the lower capacity of LoRA fine-tuning. An interesting pattern emerges: for LLLM-JEPA larger standard deviations often coincide with dips in accuracy, whereas for LLLM they tend to accompany accuracy spikes. This suggests that such fluctuations may unreliable indicators of generalization quality.
80
72.4
68.9
70
64.2
Accuracy (56)
60
50
42.5
55.0
40
50.2
52.4
48.5 8
49.1
30
LLM JEPA (Ours)
34.6
Baseline
20
Epoch
Figure 8: LLM-JEPA resists overfitting in LoRA fine-tuning, Fine-tuning with resists overfitting. When fine-tuning with LLLM loss start overfit, LLLM-JEPA kept improving. However the trend is not as stable as in full fine-tuning tuning, possibly due to limited capacity of LoRA fine-tuning.
loss (our method)
A.6 STRUCTURED REPRESENTATIONS INDUCED BY LLM-JEPA
We also examine the representation space to better understand how LLM-JEPA regularizes learned fea- tures. Specifically, we plot t-SNE embeddings for both Text and Code across three settings: the base model, a model fine-tuned with LLLM, and a model fine-tuned with CLLM-JEPA- As shown in fig. 4 clear structure emerges after fine-tuning with LLLM-JEPA. We hypothesize that CLLM-JEPA enforces structure in the representation space by constraining the mapping from Enc within narrow subspace. If this is the case, the SVD decomposition of Enc Text) Enc( Enc(Code (ode) should yield significantly smaller singular values, which is confirmed in fig. 3 Furthermore, we hypothesize that the mapping is approximately linear. To test this, compute the least-squares
to Enn(Code)
18

---

Published as a conference paper at ICLR 2026
Table 14: Fine-tuning accuracy on dataset NL-RX-SYNTH by Next Token Prediction (CLLM) loss vs. LLLM-JEPA loss (our method). Each cell is the best possible accuracy over a set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed TeTst
Model
Method
Accuracy (%) 1
y-value +
Config
LLLM
57.29 5.32
Ir 2e
1atma-3,2-1B-Inttree
1.0e 3
LII.M JJPP (ours)
71.46 + 1.34
A = 1, = 1, same b
LLLM
33.65 3.24
Ir le
gemma 2-2bbit
5.5e 3
CLLM JBPA (ours)
43.12 + 2.61
À = 2, 4. same br
LLLM
12.07 + 1.81
Ir Se -
OpenBLM- BeanBLM-LLIB-nstrruu
5.1e
LLLM- LJPA (ours)
25.40 + 2.40
X = 4, 3, same br
LLEM
87.09 + 0.36
Ir 8e - 5
OLMo-2-0425-11 IB-Istrutt
2.5e
LEPA (ours)
87.52 0.29
À 2, same Ir
Table 15:
Fine-tuning accuracy hy model 1ammm-3.2-IB-Intrruu LLLM loss vs. LLLM-JEPA loss (our method). Each cell is the best possible accuracy over set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed f-Test.
Dataset
Method
Accuracy (%) f
p-value &
Config
LLLM
22.49 1.91
2e 5
NL-RX-TURK
2.4e 4
CLLM (ours)
30.94 = 1.13
X = 1, 1, same br
LLLM
32.36 = 0.58
= 2e 5
GSM8K
9.6e
LLLM- JEPA (ours)
36.36 0.20
À
0.5, k 4, same br
LLLM
17.52 2.14
Ir = 1e 5
Spider
1.0e
LLLM-JEPA (ours)
50.55 = 2.08
1, k: same
regression error, and table suppoors this hypothesis. Together, these results suggest that LLM-JEPA promotes a near-linear transformation between Text and Code representations, which may underlie its accuracy improvements.
A.7 PERFORMANCE ACROSS MODEL SIZES
We also evaluate LLM-JEPA across different model sizes. As shown in table 17 we observe statistically significant improvements at all scales. Since there is no official 8B version of Llama-3.2, we instead use Llama-3. 3........111 1-8B-Instrue in properly terminating regular expressions. To address this, we additionally evaluate using sta rt switt t criterion -tha is, a prediction is considered correct if the generated regular expression begins with the ground-truth expression, removing the need for exact termination. Under this metric, we again observe statistically significant accuracy improvements.
where performance collapsed due to the model's difficulty
A.8 EFFICIENT HYPERPARAMETER SEARCH
We found that the optimal & values consistently cluster around 0-1 or 3-4, and moreover, when the optimum does not occur l 0 or l 3, the optimal À is always the same À that performs best at those two anchor points. This yields an efficient search strategy:
Table 16: LLM-JEPA is almost linear transformation from Enc(Text) to (Cod)).
minx
Eno(Text) Enc(Code)
Avg. Top 100 Singular
Base model
3953.11
310.73
LLLMM LLM-JEPA (Ours) i =
3035.01
341.80
4.47
94.84
LLM-JEPA (Ours) 0
4.04
16.82
19

---

Published as a conference paper at ICLR 2026
Table 17: Fine-tuning accuracy on NL-RX-SYNTH by Next Token Prediction (CLLM) loss vs. LLLM- JEPA loss (our method). Each case rums five times. Average accuracy and standard deviation are reported. We also report p-value of paired, single-tailed T-Tss. Note that Llama does not have official 3,2-8B, and we have to use 3. 3.1-8B, which has lower accuracy. Still LLM-JEPA sees significant improvement. We also evaluated on OLMo-2-7B.
Model
Method
Accuracy (%) f
p-value
Config
LLLM
57.29 5.32
Ir 2e 5
Llama-3.2-IB-Insteee
1.0c
-1.M- (ours)
71,46 + 1.34
À
1, 1, same Ir
LLLM
74.55 3.58
2r -
Llama-3.2-3 3.--uut-ntttt
0.0352
LLLM JEPA (ours)
77.16 + 3.66
À 2, same tr
LLLM
35.77 + 6.60
Ir = 2e 5
Llama-3. 3.1--8.-B1BBt-trt
0.0131
LLLM JEPA (ours)
63.57 + 16.81
À 2.0, k: same
LLEM
87.26 1 0.27
1r = 2c 5
OLMo-2-1 124--2-Insttuue
0.0845
GLLM-JERS (ours)
87.75 0.33
= 20, = 2, same Ir
k=
k=
k= 2
k=3
k 4
A 1/1024
20.25%
20.11%
20.33%
20.24%
20.34%
2 0.5
19.90%
19.64%
19.73%
20.01%
20.06%
2 1.0
19.92%
19.40%
19.72%
19.82%
19.99%
À 2.0
20.06%
20.21%
19.86%
19.75%
19.67%
4.0
20.26%
20.35%
19.79%
19.84%
19.83%
2
20.12%
20.55%
20.00%
20.34%
20.12%
À- 16.0
20.60%
20.19%
20%79%
20.71%
20.55%
32.0
20.18%
20. 93%
20. 10%
20.29%
20.50%
64.0
20.71%
20. 80.8%
19.80%
20.53%
20.36%
h. 128.0
20.79%
21. 9..4%
20.37%
20.84%
20.61%
2 1024.0
21. 29%
29%
20. 90.99%
21.42%
20.46%
Figure 9: Fine-tuning HellaSwag with Llama-3.2-18 allows À to be scaled up to 1024, with performance continuing to improve.
Evaluate all À values at k (0,3) and identify the best (A, k).
From the best (A, k), iteratively evaluate (A, h 1) until no further improvement is observed.
Under this scheme, the search cost is reduced from N M to 2N + O(1), where N is the number of À values and M is the number of & values, We verified that this procedure reliably recovers the optimal (X, across all experiments reported in the paper.
A.9 EXTREMELY HIGH JEPA LOss DROPOUT RATES
We conducted additional experiments to study the effect JEPA loss dropout rates. At extremely high dropout (1 0.0625). we observed that varying À does not yield meaningful accuracy improvements. This suggests a simplified search strategy: keep fixed. Using this approach, we found that accuracy remains stable even at dropout rate of 1 - 125, better than previously identified 0.25 (table 18)...
Table 18: Varying À does yield meaningful accuracy improvements dropout 1 0.0625.
1 0.5, A 1 0.25,
0.125, À = 0.0625,
0.0625, 2 0.0625, À 0.5
73.42 1.00 73.31 + 0.76
73.20 + 0.46
71.53
1.27
70.68
1.06
70.63 -: 1.17
20

---

Published as a conference paper at ICLR 2026
Table 19: Representation alignment leads stronger extrapolation behavior.
Config
L.M-JEPA
Base Model
Regular Fine-tuning
Cosine similarity
0.995 + 0.002
0.914
0.698 + 0.049
Accuracy
95.56 + 6.09
0
88.89 + 0.0
A.10 REPRESENTATION ALIGNMENT ANALYSTS
Figure@sshove that LLM-JEPA preserves and even improves the alignment of text representations. whereas standard fine-tuning disrupts this alignment. Intuitively, a well-aligned representation space should facilitate extrapolation and improve generalization. To test this hypothesis, constructed a controlled dataset that maps Text of the form " "lines with a number repeated k or more t times" the corresponding Code (regular expression) ([O-9]] 91) k, for k [1,9].
We measured the minimum pairwise cosine similarity among the representations. LLM-JEPA col- lapses these representations almost onto single line (min cosine similarity = 0.995), outperforming both the base model and the regularly fine-tuned model. Moreover, the regularly fine-tuned model fails to generalize to the unseen case generalize correctly. This supports the intuition that improved representation alignment leads to stronger extrapolation behavior (table 19). Note that "1, is equivalent to "+", " but the regularly fine-tuned model failed both.
(absent in the training data), whereas L.LM-EPAA can
A.11 22-NORM
The number of noomm loss in table 3 is correct. Note that the magnitude of the fn-norm loss is equivalent to the MSE loss multiplied by the embedding dimension N and for morden LLMs, N is very large. In table MSE performs reasonably well, but when switching to 4--norm the effective scale of the loss is amplified by factor of N. This dramatically increases the magnitude of the JEPA term, which can destabilize optimization and lead to collapse. This explains the severe drop in accuracy when using normm..
21


Reviews:
[{"review_id": "3tyHy3HKxB", "reviewer": "Reviewer_j7Db", "summary": "This paper proposes a novel training objective $\\mathcal{L}_{\\mathrm{LLM-JEPA}}$ for LLM training, aiming at introducing the successful JEPA insights, i.e., directly predicting from embedding space to absorb abstraction knowledge, from vision into the LLM studies, which mainly rely on the reconstruction ability on token level. LLM-JEPA recognizes the naturally paired data (e.g., text & code or Q & A in this paper) as different views of the same thing. The objective is to predict from the embeddings of one view to the embeddings of another view. Experiments show the performances of LLM-JEPA on both the reconstruction ability and predictability.", "questions": "- For the predictor, you append $k$ `[PRED]` tokens to reuse the model’s weights. Can you elaborate more on the design intuition of this setting, especially when $k>0$? Why sometimes a larger $k$ would be better?\n- In Table 3, I find that InfoNCE surprisingly performs much worse than metrics. Do you have some explanations on this? Does it just mean the predictability is more crucial than the contrastiveness in this task?\n- In this paper, both the text and code encoders are initiated with a same LLM with same weights. In prior practice in JPEA [1], the target encoder is an EMA of the context encoder weights, which can enhance the training stability. Can the EMA design be applied to LLM-JPEA?\n\n## References\n\n[1] Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture. CVPR 2023.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of introducing JEPA from vision to language is novel. Compared to the representation learning in LLM, which mainly focuses on obtaining good embeddings, LLM-JEPA doesn’t neglect its generative capabilities, and experiments show that the additional JPEA loss helps both pretraining and finetuning.\n- I like the analysis in the experiment part, including showing that next-token prediction loss cannot optimize the JPEA objective and visualizing the t-SNE and the singular value to verify the strengths of LLM-JEPA.", "weaknesses": "- Though efficient, I doubt the usage of this method, especially as its strong dependency on multi-view data, which may limit the application to real-world industrial LLMs. Do you have some blueprints of the future work for more universal scenarios (e.g., unsupervised pretraining via data augmentation on texts that don’t have paired views)?\n- Another weakness, based on my understanding, is the additional training complexity and hyperparameter search cost. From Fig. 7 in appendix, there is no clear law of the best $(\\lambda, k)$ combination, which is also mentioned in the limitation part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sqPyaVZSl1", "reviewer": "Reviewer_qoq5", "summary": "This paper proposes LLM-JEPA, a method that augments standard autoregressive LLM training (next-token prediction) with a JEPA-style embedding-space objective that forces embeddings of two views to align via a predictor network. The JEPA term is added to the generative cross-entropy loss with a balancing weight λ, and the predictor is implemented by appending small numbers of special [PRED] tokens that reuse model weights. The authors implement a custom attention mask to obtain separate view embeddings efficiently, introduce random JEPA-loss dropout to reduce compute overhead, and provide a thorough empirical study across multiple model families (Llama3, Gemma2, OpenELM, OLMo). They show consistent accuracy gains, representation improvements (t-SNE, SVD, near-linear mapping from text to code embeddings), and robustness to overfitting; they also provide ablations on loss choices, predictor placement, and loss dropout.", "questions": "1. The paper reports grid search results and suggests keeping λ·(1−α) constant with dropout, but could you provide concrete heuristics or automated tuning strategies? How sensitive are gains to λ when moving to very different view types or sequence lengths?\n2. The method uses the last-token hidden state as the embedding and appends [PRED] tokens as a tied predictor. Have you tried averaging representations, CLS-like tokens, multilayer predictors, or decoupled predictors? How robust are the results to these choices?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The formulation (LLLM + λ · JEPA) and practical predictor design (tied weights via [PRED] tokens) are simple, elegant, and directly applicable to existing transformer models without architectural surgery.\n2. Results include full fine-tuning and LoRA, pretraining experiments, several datasets (including code and QA tasks), multiple model sizes, statistical significance testing (five seeds), and useful visualizations (t-SNE, SVD) that support the claimed representational benefits.", "weaknesses": "1. Most experiments are fine-tuning or pretraining on relatively small/targeted corpora. While dropout amortizes cost, it remains unclear how LLM-JEPA behaves (performance, wall-clock, memory) at large pretraining scales used for modern foundation models.\n2. The paper presents empirical evidence (SVD, near-linear mapping), but lacks deeper theoretical analysis or causal ablations that would clarify when and why the embedding alignment improves downstream generation and reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SMZg6W2FL", "reviewer": "Reviewer_waki", "summary": "This paper proposes LLM-JEPA, a novel training objective that brings Joint Embedding Predictive Architectures (JEPA) into large language models. By combining the standard autoregressive next-token prediction loss with a JEPA-style embedding prediction loss, the model improves representation quality while maintaining generative capability.", "questions": "1. Could JEPA be extended to pure text tasks using data augmentation (e.g., paraphrase or Q–A pairs)?\n\n2. How sensitive is the method to the choice of distance metric (cosine vs. L2)?\n\n3. During large-scale pretraining, how do you plan to manage compute cost — would loss dropout scale effectively to billions of tokens?\n\n4. Did you observe any degradation in generation diversity or fluency due to the embedding-space constraint?\n\n5. Can the JEPA loss be interpreted as encouraging semantic disentanglement or latent structure emergence?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work systematically introduces JEPA into LLM training for the first time, bridging a long-standing gap between vision and language representation learning.\n\n2. Demonstrates consistent empirical improvements across diverse architectures, scales, and datasets.\n\n3. Provides clear analysis of embedding structure (t-SNE, SVD), showing how JEPA regularizes the representation space.\n\n4. Maintains generation ability while enhancing abstraction and generalization.", "weaknesses": "1. Training cost is substantially higher due to multiple forward passes (≈2× compute), as the authors acknowledge.\n\n2. Hyperparameter tuning (λ, k) appears expensive and unstable across settings.\n\n3. The approach depends on multi-view data (e.g., text–code pairs), limiting applicability to generic text-only corpora.\n\n4. The paper could discuss more about how JEPA objectives affect downstream reasoning or interpretability beyond accuracy metrics.\n\n5. It would be interesting to see whether JEPA acts more like regularization or representation alignment—the current discussion is mostly empirical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PjWbjqDvDM", "reviewer": "Reviewer_iw7d", "summary": "This paper, LLM-JEPA, brings the Joint-Embedding Predictive Architecture (JEPA) that was originally introduced in the ViT domain to LLMs (specifically, text-code generation LLMs). In the vision domain, JEPA self-supervisedly to learn to predict the representations of some target image patches, given some context image patches. The underlying assumption is that all patches represent the same thing. In this paper, LLM-JEPA self-supervisedly learn to predict the representations of the code, given prompt text, and the underlying assumption is that the prompt and the code represent the same thing. LLM-JEPA is done simply by adding additional predictor tokens and a prediction loss. The authors show that this method improves standard LLM metrics significantly without overfitting.", "questions": "-\tIn Eq.1, recommend noting somewhere what XEnt() stands for. Typically, people just write CrossEntropy. Similarly, does NTP stand for next token prediction? Please note it somewhere.\n-\tIn Line 161, is the [PRED] added after the concatenated text-code sequence or in between? (as mentioned above, a diagram would simplify things here).\n-\tIn Line 166, could the authors elaborate on what it means to use the embedding of the last predictor token? Is it the hidden_state? \n-\tLine 170 typo: “lies in obtained” -> “lies in obtaining”?\n-\tIn Line 190, could the authors explain which two forward passes? What are the purposes of each two?\n-\tIn Figure 3, the blue curve is when k=0, but in Line 166, the authors say in this case the predictor is trivial. Does being trivial mean JEPA is turned off, or mean it is most effective?\n-\tIn Figure 3, right side, we observe that even the NTP finetuning loss is almost the same at stabilization (yellow/blue), but at evaluation time their accuracy in fact differs a lot (51 vs 71). This is interesting. Do the authors have a hypothesis for the reason?\n-\tLine 292: Are the wrong/missing/extra labels compared against the GT or any valid regex? The proposed method’s output at line 292 has two additional `{}`.\n-\tLine 373: using L2 norm degrades accuracy from 71% to 2%, which is surprising how sensitive it is to the distance metric. Is this number correct?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tAlthough the JEPA idea is not new, bringing it to the LLM domain is novel.\n-\tThe authors acknowledge the underlying assumption of JEPA: they require multiple views of the *same* information as in text-code generation, but also go further to explore how it generalizes to QA tasks.\n-\tThe authors applied Loss Dropout and observed dual wins in saving compute and improving accuracy.\n-\tThe writing flows naturally (especially with the help of the natural questions), and is a pleasure to read. Having pseudo code, implementation (eg, Line 152) is a strong plus.", "weaknesses": "-\tLine 161: recommend adding a simple diagram to illustrate how the Pred() is added to an LLM transformer. The description is a bit confusing: beyond the standard transformer, is there a separate prediction network? Are the predictors also generated in k auto-regressive loops or one-shot?\n-\tRecommend showing a visualization of the mask in Line 188. Readers can imagine this looks like two upper-triangular matrices along the diagonal, but adding a diagram makes it more intuitive.\n-\tTable 1 has no description in the main text. Does it intend to show failure cases?\n-\t(Minor) The citation style does not follow the ICLR 2026 template https://iclr.cc/Conferences/2026/AuthorGuide (i.e., xxx et al.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)