Learning Effective Language Representations for Sequential Recommendation via Joint Embedding Predictive Architecture
Authors: Nguyen Anh Minh
Dung D. Le
Venue: COLM 2025 (2025)
Keywords: language models
joint embedding predictive architecture
sequential recommendation
TL;DR: JEPA4Rec improves sequential recommendation by combining joint embedding predictive architecture with language modeling, achieving better item representations and outperforming existing methods, especially in low-resource and cross-domain scenarios.

Abstract: Language representation learning has emerged as a promising approach for sequential recommendation, thanks to its ability to learn generalizable representations. However, despite its advantages, this approach still struggles with data sparsity and a limited understanding of common-sense user preferences. To address these limitations, we propose $\textbf{JEPA4Rec}$, a framework that combines $\textbf{J}$oint $\textbf{E}$mbedding $\textbf{P}$redictive $\textbf{A}$rchitecture with language modeling of item textual descriptions. JEPA4Rec captures semantically rich and transferable representations, improving recommendation performance and reducing reliance on large-scale pre-training data. Specifically, JEPA4Rec represents items as text sentences by flattening descriptive information such as $\textit{title, category}$, and other attributes. To encode these sentences, we employ a bidirectional Transformer encoder with modified embedding layers tailored for capturing item information in recommendation datasets. We apply masking to text sentences and use them to predict the representations of the unmasked sentences, helping the model learn generalizable item embeddings. To further improve recommendation performance and language understanding, we employ a two-stage training strategy incorporating self-supervised learning losses. Experiments on six real-world datasets demonstrate that JEPA4Rec consistently outperforms state-of-the-art methods, particularly in cross-domain, cross-platform, and low-resource scenarios.

Published as a conference paper at COLM 2025
Learning Effective Language Representations for Sequential Recommendation via Joint Embedding Predictive Architec- ture
Minh-Anh Nguyen, Dung D. Le
College of Engineering and Computer Science, VinUniversity, Vietnam [minh.na2, dung. ldnuniinnni.....mmmmmmmmmm edd..v
Abstract
Language representatinn learning has emerged as promising approach for sequential recommendation, thanks its ability to learn generalizable rep- resentations. However, despite its advantages, this approach still struggles with data sparsity and a limited understanding of common-sense user pref- erences. To address these limitations, propose JEPA4Rec, framework that combines Joint Embedding Predictive Architecture with language mod- eling of item textual descriptions. JEPA4Rec captures semantically rich and transferable representations, improving recommendation performance and reducing reliance large-scale pre-training data. Specifically, JEPA4Rec represents items as text sentences by flattening descriptive information such fitle, calegory, and other attributes. To encode these sentences, we utilize a bidirectional Transformer encoder with modified embedding layers specif- ically designed to capture item information in recommendation datasets. We apply masking to text sentences and use them to predict the represen- tations of the unmasked sentences, helping the model learn generalizable item embeddings. To further enhance recommendation performance and language understanding, we employ a two-stage training strategy that incorporates self-supervised learning losses. Experiments on six real-world datasets demonstrate that JEPA4Rec consistently outperforms state-of- the- art methods, particularly in cross-domain, cross- mross-platfomm, and low-resource scenarios.
1 Introduction
Sequential recommendation predicts the next item a user is likely to interact with based their past behavior. Traditional ID-based methods capture sequential patterns well but struggle with cold-start items and knowledge transfer across domains (Fang et al., 2020; Kang & McAuley, 2018; Sun et al., 2019). To address this issue, cross-domain methods leverage overlapping users or items (Tang et al., 2012; Zhu et al., 2021b), however, their real-world applicability is limited due the scarcity of shared data. Another approach utilizes modalities such as text or images, but the semantic gap between domains remains challenge (Yuan al., 2023). Pretrained language models (PLMs), trained on general data like Wikipedia (Devlin et al., 2019), often fail to align with item descriptions and generate embeddings at the sentence level, limiting their ability effectively model user preferences (Liu et al., 2023).
Our goal to leverage language representation learning for sequential recommendation while utilizing PLM knowledge. This involves three key challenges: (1) Creating flexible item text representation beyond simple attribute concatenation (Ding et al., 2021; Wang et al., 2024). (2) Learning both item sequences and common-sense user preferences for better gen- eralization. (3) Designing an efficient training strategy to bridge the text-recommendation semantic gap (Li et al., 2023b) while maintaining effectiveness in sparse data settings.
We leverage Joint Embedding Predictive Architecture (JEPA) (Assran et al., 2023; Abdelfattah & Alahi, 2024) to address these challenges. JEPA predicts abstract representations rather

---

Published as a conference paper at COLM 2025
than raw tokens, capturing meaningful semantics while avoiding low-level noise (LeCun, 2022). It consists of an encoder and a predictor: the encoder generates latent representations from context-target pairs, while the predictor learns to map context to target representations. Unlike contrastive learning (Jaiswal et al., 2020), JEPA eliminates negative samples and prevents representation collapse through asymmetric encoder design (Chen & He, 2021). Despite its success in vision tasks, JEPA remains unexplored in Natural Language Processing and recommendation (Gui et al., 2024).
Building upon this, we propose JEPA4Rec, framework that leverages JEPA for lan- guage representation learning in sequential recommendation. To enrich the semantic meaning of item representations, we trans- form item metadata (e.g., title, category, and description) into single text sentence (il- lustration in Figure 1). Effectively learning item sentence representations and captur- ing common sense in user preferences is
Tie Plo
Inne Heels)
Madtwos
Brandd
Apdle
Color
Sive
Tie Arpoda Pho
Baxi Applo
Color
Figure 1: We flatten metadata attributes and corresponding values to represent items
crucial for recommendations. Here, common sense refers to the model's ability to grasp inherent user behaviors, such as brand loyalty and category preferences, beyond explicit interaction patterns. This is where JEPA plays a central role, enabling structured and transferable representation learning. Our main contributions are as follows:
1. We construct bidirectional Transformer encoder with modified embedding layers tailored for encoding item sentences, which are then used as both the Context and Target Encoders. Additionally, we employ tokens masking strategy that selectively hides history item information at varying rates. This requires the model to reconstruct missing details from partial item information, allowing generalizable item representations and enhancing common-senss preference learning.
2. We adopt a two-stage training approach (pre-training and fine-tuning), leveraging self-supervised learning objectives tailored for both recommendation and language understanding tasks.
3. Extensive experiments real-world datasets demonstrate that JEPA4Rec consis- tently outperforms state-of-the-al methods, achieving significant improvements in recommendation performance across all datasets. Notably, JEPA4Rec requires only a fraction of the pre-training data typically used in previous studies, showcasing its efficiency in learning transferable, robust, and data-efficient item representations, particularly in cross-domain, cross-platform, and low-resource settings.
2 Related work
Sequential Recommendation aims to predict users' next interactions by modeling historical behaviors. Traditional methods, including RNNs (Hidasi et al., 2015; Li et al., 2017), CNNs (Tang & Wang, 2018), and Transformers (Sun et al., 2019; Assran et al., 2023), rely on item IDs, limiting transferability. A key limitation of this research direction is its reliance on discrete item IDs, which are inherently non non-transferable across domains and incapable of capturing the rich semantic information embedded in item metadata. In contrast, textual item representations offer more generalizable and semantically meaningful alternative. Approaches like (Hou et al., 2022; Geng et al., 2022) use PLMs to embed item descriptions, allowing knowledge to transfer across platforms and domains. However, these methods of- ten separate representation learning from user modeling, limiting their flexibility. JEPA addresses this gap by jointly learning from item metadata in natural language form, en- abling rich semantic understanding and transfer learning without reliance on shared item IDs or overlapping users (Zhu et al., 2021a; Tang et al., 2012). This makes it particularly suitable for cold-start or low-resource scenarios.
Transfer Learning for Recommendation addresses data sparsity by leveraging shared knowledge (Singh, 2020). PLMs generate universal item representations (Devlin et al., 2019; Geng et al., 2022) but require large-scale pre-training (Liu et al., 2023) and tightly couple text
2

---

Published as a conference paper at COLM 2025
Embbi
Sequence Nom Contrestive Loss
Contadl
Sidireciona Encoder
Pas Cun
Enileeding
Paesicao
Nous
fo Context
Heet
packejannin Zand Apps
Groud sruh Mixed domain
Nogallvvs
Masting
Hirry Bey moce
PPrddictor
Target
Fe Contest
Masieet I Langssage
Madesing
Badd
MASK
(a) Model Structure
(b) Pre-training Framework
Figure 2: Overview of JEPA4Rec. Left: The items are represented as text sentences. The model leverages the user's historical interactions along with partial text information to predict the full representations of the items. Specifically, the historical item sequence embedding hcL. is combined with the item position embedding D,, of item in and the embedding of the next item 1 when they are masked. These representations are then passed through the predictor reconstruct the complete representations of in and in+1, which are obtained from the Target Encoder. Right: JEPA4Rec includes the Context Encoder (fe), which encodes masked item sequences; the Target Encoder (fz), which processes the full item information and is updated via Exponential Moving Average (EMA) of the Context Encoder; and the Predictor (g0), which aims to recover the complete item representations from the partially observed inputs.
and item representations (Hou et al., 2023; 2022). Unlike contrastive loss, which encourages instance discrimination, or MLM, which focuses on local token-level semantics, JEPA4Rec introduces novel L2 Mapping Loss that directly aligns user behavior embeddings with item embeddings. This predictive alignment enables the model to reason in the embedding space rather than over discrete tokens, allowing it to capture user preferences more effectively. Through this architecture, JEPA4Rec can reconstruct complete item semantics from partial textual cues, such as fragments of product descriptions missing metadata. Furthermore, the model can learn representations that generalize across different domains, even when pre-training data is limited. This embedding-leved learning paradigm facilitates better user intent modeling, improves robustness to sparse input, and enhances cross-domain performance without requiring overlapping users shared item identifiers.
3 Methodology
In this section, we introduce JEPA4Rec, framework designed for efficient language repre- sentation learning in sequential recommendation. Building upon et al. (2023a), which em- ploys a bidirectional Transformer encoder for encoding item sentences, JEPA4Rec enhances common-sense learning in user preferences and improves generalizable item embeddings through the Joint Embedding Predictive Architecture (JEPA). As illustrated in Figure 2, JEPA4Rec incorporates two key innovations. First, its masking strategy transforms user history into text sequences and applies Masked Language Modeling (MLM). However, instead of solely learning at the token level, JEPA4Rec learns item representations the embedding space, detailed in Section 3.3.1. Second, its learning framework trains Predictor to reconstruct full item representations using history embeddings and masked item embeddings. The Target Encoder refines these representations, while sule-super
3

---

Published as a conference paper at COLM 2025
losses further enhance recommendation accuracy and language understanding, discussed in Section 3.3.2.
3.1 Problem Formulation
We address sequential recommendation with multi-domain interaction data for training or pre-training. A user's history in each domain forms sequence where each item has a unique ID and textual attributes (title, category, description). To preserve domain-specific semantics Hou et al. (2022), we keep sequences separate rather than merging them. Instead of item IDs, we represent items using textual metadata by flattening attributes (e.g,, Title, Brand) and their values (e.g,, il'hone, Apple). Each item is expressed as a sentence S, uq..,, hm where m varies based on available metadata (Figure 1). The model then learns user preferences from the sentence sequence s Su) and predicts the next item sentence 11
{i, .2, Inf/
3.2 Item representation
Model Inputs After flattening the text metadata attributes of an item, we obtain its cor- responding sentence representation Sj. In sequential recommendation, the most recently interacted items carry the most relevant information about a user's 's latest preferences (Liu et al., 2021). Consequently, we represent the interaction history as:
X = Sm.
Where [CLS] is special token used to generalize the sequence information. then fed into the encoders.
Encode Item Mechanism To encode item sentences, we follow previous works (Sun et al., 2019; Geng et al., 2022) and construct four types of embeddings. First, token embeddings A E R Xu represent the meaning of each word token, where V is the vocabulary size and d is the embedding dimension. Unlike ID-based approaches (Hua et al., 2023), JEPA4Rec uses text tokens, making the representation more flexible and domain--nddependent Second, token position embeddings B E Rd capture the position of each token in a sentence, helping the model learn word order. Third, token type embeddings CCI.s, Catributer CValue E Rd indicate whether a token belongs the [CLS] token, an attribute key, or an attribute value, which is crucial for handling repeated structures in item metadata. Finally, item position embeddings D E R encode the position of items in user's interaction sequence, where each item S; shares the same vector D, aligning all tokens within an item and facilitating sequence modeling. Details about the tokenizer can be found in Appendix B.
The input embedding for each word w in sequence dings and applying layer normalization (Ba et al., 2016):
obtained by summing four embed-
EnD LayerNorm (Aw + B + C Dw,)
The final input representation Ex consists of these embeddings for all tokens in X, including the special [CLSS token:
Ex E|css] Emy...,, Egy
Where / the maximum sequence length. To encode Ex, we use Longformer (Beltagy al., 2020), bidirectional Transformer optimized for long sequences. Similar to Longformer" document processing setup, the special token [CLS] has global attention, while other tokens rely local windowed attention. The model generates d-dimensional word representations as follows:
hcs.s, hap....... h Encoder (E(cis)), Ew. Ear
where each word representation hn E R Following standard language model practices, use hcts the sequence representation. To encode items, JEPA4Rec treats each item as a single-item sequence X = {[CLS), ,, S, and obtains its embedding h; from the sequence representation. We use the same embedding layer and Longformer encoder for both the
4

---

Published as a conference paper at COLM 2025
Context Encoder fa and Target Encoder far with the latter updated via exponential moving average for stable training and to prevent representation collapse (Chen & He, 2021).
3.3 Pre-training Framework
The goal of pre-training is to establish a strong parameter initialization for downstream tasks. Our approach integrates both language understanding and recommendation learning and enhances the model's ability to learn common sense in user preferences, improve item representation, and create generalizable item embeddings without relying on large-scale pre-training data. Therefore, JEPA4Rec is pre-trained using three key objectives: Masked Language Modeling (MLM), Mapping Representation, and Sequence-Item contrastive task.
3.3.1 Masking strategy
User history sequences, represented as sentence sequences S2,..,, Sn) are learned using the MLM approach, where text tokens are masked and predicted, but unlike traditional token-level learning, JEPA4Rec learns item embeddings in the representation space and extends MLM with a structured masking strategy.
Since we want the model to infer 4++1 based on the user's 's history and partial information from Sx+1, we mask a significant portion (50,, detailed ablation study is in Appendix G) of tokens in Sn+1 and train the model predict the full representation of fu+1- For historical sentences, excessive masking could degrade the History Sequence Embedding hcr.s, leading information loss. Therefore, we apply a 15% token masking rate, consistent with pre-trained language model studies. For both history sequences and the next item, we apply the same masking strategy BERT: (1) replacing tokens with [MASK] (80%), (2) replacing with random token (10%), and (3) keeping the original token unchanged (10%). Following this principle, after masking tokens both history item sentences and the next item, we utilize the history sequence embedding heLs to predict their full representations. However, since the number of masked tokens varies across user sequences, the number of masked items within each sequence also differs. To simplify this process, we sample one masked item per sequence and introduce learnable zero vector when no tokens are masked.
After applying this masking strategy, we obtain the full representation of one masked item from the user's history (e.g,, the n-th item in Figure 2) and the next item. These representations are then passed into the Target Encoder far where their embeddings are computed as:
h = (f(((C))),sn)) h = ([(([CL)),,.-i))
These encoded representations are then compared with their corresponding predictions generated by the Predictor to refine the learned item embeddings.
3.3.2 Learning Framework
JEPA4Rec integrates the Mapping Representation task into the training process enhance item representation learning. To achieve this, we propose a lightweight MLP--baed Pre- dictor, which utilizes the history sequence embedding hCLS along with the Item Position Embedding D,, which corresponds to masked history items. This design enables the model to predict the full representation of S; mitigating information loss in hers when encoding long sequences. The predicted representation is computed as:
h FFN, (beLs + Dr))
Furthermore, since hCLS must also infer the next item's representation based on partial information, we mask most tokens in Sn+1 to obtain h The combined information from heLS and h LM is then used to predict the full representation of S
6 11 FFN2(hcus + h (1M)
5

---

Published as a conference paper at COLM 2025
Here, the concatenation operator (+) is used to combine embeddings for simplicity. To help the model predict item representations accurately, we define the mapping loss and MLM loss as follows:
M Σ yi log (pi)
Lmap h 1 [hr+1 11 LMLM
Where p; represents the predicted probability of the masked token belonging to the i-th vocabulary word, and yj the corresponding ground truth label. The mapping loss enhances item representation learning, while the MLM loss bridges the semantic gap between the pretrained language model's knowledge and the textual information the recommendation dataset.
Another pre-training objective for JEPA4Rec is the sequence-item contrastive task (S-I), commonly used for next-item prediction (Hou et al., 2022; 2023). We treat the next items in the ground truth as positive instances, while negative instances are selected using in-batch negatives instead of negative sampling full softmax. In-batch negatives leverage ground- truth items from other sequences within the same batch, effectively serving negative instances from multiple domains when training on large datasets. This approach not only reduces computational costs since JEPA4Rec generates item embeddings dynamically rather than maintaining an item embedding table, but also enhances model generalization across domains.
siim(bcsshhanl/rr
L. 1 -log LieB esim(beLs.h))/
where sim is the cosine similarity score between 2 vectors; h is the representation of the ground truth next item; B is the ground truth item set in batch; and temperature parameter. At the pre-training stage, we use multi-task training strategy to jointly optimize JEPA4Rec:
CPT Ls + d1 LMLM + d2 mmap
Where A1 hyperparameters control the weight of the MLM and S-I task loss. The pre-trained model will be fine-tuned for new target domains.
3.4 Finetuning Framework
After pretraining, only the Context Encoder is used for fine-tuning in the target domain, with the learned user embedding encapsulating rich and transferable user preference information. We encode all item sentences to construct dynamic, learnable item matrix I, which enables probability computation for next-item prediction over the entire dataset:
Py (fn+1 hcis) = Softmax ha+1)
To reduce computational overhead, I is updated per epoch instead of every batch. For finetuning, we adopt the widely used cross-entropy loss and train the model with sequence- item contrastive learning task using fully softmax over the entire dataset based on cosine similarity between items:
LFT log
Eiel esim sicmhhhsshhi//
3.5 Discussion
We compare JEPA4Rec with prior sequential recommendation methods to highlight its key innovations and advantages. Traditional models such as SASRec (Kang & McAuley, 2018) and BERT4Rec (Sun et al., 2019) rely on trainable item ID embeddings, which are non-transferable across domains and struggle with cold-start problems due to data sparsity, Context-enhanced models like UniSRec (Hou et al., 2022), S3-Rec (Zhou et al., 2020), and VQ-Rec (Hou et al., 2023) attempt to address this by extracting item features from pretrained language models and injecting them into standalone sequential architectures. However,

---

Published as a conference paper at COLM 2025
these approaches typically separate textual understanding from sequential modeling and rely heavily on contrastive objectives or side information fusion.
JEPA4Rec addresses these limitations by leveraging JEPA to learn item-level representations directly in the embedding space, rather than the token level. This enables the model to produce semantically rich and transferable embeddings that better capture user preferences and generalize across domains. Central to our approach is an information-ricl user repre- sentation, analogous to the [CLS] token in language models, learned through two auxiliary objectives: reconstructing historical item embeddings to preserve long-range preference signals, and recovering full item information from partially masked input to simulate real- world scenarios with incomplete metadata. As result, JEPA is well-suited for settings with sparse or partial item descriptions, offering improved adaptability new domains and cold-start items (see Appendix E for detailed experiments).
4 Experiments
4.1 Experimental Setup
Datasets To evaluate JEPA4Rec's per- formance, we conduct pre-training and fine-tuning using various Amazon re- view datasets (Hou et al., 2024). The dataset statistics after preprocessing are presented in Table 1. For pre-training, we utilize data from only three categories: Automotive, Grocery and Gourmet Food, and Movies and TV, accounting for approxi- mately 35% of the dataset size used in prior studies (Li et al., 2023a; Hou et al., 2022; 2023). These categories serve as the source domain datasets.
Datasets
#Users
#ltems
Alnters.
Avg. n
Density
Pre-training
115,778
158, 006
1,250, 489
10.70
6.8 10-5
Scientific
11, 040
5,327
76, 896
6.96
1.3 10-3
Instruments
27,530
10,611
231,312
8.40
7.9 10-
Arts
260
22, 855
492, 492
8.7
388 10-4
Office
914
20
Pet
47,569
37, 970
420,662
8.84
2.3 10-4
Online Retail
4,181
3, 3,896
401,
x
Table 1: Dataset statistics
For finetuning, we test JEPA4Rec on five Amazon categories (Scientific, Instruments, Crafts, Office, Pet Supplies) to assess cross-domain generalization. Additionally, we use the On- line Retail dataset), a UK e-commerce platform with no shared users, making it a more challenging cross-setting. Following Hou et al. (2022), keep five-core datasets, filter out users/ / items with fewer than five interactions. Item text representations are built from title, categories, and brand (Amazon) Description (Online Retail).
Baselines We compare JEPA4Rec against three categories of baseline models: (1) ID-only Methods: SASRec (Kang & McAuley, 2018), BERT4Rec (Sun et al., 2019); (2) ID-text Methods: S3-Rec (Zhou et al., 2020), LlamaRec; (3) Text-only Methods: UniSRec (Hou et al., 2022), VQ-Rec (Hou et al., 2023), RecFormer (Li et al., 2023a). Detailed descriptions of these models can be found in the Appendix C.
Evaluation Settings We use NDCG@10, Recall@10, and MRR as metrics, applying a leave- one-out strategy: the latest interaction for testing, the second latest for validation, and the rest for training. The ground-truth item ranked among all items, and average scores are reported. To ensure a fair comparison with RecFormer, the state-of-the-a method, we adopt the same experimental settings as RecFormer, detailed in Appendix D. Other baselines follow prior work settings.
4.2 Overall Performance
Table 2 presents comparative analysis of JEPA4Rec against baseline methods across six different datasets. Text-only methods consistently outperform ID-only and ID-text approaches on Amazon datasets. However, on the highly dense Online Retail dataset, ID- based methods remain effective. Notably, on Instruments, Arts, and Scientific datasets,

---

Published as a conference paper at COLM 2025
Scenario Dataset
Metric
SASRec
BERT4Ree
LlamaRec
S'-Rec
UniSRec
VQ-Rec RecFormer
JEPA4Rec
Improv
R@10
0.1305
0.1061
0.1180
0.0804
0.1235
0.1361
0.1684
0.1761
4.57%
Scientific
N810
0,0797
0.0790
0.0947
0.0451
0,0862
0,0843
0.1198
0.1282
7.01%
MRR
0.0696
0.0759
0.0856
0.0392
0.0786
0.0712
0.1071
0.1190
11.11%
0.0763
0.0933
0.1363
Pet
N810
0.0569
0.0602
0.0781
0.0742
0,0702
0,0761
0.1086
0.1210
11 41%1%%
MRR
0.0507
0.0585
0.0719
0.0710
0.0650
0.0697
0.0940
0.1157
23.09%
R@10
0.0995
0.0972
0.1034
0.1110
0,1119
0.1289
0.1279
0.1347
4.49%
Instruments
N810
0.0634
0.0707
0.0767
0.0797
0.0785
0.0812
0.1001
0.1057
5.59%
Cross- Domain
MRR
0,0577
0.0677
0.0689
0.0755
0.0740
0.0776
0.0958
0.1014
5.84%
R@10
0.1342
0.1236
0.1337
0.1399
0,1333
0,1298
0,1797
0.1920
6.84%
Arts
N@10
0.0848
0.0942
0,0938
0.1026
0.0894
0.0912
0.1249
0.1442
15.45%
MRR
0.0742
0.0899
0.0847
0.1057
0.0798
0.0878
0.1187
0.1341
12.97%
R@10
0.1196
0.1205
0. 0.1201
0.1186
0.1262
0.1336
0.1599
0.1676
7.51%
Office
N810
0,0832
0.0972
0.0864
0.0919
0.1011
0.1151
0.1276
10.86%
MRR
0.0751
0.0932
0.0817
0,0957
0.08488
0.0912
0.1094
0.1185
8.31%
R@10
0.2275
0.1384
0,2361
0.2218
0.2284
0.2301
0.2355
0.2429
3.14%
Cross-
Online
N@10
0.0928
0.0478
0.1061
0.0954
0.0912
0.0913
0.1249
0.1266
1.36%
Platform
Retail
MRR
0.0901
0.0332
0.0941
0,0858
0.0799
0.0865
0.0985
0.0985
Table Performance comparison of recommendation methods across different datasets.
Somee
Text
Met
0.10
JuMAdllee Rechormer
0.27
LiisSitee
0.20
MDCG@100
SASIlec
0.06
2016
0.04
0.02
0.22
Scienffic
Arts
Instruments Dataset
Office
Pett
Target Danaset
Figure 3: Performance (Recall@10) compari- son w.r.t different pre-training datasets. Full denotes result pre-training with 3 datasets and None denotes the training from scratch.
Figure 4: Performance (NDCG@10) compari- son of three text-only methods under the zero- shot setting.
text-based models achieve the highest performance, likely due to the rich descriptive metadata available for items.
JEPA4Rec outperforms all baseline models across datasets, except for the MRR metric on Online Retail. On average, it improves Recall@10 by 6.22% and NDCG@10 by 10.06%, demon- strating its effectiveness in recommendation tasks. The results highlight the advantages of JEPA4Rec's two-stage training strategy. The pre-training phase facilitates the learning of generalizable item representations, allowing the model to grasp common user prefer- ences. The finetuning phase further enhances adaptability to new domains, all items are represented through textual descriptions, enabling seamless transfer across different recommendation scenarios.
4.3 Efficient Learning Representation Performance
Universal Pre-training Figure 3 highlights the efficiency of JEPA4Rec's pre-training strategy. This figure demonstrates that JEPA4Rec pre-trained on three datasets outperforms models pre-trained on a single dataset and finetuned on the Scientific and Online Retail domains. Additionally, it surpasses the finetuned public checkpoint of Rechormer, which was pre- trained seven Amazon datasets. Pre-training on multiple datasets allows the model to initialize with well-learned weights, leading to improved adaptation during finetuning. Notably, JEPA4Rec achieves strong results despite utilizing only 35% of the pre-training data compared to state of the art model RecFormer, demonstrating the robustness and efficiency of its pre-training approach.
8

---

Published as a conference paper at COLM 2025
0.10
JEMAfec Racformer SASitec
JEPARRe
Rectormer
NDC0@10
0.08
NDCG@10
0.10
SASRec
0.06
0.04
0.06
0.02
0.04
176
5%s
10%
50%
100%
10%
50%
100%
Training Data Ratio (Instruments)
Training Data Ratio (Scientific)
Figure 5: NDCG@10 comparison between models over different sizes training data
Variants
Scientilic
Online Retail Recall@10
NDCG@10
Recall@10
MRR
NDCG@10
MRR
(0) JEPA4Rec
0.1282
0.1761
0.1190
0.1266
0.2429
0.0985
(1) w/o MLM loss
0.1170
0.1653
0.1128
0.1118
0.2216
0.0858
w/o pre-training
0.0935
0.1365
0.0855
0.1198
0.2185
0.0965
33) w/o token type emb.
0.1251
0.1749
0.1072
0.1231
0.1755
0.0915
Table 3: Performance comparison across model variants
Zero-shot To ensure fair zero-shot evaluation, we re-trained RecFormer on the same three datasets as JEPA while keeping all hyperparameters identical. Since SASRec is ID-only and unsuitable for zero-shot settings, we trained it fully supervised on each target domain for comparison. Figure 4 demonstrates that JEPA4Rec outperforms other text-only models by 1- 5% across all five datasets. This shows that JEPA4Rec learns common-sense user preferences better than RecFormer and UniSRec. Notably, on Scientific, JEPA4Rec surpasses SASRec's fully supervised performance zero-shot settings, highlighting the power of language-based recommendation.
Low-Resource Training We compare ID-based models (SASRec, LlamaRec) with text-only models (RecFormer, JEPAARec) across varying training data ratios on both the Instruments and Scientific datasets. As shown in Figure 5, text-only models consistently outperform ID-based ones, particularly under low-resource conditions (e.g., 1% or 5% of the data). This performance gap is attributed to the ability of language-based models to utilize semantic item descriptions during pre-training, enabling better generalization to unseen items. In contrast, ID-based models assign random embeddings to new items, making it difficult to generate meaningful recommendations with sparse data. As the training data increases, ID- based models such as SASRec and LlamaRec show noticeable improvements, reducing the performance gap. Nevertheless, JEPA4Rec remains the top-performing model across all data scales, highlighting the robustness of contrastive language modeling for recommendation.
4.4 Ablation Study
4.4.1
Study of JEPA4Rec
We perform an ablation study on Scientific (cross-domain) and Online Retail (cross-platform) datasets to assess JEPA4Rec's key components, with detailed results in Table 3. (1) w/o MLM loss: Removing Masked Language Modeling (MLM) reduces performance across all metrics, highlighting its role bridging the semantic gap between Longformer's pre-trained knowledge and the recommendation dataset; (2) w / pre-training: Performance drops significantly, emphasizing the necessity of pre-training for generalizable item embeddings and common-sense has little impact on Scientific, it significantly lowers Recall@10 on Online Retail, showing its importance in distinguishing patterns within item sentences.
preference learning; (3) w/o token type embeddings: While it

---

Published as a conference paper at COLM 2025
010
090
0.08
V400010
10008300
607
1.00
667
126
2Wh
256
(a) Scientific
(b) Office
Figure 6: Effect of Masking Ratio on Performance
4.4.2 Masking Ratio Analysis
To investigate the effect of varying masking ratios on next-item prediction performance, we conducted a comprehensive ablation study by masking different proportions of tokens in next-item sentences during pre-training. Specifically, from scratch on both the Scientific dataset (representing small-scale data) and the Office dataset (representing large-scale data). The results, averaged over five runs, are presented in Figure 6. We observe a consistent pattern across both datasets: the NDCG@10 score improves as the masking ratio increases from 10% to 50%, and then degrades when further increased to 80%. This trend suggests that moderate masking (around 50%) provides the most effective learning signal for the model, balancing information retention with learning generalizable representations.
trained our JEPA4Rec model
4.4.3 Additional Experimental Results.
Further experiments are included in Appendix G, covering case studies, robustness analysis, and additional ablation. First, we examine JEPA4Rec's zero-shot capability and adaptability to partial item information. Case studies show that JEPA4Rec can accurately rank relevant items with minimal text cues, significantly outperforming RecFormer. We also provide extended zero-shot evaluations across six datasets, where JEP'AReec remains competitive despite being pre-trained on fewer domains. Lastly, ablation studies on token embedding components and loss functions demonstrate the importance of MLM loss, token structure embeddings, and contrastive learning, each contributing significantly to model effectiveness.
5 Conclusion
In this work, we propose JEPA4Rec, novel framework that integrates Joint Embedding Predictive Architecture (JEPA) with language modeling to enhance sequential recommen- dation. By representing items as text sentences and leveraging bidirectional Transformer encoder, JEPA4Rec learns semantically rich and transferable item representations while improving common-sense user preference modeling. Our framework incorporates novel masking strategy and two-stage training approach enhance recommendation accuracy and adaptability across domains. Extensive experiments on six real-world datasets demon- strate that JEPA4Rec outperforms state-of-the-art methods, particularly in cross-domain, cross-platform, and low-resoure scenarios, while requiring significantly less pre-training data. These results highlight the effectiveness and efficiency of our approach in learning generalizable and robust item representations for sequential recommendation.
10

---

Published as a conference paper at COLM 2025
References
Mohamed Abdelfattah and Alexandre Alahi. S-jepa: A joint embedding predictive architec- ture for skeletal action recognition. European Conference on Computer Vision, pp. 367-384. Springer, 2024.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint ar arXio:1607.06450, 2016.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. ar Xiv preprint ar Rtt::2004.05150,, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp. 15750-15758, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and shorf papers), pp. 4171-4186 2019.
Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. Zero-shot recommender systems. ar Xiv preprint 2rXt::2005.88318 2021.
Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Transactions on Information Systems (TOIS), 39(1):1-422 2020.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yinggiang Ge, and Yongfeng Zhang. Recommen- dation language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM conference on recommender systems, pp. 299-315, 2022.
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey self-supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session- based recommendations with recurrent neural networks. arx preprint arXiv:1511.06939, 2015.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In Pro- ceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp- 585-593, 2022.
Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. Proceedings of the ACM Web Conference 2023, pp. 1162-1171, 2023.
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation. Xio preprint ar 20X3:22403.0395, 2024.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recommendation foundation models. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pp. 195-204, 2023.
11

---

Published as a conference paper at COLM 2025
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1:22 2020.
Wang-Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEEE, 2018.
Yann LeCun. path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.
Open Review, 62(1):1-66 62, 2022.
Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. Text is all you need: Learning language representations for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Pp. 1258-1267, 2023a.
Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp. 1419-1428, 2017.
Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights. ar Xiv preprint ar Xriv:22355.11 11700, 2023b.
Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang. Is chatgpt good recommender? preliminary study. Xiv preprint ar Xrv::22304 10149, 2023.
Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S Yu. Augmenting sequential recommendation with pseudo-prior items via reversely pre-training transformer. In Proceedings of the 44th international ACM SIGIR conference on Research and development in information retrieval, pp. 1608-1612, 2021.
Monika Singh. Scalability and sparsity issues in recommender datasets: a survey. Knowledge and Information Systemis, 62(1):1-43, 2020.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Se- quential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441-1450, 2019.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 565-573, 2018.
Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. Cross-domain collaboration recommendation. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1285-1293, 2012.
Shuhan Wang, Bin Shen, Xu Min, Yong He, Xiaolu Zhang, Liang Zhang, Jun Zhou, and Linjian Mo. Aligned side information fusion method for sequential recommendation. In Companion Proceedings of the ACM Web Conference 2024, pp. 112-120, 2024.
Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? id-vs. modality-based recom- mender models revisited. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2639-2649, 2023.
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ri-Rong Wen. S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. conference information & knowledge management, pp. 1893-1902, 2020.
Proceedings of the 29th ACM international
Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. Cross-domain recommendation: challenges, progress, and prospects. arXiv preprint arXit=2103.01666, 2021a.
12

---

Published as a conference paper at COLM 2025
Yongchun Zhu, Kaikai Ge, Fuzhen Zhuang, Ruobing Xie, Dongbo Xi, Xu Zhang, Leyu Lin, and Qing He. Fransfer-meta framework for cross-domain recommendation cold- start users. In Proceedings of the 44th international ACM SIGIR conference research and development in information retrieval, pp. 1813-1817, 2021b.
A Joint Embedding Predictive Architecture
JEPA's key advantage lies in predicting ab- stract representations instead of raw pixel or token space, allowing the model to fo- cus on meaningful semantic features rather than low-level details (LeCun, 2022). As shown in Figure its architecture con- sists of an encoder fe(-) to compute in- put representations and a predictor (+(+) to estimate the representation of y based on and an auxiliary variable z, which captures transformations between them. The model minimizes the discrepancy be- tween predicted and actual embeddings via D (Ey, Pred z),, using an asymmet- ric Context and Target Encoder to prevent representation collapse (Chen & He, 2021). By operating in representation space rather than raw input, JEPA focuses on learning meaningful, generalizable features and cap- turing underlying data relationships for robust self self-supervised learning.
By
Prodictor
D(Ey, Ea)
Ey
Contoxt
Target Encodor
Encodor
Figure 7: Joint-Embedding Predictive Archi- tectures aim to estimate the representation of target input y based on the representation of a context input x, leveraging a predictor network that incorporates auxiliary variable to enhance prediction performance.
B Details of the Tokenizer Component
Taken Emb
Ajc15)
Amile
AAlipods
Ae
AjMASK
Aglond
AMMAK
Token Position Emb
Bg
B;
B2
Bg
Eg
Taken Type Emb
Ckkey
Cyalue
Cssy
Cyalue
Item Position Emb
Dg
D,
[CL.8]
Title
Airpeds
[MASK]
Brand
[MASK]]
Masking Sequence
Token Emb
ACCUS)
Arew
AAippdds
A
AquAcaa
Arpand
AquAsaq
Token Position Emb
Rg
D,
0:
Bg
De
Ds
Ds
Type
DCCLs)
Cyty
Cosaie
Cony
Cueaue
Position
Dg
(CLS)
Title
Airpods
Pro
Brand
Apple
Full Sequence
Figure 8: Tokenizer Component
Figure 8 illustrates the Tokenizer Component in detail. This component is responsible for transforming item metadata into sequence representations through a masking strategy that simulates partial observation. Specifically, it flattens item metadata and applies token-level masking to create a masked input sequence, which is then passed through the Context Encoder. Simultaneously, the complete (unmasked) input is processed by the Target En- coder. These parallel encoding paths enable the model to learn robust representations by contrasting partial and full information.
13

---

Published as a conference paper at COLM 2025
In detail, to encode item sentences, follow previous work in Sun et al. (2019); Geng et al. (2022) to construct four different types of embeddings:
Token Embedding represents the corresponding tokens, denoted by A Raxd where V. is the number of words in vocabulary and d is the embedding dimen- sion. JEPA4Rec represents items using text instead of ID tokens (Hua et al., 2023), making its size independent of the number of items and ensuring flexibility across different recommendation scenarios.
Token position embedding represents the position of token in a sequence, denoted by B E R It is designed to help transoommr-bbaee models capture the sequential structure of words.
Token type embedding identifies the origin of token within the input. It consists of three vectors, CCLS, Cauribute/ Cvalue E Rd which distinguish whether token belongs to [CLS], attribute names, or values. In recommendation datasets where attribute keys are often repeated across items, token type embedding enables the model to recognize and differentiate these recurring patterns.
Ttem Position Embedding represents the position of items in sequence, with all tokens from sentences S; represented as D; E Rd and the entire item position embedding matrix as D Ruxd where is the maximum length of user's interaction sequence. D facilitates the alignment between word tokens and their corresponding items.
C
Baselines
We compare JEPA4Rec against three categories of baseline models:
ID-only Methods: SASRec (Kang & McAuley, 2018) utilizes a directional self- attention mechanism to capture item correlations within a sequence. BERT4Rec (Sun et al., 2019) applies bidirectional Transformer with cloze-style objective for modeling user behavior.
S3-Rec (Zhou et al., 2020) leverages mutual information maxi- mization for pre-training sequential models, capturing relationships between at- tributes, items, subsequences, and full sequences. LlamaRec a two-stage ranking framework that leverages LLMs by retrieving candidate items via small-scale rec- ommenders and ranking them using a verbalizer-bar approach.
ID-text Methods:
UniSRec (Hou et al., 2022) employs text-based item represen- tations from pre-trained language model, adapting to new domains through an MoE-enhanced adaptor. VQ-Rec (Hou et al., 2023) mitigates the over-reliance on textual features in transferable recommenders by mapping item text to discrete codes, which are then used to retrieve item representations from a code embedding table (text -> code representation). We initialize them with pre-trained param- eters provided by the authors and fine-tune them on target domains. RecFormer (Li et al., 2023a) introduces framework that formulates items as text sequences and employs a bidirectional Transformer to learn language representations for sequential recommendation.
Text-only Methods:
D Experiment Settings
We use a finetuning batch size of 16, a learning rate of 5e-5, token limits of 32 per attribute and 1024 per sequence, maximum of 50 items per sequence, temperature parameter I = 0.05, and an MLM loss weight A₁ = 0.1. For pre-training, we use a batch size of 32 and Mapping Representation loss weight Ag 0.1.
14

---

Published as a conference paper at COLM 2025
0.35
JE
RecFomer
0.25
0.15
0.05
A0000
50%
25%
10%
Bosc
Target Item Information
(a) The purchase history of in Musical Instruments dataset. Comparison of the recommended item before and after revealing information. The revealed informa- tion consists of two words: "Musical" and "Amplifier".
(b) Testing the zero-shot capability of JEPA4ReC when revealing different ratios words for the target item sentences in Musical Instruments.
Figure 9: Case study and analysis of revealing information
E
Case Study
Firt case study, we analyze how text-only models adapt with partial next-item information by revealing different word ratios from item sentences in a zero-shot setting on the Instru- ments dataset, detailed in Figure 9, Both JEPA4Ree and RecFormer improve Recall@10, but JEPA4Rec shows a larger gain due to its mapping loss pre-training. Notably, JEPAs performance doubles with just 10% of item information, demonstrating its ability to leverage minimal textual cues. For a user interested in music production and guitar-related items, JEPA4Rec ranked the ground-truti item 10th initially but moved it to 1st after revealing par- tial information. This highlights its strong adaptability, making well-suited for real-world scenarios where only limited item context is available.
Second case study, to evaluate the robustness of our approach when item information is incomplete, we conducted a systematic evaluation by randomly dropping 20%, 40%, and 60% of tokens from item sequences at the target domain during the fine-tuning process. This simulates real-world scenarios where item descriptions may be incomplete or partially available. The masking is applied to the flattened item sentences containing "Title Brand Category ..." information.
Masking Ratio
Method
Scientific (NDCG@10)
Instrument (NDCG@10)
Office (NDCG@10)
RecFormer
0.1198
(1.0001
0.1151
Full info
JEPA4Rec
0.1282
0.1057
0.1276
20% masked
RecFormer
0.1145
0.0971
0.1207
JEPAARec
0.1223
0.0999
0.1242
RecHormer
0.1114
(..0954
0.1174
40% masked
JEPA4Rec
0.1204
0.0997
0.1233
RecFormer
0.1088
0.0905
0.1093
60% masked
JEPA4Rec
0.1201
0.0979
0.1197
Table 4: Robustness evaluation with different masking ratios
These results demonstrate JEPA4Rec's superior robustness when making recommendations based on partial item information. Notably, JEPA4Rec maintains stable performance even when 60% of item tokens are masked, with only modest performance degradation (e.g., from 0.1282 0.1201 on the Scientific dataset). In contrast, RecFormer shows a significant accuracy downgrade as the masking ratio increases, with performance declining more substantially (e.g., from 0.1198 to 0.1088 on the Scientific dataset). This robustness stems from JEPA ability to learn item representations in the embedding space through its mapping loss pre-training, enabling effective inference even with incomplete textual descriptions. JEPA4Rec consistently outperforms RecFormer by 3 - 5% on the NDCG metric across all masking scenarios.
15

---

Published as a conference paper at COLM 2025
F
More zero-shot Experiments
Datasets
JEPAARec
RecFormer
NDCG@10
0.0896
0.0897
Scientific
Recall@10
0.1319
0.1310
MRR
0.0810
0.0820
NDCG@10
0.0360
0.0416
Instruments
Recall@10
0.0632
0.0699
MRR
0.0316
0.0359
NDCG@10 10
0.0610
0.0722
Arts
Recall@10
0.1061
0.1090
MRR
0.0592
0.0620
NDCG@10 10
0.0568
0.0524
Pet
Recall@10
0.0782
0.0698
MRR
0.0525
0.0473
NDCG@10
0.0610
0.0551
Office
Recall@10
0.0840
0.0823
MRR
0.0462
0.0472
NDCG@10
0.0310
0.0310
Online Retail
Recall@10
0.0490
0.0592
MRR
0.0318
0.0310
Table 5: Performance comparison between JEPA4Rec and RecFormer under zero-shot setting
We conducted comparison of JEPA4Rec's zero-shot capability when pre-trained three Amazon datasets against the official public checkpoint of RecFormer, which was pre-trained seven Amazon datasets. We can observe that the recommendation capabilities of the two models are quite similar, demonstrating JEPA4Rec's 's robust training ability that does not depend on large-scale data.
G More ablation studies
G.1 Token Embedding Component Analysis
We evaluate the importance of different token embedding types:
Experiment 1: Removing MLM Loss We remove the MLM loss component to assess how much semantic information learning from textual data contributes to JEPA4Rec's recommendation performance.
Removing Token Type Embedding (C) This experiment evaluates the importance of token type embedding in distinguishing different types of textual information within item representations.
Experiment 2:
Experiment Removing Token Position Embedding (B) and MLM Loss Token position embedding crucial for language models to determine token positions. We remove both B and MLM loss to assess JEPA4Rec's recommendation capability when losing components that help learn semantic information from e-commerce text data.
Experiment 4: Removing MLM Loss, Token Position Embedding (B),, and Token Type Embedding (C) This experiment determines JEPA4Rec's performance when relying solely on item representation recovery through L2 Mapping loss.
Note on Item Position Embedding (D): We do not remove D as it is essential for determin- ing item positions in history sequences and enabling JEPA4Rec to recover complete item representations.
These results demonstrate:
1. MLM loss crucial for performance as it helps the model learn additional semantic information from the textual data of items. The performance drop when removing
16

---

Published as a conference paper at COLM 2025
Scientific Recall@
Online Retail
Variant
NDCG@10
MRR
NDCG@10
Recall
MRR
Full JEPA4Rec
0.1282
0.1761
0.1790
0.1266
0.2429
0.0985
w/o MLM loss
0.1170
0.1653
0.1128
0.1118
0.2216
0.0858
w/o Token Type (C)
0.1251
0.1749
0.1072
0.1231
0.1755
0.0915
w/o MLM Token Pos (B)
0.1102
0.15442
0.1021
0.1094
0.1898
0.0824
w/o MLM + Token Type (C)
0.0967
0.1398
0.0889
0.1019
0.1687
0.0792
Table 6: Token embedding component analysis
MLM loss (from 0.1282 to 0.1170 NDCG@10 on the Scientific dataset) shows its significant contribution understanding item semantics.
2. All embedding components are important for the model's effectiveness:
Token type embedding (C) provides discriminative power for different at- tribute patterns, with noticeable performance degradation when removed.
Token position embedding (B) is essential for understanding token sequence structure.
The combined removal of multiple components leads to progressively worse performance, confirming that each embedding type contributes unique and valuable information the model.
G.2 Ablation Studies on Loss Components
We conduct ablation studies on different loss components to understand their contributions:
Experiment 1: Pre-training when dropping contrastive loss We evaluate JEPA4Rec's ability to learn item representations the embedding space by using only the Mapping Loss and MLM during pre-training. This experiment demonstrates how well the item representation recovery mechanism alone contributes recommendation performance without the contrastive learning component.
Experiment Fine-tuning with BPR Loss During fine-tuning, we replace the contrastive loss with the well-known Bayesian Personalized Ranking (BPR) loss to assess the impact of different ranking objectives:
LBPR =
Σ (wii))Ds
Infoffui Puj)
where D5 = ((,,,)) It AJE + represents the training data with positive item i
and one sampled negative item for user u, and Pui denotes the predicted preference score.
Scientific
Online Retail
Variant
NDCGØ10
Recall@10
MRR
NDCG010
Recall@10
MRR
Full REPA4Rec
0.1282
0.1761
0.1190
0.1266
0.2429
0.0985
Pre tra-train Drop contrastive loss
0.1216
0.1634
0.1087
0.1064
0.1853
0.0892
Fine-tuning BPR Loss
0.1245
0.1723
0.1156
0.0934
0.1637
0.0865
Table 7: Ablation studies on loss components
These results demonstrate that contrastive loss with its ability to sample multiple negative items remains well-suited for JEP. The performance degradation when replacing contrastive loss with BPR loss (which samples only one negative item per positive) shows that the multi-negative sampling strategy of contrastive learning provides richer training signals. This particularly evident in the Online Retail dataset, where BPR loss shows significant performance drops across all metrics, highlighting the importance of diverse negative sampling for effective representation learning our framework.
17


Reviews:
[{"review_id": "iEYGGG9gE9", "reviewer": "Reviewer_F2XD", "summary": "This paper proposes a framework that uses textual item descriptions for the task of sequential recommendation. It explores representing items by their attributes and learning to encode them, to capture general item embeddings and subsequently exploit them to improve model performance on item recommendation in large-scale shopping datasets. \n\nThe paper provides a large set of experiments showing a range of results, detailed explanations for the pre-training framework and item embeddings. Notably, the experiments show that their trained model have a higher data efficiency than previous works, outperforming them in metrics at similar data ratios. However, the paper suffers from a lack of clarity overall, and the metric improvements are very small, questioning some of the claims of the paper and its originality.\n\nUPDATE AFTER REBUTTAL: increased the score from 5 to 7 based on the authors' response, see my full response below.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides in-depth details of the framework, easing its reproducibility.\n- It shows iterative improvements over previous state-of-the-art methods, across domains and at different ratios of training data.\n- The idea of reconstructing item information from partial descriptions described in the paper seems to allow learning generalisable item representations while exploiting text biases from model pre-training, potentially opening further avenues of research, particularly with larger models than explored in the paper.\n- Overall, the paper is solid and may be technologically impactful, releasing resources that would benefit future works in the community.", "reasons_to_reject": "- The main reason for rejecting this is the lack of clarity at many stages of the paper, particularly in Section 3. It is not obvious how each piece fits together, and requires re-reading paragraphs several times to fully understand what it is being explained. Further arguments for this point:\n    - Figure 2 does not help, as it is too high-level to explain what is being described, yet does not clearly identify the parts of the system (i.e., how do the Predictor and the Target encoder fit together in the final model? What exactly is being used to predict the final output?).\n    - This section mixes what is a contribution (or novel) and what is from previous works in the framework, leaving it up to the reader to extract the novelty (i.e., line 97 suggests that their masked language modelling is novel).\n    - The section fluctuates between explaining low-level and high-level concepts without clear motives. For example, concepts that are perhaps obvious in NLP are described in detail or mentioned, without a particular goal or any modifications to their standard use, such as token embeddings (line 124), masked language modelling, or how BERT and RoBERTa can also be used for encoding (lines 154). I think the paper would benefit from moving these less-relevant parts to the Appendix to focus on the actual contribution.\n- Some of the paper’s claims are too broad and not justified by the experiments. For example, in many parts of the paper, it claims that this framework learns common-sense user preferences (line 95, 163, 277, 296). However, I think that the framework aids a model to learn more general item representations, and thus is able to provide better recommendation predictions. I am not sure that the model learns common-sense (in the way described in the paper) or user preferences in general, and none of the experiments show this, particularly since the metric improvements against previous works are so small (~0.01 better).\n- Discussion is very shallow and does not distil (or interpret) the results. The ablation experiments are also limited, despite all the different parts introduced in Section 3. For example, only token type embeddings are ablated, despite introducing 4 types in Section 3.2. I would expect that a paper providing a framework with so many different parts would provide stronger support for these design choices, such as ablating the sequence-item contrastive task, or the mapping loss (and not just the MLM loss).\n- Novel contributions and originality are not clear throughout the paper, and the contributions themselves seem to lack support or justification. The main contributions are the encoder with modified embedding layers, the masking strategy and a 2-stage (training/fine-tuning), but neither of these is novel, and the differences with standard practices in NLP are very subtle and not explained in detail. I think the paper would benefit from rephrasing or re-structuring some parts to emphasise the contributions, perhaps targeted Figures in these sections.\n- Very limited related works section and discussion of how their framework compares to previous works (§3.5).", "questions_to_authors": "1. Can you expand your motives or support for the contributions?\n2. How is a two-stage training approach a contribution and how does it differ from standard NLP in this paper?\n3. How does the encoding item mechanism differ from previous works?\n4. Can you provide further information on the difference in training/fine-tuning from standard NLP practices? Are you applying the MLM loss for pre-training and then not for fine-tuning?\n5. When are the Predictor and Target trained? Are these frozen during fine-tuning?", "extra_scores": {}}, {"review_id": "WreTfhYTdy", "reviewer": "Reviewer_HQmU", "summary": "The paper is well structured and well written, and its main contribution is the adoption of the JEPA framework in the sequential recommendation framework, named JEPA4Rec. JEPA4Rec exploits textual description of items to capture the common sense knowledge related to user preferences based on their consumption histories. \n\nOriginality: This paper is the first at using the JEPA framework in the NLP and recommendation field, and I can some see novelty in this aspect, in addition to the customization proposed by the authors that make JEPA4Rec different from the original framework. However, this could still be considered a limited novel work, as the paper needs more details about the challenges that previous works did not tackle, while this paper did.\n\nQuality and clarity: the paper is written in a clear way and is quite easy to read, although the constraints imposed by the page limit are evident in the discussion of some parts, in particular, the related works section is too short. \nFor example, in the introduction section, the authors claim that \"the semantic gap between modalities remains a challenge\". This is not true, as there are some works in the field of recommendation that show how modalities alignment tend to improve the recommendation performance, such as LMM4Rec [1].\n\nSignificance: The reasons and practical impact of the proposed methodology are not clear and should be more emphasized: which is the problem the paper is solving, and why previous work did not solve it? The authors should have been more clear in this aspect and better frame their work in the literature, although the experiments and the results confirm their approach.\n\nOther comments:\n\nIn the pre-training phase, during the masking strategy, the authors claim to mask \"a significant portion (50%)\" of the tokens of the item to be recommended; it would be interesting to consider this portion as a sensitivity parameter, since it is not indicated why they choose this particular number and not, for example, 80% - that can still be considered as a significant portion. Or, if the authors based this choice on other works, they should declare that.\n\nAnother aspect I'm not convinced is related to the performance evaluation.\nHere, a common metric used in sequential recommendation models is the hit rate @k (hr@k), with k typically set to k={1,5} (for example, the mentioned models Bert4Rec S3-Rec user hr@1 among the other metrics). \nSince the focus of sequential recommendation is next-item prediction, hr@1 is necessary, as it tells how good the model is at recommending exaclty the next item. This make sense also based on what the authors say in the Evaluation Settings paragraph (Section 4.1), as they applied a \"leave-one-out strategy: the latest interaction for testing, the second latest for validation, and the rest for training\". On the other hand, ndcg@10 and recall@10 might be not enough good at capturing the behavior of the model in the exact next item prediction task, but \"only\" consider what happens within the next 10 recommended items. \n\n[1]: Zixuan Yi, Zijun Long, Iadh Ounis, Craig Macdonald, and Richard McCreadie. 2025. Enhancing Recommender Systems: Deep Modality Alignment with Large Multi-Modal Encoders. ACM Trans. Recomm. Syst. 3, 4, Article 52 (December 2025), 25 pages. https://doi.org/10.1145/3718099\n\nEdit: after discussion, the authors made some aspects of the paper more clear and added other metrics, and I increased my review score.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper well written and clear\n- Source code and dataset processing script provided", "reasons_to_reject": "- It can be considered a paper with limited novelty\n- It needs more detailed context and discussion about the practical impact\n- Evaluation setting and metric not necessarily suitable for the task", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HY4T04o15f", "reviewer": "Reviewer_fZrx", "summary": "In this paper, the authors propose JEPA4Rec, a framework that combines Joint Embedding Predictive Architecture with language modeling of item textual descriptions. Specifically, it represents items as text sentences by flattening descriptive information, and encodes these sentences through a bidirectional Transformer encoder with modified embedding layers tailored for capturing item information in recommendation datasets. Through extensive experiments, the authors demonstrate the benefits of the proposed model over a set of existing baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed model is easy to follow and technically sound. The authors have conducted extensive experiments to demonstrate the benefits of the proposed model over a set of existing baselines across various experiment settings.", "reasons_to_reject": "While I do not see any major flaws in this paper, I feel that there are a number of places that the authors can further improve this paper: (1) it seems that flattering the metadata attributes is an important component in the proposed framework, so will the order of these attributes affect the model outcomes? (2) I wonder if we can rely on the \"knowledge\" within the LLMs to encode the items directly for those cases where the metadata attributes are not available (3) I wonder if the authors can conduct additional analysis to study the relative contribution of four types of embeddings in the proposed model; additionally, why do the authors choose to simply sum them together? It seems that a more intuitive way is to do a weighted sum.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: JEPA4Rec is a new method for sequential recommendation that uses a Joint Embedding Predictive Architecture (JEPA) and language modeling of item descriptions to overcome data sparsity and better understand user preferences. It works by treating item information (like title and category) as text, encoding it with a Transformer, and then using masked prediction and a two-stage self-supervised training to learn rich, transferable item representations. This approach significantly improves recommendation performance, especially in situations with limited data or across different domains.

Reviewer F2XD highlights how this paper would benefit from some major edits that would help it clarify the overall contribution of the architecture and what the main changes are to the standard Jepa architecture. Some of these have been addressed by the authors’ response to this reviewer, and I would suggest that the authors make sure that the final version of the paper takes all of this into account.

Reviewer HQmU highlighted an important lack of justification of certain parameters, such as the masking ratio, as well as details regarding the evaluation metrics used. The authors provided a detailed response and added additional results which clarify the contribution. I would invite the authors to take all these comments into account and refine the final version accordingly. 

Reviewer fZrx and F2XD highlight important weaknesses regarding ablations in the experimental evaluation. For instance, the role played by the different types of embeddings or the different attribute order is not clearly studied. However, during the rebuttal phase, the authors have promptly reported additional experiments that seem to justify the choices they have made. I would invite the authors to report this in the paper (potentially in the appendix) to make sure that this weakness is mitigated. 

Overall, all the reviewers agree that this represents an interesting methodological contribution for the NLP and RecSys communities. There are many changes that have been highlighted by the reviewers, which the authors have promised to take into account in the final version of the paper.
Decision: Accept