Language Representations Can be What Recommenders Need: Findings and Potentials
Authors: Leheng Sheng
An Zhang
Yi Zhang
Yuxin Chen
Xiang Wang
Tat-Seng Chua
Venue: ICLR 2025 Oral (2025)
Keywords: Collaborative filtering
Language-representation-based recommendation
Language models
Language model representations
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields.
However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing understanding that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting a recommendation space directly from the language representation space.
Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance.
This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs.
Motivated by the finding of homomorphism, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations without ID-based embeddings.
To be specific, we incorporate several crucial components (i.e., a multilayer perceptron (MLP), graph convolution, and contrastive learning (CL) loss function) to build a simple yet effective model, with the language representations of item textual metadata (i.e., title) as the input.
Empirical results show that such a simple model can outperform leading ID-based CF models on multiple datasets, which sheds light on using language representations for better recommendation.
Moreover, we systematically analyze this simple model and find several key features for using advanced language representations:
a good initialization for item representations, superior zero-shot recommendation abilities in new datasets, and being aware of user intention.
Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.

Published as a conference paper at ICLR 2025
LANGUAGE REPRESENTATIONS CAN BE WHAT RECOMMENDERS NEED: FINDINGS AND POTENTIALS
Leheng Sheng' An Zhang Yi Zhang? Yuxin Chen' Xiang Wang Tat-Seng Chua'
National University of Singapore Uniierritty of Science and Technology of China leheng. shengēu. nus edu, anzhang@u nus edu, zy1230 (esammmml...mmm ust.. edu, en, e1143404eu nus edu, x 22aggaagg223eepgg il....aa....................... com, descts@nus .ew.seeuuuuuuuuuuu...... sg
ABSTRACT
Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing under- standing that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting recommendation space directly from the language representation space. Surprisingly, our findings demonstrate that item representations. when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome sug- gests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs.. Motivated by the finding of homomorphism, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations with- out ID-based embeddings. To be specific, we incorporate several crucial com- ponents (i.e., a multilayer perceptron (MLP), graph convolution, and contrastive learning (CL.) loss function) to build simple yet effective model, with the lan- guage representations of item textual metadata (i.e. title) as the input. Empirical results show that such a simple model can outperform leading ID-based CF mod- els on multiple datasets, which sheds light on using language representations for better recommendation. Moreover, we systematically analyze this simple model and find several key features for using advanced language representations: a good initialization for item representations, superior zero-shot recommendation abili- ties in new datasets, and being aware of user intention. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
INTRODUCTION
Language models (LMs) have achieved great success across various domains (Vaswani et al., 2017; Devlin et al., 2019; Dubey et al., 2024; OpenAI, 2023), raising critical question about the knowl- edge encoded within the language space. Recent studies empirically find that LMs extend beyond semantic understanding to encode comprehensive world knowledge about various domains, such as game states (Li et al., 2023a), lexical attributes (Vulic et al., 2020), and even concepts of space and time (Gurnee & Tegmark, 2023) through language modeling. However, in the domain of recom- mendation where the integration of LMs is attracting widespread interest (Fan et al., 2023; Li et al., 2023b; Wu et al., 2023a), it remains unclear whether LMs inherently encode relevant information on user preferences and behaviors in the language space.
Currently, one prevailing understanding holds that general LMS and traditional recommenders (e.g., collaborative filtering models (Koren et al., 2009; He et al., 2021)) encode distinct representation spaces - one for language space and the other for behavior space - but they offer the potential
An Zhang is the corresponding author.

---

Published as a conference paper at ICLR 2025
Cosine Similarity
X
LM
Z
Linear
ei
Average
C2
e
eu
Title
Language Representation
Item Represcntarion
History Item Representations
User Representation
(a) Linearly mapping language representations into the behavior space for recommendation
User Jutcetioe
Marvel sapertura mesies."
teperbore
0.65
Harasmad Moriee
Lisss Mapplaz
"My Bexstjall Laasdwate"
0.82
Tive
"Foud
LighGEN
BLMBBc liner Mappieg
(b) Performance comparison
(c) The t-SNE representations of movies and user intention in two spaces.
Figure 1: Linearly mapping item titles in language representation space into behavior space yields superior recommendation performance on Movies & TV (Ni al., 2019) dataset. (1a) The frame- work of linear mapping. (Ib) The recommendation performance comparison between leading CF recommenders and linear mapping. (Ic) The t-SNE (Van der Maaten & Hinton, 2008) visualiza- tions of movie representations, with colored lines linking identical movies or user intention across language space (left) and linearly projected behavior space for recommendation (right).
to enhance each other in downstream recommendation tasks (Liao et al., 2024). Specifically, on the one hand, when using LMs as recommenders to directly output items of interest, aligning the language space with the behavior space can significantly improve the recommendation performance (Lin et al., 2023a; Vats et al., 2024; Xu et al., 2024). Various alignment strategies are proposed, including fine-tuning LMs with user behavior data (Zhang et al., 2023e; Bao et al., 2023; Geng et al., 2022: Cui et al., 2022; Lin et al., 2023b), incorporating embeddings from traditional recommenders as a new modality of LMs (Liao et al., 2024; Zhang et al., 2023f; Yang et al., 2023; Tennenholtz al., 2024), and extending the vocabulary of LMs with item tokens (Zhu al., 2023; Zheng al., 2023; Rajput et al., 2023; Zhai et al., 2024). On the other hand, when using LMs as the enhancer to represent item content (e.g., text metadata), traditional recommenders greatly benefit from text embeddings (Yuan et al., 2022; 2023; Li et al., 2023c; Hou et al., 2024a; Liu et al., 2024b), semantic and reasoning information (Wei al., 2024; Ren et al., 2024b; Xi et al., 2023), and generated user behaviors (Zhang et al., 2023b;d). Despite these efforts, explicit studies of the relationship between language and behavior spaces remain largely unexplored the recommendation domain.
In this work, we re-examine this prevailing understanding, by exploring whether A--generated lan- guage space has inherently encoded user preferences and behaviors. Specifically, we test the possi- bility of directly deriving a behavior space from the language space that is, we assess whether the language representations of item text metadata (e.g., titles) generated by LMs can independently pre- dict user behaviors and achieve competitive recommendation performance. Positive results would imply that user behavioral patterns, such as collaborative signals (i.e.. users' preference on items reflected by their behavioral similarities) (Wang et al., 2019b), might be implicitly encoded by LMs. To test this hypothesis, we employ linear mapping (Merullo et al., 2023) to project language rep- resentations of item titles into behavior space for recommendation, Figure 1a shows. Our empirical observations and findings include:
Before linear mapping, language representation similarities (i.e., semantic textual similarities (STS) (Muennighoff et al., 2023)) may reflect user preference similarities for item contents. Con- sidering Figure 1c example, movies with themes of superheroes and monsters cluster gether in both language and behavior spaces.
After linear mapping, language representations are transformed into high-quality behavior repre- sentations, which achieve exceptional recommendation performance, Figure 1b and experimen- tal results in Section 3.2 show. Moreover, the performance improves as the language model size increases and remains relatively robust to prompt disturbances (see Section 3.2).
2

---

Published as a conference paper at ICLR 2025
Post-mapping language representations encode user behaviroal similarities beyond STS. For in- stance, while certain movies, such as those of homosexual movies (illustrated in Figure Ic), show low STS and their representations disperse in the language space, their projections through linear mapping tend to cluster together, reflecting high user preference similarities.
These findings suggest the homomorphism (Dieudonne, 1969) between the IM-generated language space and an expressive behavior space for recommendation. Motivated by this insight, we explore the possibility of building advanced collaborative filtering (CF) models based solely on the language space. To be specific, considering text metadata solely as items' pre-existing features rather than the widely-used ID information, we perform a frozen LM create the language representations of items; consequently, to map them into behavior space, and then employ contrastive loss (i.e., InfoNCE (van den Oord et al., 2018; Wu al., 2022)) optimize. We term this model AlphaRec for its simplicity and a series of good properties. Surprisingly, our empirical results show that such a simple model outperform leading ID-based CF models on multiple datasets. This result sheds light on using language representations for better recommendation.
apply trainable projector (i.e., a two-layer MLP with graph convolution)
Furthermore, systematically analyze this simple model and discover several potentials of adopt- ing language representations for recommendation. First, language representations may serve as a good initialization for item representations, with few adjustments achieve high recommendation performance (see Section 5. This is evidenced by the rapid training convergence of AlphaRec. Second, advanced language representations provide strong zero-shot recommendation capability across entirely new datasets (see Section 5.2). By co-training on multiple datasets, AlphaRec can achieve performance comparable to or even surpassing the fully-trained LightGCN (He al., 2021) on new datasets without additional training. This underscores the potential of adopting advanced language representations to develop more generalizable recommenders. Third, advanced language representations provide opportunities for perceiving user intentions to refine recommendation results (see Section 5.3). Endowed with the inherent semantic comprehension language representations, AlphaRec adjust recommendations according text- text-based user intentions, enabling recom- menders to evolve into intention-awan aware systems through a straightforward paradigm shift.
PRELIMINARY
2.1
TASK FORMULATION
Personalized recommendation aims to learn user preferences from historical behaviors (i.e., histor- ical interactions with items like view, click, purchase) and find items of interest to trigger users' future behaviors (Zhou et al., 2018). In this paper, we consider one common recommendation set- ting: collaborative filtering (CF) (Koren et al., 2022). It aims to select item E I that best matches user u's preferences based on binary interaction behaviors Y [Mus]. where yui = 1 indicates user u U has interacted with item i, and Yoi otherwise (Rendle, 2022). Scrutinizing leading CF models, summarize common paradigm tui
de (x,, x;) involving three components:
For a user-item pair (u, i), we first get their pre-existing features x,, and x,, which are usually set ID information or one-hot encodings in CF (Koren et al., 2009; Rendle, 2022; He al., 2021).
Upon x,, and x,, the representation generation module á parameterized by 0 is adopted to trans- fer them into behavior representations e,, and e,, encoding the behavioral patterns of users. Its architecture can vary widely, including ID-based embeddings (Koren et al., 2009), multilayer perceptions (He et al., 2017). graph neural networks (Wang et al., 2019b; Cai et al., 2023), and variational autoencoders (Liang et al., 2018).
Upon and e, the scoring function is used to quantify their relevance reflecting how likely user u will interact with item i. One widely-used function is cosine similarity, s s(ew, e;) Te, [+]e,
e,
(Chen et al., 2023; Wu al., 2022).
2.2 ITEM REPRESENTATION GENERATION
Here emphasize the critical role of item representation generation, which involves transforming item i's pre-existing features x; into representations e; suitable for recommendation. This process is essential, the quality of these representations directly impacts the recommendation performance.
3

---

Published as a conference paper at ICLR 2025
In this paper, we focus mainly two kinds of item representation generation tailor-made for differ- ent pre-existing features: ID- and LM-based generators.
ID-based generator. Prevailing CF models (Koren al., 2009; Rendle, 2022: Koren et al., 2022; Wang et al., 2019b; He et al., 2021; Yu ct al., 2024) typically convert the ID information of each item into one-hot encodings (e.g.. pre-existing features x;). These sparse features through a trainable generator, such as ID embedding matrices (Koren et al., 2009; Rendle, 2022) or optionally combined with graph convolution layers (Wang al., 2019b; He al., 2021; Yu al., 2024), to generate dense representations e,, Optimizing the learning of ID-based representations allows the generator to effectively learn user preferences and behaviors, leading to competitive rec- ommendation performance. However, such ID-based generators suffer several problems, such as poor domain transferability and lack of user intention-aware abilities, since one-hot encodings lack sufficient semantics beyond being identifiers (He et al., 2021).
then passed
LM-based generator. Beyond ID information, another research line (Pazzani & Billsus, 2007; Covington et al., 2016; Liu et al., 2024c; Zhang et al., 2024b; Liu et al., 2023b) explores using the text metadata of item (e.g, titles, descriptions) as pre-existing features x,. These features are fed into the LM-based generator, typically a combination of two subsequent components: (1) A frozen LM to extract i's language representation z,, such as the encoder-only LMs like BERT-style models (Devlin et al., 2019; Liu et al., 2019), the decoder-only LMs like Llama-style autoregressive models (Touvron et al., 2023b; Jiang et al., 2023) and OpenAl text embedding models (Neelakantan et al., 2022); and (2) A trainable projector map 2; into the final representation e, often using layers like graph convolution layers (He et al., 2021). Although such LM-based generators have been explored to enrich the item representations in literatures ( (Yuan al., 2023; Li et al., 2023e; Ren et al., 2024b), few studies have demonstrated that they can solely outperform ID-based generators in recommendation tasks. Worse still, the relationship between the AM-based language space and the behavior space remain largely unexplored in the recommendation domain.
3 UNCOVERING COLLABORATIVE SIGNALS IN LMS VIA LINEAR MAPPING
Do LMs inherently encode collaborative signals (i.e., users' preferences for items reflected by behavioral similarities) within their representation spaces? RQ2: If so, does the presence of such signals scale with model size, and are they robust across different settings? To investigate these questions, we use linear mapping to project language representations of item titles into a behavior space for recommendation. We detail the implementation of the linear mapping in Section 3.1. Subsequently, in Section 3.2, we empirically assess the existence and robustness of collaborative signals in language representations.
In this section, we first explore the following research questions. RQI:
3.1 LINEAR MAPPING
Linear mapping effective to study the representation properties of LMs (Merullo et al., 2023; Alain & Bengio, 2017), discovering the homomorphism (Dieudonne, 1969) between the language space and another space in the target domain. However, its application in the recommendation domain remains largely underexplored.
To bridge this gap, we train a lincar mapping matrix W to project representations from the lan- guage space into a behavior space for recommendation. High performance of this lincar mapping on the test set would indicate the presence of homomorphism between the language space and an effective behavior space, suggesting the possible existence of collaborative signals in the language representation space (Ravichander et al., 2021; Gurnee & Tegmark, 2023). The overall framework of linear mapping illustrated in Figure 1a. Specifically, we use frozen LMs to transform pre-existing item title features x; into language representations z;. To derive user representations, we compute the average of the language representations of the items user u has interacted with, denoted as 2, WoT ΣiEN, Z;, where Nç, is the set of user u's historical items. See Appendix B.2 for de- tailed procedures for obtaining language representations. The linear mapping matrix sets behavior representations of user u and item i as e. = Wz. and e; = Wz, respectively. To optimize the matrix W. adopt the InfoNCE loss (van den Oord et al., 2018) the objective function, which has demonstrated strong performance in both ID-based (Zhang et al., 2023a; Yu et al., 2024) and L.M generators (Ren et al., 2024b) (refer to equation 4 for the formula).
4

---

Published as a conference paper at ICLR 2025
Table 1: The comparison of the recommendation performance linear mapping with the classical ID-based CF baselines.
Movies TV
Video Games
Recall
NDCG
HR
Recall
NDCG
HR
Recall
NDCG
HR
MF (Rendle et al., 2012)
0.0568
0.0519
0.3377
0.0323
0.0195
0.0864
0.0437
0.0391
0.2476
B
MultVAE (1 (Liang al,, 2018)
0.0853
0.0776
0.4434
0.0908
0.0531
0.2211
0.0722
0.0597
0.3418
LightGCN (He et al., 2021)
0.08-19
0.0747
0.4397
0.1007
0.0590
0.2281
0.0723
0.0608
0,3489
Linear Mapping
BERT
0.0415
0.0399
0.2362
0.0524
0.0309
0.1245
0.0226
0.0194
0.1240
RoBERTa
0.0406
0.0387
0.2277
0.0338
0.1339
0.0247
0.0209
0.1262
Llama2-7B
0.1027
0.0955
0,4952
0.1249
0.0729
0.2746
0.0662
0.0559
0.3176
Mistral-7BB
0.1039
0.0963
0.4994
0.1270
0.0687
0,2428
0.0650
0.0544
0.3124
(ext-embedding-adae
0.0926
0.0874
0.4563
0.1176
0.0683
0.2579
0,0515
0.0436
0.2570
text-embed sost-eebeddingg- 3- large
0.1109
0. 1013
0.5200
0. 1367
0.0793
0.2928
0.0735
0.0608
0.3355
Fimheeddigg Mistral
0.1152
0.1065
0.5327
0.1370
0.0787
0.0738
0.3371
Recolly139 NUC0g.29
Recello::9
0.100
NDCGgB 20
NDCGgg20
0.05
0.12
0.185-
0.100
0.06
0.88
0
78
131
7001
71
130
7M01
70
138
7001
(a) Movic & TV
(b) Games
(c) Books
Figure 2: The recommendation performance of linear mapping with different language model sizes.
3.2 EMPIRICAL FINDINGS
Existence (RQ1). To explore the existence of collaborative signals in language representations, we test the recommendation performance of the linear mapping method. Table reports the perfor- mance yielded by post-mapping representations on three Amazon datasets (Ni et al., 20,9), compar- ing with classic ID-based CF baselines: matrix factorization (MF) (Koren et al., 2009), MultVAE (Liang et al., 2018), and LightGCN (He et al., 2021) (see baseline details in Appendix C.2). Figures (2a) (2c) depict the linear mapping performance under different LM sizes. Figure Ic demonstrates the visualization of representations before and after linear mapping. We observe that:
fost-mapping representations of advanced LMs achieve superior recommendation perfor- mance in most cases, suggesting the possible homomorphism between language spaces and behavior spaces. Specifically, advanced LMs (e.g. Llama2-7B (Touvron et al., 2023b) and text- embeddings-3-large (Neelakantan et al., 2022)) consistently perform better than leading CF mod- els (e.g. LightGCN) on most metrics. We also see that the performance improves with more recent and advanced LMs. In contrast, earlier BERT-style models (e.g.. BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019)) perform similarly to or worse than MF, indicating that LMs have only recently developed the ability encode user preference similarities effectively.
Language representations encode user preference similarities beyond semantic textual simi- larities (STS). Consider Figure Ic as an example again, homosexual movies, which differ signif- icantly in textual meaning, cluster together after linear mapping. This suggests that user prefer- ences, which are not immediately apparent from text alone, are implicitly encoded in the language space and can be uncovered through linear mapping.
Scaling and Robustness (RQ2). We also investigate whether the encoding of user preference sim- ilarities in LMs scales with model size and whether this encoding is robust in the presence of noise in the input prompts. To this end, we test the linear mapping performance of LMs of various sizes and prompts with noise:
The encoding of user preference similarities becomes more refined as model size increases, leading to better linear mapping performance. Specifically, we test the linear mapping perfor- mance across different language model sizes (7B, 13B, and 70B) the Llama2 family (Touvron et al., 2023b). Llama3 (Dubey et al., 2024) is not selected due to the lack of 13B model. As shown in Figure 2, linear mapping performance improves consistently as the model size increases from 7B to 70B, indicating that larger models capture more nuanced user behavioral patterns.
5

---

Published as a conference paper at ICLR 2025
Table 2: The robustness of language representations for recommendation.
Movies & TV
Video Games
Books NDCG
Recall
NDCG
Recall
NDCG
Recall
Title Random Noise
0.0952
0.0887 0.0955
0.4731
0.1213
0,0706
0.2722
0.0632
0.0525
0.3099
Tide Only
0.1027
0.4952
0.1249
0.0729
0.2746
0.0662
0.0559
0.3176
Language representations are relatively robust to prompt disturbances. Following previous works (Gurnee & Tegmark, 2023), we compare two prompting strategies: using item titles alone (e.g., Castlevania), and adding 5-10 random letters to the titles (e.g., Castlevania sdfhdsk). Table 2 shows that adding random noise to the item titles had minimal impact on the linear mapping performance. The prompt noise has more impact on Movies & TV since item titles in this dataset are relatively shorter than others (see Appendix C.1). This finding suggests the relative robustness of the recommendation knowledge encoded in the language representation space.
4 LEVERAGING LANGUAGE REPRESENTATIONS FOR BETTER RECOMMENDATION
This finding of possible space homomorphism (Dieudonne, 1969) and encoded collaborative signals arouse interest the following questions. RQ3: How powerful are such language representations for building advanced CF models that can outperform prevailing ID-based CF methods? To address these questions, in Section 4. we aim develop a simple yet effective CF model termed AlphaRec, which is solely based on language representations and merely incorporates three crucial components in modern CF models. After that, we evaluate its performance in Section 4.2 to demonstrate the capability of advanced language representations for recommendation.
4.1 ALPHAREC
We briefly present how this simple model AlphaRec is designed and trained. is important to highlight that we center on exploring the power of language representations for recommendation, rather than deliberately inventing new CF mechanisms. Generally, the representation generation architecture (-)-)) is simple, which only contains two-layer MLP and the basic graph convolution operation. The cosine similarity is used as the similarity function s(,,,,, and the contrastive loss InfoNCE (van den Oord et al., 2018; Wu et al., 2022) is adopted for optimization. For simplicity, adopt weweemeeddingg---largg
(Neelakantan et al., 2022) for language representation generation by default, for its excellent language understanding and representation capabilities.
Nonlinear projection. We substitute the linear matrix delineated in Section with a nonlinear MLP. Nonlinear transformation helps in excavating more comprehensive preference similarities from the language representation space (see discussions about this in Appendix C.7) (He et al., 2017). Taking the averaged language representaions of historical items as the user language representation (i.e., 2, [.T ΣiEN, z;), the initial nonlinear transformation operation be formulated as:
(0) W₂ ReLk (W1z; + bi) b2,
W2 LeakyReLU (Wi,,, + bi) + b2. (1)
Graph convolution. Graph neural networks (GNNs) show superior effectiveness for recommenda- tion (Wang et al., 2019b), owing the natural user-item graph structure in recommender systems (Wu et al., 2023b). We employ minimal graph convolution operation (He et al., 2021) to capture more complicated collaborative patterns from high-order connectivity (Wu al., 2019) as follows:
(k+1) i
Σ YEA,
(l+1)
Σ
(2)
W. W
MEN, V VINT I.
The information of connected neighbors is aggregated with a symmetric normalization term VIN.lv/Mi Here Nr, (N₂) denotes the historical item (user) set that user (item i) has inter-
acted with. The representations ( and c projected from the MLP are used as the input of the first layer. After propagating for K layers, the final behavior representation of user (item i) is obtained as the average of representations from each layer:
K
K
(k)
Cu K+1
c; =
(3)
K
+1 R=0
A==

---

Published as a conference paper at ICLR 2025
Table 3: The performance comparison with ID-based CF baselines. The improvement achieved by
AlphaRec is significant (p-value << 0.05).
Muvies
Video Games
Recall
NDCG
HR
Recall
NDCG
HR
Recall
NDCG
HR
MF (Rendic et 2012)
0,0568
0.0619
0.3377
0.0323
0.0195
0.0861
0.0437
0,0391
0.2477
MuleVAE (Liong et al,, 2018)
0.08G3
0.0776
0.4434
0.0908
0.0531
0.2211
0.0722
0.0597
0.3418
LightGCN (He ot al. 2021)
0,0819
0.0717
0.4397
0.1007
0.0590
0.2281
0.0723
0,0608
0.3418
SGL al,, 2021)
0.0916
0.0838
0,4680
0.1089
0.0634
0.2449
0.0789
0.0657
0.3734
BC Loss (Zhang ct al., 2022)
0.1039
0.0943
0.5037
0.1145
0.0668
0.2561
0.0915
0.0779
0.4045
XSImGCL. (Yu al. 2024)
0.1057
0.0984
0,5128
0.1138
0.0662
0.2550
0.0879
0.0745
0.3918
XSimGCL, (Yu et al, 2024)
0.1015
0.0951
0.5016
0.1
0.0679
0,2674
0.0900
0.0736
0.4036
KAR (Xi et al., 2023)
0.1084
0.1001
0.5134
0.1181
0.0693
0.2571
0.0852
0.0734
0.3834
RLMRec (Ren et al., 2024b)
0.1119
0.1013
0.5301
0.1384
0,0809
0.2997
0.0928
0.0774
0.4092
EMB-KNN
0.0548
0.0380
0.2916
0.0879
0.0389
0.1970
0.0434
0.0248
0.1851
AlphaRec
0
0.1144
0,5587
0.1519
0.0894
0.3207
0.0991
0,0828*
0.4185
Imp.% S over the best baseline
6.79%
5.34%
2.27%
9.12%
10.75%
5.40%
9.75%
10.51%
7.01%
Contrastive learning objective. The introduction of contrastive learning (Radford et al., 2021) another key element for the success of leading CF models. Recent research suggests that the contrast learning objective, rather than data augmentation, plays more significant role in improving recommendation performance (Yu et al., 2024; 2022; Zhang et al., 2023a). Therefore, we simply use the contrast learning object InfoNCE (van den Oord et al., 2018) the loss function without any additional data augmentation on the graph (Wu et al., 2022). With cosine similarity as the similarity function s(e,, e,)
Te, the InfoNCE loss (van den Oord et al., 2018) is written as: [e..
exp (s(Cu: (s(C:,))))
LInONNEE
Σ
log exp (s(e., e,)/T) + Ejes, exp (s(eu, e/)/T)
(4)
(u.f)EO+
Here, T is a hyperparameter called temperature (Wang & Liu, 2021), O+ {(u, Mui 1} denoting the observed interactions between users // and items I. And S, is a randomly sampled subset of negative items that user u does not adopt.
4.2 EMPIRICAL. FINDINGS
Baselines. We compare AlphaRec with leading ID-based CF baselines, assess the effectiveness of adopting advanced language representations. In addition to classic baselines introduced in section 3.2, we consider two categories of leading ID-based CF baselines, CL-based CF methods: SGL (Wu et al., 10. BC Loss (Zhang et al., 2022), XSimGCL (Yu et al., 2024) and LM-enhanced CF methods: KAR (Xi et al., 2023), RLMRec (Ren et al., 2024b). We also consider a variant of XSimGCL (i.e., XSimGCL.) where ID embeddings are replaced with language representations, and directly using language representations z, and z; for recommendation without training (i.e., EMB- KNN). See baseline details in Appendix C.2.
Recommendation capabilities (RQ3). Table 3 presents performance comparison. To alleviate the possible information leakage problem that pretraining data of LMs may include the training data, we also conduct experiments on a newly published dataset Amazon 2023 in Appendix C.6. The best-performing methods are bold, while the second-best best methods are underlined. We observe that:
Advanced language representations shows strong potentials for recommendation, which can be unleashed by appropriate model design. AlphaRec consistently outperforms leading CF baselines by large margin across all metrics on all datasets, with an improvement ranging from 6.79% to 9.75% on Recall @ compared to the best bascline. Moreover, as shown in Figure 3a, each component contributes positively (see more ablation results and details in Appendix C.3) in unleashing the power of language representations. Moreover, simply replacing the ID embeddings in XSimGCL with language representations (i.e., XSimGCL,) does not bring stable and remarkable improvement, which is consistent with previous findings (Yuan et al., 2023; Li et al., 2023c). This phenomenon highlights the importance of appropriate model design beyond simple input feature replacement. Above findings suggest that the power of advanced language representations can be unleashed by carefully designing the model, showcasing the potential to surpass prevailing IDDbased recommenders.
7

---

Published as a conference paper at ICLR 2025
20 NDC
600
LAMIGUN
MF
DC
0.08
400
SCL
Eppeches
MaRVAE
XSHGCE
Alphuattc
0.84
200
0.00
Linear Mappieg
GCN
AlphoNcc
Books
Movies TV
Video Games
(a) Ablation study on Books
(b) Training efficiency comparison
Figure 3: (3a) The effect of each component on Books dataset. (3b) The number of epochs needed for each model to converge. AlphaRec exhibits breakneck convergence speed.
The incorporation of advanced language representations can benefit traditional ID-based CF methods. We note that two LM-enhanced CF methods, KAR and RLMRee, both show im- provements over the most advanced CF methods. Nevertheless, the combination of ID-based em- beddings and language representations in these methods does not yield higher results than purely regguugg-rpreesentttinnn--o
AlphaRec. We attribute this phenomenon to their naive design for the combination ID-based embeddings and language representations, which is also highlighted by previous works (Yuan et al., 2023; Zhang al., 2024d).
5 EXPLORING POTENTIALS OF LANGUAGE REPRESENNATIONS FOR RECOMMENDATION
In this section, focus this question: What new opportunities beyond good performance ad- vanced language representations bring to recommender systems? To answer this, we systematically analyze AlphaRec and discover the following potentials of adopting such language representations. Potential 1: Good initialization for item representations (Section 5.1). Potential 2: Zero-shot abil- ity (Section 5.2). Potential Intention- aware ability (Section 5.3).
5.1 GOOD INITIALIZATION FOR ITEM REPRESENTATIONS (POTENTIAL 1)
Advanced language representations may provide good initialization for item representations, with few adjustments for effective recommendation. As shown in Figure 3b, beyond its good per- formance, AlphaRec also exhibits extremely fast convergence speed, which is comparable with or even surpasses the fastest ID-based CF methods (e.g., SGL (Wu et al., 2021) and XSimGCL (Yu al., 2024)). Moreover, recent works also suggest that, when using advanced language representa- tions to initialize ID-based item embeddings (Harte et al., 2023; Zhao et al., 2024), the performance of traditional ID-based recommenders improves significantly. We attribute the above findings to the homomorphism between the language space and good behavior space. Therefore, when using advanced language representations for initialization, only minor adjustments are needed to generate effective behavior representations for recommendation.
5.2 ZERO-SHOT ABILITY (POTENTIAL 2)
The prevailing ID-based recommenders suffer from domain transferring problems (i.e., behavior representations are highly bound with ID information (Zhu et al., 2021)). Advanced language rep- resentations may provide opportunities for learning transferable item representations (Hou et al., 2022), enabling recommenders to perform well on entirely new datasets without any ID overlap. To address this potential, we test the zero-shot recommendation ability of AlphaRec (Ding al., 202
Experimental settings. In zero-shot recommendation, there is no item or user overlap between the training set and test set (Ding et al., 2021; Zhang et al., 2024c), which is different from the research line cross-domain recommendation (Zhu et al., 2021). We jointly train AlphaRec on three source datasets (i.e., Books, Movies & TV, and Video Games), while testing it three completely new target datasets (i.e., Movielens- 1M (Harper & Konstan, 2016), Book Crossing (Lee et al., 2019), and Amazon Industrial & Scientific (Ni et al., 2019)) without further training these new datasets.
8

---

Published as a conference paper at ICLR 2025
Table 4: The zero-shot recommendation performance comparison on entirely new datasets. The
improvement achieved by AlphaRec is significant (p-value << 0.05).
Industrial & Scicatific
MovicLons-IM NDCG
Book Crossing NDCG
Recall
NDCG
HR
Recall
HR
Recall
HR
MF (Rendle et al., 2012)
0.0344
0.0225
0.0521
0.1855
0.3765
0,9634
0.0316
0.0317
0.2382
full
MulVVEE (Liung et al. 2018)
0,0751
0.0459
0.1125
0.2039
0.3741
0.9740
0.0736
0.0634
0.3716
LightGCN at, 2021)
0,0785
0.0533
0.1078
0,4017
0.0630
0.0588
Random
0.0148
0.0061
0.0248
0,0068
0.0185
0.2611
0.0039
0.0036
0.0443
X200 shot
Pop
0.0216
0.0067
0,0396
0.0253
0,0679
0.5439
0.0119
0.0101
0.1157
ZUSRec (Ding al., 2021)
0.0326
0.0272
0,0628
0.0787
0.0143
UniSiee (Hou et 2022)
0.0453
0.0350
0.0863
0.0578
0.1412
0.7135
0.0399
0.0332
0.2455
VQ-Roc (Hou al., 2023)
0.06-15
0.0410
0.0963
0.0804
0.1921
0.8167
0.0185
0,9492
0.2825
AlphaRee
0.0913*
0.0573
0.1277
0.1486
0.3215
0.9296
0.0660
0.0545
0.3381
Imp.St over the best zero -hoot buseline
157.00%
127.69%
30.29%
66.07%
64.16%
37.78%
101.55%
63.71%
47.97%
(see more details about training on multiple datasets in Appendix D.2.1). Due to the lack of zero- shot recommenders in general CF, we slightly modify three zero-shot methods in the sequential recommendation (Wang et al., 2019a), ZESRec (Hou et al., 2022), UniSRec (Hou et al., 2022), and VQ-Rec (Hou et al., 2023), as baselines. We also incorporate two strategy- baategyy-baeddddd CF methods (i.e., Random and Pop) and one method using the large language model (LLM) as zero-shot recommender (i.e., LLMRank (Hou et al., 2024b)) (see more details about baselines in Appendix D.2.2).
Findings. Table 4 presents the zero-shot recommendation performance comparison. The best meth- ods are bold and starred, while the second-best methods are underlined. We observe that:
Advanced language representations provide opportunities for learning transferable item representations. AlphaRec demonstrates strong zero-shot recommendation capabilities, compa- rable or even surpassing the fully trained LightGCN. AlphaRec performs better the Amazon Industrial & Scientific dataset, possibly because it captures user behavioral patterns of the same platform (Ni et al., 2019) through training on multiple Amazon datasets. Conversely, ZESRec and UniSRec exhibit a marked performance decrement compared with AlphaRec. We attribute this phenomenon to two aspects. On the one hand, BERT-style LMs (Devlin et al., 2019; Liu al., 2019) used in these works may not have effectively encoded user preference similarities, which is consistent with our previous findings in Section On the other hand, components designed for the next item prediction task in sequential recommendation (Kang & McAuley, 2018) may not be suitable for capturing the general preferences of users in CF scenarios. Moreover, AlphaRec also outperforms the leading LLM-based zero-shot recommender LLMRank (see Appendix D.2.3).
The zero-shot recommendation capability of advanced language representations generally benefits from an increased amount of training data, without compromising performance on source datasets. As illustrated in Table 12, the zero-shot performance of AlphaRec, when trained on a mixed dataset, is generally superior to training on one single dataset (Hou et al., 2022). More- over, we discover that AlphaRec, when trained jointly on multiple datasets, hardly experiences a performance decline on each source dataset. These results indicate the general recommendation capability of a single pre- pre-trained AlphaRec across multiple datasets. The above findings also offer a potential research path to achieve general recommendation capabilities, by incorporating more training data with more themes. See more details about these results in Appendix D.2.4.
5.3 INTENTION-AWARE ABILITY (POTENTIAL 3)
The language understanding ability in advanced language representations (especially representa- tions from LLM--based text embedding models) offers the opportunity for perceiving text-based user intentions and refining recommendations. To study the potential of intention ability, we in- troduce a new hyperparameter a in AlphaRec to combine user intentions with historical interests.
Experimental settings. To endow AlphaRec with user intention-aware ability, adopt simple paradigm shift by introducing a user intention representation e Intention In the inference stage, we obtain the language representation c. Intention for each user intention query and combine it with the original user representation to get a new user representation as ((0)) = (1 (1-aa) -e +ae Intention (Ai et al., 2017). This new user representation (00) is sent into the pre-trained AlphaRec for recommen-
dation. We test the user intention capture ability of AlphaRec on MovieLens- IM and Video Games.
In the test set, only one target item remains for each user (Ai et al., 2017). with one intention query

---

Published as a conference paper at ICLR 2025
Table The performance comparison in user intention capture.
MovicLens--he 1M NDCG@S
Video Games HRG5
HR
NDCG@S
TEM (Bi et al., 2020)
0,2738
0.1973
0.1425 0.0498
0.2212
AlphaRoc (wlo Intention)
D.0793
AlphaRec (w Intention)
0,47044
0.3738
0.2569
0.1862
User Intention: I'm looking for classic movic that delves into the world of organized crime, family loyalty, and power struggles, with iconic performances and unforgettable quotes
Target: Godfather, The (1972)
Recommendation List Intention)
Recommendation List
Sravchear((1995
Godlather, The (1972)
Schindler's List (1993)
Confidential (1997)
Star Wars: Episode V (1980)
Schindler's 's List (1993)
Fiction (1994)
Fiction (1994)
L.A. Confidential (1997)
Bravehennt (1995)
Value
(a) Case study of user intention capture
(b) Effect of a
Figure 4: User intention capture experiments on MovieLens-IM. IM. (4a) AlphaRec refines the recom-
mendations according to language- based user intention. (4b) The effect of user intention strength a.
generated by ChatGPT (OpenAI, 2023; Hou et al., 2024a) (see the details about how to generate and check these intention queries in Appendix D.3.1). We report relatively small K 5 for all metrics to better reflect the intention capture accuracy.
Findings: We report the user intention capture experiment results in Table 5, show one case study in Figure 4a, and study the effect of a in Figure 4b. We find out that:
The language understanding ability in advanced language representations enables recom- menders to perceive user intentions and refine recommendations. As shown in Table 5, the introduction of user intention (w Intention) significantly refines the recommendations of the pre- trained AlphaRec (w/o Intention). Moreover, AlphaRec outperforms the baseline model TEM (Bi et al., 2020), even without additional training on search tasks. We further conduct a case study on MovieLens- 1 M to demonstrate how AlphaRec captures the user intention (see Appendix D.3.3). Additionally, the intention-aware ability benefits from user historical interests. Figure 4b depicts the effect of a, which controls the strength of user intention. Here = 0 denotes that the user intention is neglected and = 1 denotes that the user historical interest is ignored. The convex curve in Figure 4b suggests that both user interests and user intention play vital roles. Above findings highlight the potential of adopting advanced language representations to perceive text-based user intentions and refine recommendations (see more details in Appendix D.3).
examples in
LIMITATIONS
There are several limitations unaddressed in this paper. First, this research lacks theoretical guaran- tee, and we do not contribute to designing any new components for CF models. Second, AlphaRec lacks personalized design, since there only single MLP introduced for all users. The fixed user intention in Section 5.3 is less practical, lacking exploration on personalized user intentions.
7 CONCLUSION
In this paper, we explored the relationship between language space and behavior spaces for rec- ommendation, and explored the potential for using language representations for recommendation. Empirical results suggest the possible presence of homomorphism between advanced LMs repre- sentation spaces and an effective item representation space for recommendation. Inspired by this finding, we discussed how to unleash the power of advanced language representations by devel- oping a simple yet effective CF model called AlphaRec. Moreover, by systematically analyzing AlphaRec, we explored the potentials of advanced language representations: good initialization for item representations, zero-shot ability, and intention intention-aware aware ability. Possible future work will involve exploring the space relationship from both theoretical and multimodal perspectives. We believed that this paper sheds light on rethinking the connection between language modeling and user behavior modeling, benefiting both natural language processing and recommender system communities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported by the National Science and Technology Major Project (203320122102)) the NExT Research Center, National Natural Science Foundation of China (92270114).
REFERENCES
Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W. Bruce Croft. Learning hierarchical embedding model personalized product search. In SIGIR, 2017.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR (Workshop), 2017.
Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangman He. Tallrec: An effecctive and efficient tuning framework to align large language model with recommendation. In RecSys, 2023.
Keping Bi, Qingyao Ai, and W. Bruce Croft. transformer-basee embedding model for personal- ized product search. In SIGIR, 2020.
Xuheng Cai, Chao Huang, Lianghao Xia, and Xubin Ren. Lightgel: Simple yet effective graph contrastive learning for recommendation. In ICLR, 2023.
Jiawei Chen, Junkang Wu, Jiancan Wu, Xuezhi Cao, Sheng Zhou, and Xiangnan He. Adap-T Adaptively modulating embedding magnitude for recommendation. In WWW, 2023.
Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. On softmax direct preference optimization for recommendation. CoRR, abs/2406.09215, 2024.
Zheng Chen. PALR: personalization aware Ilms for recommendation. CoRR, abs/2305.00622. 2023.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations In RecSys, 2016.
Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pre- trained language models are open-ended recommender systems. CoRR, abs/2205.08084, 2022.
Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In RecSys, 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
Jean Dieudonne. Linear algebra and geometry. Hermann, 1969.
Chenlu Ding, Jiancan Wu, Yancheng Yuan, Jinda Lu, Kai Zhang, Alex Su, Xiang Wang. and Xiang- nan He. Unified parameter-efficient unlearning for llms. CoRR, abs/2412.00388 2024.
Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. Zero-shot recommender sys- tems. CoRR, abs/2105.0m fls/2105.08318. 2021.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning, arXiv preprint urXiv:2301.0023 00234, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshomn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny

---

Published as a conference paper at ICLR 2025
Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olame Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isahel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.2178.. 2024.
Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (Imms). CoRR, 0ass/2307.02044 2023.
Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chat-rec: To- wards interactive and explainable tins-augmented recommender system. CoRR, abs/2303. 2023.
Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong He, Liang Zhang, Fajic Yuan, Jun Zhou, and Linjian Mo. Breaking the length barrier: Lim-enhanced CTR prediction in long textual user behaviors. CoRR, abs/2403. 19347, 2024.
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5), In RecSys, 2022.
Wes Gurnee and Max Tegmark. Language models represent space and time. CoRR, abs/2310.020000 2023.
F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. 114):19 2016.
Jesse Harte, Wouter Zorgdrager, Panos Louridas, Asterios Katsifodimos, Dietmar Jannach, and Mar- ios Fragkoulis. Leveraging large language models for sequential recommendation. In RecSys, 2023.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collab- orative filtering. In www, 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In SIGIR, 2021.
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD, pp. 585-593. ACM, 2022.
Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben (eds.), WWW, 2023.
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J. McAuley, Bridging language and items for retrieval and recommendation. CoRR, has/22033.0355,, 2024a.
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. ECIR, 2024b.
Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How index item ids for rec- ommendation foundation models. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel (eds.), SIGIR-AP, 2023.
12

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anme Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas 2023.
Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR,
Wang-Cheng Kang and Julian J. McAuley, Self-attentive sequential recommendation. In ICDM, 2018.
Xiaoyu Kong, Jiancan Wu, An Zhang, Leheng Sheng. Hui Lin. Xiang Wang, and Xiangnan He. Customizing language models with instance-wise for sequential recommendation. CoRR, abs/2408. 10159, 2024.
Yehuda Koren, Robert M. Bell, and Chris Volinsky. Matrix factorization techniques for recom- mender systems. Computer, 42(8):30-37, 2009.
Yehuda Koren, Steffen Rendle, and Robert M. Bell. Advances in collaborative filtering. In Recom- mender Systems Handbook, pp. 91-142. Springer US, 2022.
Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In KDD, 2020.
Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: Meta-learned user preference estimator for cold-start recommendation. In KDD, 2019.
Yuxuan Lei, Jianxun Lian, Jing Yao, Xu Huang, Defu Lian, and Xing Xic. Recexplainer: Aligning large language models for explaining recommendation models. In KDD, 2024.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In ICLR, 2023a.
Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. Large language models for generative recom- mendation: A survey and visionary discussions. CoRR, abs/2309.01157, 2023b.
Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights. CoRR, abs/2305. 11700 2023c.
Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang. Ctrl: Connect tabular and language model for ctr prediction. arXiv preprint arXiv:2306.00 02841, 2023d.
Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In WwW, 2018
Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. Llara: Large aanguag-racoommendttio assistant. In SIGIR, 2024.
Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How recommender systems benefit from large language models: survey. CoRR, abs/2306.05817, 2023a.
Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong se- quential behavior comprehension in recommendation. CoRR, abs/2308. 11131, 2023b.
Junling Liu. Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang, Is chatgpt a good recommender? A preliminary study. CoRR, abs/2304. 10149, 2023a.
Qidong Liu, Jiaxi Hu, Yutian Xiao, Jingtong Gao, and Xiangyu Zhao. Multimodal recommender systems: A survey, CoRR, abs/2302.03883, 2023b.
Qidong Liu, Xian Wu, Wanyu Wang, Yejing Wang, Yuanshao Zhu, Xiangyu Zhao, Feng Tian, and Yefeng Zheng. Large language model empowered embedding generator for sequential recom- mendation. arXiv preprint arXiv:2409. /9925, 2024a.
13

---

Published as a conference paper at ICLR 2025
Qidong Liu, Xian Wu, Xiangyu Zhao, Yejing Wang. Zijian Zhang, Feng Tian, and Yefeng Zheng. Large language models enhanced sequential recommendation for long-tail user and item. arXiv preprint arXiv:2405.
2024b.
Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. ONCE: boosting content-based recom- mendation with both open- and closed-source large language models. In Luz Angelica Caudillo- Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vas- silvitskii (eds.), WSDM, 2024c.
Shuo Liu, An Zhang, Guoqing Hu, Hong Qian, and Tat-seng Chua. Preference diffusion for recom- mendation. arXiv preprint arXiv:24100 137 2024d.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 0tps//997.11100 2019.
Zhiming Mao, Huimin Wang, Yiming Du, and Kam-Fai Wong. Unitree: unified text-to-text transformer and joint contrastive learning framework for text-based recommendation. In ACL, 2023.
Julian J. McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and com- plementary products. In KDD, 2015.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih dirembboding-mmstrrlehhhaee Yavuz. Sfr-embedding mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. URL https: ptttss://blt //blog/// sa salesforee weeeearreeeaee research com/
ffr--embeddeddmmisttreee
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In ICLR, 2023.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: massive text embed- ding benchmark. In EACL, 2023.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qim- ing Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Pet- roski Such, Kenny Hsu, Madelcine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Pe- ter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training. CoRR, abs/2201. 10005, 2022.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In EMNLP, 2019.
OpenAl. GPT-4 technical report. CoRR, 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. CoRR, abs/2311.00666 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In ICLR, 2022.
Michael Pazzani and Daniel Billsus. Content-based recommendation systems. In Peter Brusilovsky, Alfred Kobsa, and Wolfgang Nejdl (eds.), TAW, 2007.
Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-BERT: pre-training user representations for improved recommendation. In Thirty-Fifih AAAI Conference Artificial Intelligence AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. 1. AAAI, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
14

---

Published as a conference paper at ICLR 2025
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy. Recommender systems with generative retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), NeurlPS, 2023.
Abhilasha Ravichander, Yonatan Belinkov, and Eduard H. Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In EACL, 2021.
Xubin Ren, Wei Wei, Lianghao Xia, and Chao Huang. A comprehensive survey on self-supervised learning for recommendation. arXiv preprint 24Xx::2400003354, 2024a.
Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. CoRR, abs/2310.15950, 2024b.
Steffen Rendle. Item recommendation from implicit feedback. In Recommender Systems Handbook. Springer US, 2022.
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars personalized ranking from implicit feedback. CoRR, abs/1 0t///20552220 205.2618, 2012.
BPR: bayesian
Guy Tennenholtz, Yinlam Chow, Chih-Wei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulepbergenov, Deepak Ramachandran, Martin Mladenov, and Craig Boutilier. Demystifying embedding spaces using large language models. In ICLR, 2024.
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR, abs/2310. 15213, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288 2023b.
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1807 507.00348, 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Arpita Vats, Vinija Jain, Rahul Raja, and Aman Chadha. Exploring the impact of large language models on recommender systems: An extensive review. CoRR, abs/2402. 185900 2024.
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. Probing pretrained language models for lexical semantics. In EMNLP, 2020.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, 2021.
15

---

Published as a conference paper at ICLR 2025
Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun. Se- quential recommender systems: Challenges, progress and prospects. In IJCAI, 2019a.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In SIGIR, 2019b.
Yuling Wang, Changxin Tian, Binbin Hu, Yanhua Yu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou, Liang Pang, and Xiao Wang. Can small language models be good reasoners for sequential recommen- dation? CoRR, abs/2403.04260, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang. Dawei Yin, and Chao Huang. Limrec: Large language models with graph augmentation for recommen- dation. In Luz Angelica Caudillo- Mata,, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii (eds.), WSDM, 2024.
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML, 2019.
Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self- supervised graph learning for recommendation. In SIGIR, 2021.
Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, and Xiangnan He. On the effectiveness of sampled softmax loss for item recommendation. CoRR, abs/2201.022 .02327, 2022.
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang. Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. CoRR, abs/2305. 19860, 2023a.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: A survey. ACM Comput. Surv. 55(5):97:1-97:33 2023b.
Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. Towards open-world recommendation with knowledge augmentation from large language models. CoRR, abs/2306. 10933, 2023.
Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Mingchen Cai, Wayne Xin Zhao, and Ji- Rong Wen. Prompting large language models for recommender systems: A comprehensive frame- work and empirical analysis. CoRR, abs/2401.04990 2024.
Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He. Large language model can interpret latent space of sequential recommender. CoRR, abs/2310.20487 2023.
Junliang Yu. Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. Are graph augmentations necessary?: Simple graph contrastive learning for recommendation. In SI- GIR, 2022.
Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui. Nguyen Quoe Viet Hung, and Hongzhi Yin. Xsimgel: Towards extremely simple graph contrastive learning for recommendation. IEEE Trans. Knowl. Data Eng. (6(2):913-926, 2024.
Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, Yu Xu, and Xiaohu Qie. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), NeurIPS, 2022.
Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? ID- vs. modality-basee recommender models revisited. In SIGIR, 2023.
16

---

Published as a conference paper at ICLR 2025
Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi. Actions speak louder than words: Trillion- parameter sequential transducers for generative recommendations. CoRR, abs/2402.17152, 2024.
An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua. Incorporating bias-aware margins into contrastive loss for collaborative filtering. In NeurIPS, 2022.
An Zhang. Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua. Empowering collaborative filtering with principled adversarial contrastive loss. In NewrIPS, 2023a.
An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. CoRR, abs/2310.10108, 2023b.
An Zhang, Jingnan Zheng, Xiang Wang, Yancheng Yuan, and Tat-Seng Chua. Invariant collaborative filtering to popularity distribution shift. In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 May 2023, 2023c.
Zhang, Wenchang Ma, Jingnan Zheng, Xiang Wang, and Tat-Seng Chua. Robust collaborative filtering to popularity distribution shift. ACM Trans. Inf. Syst., 2024a.
Chao Zhang, Shiwei Wu, Haoxin Zhang, Tong Xu, Yan Gao, Yao Hu, and Enhong Chen. Notellm: A retrievable large language model for note recommendation. In WWW, 2024b.
Jiaqi Zhang, Yu Cheng, Yongxin Ni, Yunzhu Pan, Zheng Yuan, Junchen Fu, Youhua Li, Jie Wang, and Fajie Yuan. Ninerec: A benchmark dataset suite for evaluating transferable recommendation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024c.
Junjie Zhang, Yupeng Hou, Ruobing Xie, Wenqi Sun, Julian J. McAuley, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Agentef: Collaborative learning with autonomous language agents for recommender systems. CoRR, abs/2310.09233., 20221..
Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Rec- ommendation as instruction following: A large language model empowered recommendation ap- proach. CoRR, abs/2305.00701. 2023e.
Lingzi Zhang, Xin Zhou, Zhiwei Zeng, and Zhiqi Shen. Are ID embeddings necessary? whitening pre-trained text embeddings for effective sequential recommendation. CoRR, abs/2402.10602, 2024d.
Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: In- tegrating collaborative embeddings into large language models for recommendation. CoRR, abs/2310.194883 2023f.
Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, and Jing Wang. LANE: logic alignment of non-tuning large language models and online recommendation systems for explainable reason generation. CoRR, abs/2407.02833 2024.
Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji- Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommenda- tion. CoRR, abs/231 .09049, 2023.
Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Jungi Jin, Han Li, and Kun Gai. Deep interest network for click- thick-through rate prediction. KDD. ACM, 2018.
Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. Cross-domain recommendation: Challenges, progress, and prospects. In IJCAI, 2021.
Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. CoRR, abs/2311.01343, 2023.
17

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Representations in LMs. The impressive capabilities demonstrated by LMs across various tasks raise a wide concern about what they have learned in the representation space. Linear methods (e.g, linear probing (Merullo et al., 2023) and linear mapping (Alain & Bengio, 2017)) are important and effective approaches for interpreting and analyzing representations of LMs (Ravichander et al,, 2021). The main idea of linear methods is simple: training linear classifiers to predict some specific attributes or concepts (e.g, lexical structure (Vulic et al., 2020) ) from the representations in the hidden layers of LMs, or transforming language representations into another feature space with linear matrix. A high result of linear methods (e.g., classification accuracy on the out-of-sample test set) tends to imply relevant information has been implicitly encoded in the representation space of LMs, although this does not imply LMs directly use these representations (Ravichander et al., 2021; Gurnee & Tegmark, 2023). Recent studies empirically demonstrate that concepts such as color (Patel & Pavlick, 2022), game states (Li et al., 2023a). and geographic position are encoded in LMs. Furthermore, these concepts may even be linearly encoded in the representation space of LMs (Li et al., 2023a; Park et al., 2023).
Collaborative filtering. Collaborative filtering (CF) (Ren et al., 2024a) an advanced technique in modern recommender systems. The prevailing CF methods tend to adopt ID-based paradigm, where users and items are typically represented as one-hot vectors, with an embedding table used for lookup (Koren et al., 2009). Usually, these embedding parameters are learned by optimizing specific loss functions to reconstruct the history interaction pattern (Rendle et al., 2012). Recent advances in CF mainly benefit from two aspects, graph convolution (Wu et al., 2023b) and contrastive learning (Ren et al., 2024a; Zhang et al., 2023c). These CF models exhibit superior recommendation perfor- mance by conducting the embedding propagation (Wang et al., 2019;; He et al., 2021) and applying contrastive learning objectives (Wu et al., 2021; Cai et al., 2023; Yu et al., 2024; Zhang et al., 2024a). However, although effective, these methods are still limited, due to the ID-based paradigm. Since one-hot vectors contain no feature information beyond being identifiers, it is challenging to transfer pre-trained ID embeddings to other domains (Hou et al., 2022) or to leverage leading techniques from computer vision (CV) and natural language processing (NLP) (Yuan et al., 2023).
LMs for recommendation. The remarkable language understanding and reasoning ability shown by LMs has attracted extensive attention in the field of recommendation. The application of LMs in recommendation can be categorized into three main approaches: L.M-enhanced recommendation, LM the modality encoder, and LM-as recommender. The first rescarch direction, LM-enhanced recommendation, focuses on empowering traditional recommenders with the semantic representa- tions from LMs (Xi et al., 2023; Ren et al., 2024b; Wei et al., 2024; Geng et al., 2024; Chen, 2023; Wang et al., 2024; Hou et al., 2023; Mao et al., 2023; Qiu et al., 2021; Zhang et al., 2024d; Liu al., 2024a;d). Specifically, these methods introduce representations from LMs as additional fea- tures for traditional ID-based recommenders, to capture complicated user preferences. The second research line lies in adopting the LM as the text modality encoder, which is also known as a kind of modality-basee based recommendation (MoRec) (Yuan et al., 2023; Li et al., 2023c). These methods tend to train the LM as the text modality encoder together with the traditional recommender. In previous studies, BERT-style LMs are widely used as the text modality encoder. The third research line fails in directly using LMS as the recommender and recommends items in a text generation paradigm. Early attempts focus on adopting in-context learning (ICL) (Dong et al., 2022) and prompting pre- trained LMs (Hou et al., 2024b; Liu al., 2023a; Dai et al., 2023; Gao et al., 2023). However, such naive methods tend to yield poor performance compared to traditional models. Therefore, recent studies concentrate on fine-tuning LMs on recommendation--caate et al., 2023e; Lin et al., 2023b; Cui et al., 2022; Liu et al., 2024c; Hua et al., 2023; Chen al., 2024; Ding et al., 2024) and align the LMs with the representations from traditional recommenders as the additional modality (Liao al., 2024; Zhang al., 2023f; Yang et al., 2023; Li et al., 2023d; Kong et al., 2024).
corpus (Bao et al., 2023; Zhang
B UNCOVERING COLLABORATIVE SIGNALS IN LMS VIA LINEAR MAPPING
B. BRIEF OF USED LMS
We briefly introduce the LMs we use for linear mapping in Section 3.1.
18

---

Published as a conference paper at ICLR 2025
BERT (Devlin et al., 2019) is an encoder-only language model based the transformer archi- tecture (Vaswani et al., 2017), pre-trained on text corpus with unsupervised tasks. BERT adopts bidirectional self-attention heads to learn bidirectional representations.
RoBERTa (Liu et al., 2019) is an enhanced version of BERT. RoBERTa preserves the architecture of BERT but improves by training with more data and large batches, adopting dynamic masking, and removing the next sentence prediction objective.
Llama2-7B (Touvron et al., 2023b) is an open-source decoder-only LLM with 7 billion param- eters. Llama2 adopts grouped-query attention, with longer context length and larger size of the pre-training corpus compared with Llama-7B (Touvron et al., 2023a).
Mistral-7B (Jiang al., 2023) is an open-source pre-trained decoder-only LLM with 7 billion parameters. Mistral 7B leverages grouped-query attention, coupled with sliding window attention for faster and lower cost inference.
text rext-embedding-ade- & text-embeddings--largg (Neelakantan et al., 2022) are leading text embedding models released by OpenAl. These models are built upon decoder-only GPT models, pre-trained on unsupervised data at scale.
SFR-E Embeddingg (Meng et al., 2024) is a decoder-based text embedding model built upon the open-source L.M Mixtral-7B (Jiang al., 2023). TRR-EmmeddiiggMistte introduces hask-homogeneous batching and computes contrastive loss on "hard negatives", which brings a better performance than the vanilla Mixtral-7B model.
B.2 GENERATING ITEM REPRESENTATION FROM LMS
We present how to extract representations from LMs. For encoder-based LMs (e.g.. BERT (Devlin al., 2019) and RoBERTa (Liu et al., 2019)), we use the representation of the last hidden state cor- responding to the [CLS] token (Hou et al., 2024a). For decoder-based models (e.g.. .. Llama-7B (Tou- vron et al., 2023b; Jiang et al., 2023), Mistral-7B, and AFFR-EmbeddinggMi -Mitrrll (Meng al., 2024)), we use the representation in the last transformer block (Vaswani et al., 2017). corresponding to the last input token (Gurnee & Tegmark, 2023; Todd et al., 2023; Neelakantan et al., 2022). Especially, for the commercial closed-source model (e.g., text-embedding-ae 2aaan- and text-embeddings---lee (Neelakantan et al., 2022)), we directly call the API interface to obtain representations.
B.3 CONNECTION WITH FINE-TUNED LLM-BASED APPROACHES FOR EXPLORING THE RELATIONSIIIP BETWEEN LANGUAGE SPACE AND BELIAIIR SPACE.
In exploring the space relationship between language modeling and user behavior modeling, there is another research line that uses LLMs interpret behavior representations (e.g., item representations trained from MF (Koren et al., 2009)) from traditional recommenders (Tennenholtz et al., 2024; Yang al., 2023; Lei al., 2024). They tend to project behavior representations into the token space of LLMs. and fine-tune LLMs to generate explanations for behavior representations (c.g., generate a movie description given a movie representation trained from MF). The fact that LLMs can interpret such behavior representations indicates that the behavior space and language space can be aligned the token space level, which similar to the findings in our paper. Our paper and these studies can be viewed as complementary works on similar research goal (i.e., exploration the connection between language modeling and user behavior modeling from the representation space perspective), providing support for each other.
C LEVERAGING LANGUAGE REPRESENTATIONS FOR BETTER RECOMMENDATION
C.1 DATASETS
We incorporate six datasets in this paper. including four datasets from the Amazon platform (Ni al., 2019) (i.e., Books, Movies TV, Video Games, and Industrial Scientific), and two datasets from other platforms (i.e,, MovieLens 1M and Book Crossing). Table 6 reports the dataset statistics.
'ttpps: / httpp//aaafffnnnm opena wwtpc/occcccccccccccccccccc demp///dddd//////gnigg/..gg.
www. ama zon, com
19

---

Published as a conference paper at ICLR 2025
Table 6: Dataset statisties.
Books
Movics & TV
Industrial & Scicutific
MovicLLnn-IMM
Book Crossing
#Users
71,306
26,073
40,834
15,141
6,040
6,273
Altems
26,073
12,464
14,344
5,163
3,043
5,335
#loteractions
2,209,030
876,027
390,013
82,578
995,492
253,057
Density
0,0008
0.0026
0,0007
0,0010
0.0542
0.0076
Item Title Examples
Dismissed with Prejudice: A J.P. Beaumont Novel; Die for Love: A Jacqueline Kirby Novel of Suspense; The Cloud; Memories Before and After the Sound of Music: An Autobiography, Harry Potter and the Sorcerer's 's Stone;
Books:
Movies & TV: Batman Begins; Fantastic Four, Max Headroom: The Complete Series; Madagascar, Land of the Dead; King Kong:
Video Games: USB Microphone for RockBand or Guitar Hero (PS3, Wii, Xbox360); Com- mand &amp: Conquer: Tiberian Sun PC; Tomb Raider III: Adventures of Lara Croft; Kartia: The Word of Fate; Snowbourd Kids; Command & amp; Conquer: Tiberian Sun PC; Final Fantusy VII; Grim Fandango PC: Half-Life- PPC;
MovieLens-IN 1M: Basquiat (1996); Tin Cup (1996); Godfather, The (1972); Supercop (1992);
Manny & Lo (1996); Bound (1996); Carpool (1996);
Book Crossing: Prague A Novel; Chocolate Jesus; Wie Barney es sieht; To Kill Mock- ingbird: Sturmzeit. Roman; A Soldier of the Great War; Pride and Prejudice (Dover Thrift
Editions);
Industrial & Scientific: Jurassic Perisphinctes Ammonites from France; FS9140; Spinosaurus Dinosaur Tooth 20-30mm;; FS9410: USA Eocene, Fossil Fish (Knightia alt), A A-grdde; Delta 50-857 Charcoal Filter for 50-868; Hitachi RP30SA 7-1/2 Gallon Stainless Steel Industrial Shop Vacuum (Discontinued by Manufacturer); Makita 632002-4 14-Inch Cut-Off Wheels (5-Pack) (Disconinued by Manufactwrer); CORTER-CABLE 740001801 4 1/2-Inch by 10yd 180 Grit Adhesive- Backed Sanding Roll;
Figure 5: Example of item titles.
We divide the history interaction of each user into training, validation, and testing sets with a ratio of 4:3:3, and remove users with less than 20 interactions following previous studies (Zhang et al., 2023b). We also remove items from the test and validation sets that do not appear in the training set, address the cold start problem.
In this paper, we only use the item titles as the text description. Figure 5 gives some item title examples from different datasets.
C.2 BASELINES
We incorporate a series of ID-based CF models as our baselines for general recommendation. These models are classified classical CF methods (MF, MultVAE, and LighIGCN), CL-based CF meth- ods (SGL, BC Loss, and XSimGCL), and LM-enhanced CF methods (KAR, RLMRec). We do not consider baselines using LMs as recommenders for two practical reasons: the huge inference cost on datasets with millions of interactions and the task limitation of candidate selection (Liao et al., 2024) or next item prediction (Zheng et al., 2023). For these LM-enhanced CF methods, we adopt the leading method XSimGCL the backbone. We use the same item and user representations as we adopt in AlphaRec for KAR and XSimGCI, to make the comparison relatively fair. To evaluate the impact of the modifications, we demonstrate ablation study on RLMRec in Appendix C.5.
MF (Koren al., 2009; Rendle al., 2012) is the most basic CF model. It denotes users and items with ID-based embeddings and conducts matrix factorization with Bayesian personalized ranking (BPR) loss.
MultVAE (Liang et al., 2018) is a traditional CF model based on the variational autoencoder (VAE). It regards the item recommendation as generative process from multinomial distribution
20

---

Published as a conference paper at ICLR 2025
Rrealiy NDCGGg22
NDCGg.20
0.11
0.69
0.07
Uler Mapping W1.
ML.P
GEN
AphaRes
Ulerr Mapplag Wis CL
MLLP
Wia GCN
Apaaltec
(a) Ablation study on Movies & TV
(b) Ablation study on Video Games
Figure 6: Ablation study
and uses variational inference to estimate parameters. We adopt the same model structure as suggested in the paper: 600 - 200 + 600.
LightGCN et 2021) is light graph convolution network tailored for the recommendation, which deletes redundant feature transformation and activation function in NGCF (Wang et al., 2019b).
SGL (Wu et al., 2021) introduces graph contrastive learning into recommender models for the first time. By employing node or edge dropout generate augmented graph views and conduct contrastive learning between two views, SGL achieves better performance than LightGCN
BC Loss (Zhang et al., 2022) introduces a robust and model-agnostic contrastive loss, handling various data biases in recommendation, especially for popularity bias.
XSimGCL (Yu et al., 2024) directly generates augmented views by adding noise into the inner layer of LightGCN without graph augmentation. The simplicity of XSimGCL leads to faster convergence speed and better performance.
KAR (Xi et al., 2023) enhances recommender models by integrating knowledge from LMs. It gen- erates textual descriptions of users and items and combine the LM representations with traditional recommenders using hybrid-exped adaptor.
RLMRec (Ren et al., 2024b) aligns semantic representations users and items with the repre- sentations in CF models through contrastive loss, additional loss trained together with the CF model. The fusion of semantic information and collaborative information brings performance improvement.
C.3 ABLATION STUDY
We conduct the same ablation study as introduced in Section on Movies & TV and Video Games datasets. As illustrated in Figure 6, each component in AlphaRec contributes positively, which is consistent with our findings in Section 4. Specifically, the performance degradation caused by replacing the MLP with linear weight matrix (w/o MLP) indicates that nonlinear transformations can extract the implicit user preference similarities encoded in the language representation space more effectively. Besides, the performance also drops from replacing InfoNCE loss (Wu et al., 2022) with BPR loss (Rendle et al., 2012) (w/o CL) and removing the graph convolution (w/o GCN) suggests that explicitly modeling the collaborative relationships through the loss function and model architecture can further enhance recommendation performance.
C.4 EFFECT OF VARYING LANGUAGE REPRESENTATIONS IN ALPHAREC
We evaluate the effect of different language representations in AlphaRec by varying adopted LMs. As reported in Table 7, advanced language representations consistently yield higher performance than early BERT-style language representations. Moreover, due to the non- non-linearity and neighbor- hood aggregation introduced in AlphaRec, the performance gap between different language repre- sentations is narrowed.
21

---

Published as a conference paper at ICLR 2025
Table 7: Effect of varying language representations in AlphaRec.
Movies & TV
Video Games
Books
Recall
NDXC
HR
Recall
NDCG
HR
Recall
NDCG
HR
BERT
0.0994
0.0923
0.4873
0.0960
0.0550
0.2179
0.0719
0.0607
0.3434
RoBERTa
0.0967
0.0895
0.4793
0.0947
0.0545
0.2167
0.0710
0.0596
0.3386
AlphaRee
Llama2-7B
0.1
0.1092
0.5388
0.1395
0.0817
0,3003
0.0940
0.0793
0.4081
Mistrul-7B
0.1161
0,1097
0,5421
0.1413
0.0828
0.3020
0.0945
0.0799
0.4090
fext-mmbedding-dde--aa
0.1152
0.1083
0.5382
0.1437
0.0844
0.3062
0.0933
0.0784
0.4061
lext-embeukdings- lange
0.1221
0.1144
0.5587
0.1519
0.0894
0.3207
0.0991
0.0828
0.4185
SFR Fiii---dddding-- Minhedding-Misttall Mistral
0.1225
0.1139
0.5571
0.1521
0.0887
0.3209
0.0982
0.0820
0.4161
Table 8: Performance comparison of different versions of RLMRec
Movies & TV
Video Games
Books
Recall
NDCG
HR
Recall
NDCG
HR
Recall
NDCG
HR
XSimGCL
0.1057
0.0984
0.5128
0,1138
0,0662
0,2550
0.0879
0.0745
0.3918
RI MRee (.I.M-generated profile)
0.1046
0.0942
0.5063
0.1218
0.0696
0.2692
0.0905
0.0741
0.4049
RLMRec (This paper)
0.1119
0.1013
0.5301
0.1384
0.0809
0.2997
0.0928
0.0774
0.4092
C.5 ABLATION STUDY ON THE MODIFICATION OF RLMREC
We have modified RLMRee (Ren et al., 2024b) by using the same item and user representations as in AlphaRec to make the comparison more fair. rather than using representations of LLM --geeeerrtedd item and user profiles. To evaluate the effectiveness of this modification, we conduct an ablation study on the adopted item and user representations. Specifically, implement another version of RLMRec (Ren et al., 2024b) by using the original prompting approach to generate item and user profile, and name this version as RLMRec (LLM-generated profile). As shown in Table 8, the performance of RLMRec (LLM-generated profile) is lower than the implementation in this paper. Furthermore, the performance of incorporating LLM-generated profiles is not always stable, which may even lead to performance degradation in some cases (e.g.. Movies & TV). We attribute the performance gap to the possible noise and hallucination introduced by these generated profiles.
C.6 PERFORMANCE ON AMAZON ELECTRONIC 2023
To alleviate the information leakage problem that the early datasets may be used in the pretraining data of LMs, we consider the latest version of Amazon 2023 (Hou et al., 2024a) as the newly published dataset. This dataset was published in March 2024, which is the latest public dataset that we have access to. To further alleviate the information leakage problem, we only consider the interactions after 2022. We conduct the same experiment as in Section 4 this new dataset, and present the performance in Table 9. As shown in this table. in this new dataset, AlphaRec still consistently outperforms baselines. We attribute this to the fact that language models understand the user preference behind item title description, rather than naively remembering the training data. Therefore, AlphaRec still works fine on the latest dataset.
C.7 THE T-SNE VISUALIZATION COMPARISON
In this section, aim to intuitively explore how the MLP in AlphaRec further helps in excavating collaborative signals in language representations, compared to the linear mapping matrix. We visu- alize the item representations from LMs, post-mapping representations from AlphaRec (w/o MLP), and post-mapping representations from AlphaRec in Figure 7, where AlphaRec (w/o MLP) denotes replacing the MLP with a linear mapping matrix. We observed that movies about superhero and monster cluster in all representation spaces, indicating both AlphaRec (w/o MLP) and AlphaRec capture the preference similarities between these items and preserve the clustering relationship. The difference between AlphaRec (w/o MLP) and AlphaRec lies in the ability to capture obscure pref- erence similarities among items. As shown in Figure 7a, homosexual movies are dispersed in the language space, indicating the possible semantic differences between them. AlphaRec successfully captures the preference similarities and gathers these items in the representation space, while Al- phaRec (w/o MLP) remains some items dispersed. Moreover, AlphaRec outperforms AlphaRec (w/o MLP) by a large margin, as indicated Figure 6a. These results indicate that AlphaRec ex- hibits more fine- greinnd preference capture ability with the help of nonlinear transformation.
22

---

Published as a conference paper at ICLR 2025
Table 9: Performance of AlphaRec on Amazon Electronics 2023
Amazon Electronics 2023
Recall
NDCG
MF (Rendle al., 2012)
0.0130
0.0089
MultVAE (Liang et al. 2018)
0.0227
0.0158
0.0237
LightGCN (He et al,, 2021)
0.0237
0.0161
0.0218
SGL (Wu et al., 2021)
0.0519
0.0250
0.0551
BC Loss (Zhang
2022)
0.0548
0.0265
0.0585
XSinGCL (Yu et al., 2024)
0.0534
0.0261
0.0569
KAR al., 2023)
0.0611
0.0283
RI.MRer (Ren 2024b)
0.0633
0,0288
0.0674
AlphaRec
0.0687
0,0323
0.0732
Seperhere-res tree Van
Seper here Spider Mao
Mensarking
-40
Seperher-Spperma
Saper here Batmas
-40
Meester Coddlla
Montre Keng
Mansee King Kong
(a) LM representations
(b) AlphaRec (w/o MLP)
(c) AlphaRec
Figure 7: The t-SNE visualization of representations on Movies & TV. (7a) The item representations the LM space. (7b) The item representations obtained by replacing the MLP with linear mapping matrix in AlphaRec. (7c) The item representations obtained from AlphaRec.
D EXPLORING POTENTIALS OF LANGUAGE REPRESENTATIONS FOR RECOMMENDATION
D.1 FAST CONVERGENCE SPEED
We report the training cost of AlphaRec in this section. Table 10 reports the seconds needed per epoch and the total training cost until convergence. Here Amazon-Mix denotes the mixed dataset of Books, Movies TV, and Video Games. It's worth noting that AlphaRec converges quickly and only requires small amount of training time.
D.2 ZERO-SHOTT ABILITY
D.2.1 CO-TRAINING ON MULTIPLE DATASETS
Co-training multiple datasets is similar to training one single dataset, where the only difference lies in the negative sampling. When co-training on multiple datasets, the negative items are restricted to the same dataset as the positive item rather than the full item pool. The other training procedures remain the same with training on one single dataset.
D.2.2 BASELINES
Since previous works about zero-shot recommendation mostly focus sequential recommendation (Kang & McAuley, 2018; Wang et al., 2019a), we slightly modify three methods in sequential recommendation, ZESRec (Ding al., 2021). UniSRec (Hou et al., 2022), and VQ-Ree (Hou al., 2023) as our baselines. Specifically, we maintain the model structure as provided in the paper, and adopt the training paradigm of CF.
Random denotes randomly recommending items from the entire item pool.
23

---

Published as a conference paper at ICLR 2025
Table 10: Training cost of AlphaRec (seconds per epoch/in total).
Books
Movies
Video Games
Amazon-Mix
AlphaRec 40.1 1363.4
12.3 /47.7
2.4/
57888/
Pop denotes randomly recommending from the most popular items. Here popularity denotes the number of users that have interacted with the item.
ZESRec (Ding et al., 2021) is the first work that defines the problem of zero-shot recommendation. To address this problem. this work introduces hierarchical Bayesian model with representations from the pre-trained BERT.
UniSRec (Hou al., 2022) aims learn universal item representations from BERT, with paramet- ric whitening and a MoB-enhanced adaptor. By pre-training on multiple source datasets, UniS- Rec can conduct zero-shot recommendation on various datasets a transductive or inductive paradigm.
VQ-Rec (Hou et al., 2023) learns vector-quantized representations for items, enabling transfer- ring between datasets. A codebook is learned from language representations, for looking up item embeddings.
D.2.3
COMPARISON WITH LLMRANK
Table 11: Zero-shot performance comparison with LLMRank
MovieLens-IM
Steam
NDCG@
NDCG@5
NDCG@110
NDCG@20
NDCG@1
NDCG@s
NDCG@ 10
NDCG@20
LLMRank
0.2485
0.5302
AlphaRec
0.3919
0.6038
0.6543
0.6672
0.4450
0.6131
0.6394
0.6714
Imp
57.71%
Table 11 illustrates the zero-shot recommendation performance compared with the LLM4Rec method LLMRank. We adopt the same setting of LLMRank, equipping 19 negative items for each positive item, and evaluate the NDCG the candidate set. AlphaRec exhibits excellent zero-shot performance, significantly surpassing LLMRank. Morcover, the improvement over LLMRank ex- hibits a rising trend the K of NDCG decreases.
D.2.4 THE EFFECT OF TRAINING DATASETS
The effect the training dataset scale on zero-shot recommendation. We report the zero-shot recommendation performance differences trained different datasets in Table 12. Here AlphaRec (trained on Books) denotes training on a single Books dataset, while AlphaRec (trained on mixed dataset) denotes co-training on three Amazon datasets. Generally, training on more datasets leads to a better zero-shot performance. In addition, we observe that, for the zero-shot performance on untrained target datasets, training datasets with similar themes contribute more (e.g., Movies & TV and MovieLens-1MM
The performance comparison between training on the single dataset and the mixed dataset. In Table 13, AlphaRec (trained on single dataset) denotes training and testing on the same single dataset, while AlphaRec (trained on mixed dataset) denotes training on three Amazon datasets (i.e,, Books, Movies & TV, and Video Games) and testing on one single dataset. Generally, co-training on three Amazon datasets yields similar performance compared with training on one single dataset. The only exception lies in Video Games, which shows some performance degradation. We attribute this to the difference between the selection of т. We use т = 0.15 when trained on the mixed dataset, while the optimal 7 for Vidco Games lies around 0.2. These results indicate that single AlphaRec can capture user preferences among various datasets, showcasing a general collaborative signal capture ability.
24

---

Published as a conference paper at ICLR 2025
Table 12: The effect of the training dataset on zero-shot recommendation
Industrial & Scientific
MovieLens- IM
Book Crossing
Recall NDCG
HR
Recall
NDCG
HR
Recall
NDCG
HR
AlphaRec (trained Books)
0.0896
0.0562
0.1256
0.1218
0.2619
0.8942
0.0646
0.0532
0.3346
AlphaRec (trained Movies TV)
0.0909
0.0581
0.1266
0.1438
0.3122
0.9200
0.0471
0.0406
0.2600
AlphaRec (trained Video Games)
0.0905
0.0567
0.1225
0.1221
0.2313
0,9034
0,0412
0.0378
0.2585
AlphaRec (trained mixed dataset)
0.0913
0.0573
0.1277
0.1486
0.3215
0.9296
0.0660
0.0545
0.3381
Table 13: Performance comparison between training on the single dataset and the mixed dataset
Books NDCG
Movies & TV
Video Games
Recall
HR
Recall
NDCG
HR
Recall
NDCG
HR
AlphaRec (trained on single dataset)
0.0991
0.0828
0.4185
0.1221
0.1144
0.5887
0.1519
0.0894
0.3207
AlphaRec (trained on mixed dataset)
0.0979
0.0818
0.4147
0.1194
0.1107
0.5463
0.1381
0.0827
0.2985
D.3 INTENTION-AWARE ABILITY
D.3.1 INTENTION QUERY GENERATION
The user intention query a natural language sentence implying the target item of interest. For each item in the dataset, we generate fixed user intention query. Following the previous work (Hou et al., 2024a), we generate user intention queries with the help of ChatGPT (OpenAI, 2023). As shown in Figure 8, prompt ChatGPT in Chain-of- Chain-of-Thought (CoT) (Wei et al., 2022) paradigm and adopt the output as the user intention query. We adopt a rule-based strategy to ensure the quality of generated queries, and regenerate the wrong query. Considering the huge amount of item title text, we use ChatGPT3.5 API for generating all queries for the budget's 's sake.
D.3.2 BASELINE
AlphaRec exhibits user intention capture abilities, although not specially designed for search tasks. We compare AlphaRec with TEM (Bi et al., 2020) which falls the field of personalized search (Ai et al., 2017; McAuley et al., 2015).
TEM (Bi et al., 2020) uses transformer to encode the intention query together with user history behaviors, which enables achieve better search results by considering the user's historical interest.
D.3.3 CASE STUDY
We conduct two more case studies to verify the user intention capture ability of AlphaRec. As illustrated in Figure 9 and Figure 10, AlphaRec provides better recommendation results, assigning the target item at the top while maintaining the general user preferences.
D.3.4 EFFECT OF THE INTENTION STRENGTH ALPHA
The value of controls the balance between the user's historical interests and the user intention query. A larger incorporates more about the user intention while considering less about the user's historical interests. As shown in Figure 11, the effect a on Video Games shows a similar trend with Mo
E HYPERPARAMETER SETTINGS AND IMPLEMENTATION DETAILS
We conduct all the experiments in PyTorch with a single NVIDIA RTX A5000 (24G) GPU and a 64 AMD EPYC 7543 32-Core Processor CPU. We optimize all methods with the Adam optimizer. For all ID-ed CF methods, we set the layer numbers of graph propagation by default at 2, with the embedding size 64 and the size of sampled negative items SS, as 256. We use the early stop strategy to avoid overfitting. We stop the training process if the Recall @ 20 metric on the validation set does not increase for 20 successive evaluations. In AlphaRec, the dimensions of the input and
25

---

Published as a conference paper at ICLR 2025
Intention Query Generation
Input
You are an expert in generating queries for target movie. Please help me generate the most suitable query for the target movie within one sentence, following the given example.
Example:
TARGET: BUG-A-SALT 3.0 Black Fly Edition.
QUERY: I want a gun that can use while gardening to get rid of stink bugs, ants, flies, and spiders in my house. It needs to be amazing and help me feel less scared.
TARGET: Toy Story (1995).
Output
QUERY: I'm looking for heartwarming animated movie that follows the adventures of group of toys who come to life when their owner is not around.
Figure 8: Example of item query generation.
User Intention: I'm looking for quirky superhero movie with comedic twist that features a group of misfit heroes with unusual powers trying the day.
Target: Mystery Men (1999)
Recommendation List (w/o Intention)
Recommendation List (w Intention)
American Pic (1999)
Mystery Men (1999)
Bug's Life, A (1998)
American Pic (1999)
General's Daughter, The (1999)
Bug's Life, A (1998)
Double Jeopardy (1999)
Election (1999)
Election (1999)
Wild Wild West (1999)
Figure 9: Case study of user intention capture on MovieLens 1M
output in the two-layer MLP are 3072 and 64 respectively, with the hidden layer dimension as 1536. We apply the all-ranking strategy (Krichene & Rendle, 2020) for all experiments, which ranks all items except positive ones in the training set for each user. We search hyperparameters for baselines according to the suggestion in the literature. The hyperparameter search space is reported in Table 14. For these LM-enhanced models, KAR and RLMRec, we also search the hyperparameter of their backbone XSimGCL.
For AlphaRec, the only hyperparameter is the temperature and search it in [0.05, 2]. We report the temperature т we used for each dataset Table 15. For the mixed dataset Amazon-Mix in Section 5.2, user intention capture experiment in Section 5.3.
universal 0.15. We adopt 0.2 for the MovieLens- 1M dataset for the
looking classic game where explore gothic world filled with vampires, werewolves, and other supernatural creatures while wielding variety of unique weapons.
User Intention:
astlevanin COLLECTION
Target: Castlevania
Recommendation List (w/o Intention)
Recommendation List Intention)
inFAMOUS Playstation
Castlevania
Assassin's Creed Playstation
Hcavy Rain Greatest Hits
Assassin's Creed II Greatest Hits edition
Assassin's Creed Playstation 3
Playstation 3
inFAMOUS Playstation 3
Uncharted Among Thieves Playstation
Assassin's Creed Greatest Hits edition
Heavy Rain- Greatest Hits
Playstation
Figure 10: Case study of user intention capture Video Games
26

---

Published as a conference paper at ICLR 2025
628
618
NDCGSS
Value
Figure 11: Effect of on Video Games
Table 14: Hyperparameters search space for baselines.
In [ee Hyperparameter space
MF & LighiGCN
1e-3)
MultVAE
dropout ratio 0.2, 0.5), (0.2.
(0.2 0.4. 0.6, 0.8]
SGL
T [0.05, 2]. A ~ {0.005, 0.01, 0.05, 0.1, 0.5, P~ (0.0,,,,000000000,2,,00,00000000
BC Loss
~ [0.05, 31, [0.05, 2]
XSimGCL
[0.05, 21, {0.01, 0.05, 0.1, 0.1, 0.2, 0.5, 1.0). A~ {0.005, 0.01, 0.010 0.05, 0.1, 0.5, 1.0), [+=1
KAR
No. shared experts (3, 4. 5). No. preference experts ~ (4.5)
RLMRec
kd weight ~ [0.05, 2). kd temperature ~ (0.01, 0.05, 0.1, 0.15, 0.2. 0.5. 1J
ZESRee
A {0.01, 0.05, 0.1, 0.5, 1.0). A, (0.01, 0.05, 0.1, 0.5, 1.0)
UniSRec
Ir (3e-4, Ie-3, 3e-3, 1e-2]
TEM
1~ (2,3), head A {4, 8}
AlphaRec
T~ [0.05.
Table 15: The hyperparameters of AlphaRec
Books
Movies & TV
Video Games
Amazin Mix
0.15
0.15
0.2
0.15
Table 16: Cost for extracting language representations on Amazon Movie & TV
per 10,000 items
Time Cost
Money Cost
Llama2-7B next-embedding-3-bel l-lage
5 mins
0 S
40
0.17
27


Reviews:
[{"review_id": "PIzzwOhAyP", "reviewer": "Reviewer_V7NB", "summary": "This paper explores the relationship between language space and behavior space in the context of recommendation systems, investigating the potential of using language representations for this purpose. The proposed method, AlphaRec, demonstrates excellent performance in zero-shot recommendation scenarios.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-written and easy to follow.\n2. Leveraging textual information for recommendations and exploring zero-shot performance is both novel and interesting.\n3. The experiments are comprehensive and effectively demonstrate the proposed method's effectiveness.", "weaknesses": "1. It would be beneficial to conduct an additional ablation study to evaluate different types of text embedding models on AlphaRec.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OSy7xpk3t", "reviewer": "Reviewer_QRqF", "summary": "The paper challenges the idea that Language Models (LMs) and traditional recommenders exist in separate representation spaces. It proposes deriving a recommendation space directly from language representations. By mapping item representations from advanced LM outputs, it shows improved recommendation performance, hinting at a homomorphism between language and item representation spaces. The paper introduces collaborative filtering models based solely on language representations, excluding ID-based embeddings. Components like multilayer perceptrons, graph convolution, and contrastive learning are integrated into the model.", "questions": "Please refer to the issues mentioned in the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose using linear mapping of LLMs' representations to directly model user preferences, which is both efficient and effective.\n\n2. The authors conducted comprehensive experiments discussing the performance of AlphaRec in full-shot, zero-shot, and other scenarios, comparing it with baseline methods from various research lines.", "weaknesses": "1. From Table 1, it can be observed that the performance of AlphaRec is heavily dependent on the performance of the chosen language model. When selecting smaller language models (such as BERT, RoBERTa, etc.), the performance is less than ideal. Additionally, the costs associated with state-of-the-art methods (i.e., SFR-Embedding-Mistral, text-embeddings-3-large, etc.), including inference costs or API call expenses, cannot be overlooked, which weakens the applicability of this method.\n\n2. The authors omitted a method that performs well in zero-shot scenarios, VQ-Rec [1], when exploring zero-shot capabilities.\n\n3. In the ablation study section, it can be observed that AlphaRec without Contrastive Learning (CL) performs significantly worse than Linear Mapping, indicating that simply adding MLP and GCN on top of Linear Mapping does not lead to effective performance improvements.\n\n4. If LLMs' representations and MLP are applied to state-of-the-art ID-based methods (e.g., XSimGCL) to replace the original ID embeddings, would it achieve results comparable to those of AlphaRec?\n\n[1] Hou, Yupeng, et al. \"Learning vector-quantized item representation for transferable sequential recommenders.\" Proceedings of the ACM Web Conference 2023. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0yWQVoDoTT", "reviewer": "Reviewer_uzkZ", "summary": "This paper studies the potential of leveraging advanced language models (LMs) for recommendation tasks, exploring whether language representations can encode user preferences and collaborative signals. Traditionally, CF-based recommendation systems use ID-based embeddings to model user-item interactions. The authors propose a method that directly maps language representations of item metadata (like titles) into a behavior space to serve as effective item representations. They introduce a linear mapping approach, followed by the AlphaRec model, which combines language embeddings with collaborative filtering elements such as graph convolution and contrastive loss.\n\nGenerally, this is a very good paper provides new insights in the community: homomorphism exists between LM- based language space and user behavior space for recommendation.", "questions": "The discussion on the potential of intent-aware ability is interesting. However, as indicated in Appendix D.3.1, a fixed user intention query is generated for each item in the dataset. I would argue that this is not optimal, as different users may have different intents even when consuming the same item. Could the authors consider extending their approach to allow for dynamic query generation or user-specific intent modeling, potentially improving recommendation relevance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The topic on studying the relationship between the language space and behavior space is important to the community. The authors provide new insights to show the homomorphism between the two spaces.\n\n2. The paper’s core contribution, mapping language representations into a behavior space, is thoroughly tested and demonstrates strong empirical performance. This novel approach could reduce reliance on large datasets of historical behavior data, making the method highly applicable.\n\n3. Followed by the observation through linear mapping, the authors also propose a stronger model, AlphaRec, through non-linear mapping and GNN. Its ability to perform zero-shot recommendations and its efficient training process are well-demonstrated in multiple datasets, addressing common challenges in recommendation systems where user or item IDs change frequently. \n\n4. The paper’s evaluation covers a range of experimental setups, comparing multiple LMs, testing robustness to input noise, and conducting ablation studies. The discussion on the potential of language-based user representation (Section 5) is promising.", "weaknesses": "1. While the paper presents a compelling case for using language representations in recommendation, a discussion on how AlphaRec compares to existing fine-tuned LLM approaches would strengthen it further. For example, it would be helpful to understand whether the authors see AlphaRec as complementary or competitive with fine-tuned LLM-based approaches (e.g., [1]), and if they anticipate similar benefits, limitations, or unique strengths compared to those models.\n\n[1] Tennenholtz et al. Demystifying Embedding Spaces using Large Language Models. ICLR 2024.\n\n\n2. The authors assume that a single linear mapping can capture collaborative signals for all users, which may overlook the diversity in individual user preferences and linguistic nuances. Different users might interpret language in varied ways, and a one-size-fits-all mapping could limit the model’s ability to fully capture such personalization and diversity. How might the authors address this challenge within their framework? Exploring the possibility of user-specific mappings or personalization approaches could be a valuable direction for future work, potentially enhancing the model’s adaptability across diverse user groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMb9xpVQqp", "reviewer": "Reviewer_m7fA", "summary": "This paper explores the relationship between the language space generated by LLMs and the behavior space produced by CF models. The authors found that the two spaces exhibit a pattern of homomorphism. Based on this finding, they proposed AlphaRec, which uses item embeddings exclusively from the language space and employs a trainable projector to map them into the behavior space for recommendations. The results demonstrate that with language space-initialized embeddings, the model significantly outperforms traditional ID-based methods and other LLM-based approaches in general recommendation, zero-shot recommendation, and intention-aware recommendation.", "questions": "- Since MovieLens and Amazon Review are classical and widely used datasets publicly available on the Internet, I am concerned that they might already be part of the pretraining data for some LLMs, hence the user preference information encoded in LMs could be a potential data leakage. This could explain why BERT-style models perform worse than more recent LMs. To verify this assumption, can authors evaluate AlphaRec’s performance on a private or newly published recommendation dataset?\n\n- If authors want to claim the collaborative signals are already encoded in the language space, I wonder if authors can show the results using original language representations without any linear or nonlinear transformations? If there is valid information can show user behaviour in language space, the performance should be comparable to some traditional CF methods (like MF) or heuristic methods (e.g., Pop).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The questions explored in this paper are novel and meaningful. While many existing works have analyzed the recommendation capabilities of language models, very few have empirically investigated the differences between the representation spaces of LMs and CF models.\n\n- The evaluations are comprehensive. The authors not only investigate the recommendation capabilities of AlphaRec but also examine its performance in zero-shot and intention-aware recommendations. Additionally, the visualization of the LM and behavior spaces is a plus.", "weaknesses": "- There are some concerns that affect the soundness and the conclusions presented in the paper (see point 2 of weakness and questions below).\n\n- Some of the comparisons are not entirely fair. For example, in Figure 3 and Table 3, it is evident that when using BPR loss (which is used in LightGCN), AlphaRec’s performance is a bit worse than that of LightGCN. Given that AlphaRec shares a similar graph convolution architecture with LightGCN in its nonlinear projection, it appears that AlphaRec’s advantage over LightGCN comes from the InfoNCE loss rather than the language representations. I am wondering whether such result can support the power of language representations. By the way, I suggest aligning the implementations of KAR and RLMRec closer with AlphaRec, including the loss function and some of the modules. Without a fair comparison, it is hard to conclude that the combination of ID-based embeddings and language representations is less suitable than using pure language representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)