Title: Language Models Encode Collaborative Signals in Recommendation
Venue: NeurIPS 2024
Accepted: no
Keywords: Collaborative filtering

Language-representation-based recommendation

Language models

Language model representations
Abstract: Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. 
However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to the prevailing understanding that LMs and traditional recommender models learn two distinct representation spaces due to a huge gap in language and behavior modeling objectives, this work rethinks such understanding and explores extracting a recommendation space directly from the language representation space.
Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. 
This outcome suggests a homomorphic relationship between the language representation space and an effective recommendation space, implying that collaborative signals may indeed be encoded within advanced LMs.
Motivated by these findings, we propose a simple yet effective collaborative filtering (CF) model named AlphaRec, which utilizes language representations of item textual metadata (e.g., titles) instead of traditional ID-based embeddings. 
Specifically, AlphaRec is comprised of three main components: a multilayer perceptron (MLP), graph convolution, and contrastive learning (CL) loss function, making it extremely easy to implement and train. 
Our empirical results show that AlphaRec outperforms leading ID-based CF models on multiple datasets, marking the first instance of such a recommender with text embeddings achieving this level of performance.
Moreover, AlphaRec introduces a new text-based CF paradigm with several desirable advantages: being easy to implement, lightweight, rapid convergence, superior zero-shot recommendation abilities in new domains, and being aware of user intention.

Abstract

Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields. However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to the prevailing understanding that LMs and traditional recommender models learn two distinct representation spaces due to a huge gap in language and behavior modeling objectives, this work rethinks such understanding and explores extracting a recommendation space directly from the language representation space. Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests a homomorphic relationship between the language representation space and an effective recommendation space, implying that collaborative signals may indeed be encoded within advanced LMs. Motivated by these findings, we propose a simple yet effective collaborative filtering (CF) model named AlphaRec , which utilizes language representations of item textual metadata ( e.g., titles) instead of traditional ID-based embeddings. Specifically, AlphaRec is comprised of three main components: a multilayer perceptron (MLP), graph convolution, and contrastive learning (CL) loss function, making it extremely easy to implement and train. Our empirical results show that AlphaRec outperforms leading ID-based CF models on multiple datasets, marking the first instance of such a recommender with text embeddings achieving this level of performance. Moreover, AlphaRec introduces a new text-based CF paradigm with several desirable advantages: being easy to implement, lightweight, rapid convergence, superior zero-shot recommendation abilities in new domains, and being aware of user intention.

1 Introduction

Language models (LMs) have achieved great success across various domains [3-7], prompting a critical question about the knowledge encoded within their representation spaces. Recent studies empirically find that LMs extend beyond semantic understanding to encode comprehensive world knowledge about various domains, including game states [8], lexical attributes [9], and even concepts of space and time [10] through language modeling. However, in the domain of recommendation where the integration of LMs is attracting widespread interest [11-15], it remains unclear whether LMs inherently encode relevant information on user preferences and behaviors. One possible reason is the significant difference between the objectives of language modeling for LMs and user behavior modeling for recommenders [16-19].

Currently, one prevailing understanding holds that general LMs and traditional recommenders 35 encode two distinct representation spaces: the language space and the recommendation space 36 ( i.e., user and item representation space), each offering potential enhancements to the other for 37

(a) Linearly mapping language representations into the recommendation space

CosineSimilarity

Item Title;→

LM

Linear

Average

ei

Language

Item

HistoryItem

User

Representation

RepresentationiH

Representations

Representation

UserIntention

"I want to watch some

BATMAN

Superhero: Iron Man

0.12 1

Recall@20

Marvel superhero movies."

0.10

NDCG@20

“Batman:BadBlood"

Superhero: Batman

Monster: Godzilla

Monster: King Kong

800

Homosexual Movies

User Intention

Linear

0.06

Mapping

“MyBeautiful Laundrette"

0.04

0.02

“Foodof Love"

0.00

MF

LightGCN

Recommendation Space

(b) Performance comparison

(c) The t-SNE representations of movies and user intention in two spaces.

recommendation tasks [17, 20]. On the one hand, when using LMs as recommenders, aligning the 38 language space with the recommendation space could significantly improve the performance of 39 LM-based recommendation [14, 21-23]. Various alignment strategies are proposed, including fine40 tuning LMs with recommendation data [15, 16, 24-26], incorporating embeddings from traditional 41 recommenders as a new modality of LMs [17, 20, 27], and extending the vocabulary of LMs with item 42 tokens [18, 19, 28-31]. On the other hand, when using LMs as the enhancer, traditional recommenders 43 greatly benefit from from leveraging text representations [32-45], semantic and reasoning information 44 [46-49], and generated user behaviors [50, 51]. Despite these efforts, explicit explorations of the 45 relationship between language and recommendation spaces remain largely unexplored. 46

In this work, we rethink the prevailing understanding and explore whether LMs inherently encode 47 user preferences through language modeling. Specifically, we test the possibility of directly deriving a 48 recommendation space from the language representation space, assessing whether the representations 49 of item textual metadata ( e.g., titles) obtained from LMs can independently achieve satisfactory 50 recommendation performance. Positive results would imply that user behavioral patterns, such as 51 collaborative signals ( i.e., user preference similarities between items) [52, 53], may be implicitly 52 encoded by LMs. To test this hypothesis, we employ linear mapping to project the language 53 representations of item titles into a recommendation space (see Figure 1a). Our observations include: 54

55

56

Surprisingly, this simple linear mapping yields high-quality item representations, which achieve exceptional recommendation performance (see Figure 1b and experimental results in Section 2).

57

58

59

The clustering of items is generally preserved from the language space to the recommendation space (see Figure 1c). For example, movies with the theme of superheroes and monsters are gathering in both language and recommendation spaces.

Interestingly, the linear mapping effectively reveals preference similarities that may be implicit 60 or even obscure in the language space. For instance, while certain movies, such as those of 61 homosexual movies (illustrated in Figure 1c), show dispersed representations in the language space, 62 their projections through linear mapping tend to cluster together, reflecting their genres affiliation. 63

These findings indicate a homomorphic relationship between the language representation space of 64

LMs and an effective item representation space for recommendation. Motivated by this insight, we 65

propose a new text-based recommendation paradigm for general collaborative filtering (CF), which 66 utilizes the pre-trained language representations of item titles as the item input and the average 67 historical interactions' representations as the user input. Different from traditional ID-based CF 68

models [54, 55, 52] that heavily rely on trainable user and item IDs, this paradigm solely uses 69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

pre-trained LM embeddings and completely abandons ID-based embeddings. In this paper, to fully explore the potential of advanced language representations, we adopt a simple model architecture

consisting of a two-layer MLP with graph convolution, and the popular contrastive loss, InfoNCE

[56-58], as the objective function. This model is named good properties.

Benefiting from paradigm shifts from ID-based embeddings to language representations, AlphaRec presents three desirable advantages. First, AlphaRec is notable for its simplicity, lightweight, rapid convergence, and exceptional recommendation performance (see Section 4.1). We empirically demonstrate that, for the first time, such a simple model with embeddings from pre-trained LMs can outperform leading CF models on multiple datasets. This finding strongly supports the possibility for developing language-representation-based recommender systems. Second, AlphaRec exhibits a strong zero-shot recommendation capability across untrained domains (see Section 4.2). By co-training on three Amazon datasets (Books, Movies & TV, and Video Games) [1], AlphaRec can achieve performance comparable to the fully-trained LightGCN on entirely different platforms (MovieLens-1M [59] and BookCrossing [60]), and even exceed LightGCN in a completely new domain (Amazon Industrial), without additional training on these target datasets. This capability underscores AlphaRec's potential to develop more general recommenders. Third, AlphaRec is userfriendly, offering a new research paradigm that enhances recommendation by leveraging languagebased user feedback (see Section 4.3). Endowed with its inherent semantic comprehension of language representations, AlphaRec can refine recommendations based on user intentions expressed in natural language, enabling traditional CF recommenders to evolve into intention-aware systems through a straightforward paradigm shift.

2 Uncovering Collaborative Signals in LMs via Linear Mapping

In this section, we aim to explore whether LMs implicitly encode collaborative signals in their representation spaces. We first formulate the personalized item recommendation task, then detail the linear mapping and its empirical findings. Empirical evidence indicates a homomorphic relationship between the representation spaces of advanced LMs and effective recommendation spaces.

Task formulation. Personalized item recommendation with implicit feedback aims to select items i ∈ I that best match user u 's preferences based on binary interaction data Y = [ y ui ] , where y ui = 1 ( y ui = 0 ) indicates user u ∈ U has (has not) interacted with item i [58]. The primary objective of recommendation is to model the user-item interaction matrix Y using a scoring function ˆ y : U ×I → R , where ˆ y ui measures u 's preference for i . The scoring function ˆ y ui = s ◦ ϕ θ ( x u , x i ) comprises three key components: pre-existing features x u and x i for user u and item i , a representation learning module ϕ θ ( · , · ) parametrized by θ , and a similarity function s ( · , · ) . The representation learning module ϕ θ transfers u and i into representations e u and e i for similarity matching s ( e u , e i ) , and the TopK highest scoring items are recommended to u .

Different recommenders employ various pre-existing features x u , x i and representation learning architecture ϕ θ ( · , · ) . Traditional ID-based recommenders use one-hot vectors as pre-existing features x u , x i . The choice of ID-based representation learning architecture ϕ θ can vary widely, including ID-based embedding matrix [54], multilayer perception [61], graph neural network [52, 62], and variational autoencoder [63]. The commonly used similarity function is cosine similarity [64, 57] s ( e u , e i ) = e u ⊤ e i ∥ e u ∥·∥ e i ∥ , which we adopt in this paper.

Linear mapping. Building on the extensive knowledge encoded by LMs, we explore utilizing LMs as feature extractors, leveraging the language representations of item titles as initial item feature x i . For initial user feature x u , we use the average of the title representations of historically interacted items, defined as x u = 1 |N u | ∑ i ∈N u x i , where N u is the set of items user u has interacted with. Detailed procedures for obtaining these language-based features are provided in Appendix B.2. We select a trainable linear mapping matrix W as the representation learning module ϕ θ , setting e u = W x u and e i = W x i . To learn the linear mapping W , we adopt the InfoNCE loss [56] as the objective function, which has demonstrated state-of-the-art performance in both ID-based [65, 66] and LM-enhanced collaborative filtering (CF) recommendations [47] (refer to Equation (4) for the formula). The overall framework of the linear mapping process is illustrated in Figure 1a. We directly use linearly mapped representations e u and e i to calculate the user-item similarity s ( e u , e i ) for recommendation. High performance on the test set would suggest that collaborative signals ( i.e., user

AlphaRec for its originality and a series of

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

preference similarities between items) have been implicitly encoded in the language representation space [67, 10].

Empirical findings. We compare the recommendation performance of the linear mapping method with three classical CF baselines, matrix factorization (MF) [54, 68], MultV AE [63], and LightGCN [55] (see more details about baselines in Appendix C.2.1). We report three widely used metrics Hit Ratio (HR@ K ), Recall@ K , Normalized Discounted Cumulative Gain (NDCG@ K )) to evaluate the effectiveness of linear mapping, with K set by default at 20. We evaluate a wide range of LMs, including BERT-style models [4, 5], decoder-only language models [6, 69], and LM-based text embedding models [70, 71] (see Appendix B.1 for details about used LMs).

Table 1 reports the recommendation performance yielded by the linear mapping on three Amazon datasets [1], comparing with classic CF baselines. We observe that the performance of most advanced text embedding models ( e.g., text-embeddings-3-large [70] and SFR-Embedding-Mistral [71]) exceed LightGCN on all datasets. We further empirically prove that these improvements do not merely come from the better feature encoding ability (refer to Appendix B.3). These findings indicate the homomorphic relationship between the language representation space of advanced LMs and an effective item representation space for recommendation. Moreover, with the advances in LMs, the performance of item representation linearly mapped from LMs exhibits a rising trend, gradually surpassing traditional ID-based CF models. Representations from early BERT-style models ( e.g., BERT [4] and RoBERTa [5]) only show weaker or equal capabilities compared with MF, while the performance of decoder-only LMs ( e.g., Llama-7B [6] ) start to match MultVAE and LightGCN.

3 AlphaRec

This finding of space homomorphic relationship sheds light on building advanced CF models purely based on LM representations without introducing ID-based embeddings. To be specific, we try to incorporate only three simple components ( i.e., nonlinear projection [61], graph convolution [55] and contrastive learning (CL) objectives [56]), to develop a simple yet effective CF model called AlphaRec. It is important to highlight that our approach is centered on exploring the potential of LM representations for CF by integrating essential components from leading CF models, rather than deliberately inventing new CF mechanisms. We present the model structure of AlphaRec in Section 3.1, and compare AlphaRec with two popular recommendation paradigms in Section 3.2.

3.1 Method

We present how AlphaRec is designed and trained. Generally, the representation learning architecture ϕ θ ( · , · ) of AlphaRec is simple, which only contains a two-layer MLP and the basic graph convolution operation, with language representations as the input features x u , x i . The cosine similarity is used as the similarity function s ( · , · ) , and the contrastive loss InfoNCE [56, 57] is adopted for optimization. For simplicity, we consistently adopt text-embeddings-3-large [70] as the language representation model, for its excellent language understanding and representation capabilities.

Nonlinear projection. In AlphaRec, we substitute the linear mapping matrix delineated in Section 2 with a nonlinear MLP. This conversion from linear to nonlinear is non-trivial, for the paradigm shift from ID-based embeddings to LM representations, since nonlinear transformation helps in excavating more comprehensive collaborative signals from the LM representation space with rich semantics (see

discussions about this in Appendix C.2.3) [61]. Specifically, we project the language representation 164 x i of the item title to an item space for recommendation with the two-layer MLP, and obtain user 165 representations as the average of historical items: 166

Graph convolution. Graph neural networks (GNNs) have shown superior effectiveness for recom167 mendation [52, 55], owing to the natural user-item graph structure in recommender systems [72]. 168 In AlphaRec, we employ a minimal graph convolution operation [55] to capture more complicated 169 collaborative signals from high-order connectivity [55, 73, 74, 72] as follows: 170

The information of connected neighbors is aggregated with a symmetric normalization term 171 1 √ |N u | √ |N i | . Here N u ( N i ) denotes the historical item (user) set that user u (item i ) has inter172

acted with. The features e (0) u and e (0) i projected from the MLP are used as the input of the first layer. 173 After propagating for K layers, the final representation of a user (item) is obtained as the average of 174 features from each layer: 175

Contrastive learning objective. The introduction of contrasting learning is another key element for 176 the success of leading CF models. Recent research suggests that the contrast learning objective, rather 177 than data augmentation, plays a more significant role in improving recommendation performance 178 [66, 75, 65]. Therefore, we simply use the contrast learning object InfoNCE [56] as the loss function 179 without any additional data augmentation on the graph [76, 57]. With cosine similarity as the 180 similarity function s ( e u , e i ) = e u ⊤ e i ∥ e u ∥·∥ e i ∥ , the InfoNCE loss [56, 76, 77] is written as: 181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

Here, τ is a hyperparameter called temperature [78], O + = { ( u, i ) | y ui = 1 } denoting the observed interactions between users U and items I . And S u is a randomly sampled subset of negative items that user u does not adopt.

3.2 Discussion of Recommendation Paradigms

We compare the language-representation-based AlphaRec with two popular recommendation paradigms in Table 2 (see more discussion about related works in Appendix A).

ID-based recommendation (ID-Rec) [52, 54]. In the traditional ID-based recommendation paradigm, users and items are represented by ID-based learnable embeddings derived from a large number of user interactions. While ID-Rec exhibits excellent recommendation capabilities with low training and inference costs [62, 76], it also has two significant drawbacks. Firstly, these ID-based embeddings learned in specific domains are difficult to transfer to new domains without overlapping users and items [37], thereby hindering zero-shot recommendation capabilities. Additionally, there is a substantial gap between ID-Rec and natural languages [34], which makes ID-based recommenders hard to incorporate language-based user intentions and further refine recommendations accordingly.

LM-based recommendation (LM-Rec) [15, 16, 24]. Benefitting from the extensive world knowledge and powerful reasoning capabilities of LMs [7, 79], the LM-based recommendation paradigm has gained widespread attention [11, 13]. LM-Rec tends to convert user interaction history into text prompts as input for LMs, utilizing pre-trained or fine-tuned LMs in a text generation pattern to recommend items. LM-Rec demonstrates zero-shot and few-shot abilities and can easily understand language-based user intentions. However, LM-Rec faces significant challenges. Firstly, the LM-based model architecture leads to huge training and inference costs, with real-world deployment difficulties.

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

Additionally, limited by the text generation paradigm, LM-based models tend to perform candidate selection [17] or generate a single next item [24]. It remains difficult for LM-Rec to comprehensively rank the entire item corpus or recommend multiple items that align with user interests.

Language-representation-based recommendation. We argue that AlphaRec follows a new CF paradigm, which we term the language-representation-based paradigm. This paradigm replaces the ID-based embeddings in ID-Rec with representations from pre-trained LMs, employing feature encoders to map LM representations directly into the recommendation space. Few early studies lie in this paradigm, including using BERT-style LMs to learn universal sequence representations [37, 44], or adopting the same model architecture as ID-Rec with simple input features replacement [34, 35]. These early explorations, which are mostly based on BERT-style LMs, are usually only applicable in certain specific scenarios, such as the transductive setting with the help of ID-based embeddings [37]. This phenomenon is consistent with our previous findings in Section 2, indicating that BERT-style LMs may fail to effectively encode collaborative signals. We point out that AlphaRec is the first recommender in the language-representation-based paradigm to surpass the traditional ID-based paradigm on multiple tasks, faithfully demonstrating the effectiveness and potential of this paradigm.

4 Experiments

In this section, we aim to explore the effectiveness of AlphaRec. Specifically, we are trying to answer the following research questions:

RQ1: How does AlphaRec perform compared with leading ID-based CF methods?

RQ2: Can AlphaRec learn general item representations, and achieve good zero-shot recommendation performance on entirely new datasets?

RQ3: Can AlphaRec capture user intention described in natural language and adjust the recommendation results accordingly?

4.1 General Recommendation Performance (RQ1)

Motivation. Weaimto explore whether the language-representation-based recommendation paradigm can outperform the ID-Rec paradigm. An excellent performance of AlphaRec would shed light on the research line of building representation-based recommenders in the future.

Baselines. We only consider ID-based baselines in this section. We ignore LM-based methods due to two practical difficulties: the huge inference cost on datasets with millions of interactions and the task limitation of candidate selection or next item prediction. In addition to classic baselines ( i.e., MF, MultVAE, and LightGCN) introduced in section 2, we consider two categories of leading ID-based CF baselines: CL-based CF methods: SGL [80], BC Loss [76], XSimGCL [66] and LM-enhanced methods: KAR [48], RLMRec [47]. See more details about baselines in Appendix C.2.1.

Results. Table 3 presents the performance of AlphaRec compared with leading CF baselines. The best-performing methods are bold, while the second-best methods are underlined. Figure 2a and Figure 2b report the training efficiency and ablation results. We observe that:

AlphaRec consistently outperforms leading CF baselines by a large margin across all metrics on all datasets. AlphaRec shows an improvement ranging from 6.79% to 9.75% on Recall@20 compared to the best baseline RLMRec [47]. We further conduct the ablation study to explore the reason for its success (see more ablation results in Appendix C.2.2). As shown in Figure 2b, each component in AlphaRec contributes positively. Specifically, the performance degradation caused by replacing the MLP with a linear weight matrix (w/o MLP) indicates that nonlinear transformations can further extract the implicit collaborative signals encoded in the LM representation space.

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

Epoches

Books

Movies & TV

Video Games

Recall

NDCG

HR

Recall

NDCG

HR

Recall

NDCG

HR

MF (Rendle et al., 2012)

0

.

0437

0

.

0391

0

.

2476

0

.

0568

0

.

0519

0

.

3377

0

.

0323

0

.

0195

0

.

0864

0

.

0722

0

.

0597

0

.

3418

0

.

0853

0

.

0776

0

.

4434

0

.

0908

0

.

0531

0

.

2211

LightGCN (He et al., 2021)

0

.

0723

0

.

0608

0

.

3489

0

.

0849

0

.

0747

0

.

4397

0

.

1007

0

.

0590

0

.

2281

SGL (Wu et al., 2021)

0.0789

0.0657

0.3734

0.0916

0.0838

0.4680

0.1089

0.0634

0.2449

0.0915

0.0779

0.4045

0.1039

0.0943

0.5037

0.1145

0.0668

0.2561

0.0879

0.0745

0.3918

0.1057

0.0984

0.5128

0.1138

0.0662

0.2550

0.0852

0.0734

0.3834

0.1084

0.1001

0.5134

0.1181

0.0693

0.2571

RLMRec (Ren et al., 2024)

0.0928

0.0774

0.4092

0.1119

0.1013

0.5301

0.1384

0.0809

0.2997

AlphaRec

0.0991*

0.0828*

0.4185*

0.1221*

0.1144*

0.5587*

0.1519*

0.0894*

0.3207*

Imp.% over the best baseline

6

.

79%

5

.

34%

2

.

27%

9

.

12%

10

.

75%

5

.

40%

9

.

75%

10

.

51%

7

.

01%

(a) Training efficiency comparison

(b) Ablation study on Books

600

LightGCN

0.10

Recall@20

MF

NDCG@20

500

BCLoss

KAR

0.08

RLMRec

400

SGL

MultVAE

0.06

300

XSimGCL

AlphaRec

0.04

200

0.02

100

0

Books

Movies&TV

VideoGames

0.00

Linear Mapping w/o CL

w/oMLP

w/oGCN

AlphaRec

Moreover, the performance drop from replacing InfoNCE loss [57] with BPR loss [68] (w/o CL) and removing the graph convolution (w/o GCN) suggests that explicitly modeling the collaborative relationships through the loss function and model architecture can further enhance recommendation performance. These findings suggest that, by carefully designing the model to extract collaborative signals, the language-representation-based paradigm can surpass the ID-Rec paradigm.

The incorporation of semantic LM representations into traditional ID-based CF methods can lead to significant performance improvements. We note that two LM-enhanced CF methods, KAR and RLMRec, both show improvements over CL-based CF methods. Nevertheless, the combination of ID-based embeddings and LM representations in these methods does not yield higher results than purely language-representation-based AlphaRec. We attribute this phenomenon to the fact that the performance contribution of these methods mainly comes from the LM representations, which is consistent with the previous findings [34, 44].

AlphaRec exhibits fast convergence speed. We find that the convergence speed of AlphaRec is comparable with, or even surpasses, CL-based methods with data augmentation ( e.g., SGL [80] and XSimGCL [66]). Meanwhile, methods based solely on graph convolution (LightGCN [55]) or CL objective (BC Loss [76]) show relatively slow convergence speed, indicating that introducing these modules may not lead to convergence speed improvement. Therefore, we attribute the fast convergence speed of AlphaRec to the homomorphic relationship between the LM representation space and a good recommendation space, so only minor adjustments to the LM representations are needed for recommendation.

4.2 Zero-shot Recommendation Performance on Entirely New Datasets (RQ2)

Motivation. We aim to explore whether AlphaRec has learned general item representations [37], which enables it to perform well on entirely new datasets without any user and item overlap.

Task and datasets. In zero-shot recommendation [38], there is not any item or user overlap between the training set and test set [38, 33], which is different from the research line of cross-domain recommendation in ID-Rec [81]. We jointly train AlphaRec on three source datasets ( i.e., Books, Movies & TV, and Video Games), while testing it on three completely new target datasets ( i.e.,

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

Movielens-1M [59], Book Crossing [60], and Industrial [1]) without further training on these new datasets. (see more details about how we train AlphaRec on multiple datasets in Appendix C.3.1).

Baselines. Due to the lack of zero-shot recommenders in the field of general recommendation, we slightly modify two zero-shot methods in the sequential recommendation [82], ZESRec [37] and UniSRec [37], as baselines. We also incorporate two strategy-based CF methods, Random and Pop (see more details about these baselines in Appendix C.3.2).

Results. Table 4 presents the zero-shot recommendation performance comparison on entirely new datasets. The best-performing methods are bold and starred, while the second-best methods are underlined. We observe that:

AlphaRec demonstrates strong zero-shot recommendation capabilities, comparable to or even surpassing the fully trained LightGCN. On datasets from completely different platforms ( e.g., MovieLens-1M and Book Crossing), AlphaRec is comparable with the fully trained LightGCN. On the same Amazon platform dataset, Industrial, AlphaRec even surpasses LightGCN, which we attribute to the possibility that AlphaRec implicitly learns unique user behavioral patterns on the Amazon platform [1]. Conversely, ZESRec and UniSRec exhibit a marked performance decrement compared with AlphaRec. We attribute this phenomenon to two aspects. On the one hand, BERTstyle LMs [4, 5] used in these works may not have effectively encoded collaborative signals, which is consistent with our findings in Section 2. On the other hand, components designed for the next item prediction task in sequential recommendation [83] may not be suitable for capturing the general preferences of users in CF scenarios.

The zero-shot recommendation capability of AlphaRec generally benefits from an increased amount of training data, without harming the performance on source datasets. As illustrated in Figure 8, the zero-shot performance of AlphaRec, when trained on a mixed dataset, is generally superior to training on one single dataset [37]. Additionally, we also note that training data with themes similar to the target domain contributes more to the zero-shot performance. For instance, the zero-shot capability on MovieLens-1M may primarily stem from Movies & TV. Furthermore, we discover that AlphaRec, when trained jointly on multiple datasets, hardly experiences a performance decline on each source dataset. These findings further point to the general recommendation capability of a single pre-trained AlphaRec across multiple datasets. The above findings also offer a potential research path to achieve general recommendation capabilities, by incorporating more training data with more themes. See more details about these results in Appendix C.3.3.

4.3 User Intention Capture Performance (RQ3)

Motivation. We aim to investigate whether a straightforward paradigm shift enables pre-trained AlphaRec to perceive text-based user intentions and refine recommendations.

Task and datasets. We test the user intention capture ability of AlphaRec on MovieLens-1M and Video Games. In the test set, only one target item remains for each user [84], with one intention query generated by ChatGPT [85, 40] (see the details about how to generate and check these intention queries in Appendix C.4.1). In the training stage, we follow the same procedure as illustrated in Section 2 to train AlphaRec. In the inference stage, we obtain the LM representation e Intention u for each user intention query and combine it with the original user representation to get a new user representation as ˜ e (0) u = (1 -α ) e (0) u + α e Intention u [84]. This new user representation is sent into the freezed AlphaRec for recommendation. We report a relatively small K = 5 for all metrics to better reflect the intention capture accuracy.

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

MovieLens-1M

Video Games

HR@5

NDCG@5

HR@5

NDCG@5

0.2738

0.1973

0.2212

0.1425

AlphaRec (w/o Intention)

0.0793

0.0498

0.0663

0.0438

AlphaRec (w Intention)

0.4704*

0.3738*

0.2569*

0.1862*

(a) Case study of user intention capture

(b) Effect of

α

UserIntention:I'mlookingforaclassicmoviethat delvesintotheworld oforganizedcrime,

familyloyalty,and powerstruggles,withiconicperformances andunforgettable quotes.

1

0.4

Target:Godfather,The (1972)

0.3

RecommendationList(w/oIntention)

RecommendationList(wIntention)

Braveheart(1995)

Schindler'sList(1993)

Godfather,The (1972)

L.A.Confidential (1997)

Te

Qodlamer

1

0.2

-L.A.Confidential(1997)

Pulp Fiction (1994)

StarWars:EpisodeV(1980)

PulpFiction (1994)

Schindler'sList(1993)

1

1

1

0.1

0

0.25

0.5

Recall@5

NDCG@5

0.75

1

Valueofα

User intention capture results. Table 5 represents the user intention capture experiment results, compared with the baseline TEM [86]. Clearly, the introduction of user intention (w Intention) significantly refines the recommendations of the pre-trained AlphaRec (w/o Intention). Moreover, AlphaRec outperforms the baseline model TEM by a large margin, even without additional training on search tasks. We further conduct a case study on MovieLens-1M to demonstrate how AlphaRec captures the user (see more case study results in Appendix C.4.3). As shown in Figure 3a, AlphaRec accurately captures the hidden user intention for 'Godfather', while keeping most of the recommendation results unchanged. This indicates that AlphaRec captures the user intention and historical interests simultaneously.

Effect of the intention strength α . By controlling the value of α , AlphaRec can provide better recommendation results, with a balance between user historical interests and user intent capture. Figure 3b depicts the effect of α . Initially, as α increases, the recommendation performance rises accordingly, indicating that incorporating user intention enables AlphaRec to provide better recommendation results. However, as the α approaches 1, the recommendation performance starts to decrease, which suggests that the user historical interests learned by AlphaRec also play a vital role. The similar effect of α on Video Games is discussed in Appendix C.4.4.

5 Limitations

There are several limitations not addressed in this paper. On the one hand, although we have demonstrated the excellence of AlphaRec for multiple tasks on various offline datasets, the effectiveness of online employment remains unclear. On the other hand, although we have successfully explored the potential of language-representation-based recommenders by incorporating essential components in leading CF models, we do not elaboratively focus on designing new components for CF models.

6 Conclusion

In this paper, we explored what knowledge about recommendations has been encoded in the LM 339 representation space. Specifically, we found that the advanced LMs representation space exhibits 340 a homomorphic relationship with an effective recommendation space. Based on this finding, we 341 developed a simple yet effective CF model called AlphaRec, which exhibits good recommendation 342 performance with zero-shot recommendation and user intent capture ability. We pointed out that 343 AlphaRec follows a new recommendation paradigm, language-representation-based recommendation, 344 which uses language representations from LMs to represent users and items and completely abandons 345 ID-based embeddings. We believed that AlphaRec is an important stepping stone towards building 346 general recommenders in the future. 1 347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

References

Jianmo Ni, Jiacheng Li, and Julian J. McAuley. Justifying recommendations using distantlylabeled reviews and fine-grained aspects. In EMNLP , 2019.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In ACL , 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda B. Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In ICLR , 2023.

Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavas, and Anna Korhonen. Probing pretrained language models for lexical semantics. In EMNLP , 2020.

Wes Gurnee and Max Tegmark. Language models represent space and time. CoRR , abs/2310.02207, 2023.

Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (llms). CoRR , abs/2307.02046, 2023.

Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. Large language models for generative recommendation: A survey and visionary discussions. CoRR , abs/2309.01157, 2023.

Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. CoRR , abs/2305.19860, 2023.

Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, 395 Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How can recommender systems 396 benefit from large language models: A survey. CoRR , abs/2306.05817, 2023. 397

Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Rec398 ommendation as instruction following: A large language model empowered recommendation 399 approach. CoRR , abs/2305.07001, 2023. 400

401

402

403

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In RecSys , 2023.

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, and Xiang Wang. Llara: Aligning large language models with sequential recommenders. CoRR , abs/2312.02445, 2023.

Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. CoRR , abs/2311.01343, 2023.

Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommendation. CoRR , abs/2311.09049, 2023.

Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrating collaborative embeddings into large language models for recommendation. CoRR , abs/2310.19488, 2023.

Arpita Vats, Vinija Jain, Rahul Raja, and Aman Chadha. Exploring the impact of large language models on recommender systems: An extensive review. CoRR , abs/2402.18590, 2024.

Chengkai Huang, Tong Yu, Kaige Xie, Shuai Zhang, Lina Yao, and Julian J. McAuley. Foundation models for recommender systems: A survey and new perspectives. CoRR , abs/2402.11143, 2024.

Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Mingchen Cai, Wayne Xin Zhao, and Ji-Rong Wen. Prompting large language models for recommender systems: A comprehensive framework and empirical analysis. CoRR , abs/2401.04997, 2024.

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In RecSys , 2022.

Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems. CoRR , abs/2205.08084, 2022.

Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. CoRR , abs/2308.11131, 2023.

Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He. Large language model can interpret latent space of sequential recommender. CoRR , abs/2310.20487, 2023.

Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy. Recommender systems with generative retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NeurIPS , 2023.

Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. ONCE: boosting content-based recommendation with both open- and closed-source large language models. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM , 2024.

Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi. Actions speak louder than words: Trillionparameter sequential transducers for generative recommendations. CoRR , abs/2402.17152, 2024.

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recommendation foundation models. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel, editors, SIGIR-AP , 2023.

Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, Yu Xu, and Xiaohu Qie. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurIPS , 2022.

Jiaqi Zhang, Yu Cheng, Yongxin Ni, Yunzhu Pan, Zheng Yuan, Junchen Fu, Youhua Li, Jie Wang, and Fajie Yuan. Ninerec: A benchmark dataset suite for evaluating transferable recommendation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? ID- vs. modality-based recommender models revisited. In SIGIR , 2023.

Ruyu Li, Wenhao Deng, Yu Cheng, Zheng Yuan, Jiaqi Zhang, and Fajie Yuan. Exploring the upper limits of text-based collaborative filtering using large language models: Discoveries and insights. CoRR , abs/2305.11700, 2023.

Youhua Li, Hanwen Du, Yongxin Ni, Pengpeng Zhao, Qi Guo, Fajie Yuan, and Xiaofang Zhou. Multi-modality is all you need for transferable recommender systems. CoRR , abs/2312.09602, 2023.

Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Towards universal sequence representation learning for recommender systems. In KDD , pages 585-593. ACM, 2022.

Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. Zero-shot recommender systems. CoRR , abs/2105.08318, 2021.

Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian J. McAuley. Text is all you need: Learning language representations for sequential recommendation. In KDD , 2023.

Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian J. McAuley. Bridging language and items for retrieval and recommendation. CoRR , abs/2403.03952, 2024.

Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, WWW , 2023.

Zhiming Mao, Huimin Wang, Yiming Du, and Kam-Fai Wong. Unitrec: A unified text-to-text transformer and joint contrastive learning framework for text-based recommendation. In ACL , 2023.

Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. U-BERT: pre-training user representations for improved recommendation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 . AAAI, 2021.

Lingzi Zhang, Xin Zhou, Zhiwei Zeng, and Zhiqi Shen. Are ID embeddings necessary? whitening pre-trained text embeddings for effective sequential recommendation. CoRR , abs/2402.10602, 2024.

Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In SIGIR , 2020.

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Llmrec: Large language models with graph augmentation for recommendation. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM , 2024.

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. CoRR , abs/2310.15950, 2024.

Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. Towards open-world recommendation with knowledge augmentation from large language models. CoRR , abs/2306.10933, 2023.

Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong He, Liang Zhang, Fajie Yuan, Jun Zhou, and Linjian Mo. Breaking the length barrier: Llm-enhanced CTR prediction in long textual user behaviors. CoRR , abs/2403.19347, 2024.

An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. CoRR , abs/2310.10108, 2023.

Junjie Zhang, Yupeng Hou, Ruobing Xie, Wenqi Sun, Julian J. McAuley, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Agentcf: Collaborative learning with autonomous language agents for recommender systems. CoRR , abs/2310.09233, 2023.

Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In SIGIR , 2019.

Yang Li, Tong Chen, Yadan Luo, Hongzhi Yin, and Zi Huang. Discovering collaborative signals for next POI recommendation with iterative seq2graph augmentation. In IJCAI , 2021.

Yehuda Koren, Robert M. Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer , 42(8):30-37, 2009.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR , 2021.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018.

Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu, and Xiangnan He. On the effectiveness of sampled softmax loss for item recommendation. CoRR , abs/2201.02327, 2022.

Steffen Rendle. Item recommendation from implicit feedback. In Recommender Systems Handbook . Springer US, 2022.

F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. , 5(4):19:1-19:19, 2016.

Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: Metalearned user preference estimator for cold-start recommendation. In KDD , 2019.

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In WWW , 2017.

Xuheng Cai, Chao Huang, Lianghao Xia, and Xubin Ren. Lightgcl: Simple yet effective graph contrastive learning for recommendation. In ICLR , 2023.

Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In WWW , 2018.

Jiawei Chen, Junkang Wu, Jiancan Wu, Xuezhi Cao, Sheng Zhou, and Xiangnan He. Adapτ : Adaptively modulating embedding magnitude for recommendation. In WWW , 2023.

An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua. Empowering collaborative filtering with principled adversarial contrastive loss. In NeurIPS , 2023.

Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoc Viet Hung, and Hongzhi Yin. Xsimgcl: Towards extremely simple graph contrastive learning for recommendation. IEEE Trans. Knowl. Data Eng. , 36(2):913-926, 2024.

Abhilasha Ravichander, Yonatan Belinkov, and Eduard H. Hovy. Probing the probing paradigm: 543 Does probing accuracy entail task relevance? In EACL , 2021. 544

545

546

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: bayesian personalized ranking from implicit feedback. CoRR , abs/1205.2618, 2012.

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR , abs/2310.06825, 2023.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training. CoRR , abs/2201.10005, 2022.

Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding-mistral:enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. URL https://blog.salesforceairesearch.com/sfr-embedded-mistral/ .

Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: A survey. ACM Comput. Surv. , 55(5):97:1-97:37, 2023.

Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML , 2019.

Lei Chen, Le Wu, Richang Hong, Kun Zhang, and Meng Wang. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. In AAAI , 2020.

Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. Are graph augmentations necessary?: Simple graph contrastive learning for recommendation. In SIGIR , 2022.

An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua. Incorporating bias-aware margins into contrastive loss for collaborative filtering. In NeurIPS , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR , 2021.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR , abs/2303.18223, 2023.

Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In SIGIR , 2021.

Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. Cross-domain recommendation: Challenges, progress, and prospects. In IJCAI , 2021.

Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun. Sequential recommender systems: Challenges, progress and prospects. In IJCAI , 2019.

Wang-Cheng Kang and Julian J. McAuley. Self-attentive sequential recommendation. In ICDM , 2018.

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W. Bruce Croft. Learning a hierarchical embedding model for personalized product search. In SIGIR , 2017.

OpenAI. GPT-4 technical report. CoRR , 2023.

Keping Bi, Qingyao Ai, and W. Bruce Croft. A transformer-based embedding model for personalized product search. In SIGIR , 2020.

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR (Workshop) , 2017.

Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In ICLR , 2022.

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. CoRR , abs/2311.03658, 2023.

Xubin Ren, Wei Wei, Lianghao Xia, and Chao Huang. A comprehensive survey on selfsupervised learning for recommendation. arXiv preprint arXiv:2404.03354 , 2024.

Zheng Chen. PALR: personalization aware llms for recommendation. CoRR , abs/2305.07622, 2023.

Yuling Wang, Changxin Tian, Binbin Hu, Yanhua Yu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou, Liang Pang, and Xiao Wang. Can small language models be good reasoners for sequential recommendation? CoRR , abs/2403.04260, 2024.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234 , 2022.

Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR , 2024.

Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. Is chatgpt a good recommender? A preliminary study. CoRR , abs/2304.10149, 2023.

Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In RecSys , 2023.

Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chatrec: Towards interactive and explainable llms-augmented recommender system. CoRR , abs/2303.14524, 2023.

Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang. Ctrl: Connect tabular and language model for ctr prediction. arXiv preprint arXiv:2306.02841 , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR , abs/2310.15213, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS , 2022.

Julian J. McAuley, Rahul Pandey, and Jure Leskovec. Inferring networks of substitutable and complementary products. In KDD , 2015.

Walid Krichene and Steffen Rendle. On sampled metrics for item recommendation. In KDD , 2020.

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

A Related Works

Representations in LMs. The impressive capabilities demonstrated by LMs across various tasks raise a wide concern about what they have learned in the representation space. An important and effective approach for interpreting and analyzing representations of LMs is linear probing [67, 87]. The main idea of linear probing is simple: training linear classifiers to predict some specific attributes or concepts ( e.g., lexical structure [9] ) from the representations in the hidden layers of LMs. A high probing result ( e.g., classification accuracy on the out-of-sample test set) tends to imply relevant information has been implicitly encoded in the representation space of LMs, although this does not imply LMs directly use these representations [67, 10]. Recent studies empirically demonstrate that concepts such as color [88], game states [8]. and geographic position are encoded in LMs. Furthermore, these concepts may even be linearly encoded in the representation space of LMs [8, 89].

Collaborative filtering. Collaborative filtering (CF) [90] is an advanced technique in modern recommender systems. The prevailing CF methods tend to adopt an ID-based paradigm, where users and items are typically represented as one-hot vectors, with an embedding table used for lookup [54]. Usually, these embedding parameters are learned by optimizing specific loss functions to reconstruct the history interaction pattern [68]. Recent advances in CF mainly benefit from two aspects, graph convolution [72] and contrastive learning [90]. These CF models exhibit superior recommendation performance by conducting the embedding propagation [52, 55] and applying contrastive learning objectives [80, 62, 66]. However, although effective, these methods are still limited, due to the ID-based paradigm. Since one-hot vectors contain no feature information beyond being identifiers, it is challenging to transfer pre-trained ID embeddings to other domains [37] or to leverage leading techniques from computer vision (CV) and natural language processing (NLP) [34].

LMs for recommendation. The remarkable language understanding and reasoning ability shown by LMs has attracted extensive attention in the field of recommendation. The application of LMs in recommendation can be categorized into three main approaches: LM-enhanced recommendation, LM as the modality encoder, and LLM-based recommendation. The first research direction, LLM-enhanced recommendation, focuses on empowering traditional recommenders with the semantic representations from LMs [48, 47, 46, 49, 91, 92]. Specifically, these methods introduce representations from LMs as additional features for traditional ID-based recommenders, to capture complicated user preferences. The second research line lies in adopting the LM as the text modality encoder, which is also known as a kind of modality-based recommendation (MoRec) [34, 35]. These methods tend to train the LM as the text modality encoder together with the traditional recommender. In previous studies, BERT-style LMs are widely used as the text modality encoder. The third research line, LLM-based recommendation, directly uses LLMs as the recommender and recommends items in a text generation paradigm. Early attempts focus on adopting in-context learning (ICL) [93] and prompting pre-trained LLMs [94-97]. However, such naive methods tend to yield poor performance compared to traditional models. Therefore, recent studies concentrate on fine-tuning LLMs on recommendation-related corpus [16, 15, 26, 25, 29] and align the LLMs with the representations from traditional recommenders as the additional modality [17, 20, 27, 98].

B Linear Mapping

B.1 Brief of Used LMs

We briefly introduce the LMs we use for linear mapping in Section 2.

BERT [4] is an encoder-only language model based on the transformer architecture [3], pre-trained on text corpus with unsupervised tasks. BERT adopts bidirectional self-attention heads to learn bidirectional representations.

RoBERTa [5] is an enhanced version of BERT. RoBERTa preserves the architecture of BERT but improves it by training with more data and large batches, adopting dynamic masking, and removing the next sentence prediction objective.

Llama2-7B [6] is an open-source decoder-only LLM with 7 billion parameters. Llama2 adopts grouped-query attention, with longer context length and larger size of the pre-training corpus compared with Llama-7B [99].

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

Mistral-7B [69] is an open-source pre-trained decoder-only LLM with 7 billion parameters. Mistral 7B leverages grouped-query attention, coupled with sliding window attention for faster and lower cost inference.

text-embedding-ada-v2 & text-embeddings-3-large [70] are leading text embedding models released by OpenAI. These models are built upon decoder-only GPT models, pre-trained on unsupervised data at scale with contrastive learning objectives.

SFR-Embedding-Mistral [71] is a decoder-based text embedding model built upon the opensource LLM Mixtral-7B [69]. SFR-Embedding-Mistral introduces task-homogeneous batching and computes contrastive loss on 'hard negatives', which brings a better performance than the vanilla Mixtral-7B model.

B.2 Extracting Representations from LMs

We present how to extract representations from LMs. For encoder-based LMs ( e.g., BERT [4] and RoBERTa [5]), we use the output representation corresponding to the [CLS] token [40]. For decoder-based models ( e.g., Llama-7B [6, 69], Mistral-7B, and SFR-Embedding-Mistral [71]), we use the representation in the last transformer block [3], corresponding to the last input token [10, 100, 70]. Especially, for the commercial closed-source model ( e.g., text-embedding-ada-v2 and text-embeddings-3-large 2 [70]), we directly call the API interface to obtain representations.

B.3 Empirical Findings

We find more evidence about representations in leading LM encode collaborative signals beyond better feature encoding ability. We randomly shuffle item representations and conduct the same linear mapping experiment. As illustrated in Table 6, randomly shuffled representations, text-embeddings3-large (Random), yield similar performance with BERT, lagging largely behind the vanilla linear mapping method. These results indicate that BERT may only serve as a good feature encoder, while the latest LM may further encode collaborative signals beyond naive feature encoding.

C Experiments

C.1 Datasets

We incorporate six datasets in this paper, including four datasets from the Amazon platform 3 [1] ( i.e., Books, Movies & TV, Video Games, and Industrial), and two datasets from other platforms ( i.e., MovieLens-1M and Book Crossing). Table 7 reports the data statistics of each dataset.

We divide the history interaction of each user into training, validation, and testing sets with a ratio 716 of 4:3:3, and remove users with less than 20 interactions following previous studies [50]. We also 717 remove items from the testing and validation sets that do not appear in the training set, to address the 718 cold start problem. 719

Item Title Examples

Books: Dismissed with Prejudice: A J.P. Beaumont Novel ; Die for Love: A Jacqueline Kirby Novel of Suspense ; The Cloud ; Memories Before and After the Sound of Music: An Autobiography ; Harry Potter and the Sorcerer's Stone;

Movies & TV: Batman Begins ; Fantastic Four ; Max Headroom: The Complete Series ; Madagascar ; Land of the Dead ; King Kong;

Video Games: Fighting Force ; Tomb Raider II ; Tomb Raider ; WWFWarzone ; Kartia: The Word of Fate ; Snowboard Kids ; Command & amp; Conquer: Tiberian Sun - PC ; Final Fantasy VII ; Grim Fandango - PC ; Half-Life - PC;

MovieLens-1M: Basquiat (1996) ; Tin Cup (1996) ; Godfather, The (1972) ; Supercop (1992) ; Manny & Lo (1996) ; Bound (1996) ; Carpool (1996);

Book Crossing: Prague : A Novel ; Chocolate Jesus ; Wie Barney es sieht ; To Kill a Mockingbird ; Sturmzeit. Roman ; A Soldier of the Great War ; Pride and Prejudice (Dover Thrift Editions);

Industrial: Jurassic Perisphinctes Ammonites from France ; FS9140: Spinosaurus - Dinosaur Tooth 20-30mm ; FS9410: USA Eocene, Fossil Fish (Knightia alt), A-grade ; Delta 50-857 Charcoal Filter for 50-868 ; Hitachi RP30SA 7-1/2 Gallon Stainless Steel Industrial Shop Vacuum (Discontinued by Manufacturer) ; Makita 632002-4 14-Inch Cut-Off Wheels (5-Pack) (Discontinued by Manufacturer) ; PORTER-CABLE 740001801 4 1/2-Inch by 10yd 180 Grit Adhesive-Backed Sanding Roll ;

In this paper, we only use the item titles as the text description. Figure 4 gives some item title 720 examples from different datasets. 721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

C.2 General Recommendation

C.2.1 Baselines

We incorporate a series of CF models as our baselines for general recommendation. These models are classified as classical CF methods (MF, MultV AE, and LightGCN), CL-based CF methods (SGL,

BC Loss, and XSimGCL), and LM-enhanced CF methods (KAR, RLMRec). For these LM-enhanced CF methods, we adopt the leading CF method XSimGCL as the backbone.

MF [54, 68] is the most basic CF model. It denotes users and items with ID-based embeddings and conducts matrix factorization with Bayesian personalized ranking (BPR) loss.

MultVAE [63] is a traditional CF model based on the variational autoencoder (V AE). It regards the item recommendation as a generative process from a multinomial distribution and uses variational inference to estimate parameters. We adopt the same model structure as suggested in the paper: 600 → 200 → 600.

LightGCN [55] is a light graph convolution network tailored for the recommendation, which deletes redundant feature transformation and activation function in NGCF [52].

SGL [80] introduces graph contrastive learning into recommender models for the first time. By employing node or edge dropout to generate augmented graph views and conduct contrastive learning between two views, SGL achieves better performance than LightGCN.

BC Loss [76] introduces a robust and model-agnostic contrastive loss, handling various data biases in recommendation, especially for popularity bias.

XSimGCL [66] directly generates augmented views by adding noise into the inner layer of LightGCN without graph augmentation. The simplicity of XSimGCL leads to a faster convergence speed and better performance.

KAR [48] enhances recommender models by integrating knowledge from large language models (LLMs). It generates textual descriptions of users and items and combine the LM representations with traditional recommenders using a hybrid-expert adaptor.

(a) Ablation study on Movies & TV

(b) Ablation study on Video Games

0.16

Recall@20

NDCG@20

0.14

0.12

0.10

0.08

0.06

Linear Mapping w/o CL

w/oMLP

w/oGCN

AlphaRec

0.12

Recall@20

NDCG@20

0.11

0.10

0.09

0.08

0.07

Linear Mapping w/o CL

w/oMLP

w/o GCN

AlphaRec

(a) LM representations

(b) AlphaRec (w/o MLP)

(c) AlphaRec

001

Superhero-Iron Man

80

100

Superhero-Spider Man

75

Superhero-Superman

60

75

Superhero-Batman

50

Monster-Godzilla

40

50

Monster-KingKong

Homosexual

25

20

25

0

0

0

25

20

Superhero-Iron Man

25

Superhero-Iron Man

40

Superhero-Spider Man

Superhero-Superman

Superhero-Batman

0S-

Superhero-Spider Man

Superhero-Superman

Superhero-Batman

60

Monster-Godzilla

★

Monster-Godzilla

75

Monster-King Kong

75

Monster-King Kong

80

Homosexual

Homosexual

75

0S-

-25

0

25

50

75

001

001-

75

50

-25

0

25

50

75

100

75

0S-

25

0

25

50

75

· RLMRec [47] aligns semantic representations of users and items with the representations in CF 747 models through a contrastive loss, as an additional loss trained together with the CF model. The 748 fusion of semantic information and collaborative information brings performance improvement. 749

C.2.2 Ablation Study 750

751

752

753

We conduct the same ablation study as introduced in Section 4.1 on Movies & TV and Video Games datasets. As illustrated in Figure 5, each component in AlphaRec contributes positively, which is consistent with our findings in Section 4.1.

754

C.2.3 The t-SNE Visualization Comparison

In this section, we aim to intuitively explore how the MLP in AlphaRec further helps in excavating 755 collaborative signals in language representations, compared to the linear mapping matrix. We 756 visualize the item representations from LMs, AlphaRec (w/o MLP), and AlphaRec in Figure 6, where 757 AlphaRec (w/o MLP) denotes replacing the MLP with a linear mapping matrix. We observed that 758 movies about superhero and monster cluster in all representation spaces, indicating both AlphaRec 759 (w/o MLP) and AlphaRec capture the preference similarities between these items and preserve 760 the clustering relationship. The difference between AlphaRec (w/o MLP) and AlphaRec may lie 761 in the ability to capture obscure preference similarities among items. As shown in Figure 6a, 762 homosexual movies are dispersed in the language space, indicating the possible semantic differences 763 between them. AlphaRec successfully captures the preference similarities and gathers these items 764 in the representation space, while AlphaRec (w/o MLP) remains some items dispersed. Moreover, 765 AlphaRec outperforms AlphaRec (w/o MLP) by a large margin, as indicated in Figure 5a. These 766 results indicate that AlphaRec exhibits a more fine-grained preference capture ability with the help of 767

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

C.3 Zero-shot Recommendation

C.3.1 Co-training on Multiple Datasets

Co-training on multiple datasets is similar to training on one single dataset, where the only difference lies in the negative sampling. When co-training on multiple datasets, the negative items are restricted to the same dataset as the positive item rather than the full item pool. The other training procedures remain the same with training on one single dataset.

C.3.2 Baselines

Since previous works about zero-shot recommendation mostly focus on sequential recommendation [83, 82], we slightly modify two methods in sequential recommendation, ZESRec [38] and UniSRec [37] as our baselines. Specifically, we maintain the model structure as provided in the paper, and adopt the training paradigm of CF.

Random denotes randomly recommending items from the entire item pool.

Pop denotes randomly recommending from the most popular items. Here popularity denotes the number of users that have interacted with the item.

ZESRec [38] is the first work that defines the problem of zero-shot recommendation. To address this problem, this work introduces a hierarchical Bayesian model with representations from the pre-trained BERT.

UniSRec [37] aims to learn universal item representations from BERT, with parametric whitening and a MoE-enhanced adaptor. By pre-training on multiple source datasets, UniSRec can conduct zero-shot recommendation on various datasets in a transductive or inductive paradigm.

C.3.3 The Effect of Training Datasets

The effect of the training dataset on zero-shot recommendation. We report the zero-shot recommendation performance differences trained on different datasets in Table 8. Here AlphaRec (trained on Books) denotes training on a single Books dataset, while AlphaRec (trained on mixed dataset) denotes co-training on three Amazon datasets. Generally, training on more datasets lead to a better zero-shot performance.

The performance comparison between training on the single dataset and the mixed dataset. In Table 9, AlphaRec (trained on single dataset) denotes training and testing on the same single dataset, while AlphaRec (trained on mixed dataset) denotes training on three Amazon datasets and testing on one single dataset. Generally, co-training on three Amazon datasets yields similar performance compared with training on one single dataset. The only exception lies in Video Games, which shows some performance degradation. We attribute this to the difference between the selection of τ . We use τ = 0.15 when trained on the mixed dataset, while the optimal τ for Video Games lies around 0.2.

These results indicate that a single AlphaRec can capture user preferences among various datasets, 802 showcasing a general collaborative signal capture ability. 803

C.4 User Intention Capture 804

C.4.1 Intention Query Generation 805

Intention Query Generation

Input

You are an expert in generating queries for a target movie. Please help me generate the most suitable query for the target movie within one sentence, following the given example.

Example:

TARGET: BUG-A-SALT 3.0 Black Fly Edition.

QUERY: I want a gun that I can use while gardening to get rid of stink bugs, ants, flies, and spiders in my house. It needs to be amazing and help me feel less scared. TARGET: Toy Story (1995).

Output

QUERY: I'm looking for a heartwarming animated movie that follows the adventures of a group of toys who come to life when their owner is not around.

The user intention query is a natural language sentence implying the target item of interest. For each item in the dataset, we generate a fixed user intention query. Following the previous work [40], we generate user intention queries with the help of ChatGPT [85]. As shown in Figure 7, we prompt ChatGPT in a Chain-of-Thought (CoT) [101] paradigm and adopt the output as the user intention query. We adopt a rule-based strategy to ensure that the output query is in first person, and regenerate the wrong query. Considering the huge amount of item title text, we use ChatGPT3.5 API for generating all queries for the budget's sake.

C.4.2 Baseline

AlphaRec exhibits user intention capture abilities, although not specially designed for search tasks. We compare AlphaRec with TEM [86] which falls in the field of personalized search [84, 102].

TEM [86] uses a transformer to encode the intention query together with user history behaviors, which enables it to achieve better search results by considering the user's historical interest.

C.4.3 Case Study

We conduct two more case studies to verify the user intention capture ability of AlphaRec. As illustrated in Figure 8 and Figure 9, AlphaRec provides proper recommendation results, including the target item for the user intention at the top.

C.4.4 Effect of the Intention Strength Alpha

The value of α controls the balance between the user's historical interests and the user intention query. A larger α incorporates more about the user intention while considering less about the user's historical interests. As shown in Figure 10, the effect of α on Video Games shows a similar trend with MovieLens-1M.

C.5 Trainig Cost

We report the training cost of AlphaRec in this section. Table 10 reports the seconds needed per epoch and the total training cost until convergence. Here Amazon-Mix denotes the mixed dataset of Books, Movies & TV, and Video Games. It's worth noting that AlphaRec converges quickly and only requires a small amount of training time.

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

User Intention: I'm looking for a quirky superhero movie with a comedic twist that features a group of misfit heroes with unusual powers trying to save the day.

1

一

Target: Mystery Men (1999)

Recommendation List (w/o Intention)

1

Recommendation List (w Intention)

-AmericanPie (1999)

Mystery Men (1999)

Bug's Life,A (1998)

American Pie (1999)

General’s Daughter, The (1999)

Bug's Life, A (1998)

Double Jeopardy (1999)

Election (1999)

Election (1999)

1sM PM PM

User Intention: I'm looking for a classic video game where I can explore a gothic world filled

withvampires,werewolves,and other supernatural creatures while wielding a variety of unique weapons.

astlevumia

Target: Castlevania

COLLECTION

RecommendationList(w/oIntention)

Recommendation List (w Intention)

inFAMOUS-Playstation3

-Castlevania

-Assassin's Creed-Playstation 3

Heavy Rain - Greatest Hits

Assassin'sCreedII-GreatestHitsedition- Playstation3

-Assassin's Creed-Playstation3

-inFAMOUS -Playstation3

-Uncharted 2:AmongThieves-Playstation3

-Assassin's Creed II - GreatestHits edition -

HeavyRain-GreatestHits

Playstation 3

Target: Mystery Men (1999)

Target: Castlevania

D Hyperparameter Settings and Implementation Details

We conduct all the experiments in PyTorch with a single NVIDIA RTX A5000 (24G) GPU and a 64 AMD EPYC 7543 32-Core Processor CPU. We optimize all methods with the Adam optimizer. For all ID-based CF methods, we set the layer numbers of graph propagation by default at 2, with the embedding size as 64 and the size of sampled negative items |S u | as 256. We use the early stop strategy to avoid overfitting. We stop the training process if the Recall@20 metric on the validation set does not increase for 20 successive evaluations. In AlphaRec, the dimensions of the input and output in the two-layer MLP are 3072 and 64 respectively, with the hidden layer dimension as 1536. We apply the all-ranking strategy [103] for all experiments, which ranks all items except positive ones in the training set for each user. We search hyperparameters for baselines according to the suggestion in the literature. The hyperparameter search space is reported in Table 11. For these LM-enhanced models, KAR and RLMRec, we also search the hyperparameter of their backbone XSimGCL.

For AlphaRec, the only hyperparameter is the temperature τ and we search it in [0.05, 2]. We report the temperature τ we used for each dataset in Table 12. For the mixed dataset Amazon-Mix in Section 4.2, we use a universal τ = 0.15. We adopt τ = 0.2 for the MovieLens-1M dataset for the user intention capture experiment in Section 4.3.

E Broader Impact

The proposed AlphaRec can significantly improve the performance of zero-shot recommendation 849 and the capability of user intent capture, offering a good approach to crafting more personalized 850 recommendation results. One concern of AlphaRec is the potential for the representations generated 851 by language models can be maliciously attacked, which may result in erroneous or unexpected 852 recommendations. Therefore, we kindly advise researchers to cautiously check the quality of the 853 language representations before using AlphaRec. 854

0.25

070

0.15

0.10

0.05

Recall@5

NDCG@5

0

0.25

0.5

0.75

1

Valueofα

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state the claims made in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of this work in the Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This is an empirical article and contains no theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present all the experiment details and datasets in Appendix C, and Hyperparameters settings are reported in Appendix D. Moreover, we have uploaded the code and data we used in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide access to the data and code we used in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Datasets and data split are presented in Appendix C.1, and hyperparameters are searched according to the suggestion in the literature. See more details in Appendix D. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We validate the p-value to support the main claims of this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We conduct all the experiments in PyTorch with a single NVIDIA RTX A5000 (24G) GPU and a 64 AMD EPYC 7543 32-Core Processor CPU. And Detailed time costs are shown in Appendix C.5.

Guidelines:

The answer NA means that the paper does not include experiments.

1061

1062

1063

1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to all ethical guidelines outlined by NeurIPS. Specifically, we have ensured that our data collection methods are ethical, our experiments are conducted responsibly, and all potential biases are addressed. Additionally, we have considered the broader impacts of our work and have taken steps to mitigate any negative consequences.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We consider both the potential societal impacts and negative societal impacts, and also discuss possible mitigation strategies. Details are shown in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We incorporate six datasets, including four datasets from the Amazon platform[1](Books, Movies & TV, Video Games, and Industrial), Movielens-1M[59], and Book Crossing[60], all of which are open-source. The backend language models used in our research are BERT [4], RoBERTa [5], Llama2-7B [6], Mistral-7B [69], text-embedding-adav2 & text-embeddings-3-large [70], and SFR-Embedding-Mistral [71].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an important and open question, how much user behavior knowledge (generally captured by collaborative filtering models) are present in large language models. This has been a topic attracting significant research interest in recent years. The authors propose that simple linear mappings done on top of LM encoder representations are sufficient to capture collaborative filtering signals in recommendations, and propose a new recommendation method, AlphaRec, which takes pretrained language model content embeddings as input, transforms them via MLPs and lightweight graph convolutions, followed by a contrastive loss. The authors conduct experimental analysis for AlphaRec in both standard settings and zero-shot settings.\n\nSTRENGTHS:\n**S1**: the topic studied is important. It is generally believed that language model and collaborative filtering (recommendations) learn different representation spaces, and methods to bring the two spaces closer is of significant interest to the large community working on search, recommendations, ads, and related topics.\n\n**S2**: the particular approach proposed (linear mapping from textual space to collaborative filtering space) is understudied in prior work on LLM and (Generative) CF, despite numerous papers in recent years.\n\nWEAKNESSES:\n**W1**: the writing in this paper, esp. recommendation system paradigm related discussions, misrepresents (or ignores) significant prior work done in the field. e.g., \"AlphaRec follows a new CF paradigm, which we term the language-representation-based paradigm.\" and related writings. \n\n- Content signals and/or embeddings have been used as the dominant recommendation paradigm in the field, even well before the seminal YouTube DNN paper [1] was published (see e.g., Pazzani and Billsus, 2007 [12]). For recent examples of related work, see e.g., [10, 11] from Pinterest and Meta in KDD'22 (but one should be able to easily find similar papers in WWW KDD etc in prior years as well). \n- Replacing ResNet/ViT- or GPT-/BERT- generated embedding with LLaMa- or Mixtral- generated embeddings cannot and should not be viewed as a paradigm shift, especially given the core architecture of AlphaRec is not substationally different from prior work.\n\n**W2**: AlphaRec needs to be compared with stronger baselines. This applies to many major experiments in the paper. Here are some examples of baselines missing, which may significantly change conclusions obtained and discussions etc:\n\n- vs ID-based recommenders (Table 3).\n    - Equation (2) and line 171-173 for $N_u$ already captures set of items that a user is related to (\"user interaction history\" / \"user engagement history\") to a large extent. Thus, the authors should compare AlphaRec with at least some SotA sequential/generative recommenders, such as SASRec, BERT4Rec, TIGER, HSTU [3, 4, 6, 7]. All of them are missing in the current paper.\n    - Given AlphaRec uses the transposed item id representation - the one layer $N_i$ formulation (equation (2)), relevant work in recent years include Dual contrastive network [8] and User-centric ranking [9]. The authors should compare with or at least discuss some work in this category as related work. \n- Zero-shot performance. (Table 4)\n    - \"Book Crossing\" is not a commonly used benchmark dataset. The \"Industrial\" dataset (per citation [1] on line 273) seems to be a small-scale \"Yelp\" dataset, and should be renamed to avoid confusions.\n    - For ML-1M, the SotA approach one year ago (LLMRank [14]) already achieved 53.73 NDCG@20, significantly higher compared with 32.15  (AlphaRec) in this work. \n\n**W3**: many other formulations/experiments/writings could be significantly improved. Examples include:\n\n- The proposed task formulation does not reflect how recommendation systems work in practice. e.g., \"Line 97-99. Personalized item recommendation with implicit feedback aims to select items i ∈ I that best match user u’s preferences based on binary interaction data Y = [yui], where yui = 1 (yui = 0) indicates user u ∈ U has (has not) interacted with item i [58].\" -- here \"selecting the item that user will interact with\" is not the same as \"selecting the item with the highest reward\", as the interaction itself can be negative (e.g., disliking a recommendation, abandoning session, etc.). See [1, 2] for references.\n\n- A key contribution of this work should be the linear mapping finding. But Table 1 uses a questionable set of baselines for both LMs and CF baselines, which weakens linear mapping related claims.\n    - To claim \"Moreover, with the advances in LMs, the performance of item representation linearly mapped from LMs exhibits a rising trend, gradually surpassing traditional ID-based CF models\" -- I would expect the authors to compare with a single set of models (e.g., LLaMa-2 7B 13B 70B or GPT-3 1.3b 2.7b 6.7b 13b 175b) trained on identical data. As it stands, all models are trained and/or finetuned with different data, so a simpler hypothesis explaining the LM (Linear Mapping) trend is that people are including more and more data into LLM pretraining/finetuning stages, which happen to capture more and more aspects relevant to recommendations. \n    - On the CF side, \"MF\" \"MultiVAE\" and \"LightGCN\" do not represent SotA baselines on Amazon Review datasets (see W2). \n\n- Table 1. Please highlight the particular K used for Recall and HR metrics (hard to find in the paper, applies to other tables too).   Most work on recommendation models also report HR/NDCG/etc. over at least 2-3 Ks to help readers understand how metrics vary with different approaches.\n\n\n- Table 4. [1] should not be labeled as an \"Industrial\" dataset. The cited paper (per line 273) is Ni et al. \"Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects\" which in turn seems to be an publicly available review dataset provided by Yelp. Please use appropriate language as the current writing leads readers to think that AlphaRec is an industrially deployed system. Please refer to industrial papers (e.g., KDD ADS track papers [2, 9, 11, 12]) for how to describe testings done on publicly available industrial sampled datasets (like Yelp), vs real deployments. \n\n- Misc: Contrastive loss is widely used and should not be viewed as a contribution of AlphaRec. See [15, 11] etc.\n\nQUESTIONS:\n**Q1**: Given the paper makes very significant claims, I would strongly recommend setting up experiments that  properly corroborate those claims to avoid the paper appearing like overselling. Examples include:\n* Given linear mapping is a very weak baseline for utilizing LLMs, comparison with recent LLM4Rec approaches including in-context learning, transfer learning, etc. like [14] is necessary.\n* To justify LLMs can increasingly capture collaborating filtering signals, please fix a LLM baseline (eg LLaMa-2 or GPT-3 or Gemma) and vary parameter count (eg up to 70/175b) to show that collaborating filtering knowledge is captured with increased model complexity. Right now a simpler interpretation of Table 1 is that different models use more data, with more data used for recent LLM training.\n* Please compare with appropriate CF baselines, incl popular sequential recommenders in recent years as discussed in W2. \n\nThese experiments would make the findings/conclusions significantly more convincing, and I'm very happy to change rating if linear mapping/AlphaRec remains a competitive approach after these experiments.\n\n\n\n**Q2**: Please consider ways to reframe the CF paradigm discussions as it's unclear how AlphaRec differs from building GCNs or RNNs/Transformers on top of ResNet-/BERT-/GPT-based embeddings, which has been a popular baseline for a long time eg [5, 11, 12]. \n\n\n**References**: \n- [1] Convington et al.  Deep Neural Networks for YouTube Recommendations. RecSys'16. \n- [2] Zhou et al. Deep Interest Network for Click-Through Rate Prediction. KDD'18.\n- [3] Kang et al. Self-Attentive Sequential Recommendation. ICDM'18. \n- [4] Sun et al. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. CIKM'19. \n- [5] Hidasi et al. Session-based Recommendations with Recurrent Neural Networks. ICLR'16. \n- [6] Rajput et al. Recommender Systems with Generative Retrieval. NeurIPS'23.\n- [7] Zhai et al. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. ICML'24. \n- [8] Lin et al. Dual contrastive network for sequential recommendation. SIGIR'22. \n- [9] Zhao et al. Breaking the Curse of Quality Saturation with User-Centric Ranking. KDD'23.\n- [10] Pancha et al. PinnerFormer: Sequence Modeling for User Representation at Pinterest. KDD'22.\n- [11] Rangadurai et al. NxtPost: User To Post Recommendations In Facebook Groups. KDD'22. \n- [12] Pazzani and Billsus.  Content-based recommendation systems. In The adaptive web. 2007.\n- [13] Naumov et al. Deep Learning Recommendation Model for Personalization and Recommendation Systems. 2019. \n- [14] Hou et al. Large Language Models are Zero-Shot Rankers for Recommender Systems. ECIR'24. \n- [15] Klenitsky et al. Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec? RecSys'23.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper states that LLM encodes collaborative signals that make it easy to connect language representation space with an effective recommendation space. Thus, it proposes an effective collaborative filtering model AlphaRec that takes as input only the transformed LLM representations of textual descriptions of items and is trained by InfoNCE loss and graph neural networks. The proposed method outperforms traditional ID-based models and other LM-enhanced methods.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow. \n2. The paper conducts extensive experiments validating the effectiveness of the methods and proves the validity of the design through ablation study and anlysis.\n3. The proposed method exhibits significantly good zero-shot recommendation performance\n\nWEAKNESSES:\n1. One of the most important motivations of the work is that the paper declares large language models encode collaborative signals which indicates the advantage of using representations of large language models for recommendations compared to id embeddings. However, how the preliminary experiments prove this point is insufficiently discussed in the paper. Advanced large language encodes more semantics of the textual descriptions and thus yields better performance. Why this alone doesn't fully explain the performance gain of LLMs should be more explicitly discussed in the main paper.\n2. The novelty is limited. Using semantic embeddings of items has been widely used in recommendations. The novelty mostly lies in using the representations of large language models and the implementation details of how to make it effective when combined with traditional recommendation frameworks like non-linear transformations.\n3. The paper states that language representation-based methods have low training costs. Still, if taking into account the costs of generating language representations, the computational cost is much higher than ID-based methods.\n\nQUESTIONS:\nIt states that the improvements of using a more advanced LLM when using linear mapping of the representations do not merely come from the better feature encoding ability. However, the validating experiments in Appendix B3 is not convincing enough. Randomly shuffling the item representations eliminates the semantic relations between user-item interaction pairs and makes language models unable to leverage their semantic understanding ability, which explains the decline of the performance. Could you please elaborate more on the points you make in Appendix B3?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AlphaRec, a novel method to incorporate both knowledge from pre-trained language models and collaborative signals. Authors firstly reveal the advantages brought from pre-trained embedding model, and then propose three modules within AlphaRec. An MLP layer to transform pre-trained embedding to item-representation. A graph convolution to aggregate neighbor’s information, and the InfoNCE loss to train introduced parameters within the MLP for each dataset. Overall, the novelty of this paper lies within exploration of NLP encoded embedding on RecSys. The graph convolution and InfoNCE loss are already widely used techniques.\n\nSTRENGTHS:\n1. A good exploration on new direction (language-representation-based) RecSys\n2. Experiments are conducted from different angles for analyzing their model.\n\nWEAKNESSES:\n1. Insufficient baselines.\n2. Uncleared model name definition.\n\nQUESTIONS:\n1. Why do authors only encode titles? There is more information within your used Amazon dataset including item descriptions.\n\n2. I personally do not prefer un-informative model names such as the AlphaRec in this paper. Authors mentioned “This model is named AlphaRec for its originality and a series of good properties”. The reason seems so strange to me.\n\n3. More advanced baselines are needed to compare such as the DirectAU [1]  and GraphAU [2] \n\n[1] Wang, C., Yu, Y., Ma, W., Zhang, M., Chen, C., Liu, Y., & Ma, S. (2022, August). Towards representation alignment and uniformity in collaborative filtering. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining (pp. 1816-1825).\n[2] Yang, L., Liu, Z., Wang, C., Yang, M., Liu, X., Ma, J., & Yu, P. S. (2023, October). Graph-based alignment and uniformity for recommendation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (pp. 4395-4399).\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
This paper proposes AlphaRec, an LLM-based recommender system that utilizes language representations of item textual data for recommendations.

STRENGTHS:
+ Investigating ID paradigm and LLM paradigm is important.
+ The method is simple but seems to be effective.

WEAKNESSES:
- In this paper, what most confuses me is the usage of the terminology "collaborative filtering" throughout the paper. In traditional recommender system, collaborative filtering information means the interactions among users and items. The authors find that using LM as feature extractors to get user/item embeddings from meta-data can achieve similar results as if CF is used for recommendation. However, this seems to be fundamentally different than LM has the "collaborative information", as for most online service platforms, the interaction data should be confident and open source LMs won't be able to train on that data. Therefore, the main claim in the paper seems questionable.

- It would be beneficial if we could have results on more diverse datasets.

QUESTIONS:
Please refer to my summary of weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-18