Language Representations Can be What Recommenders Need: Findings and Potentials

Authors: Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, Tat-Seng Chua
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Collaborative filtering, Language-representation-based recommendation, Language models, Language model representations

Abstract:
Recent studies empirically indicate that language models (LMs) encode rich world knowledge beyond mere semantics, attracting significant attention across various fields.
However, in the recommendation domain, it remains uncertain whether LMs implicitly encode user preference information. Contrary to prevailing understanding that LMs and traditional recommenders learn two distinct representation spaces due to the huge gap in language and behavior modeling objectives, this work re-examines such understanding and explores extracting a recommendation space directly from the language representation space.
Surprisingly, our findings demonstrate that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance.
This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation, implying that collaborative signals may be implicitly encoded within LMs.
Motivated by the finding of homomorphism, we explore the possibility of designing advanced collaborative filtering (CF) models purely based on language representations without ID-based embeddings.
To be specific, we incorporate several crucial components (i.e., a multilayer perceptron (MLP), graph convolution, and contrastive learning (CL) loss function) to build a simple yet effective model, with the language representations of item textual metadata (i.e., title) as the input.
Empirical results show that such a simple model can outperform leading ID-based CF models on multiple datasets, which sheds light on using language representations for better recommendation.
Moreover, we systematically analyze this simple model and find several key features for using advanced language representations:
a good initialization for item representations, superior zero-shot recommendation abilities in new datasets, and being aware of user intention.
Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.

Decision: accept (oral)
Reviewer scores: 8, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1729049805109

thread_type:
official_review

<thread_object>
creator: reviewer_m7fa

<content>
summary:
This paper explores the relationship between the language space generated by LLMs and the behavior space produced by CF models. The authors found that the two spaces exhibit a pattern of homomorphism. Based on this finding, they proposed AlphaRec, which uses item embeddings exclusively from the language space and employs a trainable projector to map them into the behavior space for recommendations. The results demonstrate that with language space-initialized embeddings, the model significantly outperforms traditional ID-based methods and other LLM-based approaches in general recommendation, zero-shot recommendation, and intention-aware recommendation.

strengths:
- The questions explored in this paper are novel and meaningful. While many existing works have analyzed the recommendation capabilities of language models, very few have empirically investigated the differences between the representation spaces of LMs and CF models.

- The evaluations are comprehensive. The authors not only investigate the recommendation capabilities of AlphaRec but also examine its performance in zero-shot and intention-aware recommendations. Additionally, the visualization of the LM and behavior spaces is a plus.

weaknesses:
- There are some concerns that affect the soundness and the conclusions presented in the paper (see point 2 of weakness and questions below).

- Some of the comparisons are not entirely fair. For example, in Figure 3 and Table 3, it is evident that when using BPR loss (which is used in LightGCN), AlphaRec’s performance is a bit worse than that of LightGCN. Given that AlphaRec shares a similar graph convolution architecture with LightGCN in its nonlinear projection, it appears that AlphaRec’s advantage over LightGCN comes from the InfoNCE loss rather than the language representations. I am wondering whether such result can support the power of language representations. By the way, I suggest aligning the implementations of KAR and RLMRec closer with AlphaRec, including the loss function and some of the modules. Without a fair comparison, it is hard to conclude that the combination of ID-based embeddings and language representations is less suitable than using pure language representations.

questions:
- Since MovieLens and Amazon Review are classical and widely used datasets publicly available on the Internet, I am concerned that they might already be part of the pretraining data for some LLMs, hence the user preference information encoded in LMs could be a potential data leakage. This could explain why BERT-style models perform worse than more recent LMs. To verify this assumption, can authors evaluate AlphaRec’s performance on a private or newly published recommendation dataset?

- If authors want to claim the collaborative signals are already encoded in the language space, I wonder if authors can show the results using original language representations without any linear or nonlinear transformations? If there is valid information can show user behaviour in language space, the performance should be comparable to some traditional CF methods (like MF) or heuristic methods (e.g., Pop).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730066455673

thread_type:
official_review

<thread_object>
creator: reviewer_uzkz

<content>
summary:
This paper studies the potential of leveraging advanced language models (LMs) for recommendation tasks, exploring whether language representations can encode user preferences and collaborative signals. Traditionally, CF-based recommendation systems use ID-based embeddings to model user-item interactions. The authors propose a method that directly maps language representations of item metadata (like titles) into a behavior space to serve as effective item representations. They introduce a linear mapping approach, followed by the AlphaRec model, which combines language embeddings with collaborative filtering elements such as graph convolution and contrastive loss.

Generally, this is a very good paper provides new insights in the community: homomorphism exists between LM- based language space and user behavior space for recommendation.

strengths:
1. The topic on studying the relationship between the language space and behavior space is important to the community. The authors provide new insights to show the homomorphism between the two spaces.

2. The paper’s core contribution, mapping language representations into a behavior space, is thoroughly tested and demonstrates strong empirical performance. This novel approach could reduce reliance on large datasets of historical behavior data, making the method highly applicable.

3. Followed by the observation through linear mapping, the authors also propose a stronger model, AlphaRec, through non-linear mapping and GNN. Its ability to perform zero-shot recommendations and its efficient training process are well-demonstrated in multiple datasets, addressing common challenges in recommendation systems where user or item IDs change frequently. 

4. The paper’s evaluation covers a range of experimental setups, comparing multiple LMs, testing robustness to input noise, and conducting ablation studies. The discussion on the potential of language-based user representation (Section 5) is promising.

weaknesses:
1. While the paper presents a compelling case for using language representations in recommendation, a discussion on how AlphaRec compares to existing fine-tuned LLM approaches would strengthen it further. For example, it would be helpful to understand whether the authors see AlphaRec as complementary or competitive with fine-tuned LLM-based approaches (e.g., [1]), and if they anticipate similar benefits, limitations, or unique strengths compared to those models.

[1] Tennenholtz et al. Demystifying Embedding Spaces using Large Language Models. ICLR 2024.


2. The authors assume that a single linear mapping can capture collaborative signals for all users, which may overlook the diversity in individual user preferences and linguistic nuances. Different users might interpret language in varied ways, and a one-size-fits-all mapping could limit the model’s ability to fully capture such personalization and diversity. How might the authors address this challenge within their framework? Exploring the possibility of user-specific mappings or personalization approaches could be a valuable direction for future work, potentially enhancing the model’s adaptability across diverse user groups.

questions:
The discussion on the potential of intent-aware ability is interesting. However, as indicated in Appendix D.3.1, a fixed user intention query is generated for each item in the dataset. I would argue that this is not optimal, as different users may have different intents even when consuming the same item. Could the authors consider extending their approach to allow for dynamic query generation or user-specific intent modeling, potentially improving recommendation relevance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690283718

thread_type:
official_review

<thread_object>
creator: reviewer_qrqf

<content>
summary:
The paper challenges the idea that Language Models (LMs) and traditional recommenders exist in separate representation spaces. It proposes deriving a recommendation space directly from language representations. By mapping item representations from advanced LM outputs, it shows improved recommendation performance, hinting at a homomorphism between language and item representation spaces. The paper introduces collaborative filtering models based solely on language representations, excluding ID-based embeddings. Components like multilayer perceptrons, graph convolution, and contrastive learning are integrated into the model.

strengths:
1. The authors propose using linear mapping of LLMs' representations to directly model user preferences, which is both efficient and effective.

2. The authors conducted comprehensive experiments discussing the performance of AlphaRec in full-shot, zero-shot, and other scenarios, comparing it with baseline methods from various research lines.

weaknesses:
1. From Table 1, it can be observed that the performance of AlphaRec is heavily dependent on the performance of the chosen language model. When selecting smaller language models (such as BERT, RoBERTa, etc.), the performance is less than ideal. Additionally, the costs associated with state-of-the-art methods (i.e., SFR-Embedding-Mistral, text-embeddings-3-large, etc.), including inference costs or API call expenses, cannot be overlooked, which weakens the applicability of this method.

2. The authors omitted a method that performs well in zero-shot scenarios, VQ-Rec [1], when exploring zero-shot capabilities.

3. In the ablation study section, it can be observed that AlphaRec without Contrastive Learning (CL) performs significantly worse than Linear Mapping, indicating that simply adding MLP and GCN on top of Linear Mapping does not lead to effective performance improvements.

4. If LLMs' representations and MLP are applied to state-of-the-art ID-based methods (e.g., XSimGCL) to replace the original ID embeddings, would it achieve results comparable to those of AlphaRec?

[1] Hou, Yupeng, et al. "Learning vector-quantized item representation for transferable sequential recommenders." Proceedings of the ACM Web Conference 2023. 2023.

questions:
Please refer to the issues mentioned in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730793287676

thread_type:
official_review

<thread_object>
creator: reviewer_v7nb

<content>
summary:
This paper explores the relationship between language space and behavior space in the context of recommendation systems, investigating the potential of using language representations for this purpose. The proposed method, AlphaRec, demonstrates excellent performance in zero-shot recommendation scenarios.

strengths:
1. This paper is well-written and easy to follow.
2. Leveraging textual information for recommendations and exploring zero-shot performance is both novel and interesting.
3. The experiments are comprehensive and effectively demonstrate the proposed method's effectiveness.

weaknesses:
1. It would be beneficial to conduct an additional ablation study to evaluate different types of text embedding models on AlphaRec.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper explores the relationship between language space derived from LMs and user-behavior space in RecSys, investigating whether language representations can encode user behaviors, especially collaborative signals. The proposed method, AlphaRec, integrates language representations with collaborative filtering, showcasing strong performance in zero-shot and intention-aware recommendation scenarios.
Strengths:
- The paper tackles a meaningful and underexplored problem, providing insights into the homomorphism between language and behavior spaces in recommendation.
- The proposed method AlphaRec addresses common challenges in recommendation systems, such as frequent changes in user or item IDs, and highlights the potential of language-based models for scalable, zero-shot recommendation tasks.
- The performance of AlphaRec is thoroughly evaluated across diverse datasets and scenarios, including full-shot, zero-shot, and intention-aware recommendations. Comprehensive experiments include comparisons with various baseline methods and ablation studies, providing robust evidence of its effectiveness.

Overall, the paper presents the understanding and application of language models in recommendation systems well.