Preference Discerning in Generative Sequential Recommendation

Authors: Fabian Paischer, Liu Yang, Linfeng Liu, Shuai Shao, Kaveh Hassani, Jiacheng Li, Ricky T. Q. Chen, Zhang Gabriel Li, Xiaoli Gao, Wei Shao, Xue Feng, Nima Noorshams, Sem Park, Bo Long, Hamid Eghbalzadeh
Primary area: generative models
Keywords: Generative Retrieval, Sequential Recommendation, Preference Discerning, LLM

Abstract:
Sequential recommendation systems aim to provide personalized recommendations for users based on their interaction history. To achieve this, they often incorporate auxiliary information, such as textual descriptions of items and auxiliary tasks, like predicting user preferences and intent. Despite numerous efforts to enhance these models, they still suffer from limited personalization. To address this issue, we propose a new paradigm, which we term *preference discerning*. In *preference discerning*, we explicitly condition a generative sequential recommendation system on user preferences within its context. The user preferences are generated by large language models (LLMs) based on user reviews. To evaluate *preference discerning* capabilities of sequential recommendation systems, we introduce a novel benchmark that provides a holistic evaluation across various scenarios, including preference steering and sentiment following. We assess current state-of-the-art methods using our benchmark and show that they struggle to accurately discern user preferences. Therefore, we propose a new method named Mender (**M**ultimodal prefer**en**ce **d**iscern**er**), which improves upon existing methods and achieves state-of-the-art performance on our benchmark. Our results show that Mender can be effectively guided by human preferences, paving the way toward more personalized sequential recommendation systems. We will open-source the code and benchmarks upon publication.

Decision: reject
Reviewer scores: 6, 3, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729171159440

thread_type:
official_review

<thread_object>
creator: reviewer_fpwn

<content>
summary:
This paper first introduces a new benchmark to evaluate the model's ability to capture user preference. Then Mender is proposed to integrate LLM-generated user preferences to enhance the generative recommender system. Experiment results on its proposed benchmark show improvement on the proposed method.

strengths:
1. The motivation of this work, leveraging user preference in recommender system, is good.
2. The author conducts extensive experiments.

weaknesses:
Regarding Benchmark Design:

1. While preference-based recommendation is undoubtedly a core aspect, the practical value of the tasks such as Sentiment Following, Fine-Grained & Coarse-Grained Steering, and History Consolidation is questionable. This raises concerns about the overall contribution of the benchmark.
2. The Fine-Grained & Coarse-Grained Steering task is confusing. The paper states, “we associate these two items with different user preferences, denoted as p1 and p2, respectively,” but the relationship between p1, p2, and similar or distinct items is unclear. How are “different user preferences” determined? Additionally, in the new sequences created, why is p1 added to the sequence of very distinct items while p2 is added to the ground truth item sequence? This contradicts the earlier association of p1 and p2 with similar and distinct items, respectively. What role does the similar item play?
3. The design of the Sentiment Following task does not adequately reflect the model’s ability to follow sentiment. The description is also unclear, and I suggest the authors reorganize this section.
4. The practical value of History Consolidation is questionable, and its evaluation metric seems unnecessary. Why not train the model directly using the five preferences? The paper claims to “infer which preference is most relevant for predicting,” but there is no experimental evidence demonstrating this capability. In fact, the performance with multiple preferences is even worse than with a single preference.
5. The experimental discussion on each task, particularly Sentiment Following and History Consolidation, is insufficient.

Regarding Presentation:
1. Missing reference on line 1275.
2. Typo on line 1167: "tr iggered" should be "triggered."
3. Typo on line 401: "48.3.4%" should be corrected.
4. Method names are displayed incorrectly in lines 403-406.
5. In Table 2, the performance drop for History Consolidation on the Steam dataset seems miscalculated. The relative decline should be based on the better-performing Mender-emb, not Mender-Tok.
6. The section titled "PREFERENCE DISCERNING" in part three should likely be part of the Methodology (Section 4.2). It is unclear why this is presented as a separate section.

Regarding Experiments:
1. The selection of baselines is insufficient, with only three included. One of these, TIGER, is an ID-based model that does not leverage preference, making the comparison unfair. The two VocabExt variants either introduce a gap between randomly initialized item embeddings and semantic information, or they lack pre-trained preference understanding, making them variants of TIGER rather than fair comparisons. The authors should consider two sets of baselines: (1) preference-based recommendation models and (2) advanced TIGER variants, such as LETTER, LC-Rec.
2. The statement in line 282, “Mender-Emb allows pre-computing item and preference embeddings, resulting in improved training efficacy,” conflicts with the experimental results, as Mender-Emb consistently underperforms compared to Mender-Tok in Table 2.
3. Although the benchmark is a key contribution of the paper, there is insufficient discussion of most tasks in the experimental section, especially History Consolidation and Sentiment Following.
The lower performance of History Consolidation compared to Recommendation raises questions about the usefulness of combining five preferences versus a single preference. This casts doubt on both the validity of the preference design and the method’s ability to effectively leverage preferences. Additionally, the abnormal results on the Steam dataset lack sufficient discussion and explanation.

questions:
Please refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1729749128902

thread_type:
official_review

<thread_object>
creator: reviewer_1g5m

<content>
summary:
This paper introduces a new benchmark and proposes a novel method called Mender to evaluate and enhance the preference discerning capabilities of sequential recommendation systems. The benchmark assesses models across five dimensions, focusing on their capacity to extract and utilize user preferences from datasets. Recognizing that existing methods lack key capabilities of preference discerning, the authors propose Mender, a multimodal generative retrieval approach which effectively extracts user preferences and achieves state-of-the-art performance on the proposed benchmark. Experimental results further demonstrate the effectiveness of the method.

strengths:
1. This paper identifies a critical issue in sequential recommendation: the failure to explicitly capture and utilize user preferences, prompting the introduction of a new task: preference discerning.
2. It proposes a novel benchmark comprising five key dimensions to evaluate the preference discerning abilities of existing models.
3. The paper enhances the RQ-VAE framework by directly representing both user interaction history and preferences, while also introducing two variants that encode inputs in different ways.
4. Extensive experiments are conducted, accompanied by detailed analysis to validate the findings.

weaknesses:
1. No NDCG results for sentiment following are reported in Table 2 or the Appendix. I assume the results are close to zero, suggesting that a logarithmic scale should be used to better analyze the discrepancies between the models. 
2. The time factor is not considered, given that user preferences can change significantly over time, especially considering that the time span of the datasets can be decades. Simply limiting the user sequence to the 20 most recent items does not fully eliminate time bias. Instead, the time interval of user-item interactions should be restricted during sampling to better capture user preferences.
3. The methodology outlined in Section 4 is unclear, with similar issues arising in Section 4.1 on Fine-Grained & Coarse-Grained Steering, where the concepts are not adequately explained. In Equation 1, the entire sequence is considered, while the subsequent statement describes repeating the process for each item in the sequence, leading to ambiguity. Furthermore, in Fine-Grained & Coarse-Grained Steering, there is no reference provided to justify the validity of this approach. The sequence processing in this section also lacks rationality, as it combines a distinct item $\hat{i}_t$ with $p_1$, which represents the preference of a similar item. 
4. Experiments with only three baselines is not convincing enough, new baselines should be added, including https://arxiv.org/abs/2311.09049 (Zheng, Bowen, et al. "Adapting large language models by integrating collaborative semantics for recommendation." ICDE 2024). This paper also employs RQ-VAE, in which LLM-encoded text embedding of the item is utilized as input.

questions:
1. Given the results of Mender$_{Tok}$-Pos-Neg in Figure 5, achieving the best sentiment following results does not necessarily ensure the model's proficiency in other dimensions. Does the pursuit of high performance in sentiment following adversely affect the model's overall capabilities?
2. Why is the time factor not considered, given that user preferences can change significantly over time, especially considering that the time span of the datasets can be decades? What's the implications of not considering the time factor in the current model?
3. In Section 4,  Equation 1 already takes every item in the sequence except the last item $i_{T_{u}}$ into account, then what's the meaning of "repeating this generation process for each item in $s_{u}$" in line 203?
4. In Section 4.1, line 237, why the steering ability can be achieved by creating new sequences? Is there a reference can prove this? In Appendix D.2, line 1276, a figure reference is missing.
5. Still in Section 4.1, line 241, how is $p_{1}$ and $p_{2}$ achieved? What's the point of combining them with new sequences?
6. Still in Section 4.1, line 243, since $\hat{i}_t$ represents a distinct item, why its sequence combines with $p_1$, which represents the preference of a similar item?
7. Why the NDCG results of sentiment following are not provided in Table 2 and other tables in the Appendix?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729993555831

thread_type:
official_review

<thread_object>
creator: reviewer_2w9u

<content>
summary:
The submission focuses on sequential recommendation technique. The main contributions lie that (1) the authors proposed a LLM-based user preference generation method based on user generated reviews and (2) they propose an evaluation framework that contains five different aspects that should be taken into consideration by sequential recommendation systems.

strengths:
1. The authors would like to propose a thorough evaluation framework (the authors claim it to be a "benchmark") for sequential recommendation system, which is a nice direction.

2. The experimental results are credible and show improvement compared with some existing baselines.

weaknesses:
1. The novelty is rather limited. Extracting user preference information from their reviews is not novel. However, these existing works are not mentioned or compared by the authors. Some examples include:

User-LLM: Efficient LLM Contextualization with User Embeddings, https://arxiv.org/abs/2402.13598  (The work investigates how to capture latent user behaviors into preferences)
Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction, https://arxiv.org/abs/2305.06474 (This work investigates how LLMs comprehend user preferences and compares the performances of different LLMs in this issue.)
Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation, https://arxiv.org/abs/2408.06276 (this work proposes to extract subjective preferences from raw reviews, which is a key contribution the authors claim)

2. The proposed evaluation benchmark is rather straightforward and not so reasonable. This kind of framework should be validated by either product managers or large scale user studies. The radar chart indicates that these five dimensions are equally important, which is also not validated by any evidence. If the authors would like to propose such a framework, I would suggest compare the proposed one with actual user experiences through practical user studies.

questions:
Why do you think the five factors in your "evaluation benchmark " are equally important and are the only concerns by sequential recommendation systems?

What is the difference between your proposed preference generation framework with existing ones (as listed in "weakness" section).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722170414

thread_type:
official_review

<thread_object>
creator: reviewer_posa

<content>
summary:
This paper proposes MENDER, aiming to enhance the personalization of sequential recommendation models. Specifically, the authors first design a preference discerning paradigm based on zero-shot LLMs. With the obtained user preference, the authors construct the generative recommendation framework based on RQ-VAE. Extensive experimental results are provided to show its effectiveness.

strengths:
- Novel Benchmark. The authors suggest a new benchmark to evaluate the personalization ability of the user preference description. 
- Abundant Results. Extensive experiments are conducted. 
- Credible Reproduction. Reproduction details are available in Appendix.

weaknesses:
- Limited Technical Contribution. Overall, the framework is constructed based on existing works. The proposed method, MENDER, mainly consists of two modules, RQ-VAE and feed-forward components (Emb or Tok). Other researchers have suggested both modules, which may indicate the limited technical contribution of this work. 
- Lack of the Cross-validation on Benchmark. The suggested "holistic" benchmark is subjective and not double-checked by objective ground truth. The overall performance only reflects the indirect effectiveness of additional preference summarization, while the success of preference discerning should be further validated. 
- Inadequate Motivation. The motivation for enhancing the personalization and applying the generative recommendation is not supported. How do authors define "personalization" and examine "personalization"? Why do authors only construct the generative model? Can we integrate MENDER with discriminative models? 
- Unknown Efficiency. The efficiency of the proposed framework has not been tested.

Minor problems:
- The last block of Table 2 is in the wrong format. 
- Code is not available.

questions:
1. What is the major technical contribution of MENDER? 
2. Are there some direct and objective evaluation methods to check the effectiveness of preference-discerning results? 
3. What is the motivation for using the generative recommendation pipeline? 
4. Is MENDER a efficient method compared with traditional sequential recommendation baselines as SASRec?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722668717

thread_type:
official_review

<thread_object>
creator: reviewer_kam7

<content>
summary:
This paper aims to enhance personalized recommendations by explicitly incorporating user preferences and historical interactions. The proposed method Mender uses a pre-trained language model to generate user preferences from comments and integrates these preferences with historical data using cross-attention mechanisms. Experimental results show that Mender outperforms existing state-of-the-art methods.

strengths:
-	This paper uses four diverse datasets to ensure the generalizability and reliability of the experimental results.

-	The proposed model, Mender, is evaluated across multiple dimensions such as preference-based recommendation, sentiment following, fine-grained and coarse-grained steering, and history consolidation. The results show that Mender significantly outperforms existing state-of-the-art methods, particularly in preference guidance and sentiment following, demonstrating its robustness and effectiveness.

weaknesses:
-	The methodology section should be reorganized to provide a detailed explanation of the preference generation process. Mathematical formulations are expected to be included for explicit understanding, and pseudo-code is recommended to enhance clarity and reproducibility.
-	It is kindly recommended to add further discussion about how does the benchmark generation benefit personalization modeling.

questions:
Please refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732274864980

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to express our gratitude to all the reviewers for their invaluable and constructive feedback, which has significantly helped us improve our manuscript. We have addressed all the concerns raised in the individual responses, and have also briefly summarized them below:
- **[Kam7,posA,2w9u,1g5m,Fpwn]** To enhance clarity, we revised parts of  the introduction, merged Section 3 with Section 4, and included pseudocode (Algorithm 1) for the preference generation pipeline. We also added mathematical formulations to clarify the benchmark construction, with a particular focus on sentiment following, fine/coarse-grained steering, and history consolidation. Additionally, we provided a more in-depth discussion on the empirical evidence related to all these aspects.
- **[posA,Fpwn]** We conducted a user study with 22 participants, during which we assessed 2,200 generated preferences to evaluate their accuracy (see Appendix F). The results indicate that, on average across datasets, approximately 74% of the preferences accurately reflect the users' true preferences.
- **[Fpwn,1g5m]** We clarified the confusion regarding the LC-ReC baseline, and introduced an additional baseline that also represents the interaction history in text instead of semantic ids. MenderTok significantly outperforms this new baseline, as shown in Table 2.
- **[posA,Fpwn]** We included the training and inference times for our Mender variants in Table 7 (Appendix E), highlighting the advantages of MenderEmb and comparing it to SASRec
- **[2w9u,posA]** We clarified our contributions.
- **[Fpwn]** We included a comparison of MenderTok trained on all user preferences to demonstrate that this training setup is not advantageous, as it leads to detrimental performance compared to the standard MenderTok. This is detailed in Table 8 (Appendix E).

All changes are highlighted in red in the updated manuscript. We look forward to an engaging discussion, and welcome further feedback from the reviewers to continue improving our manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a new paradigm termed "preference discerning" for generative sequential recommendation systems, along with a novel benchmark and a new baseline model named Mender. The core idea is to condition generative models on user preferences derived from user reviews to enhance personalization in sequential recommendation tasks. The paper also evaluates the performance of current state-of-the-art methods and shows that they struggle with the proposed benchmark tasks, thereby positioning Mender as a solution that achieves improved results across these scenarios.

The paper’s strengths lie in its motivation to address the personalization gap in sequential recommendation systems and its attempt to establish a comprehensive benchmark for evaluating preference discerning capabilities. The authors have conducted extensive experiments on multiple datasets and evaluation scenarios, showing improvements over certain baselines. Furthermore, the authors made an effort to address reviewers' concerns by adding a user study, introducing new baselines, and refining methodological explanations in the rebuttal phase.

However, the weaknesses of this paper are substantial. First, the novelty of the proposed approach, both in terms of the framework and methodology, is limited. The preference generation and model architecture largely rely on existing techniques, and while the benchmark is claimed to be comprehensive, its practicality and validation remain questionable. Reviewers noted that several tasks, such as fine-grained and coarse-grained steering, sentiment following, and history consolidation, lack sufficient justification and real-world applicability. Furthermore, the benchmark axes seem subjective and were not validated by large-scale user studies or feedback from domain experts, which reduces the credibility of the proposed evaluation framework. The experimental baselines, while improved during the rebuttal, are still incomplete, as critical comparative models such as LETTER and LC-Rec are missing. Additionally, there are methodological ambiguities, particularly in the preference steering and history consolidation tasks, which make the paper difficult to follow.

The most important reason for my decision to recommend rejection is the limited technical contribution of the work combined with insufficient validation of its proposed evaluation framework. While the paper shows promise in its ambition to incorporate user preferences into sequential recommendation models, it falls short of making a substantial scientific contribution or demonstrating the practical utility of its benchmark. The lack of engagement from reviewers during the rebuttal period further suggests that the authors’ revisions and explanations were not sufficient to address the key concerns.