LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation

Authors: qidong  liu
xian  wu
yejing  wang
zijian  zhang
feng  tian
yefeng  zheng
xiangyu  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:15:12

Abstract: Sequential recommender systems (SRS) aim to predict users' subsequent choices based on their historical interactions and have found applications in diverse fields such as e-commerce and social media. However, in real-world systems, most users interact with only a handful of items, while the majority of items are seldom consumed. These two issues, known as the long-tail user and long-tail item challenges, often pose difficulties for existing SRS. These challenges can adversely affect user experience and seller benefits, making them crucial to address. Though a few works have addressed the challenges, they still struggle with the seesaw or noisy issues due to the intrinsic scarcity of interactions. The advancements in large language models (LLMs) present a promising solution to these problems from a semantic perspective. As one of the pioneers in this field, we propose the Large Language Models Enhancement framework for Sequential Recommendation (LLM-ESR). This framework utilizes semantic embeddings derived from LLMs to enhance SRS without adding extra inference load. To address the long-tail item challenge, we design a dual-view modeling framework that combines semantics from LLMs and collaborative signals from conventional SRS. For the long-tail user challenge, we propose a retrieval augmented self-distillation method to enhance user preference representation using more informative interactions from similar users. To verify the effectiveness and versatility of our proposed enhancement framework, we conduct extensive experiments on three real-world datasets using three popular SRS models. The results consistently show that our method surpasses existing baselines. The implementation code is available in Supplementary Material.

Main paper: 
 introduction: 
 The objective of sequential recommendation is to predict the next likely item for users based on their historical records [7, 53] . Owing to its wide-ranging applicability in various domains such as ecommerce [47] and social media [5] , sequential recommendation has garnered considerable attention in recent years. Given that the essence of sequential recommendation revolves around extracting user preferences from their interaction records, several innovative architectures have been proposed. For to note that the proposed framework is model-agnostic, allowing it to be adapted to any sequential recommendation model. The contributions of this paper are as follows:• We propose a large language models enhancement framework, which can alleviate both long-tail user and item challenges for SRS by introducing semantic information from LLMs.• To avoid the inference burden of LLMs, we design an embedding-based enhancement method. Besides, the derived embeddings are utilized directly to retain the original semantic relations.• We conduct extensive experiments on three real-world datasets with three backbone SRS models to validate the effectiveness and flexibility of LLM-ESR.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The contributions and scope of the paper are included in the abstract and Section 1. Please refer to the first and last paragraph of Section 1 for scope and contributions, respectively.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
problem definition: 
 The goal of the sequential recommendation is to give out the next item that users are possible to interact with based on their interaction records. The set of users and items are denoted as U = {u 1 , . . . , u i , . . . , u |U | } and V = {v 1 , . . . , v i , . . . , v |V| }, respectively, where |U| and |V| are the number of users and items. Each user has an interaction sequence, which arranges the interacted items by timeline, denoted as S u = {vEQUATIONi , . . . , vnu }. n u represents the interaction number of user u. For simplicity, we omit the superscript (u) in the following sections. Then, the problem of sequential recommendation can be defined as follows:EQUATIONFollowing the existing works related to long-tailed SRS [17, 20] , we can split the users and items into tail and head groups. Let n u and p v denote the length of the user's interaction sequence and the popularity of the item v (i.e., the total interaction number). Firstly, we sort the users and items by the values of n u and p v in descending order. Then, take out the top 20% users and items as head user and head item according to Pareto principle [4] , denoted as U head and V head . The rest of the users and items are the tail user and tail item, i.e., U tail = U \ U head and V tail = V \ V head . To alleviate the long-tail challenges, we aim to elevate the recommending performance for U tail and V tail .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: A limitation section is included in the appendix (Section D).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
llm-esr 3.1 overview: 
 The overview of the proposed LLM-ESR is shown in Figure 2 . To acquire the semantic information, we adopt LLMs to encode textual users' historical interactions and items' attributes into LLMs user embedding and LLMs item embedding. Then, two modules are proposed to augment long-tail items and long-tail users, respectively, i.e., Dual-view Modeling and Retrieval Augmented Self-Distillation. i) Dual-view Modeling: This module consists of two branches. One is semantic-view modeling, which aims to extract the semantic information from the user's interaction sequence. It first utilizes the semantic embedding layer, derived from LLMs item embedding, to encode the items.Then, an adapter is designed for dimension adaptation and space transformation. The output item embedding sequence will be fed into cross-attention for fusion and then sequence encoder to get the user representation in semantic view. The other branch is collaborative-view modeling, which transforms the interaction sequence into an embedding one by a collaborative embedding layer.Next, followed by a cross-attention and the sequence encoder, the collaborative user preference is obtained. At the end of this module, the user representations in the two views will be fused for the final recommendations. ii) Retrieval Augmented Self-Distillation: This module expects to enhance long-tail users through informative interactions of similar users. First, the derived LLMs user embedding is considered as a semantic user base for retrieving similar users. Then, similar users are fed into dual-view modeling to get their user representations, which are the guide signal for self-distillation. Finally, the derived distillation loss will be utilized as an auxiliary loss for training.The traditional SRS models are skilled in capturing collaborative signals, which can recommend for popular items well [20, 17] . However, they compromise on long-tail items due to the lack of semantics [2] . Therefore, we model the preferences of users from the dual views to cover all items simultaneously. Besides, we propose a two-level fusion to better combine the benefits from both two.Semantic-view Modeling. In general, the attributes and descriptions of items contain abundant semantics. To utilize the powerful semantic understanding abilities of LLMs, we organize the attributes and descriptions into textual prompts (the template of prompts can be found in Appendix A.1). Then, in avoid of possible inference burden brought by LLMs, we cache the embeddings derived from LLMs for usage. In specific, the embeddings can be obtained by taking out the last hidden state of open-sourced LLMs, such as LLaMA [50] , or the public API, such as text-embedding-ada-002 2 .We adopt the latter one in this paper. Let E se ∈ R |V|×d llm denotes the LLMs embedding of all items, where d llm is dimension of LLMs embedding. Then, the semantic embedding layer E se from LLMs can be used for semantic-view modeling to enhance long-tail items. However, previous works [13, 16] often adapt it as the initialization of the item embedding layer, which may ruin the original semantic relations during fine-tuning. In order to retain the semantics, we freeze the E se and propose an adapter to transform the raw semantic space into the recommending space. For each item i, we can get its LLMs embedding e llm i by taking the i-th row of E se . Then, it will be fed into the tunable adapter to get the semantic embedding:EQUATIONwhereW a 1 ∈ R d llm 2 ×d llm , W a 2 ∈ R d× d llm 2 and b a 1 ∈ R d llm 2 ×1 , b a 2 ∈ R d×1are the weight matrices and bias of adapter. Following this process, we can obtain the item embedding sequence of the user's interaction records, denoted as S se = [e se 1 , . . . , e se nu ]. Similar to a general SRS model, we employ a sequence encoder f θ (e.g., self-attention layers [51] for SASRec [18] ) to get the representation of user preference in semantic view as follows:EQUATIONwhere u se ∈ R d×1 is the user preference representation in semantic view and θ denotes the parameters of sequence encoder in an SRS model.Collaborative-view Modeling. To utilize the collaborative information, we adopt a trainable item embedding layer and supervised update it by interaction data. Let E co ∈ R |V|×d denotes the collaborative embedding layer of the item. Then, the item embedding sequence S co = [e co 1 , . . . , e co nu ] is acquired by extracting the corresponding rows from E co . To get the user preference u co in the collaborative view, we input embedding sequence to sequence encoder, i.e., u co = f θ (S co ). It is worth noting that, the sequence encoder f θ is the same one in both semantic and collaborative views for the shared sequential pattern and higher efficiency [45] . Besides, the embedding layers in the two views are in unbalanced training stages (one is pretrained, while the other is from scratch), which may lead to optimization difficulty [1] . To handle such a problem, we initialize the E co by dimension-reduced E se . The Principal Component Analysis (PCA) [43] is used as the dimension reduction method in this paper.Two-level Fusion. The effective integration of both semantic-view and collaborative-view is essential to absorb the benefits of these two. However, the direct merge of the user representations in dual views may overlook the nuanced inter-relationships between item sequences. Thus, we design a two-level fusion method for the dual-view modeling module, i.e., sequence-level and logit-level. The former aims to implicitly capture the mutual relationships between the item sequences of dual views, while the latter explicitly targets the combination of recommending abilities. In specific, we propose a cross-attention mechanism for sequence-level fusion. To simplify the description, we only take the semantic view interacting with the collaborative view for illustration, and the other view is the same. Specifically, S se is considered as the query, and S co as the key and value in attention mechanism. LetQ = S se W Q , K = S co W K , V = S co W V , where W Q , W K , W V ∈ R d×d are weight matrices.Then, the interacted collaborative embedding sequence can be formulated as follows:EQUATIONFollowing the same process of cross-attention, we can also get the corresponding semantic embedding sequence Ŝse . Finally, S se , S co are substituted by Ŝse , Ŝse to be fed into f θ (•). As for logit-level fusion, we concatenate the two-view user and item embeddings for recommendation. The probability score of recommending item j for the user u is therefore calculated as:EQUATIONwhere ":" denotes the concatenation operation of two vectors. Based on the probability score, we adopt the pairwise ranking loss to train the framework:EQUATIONwhere v + k+1 and v - k+1 are the ground-truth item and paired negative item. It is worth noting that the ranking loss may differ a little according to different backbone SRS models, e.g., sequence-to-one pairwise loss for GRU4Rec [14] .The long-tail user problem originates from the lack of enough interactions for the sequence encoder in an SRS to capture users' preferences. Thus, we propose a self-distillation method to augment the extraction capacity of the sequence encoder. Self-distillation [12, 60] is a type of knowledge distillation that considers one model as both the student and teacher for model enhancement. As for the SRS, since multiple similar users have more informative interactions, it is promising to transfer their knowledge to the target user for strengthening. Thereafter, there are two key challenges for such knowledge transfer, i.e., how to retrieve similar users and how to transfer the knowledge.Retrieve Similar Users. Previous works have confirmed that LLMs can understand the semantic meanings of textual user interaction records for recommendation [25, 10] . Based on their observation, we organize the item's title that interacted by users into the textual prompts (the template of prompts can be found in Appendix A.1). Then, similar to the derivation of LLMs item embedding E se , we can obtain and save the LLMs user embedding, denoted as U llm ∈ R |U |×d llm . It is also dubbed as the semantic user base in this paper, because the semantic relations are encoded in it. For each target user k, we can retrieve the similar user set U k as follows:EQUATIONwhere cos(•, •) is the cosine similarity function to measure the distance between two vectors. N represents the size of similar user sets, which is a hyper-parameter.Self-Distillation. As mentioned before, we design the self-distillation to transfer the knowledge from several similar users to the target user. Since the representation of user preference, i.e., u se and u co , encode the comprehensive knowledge of the user, we configure such representation as the mediator for the distillation. To get the teacher mediator, we first utilize the dual-view modeling framework (Section 3.2) to get the user representation for each similar user, denoted as {u se j , u co j }|U k | j=1 . Then, the teacher mediator is calculated by mean pooling, as the following formula:EQUATIONThe student mediator is the representation of target user k, i.e., [u se k : u co k ]. Based on the teacher and student mediators, the self-distillation loss can be formulated as:EQUATIONNote that the gradients of u se T k and u co T k are stopped, because they only provide the guidance signal instead of optimizing the model.Train. Based on the illustration in Section 3.2 and Section 3.3, we only update the collaborative embedding layer, adapter, cross-attention and sequence encoder during the training, while freezing the semantic embedding layer and semantic user base. Since the original LLMs embeddings E se and U llm are frozen, the original semantic relations get preserved well. The training loss for optimization is the combination of pairwise ranking loss and self-distillation loss, which can be written as follows:EQUATION)where α is a hyper-parameter to adjust the magnitude of self-distillation.Inference. During the inference process of the LLM-ESR, the retrieval augmented self-distillation module is exempted due to no need for the auxiliary loss. Thus, we follow the dual-view modeling process for the final recommendation by Equation (5) . Besides, since the semantic embedding layer can be cached in advance, the call for LLMs is avoided, which prevents the extra inference costs. Due to the limited space, the algorithm lies in Appendix A.2 for more clarity.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have attached the data and code used in this paper in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental settings: 
 Dataset. There are three real-world datasets applied for evaluation, i.e., Yelp, Amazon Fashion and Amazon Beauty. We follow the previous SRS works [18, 49] for preprocessing and data split. More details about the datasets and preprocessing can be seen in Appendix B.1.Baselines. To validate the flexibility, we combine the competing baselines and LLM-ESR with three well-known backbone SRS models: GRU4Rec [14] , Bert4Rec [48] and SASRec [18] . Then, two groups of baselines are compared in the experiments. One group is the traditional enhancement framework for the long-tailed sequential recommendation, including CITIES [17] and MELT [20] . The other group is the LLM-based enhancement framework, which contains RLMRec [44] and LLMInit [13, 16] . The more details about baselines are put into Appendix B.2.Implementation Details. The hardware used in all experiments is an Intel Xeon Gold 6133 platform with Tesla V100 32G GPUs, while the basic software requirements are Python 3.9.5 and PyTorch 1.12.0. The hyper-parameters N and α are searched from {2, 6, 10, 14, 18} and {1, 0.5, 0.1, 0.05, 0.01}. More details about the implementation details are in Appendix B.3. The implementation code is available at https://github.com/Applied-Machine-Learning-Lab/LLM-ESR.Evaluation Metrics. In the experiments, we adopt the metrics of Top-10 list for evaluation. Specifically, the Hit Rate (H@10) and Normalized Discounted Cumulative Gain (N@10) are used. Following [18] , we randomly sample 100 items that the user has not interacted with as the negatives paired with the ground truth for calculation of the metrics. To guarantee the robustness of the experimental results, we report the average results of the triplicate test with random seeds {42, 43, 44}.To validate the effectiveness and flexibility of the proposed LLM-ESR, we show the overall, tail and head performance on three datasets in Table 1 . At a glance, we find that the proposed LLM-ESR can outperform all competing baselines with all SRS models across all user or item groups, which verifies the usefulness of our framework. Then, we probe more conclusions by the following analysis. H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 MELT is devised to enhance tail user, but underperforms our method because of its limitations to collaborative perspective. Though LLMInit can also benefit tail users by introducing semantics, it ignores the utilization of LLMs from the user side.Flexibility. Table 1 shows that the proposed framework can get the largest performance improvements on all three backbone SRS models, which indicates the flexibility of LLM-ESR. By comparison, the other baselines incline to depend on the type of SRS. The traditional method, i.e., CITIES and MELT, tend to perform better for GRU4Rec, while LLMInit is more beneficial to Bert4Rec and SASRec.The results of the ablation study are shown in To investigate the effects of the hyper-parameters in LLM-ESR, we show the performance trend along with their changes in Figure 3 . The hyper-parameter α controls to what extent the designed selfdistillation affects the optimization. With α ranging from 1 to 0.01, the recommending accuracy rises first and drops then. The reason for the compromised performance of large α lies in that overemphasis on self-distillation will affect the convergence of ranking loss. Smaller α also downgrades the performance, which indicates the usefulness of the designed self-distillation. As for the number of retrieved users N , the best is 10. The reason is that more users can provide more informative interactions. However, too large N may decrease the relatedness of the retrieved users.For more meticulous analysis, we split the users and items into 5 groups according to sequence length n u and popularity p v , and show the performance of each group in Figure 4 . From the results, we observe that LLM-based frameworks derive increases in every user and item group, while MELT has a positive effect on some specific groups. It reflects the seesaw problem of MLET and reveals the benefit of making use of semantic embeddings from LLMs. Comparing LLMInit with LLM-ESR, LLM-ESR can get more increments on the long-tail groups (e.g., 1-4 user group and 1-9 item group), which proves the better reservation of semantic information from LLMs by our framework. The group analysis of Bert4Rec and GRU4Rec as backbones are shown in Appendix C.3.5 Related Works 
sequential recommendation: 
 The core of sequential recommendation refers to capturing the sequence pattern for the next likely item [29, 38, 31, 37, 59, 24, 23, 26, 39] . Thus, at the early stage, researchers focus on fabricating the architecture to improve model capacity. GRU4Rec [14] and Caser [49] apply RNNs and CNNs [21] for sequence modeling. Later, inspired by the great success of self-attention [51] in natural language processing, SASRec [18] and Bert4Rec [48] verify its potential in SRS. Also, Zhou et al. [65] proposes a pure MLP architecture, achieving similar accuracy but higher efficiency compared with SASRec. Despite the great progress in SRS, long-tail problems are still underexplored. As for the long-tail item problem, CITIES [17] designs an embedding inference function for those long-tail items specially. In terms of the long-tail user problem, data augmentation is the main way [36, 34] . Only one work, MELT [20] , addresses both two problems simultaneously but still sticks to a collaborative perspective. By comparison, the proposed LLM-ESR handles both the two long-tail problems better from a semantic view by introducing LLMs.Large language models [62, 42] have attracted widespread attention due to their powerful abilities in semantic understanding. Recently, There emerge several works to explore how to utilize LLMs in recommender systems (RS) [63, 28, 22, 40, 56, 57, 64, 32, 30] , which can be categorized into two lines, i.e., LLMs as RS and LLMs enhancing RS. The first line of research aims to complete recommendation tasks by LLMs directly. At the early stage, researchers tend to fabricate the prompt templates to stimulate the recommending ability of LLMs by dialogues. For example, ChatRec [10] proposes a dialogue process to complete recommendation tasks step by step. DRDT [54] integrates a retrieval-based dynamic reflection process for SRS by in-context learning [6] . LLMRerank [9] and UniLLMRec [61] fabricate the chain-of-thought prompts to target the reranking stage and whole recommendation process, respectively. Besides, some other researchers explore fine-tuning open-sourced LLMs for RS. TALLRec [2] is the first one, which fine-tunes a LLaMA-7B by parameter-efficient fine-tuning techniques [15, 33] . Some following works, including E4SRec [25] , LLaRA [27] and RecInterpreter [58] , target combining collaborative signals into LLMs by modifying the tokenization. However, this line of work faces the challenge of high inference costs. Another line, LLMs enhancing RS, is more practical, because they avoid the use of LLMs while recommending. For instance, RLMRec [44] aligns with LLMs by an auxiliary loss. AlphaRec [46] adopts LLMs embedding to enhance the collaborative filtering models. On the other hand, LLM4MSR [55] and Uni-CTR [8] propose to utilize LLMs to augment the multi-domain recommendation models. As for LLMs enhancing sequential recommendation, Harte et al. [13] and Hu et al. [16] adopt LLMs embedding as the initialization for the traditional models. The proposed LLM-ESR belongs to the latter category but further alleviates the problem of defect of semantic information. 
conclusion: 
 In this paper, we propose a large language model enhancement framework for sequential recommendation (LLM-ESR) to handle the long-tail user and long-tail item challenges. Firstly, we acquire and cache the semantic embeddings derived from LLMs, which is for inference efficiency. Then, a dual-view modeling framework is proposed to combine the semantics from LLMs and collaborative signals contained in the traditional model. It can help augment the long-tail items in SRS. Next, we design the retrieval augmented self-distillation to alleviate the long-tail user challenge. Through the comprehensive experiments, we verify the effectiveness and flexibility of our LLM-ESR.Algorithm 1 Train and inference process of LLM-ESR Get the user preference representation in semantic and collaborative views, i.e., u se and u co , respectively.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provide the details of the experimental settings, such as the data split, optimizer, etc., in the experimental setting section (Section 4.1) in the main paper and the implementation detail section (Section B.3) in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report the two-sided t-test with p < 0.05 results in the main experiments, i.e., Table 1 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the potential positive impacts that our algorithm will bring in the Introduction section, i.e., Section 1 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have attached the introduction of how to run the code and the license in the code repository.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Calculate the probability score of ground-truth and negative items by Equation (5).Calculate the ranking loss by Equation ( 6). Retrieve the similar users for each user in U B by Equation (7) .Calculate the self-distillation loss by Equation ( 9).12:Sum the ranking loss and self-distillation loss. Then, update the parameters. Get the user preference representation in semantic and collaborative views, i.e., u se and u co . 17:Calculate the probability score of each candidate item by Equation ( 5) and give out the final recommended list. 18: end forIn this section, we will refer to more details about the experimental settings.The comprehensive experiments in this paper are conducted on three common-used datasets, i.e., Yelp, Fashion and Beauty. Yelp3 is the dataset that records the check-in histories and corresponding reviews of users. We only adopt the check-in data and the attribute information of the point-of-interests. Amazon4 [41] is a large e-commerce dataset, which includes user's reviews on commodities. There are several sub-categories in this dataset and we use two of them, i.e., Fashion and Beauty.For preprocessing, we refer to the procedures in SASRec [18] . Since the sequential recommendation is often utilized for implicit interactions, we consider all review or rate records as interactions. Then, the users with fewer than three interacted items are dropped, because we do not explore the problem of cold-start users in this paper. As for the data split, the last item v nu and the penultimate item v nu-1 of each interaction sequence are taken out as the test and validation, respectively. The statistics of the three preprocessed datasets are shown in Table 3 . Backbone Models. To show the flexibility of our enhancement method, we test three popular sequential recommendation models in the experiments. The main distinction between these models refers to the sequence encoder f θ and ranking loss L Rank .• GRU4Rec [14] . It adopts the GRU as the sequence encoder, and sequence-to-one pairwise loss as the final ranking loss.• Bert4Rec [48] . Inspired by the training pattern of Bert [19] , this backbone proposes a combination between pairwise ranking loss and cloze task, which mask a proportion of items in one sequence. The sequence encoder of Bert4Rec is the stack of bi-directional self-attention layers.• SASRec [18] . Compared with Bert4rec, SASRec adopts the causal self-attention layer as the basic unit of its sequence encoder. Besides, the sequence-to-sequence pairwise ranking loss is applied for optimization during the training.There are two groups of up-to-date baselines that are compared within this paper, i.e., traditional baselines and LLM-based baselines.Traditional Baselines. This category split the users and items into long-tail and head groups at first. Then, they enhance the long-tail users or items by fabricated training procedures. Note that they only utilize the collaborative signals essentially and do not introduce any semantics.• CITIES [17] . This work devises an embedding-inference function to refine the embeddings of long-tail items specially. Such embedding-inference function is trained by head items and used for long-tail items during inference. We follow the hyper-parameters in the original paper and code 5 .• MELT [20] . MELT proposes a bilateral-branch framework to enhance the long-tail users and items. One branch is trained to generate the head user representations and enhance the tail users while inference. The other branch is to recover the embeddings of head items during training and update embeddings of tail items during inference. We refer to the implementation and the hyper-parameter settings in official code 6 .LLM-based Baselines. The methods in this line aim to combine the semantic information derived from LLMs to enhance the recommendation models.• RLMRec [44] . This baseline is one of the pioneers in utilizing the semantic embeddings derived from LLMs. However, it is designed for collaborative filtering but not sequential recommendation. For a fair comparison, we eliminate the process of profile generation during the implementation. We refer to the source code 7 of RLMRec to adapt it to sequential recommendation models.• LLMInit [13, 16] . More recent works, i.e., LLM2Bert4Rec [13] and SAID [16] , both utilize the LLMs embedding to initialize the item embedding layer in SRS models and then fine-tune it by interaction data. In this paper, we dub this way as LLMInit.We conduct all experiments on an Intel Xeon Gold 6133 platform with Tesla V100 32G GPUs. Besides, the implementation is based on Python 3.9.5 and PyTorch 1.12.0. In terms of the hyperparameter search, the criterion is N@10 on the validation set. To avoid overfitting, we adopt the early stop strategy with 20-epoch patience. For the backbone SRS models, the number of GRU layers is set to 1 for GRU4Rec, while the number of self-attention layers is fixed at 2 for SASRec and Bert4Rec. Also, the dropout rate is 0.6 for Bert4Rec. In terms of the training, the batch size and learning rate are set as 128 and 0.001 for all datasets. The embedding size is 128 for all baselines, while 64 for LLM-ESR. The reason is that there are two branches in LLM-ESR, and the half size of the other unique-branch baseline is a fair setting. Then, we choose the Adam as the optimizer. The hyperparameters N and α for LLM-ESR are searched from {2, 6, 10, 14, 18} and {1, 0.5, 0.1, 0.05, 0.01}. We find that the best choice is 10 for N and 0.1 for α for all three datasets used in this paper. H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 Furthermore, the embeddings of LLMs are derived from the API 8 named "text-ada-embedding-002" provided by OpenAI.In this section, we will show more experimental results to further analyze the flexibility and effectiveness of our LLM-ESR.For further analysis, we conduct the ablation study on the proposed LLM-ESR with Bert4Rec and GRU4Rec as the backbone SRS models. The results are shown in Table 4 and Table 5 . At first, we probe the effects of dual-view modeling by removing one of the views, denoted as w/o Co-view and w/o Se-view. From the overall performance, these two variants both underperform, which indicates the essence of the dual-view. Besides, w/o Co-view downgrades the accuracy of the head item group more, while w/o So-view harms the long-tail item group compared with LLM-ESR. This phenomenon highlights the advantages of the collaborative view and semantic view, respectively. As for distinct SRS backbone models, we find that Bert4Rec benefits more from collaborative information, because removing the collaborative view causes a more severe performance drop. By comparison, GRU4Rec can get more enhancement from the semantic view. Then, w/o SD means eliminating self-distillation. It downgrades the performance of the tail user group consistently, which indicates the proposed retrieval augmented self-distillation can actually help alleviate the long-tail user challenge. w/o Share represents using separate sequence encoders for the dual views. This variant is a little worse than applying a shared encoder, illustrating the common pattern for both views. Another advantage of the shared encoder is higher parameter efficiency. Besides, LLM-ESR without cross-attention (w/o CA) is inferior to LLM-ESR totally, which indicates the effectiveness of the sequence-level fusion.At the same time, it is risky to overfit with semantic embeddings when the textual data is scarce. To validate the robustness of our LLM-ESR, we conduct additional experiments in scenarios with limited textual data. To simulate this situation, we removed all attributes from the item descriptions except for "name" and "categories" when constructing the textual prompts for the Yelp dataset (originally using 8 attributes). This reduced the average word count of the textual prompts from 38.38 to 20.33. We used SASRec as the backbone model in these supplementary experiments, with results presented H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 H@10 N@10 Full 0.6673 0.4208 0.1893 0.0845 0.8080 0.5199 0.6685 0.4229 0.6627 0.4128 Full w/o F 0.6069 0.3664 0.1284 0.0541 0.7477 0.4584 0.6028 0.3647 0.6226 0.3730 Crop 0.6477 0.4046 0.1478 0.0675 0.7807 0.4968 0.6468 0.4058 0.6511 0.3998 Crop w/o F 0.6025 0.3630 0.1247 0.0563 0.7432 0.4563 0.6004 0.3615 0.6109 0.3786 in Table 6 . In the table, Full and Crop represent the use of the complete and cropped prompts, respectively. w/o F denotes training LLM-ESR without freezing the semantic embedding layer. The results show a decrease in performance for both Full and Crop due to the limited textual prompt. Moreover, Full w/o F and Crop w/o F yield similar results, indicating that semantic embeddings suffer from overfitting with both complete and cropped prompts. In contrast, freezing the semantic embedding layer improves performance in both scenarios and significantly benefits long-tail items, demonstrating that our design effectively alleviates the overfitting issue.To further investigate how LLMs enhance the traditional SRS models, we visualize the item embeddings of SASRec, CITIES, MELT, our LLM-ESR (concatenate the semantic embedding e se and collaborative embedding e co ), and LLM using t-SNE, as shown in Figure 5 . We group the items into four categories based on their popularity. The t-SNE figures reveal that the embeddings of SASRec, CITIES, and MELT tend to cluster according to item popularity. In contrast, the distribution of LLM embeddings is more uniform, indicating that the semantic relationships are not skewed by popularity. Furthermore, the embeddings of our LLM-ESR also show a more even distribution, validating that our method effectively corrects the embedding distribution in SRS and thus can enhance the performance of long-tail items. For a more meticulous analysis of to what extent the proposed LLM-ESR alleviates the long-tail challenges, we categorize users and items into 5 groups. The performances of each method with Bert4Rec and GRU4Rec as backbone models are shown in Table 6 and Tabel 7, respectively. Firstly, we analyze the results in different user groups. Undoubtedly, all methods perform worse for those users with fewer interactions, which highlights the long-tail user challenge. MELT can enhance the Bert4Rec well so that the performances in all groups get increased, but is incompatible with GRU4Rec and thus harms several groups. By comparison, LLMInit and our LLM-ESR can benefit all user groups consistently. Due to the better utilization of semantics from LLMs, LLM-ESR can outperform LLMInit evidently. Besides, the superiority is larger for more long-tailed users, i.e., 1-4 and 5-9 Two potential limitations should be considered for this paper. Firstly, there are two hyper-parameters for the proposed LLM-ESR, i.e., the weight of self-distillation loss α and the number of retrieved similar users N , which is time-consuming to search for the best model. Secondly, only the LLMs embedding provided by OpenAI API is validated in the experiments, but other more recent models [3, 52] may lead to better performance. Nonetheless, the experiments on various datasets and backbone models consistently validate the effectiveness of our LLM-ESR NeurIPS Paper ChecklistJustification: There is no risk of misuse of the proposed method and the datasets used in the paper are open-sourced.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original paper or attached the link to the existing assets used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.https://platform.openai.com/docs/guides/embeddingshttps://www.yelp.com/datasethttps://cseweb.ucsd.edu/~jmcauley/datasets.html#amazon_reviewshttps://github.com/swonj90/CITIEShttps://github.com/rlqja1107/MELThttps://github.com/HKUDS/RLMRechttps://api.openai.com/v1/embeddings 


Decision: Unanimous and solid reviews. Reviewers praise the experiments, approach, and relevance in real-world settings. Issues raised are mainly clarifications and seem fairly minor.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenges in sequential recommender systems (SRS), particularly the long-tail user and long-tail item issues, which complicate user experience and seller benefits in real-world applications. The authors propose the Large Language Models Enhancement framework for Sequential Recommendation (LLM-ESR) to mitigate these challenges. The LLM-ESR framework leverages semantic embeddings derived from large language models (LLMs) to enhance SRS without increasing inference load. To tackle the long-tail item problem, the framework employs a dual-view modeling approach that integrates semantics from LLMs with collaborative signals from traditional SRS. For the long-tail user issue, a retrieval augmented self-distillation method is introduced to improve user preference representation by utilizing more informative interactions from similar users.
strength_weakness: strength_weakness: - The paper does not sufficiently and deeply discuss existing work, making the motivation and core idea of the paper seem less convincing, and the innovation of the paper is also insufficient.
    
- The baselines used in the experiments are limited.
questions: questions: - In line 44, the authors mention that existing studies perform poorly due to "ignorance of the true relationship between items." What is the true relationship between items, and how does it affect recommendations?
    
- Although SASRec is a classic model, it is not reasonable to conclude that all SRSs perform poorly in long-tail scenarios solely based on SASRec. Have the authors analyzed why SASRec performs poorly in long-tail scenarios? Do models that use other techniques specifically for long-tail scenarios have this problem? What are their limitations?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a novel framework designed to address the long-tail challenges in sequential recommendation systems (SRS). By leveraging semantic embeddings from large language models (LLMs) and combining them with collaborative signals, the authors propose a dual-view modeling framework and a retrieval-augmented self-distillation method. This approach aims to enhance recommendations for both long-tail users and items without adding significant inference load. Extensive experiments on three real-world datasets demonstrate the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1.	The proposed dual-view and self-distillation methods add layers of complexity to the SRS, which may pose challenges in practical implementation.
2.	The framework assumes a certain level of similarity in user interactions, which might not hold true for highly diverse user bases.
3.	Impact on Popular Items: While the focus is on long-tail items and users, the potential impact on recommendations for popular items is not thoroughly explored.
questions: questions: 1.	Could the authors provide more details on the practical implementation challenges and how they can be mitigated?
2.	How does the framework handle highly diverse user interactions where finding similar users may be challenging?
3.	Balanced Performance: What measures have been taken to ensure that the enhancement for long-tail users and items does not adversely affect recommendations for popular items?
limitations: limitations: Discussing potential negative societal impacts, such as reinforcing biases in recommendations, would be beneficial.
paper_summary: paper_summary: The paper presents a framework that integrates large language models (LLMs) into sequential recommendation systems (SRS) to tackle the long-tail challenges. The framework includes dual-view modeling, which combines semantic embeddings from LLMs with collaborative signals, and a retrieval-augmented self-distillation method to enhance user preference representation. The authors validate their approach through extensive experiments on three real-world datasets, demonstrating significant improvements over existing methods.
strength_weakness: strength_weakness: 1) There is a risk that the semantic embeddings might overfit to the training data, especially if the textual descriptions are not diverse enough.
2) The performance of the framework might be sensitive to the choice of hyper-parameters, which is not extensively explored in the paper.
questions: questions: 1) How do the authors mitigate the risk of overfitting with semantic embeddings, especially in scenarios with limited textual data?
2) Can the authors elaborate on the hyper-parameter tuning process and its impact on the performance?
limitations: limitations: The authors have addressed several limitations, but there is room for more in-depth discussion on potential biases introduced by semantic embeddings and the sensitive to the choice of hyper-parameters.

Comments: comment: We really thank you for your valuable time and suggestions! We promise to address the issues you have referred to in the revised version.
You have resolved some of my concerns, and I am inclined to raise my score. However,  there are some issues that need further improvement in the final version.
Dear reviewer, we sincerely appreciate your valuable time and insightful suggestions on our paper again. We hope that we address your concerns by our responses. Since the reviewer-author discussion deadline is approaching, please let us know if you have any other questions. We are glad to further respond to your concerns.
We really thank you for taking the time to carefully assess our work and provide thoughtful feedback. Your suggestions greatly help improve our paper.
The authors have answered all my concerns. To this end, I prefer to recommend this paper acceptance
Thanks for your reply. Again, we really appreciate your valuable time and insightful suggestions.
Thanks for the analysis and clarification, and I will raise the rating.