On Softmax Direct Preference Optimization for Recommendation

Authors: yuxin  chen
junfei  tan
an  zhang
zhengyi  yang
leheng  sheng
enzhi  zhang
xiang  wang
tat-seng  chua
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:32:41

Abstract: Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (\textbf{S-DPO}) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO.

Main paper: 
 introduction: 
 Recommender systems aim to predict personalized rankings based on user preference data, i.e., historical interactions such as purchases, clicks, and ratings [1, 2] . Recently, leveraging the extensive world knowledge and powerful reasoning abilities of language models (LMs) [3] [4] [5] [6] , LM-based recommenders have been broadly explored [7] [8] [9] . These recommenders convert historical interaction data into language prompts and either perform in-context learning or fine-tune LMs, demonstrating notable advantages, including zero-shot and few-shot reasoning [10] [11] [12] [13] , enhanced generalization abilities [14, 15] , and rich semantic understanding [16] [17] [18] [19] . However, current LM-based recommenders typically utilize language modeling loss for personalized ranking objectives-predicting the next token-which significantly differs from the objective of modeling user preferences in recommendation tasks [20, 21] .We argue that the current objective of LM-based recommenders does not fully utilize preference data and is not optimized for personalized ranking tasks, thereby hindering recommendation performance. Most LM-based recommenders address recommendation tasks by leveraging specialized language prompts [14, 17, [22] [23] [24] , incorporating collaborative signals as a new modality [18, 19, 25] , or extending the vocabulary of LMs with item tokens [16, [26] [27] [28] [29] [30] . Typically, these recommenders pair each language prompt, including the user's historical interaction item lists, with a single positive item and then update LM parameters using language modeling loss [14, 19] . Despite being designed for recommendation tasks, these LM-based recommenders do not consider negative items and are not directly optimized for personalized rankings. Such a training paradigm fails to fully leverage user preference data and overlooks the role of negative items in recommendations, thereby impeding the alignment of LMs with user preferences.Inspired by the success of using human-labeled data to align LMs with human preferences [31] [32] [33] and advancements in direct preference optimization (DPO) [34] [35] [36] , we make progress on aligning LMs with recommendations by fine-tuning them to predict the next item in accordance with the user's preference. This preference alignment stage aims to instill ranking information into the LMs and help recommenders distinguish preferred items from negatives, rather than solely focus on positives.Towards this end, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for recommendation, connected to softmax sampling strategies [20, [37] [38] [39] , which we call S-DPO. Specifically, we first devise supervised fine-tuning to inject domain knowledge and improve LM's ability to follow the instructions before preference alignment phase, following [14, 33] . In the preference alignment stage, instead of constructing solely positive pairs, we initially pair each language prompt with both positive and randomly sampled multiple negatives to build text-based preference data. Building upon these preference data, we extend conventional DPO with the Bradley-Terry preference model [34, 40] on pairwise data to the Plackett-Luce preference model [41, 42] , which handles relative rankings among multiple samples. Furthermore, we generalize the traditional Plackett-Luce preference model, which is designed for full relative rankings, to accommodate partial rankings, a more natural fit for recommendation tasks.Benefiting from the multiple negatives in preference data, S-DPO offers three appealing properties. On the one hand, S-DPO serves as the first specialized personalized ranking loss for LM-based recommenders, effectively utilizing multiple negatives and acknowledging the importance of preference data. Empirically, we demonstrate that it provides more effective ranking gradients and better rewards for preferred items compared with DPO (cf. Section 4.2). On the other hand, we theoretically bridge the DPO loss with the pairwise BPR loss [43] over pairwise data and connect S-DPO with the softmax loss over negative sampling (also known as contrastive loss in self-supervised recommendations, which achieves state-of-the-art performance [37, 44, 45] ). This connection naturally underscores the ranking performance of S-DPO and highlights the critical role of multiple negatives. Furthermore, gradient analysis demonstrates that S-DPO has an inherent benefit of mining hard negative examples similar to contrastive learning paradigm [38] , which not only boosts the performance but also accelerates the training process (cf. Section 3.1), assuring its exceptional capabilities in recommendation tasks.Overall, our contributions can be concluded as follows:• We are among the first to point out that the widely used language modeling loss in LM-based recommendation is not designed for ranking tasks and fails to fully utilize user preference data, thereby hindering recommendation performance. • We propose S-DPO, an alternative version of DPO loss extended from the traditional Plackett-Luce preference model, incorporating multiple negatives to instill ranking information into LM and tailoring for LM-based recommenders. • We theoretically bridge S-DPO with the softmax loss over negative sampling to highlight the critical role of multiple negatives and find its inherent benefit of mining hard negatives, assuring its capabilities.Figure 1 : Framework of S-DPO. Different from existing methods which fine-tune LMs with a language modeling loss without tailoring for recommendations, S-DPO proposes to explicitly instill ranking information into LMs. To take one step further, S-DPO incorporates multiple negatives in user preference data and generalizes pairwise DPO loss to softmax ranking loss. 
preliminary: 
 In this section, we first formalize sequential recommendation as the task of aligning language models (LMs) with user preferences. Then, we discuss the general framework of current LM-based recommenders that utilizes language modeling loss to fine-tune LMs. Finally, we outline the training process widely used to align LMs with human preferences, including reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO).Task Formulation. Given the historical interactions H u of one user u in chronological order, the goal of LM-based sequential recommender M θ , where θ represents trainable parameters, is to select the item i p preferred by user u from candidate set C = {i j } N j=1 , where N is the number of candidates. This task requires that item i p be preferred over the other candidate items, denoted by I d = C\{i p }. This requirement explicitly defines a multi-negative preference understanding for LM-based recommenders, which can be formulated as follows:EQUATIONwherein ≻ u stands for the preference of user u.Fine-tuning LM-based recommenders. Current LM-based recommenders widely adopt supervised fine-tuning (SFT) [33] on recommendation-specific data to enhance their performance [7, 9] . Generally, this involves two steps: structuring recommendation data as text-based pairs and then fine-tuning LMs based on these pairs. In the first step, for user u, a recommendation task prompt x u encompasses the user's historical interactions H u , the candidate item set C, and a description of the sequential recommendation task. This prompt x u is paired with the title of the preferred item i p in the candidate set C, denoted as e p , to form the pair data (x u , e p ). In the second step, the (x u , e p ) pairs are utilized to fine-tune the LM-based recommender M θ through language modeling loss. This loss, commonly used in SFT in language modeling tasks, implicitly treats the recommendation task as predicting the next token based on preceding tokens. Formally, the objective of optimizing the LM-based recommender M θ with pair data (x u , e p ) can be formulated as:EQUATIONwhere |e p | is the number of tokens in e p , (e p ) t is the t-th token of e p and (e p ) <t is the tokens preceding (e p ) t .However, recommendation tasks are essentially user preference alignment tasks, as formalized in the above task formulation, and differ from language modeling tasks that consider only positive responses. Such a gap necessitates further exploration into aligning LM-based recommenders with user preference, an area that has been underexplored.Recent studies in natural language processing (NLP) explore the use of human-labeled pairwise data as a reward signal to align LMs with human preferences, such as RLHF [33] and DPO [34] . Specifically, the RLHF [33] pipeline adds two additional phases after the SFT phase: reward model training and reinforcement learning (RL) optimization. After obtaining the SFT model π SFT , RLHF further optimizes it with pairwise preference data.Inspired by the success of RLHF in NLP, we leverage RLHF to inject recommendation-specific user pairwise preference into LM-based recommenders. Let E = {e j } N j=1 denote the title set of candidate items, where e j denotes the title of item i j . Given two items i j , i k ∈ C, the user preference i j > u i k can be seamlessly transformed into a response preference, stipulating that e j is preferred over e k when presented with prompt x u , denoted as e j ≻ e k |x u . By sampling one dispreferred item i d from dispreferred candidate set I d , we can curate a preference dataset {(e p , e d , x u )}.After that, RLHF utilizes a preference model for preference distribution modeling, such as Bradley-Terry (BT) model [40] . This preference model assumes there is a latent function r(x u , e j ) representing the reward of prompt-response pair (x u , e j ). The bigger reward r(x u , e j ) means the more user u prefers item i. From this perspective, reward function r(x u , e j ) serves as a scoring function that quantifies the preference of user u to item i. Besides, the preference model defines a mapping from the reward function r(x u , e j ) to a preference distribution p r (e j ≻ e k |x u ). Based on preference distribution, an optimal reward function is trained by maximizing the likelihood of preference data.The training objective of this phase is as follows:EQUATIONLet π θ (e|x u ) be the probability that LM-based recommender M θ output title e given prompt x u . The final reinforcement learning phase aims to maximize the expected reward of policy while not deviate too far from the reference model, formulating the following objective for optimal policy:EQUATIONwhere D denotes the distribution of x u and π ref = π SFT .A recent study, DPO [34] , theoretically proves the optimal policy in a closed form to Eq.( 4) isEQUATIONwhich is equivalent toEQUATIONwhere Z(x u ) = e π ref (e|x u )exp 1 β r(x u , e) is the partition function.By defining p r (e p ≻ e d |x u ) as σ(r(x u , e p ) -r(x u , e d )) in Eq.( 3) according to the BT model used in RLHF and substituting term r(x u , e) in Eq.(3) with Eq.( 6), the last two phases of RLHF pipeline can be equivalently transformed into optimizing DPO loss below:EQUATIONwherein σ(x) is the sigmoid function.DPO is able to directly extract the optimal policy from pairwise preference data, making it more practical for preference alignment than RLHF. Nevertheless, DPO and RLHF are usually designed for pairwise preference. The oversight of other negative items impedes the performance of the LM-based recommenders. To bridge the gap, we expand DPO to S-DPO in recommendation tasks, in consideration of multiple negative items. 
derivation of s-dpo loss: 
 To align LM-based recommender M θ with multi-negative preference, we first derive the preference distribution and then propose a new loss function called S-DPO (depicted in Figure 1 ). For the dataset pairing one preferred item with multiple dispreferred items, we leverage the Plackett-Luce (PL) model [41, 42] to build preference distribution. Given prompt x u , K titles e 1 , e 2 , • • • , e K and a permutation τ : [K] → [K] reflecting the user preference, with τ (j) denoting the j-th element of permutation τ , the PL model estimates that the ranking e τ (1) , e τ (2) , • • • , e τ (K) turns out true, as:EQUATION)By enumerating all the permutations starting with p and calculating sum of their probability given by the PL model, the final multi-negative preference distribution p * can be derived as:p * (e p ≻ e d , ∀e d ∈ E d |x u ) = exp(r(x u , e p )) K j=1 exp(r(x u , e j )).For brevity, the complete derivation is delegated to Appendix A.1.Deriving S-DPO. By substituting reward function r(x u , e) in Eq.( 9) with Eq.( 6), the multi-negative preference distribution can be rewritten as:EQUATIONThrough plugging distribution given by Eq.( 10) in the reward learning objective in Eq.( 3), our S-DPO loss can be formulated for policy π θ as:EQUATIONNotably, when the number of candidates N is 2, which means there is only one dispreferred item, S-DPO reduces to DPO. The proof is provided in Appendix A.2.Gradient Analysis. We conduct gradient analysis on S-DPO. The gradient of L S-DPO with respect to parameters θ takes the following formulation:∇ θ L S-DPO (π θ ; π ref ) = -βE (xu,ep,E d ) σ log e d ∈E dexp(g(e d , e p , x u ))higher weight when reward deviates from preference• ∇ θ log π θ (e p |x u ) - e d ∈E d ∇ θ log π θ (e d |x u ) e ′ d ∈E d exp(g(e ′ d , e d , x u ))higher weight when reward is larger , wherein g(e j , e k , x u ) = r θ (x u , e j ) -r θ (x u , e k ) and similar to DPO, r θ (x u , e) = βlog π θ (e|xu) π ref (e|xu) is the implicit reward function defined by π θ . See Appendix A.3 for a complete derivation.Recap the DPO gradient below:∇ θ L DPO (π θ ; π ref ) = -βE (xu,ep,e d )σ(g(e d , e p , x u ))higher weight when reward is wrong• [∇ θ logπ θ (e p |x u ) -∇ θ logπ θ (e d |x u )] .Similar to DPO, the gradient of S-DPO loss increases the likelihood of the preferred item and decreases the likelihood of all the dispreferred items. Each example is also weighed by how much the implicit reward r(x u , e) deviates from the preference data. However, compared with DPO, S-DPO harnesses information of multiple dispreferred items in this weight.Moreover, S-DPO treats gradients of different negative (dispreferred) items differently by assigning the gradient of each negative item with an extra weight1 ϵ ′ d ∈E d exp(g(e ′ d ,e d ,xu)) = exp(r θ (xu,e d )) ϵ ′ d ∈E d exp(r θ (xu,e ′ d )). This term reflects the relative reward of each negative item compared with other negative items. Similar to [38] , we can categorize negative items into two categories: (1) Hard negative items, whose reward r θ (x u , e d ) = β π θ (e d |xu)π ref (e d |xu) is relatively high, making it more probable to be chosen by LM-based recommenders; (2) Easy negative items, whose reward r θ (x u , e d ) is relatively low, making it less likely to be output. For hard negative items, the extra weight termexp(r θ (xu,e d )) ϵ ′ d ∈E d exp(r θ (xu,e ′ d ))tends to be larger, leading to more decline for likelihood. This mechanism makes LM-based recommenders more discriminative and endows S-DPO with more effectiveness and stability than DPO.In this section, we will discuss the structural correlation between DPO and BPR [43] , together with S-DPO and softmax loss [39] , which demonstrates the advantage of S-DPO over DPO and language modeling loss.For user u, preferred item i p and one dispreferred i d ∈ I d , BPR loss takes the form:EQUATIONwherein f (u, i) represents preference score of user u for item i.Similarly, given dispreferred item set I d , the softmax loss takes the form:EQUATIONReview the DPO loss in Eq.( 7) and S-DPO loss in Eq. (11) . Notably, term β log π θ (e|xu) π ref (e|xu) is the implicit reward function, denoted by r θ (x u , e) in Section 3.1. According to Section 2, r θ (e, x u ) reflects the preference of user u to item i corresponding to title e. When the reference model has no knowledge about recommendation, i.e., when π ref (e|x u ) is approximately a uniform distribution, term r θ (x u , e) = β log π θ (e|xu) π ref (e|xu) exactly reveals absolute preference. Hence, r θ (x u , e) possesses a similar function to f (u, i) .From this perspective, DPO and S-DPO can be seen as special patterns of BPR and softmax loss, respectively. Given the effectiveness of BPR and InfoNCE loss in recommendation, we argue that sampled-based loss which explicitly compares preferred and dispreferred items such as DPO and S-DPO is more suitable for training LM-based recommenders than only utilizing language modeling loss. Moreover, as softmax loss works better than BPR loss in multi-negative scenarios [20] , it can be inferred that S-DPO will be more tailored for multi-negative user preference alignment than DPO. 
experiments: 
 In this section, we aim to answer the following research questions:• RQ1: How does S-DPO compare with traditional and LM-based sequential recommendation models on performance? • RQ2: How does the LM-based recommender benefit from the multiple negatives in S-DPO? • RQ3: What are the impacts of the essential parameters (β) on S-DPO?Baselines. We thoroughly compare S-DPO with three categories of recommenders in sequential recommendations: traditional recommenders (GRU4Rec [46] , Caser [47] , SASRec [48] ), LMenhanced recommenders (MoRec [49] ) and LM-based recommenders (LLaMA2 [32] , Chat-REC [15] , TALLRec [14] , LLaRA [19] ). See detailed introduction and comparison of baselines in Appendix B.Datasets. We conduct extensive experiments on three real-world benchmark datasets which differ in size and domain (Movielens [50] , Goodreads3 , and LastFM [51] ). Following standard settings of [15, 19] , we employ a commonly used metric Hit Ratio@1 (HR@1) for performance evaluation and an additional metric Valid Ratio to evaluate the LM-based methods' ability to generate appropriate responses. See detailed introductions of datasets and evaluation metrics in Appendix B.Implementation. We implement all LM-based recommenders on 4 NVIDIA A100 GPUs. For all LM-based recommenders, we conduct a supervised fine-tuning stage for a maximum of 5 epochs. For S-DPO and its variants, we conduct a preference alignment stage for a further 3 epochs. Different from existing methods, we only optimize loss on item titles and find it effective in recommendation tasks. Refer to Appendix B for more implementation details.Table 1 presents a comparative analysis of the performance of our proposed S-DPO and baselines. Bold and underlined indicate the best and the second-best performance, respectively. We observe that:• LM-based recommenders have driven impressive performance breakthroughs compared with traditional recommenders. Our results reveal that traditional recommenders outperform untuned LM-based recommenders (LLaMA, ChatRec) but fall short compared to LM-based recommenders fine-tuned on historical interactions (TALLRec and LLaRA). It is noted that untuned LM-based recommenders are limited by inadequate instruction-following capabilities (indicated by a low valid ratio) or a lack of domain-specific knowledge (indicated by a suboptimal performance), which highlights the necessity of the supervised fine-tuning stage to further ground the inherent ability of language models down to sequential recommendation tasks. Moreover, MoRec also exhibits suboptimal performance compared to its traditional variant because it leaves the reasoning ability of LM untouched. The superior performance of recent LM-based recommenders indicates the significant roles of knowledge and reasoning ability in language models for recommendation tasks in semantically informative datasets, which highlights the potential of LM-based recommenders.• Tailoring language models for recommendation task further boosts the performance of LMbased recommenders. For LM-based recommenders, the substantial performance gap between finetuned and untuned approaches emphasizes the importance of tailoring models for recommendations. TALLRec adapts LM for recommendation by supervised fine-tuning LM on historical interactions, surpassing traditional recommenders. Additionally, LLaRA consistently outperformed TALLRec across all datasets, suggesting that introducing collaborative signals through appropriate item representations is a viable direction for further adapting LM. However, existing LM-based methods adapt LM from either item representation methods or corpus construction, leaving the adaptation of optimization objectives unexplored. Instead, S-DPO aligns the language model with multi-negative user preference data by extending DPO to include a softmax ranking loss, making it a more appropriate loss function for recommendation tasks.• S-DPO consistently outperforms all traditional recommenders and the state-of-the-art LMbased recommenders on all datasets. S-DPO shows an improvement ranging from 11.10% to 47.03% on Hit Ratio@1 compared to the second-best baseline. Building on a supervised finetuning stage, we observe a further improvement to the preference alignment stage, which explicitly instills ranking information into LM and utilizes preference data with multiple negative samples. Such superior performance suggests that explicitly tailoring LM for recommendation using user preference data at the training objective level is more effective than other LM-based recommenders. By leveraging the inherent abilities of the LM and incorporating ranking information from user preference data, S-DPO effectively differentiates between preferred and less preferred items. Notably, the preference alignment stage hardly harms the inherent ability of LM, illustrated by a high valid ratio.Ablation Study. To investigate the effect of explicit ranking optimization and multiple negative samples of S-DPO, we compare it with the vanilla supervised fine-tuned model (w/o ranking), and a variant of S-DPO with only a single negative sample (w/o multi-neg), downgrading to pairwise DPO loss. The experimental results are reported in Figure 2a . We can observe that DPO can achieve an overall better performance compared to SFT, which underscores the effectiveness of instilling ranking relationships into existing LM-based recommenders. With a more effective ranking gradient provided by multiple negative samples, S-DPO can further boost performance and achieve the best among all baseline methods and variants.Study on the number of negative samples (RQ2). Benefiting from the utilization of multiple negative pairs in preference data, our S-DPO offers two empirically appealing properties compared to DPO: 1) S-DPO has more effective gradients facilitating the optimization; 2) S-DPO provides a better boost for rewards of preferred items compared to DPO. Figure 2b provides the comparison of validation loss between S-DPO and DPO, illustrating that the loss of S-DPO decreases faster and more significantly. This observation demonstrates that multiple negative pairs provide larger and more meaningful gradients for model optimization, which is attributed to the inherent benefit of S-DPO to mine negative samples [38] (cf. Section 3.1).On the other hand, we study the behavior of S-DPO which is illustrated in Figure 2c . We surprisingly find that S-DPO exhibits continually increasing rewards of preferred items that are more significant and stable than DPO, which shows better effectiveness in distinguishing preferred items and a potential of mitigating data likelihood decline issues [52, 53] .To further verify the superiority of the multiple negative samples of S-DPO compared with DPO, we select the number of negative samples from {1, 3, 5, 8, 10, 15} to conduct experiments to explore the potential of the number of negative samples, with the results depicted in Figure 3a . It can be observed that utilizing multiple negative samples allows the model to achieve better performance than with a single one. Furthermore, as the number of negative samples increases, the model's performance exhibits continual improvements. We attribute this success of S-DPO to more effective ranking gradients provided by multiple negatives which can be connected to the superior performance of contrastive loss in self-supervised recommendations [38, 39, 54] .To validate the superiority of S-DPO over the DPO variant with multi-negatives, we conduct effectiveness and efficiency comparisons. Table 2 demonstrates that introducing more negative samples benefits both DPO and S-DPO, and S-DPO achieves comparable performance with fewer training steps. We further analyze how S-DPO outperforms DPO in terms of computational efficiency. While the complexity of DPO for K negative samples is Θ(2KC M S t ), where C M + 1 denotes the base LLM's computational complexity and S t represents the size of the training data, S-DPO's complexity for the same number of negatives is reduced to Θ((K + 1)(C M + 1)S t ). This efficiency gain can be expressed by scaling the complexity of DPO by the factor 1 2 + 1 2K + 1 2C M + 1 2KC M , highlighting that S-DPO offers significant advantages, especially when working with a larger number of negative samples.In S-DPO, β is a hyperparameter controlling the deviation of LM from the base reference policy [34] . Typically, a smaller value of β implies that the language model is more heavily influenced by the preference signals and vice versa. In this section, we select the value of β from {0.1, 0.5, 1, 3, 5} to explore the effect of β on S-DPO. As indicated in Figure 3b through 3c, a higher β can achieve overall better performance in our task, while a lower β may overwhelm the model's learned knowledge from the supervised fine-tuning stage, as evidenced by both low valid ratio and hit ratio. On the other hand, an excessively large β prevents the model from effectively learning ranking relationships, leading to suboptimal performance. In all our main experiments and studies, we set β as 1 to achieve a balance between ranking signals and inherent knowledge of language models. 5 Related Work 
lm for recommendation: 
 Recent advancements in recommendation systems have increasingly incorporated Language Models (LMs) due to their extensive knowledge and robust reasoning abilities. This integration occurs primarily in two forms: LM-enhanced recommenders and LM-based recommenders. LM-enhanced recommenders utilize LM embedding as semantic representations to provide contrastive signals [55] [56] [57] [58] or utilize LM as advanced feature extractors improving the representation of user and item features [59] [60] [61] . However, these systems still rely on traditional recommenders for the final recommendation task, which leaves the reasoning ability of LM largely untouched.On the other hand, LM-based recommenders directly employ LMs for making recommendations. Early works leverage LMs' in-context learning capabilities for zero-shot or few-shot recommendations, demonstrating significant potential [10] [11] [12] 15] . However, untuned LM-based recommenders are limited by inadequate instruction-following capabilities and a lack of domain-specific knowledge. To bridge this gap, recent efforts in this category include supervised fine-tuning of LMs on the historical interactions to enhance their performance in recommendation tasks [14, 17, 23, 24, 27] . More recently, researchers have discovered that exploring item representation methods in the finetuning phase may further boost LM's ability for recommendation [30] . This branch of works includes integrating collaborative signals [18, 19, 25, [62] [63] [64] , adjusting numeric representations [22, 65, 66] or introducing additional item tokens [16, 26, 28, 29] .However, existing finetuned methods follow the training objective of language generation without any specific adjustments for personalized ranking. Different from them, S-DPO proposes to explicitly optimize item ranking information on preference data.Reinforcement Learning from Human Feedback (RLHF) [31] [32] [33] is a prevalent method of LMs to learn from human preferences. The RLHF pipeline comprises reward model training and reinforcement learning (RL) optimization, the latter of which suffers instability and inefficiency. Direct Preference Optimization (DPO) [34] bypasses the brittle RL phase via a particular reward model parameterization and is thus simpler to implement while still keeping the performance of RLHF. DPO proves to be effective in many scopes, like NLP [34, 67] and multimodal LMs [35, [68] [69] [70] . Besides, several variants have been proposed for further improvement of DPO. ΨPO [71] is a generalization of DPO loss and its representative IPO can better overcome the problem of overfitting. ODPO [36] treats preference pairs differently by stipulating that the likelihood gap of two responses should be greater than a corresponding offset value. KTO [72] utilizes prospect theory for preference alignment tasks. Other variants including GPO [73] , f -DPO [74] , RSO [75] also enhance or expand DPO in various aspects. Despite these contributions, the possibilities for leveraging and further adapting DPO for recommendation are still largely unexplored and few studies discuss extending DPO to handle multi-negative scenarios. 
limitation: 
 Despite effectiveness, there are several limitations not addressed in this paper. On the one hand, the number of negative samples is capped at 15 in our experiments. The potential of multiple negative samples hasn't been fully explored due to the limited time and computation resources. On the other hand, increasing the number of negative examples inevitably results in higher training costs, a phenomenon that becomes more pronounced as the number of negative examples grows in the context of language models. 
conclusion: 
 In this work, we devised a principled Softmax-DPO (S-DPO) loss specially tailored for LM-based recommenders, utilizing multiple negatives in preference data to explicitly instill ranking information into LM. Empirically, S-DPO surpasses all baseline models including traditional and LM-based methods on three datasets in sequential recommendation tasks while successfully providing better rewards for preferred items compared to DPO. Grounded by theoretical proof, we bridge S-DPO with the softmax loss in self-supervised recommendations, underscoring the ranking performance of S-DPO and highlighting the critical roles of multiple negatives. Also, we theoretically find that S-DPO has an inherent benefit to mine hard negatives which provide larger and more effective gradients to model optimization, assuring its exceptional capabilities in recommendation tasks. We believe that S-DPO, as a generalization of DPO, provides valuable insights for future LM-based recommenders and has the potential to benefit research fields other than recommender systems 4 . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Our language model only serves as a recommender from a given item list, which makes it impossible to be misused or dual-use. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We incorporate three datasets in our experiments including MovieLens [50] , Goodreads 6 and LastFM [51] , all of which are open-source. Also, the backend language model used in our research is LLAMA2-7B [32] , which is also an open-source model. The details of our used assets can be found in Appendix B. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing or research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The PL model takes the form:EQUATION).The ranking in multi-negative preference data is e p ≻ e d |x u , ∀e d ∈ E d . Our new preference distribution that estimates the probability of the ranking can be derived: p * (e p ≻ e d , ∀ e d ∈ I d |x u , e p , E d ) = τ ∈{τ ′ |τ ′ (1)=p} p * (τ |x u , e p , E d ) = τ ∈{τ ′ |τ ′ (1)=p} K j=1 exp r(x u , e τ (j) ) K l=j exp r(x u , e τ (l) ) = exp(r(x u , e p )) K j=1 exp(r(x u , e j )) × τ ′ ∈Per(ind(I d )) K-1 j=1 exp r(x u , e τ ′ (j) ) K-1 l=j exp(r(x u , e τ ′ (l) )) = exp(r(x u , e p )) K j=1 exp(r(x u , e j )) × τ ′ ∈Per(ind(I d )) p * (τ ′ |x u , E d ) = exp(r(x u , e p ))When N = 2, the following equations hold:EQUATION)) = -E (xu,ep,E d ) log σ -log e d ∈E d exp β log π θ (e d |x u ) π ref (e d |x u ) -β log π θ (e p |x u ) π ref (e p |x u ) = -E (xu,ep,e d ) log σ -log exp β log π θ (e d |x u ) π ref (e d |x u ) -β log π θ (e p |x u ) π ref (e p |x u ) (N = 2) = -E (xu,ep,e d ) log σ β log π θ (e p |x u ) π ref (e p |x u ) -β log π θ (e d |x u ) π ref (e d |x u ) = L DPO (π θ ; π ref ).Therefore, DPO is a special case of S-DPO.Let V (θ; e d ) = g(e d , e p , x u ) = β log π θ (e d |xu) π ref (e d |xu) -β log π θ (ep|xu)π ref (ep|xu) and the S-DPO loss takes the following form:EQUATIONThe gradient of V (θ; e d ) can be formulated as:EQUATIONUsing properties of the sigmoid function that σ ′ (x) = σ(x)(1 -σ(x)) = σ(x)σ(-x) and thus ((log σ(x)) ′ = 1 σ(x) × σ(x)σ(-x) = σ(-x), we have:∇ θ L S-DPO (π θ ; π ref ) = -E (xu,ep,E d ) ∇ θ log σ -log e d ∈E d exp(V (θ; e d )) = E (xu,ep,E d ) σ log e d ∈E d exp(V (θ; e d )) • ∇ θ log e d ∈E d exp(V (θ; e d )) ((logσ(x)) ′ = σ(-x)) = E (xu,ep,E d ) σ log e d ∈E d exp(V (θ; e d )) • e d ∈E d exp(V (θ; e d )) • ∇ θ V (θ; e d ) e ′ d ∈E d exp(V (θ; e ′ d )) = -βE (xu,ep,E d ) σ log e d ∈E d exp(V (θ; e d )) • e d ∈E d ∇ θ log π θ (e p |x u ) -∇ θ log π θ (e d |x u ) e ′ d ∈E d exp(V (θ; e ′ d ) -V (θ; e d ))(Eq. ( 16))= -βE (xu,ep,E d ) σ log e d ∈E d exp(g(e d , e p , x u )) • e d ∈E d ∇ θ log π θ (e p |x u ) -∇ θ log π θ (e d |x u ) e ′ d ∈E d exp(g(e ′ d , e d , x u )) (Definition of V (θ; e d )) = -βE (xu,ep,E d ) σ log e d ∈E d exp(g(e d , e p , x u )) • ∇ θ log π θ (e p |x u ) - e d ∈E d ∇ θ log π θ (e d |x u ) e ′ d ∈E d exp(g(e ′ d , e d , x u ))The last equation is because:e d ∈E d 1 e ′ d ∈E d exp(g(e ′ d , e d , x u )) = e d ∈E d exp(V (θ; e d )) e ′ d ∈E d exp(V (θ; e ′ d )) = 1 B Experimental Settings B.1 BaselinesWe compare the performance of S-DPO, against both traditional and LM-based baselines to showcase the effectiveness of our method. Specifically, for traditional methods, we have:• GRU4Rec [46] utilizes the GRU (Gated Recurrent Unit) architecture to model sequences, enabling effective prediction in recommendation tasks. • Caser [47] employs both horizontal and vertical convolutional operations to enhance the capture of high-order interactions within item sequences, improving recommendation accuracy. • SASRec [48] incorporates a multi-head self-attention mechanism in its self-attentive sequential recommendation model, facilitating the modeling of intricate sequential data patterns.For LM-enhanced method, we have:• MoRec [49] advances traditional recommendation systems by incorporating the modality features of items instead of the id feature. we employ BERT for the text encoder and SASRec for the recommendation architecture.For LM-based methods, we have:• LLaMA2 [32] utilized vanilla LLaMA2-7B to directly generate recommendation results through direct prompting. • Chat-REC [15] is implemented based on the framework discussed in [15] , we retain user interaction sequences consisting of item titles as use profiles for a fair comparison. We use GPT4 [76] as its primary large language model. • TallRec [14] first propose to transform interaction sequences into textual prompts and then finetunes large language models using domain-specific corpus. • LLaRA [19] combines collaborative signals from traditional recommendation systems into the fine-tuning of large language models for improved recommendation performance.To evaluate the effectiveness of S-DPO, we conduct experiments on three widely used real-world datasets: Movielens [50] , Goodreads5 , and LastFM [51] . The statistics of datasets are illustrated in Table 3 . The MovieLens dataset is widely used for movie recommendation tasks and includes user ratings and movie titles, we select the MovieLens100K dataset in our experiment. Similarly, Goodreads is sourced from a social book cataloging website, where users can explore, rate, and review a variety of books. LastFM dataset comprises users' listening history and artists' names from the Last.fm online music platform. Following [19] , we maintain their titles as textual descriptions for each dataset. For Goodreads, we remove users and books with less than 20 interactions, which keeps the same as the processing of MovieLens. For all datasets, we organize sequences chronologically before dividing the data into training, validation, and testing sets in an 8:1:1 ratio to prevent any potential information leakage.We implement all approaches with Python 3.9.7, PyTorch 2.2.2, and transformers 4.38.2 on 4 NVIDIA A100s. We select Llama2-7B [32] as the LM backbone for S-DPO. Following [19] , we randomly select prompts from several prompt formats during training and evaluation to ensure flexibility and generality. For optimization of all the traditional methods, the Adam optimizer is employed with a learning rate adjusted to 0.001, and a batch size configured at 256. All models undergo L2 regularization, with coefficients experimentally determined from [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]. In all experiments involving large language models, we train each method for a maximum of 5 epochs using a batch size of 128 and select the checkpoint with the lowest loss on the validation set as the final checkpoint. A warm-up strategy is applied to the learning rate, starting at 5% of its maximum value, and gradually adjusting it through a cosine scheduler throughout the training process. For S-DPO and all of its ablation studies, we further conduct preference training for further 3 epochs with a batch size of 128 and a learning rate of 1e-5. Setting the value of β as 1, we search the number of negative samples in [3, 5] for the main results. The effects of both factors are further explored in 4.2.Given that LMs primarily produce textual responses rather than comprehensive item rankings, we utilize a re-ranking metric in line with previous research [19] to assess recommendation performance. For each sequence, a candidate set is constructed by randomly selecting 20 non-interacted items and always includes the correct item. We assess all models based on their ability to pinpoint the correct item within this candidate set, employing the HitRatio@1 (HR@1) metric for performance evaluation. Following [19] , we also introduce an additional metric called the Valid Ratio to evaluate the LM-based methods' adherence to instructions and their ability to generate appropriate responses. Due to the difficulty LMs face in producing ranked results for candidate items, position-aware metrics like NDCG are deemed unsuitable for this evaluation. In order to examine whether the superiority of S-DPO loss over traditional language modeling loss can be generalized across different backbone language models, we conducted experiments using models with varying architectures and sizes, including LLAMA1-7b [77] , Pythia-2.8b [78] , and Mistral-7b [79] . These models were tested on two distinct datasets: LastFM and MovieLens. We evaluated three training approaches: language models fine-tuned with standard language modeling loss, and models further trained using DPO and S-DPO losses. Due to limitations in computational resources and time, we experimented with two variants of S-DPO: S-DPO (3 negatives) and S-DPO (8 negatives). As shown in Table 4 , S-DPO consistently outperformed the language modeling loss, enhancing model performance while maintaining or even improving the validity of the generated outputs. Additionally, the performance continued to improve as the number of negative samples increased from 3 to 8.We left further exploration of softmax ranking loss in LM including more negative samples and validation on various settings as future works. We believe that S-DPO, a generalization of DPO loss has the potential to benefit other research areas other than recommender systems. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide our anonymized versions of data and codes with the link in the abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The work presented in our paper aims to instill ranking information into LMs to improve the performance of LM-based recommenders. We do not foresee any negative societal impacts stemming from the outcomes of our research. We further claim our broader impact in Appendix D. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://www.goodreads.comThc extension to broader impact will be detailed in Appendix Dhttps://www.goodreads.comhttps://www.goodreads.com 


Decision: The paper extends DPO to Softmax-DPO for language model based recommendation for user preference alignment inspired by the success of softmax loss in recommendation. While reviewers expressed concerns on novelty, they appreciate the simplicity of the method and solid results, and agree the reasons to accept outweighs reasons to reject after rebuttal.

Reviews: review: paper_summary: paper_summary: Inspired by advancements in Direct Preference Optimization (DPO) and the success of softmax loss in recommendations, the paper propose Softmax-DPO (S-DPO). S-DPO enhances LM-based sequential recommenders by incorporating multiple negative samples in preference data.The paper demonstrate the superiority of S-DPO in modeling user preferences and boosting recommendation effectiveness.
strength_weakness: strength_weakness: In the recommender system domain, considering preferences in the loss function is not particularly new. However, experiments and explanations for this are lacking.

1. There are no experiments comparing S-DPO with other preference-based losses in recommender systems (BPR, InfoNCE, as the paper mentioned ). If such comparisons are not feasible, the reasons for this are insufficiently explained.

2. The experiments lacked the application of S-DPO loss to various base models. Consequently, the general applicability of S-DPO was not demonstrated.

3. Running DPO multiple times and S-DPO appear conceptually similar. Therefore, there are no experiments comparing their efficiency. A comparison of time or memory usage seems necessary.
questions: questions: Same as weaknesses

Is there a justification for not conducting the experiment described above?
limitations: limitations: LM's methodology was brought to the Recommender system. However, there seems to be a lack of discussion in the Recommender system.

BPR loss is typically not used in sequential scenarios, so it would have been beneficial to emphasize that S-DPO specifically addresses preferences in sequential contexts. Additionally, conducting more experiments to validate this aspect would have been advantageous.
paper_summary: paper_summary: The authors propose a modification to the Direct Preference Optimization (DPO) by incorporating a softmax loss to enhance the training of language model (LM)-based recommender systems.
strength_weakness: strength_weakness: 1) Novelty
- The use of multiple negatives in recommendation systems is not new and should be cited. Thus, Research Question 2 (RQ2) lacks novelty as it has already been explored. The novelty of using multiple negatives in LM-based recommender systems is uncertain.

2) Statements
- Equation 8 represents the softmax loss, which has already been applied in other recommender systems like BERT4Rec, whereas Equation 12 does not represent the softmax loss, leading to confusion.
- The claim in lines 225-226 "indicating the significant roles of knowledge and reasoning ability in language models for recommendation tasks." is not provable due to differences in training datasets.
- The statement "However, indicated by the low valid ratio and suboptimal performance, untuned LM-based recommenders are limited by inadequate instruction-following capabilities or a lack of domain-specific knowledge" in lines 228-230 is incorrect as evidenced by high ValidRatio scores of ChatRec.

3) Replicability
- The prompts used in the experiments are not provided, making replication difficult.


4) Evaluation
- NDCG is usually preferred, and metrics like @10 or @20 are more commonly used than @1 in recommendation evaluations. For example, LM-based RecSys works like LLama4Rec use @5.
- In Figure 2a, the statistical significance of the results is not specified. It is also unclear why the results are based on LLama and not LLama2.

5) Comparison
- The comparison of training epochs between LM-based recommenders (5 epochs) and S-DPO (an additional 3 epochs) is unfair.
- In Table 1, the best HR@1 is highlighted in bold, but the same should be done for the ValidRatio column even if your model does not perform the best in that aspect.
- The statistical test used should be specified.
- Optimizing hyperparameters for the proposed model without doing the same for competitor models is unfair. A fair comparison would involve equal time allocation for hyperparameter searches.
- Figure 2b shows that S-DPO decreases faster but remains higher than DPO, making the comparison questionable. Similarly, Figure 2c lacks clarity on whether the values are comparable.
-The statement "On the other hand, an excessively large β prevents the model from effectively learning ranking relationships, leading to suboptimal performance" is not supported by Figure 3b and 3c, which show consistent performance improvement.


6) Related work
- The related work section is more effective when placed after the introduction to better contextualize the state-of-the-art before presenting the new method.
- The discussion on LM for recommendation is not thorough, with many works cited together without detailed descriptions.
questions: questions: - Consider replacing "side effect" with a more positive synonym.
- The methodology section (2) appears empty; clarify if the true (new) methodology starts at section 3.1.
- Equation (1) needs clarification as i_p and i_d represent items (identifiers) and not scores nor ranks. 
- Space before subsection name in line 148
- Line 159: "mutli" --> write "multi"
- Clarify lines 174-180 regarding "hard negative items" and the theoretical basis for S-DPO's ability to mine hard negatives.
- Section 3.2 is unclear, especially the statement "The structural correlation indicates that DPO and S-DPO are more suitable for recommendation than the language-modeling loss."
- The definition of Rel.Ipv should be moved to the caption of Table 1.
limitations: limitations: - The computational effort required for using more negative samples is not considered. It would be beneficial to compute an approximate empirical complexity of the method.
- The use of the Movielens 100K dataset is a limitation due to its small size. A larger dataset like ML-1M would provide better validation while remaining computationally feasible compared to ML-20M.
paper_summary: paper_summary: proposed a softmax DPO in recommendation domain.
strength_weakness: strength_weakness: 1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.

2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?
questions: questions: 1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.

2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?
limitations: limitations: It is unclear how this framework can scale up to large scale dataset.
paper_summary: paper_summary: This paper extends DPO from pairwise (Bradley-Terry) to multi-way comparison (Plackett-Luce), where one positive example is considered better than multiple negative examples. The paper applies this approach in recommendation systems, showing promising results on several standard benchmarks.

From Rafailov et al: “although more general Plackett-Luce ranking models [ 30, 21] are also compatible with the
framework if we have access to several ranked answers”.
This paper shows the derivation of the Plackett-Luce extension. In that sense the technical contribution is somewhat incremental.
I would therefore view this more as an application paper in recommendation systems. However this is not how it is written, most of the focus is on the theoretical analysis extending DPO from Bradley-Terry to Plackett-Luce.
strength_weakness: strength_weakness: * The technical contribution seems somewhat incremental.
* Some points need clarification (see questions below).
questions: questions: 1. It is assumed that there is a single positive example which is better than all the negative ones. In reality, multiple items may satisfy users’ interests, so the negative examples should be chosen with care. In the experiments, how are negative samples chosen? This was not clear from the paper. \
Actually, in recommendations it is natural to show a slate of results and the user would click on one (or more), so the unclicked results can serve as negatives. Though this is not the type of data available in benchmarks used here such as MovieLens.

2. In Fig 2a, it seems like most of the gains in performance already exist in SFT. How come this is much better than other fine-tuned models like LLaRA?

3. Fig 2b: the performance of S-DPO seems to still be improving after 1200 iterations, why not run it longer until it stops improving?
Also, since the S-DPO loss and DPO loss are not directly comparable, you should include a figure showing HR@1 as a function of training steps for S-DPO and DPO.

4. Fig 3a: the performance seems to still be increasing, is there a number of samples for which it saturates?

5. MIssing baseline: compare the proposed approach to DPO with pairs consisting of the positive example and each negative example separately (use same negative example choice as S-DPO). This is more computationally expensive but interesting to see how it compares in terms of performance (and runtime).
Related to that, it would also be good to include a comparison of runtimes between S-DPO and DPO (perhaps show the above variant of Fig 2b in terms of runtime, not just of training steps).

Minor:
* L79: empty section
* L132: deviate -> deviating
* L175: the exp should be with the sum
* L180: “faster<space>.”
limitations: limitations: No societal impact concerns.

Comments: title: Thank you for your time
Thank you for your time
Thank you for your time
Thank you for your time
Experimental results for natural negatives
Experimental results on comparing S-DPO with other preference-based losses (BPR and InfoNCe)
Response for new questions
Sorry for leading confusion about time complexity
Sorry for leading confusion
Sorry for leading confusion
We edit the complexity
comment: We are very grateful for your valuable comments and your continual interest in our work. Your input on natural negatives has been of great assistance in refining our approach and leaving space for future exploration. We hope that our responses have addressed most of your concerns. If they have, we would appreciate it if you could consider revising your score. As the rebuttal period is ending soon, we are eager to further discuss any remaining concerns you may have. Is there anything else you’d like to clarify?
Thank you very much for your valuable feedback. Your suggestions on complexity comparison have enhanced our method. We hope our replies have resolved most of your concerns. If so, we kindly ask if you might consider revising your score. As the rebuttal period is nearing its end, we would like to discuss any remaining issues you may have. Is there anything else you'd like to discuss?
We sincerely appreciate your valuable comments. Your feedback on direct comparisons and complexity analysis is important for improving our method. We hope our responses have satisfactorily addressed most of your concerns. If this is the case, could we kindly ask you to consider adjusting your score? With the rebuttal period coming to a close, we are eager to discuss any further concerns you might have.
We greatly appreciate your valuable comments. Your insights on various base models and comparisons between DPO and S-DPO have significantly helped us improve our paper. We hope our responses have addressed most of your concerns. If they have, would you consider raising your score? If you have any remaining concerns, we would be eager to discuss them further, especially as the rebuttal period is about to end. Do you have any additional questions?
Following your suggestion, we employed GPT-4 to simulate user behaviors and used these simulated user choices as natural negatives. The experimental results are as follows:

| Method | HitRatio@1 | ValidRatio |
| ------ | ---------- | ---------- |
| SDPO-random | 0.6609 | 0.9900 |
| SDPO-natural | **0.6689**     | **0.9916**     |

The results indicate that incorporating hard negatives is beneficial for S-DPO, highlighting the critical role of negative sampling strategies. We believe that exploring negative sampling is an intriguing direction with significant potential for LM-based recommenders.

We appreciate your insightful comments and look forward to further discussion.
It is noted that traditional similarity-based training losses are not well-suited for generative language model-based recommenders, and thus cannot be directly applied in our context. To address this, we adapted BPR and InfoNCE losses within the language model training framework, employing random negative sampling on the LastFM dataset using Llama2-7b. The experimental results are summarized below:

| Method     | HitRatio@1 | ValidRatio |
| ---------- | ---------- | ---------- |
| LLAMA2     | 0.0233     | 0.3854     |
| LLAMA2-BPR     | 0.0008     | 0.0152     |
| LLAMA2-InfoNCE | 0.0029     | 0.0246     |
| LLAMA2-SDPO     | **0.6609**     | **0.9900**     |

From these results, we observe that adapting traditional losses for LM-based recommenders significantly diminishes the inherent capabilities of language models, while also failing to effectively capture domain-specific knowledge. This leads to a marked decrease in both hit ratio and valid ratio.

From this experiment, we can see that traditional recommendation losses cannot be directly adapted to LM-based recommenders. Therefore, incorporating explicit ranking information and multiple negative examples into LM-based recommenders is a nontrivial task. S-DPO addresses this challenge by introducing a partial order relationship into PL preference modeling and further deriving the effective loss with softmax structure.

We look forward to further discussing these findings with you.
Thanks for your continued interest in our work.

For the first comment. Our loss is based on our proposed preference model $p^*(e_p\succ e_d,\forall e_d\in E_d|x_d)=\frac{{\rm exp}(r(e_p,x_u))}{\sum_{j=1}^K{\rm exp}(r_\theta(e_j,x_u))}$, which is derived in the paper from the PL model. In recommendation scenarios, we follow the RLHF pipeline and finally derive this new preference model which possesses a softmax structure similar to a multinomial logit choice model.

For the second comment,  we agree that it is interesting to explore the effectiveness of natural negatives. Currently, we are working on simulating user choice with LLM to collect simulated behaviors as natural negative samples following your suggestions. The response to this question is on the way, for the necessary time to set up this experiment including training LLM. We will release our results once the experiment is finished. 

For the last comment, we appreciate your acknowledgment. All the results of additional experiments will be included in the revised version of the paper.

 If you have additional concerns, we would be more than happy to provide additional clarification. Thank you for your attention.
Thanks for clarifying the point about the loss based on partial ranking being different than PL which is based on full ranking. In that case is it not more similar to a multinomial logit choice model than to PL?

Regarding negative sampling, I understand that natural (non-random) negatives are hard to obtain from public benchmarks. You could potentially run a semi-synthetic setting where the recommender presents a slate of items and the user choice is simulated. For example, you can learn a user utility for each item based on past user ratings and then use the utility as a score in the choice model.

Thanks for adding experiments with the DPO-neg baseline, I think those should be included in the paper. In 2 out of the 3 settings DPO-neg achieves similar performance -- but with higher computational cost, so you can make that point in the paper.
We apologize for our mistakes. The revised complexity analysis is as follows:

$K$ denotes the number of negative items.

$C_\mathcal{M}$ denotes the base model $\mathcal{M}$. 

$S_t$ denotes the size of the S-DPO training set.

**Time Complexity**

| Methods | S-DPO | DPO |
|:------:|:------:|:------:|
| Complexity  | $ \Theta((K+1)(C_\mathcal{M}+1)S_t) $  | $\Theta(2KC\_\mathcal{M}S\_t)$ |
We apologize for our mistakes. The revised complexity analysis is as follows:

$K$ denotes the number of negative items.

$C_\mathcal{M}$ denotes the base model $\mathcal{M}$. 

$S_t$ denotes the size of the S-DPO training set.

**Time Complexity**

| Methods | S-DPO | DPO |
|:------:|:------:|:------:|
| Complexity  | $ \Theta((K+1)(C_\mathcal{M}+1)S_t) $  | $\Theta(2KC\_\mathcal{M}S\_t)$ |
We apologize for our mistakes. The revised complexity analysis is as follows:

$K$ denotes the number of negative items.

$C_\mathcal{M}$ denotes the base model $\mathcal{M}$. 

$S_t$ denotes the size of the S-DPO training set.

**Time Complexity**

| Methods | S-DPO | DPO |
|:------:|:------:|:------:|
| Complexity  | $ \Theta((K+1)(C_\mathcal{M}+1)S_t) $  | $\Theta(2KC\_\mathcal{M}S\_t)$ |
>**Comment 8: Unclear section**

- **Synonym.** We will replace it with more positive synonym such as "ancillary benefit" in our manuscript.
- **Empty section.** Methodology starts at Section 3.1.
- **Equation (1).** We will modify the notation "$>_u$" to "$\succ_u$" to better distinguish the comparisons between items according to the preferences of user $u$ from the comparisons between scores or ranks. 
- **Lines 174-180.** S-DPO treats gradients of different negative (dispreferred) items differently by assigning the gradient of each negative item with an extra term $\frac{1}{\sum_{e_d \in E_d}{\rm exp}(g(e^\prime_d,e_d,x_u))}$=$\frac{{\rm exp}(r_\theta(x_u,e_d))}{\sum_{e^\prime_d\in E_d}{\rm exp}(r_\theta(x_u,e^\prime_d))}$ . This term reflects the relative reward of each negative item compared with other negative items. We can categorize negative items into two categories: (1) Hard negative items , whose reward $r_\theta(x_u,e_d)$ is relatively high, making it more probable to be chosen by  LM-based recommenders; (2) Easy negative items, whose reward $r_\theta(x_u,e_d)$ is relatively low, making it less likely to be output. For  hard negative items, the extra weight term tends to be larger, leading more decline for likelihood. 
- **Section 3.2.**  "Given effectiveness of BPR and InfoNCE loss in recommendation, we argue that sampled-based loss which explicitly compares preferred and dispreferred items such as DPO and S-DPO is more suitable for training LM-based recommenders than only utilizing language modeling loss."
- Rest mistakes will be modified in the revised version, thanks for pointing out.


> **Comment 9: Complexity.**


$K$ denote the number of negative items.

$C_\mathcal{M}$ denote the base model $\mathcal{M}$. 

$S_t$ denote the size of the S-DPO training set.

**Time Complexity**

| Methods | S-DPO | DPO |
|:------:|:------:|:------:|
| Complexity  | $ \Theta((K+1)(C_\mathcal{M}+1)S_t) $  | $\Theta(2KC\_\mathcal{M}S\_t)$ |
**For Reviewer iyxD**
I would like to express my gratitude for your detailed review and the valuable feedback provided. 

>**Comment 1: Uncertain Novelty.**

We agree with you that introducing multiple negatives is important and has been widely explored in conventional recommenders, as we briefly discussed in section 3.2.
However, as your discovery, LM-based recommenders which has different training paradigm compared with traditional recommenders has just been explored, on which multiple-negatives has largely omitted and lack effectively method to introduce into training pipeline. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.

>**Comment 2: Confusing Statement**

- Clarification of Equation 12:
  Equation 12 is also an equivalent variant of the softmax loss, which can be rewritten as: $
  E_{(u,i_p,I_d)}\left[{\rm log}\frac{{\rm exp}(f(u,i_p))}{{\rm exp}(f(u,i_p)) + \sum_{i_d \in \mathcal{I}_d}{\rm exp}(f(u,i_d))}\right] $

- We will modify line 225 to "indicating the significant roles of knowledge and reasoning ability in language models for recommendation tasks in semantically informative datasets."

- Untuned LM-based recommenders achieve suboptimal performance because of inadequate instruction-following capabilities (reflected by a low valid ratio and a low hit ratio) or lack of domain knowledge (reflected by a high valid ratio but a low hit ratio). ChatRec, with GPT-4 as its backend, falls into the second case. We will modify line 228 to avoid confusion.

> **Comment 3: Replicability.**

 In fact, we have uploaded all the code and prompt used in the anonymous link " https://anonymous.4open.science/r/S-DPO-C8E0" which has been attached in the abstract of submitted manuscripts. We will further include all of our prompts in the appendix.

> **Comment 4:  Evaluation.**

- We calculate further HitRatio and NDCG, please refer to supplementary material (Table 2) for results.

- Sorry for leading confusion, the results in Figure 2a is based on LLama2 instead of LLama, which will be modified in the manuscript. 

> **Comment 5: Statistical Test.**

We utilize one-sided t-test for all of our statistical test and find the p value less than 0.05 among all shown experiments. Clarification will be included in implementation details.


> **Comment 6: Comparison.**

- It has been shown that 5 epochs is enough for model convergence. Follow your suggestions, we conduct SFT for further 3 epochs  and results show that it bring no gain to the final performance.
- We are adding a dedicated hyperparameter table to provide transparency and demonstrate that each model was given an equal opportunity for optimization. Please refer to supplementary material (Table 1).
- **Regarding Figure 2b and 2c**:
  We further ensure that the validation setting for DPO and S-DPO are consistent, allowing for a direct and fair comparison. The revised results are now included in the supplementary material (Figure 1). We compare loss, relative likelihood and absolute likelihood to validate the effectiveness of S-DPO
- **Regarding Figures 3b and 3c**:
  A very low β overly prioritizes ranking information, compromising the model's ability to follow instructions, as seen in the decreased valid and hit ratios. Conversely, an excessively high β constrains the model too much by the reference model's standards, leading to lower performance (hit ratio@1). A slight increase in valid ratio with β values greater than 1 suggests better adherence to the reference model's constraints, supporting our interpretation.

>**Comment 7: Related work.**

We agree with your suggestion and we will reorganize our revised version. The lack of in-depth discussion is due to space constraints. We will include a more detailed discussion in the appendix.