On Softmax Direct Preference Optimization for Recommendation
Authors: Yuxin Chen
Junfei Tan
An Zhang
Zhengyi Yang
Leheng Sheng
Enzhi Zhang
Xiang Wang
Tat-Seng Chua
Venue: NeurIPS 2024 poster (2024)
Keywords: Sequential Recommendation
Large Language Models
Primary area: other

Abstract: Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (\textbf{S-DPO}) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO.

On Softmax Direct Preference Optimization for Recommendation
Yuxin Chen'
Junfei Tan
An Zhang
Zhengyi Yang² Leheng Sheng' Enzhi Zhang?
Xiang Wang² Tat-Seng Chua!
National University of Singapore
University of Science and Technology of China
Hookkido University
yuxin. chen@u. edu, sober_ pheer_clleer@@mall..... ustc. edu en, anzhang@u. nus edu
leheng leheng.shengâu mus mus. mds. enzhi zhang. n6delms mwwelmm..oouuai................. thokudii.cc...
yangzhy11990ggail. com, xiangwang 2238gmai1 12ngwang12230gmai
com, dcsctsênus edu.
Abstract
Recommender systems aim predict personalized rankings based on user pref- erence data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical inter- actions into language prompts, pairing with a positive item as the target response and fine-tuning LM with language modeling loss. However, the current objective fails fully leverage preference data and is not optimized for personalized rank- ing tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement Direct Preference Optimization (DPO) human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (S-DPO) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in preference data and devise alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https: / titthuu..a. com/ chenyuxin1999/5 DPO.
Introduction
Recommender systems aim predict personalized rankings based on user preference data, i.e. historical interactions such purchases, clicks, and ratings [1, 2]. Recently, leveraging the extensive world knowledge and powerful reasoning abilities of language models (LMs) [3-6], LM-based recom- menders have been broadly explored [7-9]. These recommenders convert historical interaction data into language prompts and either perform in-context learning or fine-tune LMs, demonstrating notable
TThee authors contributed equally to this work.
An Zhang is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

advantages, including zero-shot and few-shot reasoning [10-13], enhanced generalization abilities [14, 15], and rich semantic understanding [16-19]. However, current LM-based recommenders typically utilize language modeling loss for personalized ranking Dojjetives-prediccinng the next to- won-h significantly differs from the objective of modeling user preferences in recommendation tasks [20, 21].
We argue that the current objective of LM-based recommenders does not fully utilize preference data and is not optimized for personalized ranking tasks, thereby hindering recommendation performance. Most LM-based recommenders address recommendation tasks by leveraging specialized language prompts [14, 17, 22-24], incorporating collaborative signals as a new modality [18, 19, 25], or extending the vocabulary of LMs with item tokens [16, 26-30]. Typically, these recommenders pair each language prompt, including the user's 's historical interaction item lists, with single positive item and then update LM parameters using language modeling loss [14, 19]. Despite being designed for recommendation tasks, these LM-based recommenders do not consider negative items and are not directly optimized for personalized rankings. Such training paradigm fails to fully leverage user preference data and overlooks the role of negative items in recommendations, thereby impeding the alignment LMs with user preferences.
Inspired by the success of using human-labele data to align LMs with human preferences [31-33] and advancements in direct preference optimization (DPO) [34-6]], we make progress on aligning LMs with recommendations by fine-tuning them to predict the next item in accordance with the user's s preference. This preference aligmment stage aims to instill ranking information into the LMs and help recommenders distinguish preferred items from negatives, rather than solely focus positives.
Towards this end, we incorporate multiple negatives in user preference data and devise alternative version of DPO loss tailored for recommendation, connected to softmax sampling strategies [20, 37- 391, which we call S-DPO. Specifically, we first devise supervised fine-tuning to inject domain knowledge and improve LM's ability to follow the instructions before preference alignment phase, following [14, 33]. In the preference alignment stage, instead of constructing solely positive pairs, we initially pair each language prompt with both positive and randomly sampled multiple negatives to build text- based preference data. Building upon these preference data, we extend conventional DPO with the Bradlcy-Terry preference model [34, 40] on pairwise data to the Plackett- preference model [41, 42], which handles relative rankings among multiple samples. Furthermore, we generalize the traditional Plackett-Luce preference model, which is designed for full relative rankings, to accommodate partial rankings, a more natural fit for recommendation tasks.
Benefiting from the multiple negatives in preference data, S-DPO offers three appealing properties. On the one hand, S-DPO serves as the first specialized personalized ranking loss for LM-based rec- ommenders, effectively utilizing multiple negatives and acknowledging the importance of preference data. Empirically, we demonstrate that provides more effective ranking gradients and better rewards for preferred items compared with DPO (cf. Section 4.2). On the other hand, we theoretically bridge the DPO loss with the pairwise BPR loss [43] over pairwise data and connect S-DPO with the softmax loss over negative sampling (also known as contrastive loss in self-supervised recommendations, which achieves state-of-the-as performance [37, 44, 45]). This connection naturally underscores the ranking performance of S-DPO and highlights the critical role of multiple negatives. Furthermore, gradient analysis demonstrates that S-DPO has an inherent benefit of mining hard negative examples similar to contrastive learning paradigm [38], which not only boosts the performance but also acceler- ates the training process (ef. Section 3. 3.1), assuring its exceptional capabilities in recommendation tasks.
Overall, our contributions can be concluded as follows:
We are among the first to point out that the widely used language modeling loss in LM-based recommendation is not designed for ranking tasks and fails to fully utilize user preference data, thereby hindering recommendation performance.
We propose S-DPO, an alternative version of DPO loss extended from the traditional Plackett-Luce preference model, incorporating multiple negatives to instill ranking information into LM and tailoring for LM-based recommenders.
We theoretically bridge S-DPO with the softmax loss over negative sampling to highlight the critical role of multiple negatives and find its inherent benefit of mining hard negatives, assuring its capabilities.
2

---

Task Formulation. Given the historical interactions Hy of one user chronological order, the goal of LM-based sequential recommender Mg. where 0 represents trainable parameters, is to select the item ip preferred by user u from candidate set = (ig ( where is the number of watching [History Sequence], which do you think person choose [ltem Listj?"
Supervised Fine-Tuning
Direct Preforence Optimization
Softmax-DPO
fp
4d
Sottmax
Loos
Ranking Loss
Pairwise Preference Data
Multi-Negative Preference
Figure Framework of S-DPO. Different from existing methods which fine-tune LMs with language modeling loss without tailoring for recommendations, S-DPO proposes to explicitly instill ranking information into LMs. To take one step further, S-DPO incorporates multiple negatives in user preference data and generalizes pairwise DPO loss to softmax ranking loss.
Preliminary
In this section, we first formalize sequential recommendation as the task of aligning language models (LMs) with user preferences. Then, we discuss the general framework of current LM-based recommenders that utilizes language modeling loss to fine-tune LMs. Finally, we outline the training process widely used to align LMs with human preferences, including reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO).
candidates. This task requires that item ip be preferred over the other candidate items, denoted by Ia = C (1p). This requirement explicitly defines a multi-negatiiv preference understanding for LM-based recommenders, which can be formulated as follows:
Via Ias
ip Fu id.
(I)
wherein >a stands for the preference of user u.
Fine-tuning LM-based recommenders. Current LM-based recommenders widely adopt super- vised fine-tuning (SFT) [33] recommendation
data to enhance their performance [7, 9]. Generally, this involves two steps: structuring recommendation data as text-based pairs and then fine-tuning LMs based on these pairs. In the first step, for user u, recommendation task prompt Ia encompasses the user's 's historical interactions Hu. the candidate item set C, and a description of the sequential recommendation task. This prompt z. paired with the title of the preferred item ip in the candidate set C, denoted as cp. to form the pair data (Iu,ep). In the second step, the (Iu, ep) pairs utilized fine-tune the LM-based recommender Ma through language modeling loss. This loss, commonly used in SFT in language modeling tasks, implicitly treats the recommendation task as predicting the next token based on preceding tokens. Formally, the objective of optimizing the LM-based recommender My with pair data (Zu, ep) can be formulated as:
lepl log( log(Po((ep))(tt (cp)
max
(Ku.ey) t=1
where lepl is the number of tokens in ep. (ep)s is the t-th token of e, and (ep) <e is the tokens preceding (ep).-
However, recommendation tasks are essentially user preference alignment tasks, formalized in the above task formulation, and differ from language modeling tasks that consider only positive responses. Such gap necessitates further exploration into aligning LM-based recommenders with user preference, an area that has been underexplored.
RLHF pipeline and DPO. Recent studies in natural language processing (NLP) explore the use of haman-labeled pairwise data as a reward signal to align LMs with human preferences, such as RLHF [33] and DPO [34]. Specifically, the RLHF [33] pipeline adds two additional phases after the SFT phase: reward model training and reinforcement learning (RI.) optimization. After obtaining the SFT model SPT RLHF further optimizes it with pairwise preference data.
3

---

Inspired by the success of RLHF in NLP, we leverage RLHF to inject recommendation seapeeennnnttissssssfiiee user pairwise preference into items, where Ej denotes the title of item 8j. Given two items 8j. dig E C, the user preference i; > u i can seamlessly transformed into response preference, stipulating that e, is preferred over ek when presented with prompt z.u. denoted as ej > 62 E.. By sampling one dispreferred item ig from dispreferred candidate set IJ, we can curate preference dataset {(ep. Ed, x.))-
recommenders. Let & {e, {e,}y=ı denote the title set of candidate
After that, RLHF utilizes a preference model for preference distribution modeling, such as Bradley- Terry (BT) model [40]. This preference model assumes there is latent function r(Z eÿ) representing the reward of prompt-response pair (æu, ez). The bigger reward r(Zu, e;) means the more user prefers item From this perspective, reward function r Cÿ) serves as scoring function that quantifies the preference of user u to item i. Besides, the preference model defines mapping from the reward function r(zu eq) to a preference distribution pr(ez > ex (z.)). Based on preference distribution, an optimal reward function is trained by maximizing the likelihood of preference data. The training objective of this phase as follows:
LRM (pp plog > ea(zu)]
Let (e) be the probability that LM-based recommender MM output title given prompt zu-- The final reinforcement learning phase aims maximize the expected reward of policy while not deviate too far from the reference model, formulating the following objective for optimal policy:
max BD,,DD,ennalee aw) (r( e)] [e00l~3DD.....m.] (DK..[TT(eeepp | |oo (e)x,)), TT@
which is equivalent to where 'D denotes the distribution of Tu and Treef π recent study, DPO [34], theoretically proves the optimal policy in a closed form to Eq. is
SFT
1 Z(xn)
( (rramnee):
xx (e((xw
That (e
(5)
r(zu, e) = B log x(e[Fu)
+ B Blog Z (xx)).
(6)
(re((e(
where Z(zu)= = Ee (naree/z.)ex (fr(au,))) e)) is the partition function.
By defining pr (e, > ea|zu) as (f((zu ep) r(zu, Ea)) E3) according the BT model used in RLHF and substituting term r(Eu, e) in (3) with (6), the last two phases of RLHF pipeline can be equivalently transformed into optimizing DPO loss below:
To(epp 1 log mo(ea|(u))
Teef Fror(ea( mo((e)))))) ).
LDoo (z(ee))) logo
(glog
Tzot (ro((ep(zu)
wherein (x) is the sigmoid function.
DPO is able to directly extract the optimal policy from pairwise preference data, making it more practical for preference alignment than RLHE Nevertheless, DPO and RLHF are usually designed for pairwise preference. The oversight of other negative items impedes the performance of the LM-based recommenders. To bridge the gap, we expand DPO to S-DPO in recommendation tasks, in consideration of multiple negative items.
Methodology
3.1 Derivation S-DPO loss
To align LM-based recommender Ma with multi-negative preference, we first derive the preference distribution and then propose new loss function called S-DPO (depicted in Figure l).

---

Multi-negative Preference Distribution. As mentioned in Section 2, for user there is partial ranking stipulating ip >a ia. Vid Ia in sequential recommendation tasks. Let & be the titles of dispreferred items Iz. The aforementioned partial ranking is equivalent to ep > ealxu, Vea E Ea- from which a multi-negative preference dataset (Iu, Cp: E can be curated in an analogous way to RLHF.
For the dataset pairing one preferred item with multiple dispreferred items, we leverage the Plackett- Luce (PL) model [41, 42] to build preference distribution. Given prompt Tu. K titles e1, ... e and permutation т [K] > [K] reflecting the user preference, with r()) denoting the j-th element of permutation the PL model estimates that the ranking Cr(), Cr(K) turns out true, as:
K exp (((u:CT(T)
p(T e2, eK,
II
ΣK exp(r (Ius Cr(1))
By enumerating all the permutations starting with p and calculating sum of their probability given by the PL model, the final multi- preference distribution p" can be derived as:
exp(r(xn;ep) ep))
(ep - ed, Vea Ealow)
(9)
Σラ= (x)(
For brevity, the complete derivation is delegated to Appendix A.1.
Deriving S-DPO. By substituting reward function r(z,, e) in Eq, with Eq.(6), the multi-negative preference distribution can be rewritten as:
(ep ed E Ealxt) 1 Eesecs exp B log meeglz.)) log s((plw..) Teee Tepelep/...)
(10)
Teat (eala.))
Through plugging distribution given by Eq.(10) in the reward learning objective in Eq.(3), our S-DPO loss can be formulated for policy wy as:
Te (eal(x))) log Tref (Cpp (x))
To Te(eppann
LS-DrO Wret) - -Eeee Ea)
logo
(llo
Σ edESa
exp
Blog
Tref Tret(ed(
(11)
Notably, when the number of candidates N is 2, which means there is only one dispreferred item, S-DPO reduces DPO. The proof is provided Appendix A.2.
Gradient Analysis. We conduct gradient analysis on S-DPO. The gradient of LS-DPO with respect
parameters takes the following formulation:
VoLs-DPO(To: Fret)
Vo log (o(eal explg(e's ea, zu))
log Σ (p: Ea
Vo log Toggo((pppta))
Σ
ed E
e
higher weight when reward deviates from preference
higher weight reward larger
wherein g(ey, Ck.
re( ro (#aa ex) and similar DPO, re (" e) Blog seteleu).. is the implicit reward function defined by my. See Appendix A.3 for complete derivation.
x.)) e
Recap the DPO gradient below:
VaLoPO Fref)
-33E (..
ag(ess Cps Cp, In))
[V logTo(e)|(i)) Vo logTo(ea(x:)))
higher weight when reward is wrong
Similar DPO, the gradient of S-DPO loss increases the likelihood of the preferred item and decreases the likelihood of all the dispreferred items. Each example is also weighed by how much the
5

---

implicit reward r(zu, e) deviates from the preference data. However, compared with DPO, S-DPO harnesses information of multiple dispreferred items in this weight.
Moreover, S-DPO treats gradients of different negative (dispreferred) items differently by as- signing the gradient of each negative item with extra weight Edyce, explate(1ea.m.))
oxp(ra(().ed)) Eyee, (2) This term reflects the relative reward of each negative item compared with other negative items. Similar to [38], we can categorize negative items into two categories: (1) Hard nu(sa[zu) is relatively high, making it more probable negative items, whose reward re( ex) 3 Teatlea
to be chosen by LM-based recommenders; (2) Easy negative items, whose reward ro ( ea) is relatively low, making it less likely to be output. For hard negative items, the extra weight term exp(ra(wu.ea)) tends to be larger, leading more decline for likelihood. This mechanism Eyge, ex)(
makes LM-based recommenders more discriminative and endows S-DPO with more effectiveness and stability than DPO.
3.2 Properties of S-DPO
In this section, we will discuss the structural correlation between DPO and BPR [43], together with S-DPO and softmax loss [39], which demonstrates the advantage of S-DPO over DPO and language modeling loss.
For user u, preferred item ip and one dispreferred ia E Ig, BPR loss takes the form:
CBPr (u,iptta) [log a (f(u, ip) f(u, ia))],
(12)
wherein f(u, i) represents preference score of user for item i. Similarly, given dispreferred item set Iz, the softmax loss takes the form:
Luoftmax = - -E Ia) log a (-log Σ exp (S(u, id) - f(u, M(us 1p))
(13)
(es())la)) is the implicit reward function, denoted by re( e) in Section 3.1. According to Section 2, re x.) reflects the preference of user item i corresponding title e. When the reference model has no knowledge about recommendation, i.e., when Tref (e(()) ) is approximately uniform distribution,
Review the DPO loss Eq.(7) and S-DPO loss in Eq.(11). Notably, term Glog ma(elau))
term ro (2a, e) log (e(e)w)) exactly reveals absolute preference. Hence, r T e) possesses ((e(((/)))
similar function to [u, i)
From this perspective, DPO and S-DPO can be seen special patterns of BPR and softmax loss, respectively. Given the effectiveness BPR and InfoNCE loss in recommendation, we argue that sampled-based loss which explicitly compares preferred and dispreferred items such as DPO and S-DPO more suitable for training LM-based recommenders than only utilizing language modeling loss. Moreover, softmax loss works better than BPR loss in multi-negative scenarios [20], it can be inferred that S-DPO will more tailored for multi-negative user preference alignment than DPO.
Experiments
In this section, we aim answer the following research questions:
How does S-DPO compare with traditional and LM-based sequential recommendation models on performance?
RQI:
RQ2: How does the LM-based recommender benefit from the multiple negatives in S-DPO?
RQ3: What are the impacts of the essential parameters (3) on S-DPO?
Baselines. We thoroughly compare S-DPO with three categories of recommenders. in sequential
recommendations: traditional recommenders (GRU4Rec [46], Caser [47], SASRec [48]), L.- enhanced recommenders (MoRec [49]) and LM-based recommenders (LLaMA2 [32], Chat-REC

---

Table 1: The performance comparison on three real-world datasets. "Rel.Ipv" denotes the relative improvement of S-DPO compared with baselines.
Goodreads ValidRatio
LastFM ValidRatio
MovieLens ValidRatio
HR@
ReI.Ipv
HR@1
Rel.Ipv
HR@1
Rel.Ipv
Truitional
GRUARee
0.3867
1,0000
70.91%
0.2616
1,0000
153,36%
0.3750
1,0000
40.35%
Caser
0.4174
1,0000
58.34%
0.2233
1,0000
196.82%
0.3861
1,0000
36.3/%
SASRec
0.3581
1,0000
84.56%
0.2233
1,0000
196.82%
0.3444
1,0000
52.82%
LM- MM-based
LLaMA2
0.0233
0.3845
2736. 48%
0.0246
0.3443
2596.31%
0.0421
0.4421
1150.12%
ChatRec
0.3306
1,0000
99.91%
0.3770
1,0000
75.87%
0.2000
0,9895
163.15%
MoRce
0.2877
L,000
129. 129.72%
0.1652
1.0000
301.21%
0.2822
1,0000
86.50%
TALLRcc
0.4983
0.9573
32.63%
0,4180
0,9836
58.56%
0.3895
0,9263
35.12%
LLaRA
0.5292
0.9950
24.89%
0.4508
0.9918
47.03%
0.4737
0.9684
11.10%
Ours
S-DPO
0.6609
0.9900
0.6628
0.9992
0.5263
0,9895
[15], TALLRee [14], LLaRA (19]). See detailed introduction and comparison of baselines in Appendix B.
Datasets. We conduct extensive experiments on three real-world benchmark datasets which differ in size and domain (Movielens [50], Goodreads', and LastFM [51]). Following standard settings of [15, 19], we employ a commonly used metric Hit Ratio I (HR @ 1) for performance evaluation and an additional metric Valid Ratio to evaluate the LM-based methods' ability generate appropriate responses. See detailed introductions of datasets and evaluation metrics in Appendix B.
Implementation. We implement all LM-based recommenders on NVIDIA A A100 GPUs. For all L.-based recommenders, we conduct a supervised fine-tuning stage for maximum of 5 epochs. For S-DPO and its variants, we conduct a preference alignment stage for a further 3 epochs. Different from existing methods, we only optimize loss on item titles and find it effective in recommendation tasks. Refer to Appendix for more implementation details.
4.1 Overall Performance Comparison (RQ1)
Table I presents comparative analysis of the performance of our proposed S-DPO and baselines. Bold and underlined indicate the best and the second-bes. performance, respectively. We observe that:
LM-based recommenders have driven impressive performance breakthroughs compared with traditional recommenders. Our results reveal that traditional recommenders outperform untuned LM-based recommenders (LLaMA, ChatRec) but fall short compared to LM-based recommenders fine-tuned on historical interactions (TALLRec and LLaRA). It is noted that untuned LM-based recommenders are limited by inadequate instruction- followiwn capabilities (indicated by a low valid ratio) or lack of comain-specifee knowledge (indicated by a suboptimal performance), which highlights the necessity of the supervised fine-tuning stage to further ground the inherent ability of language models down to sequential recommendation tasks. Moreover, MoRec also exhibits suboptimal performance compared to its traditional variant because it leaves the reasoning ability of L.M untouched. The superior performance of recent L-M-baedd recommenders indicates the significant roles of knowledge and reasoning ability in language models for recommendation tasks semantically informative datasets, which highlights the potential LM-based recommenders.
Tailoring language models for recommendation task further boosts the performance of LM- based recommenders. For LM-based recommenders, the substantial performance gap between fine- tuned and untuned approaches emphasizes the importance of tailoring models for recommendations. TALLRec adapts LM for recommendation by supervised fine-tuning LM on historical interactions, surpassing traditional recommenders. Additionally, LLaRA consistently outperformed TALLRec across all datasets, suggesting that introducing collaborative signals through appropriate item representations is a viable direction for further adapting LM. However, existing L.- methods adapt LM from either item representation methods or corpus construction, leaving the adaptation of optimization objectives unexplored. Instead, S-DPO aligns the language model with multi negative
hhtps//www.gedrread..com

---

wb nasting aro
DPO 5-0PO
DPO 5-DPO
DPO-Los
Chosen Roward
5.0
0.24
8.22
4.
LasiFM
Goodreads
MovieLens
1000
200
400
600
1000
Datasets
Number of Training Steps
Number of Training Steps
(c) Study preferred item reward. Figure 2: Study on S-DPO. (2a) Ablation study of S-DPO compared with SFT and DPO on three datasets. (2b) Comparison of the trend of validation loss between DPO and S-DPO on LastFM. (2c) Comparison of the reward of preferred items between DPO and S-DPO on LastFM.
(a) Ablation study.
(b) Study of validation loss.
user preference data by extending DPO to include a softmax ranking loss, making it a more appropriate loss function for recommendation tasks.
S-DPO consistently outperforms all traditional recommenders and the state-of-the-art LM- based recommenders on all datasets. S-DPO shows an improvement ranging from 11.10% to 47.03% on Hit Ratio compared the second-best baseline. Building on a supervised fine- tuning stage, we observe a further improvement to the preference alignment stage, which explicitly instills ranking information into LM and utilizes preference data with multiple negative samples. Such superior performance suggests that explicitly tailoring LM for recommendation using user preference data the training objective level more effective than other By leveraging the inherent abilities of the LM and incorporating ranking information from user preference data, S-DPO effectively differentiates between preferred and less preferred items. Notably, the preference alignment stage hardly harms the inherent ability of LM, illustrated by high valid ratio.
recommenders.
4.2 Study on S-DPO
Ablation Study. To investigate the effect of explicit ranking optimization and multiple negative samples of S-DPO, compare it with the vanilla supervised fine-tuned model (w/o ranking), and a variant of S-DPO with only a single negative sample (w/o multi-neg). downgrading to pairwise DPO loss. The experimental results are reported in Figure 2a. We can observe that DPO can achieve an overall better performance compared to SFT, which underscores the effectiveness of instilling recommenders. With a more effective ranking gradient provided by multiple negative samples, S-DPO can further boost performance and achieve the best among all baseline methods and variants.
ranking relationships into existing
Study on the number of negative samples (RQ2). Benefiting from the utilization of multiple negative pairs in preference data, our S-DPO offers two empirically appealing properties compared to DPO: 1) S-DPO has more effective gradients facilitating the optimization; 2) S-DPO provides a better boost for rewards of preferred items compared to DPO. Figure 2b provides the comparison of validation loss between S-DPO and DPO, illustrating that the loss S-DPO decreases faster and more significantly. This observation demonstrates that multiple negative pairs provide larger and more meaningful gradients for model optimization, which is attributed to the inherent benefit of S-DPO to mine negative samples [38] (cf. Section 3.1).
On the other hand, we study the behavior of S-DPO which is illustrated in Figure 2c. We surprisingly find that S-DPO exhibits continually increasing rewards of preferred items that more significant and stable than DPO, which shows better effectiveness in distinguishing preferred items and a potential of mitigating data likelihood decline issues[52, 53].
To further verify the superiority of the multiple negative samples of S-DPO compared with DPO, we select the number of negative samples from [1, 3, 5, 8, 10, 15) to conduct experiments to explore the potential of the number of negative samples, with the results depicted in Figure 3a. can be observed that utilizing multiple negative samples allows the model to achieve better performance than with a single one. Furthermore, as the number of negative samples increases, the model's performance exhibits continual improvements. We attribute this success of S-DPO to more effective ranking
8

---

Table 2: Effectiveness comparison between DPO with single negative, variant of DPO with multiple negatives and S-DPO with the same number of negatives (we set K as 3 to get the performance in this table).
Datasets
LastFM
MovieLens
Goudreads
Complexity
Measure
HitRatio@1
ValidRatio
HitRatio@1
ValidRatio
HitRatio@1
ValidRatio
DPO-Inegative
0.6342
0,9972
0.4947
0.9684
0.6381
0,9900
((2C
negatiee
0,6413
0.9964
0.4947
0.9474
0.6628
0,9900
B(2KCx
S-DPO-Knegative
0.6477
0,9980
0,5263
0.9895
0,6661
0.9950
e((K + 1JCCM + 135.))
0.665
HRt Ratiod1
0.650
1.000
0.000
0.636
0.910
Hit Ratiog1
00.600
Ratio
5.55
0.576
0.460
0.630
0.550
8.64
0.335
0.360
0.640
0.500
0.750
0.455
Numbar Negative Samples
18 13
Value
B
(a) Study of negative samples.
(b) Study of B on Hit Ratio 1.
(c) Study of B on Valid Ratio.
Figure 3: Studies on values of [3 and negative samples numbers of S-DPO on LastFM. (3a) Perfor- mance comparisons with varying numbers of negative samples (3 = 1). (3b) Performance comparisons with varying values of 3 setting negative samples number as 3. (3c) Validity comparisons with varying values of / setting negative samples number as 3.
gradients provided by multiple negatives which can be connected to the superior performance of contrastive loss in self-supervised recommendations [38, 39, 54].
To validate the superiority of S-DPO over the DPO variant with multi-negativen we conduct effec- tiveness and efficiency comparisons. Table 2 demonstrates that introducing more negative samples benefits both DPO and S-DPO, and S-DPO achieves comparable performance with fewer training steps. We further analyze how S-DPO outperforms DPO in terms of computational efficiency. While the complexity of DPO for K negative samples (2KCMM.)) where Cm + denotes the base LLM's computational complexity and S, represents the size of the training data, S-DPO's 's complexity for the same number of negatives is reduced to O((K + 1) + 1)))). This efficiency gain can be expressed by scaling the complexity of DPO by the factor 2 + 2K
I + 20' + 2KCM highlighting
that S-DPO offers significant advantages, especially when working with a larger number of negative samples.
Study on values of 3 (RQ3). In S-DPO, .3 is a hyperparameter controlling the deviation of LM from the hase reference policy [34]. Typically, smaller value of / implies that the language model is more heavily influenced by the preference signals and vice versa. In this section, we select the value of ( from 10.1, 0.5, 1, 3, 5) explore the effect of 3 S-DPO. As indicated in Figure 3b through 3c, a higher 3 can achieve overall better performance in our task, while a lower / may overwhelm the model's learned knowledge from the supervised fine-tuning stage, evidenced by both low valid ratio and hit ratio. On the other hand, an excessively large 33 prevents the model from effectively learning ranking relationships, leading to suboptimal performance. In all our main experiments and studies, we set 8 as 1 to achieve a balance between ranking signals and inherent knowledge of language models.
5 Related Work
5.1 LM for Recommendation
Recent advancements in recommendation systems have increasingly incorporated Language Models (LMs) due to their extensive knowledge and robust reasoning abilities. This integration occurs primarily in two forms: LM-enhanced recommenders and LM-based recommenders. LM-enhanced recommenders utilize LM embedding as semantie representations to provide contrastive signals [55-58] or utilize LM as advanced feature extractors improving the representation of user and

---

item features [59-61]. However, these systems still rely on traditional recommenders for the final recommendation task, which leaves the reasoning ability of LM largely untouched.
On the other hand, LM-based recommenders directly employ LMs for making recommendations. Early works leverage LMs'' in-context learning capabilities for zero-shot or few-shot recommenda- tions, demonstrating significant potential [10-12, 15]. However, untuned LM-based recommenders are limited by inadequate instruction-followiii capabilities and lack of domain domain-specific knowledge. To bridge this gap, recent efforts in this category include supervised fine-tuning of LMs on the histori-
cal interactions enhance their performance in recommendation tasks [14, 17, 23, 24, 27]. More recently, researchers have discovered that exploring item representation methods in the finetuning phase may further boost LM's ability for recommendation [30]. This branch of works includes integrating collaborative signals [18, 19, 25, 62 62-64], adjusting numeric representations [22, 65, 66] or introducing additional item tokens [16, 26, 28, 29].
However, existing finetuned methods follow the training objective of language generation without any specific adjustments for personalized ranking. Different from them, S-DPÓ proposes explicitly optimize item ranking information on preference data.
5.2 Preference Alignment of Language Models
Reinforcement Learning from Human Feedback (RLHF) [31-33] is prevalent method of LMs to learn from human preferences. The RLHF pipeline comprises reward model training and rein- forcement learning (RL) optimization, the latter of which suffers instability and inefficiency. Direct Preference Optimization (DPO) [34] bypasses the brittle RL phase via a particular reward model parameterization and is thus simpler to implement while still keeping the performance of RLHF.
DPO proves to be effective in many scopes, like NLP [34, 67] and multimodal LMs [35, 68-70]. Besides, several variants have been proposed for further improvement of DPO. PO [71] is generalization of DPO loss and its representative IPO can better overcome the problem of overfitting. ODPO [36] treats preference pairs differently by stipulating that the likelihood gap of two responses should be greater than a corresponding offset value. KTO [72] utilizes prospect theory for preference alignment tasks. Other variants including GPO [73], f-DPO [74], RSO [75] also enhance or expand DPO in various aspects. Despite these contributions, the possibilities for leveraging and further adapting DPO for recommendation are still largely unexplored and few studies discuss extending
DPO to handle multi-negative scenarios.
Limitation
Despite effectiveness, there are several limitations not addressed in this paper. On the one hand, the number of negative samples is capped at 15 in our experiments. The potential of multiple negative samples hasn't been fully explored due to the limited time and computation resources. On the other hand, increasing the number negative examples inevitably results in higher training costs, phenomenon that becomes more pronounced as the number of negative examples grows in the context of language models.
Conclusion
In this work, we devised a principled Softmax-DPe (S-DPO) loss specially tailored for recommenders, utilizing multiple negatives in preference data to explicitly instill ranking information into LM. Empirically, S-DPO surpasses all baseline models including traditional and LM-based methods on three datasets in sequential recommendation tasks while successfully providing better rewards for preferred items compared to DPO. Grounded by theoretical proof, we bridge S-DPO with the softmax loss in self-supervised recommendations, underscoring the ranking performance of S-DPO and highlighting the critical roles of multiple negatives. Also, we theoretically find that S-DPO has an inherent benefit to mine hard negatives which provide larger and more effective gradients to model optimization, assuring its exceptional capabilities in recommendation tasks. We believe that S-DPO, as a generalization of DPO, provides valuable insights for future --sed recommenders and has the potential to benefit research fields other than recommender systems".
'The extension to broader impact will be detailed in Appendix D
10

---

Acknowledgments and Disclosure of Funding
This research is supported by the National Science and Technology Major Project (2023/D0121
the NExT Research Center, National Natural Science Foundation of China (92270114), This research also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
References
[1] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Trans. Inf. Syst., 2020.
[2] Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun. Sequential recommender systems: Challenges, progress and prospects. In IJCAI, 2019.
[3] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.18225, 2023.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[5] Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, and Tat-Seng Chua. Ali-agent: Assessing Iims' alignment with human values via agent-based evaluation. In NeurIPS, 2024.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurlPS, 2022.
[7] Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (Ilms). CoRR, 0t/p2337700244 2023.
[8] Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How can recommender systems benefit from large language models: A survey. CoRR, abs/2306.05817, 2023.
[9] Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu. Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. CoRR, abs/2305. 198660. 2023.
[10] Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. chatgpt a good recommender? A preliminary study. CoRR, abs/2304.10149, 2023.
[11] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In RecSys, 2023.
[12] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR, 2024.

---

[13] An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. CoRR, abs/2310.10108. 2023.
[14] Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework align large language model with recommendation. In RecSys, 2023.
[15] Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chat- rec: Towards interactive and explainable Ilms- anssseugeented recommender system. CoRR, abs/2303. 145244, 2023.
[16] Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. CoRR, abs/2311 1.01143, 2023.
[17] Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: large language model empowered recommendation approach. CoRR, abs/2305.07001, 2023.
[18] Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He. Large language model can interpret latent space of sequential recommender. CoRR, 0tos/2310.204877 2023.
[19] Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, and Xiang Wang. Llara: Aligning large language models with sequential recommenders. CoRR, abs/2312.024455 2023.
[20] Steffen Rendle. Item recommendation from implicit feedback. In Recommender Systems Handbook, pages 143-171. Springer US, 2022.
[21] Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Mingehen Cai, Wayne Xin Zhao, and Ji-Rong Wen. Prompting large language models for recommender systems: A comprehensive framework and empirical analysis. CoRR, abs/2401.044 .04997, 2024.
[22] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In RecSys, 2022.
[23] Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems. CoRR, abs/2205.080% 2022.
[24] Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. CoRR, 0ts/2308.111311,
[25] Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrating collaborative embeddings into large language models for recommendation. CoRR, abs/2310.1948s. 2023.
[26] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy. Recommender systems with generative retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NeurIPS, 2023.
[27] Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. ONCE: boosting content- content-based recommendation with both open- and closed-soures large language models. In Luz Angelica Caudillo-Mata, Silvio I Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM, 2024.
[28] Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi. Actions speak louder than words: Trillion- parameter sequential transducers for generative recommendations. CoRR, 1bs2220 2024.
12

---

[29] Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji- Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommenda- tion. CoRR, wtw///11..aeee 1.09049, 2023.
[30] Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recommendation foundation models. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel, editors, SIGIR-AP, 2023.
[31] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In NeurIPS, 2020.
[32] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshomn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and tine-tuned chat models. CoRR, abs/2307.00 0928888,9988. 2023.
[33]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[34] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, 2023.
[35] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct pref- erence optimization of video large multimodal models from language model reward. CoRR, abs/2404.01258, 2024.
[36] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with offset. CoRR, abs/2402. 10571, 2024.
[37] An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua. Empowering collabora- tive filtering with principled adversarial contrastive loss. In NeurIPS, 2023.
[38] Jiancan Wu, Xiang Wang. Fuli Feng. Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xic. Self-supervised graph learning for recommendation. In SIGIR, 2021.
[39] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.00 0037403744 2018
[40] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
[41] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series (Applied Statistics), 1975.
[42] R. Duncan Luce. Individual Choice Behavior: A Theoretical analysis. 1959.
[43] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Thiemm, BPR: bayesian personalized ranking from implicit feedback. CoRR,
2012.
13

---

[44] Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoe Viet Hung, and Hongzhi Yin. Xsimgel: Towards extremely simple graph contrastive learning for recommendation. CoRR, 0bs/2209.025444 2022.
[45] An Zhang. Wenchang Ma. Xiang Wang, and Tat-Seng Chua. Incorporating bias-aware margins into contrastive loss for collaborative filterin.. In NeurIPS, 2022.
[46] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. In ICLR, 2016.
[47] Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 2018.
[48] Wang-Cheng Kang and Julian J. McAuley. Self-attentive sequential recommendation. In ICDM,
2018.
[49] Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? ID- vs. modality-basee recommender models revisited. In SIGIR, 2023.
[50] Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 2016.
[51] Iván Cantador, Peter Brusilovsky, and Tsvi Kufiik, editors. Proceedings of the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems, HetRec '11 Chicago, Illinois, USA, October 27, 2011, 2011. ACM.
[52] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. CORR, abs/2402.13228, 2024.
[53] Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. CoRR, Abs/2402.05369) 2024.
[54] Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. Are graph augmentations necessary?: Simple graph contrastive learning for recommendation, In SIGIR, 2022.
[55] Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Limrec: Large language models with graph augmentation for recommendation. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM, 2024.
[56] Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. CoRR, abs/2310.1592 2023.
[57] Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In WWW, pages 1662-1171. ACM, 2023.
[58] Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. Language models encode collaborative signals in recommendation. arXiv preprint arXhv:2407.0544 2024.
[59] Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. Towards open- world recommendation with knowledge augmentation from large language models. CORR, abs/2306. 10933, 2023.
open-world
[60] Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong Hc, Liang Zhang, Fajie Yuan, Jun Zhou, and Linjian Mo. Breaking the length barrier: Lim-enhanced CTR prediction in long textual user behaviors. CoRR, abs/2403. 19347, 2024.
[61] Zheng Chen. PALR: personalization aware lims for recommendation. CoRR, abs/2305.00622 2023.
14

---

[62] Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang. Ctrl: Connect tabular and language model for ctr prediction. arXiv preprint arXiv:2306.02844 2023.
[63] Xiaoyu Kong, Jiancan Wu, An Zhang. Leheng Sheng, Hui Lin, Xiang Wang, and Xiangnan He. Customizing language models with instance-wise lora for sequential recommendation. In NeurlPS, 2024.
[64] Xiaohao Liu, Jie Wu, Zhulin Tao, Yunshan Ma, Yinwei Wei, and Tat-seng Chua. Harnessing large language models for multimodal product bundling. arXiv preprint 1711v 2024.
[65] Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. VIP5: towards multimodal foundation models for recommendation. In EMNLP (Findings), 2023.
[66] Xinyu Lin, Wenjie Wang, Yongqi Li, Fuli Feng, See-Kiong Ng. and Tat-Seng Chua. A multi- facet paradigm to bridge large language model and recommendation. CoRR, abs/2310.06491, 2023.
[67] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. CoRR, abs/2401.100200, 2024.
[68] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-awaal direct preference optimization. CoRR, abs/2311.166899 2023.
[69] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations large vision language models. In AAAI, 2024.
[70] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. CoRR, abs/2312.
2023.
[71] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS, 2024.
[72] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01306, 2024.
[73] Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Gener- alized preference optimization: A unified approach to offline alignment. CoRR, abs/2402.05749, 2024.
[74] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. CoRR, abs/2309. 16240,, 2023.
[75] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. CoRR, abs/2309.06667, 2023.
[76] OpenAl. GPT-4 technical report. CoRR, abs/2303.08774 2023.
[77] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CORR, abs/2302. 13971, 2023.
[78] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling. In ICML, 2023.
15

---

[79] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, 'even Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/23 10.06825, 2023.
16

---

A Mathematical Derivations
A.1 Deriving Preference Distribution
The Pl. model takes the form:
K Z.) = II
exp (r(Iu) Er()))
p'( 22. exx
(14)
sk exp(r (+(1))
The ranking in multi-negative preference data is ey > ed ed Vea Es. Our new preference distribution that estimates the probability of the ranking can be derived:
p" (ep > ed, V ca Tu; Cp. Ed)
Σ
p" (r Cp, Ea)
(1)=
exp (r(au: ((())
Σ
II
Σ exp (((wrer()))
rE(T"(r(())
exp(r(Zur
exp (r(Ix) R Li=j exp(r(em)++1)))
II
K j=n exp(r(Eur Ej))
P'EPer(ind(Ta)) f=1
exp(r(Zu::p)
Σ r'EPer(in((aa))
p'(T'lzun
f=1 exp(r ej))
exp(r(zuu) Ep))
Σx= exp (r)
wherein (nd( denotes the indices of titles in &n and Per(ind(Ea)) denotes the set of permutations
of index set (Ea). The third equation is because a permutation of {1, 21.2... ... _K} starting with
can be divided into the prefix and subsequent permutation of the rest indices, which is exactly Per(in((Ea)).
A.2
Connection Between DPO and S-DPO
When N = 2, the following equations hold:
Ls (T- Feet)
(F1)
log ma(ea(<u) 3 log ITee (epp(())))
me(epp<t))
- -E(susppee))
log a
log Σ exp
(re((e
GgEEa
Blog Te(ealwu) log To me(ep((iu) Tret (cp (xa)
--E(zusppea)
log -log exp
(N 2)
(ea((e)
Blog [e(ep)zu)
log To Te(ea(zu)) Feet (ea (T,)
--E(zueppea)
log
Feet (c, (..)
LDPO (TT: Tret).
Therefore, DPO is special case of S-DPO.
A.3 Deriving the Gradient of S-DPO Loss
Let e() = g(ea, Tu) = B log xe(calae) following form:
3 log mo(oglau), and the S-DPO loss takes the merr(e
CS-DPO(TO: Tret)
logo
-log Σ exp(((()eaa)
(15)
caEEa

---

The gradient of V(0; ea) can be formulated as:
Vo V(0; ea) (VV [o(eala.) Volg
(16)
Using properties of the sigmoid function that (x))) = o()) - d())) a(x)o(-<)) and thus
(logo o(w) ((x))((-))) = o(-z)) we have:
Va CsLD ;
Vologo
-log
Σ
exp(V((:ea))
log Σ exp(V((;ea)) Vo log Σ exp((((0;ea))
eaEEa
edEEa
(loga( = o(-x))
Denes, exp(V((:ea)) Vo ((O
o log Σ exp(V((:eea)
=
Dejees (xp(V((0,e))
edEEa
Vologmolepll Vo log (e(e
log exp(V((:ea))
Eejees exp V (0: e/d V ((0gea))
CyEEf
CrEEd
(Eq. (16))
Vo logooo (e Vo log Tn (ea (
-
log Σ (p.
Σ
Eejecn exp(g(e's ed, I.))
EdEla
edEEa
(Definition of V(0; cx)
a (oog Σ explo(es.Cp:zul) edEEd
Volog
= - oe (p))))))
log Volog To(eplau) - Σ
EDERa Eeges, explo(e4, Ed, Rw))
The last equation is because:
Σ eaEEa
1
Eeses, exp(V((:ea))
Eejees explgle,, ea, xo))
Edges, cxp(V
B Experimental Settings
B.1 Baselines
We compare the performance of S-DPO, against both traditional and LM-based baselines to showcase the effectiveness of our method. Specifically, for traditional methods, we have:
GRU4Rec [46] utilizes the GRU (Gated Recurrent Unit) architecture to model sequences, enabling effective prediction in recommendation tasks.
Caser [47] employs both horizontal and vertical convolutional operations to enhance the capture of high-order interactions within item sequences, improving recommendation accuracy.
SASRec [48] incorporates multi-head self-attention mechanism in its self-attentive sequential recommendation model, facilitating the modeling of intricate sequential data patterns.
For LM-enhanced method, we have:
MoRec [49] advances traditional recommendation systems by incorporating the modality features of items instead of the id feature. employ BERT for the text encoder and SASRec for the recommendation architecture.
For LM-based methods, we have:
LLaMA2 [32] utilized vanilla 1.LMM2-7B to directly generate recommendation results through direct prompting.
18

---

Table Statistics of datasets.
Dalaset
MovieLens
Goodreads
LaslFM
#Sequence
943
6,031
1,220
#Items
1,682
4,500
4,606
#lnteractions
100.000
220,100
73,510
Chat-REC [15] is implemented based on the framework discussed in [15], we retain user interaction sequences consisting of item titles as use profiles for a fair comparison. We use GPT4 [76] as its primary large language model.
TallRec [14] first propose to transform interaction sequences into textual prompts and then fine- tunes large language models using domain-specific corpus.
LLaRA [19] combines collaborative signals from traditional recommendation systems into the fine-tuning of large language models for improved recommendation performance.
B.2 Datasets
To evaluate the effectiveness of S-DPO, we conduct experiments on three widely used real-world datasets: Movielens [50], Goodreads", and LastFM [51]. The statistics of datasets are illustrated in Table 3. The MovieLens dataset is widely used for movie recommendation tasks and includes user ratings and movie titles, we select the MovieLens 100K dataset in our experiment. Similarly, Goodreads is sourced from social book cataloging website, where users can explore, rate, and review variety of books. LastEM dataset comprises users' listening history and artists' names from the Last. fm online music platform. Following [19], we maintain their titles as textual descriptions for each dataset. For Goodreads, we remove users and books with less than 20 interactions, which keeps the same as the processing MovieLens. For all datasets, we organize sequences chronologically before dividing the data into training, validation, and testing sets in an 8:1:1 ratio to prevent any potential information leakage.
B.3 Implementation Details
We implement all approaches with Python 3.9.7, PyTorch 2.2.2, and transformers 4.38.2 on 4 NVIDIA A100s. We select Llama2-7B [32] as the LM backbone for S-DPO. Following [19], we randomly select prompts from several prompt formats during training and evaluation to ensure flexibility and generality. For optimization of all the traditional methods, the Adam optimizer employed with learning rate adjusted to 0.001, and batch size configured at 256. All models undergo L2 regularization, with coefficients experimentally determined from [le-3, le-4, 1e-5, 1e-6, Ie-7j. In all experiments involving large language models, we train each method for maximum of 5 epochs using a batch size of 128 and select the checkpoint with the lowest loss on the validation set as final checkpoint. A warm-up strategy is applied to the learning rate. starting at 5% of its maximum value, and gradually adjusting it through a cosine scheduler throughout the training process. For S-DPO and all of its ablation studies, we further conduct preference training for further 3 epochs with a batch size of 128 and a learning rate le-5, Setting the value of 3 as 1, we search the number of negative samples in [3,5] for the main results. The effects of both factors further explored in 4.2.
B.4 Evaluation Metrics
Given that LMS primarily produce textual responses rather than comprehensive item rankings, we utilize a re-ranking metric in line with previous research [19] to assess recommendation performance. For each sequence, candidate set constructed by randomly selecting 20 non-interacted items and always includes the correct item. We assess all models based on their ability to pinpoint the correct item within this candidate set, employing the HitRatio@ (HR @ 1) metric for performance evaluation. Following [19], we also introduce an additional metric called the Valid Ratio to evaluate the LM- based methods' adherence to instructions and their ability to generate appropriate responses. Due to the difficulty LMs face in producing ranked results for candidate items, position-awe aware metrics like NDCG are deemed unsuitable for this evaluation.
$ "tttps//www.goodreas.oom
19

---

Table 4: The performance comparison among three different backbone language models on LastEM and Ens.eLens.
ILaMAI-7
Mistral-71
Pythia-2.88
HR@1
ValidRatio
HR
ValidRatio
HR@1
ValidRatio
LastFM
Vanilla
0.0465
0.0633
0.0265
0.3648
Language Modeling
0.5980
0.9880
0.7828
0,9992
0.1611
0.4281
DPO
0.6084
0.9976
0.7415
0.9964
0.1896
0.4220
S-DPO (3 negatives)
0.6285
0.9976
0.7679
0.9972
0.1948
0.4689
S-DPO (8 negatives)
0.6365
0.9988
0.7820
0.9972
0.2200
0.4685
MovieLens
Vanilla
0.0316
0.5158
0.0842
0.6737
0.0421
0.4421
Language Modeling
0.3895
0.9684
0.4211
0.9895
0.1053
0.5684
DPO
0.3789
0.9684
0.4421
0.9684
0.1271
0.8449
S-DPO (3 negatives)
0.9474
0.4421
0,1271
S-DPO negatives)
C Study on Backhone Language Models
In order to examine whether the superiority of S-DPO loss over traditional language modeling loss can be generalized across different backbone language models, we conducted experiments using models with varying architectures and sizes, including LLAMA1-7b [77]. Pythia-2.8h [78], and Mistral-7b [79]. These models were tested on two distinct datasets: LastFM and MovieLens. We evaluated three training approaches: language models fine-tuned with standard language modeling loss, and models further trained using DPO and S-DPO losses. Due to limitations in computational resources and time, we experimented with two variants of S-DPO: S-DPO (3 negatives) and S-DPO (8 negatives). As shown in Table 4, S-DPO consistently outperformed the language modeling loss, enhancing model performance while maintaining or even improving the validity of the generated outputs. Additionally, the performance continued improve as the number of negative samples increased from 3 to 8.
D Broader Impact
We left further exploration of softmax ranking loss LM including more negative samples and validation various settings as future works. We believe that S-DPO, generalization of DPO loss has the potential to benefit other research areas other than recommender systems. This paper presents work whose goal is advance the field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claim including both empirical results and theoretical findings. Refer Section for more details.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly discuss the limitations of work in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

We provide the full set of assumptions, together with complete and correct proofs of our theory in Section Section ?? and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide extensive implementation details in Section B.3 including package version, device, datasets and hyperparameter setting.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide our anonymized versions of data and codes with the link in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce ceroolicggiideee CodeeSbbm ssi aasionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the details about the experimental setting and datasets can be found in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We validate the p-value of experiment in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of computation resource needed to reproduce the experiment and also the number epochs we conduct in Section B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We strictly follow the NeurIPS Code of Ethics to conduct our research. Specifically, we have taken steps to ensure that our data collection methods are ethical, our experiments are conducted with responsibility, and all potential biases are thoroughly addressed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work presented in our paper aims to instill ranking information into LMs improve the performance of LM-based recommenders. We do not foresee any negative societal impacts stemming from the outcomes of our research. We further claim our broader impact in Appendix D.
Guidelines:

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks. Our language model only serves recom- mender from a given item list, which makes it impossible to misused or dual-use.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We incorporate three datasets in our experiments including Moviel.ens [50], Goodreads and LastFM [51], all of which are open-source. Also, the backend language model used in our research is LLAMA2-7B [32], which is also an open-source model. The details of our used assets can be found Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
hntps//www.geddreaas.ccom
25

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: We won't release any assets our research except necessary checkpoint and processed data in order to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
26

---

The answer NA means that the paper does not involve crowdsourcing or research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "SM9Urq5E5X", "reviewer": "Reviewer_gGah", "summary": "Inspired by advancements in Direct Preference Optimization (DPO) and the success of softmax loss in recommendations, the paper propose Softmax-DPO (S-DPO). S-DPO enhances LM-based sequential recommenders by incorporating multiple negative samples in preference data.The paper demonstrate the superiority of S-DPO in modeling user preferences and boosting recommendation effectiveness.", "questions": "Same as weaknesses\n\nIs there a justification for not conducting the experiment described above?", "limitations": "LM's methodology was brought to the Recommender system. However, there seems to be a lack of discussion in the Recommender system.\n\nBPR loss is typically not used in sequential scenarios, so it would have been beneficial to emphasize that S-DPO specifically addresses preferences in sequential contexts. Additionally, conducting more experiments to validate this aspect would have been advantageous.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. DPO, a method that considers preferences in NLP, has been extended to LM-based recommender systems, expanding NLP techniques to the recommender system domain.\n\n2. Building upon related previous work such as reinforcement learning from human feedback (RLHF) and DPO, the motivation and approach of this paper are well-founded.", "weaknesses": "In the recommender system domain, considering preferences in the loss function is not particularly new. However, experiments and explanations for this are lacking.\n\n1. There are no experiments comparing S-DPO with other preference-based losses in recommender systems (BPR, InfoNCE, as the paper mentioned ). If such comparisons are not feasible, the reasons for this are insufficiently explained.\n\n2. The experiments lacked the application of S-DPO loss to various base models. Consequently, the general applicability of S-DPO was not demonstrated.\n\n3. Running DPO multiple times and S-DPO appear conceptually similar. Therefore, there are no experiments comparing their efficiency. A comparison of time or memory usage seems necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**For Reviewer gGah**\n\nYour main suggestions about considering additional base models help us substantiate wide applicability of S-DPO.\n\n>**Comment1: Lack comparison with traditional preference-based losses.**\n\nThank you for your insightful question, which raises a profound and previously unexplored issue: whether traditional preference-based losses can be directly applied to LM-based recommenders.\n\nFollowing your suggestion, we attempted to train LM-based recommenders using BPR loss and InfoNCE loss. \nHowever, due to the significant differences between losses designed for language models (e.g., SFT, DPO) and discriminative preference-based losses, and given the time constraints of the rebuttal period, we have not yet obtained results within a reasonable range. We will continue to adjust parameters and training methods over the next discussion week to provide a more reliable conclusion.\nWe greatly appreciate your question, looking forward to more discussions with you.\n\nAdditionally, we want to emphasize that we have compared preference-based losses in traditional recommenders, such as GRU4Rec-BPR.\n\n\n\n> **Comment 2: General applicability of S-DPO** \n\nGreat point! Following your suggestion, we have **extended our experiments** to include more base models.\n\nWe selected language models with different architectures (**LLAMA1-7b, Pythia-2.8b, Mistral-7b**), on varying datasets (**LastFM, MovieLens**), and of different sizes to perform experiments. \nWe compared untuned LMs, LMs with only SFT as the training loss, and LMs with DPO and S-DPO for further training. \nDue to computational resource and time limitations, we experimented with S-DPO using **3 and 8** negative samples on two datasets. \n\n\n**Results on LastFM** \n\n**LLAMA1-7b**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| LLAMA1 | 0.0465 | 0.5872 |\n| SFT | 0.5980 | <u>0.9980<u> |\n| DPO | 0.6084 | 0.9976  | \n| S-DPO-3neg | <u>0.6285</u> | 0.9976  |\n| S-DPO-8neg | **0.6365** | **0.9988** |\n\n**Pythia-2.8b**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| Pythia | 0.0265 | 0.3648 |\n| SFT | 0.1611 | 0.4281 |\n| DPO | 0.1896 | 0.4220 |\n| S-DPO-3neg | <u>0.1948</u> | **0.4689** |\n| S-DPO-8neg | **0.2200** | <u>0.4685</u> |\n\n**Mistral-7b**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| Mistral | 0.0633 | 0.7475|\n| SFT | **0.7828** | **0.9992** |  |\n| DPO | 0.7415 | 0.9964 |  |\n| S-DPO-3neg | 0.7679 | <u>0.9972</u> |  |\n| S-DPO-8neg | <u>0.7820</u> | <u>0.9972</u> |  |\n\n**Results on MovieLens**\n\n**LLAMA1-7b**\n\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| LLAMA1 | 0.0316 | 0.5158 |\n| SFT | 0.3895 | **0.9684** |\n| DPO | 0.3789 | **0.9684** |\n| S-DPO-3neg | **0.4526**  | 0.9474 |\n| S-DPO-8neg | **0.4526** | 0.9579 |\n\n**Pythia-2.8b**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| Pythia | 0.0421 | 0.5895 |\n| SFT | 0.1053 | 0.5684 |\n| DPO | <u>0.1271</u> | <u>0.8449</u> |\n| S-DPO-8neg | **0.1474** | **0.8737** |\n\n**Mistral-7b**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| Mistral | 0.0842 | 0.6737 |\n| SFT | 0.4211 | 0.9894 |\n| DPO | <u>0.4421</u> | 0.9684 |\n| S-DPO-3neg | <u>0.4421</u> | **0.9895** |\n| S-DPO-8neg | **0.4947** | **0.9895** |\n\nThe experimental results indicate that DPO generally enhances the performance of SFT across different language models, demonstrating the importance of preference information for recommendation tasks. \nBy incorporating multiple negative examples, S-DPO can achieve further performance improvements on top of DPO. \nMoreover, as the number of negative examples increases, the model's performance also improves. \n\n\n\n\n\n>**Comment 3: Comparison between DPO and S-DPO.**\n\nWe appreciate your valuable comment. \nTo address this, we have **added effectiveness and efficiency comparisons between DPO and S-DPO**. \n\n$K$: the number of negative items.\n\n$C_\\mathcal{M}$: the complexity of base model $\\mathcal{M}$. \n\n**Time Complexity**\n| Methods | S-DPO | DPO |\n|:------:|:------:|:------:|\n| Complexity | $O((K+1)(C_\\mathcal{M}+1))$ | $O(2K C_\\mathcal{M} S\\_t)$ |\n\n The complexity of S-DPO scales with the factor $\\frac{1}{2}+\\frac{1}{2K}+\\frac{1}{2C_\\mathcal{M}}+\\frac{1}{2KC_\\mathcal{M}}$ compared to DPO. Since $C_\\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means more efficiency S-DPO will posses compared with DPO.\n\nEmpirically, When both considering 3 negative examples and trained on goodreads dataset with 4 A100s, it will take 25h for S-DPO and 53h for DPO, indicating the efficiency of our method.\n\nAdditionally, we conducted experiments on three datasets compare DPO with multiple negatives and S-DPO on LLAMA2-7b. \n\n**LastFM**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| DPO | 0.6342 | 0.9972 |\n| DPO-3neg | 0.6413 | 0.9964 |\n| S-DPO-3neg | **0.6477** | **0.9980** |\n\n\n**MovieLens**\n| Methods | HR@1 | ValidRatio | \n|:------:|:-------:|:------:|\n| DPO | 0.4947 | 0.9684 |\n| DPO-3neg | 0.4947 | 0.9474 |\n| S-DPO-3neg | **0.5263** | **0.9895** |\n\n**Goodreads**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| DPO | 0.6381 | 0.9900 |\n| DPO-3neg | **0.6661** | 0.9900 |\n| S-DPO-3neg | 0.6628 | **0.9950** |\n\nThe results show that S-DPO still achieves leading or comparable performance with less complexity. This can be attributed to S-DPO considering more negative examples during single gradient update, resulting in more effective gradients compared to DPO. \n\n> **Limitation 1: Lack of discussion in the recommender system.** \n\nWe are not sure we fully understand your point. We discussed the LM-based recommender and traditional recommender in related works.\n\n> **Limitation 2: conducting more experiments and emphasizing S-DPO specifically addresses preferences in sequential recommendation**\n\nThanks for your valuable comments! Following your suggestion, we added more experiments and will emphasize the important role of S-DPO in sequential recommendation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ziaV5rJTkO", "reviewer": "Reviewer_iyxD", "summary": "The authors propose a modification to the Direct Preference Optimization (DPO) by incorporating a softmax loss to enhance the training of language model (LM)-based recommender systems.", "questions": "- Consider replacing \"side effect\" with a more positive synonym.\n- The methodology section (2) appears empty; clarify if the true (new) methodology starts at section 3.1.\n- Equation (1) needs clarification as i_p and i_d represent items (identifiers) and not scores nor ranks. \n- Space before subsection name in line 148\n- Line 159: \"mutli\" --> write \"multi\"\n- Clarify lines 174-180 regarding \"hard negative items\" and the theoretical basis for S-DPO's ability to mine hard negatives.\n- Section 3.2 is unclear, especially the statement \"The structural correlation indicates that DPO and S-DPO are more suitable for recommendation than the language-modeling loss.\"\n- The definition of Rel.Ipv should be moved to the caption of Table 1.", "limitations": "- The computational effort required for using more negative samples is not considered. It would be beneficial to compute an approximate empirical complexity of the method.\n- The use of the Movielens 100K dataset is a limitation due to its small size. A larger dataset like ML-1M would provide better validation while remaining computationally feasible compared to ML-20M.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper is well-written.\n- The mathematical formulations are clear and appreciated.\n- The proposed method is fairly simple yet effective.\n- The results are convincing.", "weaknesses": "1) Novelty\n- The use of multiple negatives in recommendation systems is not new and should be cited. Thus, Research Question 2 (RQ2) lacks novelty as it has already been explored. The novelty of using multiple negatives in LM-based recommender systems is uncertain.\n\n2) Statements\n- Equation 8 represents the softmax loss, which has already been applied in other recommender systems like BERT4Rec, whereas Equation 12 does not represent the softmax loss, leading to confusion.\n- The claim in lines 225-226 \"indicating the significant roles of knowledge and reasoning ability in language models for recommendation tasks.\" is not provable due to differences in training datasets.\n- The statement \"However, indicated by the low valid ratio and suboptimal performance, untuned LM-based recommenders are limited by inadequate instruction-following capabilities or a lack of domain-specific knowledge\" in lines 228-230 is incorrect as evidenced by high ValidRatio scores of ChatRec.\n\n3) Replicability\n- The prompts used in the experiments are not provided, making replication difficult.\n\n\n4) Evaluation\n- NDCG is usually preferred, and metrics like @10 or @20 are more commonly used than @1 in recommendation evaluations. For example, LM-based RecSys works like LLama4Rec use @5.\n- In Figure 2a, the statistical significance of the results is not specified. It is also unclear why the results are based on LLama and not LLama2.\n\n5) Comparison\n- The comparison of training epochs between LM-based recommenders (5 epochs) and S-DPO (an additional 3 epochs) is unfair.\n- In Table 1, the best HR@1 is highlighted in bold, but the same should be done for the ValidRatio column even if your model does not perform the best in that aspect.\n- The statistical test used should be specified.\n- Optimizing hyperparameters for the proposed model without doing the same for competitor models is unfair. A fair comparison would involve equal time allocation for hyperparameter searches.\n- Figure 2b shows that S-DPO decreases faster but remains higher than DPO, making the comparison questionable. Similarly, Figure 2c lacks clarity on whether the values are comparable.\n-The statement \"On the other hand, an excessively large β prevents the model from effectively learning ranking relationships, leading to suboptimal performance\" is not supported by Figure 3b and 3c, which show consistent performance improvement.\n\n\n6) Related work\n- The related work section is more effective when placed after the introduction to better contextualize the state-of-the-art before presenting the new method.\n- The discussion on LM for recommendation is not thorough, with many works cited together without detailed descriptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "90sRM8n6mB", "reviewer": "Reviewer_AMu8", "summary": "proposed a softmax DPO in recommendation domain.", "questions": "1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.\n\n2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?", "limitations": "It is unclear how this framework can scale up to large scale dataset.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper mainly focus on softmax DPO for recommendation\n2. The paper is well wrriten with good experimental validation.\n3. The source code is avaliable.", "weaknesses": "1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.\n\n2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**For Reviewer AMu8**\n\nThanks for your time and feedback. To address your concerns, we present the point-to-point responses as follows. Looking forward to more discussions with you.\n\n>**Comment 1: Uncertain Novelty.**\n\nGenerative LM-based recommenders has recently been explored which aim to directly generate the preferred items by first encoding textual prompt then autoregressively decoding, which is different from traditional recommenders which calculating similarity among user and item numerical representations. With such differences, the training losses between LM-based recommenders and conventional recommenders have connections but they are strictly under different settings.\n\nFor LM-based recommenders, introducing ranking information and multiple-negatives has been largely omitted. Besides, there lacks effective methods to introduce multiple negatives into the training pipeline of LM-based recommenders. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.\n\nAdapting DPO to recommendation is a non-trivial task. Conventional DPO utilizes the Bradley-Terry (BT) or the Plackett-Luce (PL) preference model and focus on absolute ranking relationships among data samples, while we also utilize the PL preference model but focus on partial ranking relationships specifically for recommendation data and derive a new preference distribution from the PL model.\nMoreover, we also provide theorectical analysis to connect pairwise DPO with BPR loss and connect S-DPO with softmax loss in recommendation such as InfoNCE. Also, we further analyze the gradient of S-DPO loss and theorectically prove that mining hard negatives is one of the reasons behind S-DPO to provide effective gradient.\n\n>**Comment 2: Uncertain Complexity.**\n\nTo address your concern, we have now included analysis  of efficiency comparison between traditional DPO and our S-DPO, focusing on theoretical analysis  and empirical results . This comparison highlights the efficiency gains achieved by S-DPO, supporting its practical value.\n\nLet $K$ denote the number of negative items taken into consideration by S-DPO and $C_\\mathcal{M}$ denote the time complexity of the chosen base model $\\mathcal{M}$. The time complexity of computing S-DPO loss is $\\Theta((K+1)C_\\mathcal{M}+K+1)=\\Theta((K+1)(C_\\mathcal{M}+1))$. \n\nFor the DPO loss, the complexity of computing a single positive-negative pair is $\\Theta(2C_\\mathcal{M})$. \n\nBesides, when an equal number of negative items are added to the DPO training set, the total number of the DPO training set is $K$ times of that of the S-DPO training set. Therefore, let $S_t$ denote the size of the S-DPO training set. The time complexity of training with DPO loss scale to $\\Theta(2C_\\mathcal{M}\\times KS_t)=\\Theta(2KC_\\mathcal{M}S_t)$.  Although this complexity is in the same magnitude as the time complexity of training with S-DPO loss, the latter scales the complexity of training with DPO loss with the factor $\\frac{1}{2}+\\frac{1}{2K}+\\frac{1}{2C_\\mathcal{M}}+\\frac{1}{2KC_\\mathcal{M}}$.  Since $C\\_\\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means the more efficiency S-DPO will posses compared with DPO.\n\n**Time Complexity**\n\n| Methods | DPO-1neg | S-DPO | DPO |\n|:------:|:-------:|:------:|:------:|\n| Complexity | $\\Theta(2C_\\mathcal{M}S_t)$ | $\\Theta((K+1)(C_\\mathcal{M}+1))$ | $\\Theta(2KC\\_\\mathcal{M}S\\_t)$ |\n\n> **Comment 3: Large scale Dataset**\n\nWe have used large scale benchmark dataset such as goodreads. For extremely large datasets, it might be a common issue for researchers having difficulties to implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zvboCDIv36", "reviewer": "Reviewer_6tmb", "summary": "This paper extends DPO from pairwise (Bradley-Terry) to multi-way comparison (Plackett-Luce), where one positive example is considered better than multiple negative examples. The paper applies this approach in recommendation systems, showing promising results on several standard benchmarks.\n\nFrom Rafailov et al: “although more general Plackett-Luce ranking models [ 30, 21] are also compatible with the\nframework if we have access to several ranked answers”.\nThis paper shows the derivation of the Plackett-Luce extension. In that sense the technical contribution is somewhat incremental.\nI would therefore view this more as an application paper in recommendation systems. However this is not how it is written, most of the focus is on the theoretical analysis extending DPO from Bradley-Terry to Plackett-Luce.", "questions": "1. It is assumed that there is a single positive example which is better than all the negative ones. In reality, multiple items may satisfy users’ interests, so the negative examples should be chosen with care. In the experiments, how are negative samples chosen? This was not clear from the paper. \\\nActually, in recommendations it is natural to show a slate of results and the user would click on one (or more), so the unclicked results can serve as negatives. Though this is not the type of data available in benchmarks used here such as MovieLens.\n\n2. In Fig 2a, it seems like most of the gains in performance already exist in SFT. How come this is much better than other fine-tuned models like LLaRA?\n\n3. Fig 2b: the performance of S-DPO seems to still be improving after 1200 iterations, why not run it longer until it stops improving?\nAlso, since the S-DPO loss and DPO loss are not directly comparable, you should include a figure showing HR@1 as a function of training steps for S-DPO and DPO.\n\n4. Fig 3a: the performance seems to still be increasing, is there a number of samples for which it saturates?\n\n5. MIssing baseline: compare the proposed approach to DPO with pairs consisting of the positive example and each negative example separately (use same negative example choice as S-DPO). This is more computationally expensive but interesting to see how it compares in terms of performance (and runtime).\nRelated to that, it would also be good to include a comparison of runtimes between S-DPO and DPO (perhaps show the above variant of Fig 2b in terms of runtime, not just of training steps).\n\nMinor:\n* L79: empty section\n* L132: deviate -> deviating\n* L175: the exp should be with the sum\n* L180: “faster<space>.”", "limitations": "No societal impact concerns.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is technically sound and mostly clearly written.\n* The experimental results look very promising.", "weaknesses": "* The technical contribution seems somewhat incremental.\n* Some points need clarification (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**For Reviewer 6tmb**\n\nWe appreciate your comments, some of which inspires us to greatly improve our paper. Below we provide the point-to-point responses to address your concerns and clarify the misunderstandings of our proposed method. If you have additional questions, we would be pleased to discuss them with you.\n\n>**Comment 1: Uncertain contribution**\n\nWe acknowledge your statement that original DPO has mentioned the utilization of preference models such as Bradley-Terry and Plackett-Luce to model strict ranking relationships among ranked data samples. We claim that S-DPO is a alternated version of DPO deriving a new preference model from the PL model, which is different from the original PL model and has a softmax structure. The the new preference model focuses on partial ranking relationships among preferred items and dispreferred items, which widely exist in recommendation data.\nSuch inherent different in data characteristics makes it nontrivial to properly adapt DPO in recommendation while introducing multiple negatives which is important for recommenders.\nBesides adapting softmax DPO to recommendation data, we also provide theorectical analysis to connect pairwise DPO with BPR loss and connect S-DPO with softmax loss in recommendation such as InfoNCE, ensuring its superior performance. Also, we further analyze the gradient of S-DPO loss and theorectically prove that mining hard negatives is the reason behind S-DPO to provide effective gradient. \n\nLM-based recommenders aim to directly generate the preferred items by first encoding textual prompt then autoregressively decoding, which is different from traditional recommenders which calculate similarity among user and item numerical representations. With such differences, the training losses between LM-based recommenders and conventional recommenders have connections but are strictly under different settings. So traditional loss functions like BPR and InfoNCE are unsuitable for LM-based recommenders.\n\nFor LM-based recommenders, introducing ranking information and multiple-negatives has been largely omitted. Besides, there lacks effective methods to introduce multiple negatives into the training pipeline of LM-based recommenders. To our knowledge, we are among the first to point out that multi-negatives is important in LM-based recommenders and propose to effectively instill partial ranking information into LM-based recommenders through an alternated softmax version of DPO.\n\n>**Comment 2: Negtive sampling.**\n\nYou are right that negative sampling is a critical area of exploration in recommender systems. In this work, negative samples were randomly selected. We acknowledge the importance of more sophisticated negative sampling strategies, such as popularity-based and similarity-based negative sampling, which we leave as future work.\nThe type of data you mentioned, where unclicked results serve as negatives, is indeed common in real-world industrial datasets but is challenging to obtain in benchmark datasets like MovieLens. Within our experimental setting, random sampling serves as a simple yet effective strategy.\n\n>**Comment 3: Gain of performance.**\n\nAs line 215-217 mentioned, we optimize loss only on item title and find it effective in recommendation data. For other LM-based baselines, we adopt their official implementation for a fair comparison.\n\n>**Comment 4: Saturation of negative samples.**\n\nWe believe that the performance will further improve after more training iterations but as mentioned in limitation section that our computation resourse is limited, so we run all experiments for 3 epochs for fair comparson. We also believe that introducing more negatives may bring more gain but we can only explore part of it due to the same limitation in our computation resources.\n\n>**Comment 5: Efficiency.**\n\nWe appreciate your point. To address this, we have now included effectiveness and efficiency comparisons between DPO and S-DPO. \n\n$K$ denote the number of negative items.\n\n$C_\\mathcal{M}$ denote the complexity of base model $\\mathcal{M}$. \n\n**Time Complexity**\n\n| Methods | S-DPO | DPO |\n|:------:|:------:|:------:|\n| Complexity | $\\Theta((K+1)(C_\\mathcal{M}+1))$ | $\\Theta(2KC\\_\\mathcal{M}S\\_t)$ |\n\n The complexity of S-DPO scales with the factor $\\frac{1}{2}+\\frac{1}{2K}+\\frac{1}{2C_\\mathcal{M}}+\\frac{1}{2KC_\\mathcal{M}}$ compared to DPO. Since $C\\_\\mathcal{M}$ is usually large for LLMs, the bigger $K$ is, the smaller the factor is, which means the more efficiency S-DPO will posses compared with DPO.\n\nEmpirically, When both considering 3 negative examples and trained on goodreads dataset with 4 A100s, it will take 25h for S-DPO and 53h for DPO, indicating the efficiency of our method.\n\nAdditionally, we conducted experiments on three datasets compare DPO with multiple negatives and S-DPO on LLAMA2-7b. \n\n**LastFM**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| DPO | 0.6342 | 0.9972 |\n| DPO-3neg | 0.6413 | 0.9964 |\n| S-DPO-3neg | **0.6477** | **0.9980** |\n\n\n**MovieLens**\n| Methods | HR@1 | ValidRatio | \n|:------:|:-------:|:------:|\n| DPO | 0.4947 | 0.9684 |\n| DPO-3neg | 0.4947 | 0.9474 |\n| S-DPO-3neg | **0.5263** | **0.9895** |\n\n**Goodreads**\n| Methods | HR@1 | ValidRatio |\n|:------:|:-------:|:------:|\n| DPO | 0.6381 | 0.9900 |\n| DPO-3neg | **0.6661** | 0.9900 |\n| S-DPO-3neg | 0.6628 | **0.9950** |\n\nThe results show that S-DPO still achieves leading or comparable performance with less complexity. This can be attributed to S-DPO considering more negative examples during single gradient update, resulting in more effective gradients compared to DPO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are delighted to see the contributions of our paper have been acknowledged by the majority of the Reviewers. Specifically, we appreciate the Reviewers' recognition of our clarity in presentation (iyxD, AMu8, 6tmb), well-founded theoretical analysis (gGah, iyxD, 6tmb) and effectiveness (iyxD, AMu8, 6tmb).

We appreciate all the reviewers for their valuable comments and suggestions. This helped improve our submission and better strength our claims. Taking into account suggestions of Reviewers, we have summarized the updates to the paper as follows:

- **Experiments on three language model backbones compared S-DPO with DPO and SFT.**  Following the suggestions of Reviewer gGah, we have incorporated three additional language model backbones to validate the generalization ability of S-DPO.
- **Comparison experiments of effectiveness and efficiency analysis of DPO and S-DPO under the same negative samples.** Addressing the concerns raised by Reviewers gGah, iyxD, AMu8 and 6tmb, we have conducted experiments to validate the effectiveness of S-DPO with better efficiency compared with DPO.
- **More detailed explanations.** In response to Reviewers gGah, iyxD, AMu8 and 6tmb, we provide detailed explanation to clarify some of our statements, address questions and provide more understanding of our method.
- **Details about S-DPO.** We have incorporated detailed discussion about S-DPO, including hyperparameters selection, data likelihood decline issue and gradient analysis, to address the concerns of Reviewers iyxD and 6tmb.

We have tried our best to address the main concerns raised by reviewers in limited time and we hope that these improvements will be taken into consideration. We also present the point-to-point responses for each reviewer below.

Meta-review: The paper extends DPO to Softmax-DPO for language model based recommendation for user preference alignment inspired by the success of softmax loss in recommendation. While reviewers expressed concerns on novelty, they appreciate the simplicity of the method and solid results, and agree the reasons to accept outweighs reasons to reject after rebuttal.
Decision: Accept (poster)