Title: On Softmax Direct Preference Optimization for Recommendation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Sequential Recommendation

Large Language Models
Abstract: Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (\textbf{S-DPO}) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO.

On Softmax Direct Preference Optimization for Recommendation

Abstract

Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO ( S-DPO ) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO .

1 Introduction

Recommender systems aim to predict personalized rankings based on user preference data, i.e., historical interactions such as purchases, clicks, and ratings [1, 2]. Recently, leveraging the extensive world knowledge and powerful reasoning abilities of language models (LMs) [3-6], LM-based recommenders have been broadly explored [7-9]. These recommenders convert historical interaction data into language prompts and either perform in-context learning or fine-tune LMs, demonstrating notable

advantages, including zero-shot and few-shot reasoning [10-13], enhanced generalization abilities [14, 15], and rich semantic understanding [16-19]. However, current LM-based recommenders typically utilize language modeling loss for personalized ranking objectives-predicting the next token-which significantly differs from the objective of modeling user preferences in recommendation tasks [20, 21].

We argue that the current objective of LM-based recommenders does not fully utilize preference data and is not optimized for personalized ranking tasks, thereby hindering recommendation performance. Most LM-based recommenders address recommendation tasks by leveraging specialized language prompts [14, 17, 22-24], incorporating collaborative signals as a new modality [18, 19, 25], or extending the vocabulary of LMs with item tokens [16, 26-30]. Typically, these recommenders pair each language prompt, including the user's historical interaction item lists, with a single positive item and then update LM parameters using language modeling loss [14, 19]. Despite being designed for recommendation tasks, these LM-based recommenders do not consider negative items and are not directly optimized for personalized rankings. Such a training paradigm fails to fully leverage user preference data and overlooks the role of negative items in recommendations, thereby impeding the alignment of LMs with user preferences.

Inspired by the success of using human-labeled data to align LMs with human preferences [31-33] and advancements in direct preference optimization (DPO) [34-36], we make progress on aligning LMs with recommendations by fine-tuning them to predict the next item in accordance with the user's preference. This preference alignment stage aims to instill ranking information into the LMs and help recommenders distinguish preferred items from negatives, rather than solely focus on positives.

Towards this end, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for recommendation, connected to softmax sampling strategies [20, 3739], which we call S-DPO . Specifically, we first devise supervised fine-tuning to inject domain knowledge and improve LM's ability to follow the instructions before preference alignment phase, following [14, 33]. In the preference alignment stage, instead of constructing solely positive pairs, we initially pair each language prompt with both positive and randomly sampled multiple negatives to build text-based preference data. Building upon these preference data, we extend conventional DPO with the Bradley-Terry preference model [34, 40] on pairwise data to the Plackett-Luce preference model [41, 42], which handles relative rankings among multiple samples. Furthermore, we generalize the traditional Plackett-Luce preference model, which is designed for full relative rankings, to accommodate partial rankings, a more natural fit for recommendation tasks.

Benefiting from the multiple negatives in preference data, S-DPO offers three appealing properties. On the one hand, S-DPO serves as the first specialized personalized ranking loss for LM-based recommenders, effectively utilizing multiple negatives and acknowledging the importance of preference data. Empirically, we demonstrate that it provides more effective ranking gradients and better rewards for preferred items compared with DPO ( cf. Section 4.2). On the other hand, we theoretically bridge the DPO loss with the pairwise BPR loss [43] over pairwise data and connect S-DPO with the softmax loss over negative sampling (also known as contrastive loss in self-supervised recommendations, which achieves state-of-the-art performance [37, 44, 45]). This connection naturally underscores the ranking performance of S-DPO and highlights the critical role of multiple negatives. Furthermore, gradient analysis demonstrates that S-DPO has an inherent benefit of mining hard negative examples similar to contrastive learning paradigm [38], which not only boosts the performance but also accelerates the training process ( cf. Section 3.1), assuring its exceptional capabilities in recommendation tasks.

Overall, our contributions can be concluded as follows:

We are among the first to point out that the widely used language modeling loss in LM-based recommendation is not designed for ranking tasks and fails to fully utilize user preference data, thereby hindering recommendation performance.

We propose S-DPO, an alternative version of DPO loss extended from the traditional Plackett-Luce preference model, incorporating multiple negatives to instill ranking information into LM and tailoring for LM-based recommenders.

We theoretically bridge S-DPO with the softmax loss over negative sampling to highlight the critical role of multiple negatives and find its inherent benefit of mining hard negatives, assuring its capabilities.

X:“Afterwatching

g [History Sequence], which movie do you think the person will choose next from [

[Item List]?"

SupervisedFine-Tuning

DirectPreferenceOptimization

Softmax-DPO

id1

Autoregressive

Pairwise

Softmax

Loss

Ranking Loss

RankingLoss

LM

LM

Father

God

Father

God

Strays

Father

God

Leo

Strays

Trolls

TargetItem

PairwisePreferenceData

Multi-NegativePreferenceData

2 Preliminary

In this section, we first formalize sequential recommendation as the task of aligning language models (LMs) with user preferences. Then, we discuss the general framework of current LM-based recommenders that utilizes language modeling loss to fine-tune LMs. Finally, we outline the training process widely used to align LMs with human preferences, including reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO).

Task Formulation. Given the historical interactions H u of one user u in chronological order, the goal of LM-based sequential recommender M θ , where θ represents trainable parameters, is to select the item i p preferred by user u from candidate set C = { i j } N j =1 , where N is the number of candidates. This task requires that item i p be preferred over the other candidate items, denoted by I d = C \{ i p } . This requirement explicitly defines a multi-negative preference understanding for LM-based recommenders, which can be formulated as follows:

wherein ≻ u stands for the preference of user u .

Fine-tuning LM-based recommenders. Current LM-based recommenders widely adopt supervised fine-tuning (SFT) [33] on recommendation-specific data to enhance their performance [7, 9]. Generally, this involves two steps: structuring recommendation data as text-based pairs and then fine-tuning LMs based on these pairs. In the first step, for user u , a recommendation task prompt x u encompasses the user's historical interactions H u , the candidate item set C , and a description of the sequential recommendation task. This prompt x u is paired with the title of the preferred item i p in the candidate set C , denoted as e p , to form the pair data ( x u , e p ) . In the second step, the ( x u , e p ) pairs are utilized to fine-tune the LM-based recommender M θ through language modeling loss. This loss, commonly used in SFT in language modeling tasks, implicitly treats the recommendation task as predicting the next token based on preceding tokens. Formally, the objective of optimizing the LM-based recommender M θ with pair data ( x u , e p ) can be formulated as:

where | e p | is the number of tokens in e p , ( e p ) t is the t -th token of e p and ( e p ) <t is the tokens preceding ( e p ) t .

However, recommendation tasks are essentially user preference alignment tasks, as formalized in the above task formulation, and differ from language modeling tasks that consider only positive responses. Such a gap necessitates further exploration into aligning LM-based recommenders with user preference, an area that has been underexplored.

RLHF pipeline and DPO. Recent studies in natural language processing (NLP) explore the use of human-labeled pairwise data as a reward signal to align LMs with human preferences, such as RLHF [33] and DPO [34]. Specifically, the RLHF [33] pipeline adds two additional phases after the SFT phase: reward model training and reinforcement learning (RL) optimization. After obtaining the SFT model π SFT , RLHF further optimizes it with pairwise preference data.

Inspired by the success of RLHF in NLP, we leverage RLHF to inject recommendation-specific user pairwise preference into LM-based recommenders. Let E = { e j } N j =1 denote the title set of candidate items, where e j denotes the title of item i j . Given two items i j , i k ∈ C , the user preference i j > u i k can be seamlessly transformed into a response preference, stipulating that e j is preferred over e k when presented with prompt x u , denoted as e j ≻ e k | x u . By sampling one dispreferred item i d from dispreferred candidate set I d , we can curate a preference dataset { ( e p , e d , x u ) } .

After that, RLHF utilizes a preference model for preference distribution modeling, such as BradleyTerry (BT) model [40]. This preference model assumes there is a latent function r ( x u , e j ) representing the reward of prompt-response pair ( x u , e j ) . The bigger reward r ( x u , e j ) means the more user u prefers item i . From this perspective, reward function r ( x u , e j ) serves as a scoring function that quantifies the preference of user u to item i . Besides, the preference model defines a mapping from the reward function r ( x u , e j ) to a preference distribution p r ( e j ≻ e k | x u ) . Based on preference distribution, an optimal reward function is trained by maximizing the likelihood of preference data. The training objective of this phase is as follows:

Let π θ ( e | x u ) be the probability that LM-based recommender M θ output title e given prompt x u . The final reinforcement learning phase aims to maximize the expected reward of policy while not deviate too far from the reference model, formulating the following objective for optimal policy:

where D denotes the distribution of x u and π ref = π SFT .

A recent study, DPO [34], theoretically proves the optimal policy in a closed form to Eq.(4) is

where Z ( x u ) = ∑ e π ref ( e | x u )exp ( 1 β r ( x u , e ) ) is the partition function.

By defining p r ( e p ≻ e d | x u ) as σ ( r ( x u , e p ) -r ( x u , e d )) in Eq.(3) according to the BT model used in RLHF and substituting term r ( x u , e ) in Eq.(3) with Eq.(6), the last two phases of RLHF pipeline can be equivalently transformed into optimizing DPO loss below:

wherein σ ( x ) is the sigmoid function.

DPO is able to directly extract the optimal policy from pairwise preference data, making it more practical for preference alignment than RLHF. Nevertheless, DPO and RLHF are usually designed for pairwise preference. The oversight of other negative items impedes the performance of the LM-based recommenders. To bridge the gap, we expand DPO to S-DPO in recommendation tasks, in consideration of multiple negative items.

3 Methodology

3.1 Derivation of S-DPO loss

To align LM-based recommender M θ with multi-negative preference, we first derive the preference distribution and then propose a new loss function called S-DPO (depicted in Figure 1).

which is equivalent to

Multi-negative Preference Distribution. As mentioned in Section 2, for user u , there is a partial ranking stipulating i p ≻ u i d , ∀ i d ∈ I d in sequential recommendation tasks. Let E d be the titles of dispreferred items I d . The aforementioned partial ranking is equivalent to e p ≻ e d | x u , ∀ e d ∈ E d , from which a multi-negative preference dataset { x u , e p , E d } can be curated in an analogous way to RLHF.

For the dataset pairing one preferred item with multiple dispreferred items, we leverage the PlackettLuce (PL) model [41, 42] to build preference distribution. Given prompt x u , K titles e 1 , e 2 , · · · , e K and a permutation τ : [ K ] → [ K ] reflecting the user preference, with τ ( j ) denoting the j -th element of permutation τ , the PL model estimates that the ranking e τ (1) , e τ (2) , · · · , e τ ( K ) turns out true, as:

By enumerating all the permutations starting with p and calculating sum of their probability given by the PL model, the final multi-negative preference distribution p ∗ can be derived as:

For brevity, the complete derivation is delegated to Appendix A.1.

Deriving S-DPO. By substituting reward function r ( x u , e ) in Eq.(9) with Eq.(6), the multi-negative preference distribution can be rewritten as:

Through plugging distribution given by Eq.(10) in the reward learning objective in Eq.(3), our S-DPO loss can be formulated for policy π θ as:

Notably, when the number of candidates N is 2, which means there is only one dispreferred item, S-DPO reduces to DPO. The proof is provided in Appendix A.2.

Gradient Analysis. We conduct gradient analysis on S-DPO. The gradient of L S -DPO with respect to parameters θ takes the following formulation:

wherein g ( e j , e k , x u ) = r θ ( x u , e j ) -r θ ( x u , e k ) and similar to DPO, r θ ( x u , e ) = β log π θ ( e | x u ) π ref ( e | x u ) is the implicit reward function defined by π θ . See Appendix A.3 for a complete derivation.

Recap the DPO gradient below:

Similar to DPO, the gradient of S-DPO loss increases the likelihood of the preferred item and decreases the likelihood of all the dispreferred items. Each example is also weighed by how much the

implicit reward r ( x u , e ) deviates from the preference data. However, compared with DPO, S-DPO harnesses information of multiple dispreferred items in this weight.

Moreover, S-DPO treats gradients of different negative (dispreferred) items differently by assigning the gradient of each negative item with an extra weight 1 ∑ ϵ ′ d ∈E d exp( g ( e ′ d ,e d ,x u )) =

exp( r θ ( x u ,e d )) ∑ ϵ ′ d ∈E d exp( r θ ( x u ,e ′ d )) . This term reflects the relative reward of each negative item compared with other negative items. Similar to [38], we can categorize negative items into two categories: (1) Hard negative items, whose reward r θ ( x u , e d ) = β π θ ( e d | x u ) π ref ( e d | x u ) is relatively high, making it more probable to be chosen by LM-based recommenders; (2) Easy negative items, whose reward r θ ( x u , e d ) is relatively low, making it less likely to be output. For hard negative items, the extra weight term exp( r θ ( x u ,e d )) ∑ ϵ ′ d ∈E d exp( r θ ( x u ,e ′ d )) tends to be larger, leading to more decline for likelihood. This mechanism makes LM-based recommenders more discriminative and endows S-DPO with more effectiveness and stability than DPO.

3.2 Properties of S-DPO

In this section, we will discuss the structural correlation between DPO and BPR [43], together with S-DPO and softmax loss [39], which demonstrates the advantage of S-DPO over DPO and language modeling loss.

For user u , preferred item i p and one dispreferred i d ∈ I d , BPR loss takes the form:

wherein f ( u, i ) represents preference score of user u for item i .

Similarly, given dispreferred item set I d , the softmax loss takes the form:

Review the DPO loss in Eq.(7) and S-DPO loss in Eq.(11). Notably, term β log π θ ( e | x u ) π ref ( e | x u ) is the implicit reward function, denoted by r θ ( x u , e ) in Section 3.1. According to Section 2, r θ ( e, x u ) reflects the preference of user u to item i corresponding to title e . When the reference model has no knowledge about recommendation, i.e., when π ref ( e | x u ) is approximately a uniform distribution, term r θ ( x u , e ) = β log π θ ( e | x u ) π ref ( e | x u ) exactly reveals absolute preference. Hence, r θ ( x u , e ) possesses a similar function to f ( u, i ) .

From this perspective, DPO and S-DPO can be seen as special patterns of BPR and softmax loss, respectively. Given the effectiveness of BPR and InfoNCE loss in recommendation, we argue that sampled-based loss which explicitly compares preferred and dispreferred items such as DPO and S-DPO is more suitable for training LM-based recommenders than only utilizing language modeling loss. Moreover, as softmax loss works better than BPR loss in multi-negative scenarios [20], it can be inferred that S-DPO will be more tailored for multi-negative user preference alignment than DPO.

4 Experiments

In this section, we aim to answer the following research questions:

RQ1: How does S-DPO compare with traditional and LM-based sequential recommendation models on performance?

RQ2: How does the LM-based recommender benefit from the multiple negatives in S-DPO?

RQ3: What are the impacts of the essential parameters ( β ) on S-DPO?

Baselines. We thoroughly compare S-DPO with three categories of recommenders in sequential recommendations: traditional recommenders (GRU4Rec [46], Caser [47], SASRec [48]), LMenhanced recommenders (MoRec [49]) and LM-based recommenders (LLaMA2 [32], Chat-REC

[15], TALLRec [14], LLaRA [19]). See detailed introduction and comparison of baselines in Appendix B.

Datasets. We conduct extensive experiments on three real-world benchmark datasets which differ in size and domain (Movielens [50], Goodreads 3 , and LastFM [51]). Following standard settings of [15, 19], we employ a commonly used metric Hit Ratio@1 (HR@1) for performance evaluation and an additional metric Valid Ratio to evaluate the LM-based methods' ability to generate appropriate responses. See detailed introductions of datasets and evaluation metrics in Appendix B.

Implementation. We implement all LM-based recommenders on 4 NVIDIA A100 GPUs. For all LM-based recommenders, we conduct a supervised fine-tuning stage for a maximum of 5 epochs. For S-DPO and its variants, we conduct a preference alignment stage for a further 3 epochs. Different from existing methods, we only optimize loss on item titles and find it effective in recommendation tasks. Refer to Appendix B for more implementation details.

4.1 Overall Performance Comparison (RQ1)

Table 1 presents a comparative analysis of the performance of our proposed S-DPO and baselines. Bold and underlined indicate the best and the second-best performance, respectively. We observe that:

LM-based recommenders have driven impressive performance breakthroughs compared with traditional recommenders. Our results reveal that traditional recommenders outperform untuned LM-based recommenders (LLaMA, ChatRec) but fall short compared to LM-based recommenders fine-tuned on historical interactions (TALLRec and LLaRA). It is noted that untuned LM-based recommenders are limited by inadequate instruction-following capabilities (indicated by a low valid ratio) or a lack of domain-specific knowledge (indicated by a suboptimal performance), which highlights the necessity of the supervised fine-tuning stage to further ground the inherent ability of language models down to sequential recommendation tasks. Moreover, MoRec also exhibits suboptimal performance compared to its traditional variant because it leaves the reasoning ability of LM untouched. The superior performance of recent LM-based recommenders indicates the significant roles of knowledge and reasoning ability in language models for recommendation tasks in semantically informative datasets, which highlights the potential of LM-based recommenders.

Tailoring language models for recommendation task further boosts the performance of LMbased recommenders. For LM-based recommenders, the substantial performance gap between finetuned and untuned approaches emphasizes the importance of tailoring models for recommendations. TALLRec adapts LM for recommendation by supervised fine-tuning LM on historical interactions, surpassing traditional recommenders. Additionally, LLaRA consistently outperformed TALLRec across all datasets, suggesting that introducing collaborative signals through appropriate item representations is a viable direction for further adapting LM. However, existing LM-based methods adapt LM from either item representation methods or corpus construction, leaving the adaptation of optimization objectives unexplored. Instead, S-DPO aligns the language model with multi-negative

(a) Ablation study.

(b) Study of validation loss.

(c) Study of preferred item reward.

0.8

0.7

w/o ranking

w/omulti-neg

S-DPO

DPO-LOSS

0.28

S-DPO

DPO

Reward

6.0

S-DPO

DPO

HitRatio@1

0.6

0.5

0.26

5.5

0.4

Validation

Chosen

5.0

0.3

0.24

0.2

4.5

0.22

0.1

4.0

0.0

LastFM

Datasets

Goodreads

MovieLens

200

NumberofTrainingSteps

400

600

800

1000

0

200

NumberofTrainingSteps

400

600

800

1000

user preference data by extending DPO to include a softmax ranking loss, making it a more appropriate loss function for recommendation tasks.

S-DPO consistently outperforms all traditional recommenders and the state-of-the-art LMbased recommenders on all datasets. S-DPO shows an improvement ranging from 11.10% to 47.03% on Hit Ratio@1 compared to the second-best baseline. Building on a supervised finetuning stage, we observe a further improvement to the preference alignment stage, which explicitly instills ranking information into LM and utilizes preference data with multiple negative samples. Such superior performance suggests that explicitly tailoring LM for recommendation using user preference data at the training objective level is more effective than other LM-based recommenders. By leveraging the inherent abilities of the LM and incorporating ranking information from user preference data, S-DPO effectively differentiates between preferred and less preferred items. Notably, the preference alignment stage hardly harms the inherent ability of LM, illustrated by a high valid ratio.

4.2 Study on S-DPO

Ablation Study. To investigate the effect of explicit ranking optimization and multiple negative samples of S-DPO, we compare it with the vanilla supervised fine-tuned model (w/o ranking), and a variant of S-DPO with only a single negative sample (w/o multi-neg), downgrading to pairwise DPO loss. The experimental results are reported in Figure 2a. We can observe that DPO can achieve an overall better performance compared to SFT, which underscores the effectiveness of instilling ranking relationships into existing LM-based recommenders. With a more effective ranking gradient provided by multiple negative samples, S-DPO can further boost performance and achieve the best among all baseline methods and variants.

Study on the number of negative samples (RQ2). Benefiting from the utilization of multiple negative pairs in preference data, our S-DPO offers two empirically appealing properties compared to DPO: 1) S-DPO has more effective gradients facilitating the optimization; 2) S-DPO provides a better boost for rewards of preferred items compared to DPO. Figure 2b provides the comparison of validation loss between S-DPO and DPO, illustrating that the loss of S-DPO decreases faster and more significantly. This observation demonstrates that multiple negative pairs provide larger and more meaningful gradients for model optimization, which is attributed to the inherent benefit of S-DPO to mine negative samples [38] ( cf. Section 3.1).

On the other hand, we study the behavior of S-DPO which is illustrated in Figure 2c. We surprisingly find that S-DPO exhibits continually increasing rewards of preferred items that are more significant and stable than DPO, which shows better effectiveness in distinguishing preferred items and a potential of mitigating data likelihood decline issues[52, 53].

To further verify the superiority of the multiple negative samples of S-DPO compared with DPO, we select the number of negative samples from {1, 3, 5, 8, 10, 15} to conduct experiments to explore the potential of the number of negative samples, with the results depicted in Figure 3a. It can be observed that utilizing multiple negative samples allows the model to achieve better performance than with a single one. Furthermore, as the number of negative samples increases, the model's performance exhibits continual improvements. We attribute this success of S-DPO to more effective ranking

.

Datasets

LastFM

MovieLens

Goodreads

Complexity

Measure

HitRatio@1

ValidRatio

HitRatio@1

ValidRatio

HitRatio@1

ValidRatio

DPO-

1

negative

0.6342

0.9972

0.4947

0.9684

0.6381

0.9900

Θ(2

C

M

S

t

)

DPO-

K

negative

0.6413

0.9964

0.4947

0.9474

0.6628

0.9900

Θ(2

KC

M

S

t

)

S-DPO-

K

negative

0.6477

0.9980

0.5263

0.9895

0.6661

0.9950

Θ((

K

+1)(

C

M

+1)

S

t

)

(a) Study of negative samples.

(b) Study of

β

on Hit Ratio@1.

(c) Study of

β

0.665

Hit Ratio@1

0.650

1.000

0.625

0.660

0.950

Ratio@1

1

0.655

Ratic

?

1

0.600

0.575

Rat

0.900

0.650

R

0.550

Valic

0.850

Hit

0.645

0.525

0.800

0.500

0.640

0.475

HitRatio@1

0.750

2

4

6

8

10

12

14

0

1

2

3

4

5

0

1

2

3

4

5

NumberofNegativeSamples

Valueofβ

Valueofβ

gradients provided by multiple negatives which can be connected to the superior performance of contrastive loss in self-supervised recommendations [38, 39, 54].

To validate the superiority of S-DPO over the DPO variant with multi-negatives, we conduct effectiveness and efficiency comparisons. Table 2 demonstrates that introducing more negative samples benefits both DPO and S-DPO, and S-DPO achieves comparable performance with fewer training steps. We further analyze how S-DPO outperforms DPO in terms of computational efficiency. While the complexity of DPO for K negative samples is Θ(2 KC M S t ) , where C M +1 denotes the base LLM's computational complexity and S t represents the size of the training data, S-DPO's complexity for the same number of negatives is reduced to Θ(( K +1)( C M +1) S t ) . This efficiency gain can be expressed by scaling the complexity of DPO by the factor 1 2 + 1 2 K + 1 2 C M + 1 2 KC M , highlighting that S-DPO offers significant advantages, especially when working with a larger number of negative samples.

Study on values of β (RQ3). In S-DPO, β is a hyperparameter controlling the deviation of LM from the base reference policy [34]. Typically, a smaller value of β implies that the language model is more heavily influenced by the preference signals and vice versa. In this section, we select the value of β from {0.1, 0.5, 1, 3, 5} to explore the effect of β on S-DPO. As indicated in Figure 3b through 3c, a higher β can achieve overall better performance in our task, while a lower β may overwhelm the model's learned knowledge from the supervised fine-tuning stage, as evidenced by both low valid ratio and hit ratio. On the other hand, an excessively large β prevents the model from effectively learning ranking relationships, leading to suboptimal performance. In all our main experiments and studies, we set β as 1 to achieve a balance between ranking signals and inherent knowledge of language models.

5 Related Work

5.1 LMfor Recommendation

Recent advancements in recommendation systems have increasingly incorporated Language Models (LMs) due to their extensive knowledge and robust reasoning abilities. This integration occurs primarily in two forms: LM-enhanced recommenders and LM-based recommenders. LM-enhanced recommenders utilize LM embedding as semantic representations to provide contrastive signals [55-58] or utilize LM as advanced feature extractors improving the representation of user and

item features [59-61]. However, these systems still rely on traditional recommenders for the final recommendation task, which leaves the reasoning ability of LM largely untouched.

On the other hand, LM-based recommenders directly employ LMs for making recommendations. Early works leverage LMs' in-context learning capabilities for zero-shot or few-shot recommendations, demonstrating significant potential [10-12, 15]. However, untuned LM-based recommenders are limited by inadequate instruction-following capabilities and a lack of domain-specific knowledge. To bridge this gap, recent efforts in this category include supervised fine-tuning of LMs on the historical interactions to enhance their performance in recommendation tasks [14, 17, 23, 24, 27]. More recently, researchers have discovered that exploring item representation methods in the finetuning phase may further boost LM's ability for recommendation [30]. This branch of works includes integrating collaborative signals [18, 19, 25, 62-64], adjusting numeric representations [22, 65, 66] or introducing additional item tokens [16, 26, 28, 29].

However, existing finetuned methods follow the training objective of language generation without any specific adjustments for personalized ranking. Different from them, S-DPO proposes to explicitly optimize item ranking information on preference data.

5.2 Preference Alignment of Language Models

Reinforcement Learning from Human Feedback (RLHF) [31-33] is a prevalent method of LMs to learn from human preferences. The RLHF pipeline comprises reward model training and reinforcement learning (RL) optimization, the latter of which suffers instability and inefficiency. Direct Preference Optimization (DPO) [34] bypasses the brittle RL phase via a particular reward model parameterization and is thus simpler to implement while still keeping the performance of RLHF.

DPO proves to be effective in many scopes, like NLP [34, 67] and multimodal LMs [35, 68-70]. Besides, several variants have been proposed for further improvement of DPO. Ψ PO [71] is a generalization of DPO loss and its representative IPO can better overcome the problem of overfitting. ODPO [36] treats preference pairs differently by stipulating that the likelihood gap of two responses should be greater than a corresponding offset value. KTO [72] utilizes prospect theory for preference alignment tasks. Other variants including GPO [73], f -DPO [74], RSO [75] also enhance or expand DPO in various aspects. Despite these contributions, the possibilities for leveraging and further adapting DPO for recommendation are still largely unexplored and few studies discuss extending DPO to handle multi-negative scenarios.

6 Limitation

Despite effectiveness, there are several limitations not addressed in this paper. On the one hand, the number of negative samples is capped at 15 in our experiments. The potential of multiple negative samples hasn't been fully explored due to the limited time and computation resources. On the other hand, increasing the number of negative examples inevitably results in higher training costs, a phenomenon that becomes more pronounced as the number of negative examples grows in the context of language models.

7 Conclusion

In this work, we devised a principled Softmax-DPO (S-DPO) loss specially tailored for LM-based recommenders, utilizing multiple negatives in preference data to explicitly instill ranking information into LM. Empirically, S-DPO surpasses all baseline models including traditional and LM-based methods on three datasets in sequential recommendation tasks while successfully providing better rewards for preferred items compared to DPO. Grounded by theoretical proof, we bridge S-DPO with the softmax loss in self-supervised recommendations, underscoring the ranking performance of S-DPO and highlighting the critical roles of multiple negatives. Also, we theoretically find that S-DPO has an inherent benefit to mine hard negatives which provide larger and more effective gradients to model optimization, assuring its exceptional capabilities in recommendation tasks. We believe that S-DPO, as a generalization of DPO, provides valuable insights for future LM-based recommenders and has the potential to benefit research fields other than recommender systems 4 .

Acknowledgments and Disclosure of Funding

This research is supported by the National Science and Technology Major Project (2023ZD0121102), the NExT Research Center, National Natural Science Foundation of China (92270114). This research is also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.

References

Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Trans. Inf. Syst. , 2020.

Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun. Sequential recommender systems: Challenges, progress and prospects. In IJCAI , 2019.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR , abs/2303.18223, 2023.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, and Tat-Seng Chua. Ali-agent: Assessing llms' alignment with human values via agent-based evaluation. In NeurIPS , 2024.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurIPS , 2022.

Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. Recommender systems in the era of large language models (llms). CoRR , abs/2307.02046, 2023.

Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. How can recommender systems benefit from large language models: A survey. CoRR , abs/2306.05817, 2023.

Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models for recommendation. CoRR , abs/2305.19860, 2023.

Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. Is chatgpt a good recommender? A preliminary study. CoRR , abs/2304.10149, 2023.

Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. Uncovering chatgpt's capabilities in recommender systems. In RecSys , 2023.

Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In ECIR , 2024.

An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua. On generative agents in recommendation. CoRR , abs/2310.10108, 2023.

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In RecSys , 2023.

Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. Chatrec: Towards interactive and explainable llms-augmented recommender system. CoRR , abs/2303.14524, 2023.

Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li. Collaborative large language model for recommender systems. CoRR , abs/2311.01343, 2023.

Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. Recommendation as instruction following: A large language model empowered recommendation approach. CoRR , abs/2305.07001, 2023.

Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He. Large language model can interpret latent space of sequential recommender. CoRR , abs/2310.20487, 2023.

Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, and Xiang Wang. Llara: Aligning large language models with sequential recommenders. CoRR , abs/2312.02445, 2023.

Steffen Rendle. Item recommendation from implicit feedback. In Recommender Systems Handbook , pages 143-171. Springer US, 2022.

Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Mingchen Cai, Wayne Xin Zhao, and Ji-Rong Wen. Prompting large language models for recommender systems: A comprehensive framework and empirical analysis. CoRR , abs/2401.04997, 2024.

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5). In RecSys , 2022.

Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language models are open-ended recommender systems. CoRR , abs/2205.08084, 2022.

Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation. CoRR , abs/2308.11131, 2023.

Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. Collm: Integrating collaborative embeddings into large language models for recommendation. CoRR , abs/2310.19488, 2023.

Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy. Recommender systems with generative retrieval. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, NeurIPS , 2023.

Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu. ONCE: boosting content-based recommendation with both open- and closed-source large language models. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM , 2024.

Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi. Actions speak louder than words: Trillionparameter sequential transducers for generative recommendations. CoRR , abs/2402.17152, 2024.

Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen. Adapting large language models by integrating collaborative semantics for recommendation. CoRR , abs/2311.09049, 2023.

Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang. How to index item ids for recommendation foundation models. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel, editors, SIGIR-AP , 2023.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In NeurIPS , 2020.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS , 2022.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS , 2023.

Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimization of video large multimodal models from language model reward. CoRR , abs/2404.01258, 2024.

Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. CoRR , abs/2402.10571, 2024.

An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua. Empowering collaborative filtering with principled adversarial contrastive loss. In NeurIPS , 2023.

Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. Self-supervised graph learning for recommendation. In SIGIR , 2021.

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR , abs/1807.03748, 2018.

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 1952.

R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 1975.

R. Duncan Luce. Individual Choice Behavior: A Theoretical analysis . 1959.

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. BPR: bayesian personalized ranking from implicit feedback. CoRR , abs/1205.2618, 2012.

Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoc Viet Hung, and Hongzhi Yin. Xsimgcl: Towards extremely simple graph contrastive learning for recommendation. CoRR , abs/2209.02544, 2022.

An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua. Incorporating bias-aware margins into contrastive loss for collaborative filtering. In NeurIPS , 2022.

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. In ICLR , 2016.

Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM , 2018.

Wang-Cheng Kang and Julian J. McAuley. Self-attentive sequential recommendation. In ICDM , 2018.

Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. Where to go next for recommender systems? ID- vs. modality-based recommender models revisited. In SIGIR , 2023.

F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst. , 2016.

Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik, editors. Proceedings of the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems, HetRec '11, Chicago, Illinois, USA, October 27, 2011 , 2011. ACM.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. CoRR , abs/2402.13228, 2024.

Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. CoRR , abs/2402.05369, 2024.

Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen. Are graph augmentations necessary?: Simple graph contrastive learning for recommendation. In SIGIR , 2022.

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Llmrec: Large language models with graph augmentation for recommendation. In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, WSDM , 2024.

Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. CoRR , abs/2310.15950, 2023.

Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao. Learning vector-quantized item representation for transferable sequential recommenders. In WWW , pages 1162-1171. ACM, 2023.

Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. Language models encode collaborative signals in recommendation. arXiv preprint arXiv:2407.05441 , 2024.

Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu. Towards open-world recommendation with knowledge augmentation from large language models. CoRR , abs/2306.10933, 2023.

Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong He, Liang Zhang, Fajie Yuan, Jun Zhou, and Linjian Mo. Breaking the length barrier: Llm-enhanced CTR prediction in long textual user behaviors. CoRR , abs/2403.19347, 2024.

Zheng Chen. PALR: personalization aware llms for recommendation. CoRR , abs/2305.07622, 2023.

Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang. Ctrl: Connect tabular and language model for ctr prediction. arXiv preprint arXiv:2306.02841 , 2023.

Xiaoyu Kong, Jiancan Wu, An Zhang, Leheng Sheng, Hui Lin, Xiang Wang, and Xiangnan He. Customizing language models with instance-wise lora for sequential recommendation. In NeurIPS , 2024.

Xiaohao Liu, Jie Wu, Zhulin Tao, Yunshan Ma, Yinwei Wei, and Tat-seng Chua. Harnessing large language models for multimodal product bundling. arXiv preprint arXiv:2407.11712 , 2024.

Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang. VIP5: towards multimodal foundation models for recommendation. In EMNLP (Findings) , 2023.

Xinyu Lin, Wenjie Wang, Yongqi Li, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. A multifacet paradigm to bridge large language model and recommendation. CoRR , abs/2310.06491, 2023.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. CoRR , abs/2401.10020, 2024.

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. CoRR , abs/2311.16839, 2023.

Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI , 2024.

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. CoRR , abs/2312.10665, 2023.

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In AISTATS , 2024.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: model alignment as prospect theoretic optimization. CoRR , abs/2402.01306, 2024.

Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. CoRR , abs/2402.05749, 2024.

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints. CoRR , abs/2309.16240, 2023.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. CoRR , abs/2309.06657, 2023.

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In ICML , 2023.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR , abs/2310.06825, 2023.

A Mathematical Derivations

A.1 Deriving Preference Distribution

The PL model takes the form:

.

The ranking in multi-negative preference data is e p ≻ e d | x u , ∀ e d ∈ E d . Our new preference distribution that estimates the probability of the ranking can be derived:

wherein ind( E d ) denotes the indices of titles in E d and Per(ind( E d )) denotes the set of permutations of index set ind( E d ) . The third equation is because a permutation of { 1 , 2 · · · , K } starting with p can be divided into the prefix p and a subsequent permutation of the rest indices, which is exactly Per(ind( E d )) .

A.2 Connection Between DPO and S-DPO

When N = 2 , the following equations hold:

Therefore, DPO is a special case of S-DPO.

A.3 Deriving the Gradient of S-DPO Loss

Let V ( θ ; e d ) = g ( e d , e p , x u ) = β log π θ ( e d | x u ) π ref ( e d | x u ) -β log π θ ( e p | x u ) π ref ( e p | x u ) and the S-DPO loss takes the following form:

The gradient of V ( θ ; e d ) can be formulated as:

Using properties of the sigmoid function that σ ′ ( x ) = σ ( x )(1 -σ ( x )) = σ ( x ) σ ( -x ) and thus ((log σ ( x )) ′ = 1 σ ( x ) × σ ( x ) σ ( -x ) = σ ( -x ) , we have:

∇

The last equation is because:

B Experimental Settings

B.1 Baselines

We compare the performance of S-DPO, against both traditional and LM-based baselines to showcase the effectiveness of our method. Specifically, for traditional methods, we have:

GRU4Rec [46] utilizes the GRU (Gated Recurrent Unit) architecture to model sequences, enabling effective prediction in recommendation tasks.

Caser [47] employs both horizontal and vertical convolutional operations to enhance the capture of high-order interactions within item sequences, improving recommendation accuracy.

SASRec [48] incorporates a multi-head self-attention mechanism in its self-attentive sequential recommendation model, facilitating the modeling of intricate sequential data patterns.

For LM-enhanced method, we have:

MoRec [49] advances traditional recommendation systems by incorporating the modality features of items instead of the id feature. we employ BERT for the text encoder and SASRec for the recommendation architecture.

For LM-based methods, we have:

LLaMA2 [32] utilized vanilla LLaMA2-7B to directly generate recommendation results through direct prompting.

Chat-REC [15] is implemented based on the framework discussed in [15], we retain user interaction sequences consisting of item titles as use profiles for a fair comparison. We use GPT4 [76] as its primary large language model.

TallRec [14] first propose to transform interaction sequences into textual prompts and then finetunes large language models using domain-specific corpus.

LLaRA [19] combines collaborative signals from traditional recommendation systems into the fine-tuning of large language models for improved recommendation performance.

B.2 Datasets

To evaluate the effectiveness of S-DPO, we conduct experiments on three widely used real-world datasets: Movielens [50], Goodreads 5 , and LastFM [51]. The statistics of datasets are illustrated in Table 3. The MovieLens dataset is widely used for movie recommendation tasks and includes user ratings and movie titles, we select the MovieLens100K dataset in our experiment. Similarly, Goodreads is sourced from a social book cataloging website, where users can explore, rate, and review a variety of books. LastFM dataset comprises users' listening history and artists' names from the Last.fm online music platform. Following [19], we maintain their titles as textual descriptions for each dataset. For Goodreads, we remove users and books with less than 20 interactions, which keeps the same as the processing of MovieLens. For all datasets, we organize sequences chronologically before dividing the data into training, validation, and testing sets in an 8:1:1 ratio to prevent any potential information leakage.

B.3 Implementation Details

We implement all approaches with Python 3.9.7, PyTorch 2.2.2, and transformers 4.38.2 on 4 NVIDIA A100s. We select Llama2-7B [32] as the LM backbone for S-DPO. Following [19], we randomly select prompts from several prompt formats during training and evaluation to ensure flexibility and generality. For optimization of all the traditional methods, the Adam optimizer is employed with a learning rate adjusted to 0.001, and a batch size configured at 256. All models undergo L2 regularization, with coefficients experimentally determined from [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]. In all experiments involving large language models, we train each method for a maximum of 5 epochs using a batch size of 128 and select the checkpoint with the lowest loss on the validation set as the final checkpoint. A warm-up strategy is applied to the learning rate, starting at 5% of its maximum value, and gradually adjusting it through a cosine scheduler throughout the training process. For S-DPO and all of its ablation studies, we further conduct preference training for further 3 epochs with a batch size of 128 and a learning rate of 1e-5. Setting the value of β as 1, we search the number of negative samples in [3,5] for the main results. The effects of both factors are further explored in 4.2.

B.4 Evaluation Metrics

Given that LMs primarily produce textual responses rather than comprehensive item rankings, we utilize a re-ranking metric in line with previous research [19] to assess recommendation performance. For each sequence, a candidate set is constructed by randomly selecting 20 non-interacted items and always includes the correct item. We assess all models based on their ability to pinpoint the correct item within this candidate set, employing the HitRatio@1 (HR@1) metric for performance evaluation. Following [19], we also introduce an additional metric called the Valid Ratio to evaluate the LM-based methods' adherence to instructions and their ability to generate appropriate responses. Due to the difficulty LMs face in producing ranked results for candidate items, position-aware metrics like NDCG are deemed unsuitable for this evaluation.

C Study on Backbone Language Models

In order to examine whether the superiority of S-DPO loss over traditional language modeling loss can be generalized across different backbone language models, we conducted experiments using models with varying architectures and sizes, including LLAMA1-7b [77], Pythia-2.8b [78], and Mistral-7b [79]. These models were tested on two distinct datasets: LastFM and MovieLens. We evaluated three training approaches: language models fine-tuned with standard language modeling loss, and models further trained using DPO and S-DPO losses. Due to limitations in computational resources and time, we experimented with two variants of S-DPO: S-DPO (3 negatives) and S-DPO (8 negatives). As shown in Table 4, S-DPO consistently outperformed the language modeling loss, enhancing model performance while maintaining or even improving the validity of the generated outputs. Additionally, the performance continued to improve as the number of negative samples increased from 3 to 8.

D Broader Impact

We left further exploration of softmax ranking loss in LM including more negative samples and validation on various settings as future works. We believe that S-DPO, a generalization of DPO loss has the potential to benefit other research areas other than recommender systems. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claim including both empirical results and theoretical findings. Refer to Section 1 for more details.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly discuss the limitations of our work in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions, together with complete and correct proofs of our theory in Section 2, Section ?? and Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide extensive implementation details in Section B.3 including package version, device, datasets and hyperparameter setting.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our anonymized versions of data and codes with the link in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the details about the experimental setting and datasets can be found in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We validate the p-value of experiment in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of computation resource needed to reproduce the experiment and also the number of epochs we conduct in Section B.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We strictly follow the NeurIPS Code of Ethics to conduct our research. Specifically, we have taken steps to ensure that our data collection methods are ethical, our experiments are conducted with responsibility, and all potential biases are thoroughly addressed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work presented in our paper aims to instill ranking information into LMs to improve the performance of LM-based recommenders. We do not foresee any negative societal impacts stemming from the outcomes of our research. We further claim our broader impact in Appendix D.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks. Our language model only serves as a recommender from a given item list, which makes it impossible to be misused or dual-use.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We incorporate three datasets in our experiments including MovieLens [50], Goodreads 6 and LastFM [51], all of which are open-source. Also, the backend language model used in our research is LLAMA2-7B [32], which is also an open-source model. The details of our used assets can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We won't release any assets in our research except necessary checkpoint and processed data in order to reproduce our experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing or research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends DPO from pairwise (Bradley-Terry) to multi-way comparison (Plackett-Luce), where one positive example is considered better than multiple negative examples. The paper applies this approach in recommendation systems, showing promising results on several standard benchmarks.\n\nFrom Rafailov et al: “although more general Plackett-Luce ranking models [ 30, 21] are also compatible with the\nframework if we have access to several ranked answers”.\nThis paper shows the derivation of the Plackett-Luce extension. In that sense the technical contribution is somewhat incremental.\nI would therefore view this more as an application paper in recommendation systems. However this is not how it is written, most of the focus is on the theoretical analysis extending DPO from Bradley-Terry to Plackett-Luce.\n\nSTRENGTHS:\n* The paper is technically sound and mostly clearly written.\n* The experimental results look very promising.\n\nWEAKNESSES:\n* The technical contribution seems somewhat incremental.\n* Some points need clarification (see questions below).\n\nQUESTIONS:\n1. It is assumed that there is a single positive example which is better than all the negative ones. In reality, multiple items may satisfy users’ interests, so the negative examples should be chosen with care. In the experiments, how are negative samples chosen? This was not clear from the paper. \\\nActually, in recommendations it is natural to show a slate of results and the user would click on one (or more), so the unclicked results can serve as negatives. Though this is not the type of data available in benchmarks used here such as MovieLens.\n\n2. In Fig 2a, it seems like most of the gains in performance already exist in SFT. How come this is much better than other fine-tuned models like LLaRA?\n\n3. Fig 2b: the performance of S-DPO seems to still be improving after 1200 iterations, why not run it longer until it stops improving?\nAlso, since the S-DPO loss and DPO loss are not directly comparable, you should include a figure showing HR@1 as a function of training steps for S-DPO and DPO.\n\n4. Fig 3a: the performance seems to still be increasing, is there a number of samples for which it saturates?\n\n5. MIssing baseline: compare the proposed approach to DPO with pairs consisting of the positive example and each negative example separately (use same negative example choice as S-DPO). This is more computationally expensive but interesting to see how it compares in terms of performance (and runtime).\nRelated to that, it would also be good to include a comparison of runtimes between S-DPO and DPO (perhaps show the above variant of Fig 2b in terms of runtime, not just of training steps).\n\nMinor:\n* L79: empty section\n* L132: deviate -> deviating\n* L175: the exp should be with the sum\n* L180: “faster<space>.”\n\nLIMITATIONS:\nNo societal impact concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
proposed a softmax DPO in recommendation domain.

STRENGTHS:
1. The paper mainly focus on softmax DPO for recommendation
2. The paper is well wrriten with good experimental validation.
3. The source code is avaliable.

WEAKNESSES:
1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.

2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?

QUESTIONS:
1. The novely of this work is not high, it seems that they mainly adapt DPO to the recomemndation area, most of the concept are from DPO.

2. what is the complexith of S-DPO, and how can it scale up the large-scale dataset?

LIMITATIONS:
It is unclear how this framework can scale up to large scale dataset.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a modification to the Direct Preference Optimization (DPO) by incorporating a softmax loss to enhance the training of language model (LM)-based recommender systems.\n\nSTRENGTHS:\n- The paper is well-written.\n- The mathematical formulations are clear and appreciated.\n- The proposed method is fairly simple yet effective.\n- The results are convincing.\n\nWEAKNESSES:\n1) Novelty\n- The use of multiple negatives in recommendation systems is not new and should be cited. Thus, Research Question 2 (RQ2) lacks novelty as it has already been explored. The novelty of using multiple negatives in LM-based recommender systems is uncertain.\n\n2) Statements\n- Equation 8 represents the softmax loss, which has already been applied in other recommender systems like BERT4Rec, whereas Equation 12 does not represent the softmax loss, leading to confusion.\n- The claim in lines 225-226 \"indicating the significant roles of knowledge and reasoning ability in language models for recommendation tasks.\" is not provable due to differences in training datasets.\n- The statement \"However, indicated by the low valid ratio and suboptimal performance, untuned LM-based recommenders are limited by inadequate instruction-following capabilities or a lack of domain-specific knowledge\" in lines 228-230 is incorrect as evidenced by high ValidRatio scores of ChatRec.\n\n3) Replicability\n- The prompts used in the experiments are not provided, making replication difficult.\n\n\n4) Evaluation\n- NDCG is usually preferred, and metrics like @10 or @20 are more commonly used than @1 in recommendation evaluations. For example, LM-based RecSys works like LLama4Rec use @5.\n- In Figure 2a, the statistical significance of the results is not specified. It is also unclear why the results are based on LLama and not LLama2.\n\n5) Comparison\n- The comparison of training epochs between LM-based recommenders (5 epochs) and S-DPO (an additional 3 epochs) is unfair.\n- In Table 1, the best HR@1 is highlighted in bold, but the same should be done for the ValidRatio column even if your model does not perform the best in that aspect.\n- The statistical test used should be specified.\n- Optimizing hyperparameters for the proposed model without doing the same for competitor models is unfair. A fair comparison would involve equal time allocation for hyperparameter searches.\n- Figure 2b shows that S-DPO decreases faster but remains higher than DPO, making the comparison questionable. Similarly, Figure 2c lacks clarity on whether the values are comparable.\n-The statement \"On the other hand, an excessively large β prevents the model from effectively learning ranking relationships, leading to suboptimal performance\" is not supported by Figure 3b and 3c, which show consistent performance improvement.\n\n\n6) Related work\n- The related work section is more effective when placed after the introduction to better contextualize the state-of-the-art before presenting the new method.\n- The discussion on LM for recommendation is not thorough, with many works cited together without detailed descriptions.\n\nQUESTIONS:\n- Consider replacing \"side effect\" with a more positive synonym.\n- The methodology section (2) appears empty; clarify if the true (new) methodology starts at section 3.1.\n- Equation (1) needs clarification as i_p and i_d represent items (identifiers) and not scores nor ranks. \n- Space before subsection name in line 148\n- Line 159: \"mutli\" --> write \"multi\"\n- Clarify lines 174-180 regarding \"hard negative items\" and the theoretical basis for S-DPO's ability to mine hard negatives.\n- Section 3.2 is unclear, especially the statement \"The structural correlation indicates that DPO and S-DPO are more suitable for recommendation than the language-modeling loss.\"\n- The definition of Rel.Ipv should be moved to the caption of Table 1.\n\nLIMITATIONS:\n- The computational effort required for using more negative samples is not considered. It would be beneficial to compute an approximate empirical complexity of the method.\n- The use of the Movielens 100K dataset is a limitation due to its small size. A larger dataset like ML-1M would provide better validation while remaining computationally feasible compared to ML-20M.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by advancements in Direct Preference Optimization (DPO) and the success of softmax loss in recommendations, the paper propose Softmax-DPO (S-DPO). S-DPO enhances LM-based sequential recommenders by incorporating multiple negative samples in preference data.The paper demonstrate the superiority of S-DPO in modeling user preferences and boosting recommendation effectiveness.\n\nSTRENGTHS:\n1. DPO, a method that considers preferences in NLP, has been extended to LM-based recommender systems, expanding NLP techniques to the recommender system domain.\n\n2. Building upon related previous work such as reinforcement learning from human feedback (RLHF) and DPO, the motivation and approach of this paper are well-founded.\n\nWEAKNESSES:\nIn the recommender system domain, considering preferences in the loss function is not particularly new. However, experiments and explanations for this are lacking.\n\n1. There are no experiments comparing S-DPO with other preference-based losses in recommender systems (BPR, InfoNCE, as the paper mentioned ). If such comparisons are not feasible, the reasons for this are insufficiently explained.\n\n2. The experiments lacked the application of S-DPO loss to various base models. Consequently, the general applicability of S-DPO was not demonstrated.\n\n3. Running DPO multiple times and S-DPO appear conceptually similar. Therefore, there are no experiments comparing their efficiency. A comparison of time or memory usage seems necessary.\n\nQUESTIONS:\nSame as weaknesses\n\nIs there a justification for not conducting the experiment described above?\n\nLIMITATIONS:\nLM's methodology was brought to the Recommender system. However, there seems to be a lack of discussion in the Recommender system.\n\nBPR loss is typically not used in sequential scenarios, so it would have been beneficial to emphasize that S-DPO specifically addresses preferences in sequential contexts. Additionally, conducting more experiments to validate this aspect would have been advantageous.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}