Ordinal Preference Optimization: Aligning Human Preferences via NDCG

Authors: Yang Zhao, Yixin Wang, Mingzhang Yin
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Models, Human Preferences Alignment, Listwise, Learning to Rank

Abstract:
Aligning Large Language Models (LLMs) with diverse human preferences is a pivotal technique for controlling model behaviors and enhancing generation quality. Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and their variants optimize language models by pairwise comparisons. However, when multiple responses are available, these approaches fall short of leveraging the extensive information in the ranking given by the reward models or human feedback. In this work, we propose a novel listwise approach named Ordinal Preference Optimization (OPO), which employs the Normalized Discounted Cumulative Gain (NDCG), a widely-used ranking metric, to better utilize relative proximity within ordinal multiple responses. We develop an end-to-end preference optimization algorithm by approximating NDCG with a differentiable surrogate loss. This approach builds a connection between ranking models in information retrieval and the alignment problem. In aligning multi-response datasets assigned with ordinal rewards, OPO outperforms existing pairwise and listwise approaches on evaluation sets and general benchmarks like AlpacaEval. Moreover, we demonstrate that increasing the pool of negative samples can enhance model performance by reducing the adverse effects of trivial negatives.

Decision: reject
Reviewer scores: 6, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1729687416019

thread_type:
official_review

<thread_object>
creator: reviewer_9n1z

<content>
summary:
This paper presents OPO, a novel listwise approach that leverages the Normalized Discounted Cumulative Gain (NDCG), a widely-used ranking metric in Learning To Rank, to align LLM with human preference. By incorporating a differentiable surrogate loss function, OPO effectively utilizes the ranking information derived from multiple model responses.

strengths:
- The paper is generally well-organized and clearly written.
- The paper conducts comprehensive experimental with detailed examples, and the implementation code is publicly available.

weaknesses:
- Contributions of this paper are quite limited and appear to be incremental. The key technical components, including Neural NDCG objective [1], Sinkhorn scaling [3], and Neural Sort [2], have been previously established in the literature. Sections 3.1 and 3.2 present a reformulation of the established Neural NDCG formulation (Section 4.1 and 4.2).

- Missing discussion on how this work is novel compared to other listwise method, such as LiPO[4]. LiPO uses the DCG weighting scheme, which inherently captures the relative proximity within multiple responses. Additionally, the discussion of current listwise methods in line 53 requires further clarification to better distinguish their differences and respective limitations.

- While the paper claims to establish an explicit connection between LLM alignment and ranking model training, this theoretical connection is inadequately elaborated in the paper. Moreover, similar insights have been previously explored in LiPO's work.


- In traditional learning-to-rank tasks, NDCG serves primarily as an evaluation metric, with standard LTR loss functions only loosely correlated to these evaluation criteria. The use of differentiable NDCG surrogate loss helps bridge this optimization-evaluation gap in ranking tasks. However, LLM is commonly evaluated on open benchmarks rather than ranking metrics, and the paper lacks theoretical justification in two aspects: why NDCG optimization would lead to better preference alignment, and what motivates the choice of NDCG as the optimization objective.

- The primary motivation is that NDCG can better utilize relative proximity within ordinal multiple responses. However, in lines 207-208, the paper does not explain why "the gain function should be proportional to the label" can effectively capture the relative ranking of different responses. Moreover, lines 209-211 indicate that recommendation tasks focus on top-ranked items. This design stems from NDCG's discount function, which is derived from an approximation of human attention decay. However, the paper lacks discussion on whether such a discount function is appropriate for preference optimization tasks. 

- The paper could benefit from comparing different Learning to Rank objectives, such as Neural Sort (which Neural NDCG builds upon) [2], PiRank[5] and other ranking methods.





[1] NeuralNDCG: Direct Optimisation of a Ranking Metric via Differentiable Relaxation of Sorting.

[2] Stochastic Optimization of Sorting Networks via Continuous Relaxations

[3] A relationship between arbitrary positive matrices and dou-
bly stochastic matrices.

[4] LiPO: Listwise Preference Optimization through Learning-to-Rank.

[5] PiRank: Scalable Learning To Rank via Differentiable Sorting.

questions:
1. In MT-Bench experiments, why the reported results differ from those presented in other papers.

2. In the ablation studies of Neural NDCG, while different gain functions are compared, the paper does not explore the impact of different discount functions.

3. Table 5 suggests that diverse negative samples are more crucial than varied positive samples. However, the performance gain of All Pairs over Single Pair might simply result from increased training data.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730039766702

thread_type:
official_review

<thread_object>
creator: reviewer_zehe

<content>
summary:
The paper proposes to use a listwise objective for LM alignment when multiple responses for a prompt is available. The objective is an existing one from the IR field. The proposed method is evaluated on datasets such as AlpacaEval and MT-Bench against baselines that focus on objective alternatives with the same backbones.

strengths:
1. It is good to further study the listwise perspective of the important LLM alignment problem. However, there seem to have some factual misunderstanding of the literature discussed below.

2. The study of diversity of negative samples is interesting and has value. 

3. The paper essentially performs a study of additional listwise objectives for LM alignment. I think this is good to do, though I am not convinced of the significance and the trustworthiness of the empirical results, see below.

weaknesses:
1. The major issue of this paper is its contribution. There are *wrong* claims such as claiming LiPO to be pairwise approaches, while they are actually listwise - for example, LambdaRank is considered as a *listwise* objective in the information retrieval literature and it is a proxy of NDCG, which the authors avoid discussing. This is because the weights use *listwise* information so they are no longer pairwise. It is even more confusing that LiPO pretty much has the exact same idea (while being public almost a year ago) to optimize NDCG. The paper uses another *existing* surrogate loss to do that. In the IR literature, this surrogate loss is not as acknowledged as the one studied in the LiPO work. The first two major contributions (of out 3) may even be put in the LiPO paper without any inconsistency. 

2. Despite the large overlap with existing work, the paper does not have a convincing argument on why the deployed objective is a better choice, so the experiments are at the risk of cherry-picking. And given extensive work on such topics, the contribution is quite limited to meet the standard of a top-tier ML venue. Note that even if there's a argument specific to the objective, the contribution is still likely limited since the formulation and general objective (e.g., optimizing NDCG) are more important and the loss alternatives are not that different (as studied intensively in the IR literature).

3. Besides the formulation and method, the additional value provided by the paper is also relatively limited. For example, it is still limited to offline alignment (which might be ok itself, the issue is it does not add value based on existing work as discussed above).

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730121708949

thread_type:
official_review

<thread_object>
creator: reviewer_bleq

<content>
summary:
This paper proposes Ordinal Preference Optimization (OPO), a listwise method that adapts the existing learning-to-rank (LTR) methods to optimize LLMs based on multiple response rankings. By adapting the existing learning-to-rank (LTR) methods such as NeuralNDCG with DPO, OPO can directly optimize under the multi-response datasets.Experiments show the effectiveness of the proposed OPO.

strengths:
1. The proposed OPO method is straightforward and intuitive.
2. Experiments with various pairwise and listwise baselines demonstrate the effectiveness of OPO.
3. The codes and scripts are provided in the supplementary materials, enhancing the reproducibility of this work.

weaknesses:
1. The motivation for OPO is not clear. It is only briefly mentioned (e.g., line 53 "which is relatively simplistic") and requires a more detailed comparison with existing listwise approaches to highlight the motivation of the proposed new listwise method.
2. The proposed OPO directly adapts existing learning-to-rank (LTR) methods (e.g., NeuralNDCG) to optimize DPO with NDCG, lacking novelty. In fact, as listed in this paper, other listwise LTR methods can also applied to the DPO.
3. The paper lacks theoretical analysis and comparative discussion of the proposed OPO method, which would strengthen its contribution and clarify its advantages over existing methods.

questions:
1. Why was NeuralNDCG chosen as the surrogate loss for optimizing NDCG, rather than other direct NDCG optimization losses?
2. Given the potential sensitivity of the win rate metric with GPT4, was a t-test conducted to assess its significance, and were selection biases (e.g., sensitivity to AB vs. BA results) considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702361820

thread_type:
official_review

<thread_object>
creator: reviewer_eyvn

<content>
summary:
The paper introduces a new surrogate objective function for training generative models using reinforcement learning, where the objective function is based on NDCG ranking rewards of multiple candidate answers. The paper argues that strong RL training methods often depend on designing contrastive reward functions, which utilize the comparative values rather than the absolute values of multiple answers for self-normalization and uncertainty quantification purposes. The proposed NDCG objective is an extension of this idea, where the detailed ranking positions for a list of answers are considered at the same time.

To optimize the NDCG objective, the paper further introduces a differentiable Neural Sort operator (Grover et al 2019), where the ranking steps are relaxed into a probabilistic distribution matrix using softmax operators. The paper then compares the proposed Neural-NDCG method with other popular NDCG optimization methods, such as LambdaRank and simple pairwise comparison baselines. The paper shows the generality of the proposed method by fine-tuning medium-scaled LLMs on ListUltraFeedback dataset and evaluating them on AlpacaEval and MT-Bench datasets.

strengths:
Originality: The paper introduces a new way to train RLHF models by extending pairwise reward functions into list-wise ranking functions. The paper then introduces neural-sort operator to replace the current SOTA LambdaRank method. The introduction of the neural-sort operator looks natural and the intuitions are clear.

Quality: The proposed methods are evaluated on recent models and relevant benchmarks. The experiments and the ablation studies are well-designed. They serve as a good template for researchers setting up similar experiments.

weaknesses:
It does not seem very clear from the paper, but the proposed RL training steps seem to be offline. According to Eq (2), the learned policies are only fitted on the likelihoods of the pre-generated answers, instead of the self-generated answers. It is unclear to me whether the offline optimization paradigm presents a performance gap when the models are evaluated online or when they generate answers in the real world.

As a minor point, the NeuralSort algorithm (5) could use more discussions. For example, the paper could include more details about the probabilistic P_sort matrix in the numerical examples.

questions:
Regarding online vs offline RL, can the authors:
* Clarify whether the proposed method is intended for offline or online RL.
* Discuss potential limitations of using an offline approach
* Compare the proposed method to online RL alternatives to verify if potential performance gaps may exist
* Give concrete examples on how the proposed method perform in real-world online generation scenarios.

Regarding NeuralSort details, please:
* Include a step-by-step example of how the probabilistic P_sort matrix is computed in Appendix A
* Clarify the benefits of probabilistic permutations. Will having ties or near-ties allow better characterization of the ranking uncertainties?
* Discuss potential limitations of probabilistic permutations. Will the P_sort matrix lead to miscalibration, where all of the generated matrix may degenerate? For example, having the same [0,1,0,0] rows in the entire matrix?
* Discuss uniqueness of the solutions. Are there other equivalent formulations to achieve neural-sort? Will these equivalent formulations alleviate some of the limitations? In what sense is the proposed algorithm the most obvious choice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732523334420

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to shared concerns.

comment:
# Response to All:

We thank the reviewers for your positive and constructive feedback. Your comments signiﬁcantly help us improve the paper. We summarize responses to the shared concerns as follows:

## 1. Motivation and Theoretical analysis

To additionally analyze why OPO outperforms other methods, we counted their correct ranking pairs and successful ranking flips (i.e., incorrect to correct) during training. OPO exhibited significantly higher efficiency in flipping incorrect pairs compared to DPO and LiPO. We can also theoretically prove that the optimal ranking accuracy of NDCG-based methods is bounded by the ranking accuracy of the reference model. But to achieve such an optimal policy, OPO doesn’t necessarily maximize the reward margin like DPO and LiPO, which might be a reason that leads to higher efficiency in increasing the ranking accuracy metric. We can show that OPO has a higher speed in correcting incorrect ranking accuracy pairs (i.e., make $\frac{\pi\_\theta(y_w|x)}{\pi\_\theta(y_l|x)}>1$ given $\frac{\pi\_{\text{ref}}(y_w|x)}{\pi\_{\text{ref}}(y_l|x)}<1$) than DPO and LiPO. The detailed results are provided in Appendix A.9.


## 2. Online vs. Offline

OPO is an offline alignment algorithm, so it also faces the challenge of fitting on sub-optimal pre-generated training data like other offline methods. However, it can be applied in an iterative paradigm, which has been proven to be effective in the industry. [1] Traditional iterative pairwise methods sample best-of-n responses and assign them reward scores with the separated reward model, then they choose preferred response $y_w$ and non-preferred response $y_l$ based on assigned rewards and apply alignment algorithms like DPO on these response pairs. For OPO, it shares the same sampling-rewarding stage and the only difference is that we input a list of K responses with their assigned reward scores and employ listwise algorithms on them.

[1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

## 3. Supplement Experiments

To comprehensively investigate the LTR literature, we added additional baselines of BCE, RankNet, ListNet, and PiRank. We also supplemented ablation studies on the key components of OPO. The results shown below indicate that OPO consistently outperforms these methods.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel surrogate objective function for reinforcement learning-based generative model training, leveraging NDCG ranking rewards to evaluate multiple candidate answers. This approach extends contrastive reward functions by incorporating detailed ranking positions for better self-normalization and uncertainty quantification. The authors optimize the NDCG objective using the differentiable Neural Sort operator and demonstrate the method’s generality by fine-tuning language models on ListUltraFeedback and evaluating them on AlpacaEval and MT-Bench datasets, outperforming existing NDCG optimization methods like LambdaRank.

I agree with the common concerns of the reviewers (Reviewer Eyvn, bLeQ, Zehe, 9n1Z) regarding the motivation of the paper over DPO motivation. Furthermore, the novelty of the proposed methods is unclear over the existing LTR algorithms. In general, the significance and efficacy of listwise objectives for LM alignment are not very convincing, even from the empirical results. Based on the common consensus, therefore, we suggest the authors submit to the next suitable venue with all the necessary modifications.