Optimizing Preference Alignment with Differentiable NDCG Ranking

Authors: Jiacong Zhou, Xianyun Wang, Jun Yu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Language models; Human preferences alignment

Abstract:
Aligning large language models with human preferences improves interaction quality and safety by ensuring outputs better reflect human values. A promising strategy involves Reinforcement Learning from Human Feedback (RLHF), starting with collecting and ranking responses generated by a supervised fine-tuning model to refine alignment. Current methods (DPO) focus on learning from pairwise preference data, categorizing responses into preferred and less preferred pairs, and optimizing by maximizing pairwise margins. Recent studies have uncovered a substantial discrepancy between the theoretical aspirations of preference learning and its real-world results. Current preference alignment techniques underperform expectations, with ranking accuracies below $60\%$ on standard datasets. This suggests existing methods inadequately capture ideal preference relationships within sequences. To address this challenge, this paper introduces \underline{D}irect \underline{R}anking \underline{P}reference \underline{O}ptimization (DRPO), a novel method that views human preference alignment as a Learning-to-Rank (LTR) task. DRPO leverages NDCG, a widely used LTR metric, to optimize the ranking of responses within lists based on preference data, thereby enhancing ranking accuracies. Due to the nondifferentiability of NDCG, we propose diffNDCG loss, a differentiable approximation facilitated by a sorting network to simulate NDCG. Furthermore, to improve the quality of generated response, we propose a novel margin-based Adaptive Rank Policy Score. Extensive experiments have shown that DRPO outperforms existing baseline methods, enhancing the quality of the generated responses.

Decision: reject
Reviewer scores: 6, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730153837546

thread_type:
official_review

<thread_object>
creator: reviewer_mwpe

<content>
summary:
The paper introduces a novel method to improve the alignment of large language models (LLMs) with human preferences. This method, called Direct Ranking Preference Optimization (DRPO), views human preference alignment as a Learning-to-Rank (LTR) task and employs the Normalized Discounted Cumulative Gain (NDCG) metric for direct optimization from ranked response lists.

Contributions of the paper include:

**Novel Method**: DRPO treats human preference alignment as a listwise ranking problem, using ranking preference data to optimize LLMs.

**Adaptive Rank Policy Score**: A new ranking score computation strategy is introduced to dynamically adjust score margins based on relative positions.

**Differentiable NDCG (diffNDCG)**: A differentiable approximation of the NDCG metric is developed to serve as the loss function, allowing for more accurate alignment of LLM responses with human preferences.

strengths:
1. The performance gain from the proposed method appears to be quite significant.
2. The code and models are publicly available for review.
3. Some revise has been incorporated. I’ve reviewed the paper previously for a different venue, where I’ve raised questions on the choice of discount factors and comparison with other differentiable approximation methods of NDCG. Both have been incorporated in the main text as ablation studies.

weaknesses:
1. However, some other discussion in the previous review has been ignored. For example, in the previous review, I’ve asked for a novelty discussion against LiPO work. The discussion is still missing. The authors only mention “which (diffNDCG) can utilize list data more effectively than existing methods” in Line 812 and no discussion like “reward hacking challenge” is mentioned. Is that because diffNDCG is not a proper solution to the challenge as well? Or due to some other reasons, which are unclear to me.
2. The main issue of the work is that the proposed method appears very significantly better than other differentiable ranking approaches. This is extraordinary, but hard for me as a reader why it’s so much more effective than others. Because of better differentiable quality (what in specific)? Or because the experiments were done on some specific settings? Like Qwen 0.5B and 1.8B. Both sizes are a bit too small for functioning Large Language Models. For example, the results on Mistral-7B are less significant.
3. The results are a bit too dense. Very tiny fonts, hard to read and follow.

questions:
1. In Figure 3, it appears that LiPO grows with list size a bit faster than DRPO, will they at some point cross in terms of RM win rate, or the gap will keep eventually?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730587388491

thread_type:
official_review

<thread_object>
creator: reviewer_nu6a

<content>
summary:
The paper proposes a DRPO method, a modification of DPO, for aligning Large Language Models (LLMs) with human preferences. The core idea of the paper is to leverage the NDCG metric and corresponding continuous proxies to augment the DPO approach, which relies solely on pairwise preference. The authors introduce the "Adaptive Rank Policy Score" (Equation (5)), an approach that assigns each candidate (LLM response) a score that combines the log-likelihood of the response (according to the current policy) with position factors. The policy is then updated by comparing the generated order with the ground truth ranking via a differentiable NDCG proxy, diffNDCG. My vote is a "reject" due to issues in the evaluation methodology and the lack of a rigorous analysis of the proposed scoring scheme. I found the empirical evaluations to be limited, with no insights on the assumptions under which the proposed approach might generalize to a broader setting. I believe the paper is not ready for presentation at ICLR.

strengths:
The idea of using the log likelihoods as the gains in the NDCG metrics is interesting, and might have an impact.

weaknesses:
In the paragraph on line 061, the authors argue that "win rate" is more correlated with NDCG metrics. However, the evidence they provide in Figure 4 and Table 11 is limited. The main issue with Figure 4 is that the numbers are conditioned on the use of the proposed DRPO method. Thus, Figure 4 does not provide information on what might happen with a different method. From this perspective, Figure 4 should be removed from the paper. In Table 11, the authors calculate Pearson's correlation between NDCG and "win rate"; however, the table lacks the correlation with the pairwise approach used in the original DPO. Table 11 does not support differentiation between the pairwise approach and NDCG ranking. Furthermore, the proper way to compare correlation between ranked lists is the Kendall Tau distance.

I somewhat disagree with the statement on line 096. A well-established loss from the LTR community is LambdaRank, which relies on pairwise comparisons and builds a preference model somewhat similar to DPO. Given this connection, the original DPO paper optimizes for NDCG metrics via pairwise preferences. While it might not be an ideal proxy (as it does not use the lambda weights), improving pairwise alignment does lead to better ranking.

On line 102, the authors discuss the contribution of "a novel differentiable NDCG" metric. This contribution is incremental, as it is a straightforward application of sorting networks (Petersen et al. 2021) to the optimization of NDCG metrics.

The main contribution of the paper is the scoring approach in Equation (5). The moving average of scores for a specific position relies on assumptions that the authors did not discuss. For example, I expect the ARPS scoring scheme to perform poorly if the distribution of examples changes drastically from one instance to another, as might be expected when aligning over different types of tasks or questions. 
Table 1 contains imprecision that makes it challenging to assess the results. The entries for ListNet, PiRank, and Neural Sort are actually the PPR and ARP methods with these methods as subroutines (see line 406). However, is it PPR or ARP? Line 406 states both ("and"). Additionally, I find it misleading to use the names of these subroutines in the context of PPR and ARP; please be explicit about this distinction.

One of the main challenges in assessing the quality of the proposed methods is that the ground truth is obtained via other models. The authors acknowledge this in Appendix B, but I suggest moving this acknowledgment to the beginning of the paper to provide clarity upfront about what the reader can expect. As it stands, the proposed methodology might capture biases in the evaluation models. If one switches the evaluation models, the numbers could change unpredictably.

I would like to point out that the choice of alpha in Equation (8) is critical. If the authors make comparisons with PiRank, they need to specify the temperature used in the experiment. The alpha in diffNDCG relates to the temperature in PiRank, and adjusting it controls the trade-off between convergence speed and precision of the final estimates.

questions:
I encourage the authors to be more precise in their mathematical notation. In equation (5) the quantity $V_{q(y)}$ appears on both sides of the equation, which would reduce the quantity to a linear combination of $\log\pi$. Please add iteration indexing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609318358

thread_type:
official_review

<thread_object>
creator: reviewer_uelb

<content>
summary:
1. The paper introduces an approach with differentiable NDCG (diffNDCG) ranking with the aim to improve preference alignment in large language models by optimizing for new ranking-specific metrics. The objective is to improve on simpler optimization objectives that rank items for a recommendation system, e.g. pairwise comparisons of items for a given query, which treat items within a list as independent overlooking the relationship between them. 

2. The paper provides a background on traditional preference-alignment models and identifies their limitations to motivate the need for diffNDCG.

3. The diffNDCG approach uses a soft permutation matrix to approximate a ranking order that is differentiable, which allows for gradient-based optimization on the NDCG metric.  

4. The differentiable sorting network component (DRPO) offers a way to rank items simultaneously across the entire list based on predicted scores and departs from traditional pairwise comparisons.

strengths:
The authors present a novel approach to preference alignment by introducing differentiable NDCG (diffNDCG), which addresses the challenges of standard ranking models by aligning with the structure of human preference more closely. The paper provides a good motivation for diffNDCG, noting its potential to capture the fact that items in a ranked list are not independent of one another.  The differentiable sorting component, DRPO is what sets it apart from traditional methods.

weaknesses:
1. **Model Misspecification**: The reliance on score-based probabilities could introduce misspecification, as it assumes scores fully capture ranking probabilities through an assumes scoring function. 

2. **Biases in Data** Furthermore, if the scoring function is miscalibrated (i.e. depends heavily on what function one is applying to obtain probabilities) or biased (the function weights items in a higher position more), diffNDCG could actually reinforce biases (like position bias), potentially overemphasizing higher-ranked items based on position rather than relevance. Since there's a vast literature on ranking and bias, and this metric's aim is to improve ranking, then this warrants some discussion. 

3. **Stand-alone Tables and Experimental Detail**: The tables lack guidance and notes making it difficult to get takeaways without referring back to the text. Tables 2 and 3, for example, do not stand alone. DRPO, ARP, etc. should all be defined in table notes. It is not immediately apparent which metrics indicate improvement or if higher values signify better performance. This should all be in the tables notes. Including confidence intervals and significance tests for key results would also help support the robustness of observed improvements rather than percentage changes. 

4. **Experimental Setup and Parameter Transparency**: I'm not clear on the experimental setup. What are the models validated on? Data with labels? Are the clicks simulated? Perhaps this is obvious or self explanatory, but I think should be made clear for any reader.

5. **Writing:* I feel Sections 1-3 could be more to the point, or that we could arrive to the core of the paper that begins us to 3.2 sooner. This would create space for experiments from the Appendix. 

The paper would benefit from reducing the amount of passive voice used (rather than: "have demonstrated" "have been proposed" "is developed" "progress has been made" "has been widely adopted") use "they demonstrate" "they propose"). The extensive use of hte passive voice makes it more difficult to follow the paper.

questions:
The authors could address the method’s potential for amplifying biases (position, selection or algorithmic biases), especially if the scoring function is miscalibrated. 

Sections 1-3 could be more to the point, or we could arrive to the core of the paper that begins us to 3.2 sooner. This would create space for experiments from the Appendix, including how they set up the validation data.

Tables should be more self-contained with definitions of acronyms and metrics, and notes on takeaways.

The paper would benefit from reducing the amount of passive voice being use (rather than: "have demonstrated" "have been proposed" "is developed" "progress has been made" "has been widely adopted") use "they demonstrated" "they proposed"). It detracts from the clarity of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730838281857

thread_type:
official_review

<thread_object>
creator: reviewer_d9sn

<content>
summary:
This paper presents a novel approach, Direct Ranking Preference Optimization (DRPO), for aligning Large Language Models (LLMs) with human preferences using ranking preference data. The authors argue that existing methods, primarily focused on pairwise comparisons, fail to adequately capture the nuanced relationships inherent in ranked lists of responses and capture the importance of top ranked responses. DRPO tackles this challenge by framing preference alignment as a Learning-to-Rank (LTR) task and directly optimizing for the Normalized Discounted Cumulative Gain (NDCG) metric, commonly used in LTR.

DRPO consists of three core components:
● Adaptive Rank Policy Score: This component replaces the traditional Policy Reference Ratio score in DPO with a mechanism that prioritizes maximizing the likelihood of preferred responses while incorporating adaptive score margins based on their positions in the ranked list.
● Differentiable Responses Ranking: A differentiable sorting network, specifically an odd-even sorting network, is used to rank responses based on their calculated scores, making the ranking process differentiable.
● DiffNDCG Loss: Due to the non-differentiability of NDCG, the authors propose a differentiable approximation, diffNDCG, which leverages permutation matrices from the sorting network to simulate the NDCG metric.

Finally, extensive experiments on datasets like Anthropic's Helpful Harmless (HH), UltraFeedback, and VLFeedback are used to illustrate that DRPO outperforms baseline methods in metrics like win rates on model based comparisons like GPT-4 and Reward Model Win Rates. The authors also provide an analysis of the time complexity, an ablation study, and computational efficiency of their method.

strengths:
- Novel framing of preference optimization: The paper tackles the crucial problem of LLM preference alignment from a new angle by adopting a listwise ranking approach and optimizing for NDCG, which aligns well with the goal of prioritizing highly preferred responses, and performs very well in practice.

- Technical Soundness: The proposed methodology is technically sound, employing a differentiable sorting network and a differentiable approximation of NDCG to enable end-to-end training.  

- Strong Empirical Results: The experimental results are compelling, showing consistent improvements over existing methods across different model sizes and datasets.

- Comprehensive Analysis: The paper includes a thorough ablation study analyzing the impact of individual components, including the Adaptive Rank Policy Score and different discount factors used in diffNDCG. I also appreciate that the reviewers also try existing methods with ARP score and show the positive impact.

weaknesses:
Clarity in Ranking Score Computation: While the concept of Adaptive Rank Policy Score is interesting, the description and motivation behind its specific formulation could be clearer. Providing more intuition and justification for the choices made in Equations (4) and (5) would enhance understanding. The paper goes back and forth between Section 3 and future sections (including results). Authors should reconsider the style of explaining different choices and make the sections self-contained. 

- Choice of Sorting Network: The authors opt for the odd-even sorting network due to its "simplicity and ease of implementation". However, they mention other variants with potentially better time complexities. Exploring and comparing different sorting network architectures could strengthen the paper.

- Reliance on availability of scores: DRPO relies on the availability of scalar scores for each response in the ranked list of responses for supervision. This seems to be a limitation when you only have access to ordered pairs (for example, in the Anthropic HH dataset). Does the reliance on DeBERTa in that case change whether the model is able to learn? Could using GPT4 or a larger model make a difference?

- Real-world Human Evaluation: The reliance on reward models as a proxy for human evaluations, while acknowledged as a limitation, could be further addressed. Supplementing the results with some real-world human evaluation of generated responses would bolster the claims. Also, is there a 

- Generalizability: The experiments focus on specific datasets. Evaluating DRPO on a wider range of preference alignment tasks and datasets would provide further evidence of its generalizability. See questions below.

questions:
- How sensitive is DRPO to the choice of hyperparameters, particularly those involved in the Adaptive Rank Policy Score and the differentiable sorting network? Policy optimization can be a little unstable so it would be good to present some guide to the reader around the list of hyperparameters and setting them. 
- Are there particular scenarios where DRPO is better to optimize for human preference? There are plenty of other preference datasets for LLMs. Can the authors provide a reason for selecting these datasets, and/or the kind of preference datasets where DRPO is expected to work better. 
- What are the potential avenues for future research building upon DRPO? For example, could DRPO be adapted for scenarios with dynamic or evolving human preferences?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Direct Ranking Preference Optimization (DRPO), a novel approach for alignment from human feedback by framing it as a learning-to-rank task optimizing a differentiable approximation of NDCG (diffNDCG. The reviewers recognized the following strengths:
- Novel approach by applying sorting network for differentiable NDCG, and by introducing Adaptive Ranking Policy Score.
-  Experimental results show significant performance gain.

Weaknesses:
- Novelty and contribution compared to prior works are not clear.
- The reason for the significant performance gain should be analyzed and discussed in detail.
- Assuming access to reliable ranking scores seems strong, where in practice the score could be biased or unavailable.
- The contribution of diffNDCG might be overstated. 

This is a borderline paper with final ratings of 6, 6, 6, 3. During the discussion, the negative reviewer recognized the performance gain but was not full convinced by the rebuttal. AC agrees with the positive reviews regarding the strengths.  However, AC also agrees with the concerns raised by the negative reviewer such as potential overstatement of contribution and novelty against existing works. The authors are strongly encouraged to carefully revise the final version to address these concerns, which will further enhance the paper. Overall, AC recommends rejection.