Title: Risk-aware Direct Preference Optimization under Nested Risk Measure
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

LLM Alignment

Risk-Aware
Abstract: When fine-tuning pre-trained Large Language Models (LLMs) to align with human values and intentions, maximizing the estimated reward can lead to superior performance, but it also introduces potential risks due to deviations from the reference model's intended behavior. Most existing methods typically introduce KL divergence to constrain deviations between the trained model and the reference model; however, this may not be sufficient in certain applications that require tight risk control. In this paper, we introduce Risk-aware Direct Preference Optimization (Ra-DPO), a novel approach that incorporates risk-awareness by employing a class of nested risk measures. This approach formulates a constrained risk-aware advantage function maximization problem and then converts the Bradley-Terry model into a token-level representation. The objective function maximizes the likelihood of the policy while suppressing the deviation between a trained model and the reference model using a sequential risk ratio, thereby enhancing the model's risk-awareness. Experimental results across three open-source datasets: IMDb Dataset, Anthropic HH Dataset, and AlpacaEval, demonstrate the proposed method's superior performance in balancing alignment performance and model drift.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Risk-Aware Direct Preference Optimisation (Ra-DPO), a new method for aligning LLMs with human preferences while taking risk into account. The method builds on token-level DPO (TDPO) and incorporates a nested risk measure from risk-sensitive reinforcement learning.\nThe authors want to include risk-awareness into LLM-training and take it into consideration, rather than simply maximising expected reward.\nIn the paper, risk is defined as the deviation (KL-divergence) from the original reference policy. The proposed loss function aims to maximise the likelihood of preferred responses while simultaneously penalising this token-level model drift.\nThe authors provide theoretical justification for their approach, deriving the loss function from a risk-aware advantage maximisation problem.\nThey present results of applying their method on IMDb dataset and Anthropic HH dataset, and evaluate the resulting models using AlpacaEval. They argue Ra-DPO achieves a better balance between alignment performance (reward accuracy, win-rate) and KL-divergence compared to the baselines.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n- Paper is clear, easy to read, well structured\n- The paper correctly identifies a limitation in current methods (DPO) as they are risk neutral (simply optimise for mean reward) and that TDPO, while moving to a more granular token-level approach, remains risk neutral. Proposing a risk-aware alternative method makes sense\n- The structure of the experiments is sound, training on IMDb, Anthropic and evaluating on AlpacaEval and Ra-DPO shows marginal improvements on the baseline (in AlpacaEval). In Figure 3 they also show improved reward accuracy and KL-divergence from the reference model compared to TDPO. I would have been interested to see the performance of DPO here too\n\nWEAKNESSES:\n- I fundamentally disagree with the authors' framing of \"risk\" as divergence from the reference policy. The whole point of preference optimisation is to make the model less harmful, more helpful and honest. Quite a few papers have shown the KL divergence is useful to keep the model from moving completely out of distribution rather than to maintain any good qualities [1]. A reference model is not inherently \"safe\" or \"correct\". Constraining the policy to stay close to it can limit the model's ability to learn and adapt to the preferences it is being trained on. So much that recent, high-performing offline preference optimisation loss functions (such as SimPO [2]) completely scrap the KL-divergence term and instead just add an SFT term to the loss. This also has memory advantages (as you don't have to keep a copy of the model weights in memory)\n- The paper's motivation that \"a significant deviation from the reference model typically implies the degradation of superior decision-making and reasoning capabilities\" is a strong claim presented without evidence.\n- This narrow definition of risk is acknowledged, but only in the appendix\n- Unconvincing Empirical Results: the improvements are marginal and no extensive hyperparameter search is performed. I understand the computational constraints, but given the marginal improvements I don't think the current evaluation is enough\n- beta of 0.1 is quite low for DPO I think? I've seen 0.5 being used usually\n\n[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study - Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, Yi Wu \n[2] SimPO: Simple Preference Optimization with a Reference-Free Reward - Yu Meng, Mengzhou Xia, Danqi Chen\n\nQUESTIONS:\n- How were the hyperparameters for the LLM experiments chosen?\n- Could you run the LLM experiments on across a few more hyperparameter values? (beta, learning rate)\n- Could you include performance of DPO (maybe with several different betas) in Figure 2?\n- i think the paper could be improved by maybe using an external \"risk evaluation\" model to score risk rather than using drift from the reference model\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Risk-aware Direct Preference Optimization (Ra-DPO), a LLM alignment method that includes risk-awareness by using a class of nested risk measures. The main idea of Ra-DPO is to incorporate the nested risk measure such as Conditional Value-at-Risk (CVaR) and Entropic Risk Measure (ERM) into Token-level Direct Preference Optimization (TDPO) to avoid the model drift from the reference model. This paper evaluates Ra-DPO on IMDb, Anthropic HH, and AlpacaEval. The experiment results show that Ra-DPO can improve the alignment performance, while avoiding the model drift.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this paper can be summarized as follows:\n- S1. The authors derive the loss function of Ra-DPO by using the thorough theoretical analysis.\n- S2. Even though the evaluation benchmarks (IMDb and Anthropic HH) are relatively simple, the authors provide comprehensive experiment results on those datasets.\n\nThe weaknesses of this paper can be summarized as follows:\n- W1. Even though this paper is theoretically strong, the proposed method can be considered as a direct extension of the previous work TDPO (Token-level Direct Preference Optimization, ICML 2024).\n- W2. The loss function of Ra-DPO includes the term that calculates $D_{SeqRR}(x, y; \\pi_{ref} | \\pi_{\\theta})$ over the generated tokens $z$. This increases the computational complexity during training time.\n- W3. Even though Ra-DPO can effectively control model drift, the model performance does not seem outperforming. For example, in experiments on AlpacaEval (i.e., Table 1), the win-rate of DPO, TDPO_1, and Ra-DPO_1 are 52.1, 51.9, and 53.5, respectively.\n\nQUESTIONS:\n- Q1. How does Ra-DPO compare to the other method such as DPO and TDPO in terms of computational complexity?\n- Q2. Is there any trade-off in incorporating risk-awareness into DPO or TDPO?\n- Q3. In Figure 3, why does Ra-DPO with ERM work worse than Ra-DPO with CVaR?\n\nLIMITATIONS:\nThe authors provide the limitations of their work in Section D.1 (i.e., Limitations) of Appendix. However, I am not sure that the limitation section can be in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors highlight a couple issues with industry standard techniques:\n1. The lack of granular token-level likelihood and optimization (issues faced by DPO and its variants). \n2. The lack of consideration of the overall distribution (i.e. the tails) during the reward maximization approach for DPO variants that do not suffer from the issue highlighted in #1 (such as for TDPO, which only optimizes the expected reward/the mean). Many variants that do address this fail to address #1.\n\nThey are proposing a new method that hits #1 and #2, and have termed this Ra-DPO. They are proposing a new method that considers the shape of the reward distribution when maximizing the expected return. They do this by replacing the expectation by a nested risk functional which pushes the downstream reward return distribution through a coherent risk analysis (to maintain convexity), by focusing on the worst outcomes (to focus on the tails parameterized by mu) and by propagating that token by token (basically if any downstream token contains a high risk tail, the current token reflects it). The authors also proceed to show the efficacy of it through empirical trials. \n\nThe contribution are two new objectives (Ra-DPO1/2) that directly address both of the issues with current widely adopted optimization objectives, and the theory that underpins those assertions + empirical results for those two new objectives compared to the currently adopted objectives.\n\nSTRENGTHS AND WEAKNESSES:\nQuality and Clarity: The quality of the paper is strong in my own opinion. I see a strong narrative whereby the authors start with drafting the limitations of the current methodologies in industry along two relevant axis (i.e. capturing the entire reward distribution and the need for token-wise maximization) and then directly address the limitations through a new formulation that is theoretically underpinned rigorously. The proofs are well written and is relatively easy to follow, and the limitations and workarounds are clearly indicated. I also appreciate the augmentations made to the formulations for mechanistic purposes, making it easier to adopt. \n\nThe practicality study of the methodology needs more work in my opinion to get a strong significance score. I fully acknowledge the three dataset (IMDB, Anthropic-HH, and AlpacaEval) + Two Model family (Pythia-XX,  GPT2) experiments, but they seem a bit limited due to how dated they are, and how saturated the results are.\n\nThe baselines are the most significant weakness of this paper, with IMDB (Figure 2) showing very saturated rewards across hyperparameters (almost impossible to discern impact). Figure 3 and Table 1 are more meaningful, but requires more model baselines. The varying in models is appreciated, but the authors do not sufficiently explore the connection of the base models to the task being studied despite relying so heavily on the reference policy entropy. I can conceive of a situation where the base policy is very peaky (due to heavy SFT, etc.) and the sampling of the tail being more difficult. This is especially true when the data being optimized is very in-distribution to the base reference policy SFT data. This relation does not seem to be comprehensively studied. It would be helpful to see how this approach practically translates to those situations, which are relatively common. Also, I don't see direct metrics that address their core claims, adding in more tail-sensitive practical metrics would be helpful here to directly address the claims that are made initially, instead of solely adding E2E numbers since they do claim that capturing the entire reward distribution is a big part of their contribution. Having concrete metrics that directly show it would be helpful. Overall strong paper, but this would improve it. \n\nThe idea of jointly optimizing for the reward distribution shape and a per-token reward maximization is fantastic. Even beyond safety, the tail is a critical part of the out of distribution quality, and ensuring minimal degradation with respect to the base policy is very meaningful (more so on verifiable and reasoning-benefitting tasks, which isn't covered here). Having a way of optimizing for both jointly is good contribution -- the only major limitation of the paper being the practicality and feasibility study comprehensiveness.\n\nQUESTIONS:\n1. How does the efficacy of this metric change with the base model's entropy? How does it compare to when the base model is very peaky on the data this is being applied to versus when it is more OOD to the base model? I think this is my main concern with this approach, since it relies quite heavily on the base model's entropy and the ability to sample the tail. An explicit section on this (with respect to having concrete metrics from the base model itself to understand its capability and behavior on the evaluated task) would be helpful. \n2. How does the gradients and the E2E efficacy look on tasks with rewards that benefit from varying answer lengths? Some tasks require longer answers and others shorter, how does the efficacy of the methodology wrt the baselines shift on these tasks? I do not think the length controlled scores from AlpacaEval satisfactorily covers this, as the target of this question is the potential for noisy gradients as the length increases for tasks that demand it, while the AlpacaEval communicates better accuracy at lower generation lengths. \n3. Can you probe Ra-DPO on benchmarks that explicitly measure the rare but severe failure modes and report more tail-focused metrics from these? Ideally, it would be good to have some way to qualify the shift in model's answers specifically made possible by capturing the entire reward distribution -- including the tail.\n\nLIMITATIONS:\nBroadly yes, but the rigor of the baselines is a limitation of the paper. The paper is good and the idea is novel, but the paper can be made stronger by having more practical feasibility studies of the approach. In general, being able to provide reasoned guidance about the the dependency on the reference entropy, the gradient noise response with respect to the generation length, and more tail-oriented studies would greatly strengthen the paper. It is possible to do this with limited compute by primarily having good coverage of base models and tasks along these axis in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}