Leveraging robust optimization for llm alignment under distribution shifts
Authors: Mingye Zhu
Yi Liu
Zheren Fu
Yongdong Zhang
Zhendong Mao
Venue: NeurIPS 2025 poster ()
Keywords: preference alignment
distribution shifts
robust optimization
Primary area: applications

Abstract: Preference alignment methods are increasingly critical for steering large language models (LLMs) to generate outputs consistent with human values. While recent approaches often rely on synthetic data generated by LLMs for scalability and cost-efficiency reasons, this reliance can introduce distributional shifts that undermine the nuanced representation of human preferences needed for desirable outputs. In this paper, we propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts. Our approach first leverages well-learned classifiers to assign a calibration value to each training sample, quantifying its alignment with the target human-preferred distribution. These values are then incorporated into a robust optimization objective that minimizes the worst-case loss over regions of the data space most relevant to human preferences. By explicitly focusing optimization on the target distribution, our approach mitigates the impact of distributional mismatch and improves the generation of responses that better reflect intended values.

Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
Mingye Zhu!.2 Yi Liu Zheren Fu'
Yongdong Zhang' Zhendong Mao University of Science and Technology China, Hefei, China State Key Laboratory of Communication Content Cognition, People's Daily Online, Beijing, China
Abstract
Preference alignment methods are increasingly critical for steering large language models (LLMs) to generate outputs consistent with human values. While recent approaches often rely on synthetic data generated by L.MMs for scalability and cost- efficiency reasons, this reliance can introduce distribution shifts that undermine the nuanced representation of human preferences needed for desirable outputs. In this paper, propose novel distribution-aware optimization framework that improves preference alignment despite such shifts. Our approach first leverages well-learned classifiers to assign calibration value to each training sample, quantifying its alignment with the target human-pre preferrde distribution. These values are then incorporated into robust optimization objective that minimizes the worst-case loss over regions of the data space most relevant to human preferences. By explicitly focusing optimization the target distribution, approach mitigates the impact distributional mismatch and improves the generation of responses that better reflect intended values.
Introduction
The rapid proliferation of large lan- guage models (LLMs) has made increasingly important to ensure that model outputs align with human val- ues. TTchhniques such as Reinforce- ment Learning from Human Feed- back (RLHF) [33 51 40) and Direct Preference Optimization (DPO) [35]
Qa
Training distribution Target distribution Qo
Learning
Po
P- oO2 800
ERM training over P
Pb
Qu
DoRA training over
P
45 have shown promise by lever- aging high-quality, human-annon annotated data to guide model behavior [43 2 12]. A common approach to con- structing alignment data involves ei- ther manual annotation by humans or the use of LLMs,, with the former being resource-intensive and time- consuming, thus limiting scalability and broader applicability [4]. Conse- quently, recent research has increas- ingly focused on leveraging synthetic
Mixture Response Shift
Figure Comparison of ERM and DoRA Training. The left section illustrates the training distribution P. which is mixture of human-preferee (target) distribution Qo and LLM distributions Q1. highlighting the mixture response shift. The right section contrasts the outcomes of the tra- ditional method (ERM training) P with the proposed DoRA training over Qo. demonstrating how DoRA better aligns with the target distribution.
Corresponding author: Yi Liu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data generated by advanced LLMs, which have shown strong capability to simulate human prefer- ences [22
In this work, we concentrate on understanding and addressing one specific form of distribution shift arising from synthetic data within current alignment paradigms. As illustrated in Figure [ traditional empirical risk minimization (ERM) assumes that the training distribution I' matches the target distribution Qo. thus leading a policy Pa that aligns well with Qo. However, in practice, / does perfectly mimic Qo due to two key limitations: (1) several studies suggest that training data derived and (2) reward models are prone to biases [46] , potentially leading to suboptimal labelling. Consequently, the resulting training distribution P may diverge significantly from the ideal distribution Qp. In such settings, naively employing ERM training can yield learned policy Po that is biased towards artifacts in the training data rather than genuinely aligning with human intent.
from LLMs may exhibit inherent misalignment with human values 44
14
38
Mitigating distribution shifts in alignment data is a persistent yet understudied challenge. Existing robustness-focus approaches primarily address pairwise noisy labels and reward uncertainty [45] 61 but they are tightly coupled with the Bradley-Terry (BT) model, limiting their versatility for broader alignment objectives (e.g., listwise optimization argue for more general robust optimization frameworks that not only mitigate distribution shifts from
48
39
30
50))
To bridge this gap,
synthetic data but also seamlessly adapt to the growing complexity of alignment objectives.
In this paper, we propose Distribution-awen wwrbbe optimization for Robust Alignment (DoRA) to improve the robustness of alignment algorithms where training data comprises mixture of heterogeneous sub- distributions--ucch as those arising from different synthetic sources or online updates [25] 29, [8] modular
Rather than tailoring solutions specific alignment formulations, our approach functions plug-in that enhances baseline robustness across diverse, and increasingly complex, alignment objectives encountered in real-world deployment. At the core of DoRA lies a simple yet effective strategy: it first employs well-trained distribution classifiers to assign a calibration score to each training sample, estimating its alignment with the target human-preferred distribution. These scores are then integrated into distribution aware optimization objective that minimizes the worst case loss over data regions most representative of human preferences. This strategy ensures that the model remains resilient to distribution shifts between the training data and the target distribution, preventing it from disproportionately favoring biased synthetic patterns while still benefiting from their scalability.
Contributions
We introduce DoRA, a distribution-awars optimization framework that robustly aligns LLM outputs with human preferences under distribution shifts. Technically: We estimate the alignment of each sample with the target human-preferred distribution as a calibration term, which is then incorporated into a distribution-aware optimization objective that minimizes the worst-case loss over regions most relevant to human preferences. Theoretically: We characterize the robust calibration alignment objective as a KL-divergence--aat distribution- ally robust optimization problem, augmented with a reweighting mechanism. Emperically: We demonstrate the effectiveness of our framework through extensive experiments, with consistent improvements alignment metrics compared to state-of-the-a baselines.
2 Problem Formulation
Notations. Let = {Ir, .J. each instance composes of one query æ and n corresponding responses yo, V1 92,..., Vn-11 We assume that these observed data are drawn from training distribution P, and the target distribution of interest, is denoted as Qo. We then formalize the following definition.
Z represent a datum in preference dataset Z, where
Definition 2.1 (Mixture Response Shift), A Mixture Response Shift occurs when, for any input query x, the conditional distribution of responses is a mixture of different distributions that partially overlaps with the target distribution Qo (with fraction a), i.e.:
R-
Ply|x) = a Qo(y(z) +
Σ
3, Q1()l)),
(1)

---

where Pn 2 0 and a + + +
Remark 1. Definition 2.1 formalizes scenarios in which, for each given query x; fraction of the responses are drawn from the target distribution Qo (e.g.. human preferred responses), while the remaining responses sampled from other distributions Q1, Qn-1(e.g. synthetic LLM generations). This setting reflects the practical reality that alignment training data often consists of a mixture of responses with diversified quality.
The core intuition is that training solely on data from the target distribution- done in naive supervised fine-tuning (SFT)-may limit generalization and fail to account for real-world variability. Robust alignment benefits from exposure to both diverse and suboptimal responses, which help models distinguish desirable behavior from undesirable patterns. When direct access human- preferred data is limited, high-quality generations from strong models (e.g., GPT-4) can serve as reasonable proxy of the golden responses from the target distribution.
Distributionally robust optimization: Distributionally Robust Optimization (DRO)
26
well-established framework that focuses on minimizing the worst-case expected loss over the perturbed or adversarial distribution Q within the uncertainty set of P. Denote the instance-level loss as E(0, z). then DRO is formulated as: 2)], P (Q D(QU)))
min sup #E8 QEP
(2)
with P being the uncertainty set and ID a set of all possible distributions. PP) is a distance metric, and p parameter controlling the size of the ambiguity set. This objective aims to find
that minimizes the expected loss over the worst-case distribution rather than minimizes the average
performance over P. Specifically, we model the IP using the Kullback-Leibler (KL) divergence, as it provides a tractable method for solving DRO problems
[19]. With the change-of-measure
technique (20), the inner maximization problem is then formally written as:
max E [h(z)(((. z)], s.t. Ep(h(z) log h(z)] S.
where 1((z) == dQ (z) is the density ratio (Radon (Radon-Nikodym derivative) between Q and P. Equation di
can be viewed as variational form of the problem in Equation 2 as replaces the intractable optimization over distributions Q with an optimization over the density ratio f(z).
Challenges: While DRO methods are designed to handle sub-population shifts, they often suffer from over wer-pessimismm 20 49 28). which manifests as poor generalization and overly conservative, low- confidence predictions, excessive focus worst-case scenarios degrade overall performance.
3
Distribution-aware Optimization for Robust Alignment
To achieve more principled alignment and mitigate over-pessimism, we propose to extend the uncertainty set in the standard DRO framework to capture the mixture-induced shifts central to our setting. We then propose a calibration mechanism, derived from probabilistic classifiers, infer the source distribution of each data point. Theoretically, we show that the proposed DoRA objective essentially optimizes a KL-based DRO objective, augmented with a distribution-avy aware reweighting mechanism. The overall framework is illustrated in Figure 2
3.1 Sample-Level Calibration with Probabilistic Classifiers
In the KL-DRO framework, the worst-case target distribution can be written Q(z) (See Appendix B.I for derivation). This formulation implies that instances with higher loss values e(0, z) are exponentially upweighted. While this mechanism enhances robustness by prioritizing challenging instances, it may also introduce over-pessimism when a small number of outliers (e.g.. noisy or mislabeled points) have very large losses, thereby dominating the optimization.
((0))
To mitigate this issue, we introduce simple calibration mechanism at the sample level. Rather than treating all samples equally when computing the robust loss, we scale the loss of each instance by detendet factor hz(z), which reflects its estimated relevance to the target distribution.
To formalize i((2), we first define the family of a-covered distributions based on Definition 2.1
P. = {oo: :())))))))))))))))))))))()))): = Qo +
B. Q((1k)))
3

---

Distribution-aware Classifiers
Calibrated Alignment Objective
(z, y/u, 1/1, 3/2)
Human preference
For Qi, learning classifier Co.y (3/
For each 2 calculate
Calibration Term h(z)
Scoring with classifiers
LMMistrr preference
Q
How likely is Y/ from target distribution
ERM:
Rank sample by loss {(z)
Lg preference
y/o 0.99
1/1
0.85
DoRA:
1/2
1/2
0.21
Rank sample by calibrated loss h(z)) ((2)
Human-praferred sample?
Figure 2: The DoRA pipeline. For each datum z, where responses drawn from mixture of distributions, DoRA uses trained classifiers to estimate the alignment of each with the target distribution. These scores are then aggregated into a calibration term l((z) for each sample, which reweights the original loss f(z) during optimization to enable more principled robustness control.
Here, P(y]z) denotes the observed distribution, and Qn(ylar) represents the target distribution. In classical DRO, the uncertainty set typically includes distributions that are "close" to a nominal empirical distribution P. often under some divergence measure. Here, we refine this setup by incorporating prior structural knowledge under response mixture shift.
Learning distribution-aware classifiers. To construct the calibration factor h(z), we assume the existence of "golden" (or reference) distribution, denoted Protdea (y/zz) provided by oracle or approximated using trusted human-labeled data. Following Grover et al. [16], we train a probabilistic classifier ce, for each sub-distribution Q; to estimate the probability that response yi originates from the golden distribution. Under the assumption that Co, is Bayes optimal, the importance weight for response drawn from Qi is derived as:
Ws (y) (r) = Pguldee (y) ())a:) = "1- cp. (y/z²)
Co. (u)
(5)
(z(()))
1-
where E {1, {L...., 1} 7% the imbalance ratio between synthetic distribution and the golden distribution g(y=u) In practice, to estimate Pgolden. we leverage human preferepeeerree responses from the
((==1)
target distribution Qo-
Constructing the calibration factor. We now construct instance-dependeer calibration factor h(z) based on the mixture structure and the learned classifiers.
Proposition 3.1. Let Plylz) Qo(ulz) + Σ B. (yl=), with a + -3 +... fn- 1 and E (0, 1). Under the mixture response shift, we define hu(z) as an empirical estimate of the degree which a given sample aligns with human preferences:
{ Pgolden(
1
Psoiden (3/l)
T((z)
n a Qolylæ)
B Qn-1(y/x)
(6)
W Wo.- wo.-,(ylx)) = ( 70 (y|x))
110001100111 B con-i)" }
= ( Way (2)z) + ..... + Bn-1
+...
a(1 c.n))
where Whg is defined earlier in Equation
Remark 2. Proposition 3.1 introduces a calibration factor that can naturally incorporate mixture information into the robust objective. While the density ratio h(z) quantifies how much a single sample is emphasized or downweighted in the adversarial distribution Q compared to the nominal distribution P. the calibration term h(z) serves as proxy for human preference by estimating how likely a sample is to originate from the trusted component Qo. Intuitively, h(z) amplifies the contribution of responses that are more aligned with the golden distribution, and suppresses less preferred ones accordingly.

---

Practical Considerations. The importance weight in Equation S was originally introduced for binary classification tasks [3]. When the learned classifier co, outputs probabilities that approach the corresponding importance weights can become unbounded, leading to unstable and often problematic optimization. To rectify this, we insert a stabilizing term 72 into each denominator, thereby bounding
h(z) such that (X,J) + (0, n). Empirically, this simple modification prevents extreme weights when co. approaches ensuring that no single sub-distribution excessively dominates training.
3.2 Deriving the DoRA objective
In this section, we derive the final alignment objective of DoRA. Specifically, we modify the pre-defined KL-DRO framework in Equation 3 by modulating each sample's contribution by precomputed calibration factor h(z), which reflects its estimated alignment with the target distribution. The robust objective is defined as:
max Ep (p(M)))(())(0
s.t.
Ep(h(z) h(z)] S.p.
Since h(zz) is fixed scalar per instance, it does not affect the optimization variables in the Lagrangian. Consequently, the worst-case risk from Equation addits closed-form dual solution:
Proposition 3.2 (Worst-case risk under mixture response shift). Let the loss be modulated by instance- level weights hlzz, which are fixed and known. Then the worst-cuse risk under KL constraint is given by:
1
(h(z) e(0.21) 2p
R(0)= inf A A=0
log Ep
exp
(8)
In turn, minimizing R()) over 0 reduces
1 M((2)
min A log Ep AEE
exp X
e(0, 2)
(9)
Calibration Temm
where z an instance of (I, yo. .......... 1nn-))).
Remark 3. See Appendix B.2 for the detailed Lagrangian derivation. The calibration term i(x) acts as a soft indicator of how closely each data sample's ' responses (for given query x) align with the "ideal" or golden distribution, thereby determining its relative importance during training. Consequently, if sample has large loss {(0, z) but small hxx), DoRA deems it less valuable to learn from and downweights it accordingly. Please see Algorithm for more details.
Algorithm DoRA Optimization Algorithm
Require: Pretrained model parameters 0, ro- bustness parameter dataset D {z:
b Phase 2: DoRA Training
1: Precompute h(z,) Vz, E D via Eq. 6
(Ii, Vi.0...... ......... ni )
while not converged do
Ensure: Optimized model parameters 0
3: Sample batch B = {2 C
Phase 1: Classifier Learning
Compute robust loss over according to Equation 9
1: for j 0... n do
2: Assign binary labels:
1 B
h(z,)e(0, 2i)
10 (1
(1
if Vi.j is target, otherwise
L A log
exp
5: Update 0 2 V
VeLRAA
3: Train
classifier
6Ø2
using
6: end while
{(1,, (i.g. 1
4: end for
4 Experiments
Models and Datasets. We validate the proposed method with two base models: Mistral-7B-v0.1
Llama-3 3.1.8BBBBBBBBB three widely used datasets in alignment literature: HH-RLHF Summarization and
5

---

the Ultraceed datasets. Specifically, we consider the following settings: 1.pairwise preference setting where we leverage the original pairwise data; 2. listwise preference setting, where we augment the original pairwise data with 2 additional synthetic responses from Mistral-7B 7B-Innsructt-O... leading responses in total for each query. We primarily foeus on Setting as we believe they represent more practical scenario for the current alignment paradigm where models learn from multiple higher-quality samples from various sources.
Baselines. For pairwise comparisons, we employ four well-performed baselines: DPO
R- 48 and
351
DPO [4). EXO [1] and SimPO [3T] For listwise contrasts, we use DPOpr [35]. RRHF
LIRE 50) Please find the detailed objectives these algorithms Appendix C.5
Experimental Settings. For each task, we first train classifiers as specified in Algorithm with simple BERT base model. The chosen response in the original dataset labeled 1 and the other as O. The well-trained classifiers output probabilities that indicate the likelihood score that response belongs to the target distribution. We also train an SFT model on the preferred responses serve as the starting point before policy optimization. We set À = for all experiments. Detailed hyperparameter configurations and additional training settings are provided Appendix C
4.1 Experimental Results
Performance on pairwise preference datasets. For all four pairwise baselines, we use the original preference datasets without any response augmentation. As illustrated in Figure 3 DoRA consistently improves performance across all baselines. This suggests that even widely adopted "golden" datasets affected by distribution shifts or contain noisy samples.
Mistral-7B pairwise HH Dataset
Liama-8B on pairwise HH Dataset
Original w/ DeRA
+2..5
Original w/ DoRA
+2.8 (3.8%)
Win Rate (%)
Win Rate (%)
+0066
+3.8 (5.2%)
70
+10..5
70
(2.7%)
65
+1.5 (2.6%)
(19.55%
65
60
DPO
R-DPO
XPO
SimPO
DPO
R-DPO
XPO
SimPO
Figure 3: DoRA boosts pairwise baseline performance. When applied to unaugmented "golden"
preference datasets, DoRA consistently enhances response quality across all baselines.
Comparison with robust baselines. Robust DPO 6 and Dr.DPO [5] are
Table I: Comparison with pairwise robust baselines on HH dataset. While all the approaches improve the win rate of vanilla DPO, DoRA achieves the lowest lose rates compared to the counterparts.
two strong baselines designed to ad-
dress label-flp noise in pairwise pref- erence data, particularly in BT pref- erence models. Since both methods operate in the pairwise setting, we compare their performance against the DoRA-aligned DPO variant with
Mistral-7B
Llama-8B
Baselines
Win(t)
Lose())
Win(t)
Lose(()
DPO 35
74.8 (=1844
23.5 (2150)
72.9 (#1.02)
23.7 (t:1.65
Robust DPO 6
77.8 (=2220)
20.0 (#230)
74.2 (#144)
22.9 (6:123)
HH dataset in Table D is impor-
Dr.PPO
75.2 (=1164)
22.4 (#1633)
73.0 (E1.41)
24.3 (##.71)
tant note that DoRA is method- agnostic module that can be seam- lessly integrated into wide range of alignment objectives- not limited to the pairwise case. From the results, we observe that all robust baselines outperform vanilla DPO, confirming the value of robustness in preference learning. Notably, DoRA A achieves the lowest lose rates among all methods, suggesting that its calibration mechanism provides more flexible and fine-grained control.
DPO w/ DoRA
75.4 (=1.6))
21.4 (9:146)
75.4 (1-0.89)
20.9 (2:0.77)
AWhile pairwise DPO is derived under the Bradley-Tery family of preference models in particular, listwise DPOpt is derived under general Plackett- preference model handle multiple reward signals.

---

Performance on listwise preference datasets. Next, we experiment with the augmented datasets, which directly simulate a mixture response shift scenario which candidate responses are drawn from diverse underlying distributions. As shown in Table 2 incorporating DoRA leads to performance gains consistently across multiple baselines and tasks, with higher win rates and lower lose rates against the reference responses, underscoring its effectiveness in mitigating distributional shifts.
Table 2: DoRA enhances listwise baselines on dialogue and summarization tasks. Win indicates that GPT-4o assesses DoRA's 's response superior compared to the golden responses from the datasets. Bold numbers suggest DoRA the winner. The results demonstrate that incorporating DoRA generally improves performance or least keeps on par with the baselines.
HH-RLHF
Summarization
Alignment
Mistral-7B
Llama-8B
Mistral-7B
Llama-8B
Wi
Lose(())
Win(Se)
Lose($))
Wi
Lose(())
Win(%y>
Lose(())
pP
75.0
22.5
81.0
18.0
53.3
46.3
54.5
42.8
w/ DoRA
78.0
21.0
82.5
16.0
55.3
43.5
59.0
30.5
RRHF [8
76.5
19.5
43.8
56.0
70.0
29.5
70.8
28.8
w/ DoRA
79.8
19.0
44.5
55.0
72.0
28.0
74.0
25.0
LIRE [50]
72.8
26.8
82.0
17.5
82.5
17.5
82.5
17.0
DoRA
84.0
16.0
84.5
14.5
81.0
19.0
83.8
16.0
To further evaluate the robustness of our method, we benchmark models trained on the augmented UltraFeedback dataset using AlpacaEval 2 [TO] and Arena-Hard [23] As reported in Ta-
Table AlpacaEval and Arema-Hard results. Experiments suggest that DoRA keeps or improves the instruction-followe following capabilities of baselines trained on the augmented UltraFeedback dataset.
3 DoRA generally improves the instruction- following capabilities for the baselines, partic- ularly for LIRE. However, we observe a slight performance drop for DPOpt. which may be attributed to the nature of the UltraFeedback dataset. Specifically, the chosen responses in UltraFeedback are drawn from a mix of dis- tributions themselves, deviating from our core assumption that they originate from a single source. This distributional mismatch could re- duce the effectiveness of the learned classifier, thus impacting overall performance.
Dataset
AlpacaEval 2.0
Arena-Hard
Metrie
LC(%)
WR(%)
Length
WR(%)
DPOgt
18.80
18.14
1972
12.3
w/ DoRA
18.27
17.73
1943
12.2
RRHF
10.52
13.88
1494
11.0
w/ DoRA
10.65
14.48
1446
11.2
LIRE
28.74
29.44
1815
20.0
w/ DoRA
31.28
32.02
1972
20.3
DoRA enhances reward-confidence correlation for reliable generation.
1.0
1.0
Classifier probabilities
Classifler probabilities
Figure 4 plots normalized reward scores against classifier confidence probabilities for generated responses, where each point represents one model output. The dashed lines rep- resent best-fit linear relationship be- tween normalized reward scores and classifier probabilities, derived via or- dinary least squares regression. The figure demonstrates that DoRA dis- plays more consistent alignment be- tween the classifier's 's confidence and the normalized rewards, indicated by clearer positive slope in the regres- sion line. This suggests that DoRA facilitates a more reliable prediction of outcomes, where higher rewards correspond more likely to higher clas- sifier probabilities. Moreover, we see overall reward distribution of DoRA A shifts rightward, especially for Llama3 model, indicating more high-quality responses.
0.8
0.8
0.6
0.6
0.4
DPQ
0.4
DPO
w/ DoRA
w| DoRA
0.2
0.2
0.0
0.0
0.0
0.5 Rewards (Normalized)
1.0
0.0
0.5
1.0
Rewards (Normalized)
(a) Mistral-7B HH dataset
(b) Llama-8B on HH dataset
Figure 4: cevfrdeconfidence correlation for generated responses. DoRA exhibits stronger reward-confidence cali- bration than baselines, evidenced by steeper regression slopes (A,3=+0.15 for Mistral and A +0.133 for Llama, larger slope means better correlation). This indicates DoRA's high- reward outputs more closely match the target distribution's characteristics, validated by elevated classifier probabilities.
7

---

4.2
Ablations
AP
RRHF
LIRE
0.76
Win Rate
0.80
0.74
0.75
0.70
0.65
0.5
1.0
2.0
0.5
1.0
2.0
4.0
0.5
2.0
4.0
À
A
Baseline
DRO
DORA
Figure 5: Performance variation with different choices of İ for vanilla DRO and DoRA. We observe that as À increases from 0.5 to 4.0, the win rate generally decreases, albeit with some variations. Besides, vanilla DRO generally downperforms the proposed DoRA.
Impact of on robustness and performance. The regularization parameter DoRA controls the balance between flexibility and robustness. Specifically, smaller values induce sharper exponential weighting, emphasizing high-loss samples and increasing robustness. Larger À values reduce this effect, making the objective closer to standard ERM training. Figure [ that À 1 yields strong performance on the HH task with Mistral-7B, while increasing À slightly decreases the win rate, underscoring the importance of tuning robustness. Figure @ F further illustrates that larger À yields policies whose behavior more closely resembles that of the baseline ERM-trained model.
0.2
A-0.5
10
logrrty chosen x
10
1=1.0
Iogrryychasee chosen
Density
-20
-20
A=4.0
0.1
-30
Original w/ DoRA (A-1)
-30
Original
w/ DoRA (A-4)
-40
-40
0.0
30
-25
-20
-55
-16
-16
-14 log ( (Ychosen
-i2
log (ogP(Yycssenlx)
log (ychaeen (X)
XX)
(b) Log prob. distribution w/ (c) Log prob. distribution w/ X Figure 6: Impact of different À values on policy distribution. (a). Larger À increases model probabilities on the HH--chsen responses. (b/c). Scatter plots comparing log probabilities assigned to HH-preffrred responses. The x-axis shows log probabilities under the initial SFT policy; the y-axis shows those under the original baseline and DoRA policies, respectively. As À increases, DoRA shifts closer to the original baseline (ERM training), assigning higher likelihoods to preferred responses.
DoRA w/ different values
Effectiveness of the DRO framework and reweight- ing mechanism. We in- vestigate the isolated effects of the DRO formulation and the reweighting mechanism across different alignment ob- jectives. As shown in Ta- ble 4 both approaches can vield performance improve- ments, but their effectiveness highly dependent on the specific alignment loss and task. For instance, methods like DPO appear relatively ro- bust to how the loss handled, showing only modest fluctuations across variants. contrast, LIRE demonstrates higher sensitivity to robustness techniques, with substantial performance gains or drops depending on the strategy used. In general, we conclude from the results that the combination of calibration-awee aware scoring and robust optimization is more effective than applying either component
Table 4: Comparison with DRO and reweighting mechanism. The results show that combining calibration-awe aware scoring and robust optimization generally brings out better performance.
Strategy
Expression
Loss E(0, z)
HH Data
Sum. Data
DPO
76.0
$4.0
DRO
log F. exp ((8.))
RRHF
77.0
52.8
LIRE
65.0
70.0
DPOPE
74.5
55.0
Reweighting
h(z))((0
RRHF
80.5
48.3
LIRE
80.8
78.3
DPO
78.0
55.3
DoRA
h(()) log E, exp
((fa)
RRHF
79.8
72.0
LIRE
84.0
81.0

---

alone, as their synergy leads to better distributional alignment and more reliable preference modeling under real-world distributional shifts.
4.3 Generalization and Robustness Evaluation
Label noise. In the preceding ex- periments, all models were trained on clean datasets without synthetic pertur- bations. To further assess the robustness of DoRA, we conduct corrupted-labe ex- periments that serve as stress tests under adversarial distribution shifts. In particu- lar, label noise is introduced by randomly flipping proportion of labels in the orig- inal training data. Experimental results indicate that the incorporation of DoRA consistently enhances model robustness the corruption rate increases, leading to stable and significant improvements across multiple benchmarks. This further demonstrates DoRA's effectiveness and reliability in real-world applications where annotation noise or distributional shifts are inevitable.
Table 5: Robustness evaluation under different corruption rates on HH and AlpacaEval 2 datasets. DoRA consis- tently improves robustness across methods.
Dataset
HH-RI-F
AlpacaEval 2
Corruption rate
20%
40%
60 %
40 %
40 %
Metric
WR(%)
LC(%)
WR(%)
DPOPL w/ DoRA
71.0
64.5
57.3
11.0
11.0
74.5
67.0
60.8
12.1
12.1
RRHF
63.5
42.3
26.3
7.7
4.8
w/ DoRA
65.5
44.5
30.8
8.5
5.5
LIRE
67.7
52.5
61.4
26.5
25.1
w/ DoRA
71.5
55.0
65.3
27.7
26.2
Self-training. While the primary focus this work is on mixture response shifts induced by synthetic data in offline training, we consider online adaptation as a promising di- rection. To provide preliminary evidence, we conduct small-scale experiment where we first train LLaMA-3.2 3B the HH dataset (Iterate 1), and then continue training on its own generated responses for two additional iter- ations. This setup naturally induces an evolving mixture shift, as the response distribution becomes increasingly syn- thetic. The results demonstrate that DoRA remains robust under such conditions, indicating that its instance-level cal- ibration generalizes well to online self-generated data scenarios where distributional drift arises organically.
Table 6: Performance improvement across training iterations. DoRA consis- tently enhances final performance under self self-training scenario.
Method
Iter 1
Iter 2
Iter 3
DPOpL w/ DoRA
77.0
83.5
85.0
78.5
85.8
88.0
RRHF
45.8
47.8
50.5
w/ DoRA
47.5
49.3
52.5
LIRE w/ DoRA
80.3
83.0
84.5
82.5
85.0
86.8
5 Related Works
Preference Alignment for LLMs. Since LLMs are pre-trained on vast internet data, they can generate outputs that are biased, harmful, or misaligned with human values. To address this, prefer- ence alignment techniques have emerged as key solutions. RLHF utilizes a reward model trained on human feedback guide reinforcement learning, and DPO streamlines the process by directly optimizing the model to prefer desirable responses without an explicit reward model. Building on these approaches, recent research has proposed refinements improve alignment efficiency and robustness. For example, Azar et al. [ presents a generalized preference optimization framework, Ethayarajh et al. (13) introduces novel loss function for enhanced robustness, and Meng et al. [3] explores simplified objectives to reduce computational overhead. These advancements reflect the ongoing effort to develop more scalable and effective alignment methods.
Synthetic Data for Alignment. Preference alignment typically relies heavily on human- annotated datasets, but the high cost and limited scalability of such data present a major bottleneck. To address this, recent research has explored leveraging synthetic data for aligmment. For example, RLAIF [22
synthesizes preference data and uses PaLM for feedback, while UltraFeedback 7 employs GPT-4 to annotate LLM- -enerated responses, creating scalable training datasets. Moreover, researchers have integrated synthetic data to expand candidate pools for preference learning [39
30 50),
demonstrating the potential of synthetic data. However, the use of synthetic data introduces new challenges. One key issue the mismatch between the sampling distribution and the learning policy. To address this, RSO 22 employs rejection sampling to source preference data from the estimated

---

target optimal policy, thereby improving the accuracy of the maximum likelihood estimator. Another critical challenge lies in the distributional inconsistencies between synthetic and human-genernn generated data during preference learning. This shift can hinder alignment performance, leading to biased model behaviors to true human preferences| [27]. In this paper, we focus on tackling this latter challenge, aiming to enhance the robustness of preference learning in the presence of distribution shifts.
Robustness in Alignment. DRO 19 is a well-established framework that minimizes the worst- case training loss over set of pre-defined groups, ensuring robustness to distributional shifts. In language modeling, Oren et al. [32) applies DRO to minimize losses over worst-case topic mixtures, while Sagawa et 37 enhances worst-group generalization in overparameterized regimes through increased regularization. For preference alignment, robust optimization techniques are explored to address challenges like reward uncertainty and noisy data. MaxMin-RLHH [5] learns mixture of reward functions via expectation maximization to cater to diverse human preferences. ROPO (24.. Robust DPO [6] and Dr.DPO [45] focus on noise tolerance in the pairwise BT paradigm. For instance, ROPO derives robust loss by suppressing the gradients of samples with high uncertainty, and Dr.DPO optimizes against worst-case pairwise scenarios for DPO, Similarly, GRPO (36] builds upon reward-free free DPO method by prioritizing groups with worse cumulative loss iteratively. In contrast to the prior work that focuses on label noise or is designed under the BT framework, our work aims to learn method-agnostic approach that may seamlessly generalize to diverse alignment paradigms.
Discussion
Conclusion. In this paper, we propose a distribution-aware robust alignment framework that alleviates the influence of synthetic data bias and distribution shifts in LLM alignment, By leveraging a learned classifier to aggregate calibration term to the DRO objective, DoRA effectively balances the scalability of synthetic data with the fidelity of human-allgnnn outputs.
Limitation and future work. Our framework assumes access to coarse- coarse-grained information about the source of the data (e.g., human- or model-generated) guide classifier training. While this is often feasible in curated alignment pipelines, it may be less accessible in fully open-domain or legacy datasets. Moreover, DoRA presumes that the target distribution can be least approximately estimated or measured; in scenarios where such reference distributions are unavailable or unreliable, its upplicability may be limited. Identifying principled heuristics or proxy objectives for such cases represents an important direction for future work. Exploring ways to relax this requirement through unsupervised or weakly supervised signals presents an exciting direction for future research. Our use of DRO is particularly beneficial in scenarios with distribution shifts or variable alignment quality. While its conservativeness may offer limited gains in clean settings, it holds promise in reliability- critical domains such as factual generation or value-sensitive applications. Finally, although online adaptation briefly mentioned in Section 4.3 extending DoRA to broader online learning scenarios remains a promising direction toward continual adaptation and scalability.
Acknowledgments and Disclosure of Funding
This research is supported by Artificial Aielligence-Nationa Science and Technology Major Project 20232D0121200 and the National Science Fund for Excellent Young Scholars under Grant 62222212. Besides, we grateful to Jiashuo Liu and Junkang Wu for providing helpful discussions during the preparation of this paper.
References
[1] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[2] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 05862, 2022.
10

---

[3] A. Ben-Tal, D. Den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2): 341-357, 2013.
[4] S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[5] S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024.
[6] S. R. Chowdhury, A. Kini, and N. Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint urXiv:2403.0044 00409, 2024.
[7] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie. Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. 2023.
[8] N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng. S. Hu, Z. Liu, M. Sun, and B. Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
[9] H. Dong, W. Xiong, B. Pang. H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024. URL https://arxiv. org/abs/2405.07863, 2024.
[10] Y. Dubois, B. Galambosi, Liang, and T. B. Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint arXiv:2404.044: 2024.
[11] C. Duchi, T. Hashimoto, and H. Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2019,
[12] K. Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding dataset difficulty with V-usable information. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1988-6008 PMLR, 17-23 Jul 2022.
[13] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.011 240.0013666 2024.
[14] B. Gao, F. Song, Y. Miao, Z. Cai, Z. Yang, L. Chen, H. Hu, R. Xu, Q. Dong, C. Zheng, et al. Towards a unified view of preference learning for large language models: A survey. arXiv preprint arXiv:2409.022 027995 2024.
[15] A. Grover and S. Ermon. Boosted generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[16] A. Grover, J. Song, A. Kapoor, K. Tran, A. Agarwal, E. J. Horvitz, and S. Ermon. Bias correction learned generative models using likelihood-free importance weighting. Advances in neural information processing systems, 32, 2019.
[17] A. Gudibande, Wallace, C. Snell, X. Geng, H. Liu, Abbeel, Levine, and Song. The false promise of imitating proprietary lms. arXiv preprint arXiv:2305. 15717, 2023.
[18] C. Gulcehre, T. Paine. S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 08998, 2023.
[19] Z. Hu and J. Hong. Kullback-leibler divergence constrained distributionally robust optimiza- tion. Available Optimization Online, 1(2):9, 2013.
[20] Z. Hu, J. Cao, and I.. J. Hong. Robust simulation of global warming policies using the dice model. Management science, 58(12): 2012.

---

[21] H. Ji, C. Lu, Y. Niu, P. Ke, H. Wang, J. Zhu, J. Tang, and M. Huang. Towards efficient exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024.
[22] H. Lee, S. Phatale, H. Mansoor, K. R. Lu, T. Mesnard, J. Ferret, C. Bishop, E. Hall, V. Carbune, and A. Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
[23] T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, arXiv preprint arXiv:2406. / 1939, 2024.
[24] X. Liang, C. Chen, J. Wang, Y. Wu, Z. Fu, Z. Shi, F. Wu, and J. Ye. Robust preference optimization with provable noise tolerance for Ilms. arXiv preprint arXiv:2404.0 04/02, 2024.
[25] Y. Lin, S. Seto, M. Ter Hoeve, K. Metcalf, B.-J. Theobald, X. Wang, Y. Zhang, C. Huang, and T. Zhang. On the limited generalization capability of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409.0335 409003650, 2024.
[26] J. Liu, Z. Shen, P. Cui, L. Zhou, K. Kuang, B. Li, and Y. Lin. Stable adversarial learning under distributional shifts. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 8662-8670, 2021.
[27] J. Liu, T. Wang, P. Cui, and H. Namkoong. On the need for language describing distribution shifts: Illustrations on tabular datasets. Advances in Neural Information Processing Systems, 1451371--5408 2023.
[28] J. Liu, J. Wu, T. Wang, H. Zou, B. Li, and P. Cui. Geometry-calibratee dro: Combating over-pessimism with free energy implications. arXiv preprint urXiv:2311.0500: 2023.
[29] T. Liu, Y. Zhao, R. Joshi, M. Khalman, M. Saleh, J. Liu, and J. Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657 2023.
[30] T. Liu, Z. Qin, J. Wu, J. Shen, M. Khalman, R. Joshi, Y. Zhao, M. Saleh, S. Baumgartner,
J. Liu, et al. Lipo: Listwise preference optimization through learning-to-rant arXiv preprint arXiv:2402.01878, 2024.
[31] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
[32] Y. Oren, S. Sagawa, T. B. Hashimoto, and P. Liang. Distributionally robust language modeling. arXiv preprint 1009:1909.0000 2019.
[33] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[34] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19153 2024.
[35] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[36] S. S. Ramesh, Y. Hu, I. Chaimalas, V. Mehta, P. G. Sessa, H. B. Ammar, and I. Bogunovic. Group robust preference optimization in feward-free rlhf. arXiv preprint 2024.
[37] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint
2019.
[38] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. 17493, 2023.
12

---

[39] F. Song, B. Yu, Li, H. Yu, F. Huang, Y. Li, and H. Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998, 2024.
[40] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances Neural Information Processing , 33:3008-388 3021, 2020.
[41] M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[42] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 13:18583-18599 2020.
[43] M. Völske, M. Potthast, S. Syed, and B. Stein. Tl: dr: Mining reddit to learn automatic summarization. Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63, 2017.
[44] K. Wang, J. Zhu, M. Ren, Z. Liu, S. Li, Z. Zhang, C. Zhang, X. Wu, Q. Zhan, Q. Liu, et al. survey on data synthesis and augmentation for large language models. arXiv preprint arXiv:2410. /289,, 2024.
[45] Wu, Y. Xie, Z. Yang. J. Wu, J. Chen, Gao, B. Ding, X. Wang, and X. He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint arXiv:2407.07888 2024.
[46] S. Xu, W. Fu, J. Gao, W. Ye, W. Liu, Z. Mei, Wang, C. Yu, and Wu. dpo superior to ppo for Ilm alignment? comprehensive study, arXiv preprint arXiv:2404. 10719, 2024.
[47] Z. Ye, F. Greenlee-Scott, M. Bartolo, P. Blunsom, J. A. Campos, and M. Gallé. Improving reward models with synthetic critiques. arXiv preprint arXiv:2405.208800 2024.
[48] Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint 2rX5v:2304.05302, 2023.
[49] R. Zhai, C. Dan, Z. Kolter, and P. Ravikumar. Doro: Distributional and outlier robust opti- mization. In International Conference on Machine Learning, pages 12345-12355. PMLR, 2021.
[50] M. Zhu, Y. Liu, L. Zhang, Guo, and Z. Mao. Lire: listwise reward enhancement for preference alignment. arXiv preprint arXiv:2405. 13516, 2024.
[51] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint 190v:0900.08593, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly depict the distribution shift problem in the current alignment paradigm and what this paper proposes to mitigate the problem.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have included Limitation part in the Discussion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The proofs and mathematical derivations are numbered and cross cross-referenced in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included the Algorithm settings in Appendix C
| iin the main paper and the hyper-parameters
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We finalizing the codebase and will release
finished.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We include detailed experimental setting and the hyper--arameters settings in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We include error bars for subset of experiments in Table [ illustrate variability, based on multiple runs. Due to the high cost of GPT-4o evaluations, we limit the use of error bars to these partial results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The relevant information is provided in Appendix
C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: btttaps://neuriiie www.rpsu.p.../. htcppps.ccpppppe cc/puullic/thhicccuueeeel.....
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discusses the broader impacts in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The licenses used in the paper stated in Appendix C and the adopted benchmarks are accordingly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
20

---

A Broader Impacts
As LLMs grow more capable, they also pose escalating risks insku-ineluding misinformation, deception, bias, and harmful content- potentially severe societal consequences. To steer model outputs toward human values and intentions, developing robust techniques for ethical alignment has become imperative. Significant research efforts now target ethical Al frameworks spanning data curation, algorithmic design, and deployment safeguards. Our work aims to advance this critical frontier, enhancing the safety and robustness of LLMs for real-world applications.
B
Mathematical Derivation
B.1 The worst-case distribution in KL-DRO
In KL-DRO, consider the following optimization problem:
max
Ee(e(x)) subject to Sp.
Q
where ((x) the loss function. Let À 2 0 be Lagrange multiplier for the KL constraint, we have the following Lagrangian relaxation:
L = Eq [<][((>] ( A(DxL(Q11P) p).
then substitute (QI(Q = =Eq In Q(x)) P(z)
c / Q())((a)) de
(foce)
Q(x) dr - P)
P(x)
Vary L with respect to (1x). The functional derivative is:
SC 8Q(x)
((x) A In (xx)
+ +1) P(œ)
Set this derivative to zero:
((x) - > (in P(x)
(In Q(x)
+ = 0.
We are left with
((x) (x)
1(0)
= e
P(x)
The Lagrange multiplier À is implicitly tied to For simplicity, redefine + 3.4 leading to:
Q(x) x (())e
B.2
Dual Optimization
We proceed with the robust objective in Equation 7 First we introduce a Lagrange multiplier À 2 0 for the KL constraint and p IR for the normalization constraint Ee[hh] 1. The Lagrangian becomes:
C(h, Ep [(()) h((z) h(z).((0,2) X (EE p) r (Er[h(z)] 1)
=Ep [(z)) (f(z))((, Alog h(z) 11) + Ap +
To find the optimal A", we solve the variational problem by setting the functional derivative of L with respect h(z) zero:
SC 8h(z) = h(z) z) A(1 + log h(z))
Solving for (z2), get:
21

---

h² (z) exp (1(z)) ((0,z) - -1 -
We denote the normalizing constant:
Z : Ep exp (Hile) ((0 (00.))
(²(2) z exp (H5lz)- e(0.2)
Now substitute he back into the original objective:
Ep(h (2) M(2) ((0, = RR Ep M(z). ((0,2) exp (Hh(e)) -(0(2))(0022)
Now the simplified dual expression can written as:
{ (Ap + Alog Ep exp (l(z) e(0, (5(2))((0.22))
-(0.2))]
inf Aso
B.3 Convergence Analysis
In this section, analyze the convergence properties of the DoRA formulation. In particular, show that under suitable conditions on the loss function (convexity and smoothness) and the density ratios, robust objective converges global optimum via gradient-based methods.
Robust Objective Formulation. First we define
1(0.2) z) A
)))
f(0) loo
Ep exp
The function S(0) is the well-known log-sum-exp (LSE) function, which is a smooth convex approxi- mation of the maximum. Assume that for every the loss function ((0, z) h(z)) 8(0, 2) is convex and has Lipschitz continuous gradient with constant L:
|V((0,,)) z) vI(02. z) s L 1 02ll. v01,02.
Since the exponential function convex and increasing, the mapping
1(0.
exp
A
convex for each Taking the expectation over P. obtain that
1(0, z)
( = Ep
exp
convex in Moreover, since the logarithm is monotonic transformation, f(0) = log (0) is also convex. The gradient of f(0) is given by
1 Ā
Ep
exp
[10.z)
vI(0,2) 2)
Vf(0)
Ep
cxp
[(0.x))
This expression can be interpreted as weighted average of VI(0, 2), where the weights
((0.z)
exp
(z
((9.z)
Ep
exp
form softmax distribution. Standard arguments for the LSE function then imply that (f( is Lipschitz continuous with constant L' (which depends on and X). Consequently, the scaled function
F(0) Xh f(0)
22

---

is both convex and smooth. By applying gradient descent with an appropriate constant step size ? 1/L', we obtain the convergence guarantee:
F(OT) - F(0)) - 11
27
where Or is the parameter after iterations, 0 is the global minimizer of F(0). and 0g is the initial parameter. This result ensures that DoRA converges general convex case. Moreover, DoRA follows the (non-) coon-xitvyxxty properties of the baseline, and since it be viewed an LSE transformation of the baseline, its convergence behavior is expected to be similar. Specifically, DoRA converges at the same rate of O(1/T) as the baseline in the convex setting. In non-convex cases, while the convergence guarantees may weaker, the convergence trajectory is anticipated to be comparable, with the LSE transformation potentially affecting factors like smoothness or step-size dependence but fundamentally altering the convergence order.
global optimum rate of O(1/T) in the
C Implementation Details
C.1 Data Generation
In this section, we introduce the data generation pipeline and how we develop a controlled setting under mixture response shift. Specifically, we sample 2 additional synthetic responses using Mistral- 7B-Instruct-vO.3 leveraging the queries the original dataset. The temperature is set to 0.8 and repetition penalty is set to 1.1 during sampling. Then combine the 2 synthetic responses well as the pairwise responses from the original dataset, leading to 4 responses in total for each query. All the datasets subject the terms of the MIT License, except for the AlpacaEval benchmark which subject to the Apache-2.0 license. All these datasets and benchmark are utilized in accordance with their intended purposes.
Reward models
Task
Preference Accuracy (%)
HH-trained GPT-J Sum-trained DeBERTa-larg
HH-RLHF
74.6
Summarization
89.2 Ultrafeedback
UltraFeedback trained UltraRM- 13B
74.6
Eurus-RMM76 (top 10 for Chat)
HH-RLHF
65.0
Table 7: Preference accuracy of various reward models on the corresponding tasks.
C.2
Reward models
For augmented listwise contrasts, we utilize proxy reward models GPT-J DeBBRTa-large and UltaRM-13B to score the dialogue, summarization and UltraFeddbckk datasets, respectively. Note that the reward models are intentionally trained for these tasks, with very high accuracy distinguishing between good and bad samples shown Table 7 We also observe that while the top-ranked models on the RewardBench leaderboard are trained on mixed datasets and may perform better on more challenging or diverse tasks, they may not necessarily outperform the selected reward models by great margin under our settings. This indicates that these selected reward models demonstrate sufficient discriminative power for our specific experimental context.
Hyperparameters
DPO
R-DPO
EXO
SimPO
DPOM.
RRHF
LIRE
HH-RLHF
8=0.1
B=0.1,
a=0.01
$=0.5
8=2.0,y=0.8
8=0.1
a=1.0
T=2.0
Summarization
3=0.5
a=0.5
T=1.0
UltraFeedback
3=0.1
a=1.0
T=2.0
Table 8: Hyperparameters for different baselines and tasks.
23

---

C.3 Hyperparameter settings
General training settings. We begin by performing SFT on the selected responses for each task, following the default hyperparameter configurations from the DPO codebase, All experiments are run on 80GB A100 GPUs with a batch size of 32. For pairwise training, we adopt learning rate of 3e-7 for the Mistral-7B base model and 6e-7 for the LLLaa-8B base model, in line with Meng al. [31. In the listwise setting, we apply LoRA with a learning rate of 2e-5 for Mistral and 4e-5 for Llama. For Ulraaeeddback, we default to full fine-tuning. For classifier training, we set the learning rate 2e-5 and train for epochs.
Baseline-specific hyperparameters. Table summarizes the hyperparameter settings used for different models and tasks. Whenever available, we adopt the default values specified in the original papers. In cases where default values are not provided, we conduct preliminary experiments with range of hyperparameter choices and select the configuration that yields the best performance. For instance, in the Summarization task, we find that setting a to 0.5 yields much better results than keeping it as 1.
Decoding hyperparameters. We adopt fixed sampling strategy across all experiments to ensure consistency in response generation. Specifically, we set the temperature to 0.8, top-k to 50, and top-p to 0.9 during sampling. For maximum new tokens, we use 128 for dialogue and 512 for summarization tasks, while setting 1024 for the AlpacaEval 2.0 benchmark and 4096 for Arena-Hard bench.
C.4 Evaluation prompts
For HH and Summarization tasks, adapt the evaluation prompts from Rafailov et al. [35 using GPT-40, and compute the win rates and lose rates with 400 randomly selected test queries, with the order randomly swapped.
[H--LLFF]: For the following query to chatbot, which response is more helpful?
Query: <the user query>
Response A: <response A>
Response B: <response B>
FIRST, state only 'A' or 'B' to indicate which response is more helpful, state 'C' if its tie. SECOND, provide mee-sentence comparison of the two responses and explain which you feel is more helpful. Your response should use the format: More helpful: <' A'A' or 'B' or 'C'> Comparison:
<one-sentence comparison and explanation
(Summarization]: Which of the following summaries does better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise.
Post: <post>
Summary A: <summary A>
Summary B: <summary B>
FIRST, state only 'A' or 'B' to indicate which summary is preferred, state 'C' its tie. SECOND, provide one-sentence comparison of the two summaries, explaining which
24

---

you prefer and why. Your response should use the format: Preferred: <'A or 'B" or 'C'> Comparison: <one-sentence comparison and explanation
C.5
Baseline objectives
In this paper, we primarily focus on three baseline methods in preference alignment that employ list-wise contrastive optimization. Each of these methods optimizes distinct objective function designed to enhance alignment with human preferences. The mathematical formulations for these optimization objectives are presented below, and refer readers the original papers for a more detailed discussion:
DPOpL [35] (derived under the Plackett- Luce model) avoids explicit reward modeling by directly optimizing the policy using a ranking-baas baaseng-bsedddd loss. The loss encourages the policy ny to assign higher probabilities preferred responses relative to reference policy Tee
exp (glog
na (a (M) Fud(Br(4))
(po
log II k=1
K j=k
exp 3 log
(V-
combines ranking loss with an SFT loss. The ranking loss ensures that higher-ranker responses have higher probabilities, while the SFT loss ensures the model stays close to the reference behavior:
RRHF [8
Σ, log To (V(, Vi,se)
pj
||
IRRIF
max(0, Pÿ) ISFT-
R Rg
LIRE [50] uses a softmax-bes based probability distribution over responses, weighted by their rewards, to guide the policy. The loss encourages the policy assign higher probabilities to responses with higher rewards:
exp (+ log To ( (y)=))
Pao (y|E, A)
Σ exp ( log To (y)))=))
IIRE
Pao (yy x. A)Rj.
where represents the loss objective of the methods and K is the number of responses for each query.
These baselines provide a strong foundation for evaluating preference alignment techniques, allowing us to systematically assess the effectiveness of DoRA in mitigating distributional shifts and enhancing robustness.
D Qualitative analysis
In this section, we provide some model answers aligned with different strategies for a better under- standing. Specifically, we show real Mistral model generations from the HH-RLHF dataset, where incorporating DoRA provides responses that align better with target answers.
25

---

HH-RLHF
Query:
Human: How do you make your own pizza dough? Assistant:
Golden
You start with flour, salt, and water. You mix the ingredients together until you get a dough you can work with. Then you knead the dough to make it smooth.
PP
To make homemade pizza dough, you'l want to start with warm water and yeast. You'll want to add flour, salt, and oil to the water and yeast, and mix everything together until you have a smooth dough. You'll want to knead the dough on a floured surface until it's smooth and elastic. Then you'll IIl'l want roll it out into thin rectangle, and place it on a baking sheet. You'll II want to brush the dough with olive oil, and sprinkle it with garlic powder and salt. Then you'll want to top the dough with... To make homemade pizza dough, you need to combine flour, water, salt, and yeast. You can then knead the dough for a few minutes until it becomes elastic. Then you can shape the dough into round, place it on a baking sheet, and top it with your favorite pizza toppings. Bake the pizza in a hot oven until the crust is golden brown and the cheese is melted.
w/ DoRA
RRHF
I've heard that there are some people who make their own pizza dough. I'm not one of them. I don't I think it's too difficult, but I think it's probably more trouble than it's worth. just buy store-bought dough. I've heard that there are some people who make their own pizza dough. They often say it's better than store-bought. It's just flour, water, salt, and yeast, so you can easily make it yourself.
w/ DoRA
LIRE
To make your own pizza dough, start by mixing together flour, salt, and yeast in bowl. Then, add water and oil, and mix until smooth dough is formed. Next, knead the dough on a floured surface until it is smooth and elastic. Finally, divide the dough into two equal balls, and let them rise for 30 minutes. After that, you can roll out the dough and top it with your favorite pizza toppings!
w/ DoRA
To make your own pizza dough, start by mixing together flour, salt, and water in a large bowl. Then, knead the dough until it is smooth and elastic. Finally, let the dough rise for about 30 minutes, and then you can roll it out and top it with your favorite toppings.
Table 9: Given the query on how to make a pizza dough, the golden response is brief and simple. By incorporating DoRA, the responses tend to be more concise and resemble the target response more compare to the baselines, giving more direct and brief answers.
D.1 Loss during the Optimization Process
In this section, we visualize the training loss between baselines and DoRA. Figure 7 illustrates the training loss curves for DPOpL and DoRA with Mistral model across dialogue and summarization tasks. We observe that both methods exhibit stable convergence over iterations despite different loss scales, suggesting that DoRA enforces different learning trajectory compared to standard training.
Training Loss Over Iterations
Training Loss Over Iterations
Baseline Train Loss
Baseline Train Loss
DoRA Train Loss
0.7
DoRA Train Loss
1.0
Loss Value
0.6
0.5
0.6
200
400
800
1000
1200
200
400
800
1000
1200
Step
Step
(a) HH
(b) Summarization
Figure 7: Training Loss Comparison Between Baseline and DoRA. This figure shows that DoRA
exhibits stable convergence over iterations, despite different loss values compared standard training.
26

---

E Further discussion on RSO
As mentioned earlier in the related work, key issue in offline Maximum Likelihood Estimation (MLE) training is the mismatch between the sampling distribution and the learning policy. This arises because the maximum likelihood estimator of the target optimal policy requires labeled prefer- ence pairs sampled from that policy. To address this, Statistical Rejection Sampling Optimization (RSO) (29] employs rejection sampling to source preference data from the estimated target optimal policy, thereby improving the accuracy of policy estimation during training.
It is worth noting that RSO tackles data bias from different perspective compared to this paper. Specifically, it aims to make the optimization process more "on-policy" by sourcing preference data that better aligns with the estimated target optimal policy during MLE. Rejection sampling used to approximate the distribution of preferred responses by filtering samples from a proposal distribution (e.g., the current policy ı based on a preference model.) While we in this paper focus on mixture response shift and the bias in synthetic data compared to human-pret human-preferred responses.
Despite targeted on different perspectives, thought would be intriguing to compare these two methods. In preliminary experiments, we adapted the RSO technique by sampling 8 responses per prompt (compared to 64 in the original RSO paper due to computational constraints) from Alpaca the HH-RLHF dataset, then selected 4 responses for downstream training with Mistral hase model. As shown in Table I0 while RSO outperforms baseline methods in some cases (c.g., RRHF), consistently underperforms DoRA and incurs significant computational overhead from additional generation sampling.
HH-RLHF
Baselines
Methods
Win(t)
Lose(L)
RSO
54.0
8.5
DPOpt
DoRA
58.5
9.5
RSO
46.5
RRHF
DoRA
43.5
18.0
RSO
56,0
13.5
LIRE
DoRA
57.5
11.0
Table 10: Comparison of RSO and DoRA on HH-RLHF. Results show that DoRA generally
outperforms RSO across DPO and LIRE baselines, achieving higher win rates and lower lose rates.
27


Reviews:
[{"review_id": "QtIcZOtHsI", "reviewer": "Reviewer_2E9q", "summary": "The paper consider the problem of robust preference alignment in LLM. The approach is to first learn a classifier to assign a calibration value to each training sample, and then leverage it to the target human-preferred distribution. The objective is formulated as a distributionally robust optimization with respect to KL divergence.", "questions": "1. Construction of $\\tilde h(z)$ is not completely clear to me. How does one compute the constants $\\gamma_j$ and $\\beta_j$?\n\n2. How well does this method generalize to datasets not used for training? Did the authors test the aligned model on a dataset that has not been used for training (not considering train-test split)?\n\n3. I see that golden/reference distribution is approximated using trusted human-labeled data. Clarifying questions, is the \"golden\" distribution same as the target/preferred distribution? How one may obtain such a data?", "limitations": "Please see the **Strengths And Weaknesses** and **Questions** sections.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** The paper consider a distribution-aware optimization framework to robustly align LLM outputs with human preferences under distribution shifts. The framework relies on the mixture response shift assumption, which posits that the training data comprises a mixture of heterogeneous sub-distributions. Then the procedure is broken into two parts: 1. employ a calibration score to each training sample, estimating its alignment to human preferences, and 2. integrate the calibration scores to a distribution-aware optimization objective that minimizes the worst case loss. \n\n**Weaknesses:** I have some concerns about clarity of the expositions and generalizability of the experimental results. Please see my questions in the **Questions** section.", "was_revised": true, "final_justification": "The authors have address my comments, so I have raised my evaluation score.", "rebuttal": "Thank you for the insightful comments and helpful suggestions! We hope that our responses below help alleviate the concerns that were raised.\n\n> $Q_1$. \"Construction of $\\tilde{h}(z)$ is not completely clear to me. How does one compute the constants $\\gamma_j$ and $\\beta_j$ ?\"\n\nThanks for the detailed review. We clarify the definitions of $\\beta_j$ and $\\gamma_j$ as follows:\n\n  $\\beta_j$ denotes the **relative propotion** of sub-distribution $Q_j$ in the dataset;\n\n $\\gamma_j$ is defined as the **imbalance ratio** between a synthetic distribution and the golden distribution $\\frac{q(y=0)}{q({y=1})}=\\frac{\\beta_j}{\\alpha}$.\n\n   For example, suppose for a given query, the responses set includes: 1 response from golden distribution $Q_0$, 2 responses from $Q_1$, and 3 responses from $Q_2$, then $\\beta_1=\\frac{2}{1+2+3}=\\frac{1}{3},\\gamma_1$ = $\\frac{\\beta_1}{\\alpha}=\\frac{\\frac{1}{3}}{\\frac{1}{1+2+3}}=2$. Then $\\tilde{h}(z)$ can be computed according to Eq. (6), where $c_{\\phi_j}$ represents the score from the learned classifier for sub-distribution $Q_j$. \n\n>$Q_2$: \"How well does this method generalize to datasets not used for training? ...\"\n\nThank you for the insightful question. **Table 3 (line214)** reports results on **AlpacaEval 2** and **Arena-Hard** using models trained on **UltraFeedback**, where the test queries come from a different distribution to the training data. These results demonstrate that DoRA generalizes effectively when evaluated out-of-distribution. \n\nWe *further* evaluate generalization capability under the following scenarios:\n(1) Training on HH-RLHF and testing on summarization (different tasks), and\n(2) Training with label noise (40% of labels randomly shuffled).\nAs shown in the table below, DoRA improves baseline performance in all cases, demonstrating robust generalization.\n\n| Training data | HH-RLHF       | UltraFeedback (w/ noisy label) |              |\n| ------------- | ------------- | ------------------------------ | ------------ |\n| Test Data | Summarization | AlpacaEval 2                   |              |\n| Metrics       | Win rate(%)   | Length-controlled Win rate (%) | Win rate (%) |\n| DPO           | 36.3          | 11.00                          | 10.99        |\n| +DoRA         | 39.0          | 12.07                          | 12.06        |\n| RRHF          | 23.5          | 7.67                           | 4.84         |\n| +DoRA         | 27.5          | 8.50                           | 5.51         |\n| LIRE          | 43.8          | 26.52                          | 25.06        |\n| +DoRA         | 49.5          | 27.65                          | 26.18        |\n\n>$Q_3$. \"Clarifying questions, is the \"golden\" distribution same as the target/preferred distribution? How one may obtain such a data?\"\n\nYes, in our setting, the “golden” distribution corresponds to the positive class when training the probabilistic classifiers. It is constructed with samples drawn from the target/preferred distribution $Q_0$.\nIn practice, we approximate this golden distribution using \"chosen\" responses from trusted, high-quality preference datasets, where preferences are made either by humans or by a very strong LLM such as GPT‑4. These responses serve as a reasonable proxy for the target distribution.\n\n   Thank you again for the detailed review and insightful feedback, and we will include these clarifications in the revised manuscript. We believe your input has already helped improve the paper and look forward to engaging with you further during the discussion period."}, {"review_id": "PuwFqd2B1l", "reviewer": "Reviewer_1ReX", "summary": "The paper introduces DoRA (Distribution-aware Optimization for Robust Alignment), a framework aiming to address the LLM alignment in the presence of distribution shifts. Taking the modern trend of using synthetic preference data as an example, the paper propose the method that model remains resilient to distribution shifts between the training data and the target distribution while still benefiting from their scalability.\nThe authors propose a two-phase approach that first uses distribution classifiers to estimate the likelihood that training samples originate from the target distribution Q₀, then incorporates these calibration scores into a KL-divergence-based Distributionally Robust Optimization objective to reweight the loss function toward samples better aligned with the target distribution.\nEmpirical evaluation across multiple alignment tasks demonstrates that DoRA consistently improves model performance and win rates compared to various baseline approache", "questions": "- Could the authors elaborate more on the conceptual (and technical) novelty of introducing a sample-level classifier within the DRO framework? In particular, how does this differ fundamentally from prior work, and what capabilities does it enable beyond existing methods?\n- How does DoRA perform under adversarial distribution shifts or highly corrupted synthetic data? An ablation study with listedwise dataset from LLMs of varying quality might be helpful.\n- Although the author mentioned about the flexibility of the proposed method with diverse distributional shift, the author only demonstrated on listwise preference setting with synthetic response. Other source such as combining different datasource with different quality, online updates would help strengthen the contribution.\n- Since DoRA emphasizes Q₀-labeled samples, how do the authors ensure that this does not amplify label noise or miscalibration? What happens when classifier calibration itself is unreliable?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The proposed framework is well-motivated, addressing a crucial challenge in alignment training in the divergence between synthetic and target distributions.\n- The expansion beyond traditional pairwise noise robustness to broader distributional concerns is a timely and necessary shift in the alignment literature.\n- The proposed method is method-agnostic and can be integrated into diverse alignment objectives, including different synthetic sources or online updates\n- The writing is generally clear and well-structured. Key concepts such as mixture response shift, KL-DRO, and the calibration mechanism are explained clearly. The proposed method is demonstrated in a both principled and practical way.\n- Experiments span pairwise and listwise settings across multiple datasets and models, showing consistent (although sometimes modest) gains in alignment quality.\n\n### Weaknesses\n- The contribution of introducing new sample-level classifier on existing method DRO offers limited novelty and add marginal contributions. \n- While the paper provides theoretical motivation, the empirical gains are sometimes marginal (e.g., +0.6% in win rate for DPO with Mistral-7B), raising questions about cost-effectiveness. The added complexity of training additional classifiers and additional inference might not be justified.\n- There is insufficient clarity on how DoRA’s calibration is different from reward models. Both attempt to score sample quality and may collapse to a combined reward models. The classifier based approach still hold the limitation of reward models that are prone to biases, potentially leading to suboptimal labelling. The authors should discuss this distinction more clearly.\n- The approach might amplify label noise rather than mitigate it, since it emphasizes Q₀ without validating whether those labels are truly correct.\n\nMinor feedback on Clarity:\n- The details about win rate (e.g. using GPT-4o) is introduced first in Table 2. Earlier context would help orient readers.\n- Figure 1 misleadingly depicts $Q_₁$ as easily separable from $Q_₀$, contradicting evidence that modern LLMs approximate human preferences well. In addition, unclear definition of $Q_1$ makes P unclear in Figure1, which prevent it from delivering the clear picture of overall method and its strength.\n- notation of $h(z)$ vs. $h^{\\sim} (z)$ is confusing, as one refers to the Radon-Nikodym derivative and the other to the calibration term.", "was_revised": true, "final_justification": "The concern about noise from the newly introduced learned classifier is partially addressed by experimental results. The author demonstrates that the classifier's performance gains outweigh its drawbacks, supported by the rationale of the stabilizing factor and empirical results on Q2, Q4, and Q5. Assuming the proposed revisions and additional experiments are included in the final version, I am updating my score slightly more positively.", "rebuttal": "Thanks for the insightful comments and helpful suggestions! We hope that our responses help alleviate the concerns that were raised.\n\n> $Q_1$: \"The contribution of new sample-level classifier on existing DRO framework\"\n\nThanks for asking. Our method is not merely a straightforward extension of DRO by adding a classifier. Instead, it introduces a fundamentally new formulation and training objective for robust alignment under synthetic data mixtures:\n\n(i) **Problem reframing**: We step beyond traditional empirical risk minimization (ERM) approaches used in alignment, and formally define a new robustness setting: mixture response shift. This setting, though increasingly common in practice due to the prevalence of synthetic data, has been *under-explored* in the literature.\n\n(ii) **Robust objectives with calibration factor**: Building on this, we derive the robust objective for existing alignment methods by introducing a *calibration factor* into the uncertainty set, where the probabilistic classifiers are leveraged to compute these factors.\n\n(iii) **Closed-form solution with over-pessimism mitigation:** The resulting objective admits a *closed-form dual solution*, and directly mitigates the over-pessimism commonly observed in classical DRO, where the model can over-focus on outliers or noisy examples. \n\nWe also conclude some key differences with previous DRO literature in the following table.\n\n| Method              | Robust to                                                    | Optimization Focus                                           | Granularity of trust | pairwise & listwise? | Over‑pessimism control                        |\n| ------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------- | -------------------- | --------------------------------------------- |\n| **Standard KL‑DRO** | General distribution shift          | Worst-case loss over the entire training distribution        | Global               | Pairwise only        | No (often over‑pessimistic)                   |\n| **CVaR-DRO**        | Subpopulation shift in covariates ($\\mathcal{X}$)            | Worst-case loss over any $\\alpha-$sized subset of the training set | Per‑group            | Pairwise only        | Limited (moderates extremes but still coarse) |\n| **DoRA (ours)**     | Response/output shift in $\\mathcal{Y}$ (mixture of output distributions per query) | Worst-case loss over target distribution alignment           | Per‑sample           | Yes                  | Yes                                           |\n\n> $Q_2$: \"Performance under adversarial distribution shifts/highly corrupted synthetic data?\"\n\nThis is a good point. We focus on “clean” benchmark datasets (no synthetic corruptions) in the main paper, but we are happy to add the following corrupted-label experiments to serve as stress tests to confirm DoRA’s ability under adversarial shifts. Specifically, we introduce corruption to the data by randomly flipping the labels. Experiments show that incorporating DoRA consistently improves robustness under increasing label corruption, yielding consistent gains on different benchmarks.\n\n   | Dataset               | HH       |          |          | AlpacaEval 2 |          |\n   | --------------------- | -------- | -------- | -------- | ------------ | -------- |\n   | Corruption rate| 20%      | 40%      | 60%      | 40%          | 40%      |\n   | Metrics| Win rate (%)| Win rate (%)| Win rate (%)| Length-controlled Win rate (%) | Win rate (%)|\n   | DPO | 71.0     | 64.5     | 57.3     | 11.0         | 11.0     |\n   | +DoRA| 74.5     | 67.0     | 60.8     | 12.1         | 12.1     |\n   | RRHF | 63.5     | 42.3     | 26.3     | 7.7          | 4.8      |\n   | +DoRA  |65.5     | 44.5     | 30.8     | 8.5          | 5.5      |\n   | LIRE    | 67.7     | 52.5     | 61.4     | 26.5         | 25.1     |\n   | +DoRA   | 71.5     | 55.0     | 65.3     | 27.7         | 26.2     |\n\n We will add these new experimental results to the revised manuscript and highlight them in the discussion.\n\n>$Q_3$: \"how DoRA’s calibration is different from reward models.\"\n\nDoRA's calibration is fundamentally different from reward models, and the two operate on **different axes** of the alignment pipeline. First, DoRA relies on a learned probabilistic binary classifier which estimates *distribution membership*, not *human-preference utility*. While reward models directly score “how good a sample is”, DoRA’s classifier scores \"how likely a sample comes from the target distribution\". Therefore, DoRA’s calibration factor is a **density ratio proxy**, not a reward score. Its role is to modulate the DRO uncertainty set (to determine how much robustness weight to assign to each sample).\n\n We agree that any learned model can be biased if trained on unrepresentative or mislabeled data. However, reward model bias directly impacts what the model learns to generate. If it reflects skewed or noisy preferences, it can misdirect optimization and lead to alignment failures.\nIn contrast, our classifier only **distinguishes between sources** (e.g., human vs. synthetic), enabling selective robustness without steering generation itself. Moreover, the impact of classifier miscalibration is **bounded**.\n   As we describe in Line 146 and Eq. (6), we introduce a *stabilizing factor* (e.g., a small additive term in the denominator) that prevents any individual sample from dominating the optimization. This ensures that even overconfident classifier outputs do not result in extremely unstable or biased learning.\n\n >$Q_4$:\"performance on combining different datasource with different quality, online updates\"\n\nThanks for raising this point. While the primary **focus** of this paper is on **mixture response shift with synthetic data** in offline training, we briefly mention *online updates* as a promising direction in “Limitation and Future Work” section.\n\n To illustrate DoRA’s robustness in such scenarios, we conduct a small-scale experiment: we train LLama3.2‑3B on the HH dataset (Iter 1), then continue training on its own generations for two more iterations. This setup induces an evolving mixture shift as the response distribution becomes increasingly synthetic. The results show that DoRA remains effective under this shift, suggesting that its instance-level calibration also generalizes to online or self-generated data settings, where distribution drift occurs naturally.\n\n   |       | Iter 1 | Iter 2 | Iter 3 |\n   | ----- | ------ | ------ | ------ |\n   | DPO   | 77.0   | 83.5   | 85.0   |\n   | +DoRA | 78.5   | 85.8   | 88.0   |\n   | RRHF  | 45.8   | 47.8   | 50.5   |\n   | +DoRA | 47.5   | 49.3   | 52.5   |\n   | LIRE  | 80.3   | 83.0   | 84.5   |\n   | +DoRA | 82.5   | 85.0   | 86.8   |\n\n>$Q_5$: \"how do the authors ensure that this does not amplify label noise or miscalibration? What happens when classifier calibration itself is unreliable?\"\n\nThanks for asking, and we try to answer this from the following two perspectives.\n\n1. \"Label noise\": We first clarify that $Q_0$ denotes the **target distribution** (trusted human-preferred responses from **known** sources/ground truth), while label noise in the alignment context typically refers to incorrect “chosen/rejected” annotations.\nFrom this perspective, if label noise exists in the *original dataset*, DoRA can actually **mitigate its impact** by downweighting those samples that are distributionally misaligned with the target (i.e., with low calibration scores).\n\n2. \"Classifier accuracy\": If the concern instead refers to errors in the classifier itself, we agree that any learned model may be imperfect. However, we observe that our lightweight classifiers achieve consistently high accuracy across datasets, as shown below, indicating reliable calibration in practice:\n\n|  Dataset        | HH-RLHF | Summarization | UltraFeedback |\n| --------- | ------- | ------------- | ------------- |\n| Test Acc. | 94.8%   | 93.9%         | 92.7%         |\n\n\n\n>\"Minor feedback on Clarity\"\n\n   1. Thanks for pointing this out and we will place the explanation of \"Win Rates\" earlier in the main paper.\n\n   2. >\"misleadingly depicts  $Q_1$ as easily separable from $Q_0$, contradicting evidence that modern LLMs approximate human preferences well.\" \n\nThank you for the feedback. We clarify that **Figure 1 illustrates distributional separability, not preference similarity.** While it is true that modern LLMs (e.g., GPT‑4) may approximate human preferences in aggregate utility—as reflected by similar reward scores—this does **not** imply that their response distributions are identical. \nIn fact, we find that $Q_1$ (synthetic responses) and $Q_0$ (human-preferred responses) often exhibit distinct generation patterns, which are learnable by a classifier with over 90% accuracy in our setting. This supports the distribution-level separability shown in Figure 1.\n\nWe will revise Figure 1 to (i) explicitly clarify the meaning of $Q_1$ and $P$ in the caption and (ii) adjust the visual to better reflect realistic overlaps while still communicating the method’s key intuition.\n\n  3. >\"notation of $h(z)$ vs. $\\tilde{h}(z)$ is confusing...\"\n\nWe clarify that $h(z)$ denotes the Radon-Nikodym derivative $\\frac{d Q_0}{dQ}$, which expresses the ideal density ratio between the two distributions, but is generally *incomputable in practice*. The symbol $\\tilde{h}(z)$ refers to our *practical, learned approximation* of $h(z)$, obtained via a calibrated probabilistic classifier.\n\n  Thus, the notation distinction emphasizes that $\\tilde{h}(z)$ is an implementable surrogate, not the true derivative. We will clarify this in the final version by explicitly linking the two in notation and explanation.\n\nThanks again for the questions, and we hope the above responses can address the confusion or misunderstanding. We are happy to make further clarifications and we welcome further discussion."}, {"review_id": "i86TmwHqeF", "reviewer": "Reviewer_Jipr", "summary": "This paper proposes DoRA (Distribution-aware optimization for Robust Alignment) to address distribution mismatch in preference learning. The core issue is that maximum likelihood estimators for optimal policies require samples from the target distribution, but training data typically contains a mixture of human-generated and LLM-generated responses from various sources.\n\nDoRA uses distributionally robust optimization to handle this distribution shift. The method trains classifiers to identify the source distribution of each sample, then incorporates these predictions as importance weights in a robust optimization objective. This transforms the standard sum-of-log-exponentials loss into a log-sum-exponential form that minimizes worst-case expected loss. The approach is method-agnostic, working with various preference learning algorithms beyond DPO.\n\nExperiments on HH-RLHF and UltraFeedback datasets using Mistral-7B and Llama-8B show improvements over baselines like Robust DPO and Dr.DPO across most settings.", "questions": "* (traditionally) Log-sum-exponential objectives are notoriously difficult to optimize, prone to numerical instability and vanishing gradients. While the authors add a 1/n stabilizing term, insufficient discussion of **practical** optimization considerations may hinder reproducibility\n* Consider renaming the method, as \"DoRA\" is already used for a LoRA variant in parameter-efficient training.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- **Technically sound approach** with clear motivation addressing a practical problem in preference learning\n- **Method-agnostic approach** that extends beyond DPO to various preference learning algorithms\n- **Well-organized presentation** with focused contributions and thorough experimental evaluation\n\n## Weakness\n* **Limited practical gains vs. complexity**: Requires training separate BERT classifiers for each source distribution, adding significant implementation complexity for small improvements on ArenaHard and AlpacaEval 2.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and helpful suggestions!  We are grateful for the positive rating and hope that our responses alleviate the questions that were raised.\n\n> $Q_1$: Limited practical gains vs. complexity: Requires training separate BERT classifiers for each source distribution, adding significant implementation complexity for small improvements on ArenaHard and AlpacaEval 2.\n\nThank you for raising this point. We acknowledge that DoRA introduces some additional implementation overhead due to training source-specific classifiers. However, these classifiers are lightweight, fast to train, and only trained once per source. Their outputs can be **pre-computed and cached**, resulting in **negligible overhead during policy optimization**.\n\nIt’s also important to note that our current experiments are based on relatively clean datasets, which may understate DoRA’s practical utility. As a robustness control method, DoRA’s benefits become more apparent under *noisy or significantly-shifted distributions*. We show this in our stress tests (see the table below), including: 1. larger shifts in response (we use a model distributionally far from the target (Alpaca‑7B) to generate the synthetic responses. 2. larger label noise (preference labels are randomly corrupted in 40% of the training set).\n\nIn both cases, DoRA consistently improves baseline performance on AlpacaEval 2. We expect its value to be even greater in real-world or evolving deployment scenarios, where data often includes noisy synthetic outputs or imperfect human annotations. Overall, DoRA provides strong robustness gains at a modest and manageable implementation cost.\n\n| AlpacaEval 2 | Larger shifts in response space |              | Larger label noise             |              |\n| ------------ | ------------------------------- | ------------ | ------------------------------ | ------------ |\n| **Methods**  | Length-controlled Win rate (%)  | Win rate (%) | Length-controlled Win rate (%) | Win rate (%) |\n| DPO          | 5.92                            | 5.50         | 11.00                          | 10.99        |\n| +DoRA        | 6.61                            | 6.03         | 12.07                          | 12.06        |\n| RRHF         | 3.16                            | 2.79         | 7.67                           | 4.84         |\n| +DoRA        | 4.73                            | 3.99         | 8.50                           | 5.51         |\n| LIRE         | 6.51                            | 7.47         | 26.52                          | 25.06        |\n| +DoRA        | 14.94                           | 11.75        | 27.65                          | 26.18        |\n\n\nWe are happy to include this in the discussion and provide more ablation/analysis in the updated version. Thanks again for the feedback.\n\n> $Q_2$: \"insufficient discussion of practical optimization considerations\"\n\nWe appreciate this thoughtful comment. To reduce the numerical instability of log-sum-exp–based objectives, we adopt a stabilizing term $\\frac{1}{n}$ as an offset to the denominator (Line 149). This ensures that $\\tilde{h}(z)$ is bounded in $(0,n)$, ensuring no single sub-distribution can dominate training. Besides, we also leverage the LSE trick by subtracting the max value before exponentiating to maintain numerical stability in practice. These techniques help stabilize training across all tasks in our experiments. We will include more details in the updated version, and we are happy to release the code to ensure reproducibility.\n\n>$Q_3$: \"Consider renaming the method, as \"DoRA\" is already used for a LoRA variant in parameter-efficient training.\"\n\nThanks for the kind advice. We will consider another name to avoid confusion in the updated version.\n\n Thanks again for the questions, and we hope the above responses can mitigate the concerns raised. We are happy to make further clarifications and we welcome further discussion."}, {"review_id": "Vr1j5PQPlr", "reviewer": "Reviewer_8Cax", "summary": "This paper proposes DoRA (Distributionally Robust Alignment), a new preference learning framework inspired by distributionally robust optimization (DRO), applied to LLM alignment. The key idea is to model preference optimization as minimizing regret over a worst-case distribution shift between the original preference dataset and the LLM’s evolving policy. To do this, the authors train classifiers to estimate the density ratio between the preferred and dispreferred responses, use those to compute importance weights, and apply a regularized regret loss to update the model. The method is evaluated primarily on the HH-RLHF dataset with LLaMA-7B, showing improved win rate over DPO and SimPO.", "questions": "Can the authors clearly define how the HH dataset win rate is computed? Is it from GPT-4 comparisons, pairwise preferences, or external rankers?\n\nWhat specific role does the classifier play in estimating the density ratio? How is it trained and validated? More clarity and ablations are needed here.\n\nWhy is the method described as LLM-specific? Apart from the use of preference data from LLM outputs, the algorithm seems agnostic to language modeling.\n\nCan the authors provide evaluation on more diverse benchmarks (e.g., MT-Bench, Arena-Hard, AlpacaEval)? One win rate metric is insufficient to support alignment claims.\n\nGiven the added complexity, how does DoRA perform under simpler ablations? For example, removing the density ratio weighting or using a simpler robustness penalty.", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The formulation is grounded in well-studied robust optimization theory.\n\nWeaknesses:\n\nDisconnect between method and LLM context:\nAlthough the paper positions itself as a contribution to LLM alignment, the methodology appears to be a generic instance of distributionally robust optimization (DRO) applied to a preference dataset. The classifier-based density ratio estimation and reweighting procedure is standard in DRO literature, and the paper does not explain how the unique challenges of LLM alignment inform its algorithmic design. There is little adaptation to the sequential or structured nature of LLM outputs. As a result, the method feels like a transplant from a different domain rather than a carefully tailored alignment strategy.\n\nUnclear evaluation metric:\nThe primary metric reported in the experiments is “HH dataset win rate,” but the paper fails to explain how this metric is computed. Is the win rate measured via GPT-4 preferences, a learned reward model, or some internal heuristic? What exactly constitutes a \"win\" in this setting? Against which model? Are responses sampled once per prompt or averaged over multiple runs? Without a clear description, it's hard to judge the reliability or fairness of this metric. Given that the win rate is the only reported outcome, this lack of transparency significantly weakens the empirical claims.\n\nLimited empirical validation:\nThe evaluation is restricted to a single dataset (HH-RLHF) and a single win rate metric. No comparisons are made on other standard benchmarks used for LLM alignment evaluation, such as MT-Bench, AlpacaEval, Arena-Hard, or Open LLM Leaderboard metrics. Additionally, there is no breakdown of performance across prompt categories or qualitative error analysis. This narrow evaluation makes it hard to judge whether the proposed method offers any practical advantages in broader or more realistic alignment settings.\n\nUnnecessarily complex and opaque setup:\nThe proposed method requires training separate classifiers to estimate the density ratio between preferred and dispreferred responses. These are then used to reweight training samples when optimizing the policy. However, the motivations for these design choices are not clearly justified, and the pipeline introduces substantial complexity. The use of SFT initialization is briefly mentioned but not analyzed. Overall, the system feels overengineered relative to the simplicity of existing approaches like DPO, and it is unclear whether the added complexity yields meaningful or robust benefits.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the questions, and we hope the following responses can address the confusion or misunderstanding.\n\n>$Q_1$: \"Can the authors clearly define how the HH dataset win rate is computed? Is it from GPT-4 comparisons, pairwise preferences, or external rankers?\"\n\nAs noted in line 550, we use **GPT-4o** to compute win rates according to the provided evaluation prompt. To improve clarity, we will move this specification to the “Experimental Settings” section in the main paper.\n\n>$Q_2$: \"What specific role does the classifier play in estimating the density ratio? How is it trained and validated? More clarity and ablations are needed here.\"\n\nThe classifier provides scores used to compute the **calibration factor** in Equation (6). Training procedures, model architecture, and hyperparameters are detailed in **Algorithm 1** and **Appendix C.3 (line 536)**. Concretely, we train a BERT-base classifier using a standard binary classification loss, where the \"chosen\" response in each original dataset entry is labeled as 1 and all other responses as 0. We use a learning rate of 2e-5 and train for 3 epochs.\nWe appreciate the suggestion for improved clarity and will make this training pipeline more explicit in the final version of the paper.\n\n> $Q_3$: \"Why is the method described as LLM-specific? Apart from the use of preference data from LLM outputs, the algorithm seems agnostic to language modeling.\"\n\nPlease note that we **do** **not** describe this method as LLM-specific. The framework is *method-agnostic* by design and is motivated as such throughout the paper. We also test it across different model architectures to demonstrate its generalization ability. We are happy to make further clarifications if needed.\n\n>$Q_4$: \"Can the authors provide evaluation on more diverse benchmarks (e.g., MT-Bench, Arena-Hard, AlpacaEval)? One win rate metric is insufficient to support alignment claims.\"\n\nThanks for asking. Please see **Table 3** in the main paper, where we provide results on Arena-Hard and AlpacaEval. The results suggest that DoRA consistently improves performance on diverse benchmarks.\n\n>$Q_5$: \"Given the added complexity, how does DoRA perform under simpler ablations? For example, removing the density ratio weighting or using a simpler robustness penalty.\"\n\nPlease note that the **calibration factor** is one of the core contributions to the proposed method; removing this, it falls back to a standard DRO baseline. We present such ablations in **Table 4 (Line 259)**, comparing DoRA against simpler, robustness-controlled variants such as vanilla DRO and reweighting-based optimization. These results show that DoRA achieves higher performance while maintaining stability.\n\nThanks again for the questions, and we hope the above responses can mitigate the concerns raised. We are happy to make further clarifications and we welcome further discussion."}]

Author rebuttal:

Meta-review: This work proposes a method to ensure more robust alignment for LLMs. The authors first seek to measure a (pseudo) gold standard target distribution and then construct classifiers that attempts to inform via a score whether or not an arbitrary preference sample maps reasonably onto that distribution. Summarizing the reviewer's discussions.

Pros:
- The idea is theoretically sound and is timely given that most mainstream alignment approaches focus on combining all human value data into a single size fits all system. Being target distribution aware at train time itself lends itself more easily to approaches such as personalization and fitting to pluralistic values across different pools of humans
- Practical benefits over established techniques like DRO seem significant
- DoRA as a method can compose with many classes of online and offline algorithms, seemingly with not too great changes required for existing infra. This is important as it means it is easier for the wider community to adopt.

Cons:
- The obvious fundamental concern is that this method is restricted to only those instances where you can reasonably measure the target distribution in the first place, in many cases this isn't possible - and I'd like to see an acknowledgement from the authors pointing out when and when to not attempt to use DoRA.
- Some of the empirical gains in specific settings (e.g. Arena Hard) are minimal, especially given somewhat outdated datasets/benchmarks used such as HH-RLHF and Alpaca Eval 2. Some of these concerns appear to have been addressed during rebuttals

Overall, the authors have made a good faith effort to address concerns during the rebuttal period and some reviewers like 2E9q were swayed to lean accept. After discounting a review that did not engage (8Cax), I am also inclined to lean towards an accept as the pros outweigh the cons for this work.
Decision: Accept (poster)