Importance Weighting for Aligning Language Models under Deployment Distribution Shift
Authors: Thanawat Lodkaew
Tongtong Fang
Takashi Ishida
Masashi Sugiyama
Venue: Accepted by TMLR ()

Abstract: Aligning language models (LMs) with human preferences remains challenging partly because popular approaches, such as reinforcement learning from human feedback and direct preference optimization (DPO), often assume that the training data is sufficiently representative of the environment in which the model will be deployed. However, real-world applications frequently involve distribution shifts, e.g., changes in end-user behavior or preferences during usage or deployment, which pose a significant challenge to LM alignment approaches. In this paper, we propose an importance weighting method tailored for DPO, namely IW-DPO, to address distribution shifts in LM alignment. IW-DPO can be applied to joint distribution shifts in the prompts, responses, and preference labels without explicitly assuming the type of distribution shift. Our experimental results on various distribution shift scenarios demonstrate the usefulness of IW-DPO.

Published in Transactions on Machine Learning Research (07/2025)
Importance Weighting for Aligning Language Models under Deployment Distribution Shift
Thanawat Lodkaew The University of Tokyo, Japan
lodkaew Oodkaeu-mmskk...ooooooo....
Tongtong Fang
fang fang ac..j
The Institute of Statistical Mathematics, Japan
Takashi Ishida
ishi@k..ttkkyo.cc..pp
RIKEN, Japan
The Universily of Tokyo, Japan
Masashi Sugiyama
sugi@k..-lokyo..e.
RIKEN, Japan
The University of Tokyo, Japan
Reviewed on OpenReview: https: // openreview. net/ forum? id= C7QWN4AXUP
Abstract
Aligning language models (LMs) with human preferences remains challenging partly because popular approaches, such as reinforcement learning from human feedback and direct prefer- ence optimization (DPO), often assume that the training data sufficiently representative of the environment which the model will be deployed. However, real-world applications frequently involve distribution shifts, e.g., changes in end-user behavior or preferences dur- ing usage deployment, which pose significant challenge to LM aligmment approaches. In this paper, propose an importance weighting method tailored for DPO, namely IW-DPO, to address distribution shifts in LM alignment. IW-DPO can be applied to joint distribu- tion shifts in the prompts, responses, and preference labels without explicitly assuming the type of distribution shift. Our experimental results on various distribution shift scenarios demonstrate the usefulness of IW-DPO.
1 Introduction
While language models (LMs) have been rapidly increasing their language generation capabilities in recent years, aligning them with human values and norms remains challenging task (Shen et al. 2023) Among the various approaches for aligument, reinforcement learning from human feedback (RLHF) has demonstrated
considerable success in aligning LMs with human preferences (Ziegler et al.) 2019 Stiennon et al.) 2020
Ouyang et al., p022 However, it involved a rather complex training pipeline: reward modeling (RM) from preference data and optimization of an LM using learned reward model and a reinforcement lcarning (RL) algorithm. To reduce this complexity, [Rfailov et al. (2024) developed a simple yet effective optimization approach, namely direct preference optimization (DPO). DPO directly optimizes the LM without the need for RM and RL, thus making it simpler and faster.
DPO has been demonstrated to be an effective method for fine-tuning LMs to generate responses that align with human-desiree outputs, leading to the creation of several widely used foundation LM families, such as Llama (Grattafiori et al.] 2024), Phi-4 (Abdin al.) (024)) Qwen2 (Yang et al. 2024). and DeepSeck (Bi et al.) 2024). Like other machine learning algorithms (Quiñonero-Cande Candela
2008 Pan & Yang. 2009; Sugiyama Kawanabe 2012), however, DPO typically suffers from various distribution shifts that present

---

Published in Transactions on Machine Learning Research (07/2025)
Direct Preference Optimization (DPO)
Importance-Weighted Direct Preference Optimization (IW-DPO)
Training data
Log probs.
Training data
Log probs.
DPO
EPO
B
w
Log probs.
Validation data
min pulaannee.) (DDOO (2,, y1, y22 b)]
min 3 Pal (w0 (a, y1, y2, bDr (ar, 91, 92, b)]
Figure 1: DPO optimizes for the training distribution by using only the training data, while IW-DPO optimizes for the test distribution by additionally using tiny amount of data (i.e., validation data) sampled from the test distribution to estimate weights and reweight training losses. weight estimation, the log probabilities of the training data and those of the validation data are passed through a transformation function t, and the transformed data are then used compute the importance weights.
challenge in aligning with human-desired responses, underscoring the need for the development of a method that can effectively address such practical difficulties.
Recent studics have attompted to address the issue of distribution shifts in DPO, where the LM being optimized gradually deviates from the initial reference model (the LM used as initial weights for training) as training progresses
fixed offline preference dataset, which refer to as model distribution shift. For instance, Sun et al. (2023) proposed a way to address the difference between the reward distribution of the LM and that of the reference model. Gou & Nguyen (2024). Zhou et al. (2024) and Xu et al. (2024) explored a phenomenon in which the output (also called sample, response or completion in various literature) distribution the LM changes, causing to diverge from the distribution present in the fixed offline preference dataset. Similarly, Dou et al. (2024) examined how output distribution shifts negatively impact the performance of the reward model, diminishing its ability to distinguish between responses.
In contrast, work addresses fundamentally diffcrent form of distribution shift which we call the deploy- ment distribution shift, where the environment changes in ways not rellected in the training dalaset. Such shifts can arise from real-world usage or deployment, such as changes in end-user behavior or preferences. For the remainder of this paper, we will use the term "distribution shift" to denote this phenomenon. Wo characterize the factors that cause distribution shifts in LM alignment and, accordingly, systematically define the types of distribution shifts. Specifically, in the context of LM alignment, preference data typically con- sists of three elements: prompt, response, and preference label. Various types of distribution shifts between the training distribution and post-deployment, i.c., the test distribution, factors. When distribution shift. occurs, training on the training dataset means optimizing for the training distribution, which may result in poor performance on the test distribution. [Son et al. 2025) explored a shift in one of these factors, the preference shift problem, but focused on an online setting, whereas we assume to have a fixed offline preference dataset for training. We provide a detailed explanation of the definition of distribution shift, the contributing factors, and the types of distribution shifts in Section 3.1
arise from one or more of these
For solving distribution shift problems, importance weighting is a powerful tool that estimates a test-over- training density ratio as weights and uses these weights to reweight the training losses (Sugiyama & Kawan- abe 2012). Later, dynamic importance weighting (DIW) was proposed as modern implementation of importance weighting, which makes it well suited for deep learning (Fang et al. 2020 However, DIW mainly focuses on classification, and its effectiveness in large-scale machine learning problems such as LM alignment has yet to be investigated.
In this paper, inspired by DIW, we propose an importance- weighted DPO, namely IW-DPO, to solve the distribution shifts in LM alignment. An overview of our method compared to the original DPO is shown
2

---

Published in Transactions on Machine Learning Research (07/2025)
in Figure IW-DPO estimates importance weights for training instances and uses them to up/down-- weight training instances that are distribution and more aligned with the test distribution. To estimate the importance weights, IW-DPO uses transformation function t derived from the LM to convert raw preference data into low-dimensional representations. It then performs existing density ratio estimatiio methods, such as kernel mean matching (KMM) (Huang et al. 2006), Kullback-Leible importance estimation procodure (KLIEP) (Sugiyama et al., 2007) and relative unconstrained least-squares importance fitting (RuLSIF) (Yamada et al. 2011]; on the transformed data.
to ensure that the LM is not overfitted to the training
A significant advantage of IW-DPO is its capability to handle joint distribution shifts without requiring prior knowledge of the types of distribution shifts involved, making it particularly valuable for practical applications. To evaluate its effectiveness, we design and conduct experiments under various distribution shift scenarios in LM alignment. The results show a great potential of IW-DPO in handling practical distribution shift problems.
2 Related Work and Background
In this section, we first explore various approaches to importance weighting in LMs and then provide the background information on seward-based and reward-free RLHF. See Appendix D for a further discussion of related work.
2.1 Importance Weighting in LMs
Several approaches based on importance weighting have been proposed for language modeling. Grangier al. (2023) proposed importance weighting method for LM pre-training and fine-tuning, where importance weights estimated by separate weighting model trained jointly with the LM. While they concentrated on LM pre-training and fine-tuning, our emphasis is on LM alignment, particularly preference optimization. Moreover, our IW-DPO utilizes a transformation function from the LM for weight estimation, eliminating the need for joint training of a weighting model as they did. Jiang et al. (2024) applied importance weighting as a form of importance sampling to filter out self-generated examples that deviate from the desired distri- bution, aiming for iel--mmproeeeent in LMs. However, their approach is constrained to datasets or tasks with clear, definitive answers because it includes components such self-consistency and majority voting, whereas our focus is on weighted preference optimization (WPO). Their approach involves reweighling training instances to address the distribution shift between the output distribution of the LM and the distribution presented in the train- ing preference dataset. While WPO focuses on the model distribution shift, we focus on the deployment distribution shift. In addition, while WPO uses length- of all predicted tokens in the response) as weights, IW-DPO estimates weights by using a density ratio estimation method. [Sow et al. (2025) introduced an importance weighting method for LM pre-training with weight estimation based training losses; however, they did not account for any distribution shifts. Addi- tionally, their weight. imaationn method computes importance weights solely based on information from the training examples, specifically using training loss values. In contrast, our weight estimation process accounts for using information from both training and test distributions, thereby optimizing specifically for the test distribution.
open-ended tasks. Zhou et al. (2024) proposed an extension of DPO, namely
sequence probabilities (i.e., probabilities
2.2 RLHF
In reward-basod RLHF, following the pipcline in Sticnnon et al. (2020) we first construct a reward model that approximates human preferences based on a pair of responses (y1 y2) to a given prompt xx Human annotators express a preference for one response over the other, referred to as preference label b, which is used train the reward model. We define b +1 if y is preferred, and b = - if V/ is preferred. One common approach for modeling human preferences is the Bradley- Terry model (Bradley
Reward-based RLHF
Some R..HF pipelines, such those in Ziegler al. (2019) and Ouyang et (2022), may utilize more than two responses.
3

---

Published in Transactions on Machine Learning Research (07 (025))
& Terry 1952.
which defines the preference probability expressed
p(b æ, U1, V2) = a (b (x, r" (2.))))).
(1)
where r" is latent reward model and o(u) 1+) is the sigmoid function. We given preference dataset D {(x,, vt v2, b') of N instances. During the RM phase, we aim to optimize the following objective to train a reward model Ty parameterized by th:
min (( I- (b (rp(a, (2 92)))l
After training the reward model, we procoed to the RL phase where we consider optimizing an LM Ty parameterized by 8 A The goal of this phase to maximize the expected reward assigned to the generated response of the LM me while ensuring that it does not drift too far from the reference model Tref- This be done by utilizing proximal policy optimization (Schulman et al. 2017), which results in the following objective:
max Eww.....wDattt...... (ro v)] 3I (To( x)) Tie(( x)], <gg
where is a response generated by given prompt sampled from Dz = for [a')_= and DKL is the -ullback-Leibler (KT.) divergence and ensures that the LM does not diverge too far from the reference model, as controlled by a hyperparameter 3 > O.
Reward-free RLHFF DPO [Rafailov al. 2024) simplifies the RLHF process by directly optimizing the LM using human preference data, without the need for RM and RL. The derivation of the DPO loss begins by reparameterizing the reward function in terms of the LM and the reference model Tref, resulting in an implicit reward function r. We can then express the probability of human preferences in terms of the LM directly, thereby bypassing the need to fit an explicit reward model (Rafailov et al. 2024 This results in the DPO loss, which is defined as
(DPO(Z, U/, 12. - log (b- (r(z, 91) r(x, y2))),
(4)
where the implicit reward function r is given by
r(z,y) 8 log Te x)
(5)
Teat (y) x))
Going forward, we will omit x Eq. (5) for simplicity. In practice, a simple way derive wo and Rref is to initialize them supervised fine-tuned LM (Ralailov al. 2024), which we will refer to as supervised fine-tuning (SFT)
3 Proposed Method
In this section, we introduce the mechanism of IW-DPO. We begin by providing explanation of the definition of distribution shift and formulating the objective that we aim to optimize. Next, we describe how to optimize this objective using IW-DPO. Finally, we present two variants of IW-DPO.
3.1 Problem Setting
Distribution shift A shift in the distribution of the data is defined as the underlying joint density of the 32, b) differing from that of the test preference data, pec(x, y1, 32, b), i.e.,
training preference data prr( yr, V2, Pur (z. Vr, V2, b) + Pte y1, y2. b).
"Given a prompt 2, my generates a response y in an auto-rogressive manner characterized by mo(y) x) II, no (By æ. 945))
where yj is the j-th token in the response and Vej the tokens in the response prior My [Xu all 2024 Inn RLHF, SFT typically involves fine-tuning an LM on pairs of prompts and their corresponding responses Ouyang
2022
4

---

Published in Transactions on Machine Learning Research (07/2025)
Factors of distribution shift The factors con-
Table 1: Factors and potential distribution shift types. Specifying the type of shift can be challenging due to complex relationships among factors. Factors 1, 2, and 3 represent the prompt, the response, and the prefer- ence label, respectively.
tributing to distribution shift. can be categorized by expressing the joint density as p(z, yı, y1,32, b) = p(z)p(y1. 1/2 z)p(b 2, y1, V2) and studying cach component individually: 1) Prompt: A change in prompts may arise from a shift the domain of in- terest, such as from culinary topies to agricultural practices. Formally, this sort of change can be ex- pressed as (a) F Ptc(x). 2) Response: A change in responses may result from shift in the preferred response style. For instance, whereas helpful re- sponses were previously expected, there is now an expectation for responses be both helpful and harmless. Formally, this change in responses can be expressed (1r z) * (te 3/ z). 3) Preference label: A change in user preferences can lead to shift in the distribution of preference labels, even if the prompts and responses remain unchanged. Formally, this change in preference la- bels can be expressed as Ptr(b x, (1,41,3/2) + Pro 2, v1.)2).
Factor
Type of shift
1
2
a No shift
b Full shift
/
V
c Prompt shift
/
d Response shift
/
Preference label shift
f Prompt + response shift
g
Prompt + preference label shift
h Response + preference label shift
A distribution shift can be caused by one more of these factors, resulting in diffcrent types of distribution shifts. In this paper, consider the full distribution shift, which includes all seven specific types as special cases. We show the relationship between the causes and all possible distribution shift types in Table [ Although there are multiple factors and distribution shift types, we will demonstrate that our method can effectively address such distribution shift problems without requiring knowledge of the underlying factors or specific types of distribution shifts.
Learning objective Ideally, the LM my should be learned by optimizing the following objective:
(mo) E (I, y1 y1.y2, b)]
(6)
When the training and test distributions differ, training solely on the training data implies that we are opti- mizing for the training distribution, which may lead suboptimal performance on the test distribution. In addition to a training preference dataset from the training distribution Drr {(xtr,, .v tr, tr, b.tr)) Ltd. 3/2
(r, Vr. V/2 b), our problem setting further assumes the availability of a validation preference dataset from the test distribution D, = { Mi M3'" erably smaller than that of Diz, ie., N, < Nir- This reflects a real-world situation in which it may be possible collect limited amount of preference data from the test distribution. We can use D, to directly approximate Eq. (6), but it may not be accurate due to the limited sample. Hence, our goal is to utilize both D.r and Dv to learn no that makes Eq. (6) small. Given that the size of D. is tiny, we anticipate that utilizing Dy during training with Dr, will yield better performance than training with D, alone (Fang et al. 2020).
vi vi ). ~ Pro(I, V1, y2, b). However, the size of D. is consid- ii.d
3.2 Importance-Weightern DPO
In this section, we first present the derivation of the training objective and then describe the procodure for weight estimation.
3.2.1 Training Objective
To make the learning objective Eq. (6) small, we propose an importance-weightte DPO method, which we call IW-DPO. Let supp(p) denote the support of a density p over (z, y1, 31, b), defined as the set of points where p assigns nonzero probability density. As is standard in importance weighting, we assume that the
5

---

Published in Transactions on Machine Learning Research (07/2025)
support of the training distribution covers that of the test distribution; that is, supp(pre) S supp(Pu)) Under this assumption, the importance weight w" (z, U1, 122 = (te v1, U2, (2, (p/ vn, U2, is well-defined, and the following proposition holds.
Proposition 1 Given the true importance weight w" and To that minimize the importancc-wweighted risk on the training distribution [TT w²), the risk on the test distribution (Eq. (6) can be expressed as
(mT) = (x.....22)))))) [w" (2r, u1, b) (2r, y1, y2, b)] Tnr(mo, w²).
The proof is shown in Appendix A Proposition D implies that minimizing the importance- risk on the training distribution is equivalent to minimizing the risk on the test distribution. In practice, it is necessary to estimate w" since it is unknown. Tur can be approximated by the weighted empirical loss over the training distribution. Formally, an inportance-weigghled empirical version of J (as defined in Eq. 6 is given by
N.r (mT) Nur Σ nh EDro( Vi 1/2 ,8
where wh is the empirical importance weight of the i-th training instance. If empirical weights wh equal the true importance weights w", then- by Proposition | I is an unbiased estimator of J. More generally, when a are consistent estimates of w", I remains a consistent (asymptotically unbiased) estimator.
Why is importance weighting important? As discussed in Proposition | when considering w, we can ensure that minimizing the risk on the training distribution aligns with minimizing the risk on the test distribution, effectively optimizing Ty for the test distribution. However, in the absence of w (i.c., when ming, (2.1 ( V1, y2, b)], there is no guarantee that minimizing the risk the training distribution will correspond to minimizing the risk on the test distribution. In our experiments, we show that even though the estimation of w is not perfect resulting in some relevant examples being down- weighted (i.e., w < 1) and some irrelevant examples being up-weighted (i.e., w > 1u-our methods still consistently outperform the baseline methods. For more details, see Section 4.2.1 particularly Figure 2
3.2.2 Weight Estimation
The key challenge then becomes how to estimate the importance weights. Especially when working with complex data requiring deep models, estimating importance weights also requires powerful models capable of handling such data. One straightforward approach is to model the importance weights (xx 31, y2, 6) directly with a deep neural network, which requires joint training of both an LM and a separate weighting model (Grangier el al. 2023). In contrast, we adopt simpler approach that uses a transformation function derived from the LM transform the inputs into the low-dimensional transformed data (Fang et al., 2020).
In particular, introduce transformation function t (I. y1, 32,6) + that transforms Die and D, into set of transformed training data 3., ( 2tr.Nrr ,...,
and transformed validation data 2, {a 2..N,, We then estimate importance weights by applying a density ratio estimation method to the transformed data Zir and Z,. Weight estimation on the transformed data is expected to be easier than that on the raw data.
While several density ratio estimation methods can be applicd, we use KMM HHang et al. 2006) to illustrate how to derive the importance weights for clarily and brevily. In our experiments, we also explore alternative methods, including KLIEP (Sugiyama et al. 2007) and RuLSIF (Yaaada et al. 2011). and discuss them in Section 4.2.4
In KMM, our objective is to determine importance weights wtr,1 ...., Nr that ensure the mean embedding of the training distribution is approximately equal to that of the test distribution within a reproducing kernel Hilbert space H. is known that there exists a fcature map Φ: Rdd H such that k(z, z') (Ø(2). d(2')) where d the dimension of the transformed data 2 and (--hm represents the inner product in (Smola al. 2007). Then, let. per ( w1. (x)) [d(=)] and phe = wr [d(=)] represent the kernel embeddings
6

---

Published in Transactions on Machine Learning Research (07 (07/2025)
of (tr V1, 22. w(z) and (2, M1. V2, b) in H. respectively. Subsequently, KMM aims to minimize the discrepancy between Mer and Mee, which can be estimated with two empirical means as follows:
||pe=- Meellz
1 Nu
Σ
w trii (2
Nv Σ (2
1 w Kw Nº
k w + const., N
2
(8)
H
where w = ........... Mu]] is the weight vector, k; = No. ΣØ= &o ( Kij = k(2"" 2")d), and
N.
"const." is a constant that does not depend on w. As a kernel lunction, we use the radial basis function (RBF) [uhhmann 2000) in this work, ie., k(z,z") exp (->llz 2'113), where y 0 is the kernel width parameter. More formally, KMM solves the following quadratie optimization problem for w=
Nu word
min w Kw hTw wow + Al subject to w E [0, B] and 2
Se.
(9)
where À > is the & regularization hyperparameter, B slack variable.
upper bound of the weights, and
0 is
As the use of (z, 32, b) is not straightforward for density ratio estimation, it is necessary to properly define the transformation function t. In Section 3.3 we will explain different choices of t.
3.3
Choices of Transformation Function
Here we explain how we can use EDP (Eq. and r (Eq. 5
as t.
3.3.1 Loss
Fang et al. (2020) suggested using loss values (i.e., using EDP (Eq. 4 to estimate importance weights, which in our case corresponds to t (x, 81. 32,6) b) + (D yı y2, We denote this method as IW-DPO- Loss or IW-DPO-I. for short.
Issue of IW-DPO-L Using loss values for weight estimation can be problematic, because DPO is not invertible. For example, a loss value can be associated with multiple instances (x. y1, 3/2 b) as long as their reward margins (i.e., r(y1) - r(z2)) are identical. As stated in Fang et al. (2020), cannot be arbitrarily defined but it must idcally satisfy three properties: fixed, deterministic, and invertible. Although 4ro is fixed and deterministic, it is not invertible, and thus technically not proper transformation function.
3.3.2 Reward
To avoid the issue of IW-DPO-L, we propose utilizing reward values (i.c., utilizing r (Eq. (5)) place of loss values transformed data during weight estimation. Intuitively, reward values provide more direct information, making them more effective for density ratio estimation using data from training and test distributions. Since we have two responses (2), for each prompt we suggest using the reward values of both responses because we may lose information if we use only one of them. Formally, we have t (z. y1, 32, b) + f(x, y1, y2, b), where f(c, yı, V2, b) = (r(1)), r(yz)) is tuple-valued function. While using the loss function is problematic due to its non-invertibility discussed in Section 3.3.1 we use the reward values avoid the issue.
Kernel combination Given that f does not outpul a scalar but a tuple of two reward values, we have }}, where žn and zyy cor- respond to r(y1) and r(y2), respectively, and we cannot compute k directly. To address this, we com- pute two kernels for žn, and zyz separately and combine them. Specifically, we combine the ker- nels by multiplying them together. Then, in Eq. 19), we have k; No. Exn k6 12 2x)) ) 2xd) and N.
E. = [(=tr.1 - )...), ),...,((It.,,,, Net,,Nr, 2ip str.Nu)] and 2, = [(=y]1 1 2y,1), )...),,.. (2y, Sya Ny.N. ,y,
Kú k(2t1,4 a0r )k 2
,uJ).

---

Published in Transactions on Machine Learning Research (07/2025)
Weight normalization There is constraint that the mean of the weights must be 1, i.e., 1/ 1/Nr Ei= w = since the expectation of the true weights is 1:
B pu (w> (x, U1 U1, b)] [p
(10)
In practice, the mean of the weights does not have to be equal to 1; however, it is typically forced to be close to to ensure that the reweighting is performed properly. However, we observe empirically that the direct use of reward values fails to satisfy the constraint, c.g., the mean of the weights far from 1. Refer to Section 4.2.2 especially Figure 3 for empirical evidence. To ensure that we satisfy the constraint, we propose normalize the weights as a post-processing of weight estimation. Given w, let |w| denote its cardinality. We compute its normalized version v = [ :.... (ett,Ne),, where à = w/ w Siw ty x |wo]. We refer to the method that uses this weight normalization as IW-DPO-Reward or IW-DPO-R for short. In Section 4.2.2 we show that the weight normalization process is essential for improving the performance.
Warmup phase Before initiating the loss reweighting process, it is essential to train the LM for brief period, specif- ically the first few examples of the dataset. This initial training phase, re- ferred to as the warmup phase, helps the model to stabilize and learn basic patterns from the data. We manage this process using the hyperparameter warmup_examples examples, es, which determines the number of examples used during warmup.
Algorithm 1 IW-DPO
1: Finish warmup phase
Define as IDPO (for IW-DPO-L) or i (for TW-DPO-R)
3: Define the batch sizes NB., and Ns, Define the number of training epochs E
5: for = to E do
for Batch Bur = { (x""",0)) bri trd .Bt.)) ) Ltd. Dir do :31 :122
6:
7:
Sample batch B, = {x""
vi Vi 1222 _b".)) ,N,, iid Dr Obtain Zu with respect to Bu, and Z, with respect to B, Estimate w with Zu, and Z, as inputs
8:
9:
10:
Obtain vó by normalizing
The importance of this warmup phase
Obtain per-instance losses [Dro: "ro MoNee
11:
lies in its ability to enhance the infor-
12:
Obtain J by reweighting the per-instance losses with t Compute the gradients with J
mativeness of the reward values, which
13:
are crucial for the subsequent weight esti-
14:
Update the model parameters using the computed gradients 15: end for
mations. Without this phase, the reward
values may be poorly calibrated and lack meaningful information; they could ap- pear random values, leading to inaccurate weight estimations. It is important to note that IW-DPO-L also requires this warmup phase.
16:
end for
All processcs of IW-DPO, including data transformation, weight estimation, and loss reweighting, are per- formed in a mini-batch--wsse manner. Given that validation instances are employed for each mini-batch training and N. < Nu. it is inevitable that validation instances will run out before training instances. Con- sequently, we continually sample mini-batches of validation instances from D, until the training complete. We show the algorithm in Algorithm
4 Experiments
In this section, we first demonstrate the effectiveness of our proposed methods across several datasets that encompass different distribution shift scenarios. Additionally, compare methods against WPO (Zhou
ct al.) 2024). Following this, we conduct a scries of empirical investigations, including: (i) a comparison of the importance weights estimated by IW-DPO-L and IW-DPO-R; (ii) an ablation study on the effect of
weight normalization; (iii) an analysis the relationship between the performance and the severity levels of distribution shift; and (iv) an evaluation of performance under different density ratio estimators. For
details on hyperparameter tuning for DPO, IW-DPO-L, and IW-DPO-R, please refer to Appendix B.1 See A Appendix B.2 for the number of instances for the training, validation, and test sets.
8

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Summarized experimental sctups. As discussed in Table it is unclear exactly which type of shift these scenarios fall into. For the datasets, see Dai et al. (2024) and Ji et al. 2023) for SafeRLHF, Ethayarajh
et al. (2022) for SHP, and Huang & Yang 2023 for CALL For the models, see Grattafiori et al. (2024) for Llama Instuc,, Biderman et al.
2023 for Pythia-1.4B, and Riviere al. (2024) for Gemma 2-9B.
Dataset & Model
Training distribution
Test distribution
Scenario
Shift type
Helpful-Harmful responses
SafeRLHF & Liama A.1-8B-Instruc,
Helmful-Harmless Harmless LM
Helpful-Harmlest responses
+ Helpful-Harme Harmless responses
or h
Science fiction-domain
SHP & Gemma 2-9B
prompts +
Science
Science-domain prompts
b
LM
Science-domain prompts
American preference labels
CALI & Pythia-1.4B
Culture-Aware LM
Indian preference labels
+
Indian
preference labels
4.1 Benchmark Experiments on Distribution Shift Scenarios
4.1.1
Experimental Setups
We construct three distribution shift scenarios covering all of the factors discussed in Section 3.1 each involving simple mixture of two distributions to represent static distribution changes. While simplified, these scenarios effectively capture certain characteristics of real-world distribution shifts relevant to study. We summarize experimental setups in Table 2 For each scenario, train an SFT model using both Du and D.. Following Rafailov et al. (2024), we use preferred responses- often referred to as chosen responses as the corresponding responses for prompts in both datasets.
Helpful-Harmless LM In this scenario, we assume that we have a preference dataset for optimizing an LM to serve responses that are as helpful as possible. However, safety another criteria often used when applications. Therefore, we aim to train an LM to produce responses
using LMs for conversation-lype that are both helpful and harmless, The training instances are labeled based on helpfulness only, regardless of how harmful it may be. Specifically, the datasct contain instances whose responses are helpful and harm- less (relevant instances) and instances whose responses are helpful but not harmless (irrelevant instances). Conceptually, we want. to train the LM to be helpful and harmless by using IW-DPO to up-weight relevant instances and down- wein- irrelevant instances during training.
Construction of Dtr, D, and Dic: We employ the SafeRLHF dataset, where each instance contains a question and pair of responses. In addition to preference labels based on helpfulness, the SafeRIHF dataset (Dai et al.) 22024] Ji al.) 2023) includes a safety label for each response indicating whether the response is harmless harmful. We use these safety labels to partition the SafeRLHF datasct into two sets: the Helpful-Harmful set, which contains chosen responses that are helpful but not. harmless, and the
4The URLs https: http://gggggggg....e//
for SafeRLHE, https https://uugginggaaa for Gemma-2-gh https://gittuubbcccttm www.s/huuu...c......S.. www. -LL http:/AAATT-LL//ttttraalalll ttrp/uultrralllyvaeee....
1aamataallamaLLaamaaa 1-88- 1.ntsrnnnt for Llama I.sssut- https://mugginggacce.u co/ taataseta/staaforddaa
SHP, httpp::/huuggingffcc gotgggoglllleaaaaee/ blob/mainn/aata/ttaa.tt for CALI and http:::/waggiiaae..ee///eeeeeeee
Pythia- 1.ii----4...

---

Published in Transactions on Machine Learning Research (07/2025)
Helpful-Harmless set, which consists of chosen responses that are both helpful and harmless. In each set, any rejected response may be either harmful or harmless. We further divide the Helpful-Harmless set into three sets: Harmlless training set, Helpful-Harmless validation set, and Helpful-Harmless test set. We then create the training dataset Dir by combining the Helpful-Harmfin set and the Helpful-Harmless training set. The amount of the Helpful-Harmless training data that we use is 25% of the training dataset. While the Helpful-Harmless validation set usod as the validation datasct Dy, the Helpful-Harmless test set used as Dio for evaluation. D. fifty times smaller than Dir.
Evaluation: We assess the effectiveness of all methods in terms of helpful and harmless response generation, which can be done by asking a human evaluator to determine which response is better in terms of helpfulness and harmlessness: the reference response or the generated response. 5 Since this would be exhausting for the human evaluator, we align with previous studies (Rafailov et al. 2024 Dai et al.) 2024 Ethayarajh et al. 2024) in conducting a GPT-4 evaluation. Specifically, we employ GPT-4o mini as a human proxy evaluator. The evaluator evaluates n test instances. See Appendix [B3.1] for the prompt template. Following this, we have the number of instances where generated responses are preferred over chosen responses Twwin- Then, we compute win rate Awin/n. n..
Science LM In this scenario, assume that we have a preference dataset that is mixed with science fiction-domain prompts (and responses) and science-domain prompts (and responses). Basically, science uses observation and experimentation to understand the natural world, while science fiction imagines futuristic scenarios based on scientific concepts. We aim to use this dataset to build an LM that produces helpful responses with respect to science. Specifically, when the LM is queried, we expect to receive a helpful response based on the natural world, rather than imaginative scenarios or idcas. If we allow the proportion of science fiction data in the training dataset to have large contribution to the LM training, the LM would still produce responses that may not be grounded in reality, but involve speculative elements that may not currently exist or be feasible. Conceptually, we want to train the LM to be helpful on the domain of science by using IW-DPO to up-weight relevant instances (science) and down-weight irrelevant instances (science fiction) during training.
Construction of Der, D. and Dro: The SHP dataset [Ethayarash et al. 2022) consists of questions and responses from 18 different domains, including science and science fiction, which are the domains we use in this scenario. Fach instance contains question and pair of responses: chosen response and a rejected response. To prepare the training, validation and test datasets, we first extract instances of the two domains from the SHP datasel and treat them as two different sets: Science set and Science Fiction set. We then randomly split the Science set into three sets: Science training set, Science validation set and Science test set. The combination of the Seience training set and the Science Fiction set is used as the training dataset Dur where the amount of the Science training data is 25% of the training dataset. The Science validation set is used as the validation dataset Dy. The evaluation is performed on the Science test set Dro- Dy is fifty times smaller than Dur.
Evaluation: Similar to the Helpful- Harmless LM scenario, evaluate all methods by win rates. The evaluator is asked to decide which response is more helpful. See Appendix B.3.2 for the prompt template.
Multure-Award LM In this scenario, we assume that we need an LM that is aware of Indian culture, e.g., the LM will be used in India or for people who want to study Indian culture. However, the preference dataset we have may contain a proportion of preferences that are not aligned with Indian culture, but rather with another culture, e.g., American culture (see Table 3). Specifically, the dataset is a mixture of preferences basod on Indian culture and those based on American culture. We aim to use this dataset to train an LM to be aligned with Indian culture. Specifically, when the LM is asked to give an opinion, we expect to get response that is aware of Indian culture. If allow the proportion of American culture data to have a large contribution to the LM training, the LM would still be biased towards American culture, leading to
Simiilar to Die and Dr, each test instance in Dee consists of prompt paired with one chosen response and rejected response. For our evaluation, we focus the chosen response, treating it the reference response that represents the desired output for the test distribution. During the evaluation process, we provide the trained LM with a prompt sampled from Die and use generate response. We then compare this generated response to the reference response associated with the prompt. hhttps///ppeas.. www//ooo/iaeeeeeeee coom/inde//gge gpt go---o--ttttt 4o---oo-nni wow--iniavvvannnnneeeeeaieaa--e- advancing addaancing-cost-eee cost-efficiiettt.... inteenteitteellgennee
10

---

Published in Transactions on Machine Learning Research (07/2025)
Table 3: An example from Huang & Yang (2023) of how the preferences of American and Indian cultures differ in terms of tipping norms.
Preference label
Prompt and responses
American Indian
What do you think about this statement: tipping is customary? This statement is very likely true.
b =1
b=
= This statement is very likely false.
Table 4: Performance of various methods across three distribution shift scenarios. The numbers represent win rates (%) for the Helpful-Harmler LM and Science LM scenarios, while they denote match rates (%) for the Culture-Aware LM scenario. The best performances are indicated bold, and an asterisk (*) denotes the methods oquivalent to the best method based on a 5% t-test.
Method
Helmfuls LM
Science LM
Culture-Aware LM
SFT w/ Dr
56.40 + 5.12
47.06 = 5.59
31.72 3.13
DPO w/ D.
60.48 + 4.25
53.20 + 5.14
32.15 + 3.56
DPO w/ Dr
68.71 + 3.45
63.79 3.45
35.62 + 0.97
WPO Zhou et al. 2024 w/ D,+
70.26 4.05
64.84 5.22
36.41 + 1.25
IW-DPO-L
70.50 3.46
65.88 + 6.96*
36.49 + 1.39*
IW-DPO-R
72.28 4.62
70.59 + 3.01
36.92 1.77
misleading responses regarding Indian culture. Conceptually, we want to train the LM to be helpful and aware of Indian culture by using IW-DPO to up-weight relevant instances (Indian culture) and down- weight irrelevant. instances (American eulture) during training.
Construction of Dir, Dv and Dio: The CALI dataset (Huang & Yang 2023) contains premises, hypotheses, and labels (very likely true cene/neurall/wee premise and a hepothesis. These labels are collected from two groups of people, Americans and Indians. To use the CALI dataset for our distribution shift scenario, we create two preference datasets, US set and India set. In each set, each instance consists of prompt asking about the relationship between a given premise and a corresponding hypothesis, a pair of responses, and a preference label. We further divide the India set into India training set, India validation set, and Indía test set. We use the US set and the India training set Dir, where the amount of the India training data is 30% of the training dataset. The India validation set is used as Dr, which is fifty times smaller than D,,. We test the performance on the India test set D...
likely false) indicating the relationship between each pair of a
Evaluation: We simply compare the chosen responses with the generated responses to see if they match. We use n test instances. Following this, we have the number of instances, where the generated responses match the chosen responses Omatch- Then, we compute a match rate as mmatcc/n.
4.1.2 Results
We compared IW-DPO-L and IW-DP-RR against three baselines across three scenarios. The first baseline reflocts the performance of the SFT model alone, which we refer as SFT w/ Prr The second bascline involves fine-tuning based on the SFT model using combined set of training and validation data, which refer to as DPO w/ Dr, The final baseline entails fine-tuning the SFT model with validation data only, referred to as DPO w/ D,, All experiments were repeated three times with different random seeds. To evaluate the quality of text generation, performance was measured over five rounds of text generation using different sampling seeds.
The results presented in Table I show the performance of various fino-tuning methods across three distribu- tion shift scenarios: Harmllls LM, Science LM, and Culture-Aware LM. Starting from the bascline SFT method, which showed lower performance due to lack of preference optimization and limited adaptabil-
11

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
WW-FF-1
Ite maaid Relevant
Freleaadd Relevant
000000
000
40
00MMBOOO 000
080000000
2.0 Estimatted weightt
3.5
0.0
1.0
3.5
Est teiight
Inelecent Relevant
Freleven Relovant
120
140
100
80
60
40
20
5
weight
Figure 2: Distributions of estimated weights for the Helpful-Harmless LM scenario. Here, "irrelevant" refers to Helpful-Harmful below display the distributions of weights estimated by IW-DPO-L and IW-DPO-R for relevant and irrelevant instances, whereas the box plots above facilitate comparisons between the estimated weights of relevant and irrelevant instances. Small circles in the box plots indicate outliers. The x-axis represents the estimated weight values for both the histogram and box plots, while the y-axis indicates the number of instances for the histogram plots.
Harmful response data, while "relevant" denotes Helpful-Harmless response data. The histograms
IW-DPOLL
IW-DPO-R
Nomalizzttinn
Norma Ization
No
of hini20schs
2.0
200
3.00
1.00
0.959
0.995
1.000
1.005
1,010
1.015
1,020
1.025
0.6
0.7
0.8
0.9
1.0
Meen weight
Mean weight
Figure Distributions of mean weights under the Helpful-Harmless LM scenario. As discussed Sec- tion 3.3.2. the mean of the estimated weights should be very close for each training mini-batch. For IW-DPO-L, the mean weights hover around 1 without weight normalization. In contrast, IW-DPO-R shows mean weights distributed between approximately 0.6 and 1.1 without weight normalization. However, with weight normalization, we can ensure that the mean weight of cach mini-batch is very close to 1 for both IW-DPO-L and IW-DPO-R.
ity, DPO without Drr showed small gains. In contrast, DPO with D.r achieved significant improvements, highlighting the benefits of integrating both training and validation datasets during training. In particular, our proposed methods, IW-DPO-L and IW-DPO-R, further improved their performance, with IW-DPO-R achieving the highest performance in all scenarios. While the improvements were substantial in the Helpful- Harmless LM and Science LM scenarios, they were more modest in the Culture Aware LM scenario, with an increase approximatoly 1% over standard DPO and less than 1% over WPO. We also evaluated the per- formance of our proposed methods in smaller LMs for the Helpful-Harmless Additional results be found in Appendix C
Harmless LM and Science LM scenarios.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
4.2 Empirical Analysis of the Proposed Method
4.2.1
Comparison of Estimated Importance Weights from IW-DPO-L and IW-DPO-R
As discussed in Section 3.3.2 we assert that utilizing reward values yields more accurate weight estimations
and, consequently, better text generation results compared to using loss values. This is supported by the results presented in Table 1 which illustrates the superior performance of IW-DPO-R over IW-DPO-L. Additionally, Figure supports
@upuressupports this claim by displaying the weight distributions of IW-DPO-L, and IW-DPO- R. While IW-DPO-L exhibited relatively uniform up-weighting of relevant instances and down-weighting of irrelevant ones, IW-DPO-R clearly demonstrated stronger up-weighting of relevant instances and down- weighting of irrelevant instances.
4.2.2 Impact of Weight Normalization
To evaluate the impact of the weight normalization the performance of our methods, we conducted an ablation study under the Helpful-Harmless LM scenario compar- ing the results obtained with and without weight normal- ization. Figure 3 displays the distributions of the means of the estimated weights across mini-batches. The com- parative results in Table indicate that the weight nor- malization improved the performance of IW-DPO-R, as evidenced by the higher win rates of IW-DPO-R over IW- DPO-R without weight normalization. This underlines the importance of weight normalization in IW-DPO-R. In other words, is very important to make sure that the mcan of the weights is close and equal to or technically satisfying Eq. (10). Similarly, the win rate of IW-DPO-L improved with weight normalization compared to IW-DPO-L without weight normalization, although the improvement was very small. These findings underscore the beneficial role of the weight normalization in enhancing the performance of IW-DPO methods.
Table 5: Performance of different methods with and without normalization. Best performances are indicated in bold, and an asterisk (*) denotes the methods equivalent to the best method based on a 5% t-test.
Method
Normalization
Win rate (%)
IW-DPO-L IW-DPO-L
x
69.35 3.38*
/
70.50 + 3.46
IW-DPO-R
X
69.19 = 3.43
IW-DPO-R
/
72.28 4.62
4.2.3 Analysis of Performance under Distribution Shift Levels
We conducted a study to observe the performance of our methods under different severity levels of dis- tribution shift. Understanding how different de- grees of distribution shift affect performance cru- cial for evaluating the robustness of our methods in real-worid scenarios. To do so, we intention- ally introduced controlled distribution shift levels in the Helpful-Harmless LM scenario. We defined
SFT
DPO
WN-DPO-L
NW-DPO-R
Win rate (%)
range of shift severity levels characterized by varying amounts of Helpful-Harmless data (relevant data) drawn from the test distribution in the train- ing dataset Dtr, while keeping its size unchanged. Specifically, the amount of relevant data was 25%, 15%, 5%, and 0% of the training dataset for low, medium, high, and complete shift levels, respec- tively. Note that the size of the validation datasot D. was fixed to be fifly limes smaller than D.,. Our methods were evaluated under these conditions, and performance was recorded for each severity level. The results of our investigation are summarized in Figure 4 As the amount of distribution shift in-
Low (25%)
Medium (35%)
High (5%) relevant data)
Complete (0%)
Figure 4: Analysis of the win rate a function of the amount of data from the test distribution in the training dataset. The plots illustrate how variations in distribution shift level affect the performance results. Note that SET and DPO represents SET w/ Du+D. and DPO w/ DD, respectively.
13

---

Published in Transactions on Machine Learning Research (07/2025)
creases (the amount of relevant data decreases), we observed consistent detcrioration in model performance, highlighting the challenges associated with specialization on the test distribution. Additionally, when the training and test distributions are completely different (0% of the amount of relevant data), all methods failed to adapt to the test distribution, as evidenced by similar performance to the SFT model. Overall, the deterioration behavior observed in this study highlights the importance of developing methods that can mitigate the negative effects of distribution shifts.
4.2.4 Performance under Different Density Ratio Estimators
We examined the robust.ness of weight estimation across various density ratio estimators. Specifically, we compared three methods: KMM (Huang et al. 2006), KLIEP (Sgiyama et al. 2007). and RuLSIF (Yamada ett al. 2011). As shown in Table 6 we assessed the performance of both IW-DPO-L and IW-DPO-R under these methods. Although RuLSIF demonstrates superior performance in many cases, our t-test results indicate that the choice of density ratio estimation method does not significantly affect overall performance.
We suspect that one potential reason for these comparable results may be the limited amount of data available for conducting density ratio estimation in each mini-batch. Specifically, in our experiments, utilized small batch sizes due to the large size of our models, e.g., 8 for both the training and validation batch sizes. This constraint may have hindered the ability of the methods to perform differently given such a small amount of data. Investigating the robustness of various density ratio estimation methods in relation to the amount of data available would be an interesting direction for future research.
5 Conclusion
In this work, we addressed the issue of distribution shift between training and test datasets in language model (LM) alignment, particularly in direct preference optimization (DPO). We showed that such distribution shift can occur due to one or more changes in prompts, responses and preference labels. Moreover, since there are several types of distribution shifts, often difficult to identify the type of distribution shift we are addressing. A notable advantage of the proposed importance weightedd DPO (IW-DPO for short) method is its ability to handle joint distribution shifts in general manner, without the need to know the type of shift. IW-DPO assumes the availability of limited amount of data from the test distribution (validation data), in addition to a larger amount of data from the training distribution (training data). During training, IW-DPO performs density ratio estimation using training and validation data to estimates importance weights and then reweights the training instances so that the LM training can be more inffuenced by those instances that are useful for alignment with the test distribution. We investigated two types of data used for density ratio estimation-
values (TW-DPO-Loss or IW-DPO-T) and reward values (IW-DPO-Reward or IW- DPO-R). To evaluate IW-DPO-L and IW-DPO-R, we conducted experiments on different distribution shift scenarios using different datasets, and the results demonstrated the effectiveness of our methods, especially IW-DPO-R.
6 Limitations and Future Work
Unelcar justification of importance weighting Originally, importance weighting was justified only for misspecified models for which the empirical error cannot be zero in general (Sugiyama & Kawanabe, 2012); for over- peeepparaetterieed models, the empirical error can become zero and then importance weighting no longer affects the training objective. In the context of LM alignment, the use of importance weighting may still be justified when only the final layer of a neural serwork-basce model is fine-tuned (i.e., when using a linear model). However, its justification becomes less clear when the entire model is updated, which is often the case with fully fine-tuned LMs using DPO. Future work could theoretically investigate the behavior of importance weighting for fully updated neural network-basod models.
Restrictive assumption of the support of the test distribution In this work, we assume that the support of the test distribution is fully contained within that of the training distribution. Although this assumption is standard in importance weighting, it does not account for more practical and challenging
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 6: Performance IW-DPO-L and IW-DPO-R under different density ratio estimation methods. Best performances are indicated in bold, and an asterisk (*) denotes the methods equivalent the best method based on a 5% t-test.
Scenario
Method
Density ratio estimator
Win/Match rale (%)
KMM
70.50 + 3.46
IW-DPO-L
KLIEP
70.10 + 4.39
RuLSIF
72.28 + 4.94
Helpful-Harme Harmless LM
KMM
72.28 = 4.62*
IW-DPO-R
KLIEP
71.88 = 4.20*
RuLSIF
73.19 + 3.39
KMM
65.88 = 6.96*
IW-DPO-L
KLIEP
68.10 + 2.66
RuLSIF
67.58 + 3.32*
Science LM
KMM
70.59 + 3.01
IW-DPO-R
KLIEP
69.28 = 4.45*
RaLSIF
70.59 = 4.68*
KMM
36.49 = 1.39*
IW-DPO-L
KLIEP
37.83 + 2.68
RuLSIF
36.45 + 0.70*
Aulte LM
KMM
36.92 + 1.77*
IW-DPO-R
KLIEP
36.25 + 1.36*
RuLSIF
38.38 + 1.46
scenarios in which the support of the lest distribution is broader only partially overlaps with that of the training distribution. Addressing such cases would require fundamentally different approaches. While the current work focuses on managcable distribution shifts under this assumption for the sake of tractability, we consider extending our methods to accommodate partial or non-overlapping supports as a promising direction for future research.
Relatively simple distribution shift scenarios Our experiments are conducted in controlled settings involving simple mixtures two distributions, modeling static distribution shifts that capture certain aspects of real-world scenarios. However, real-world distributions can shift and evolve dynamically over time, often resulting in more complex changes than those modeled in this paper. While our method currently limited to static distribution changes, recognize the significance of dynamic shifts and consider extending approach to address such scenarios important direction for future work.
7 Acknowledgments
We thank Johannes Ackermann and Nontawat Charoenphakdes for the helpful discussions. This work was partially supported by JST ASPIRE Grant Number JPMJAP2405. TL was supported by the Institute for Al and Beyond at the University Tokyo. TF was supported by KAKENHI Grant Number 23KJJ0338. TI was supported by KAKENHI Grant Number 22K17946.
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasckar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim,
15

---

Published in Transactions on Machine Learning Research (07/2025)
Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 Technical Report. Vixiv preprint or Niu:2412.08900 2024.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant Risk Minimization. arXiv preprint or xrXio:1907.02500 2020.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wn, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingehuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint NKNi:2401.0295 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In ICML, 2023.
Ralph Allan Bradley and Milton E Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324 345, 1952.
Martin Dietrich Buhmann. Radial Basis Functions. Acta Numerica, 9:1-38, 2000.
Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, and Wenpin Tang. MallowsPO: Fine-Tune Your LLM with Preference Dispersions. In ICLR, 2025.
Josef Dai, Xuchai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe Reinforcement Learning from Human Feedback. In ICLR, 2024.
Shihan Dou, Yan Liu, Enyu Zhou, Tianlong Li, Haoxiang Jia, Limao Xiong, Xin Zhao, Junjie Ye, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. MetaRM: Shifted Distributions Alignment via Meta- MeranLe arXiv preprint ar 00.00440 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding Dataset Difficulty with V-Usable Information. In ICML, 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment as prospect thcoretic optimization. In ICML, 2024.
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking Importance Weighting for Deep Learning under Distribution Shift. In NeurIPS, 2020.
Qi Gou and Cam-Tu Nguyen. Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model. arXiv preprint arXiv:2403.19446 19443, 2024.
David Grangier, Pierre Ablin, and Awni Hannun. Adaptive Training Distributions with Scalable Online Bilevel Optimization. arXiv preprint 21X1122811.11973, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archic Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra,
16

---

Published in Transactions on Machine Learning Research (07 2205))
Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmén, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Ilyann Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyn Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rentala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Peng- wei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileamu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- seini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarck Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswamii, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Bhing-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem FOss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Fissa Jamil, Flaine Montgomery, Fleonora Presani, Fmiil Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik
17

---

Published in Transactions on Machine Learning Research (07/)025)
Vecraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Takshya Garg, Lavender A, Teandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshov, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battcy, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mchta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Tonescu, Vlad Poenaru, Vlad Tiberin Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosubrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 Herd of Models. Xix preprint or 0rXiv:2407.2178 2024.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting Sample Selection Bias by Unlabeled Data. In NeurIPS, 2006.
Jing Huang and Diyi Yang. Culturally Aware Natural Language Inference. In EMNLP, 2023.
Jiaming Ji, Mickel Liu, Juntao Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. BeaverTails: Towards Improved Safety Alignment of LLM via Human- Preference Datasct. In NeurIPS, 2023.
Chunyang Jiang, Chi min Chan, Wei Xue, Qifeng Liu, and Yike Guo. Importance Weighting Can Help Large Language Models Sel--mprove. ar preprint arXiv:2408.0083 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kclton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback. In NewrIPS, 2022.
Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. TEFE Transactions on Knowledge and Data Engineering, 22(10):1345 1359, 2009.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing Rein- forcement Learning from Human Feedback with Variational Preference Learning. In NewrIPS, 2024.
Joaquin Cuinonero-Candela Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Datnset Shift in Machine Learning. MIT Press, 2008
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In NewrIPS, 2024.
18

---

Published in Transactions on Machine Learning Research (07/ 2205))
Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya. Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantamu Thakoor, Jean- Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette- Choo, Danila Sinopalnikov, David Weinberger, Dim- ple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Frica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimezak-Pluuiiiska,, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Njazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre,, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh han, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ra- mona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Me Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhilao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving Open Language Models at Practical Size. arXiv preprint Xiv:2400 00118, 2024.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In ICLR, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Opti- mization Algorithms. arXiv preprint arXi::1707.063/7 2017.
Tianhao Shen, Renren Jin, Yufci Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large Language Model Alignment: A Survey. arXiv preprint ar 15025, 2023.
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A Hilbert Space Embedding for Distribu- tions. In ALT, 2007.
Scongho Son, William Bankes, Sayak Ray Chowdhury, Brooks Paige, and Tlija Bogunovic. Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift. In ICML, 2025.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L. Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. Position: A Roadmap to Pluralistic Alignment. In JCML, 2024.
Daouda Sow, Herbert Woisetschläger, Saikiran Bulusu, Shiqiang Wang, Hans Arno Jacobsen, and Yingbin Sample Reweighting for Improved Large Language Model Pretraining. In
Liang. Dynamic ICLR, 2025.
19

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. In NewrIPS, 2020.
Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Slationary Enwironments: Introduc- tion to Covoriate Shift Adaptation. MIT press, 2012.
Masashi Sugiyama, Shinichi Nakajina, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct Importance Estimation With Model Selection and Its Application to Covariate Shift Adaptation. In NewrIPS, 2007.
Zhongkai Sun, Yingxue Zhou, Jie Hao, Xing Fan, Yanbin Lu, Chengyuan Ma, Wei Shen, and Chenlei Guo. Improving Contextual Query Rewrite for Conversational AI Agents through User-preference Feedback Learning. In EMNLP, 2023.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep Domain Confusion: Maxi- mizing for Domain Invariance. arXiv preprint 4rXiv:1412.9474, 2014.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO Superior to PPO for LLM Alignment? A Comprchensive Study. In ICML, 2024.
Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative Density-Ratio Estimation for Robust Distribution Comparison. In NewrIPS, 2011.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingleng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report. arXiv preprint ar Xrv::24407 106711 2024.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. WPO: Enhancing RLHF with Weighted Preference Optimization. In EMNLP, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. ar preprint ar arXiv:1909.08503 2019.
A Proof of Proposition
We show that the inportance-weighted risk on the training distribution Tu (TO, w*) is equivalent to the risk on the test distribution (TT) as follows:
[(TT) = Jee( w")
= Eppe (zimn (z. wa.b) (x, y2, b) yn, (2. b)], "1'
f w² (r. U1. y2, b) (z. 81, U2 (ap vr, y2, b) dr dyn dyz,
Σ
bE(+1,-1)
J Pro 11. 1/2 b)
Σ
IDPO (I, U1,
w2, 81, b) (x, æ1 y2, b) dx dy dy2.
Per v1. (r.(x.11.12.
bE(+),-1)
= Σ
[
Pro(o, Un, u2, (2o U1, V2. b) dx dyn dyz.
bE(+1,-1)
Em.. (4DOO yr, b2)
20

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 7: Default hyperparameter settings.
Hyperparameter
DPO
IW-DPO-L
IW-DPO-R
B (for Eq. À (for Eq. (9
0.1
0.1
0.1
0.1
0.1
forr R.FF.
0.1
0.1
warmup_ warmup_examples
1024
1024
Table 8: Sizes of the datasets used for training and testing in each scenario.
Training set (Due)
Dataset
Validation set (D.)
Test set (Die)
(Irrelevant examples + Relevant examples)
15000
Helpful-Harmler LM
300
512
(11250 - 3750)
15000
Science LM
300
512
(11250 - 3750)
1048
Culture-Aware LM
20
512
(729 + 319)
Thus, the proof is complete.
B Further Details on Experimental Settings
B.1 Hyperparameter Tuning
The default hyperparaaetter settings are presented in Table In our experiments, we fixed B for all methods and aarmup_examplee for our proposed methods, while tuning the hyperparameters and A. Specifically, we explored the range of {0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0) for both hyperparameters. However, we empirically observed that using the default values for and often resulted in better performance for IW-DPO-L and IW-DPO-R compared to the baselines.
B.2 Sizes of the Datasets Used for Training and Testing
We prepared the training, validation, and test datasets, with their respective sizes displayed in Table S In the Helpful-Harmless LM and Science LM scenarios, the proportion of relevant examples is 25%, while in the Cultural- LM scenario, is approximately 30%. This higher percentage is due to the small size of the CALI dataset used for the Cultural-Aware LM scenario. Reducing the number of relevant examples further would leave us with insufficient irrelevant examples to maintain total training set size of over 1,000.
GPT-4 Prompt Templates B.3
B.3.1
Helpful- Helmful-Harmless LM
We modified a prompt template from Rafailov et (2024) with information about the helpfulness and harmlessness of responses from Dai al. (2024) and Ji et al. (2023).
For the following query to chatbot, determine which response is more helpful and harmless.
**Qury::** {query)
**Response A:** (response A}
**Response B:** (response_i
21

---

Published in Transactions on Machine Learning Research (07 (07/2025)
FIRST, provide one-sentence comparison of the two responses, explaining which response fulfills the criteria of being both helpful (offering clear, relevant, and accurate information) and harmless (ensuring no potential harm, offense, or misinformation is present). In your evaluation, both responses for potential harmful implications, considering the following categories:
Endangering National Security
2. Insulting Behavior
3. Discriminatory Behavior
4. Endangering Public Health
5. Copyright Issues
6. Violence
7. Drugs
8. Privacy Violation
9. Economic Crime
10. Mental Manipulation
11. Human Trafficking
12. Physical Harm
13. Sexual Content
14. Cybercrime
15. Disrupting Public Order
16. Environmental Damage
17. Paychological Harm
18. White-Collar Crime
19. Animal Abuse
line, state only "A" "B" to indicate which response is helpful and harmless. Please prioritize harmlessness in your evaluation.
SECOND,
Use the following format:
Comparison: <one-sentence comparison and explanation>
More helpful and harmless: <"A" or "B">
B.3.2
Science LM
We modified prompt template from Rafailov et al. 2024 by incorporating additional information about helpfulness based on scientific principles.
For the following query to chatbot, determine which response is more helpful.
**Query:** {query}
**Response A:** (response_A) _A}
**Response B:** (response_Bi
FIRST, provide one-sentence comparison of the two responses, explaining which response is more helpful by indicating that offers accurate information based on scientific understanding and the natural world, while avoiding imaginative scenarios speculative ideas. SECOND, on new line, state only "A" or "B" to indicate which response is more helpful.
Use the following format:
Comparison: Kone-sentence comparison and explanation, focusing on accuracy and grounding in the natural world>
More helpful: <"A" "B">
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 9: Performance of various methods employing different LMs the Helpful-Harmnn Harmless LM and Science LM scenarios. Best performances are indicated in bold, and an asterisk (*) denotes the methods equivalent the best method based on a 5% t-test.
Scenario
Model
Method
Win rate (%)
SFT w/ Du+D, DPO D. DPO w/ Du+DD IW-DPO-L IW-DPORR
12.48 + 3.36
13.62 + 3.91
Pythia-2.8B
41.78 + 4.08
44.83 = 4.76
49.70 4.10
Helpful- -arfmllsss LM
SFT w/ Du DPO w/ D. DPO w/ De,+D,
56.40 + 5.12
60.48 + 4.25
Llama
08.71
IW-DPO-L IW-DPO-R
70.50
72.28 4.62
SFT w/ DirtD
37.25 + 6.19
DPO w/ D. DPO w/ Du,DD,
38.30 + 4.33
Gemma 2-2B
43.79 3.38
IW-DPO-L IW-DPO-R
46.93 3.42"
47.58 = 2.46
Science LM
SFT w/ Du+DD DPO w/ D. DPO w/ Do IW-DPO-L IW-DPO-R
47.06 + 5.59
53.20 5.14
Gemma 2-9B
63.79 3.45
65.88 + 6.96
70.59 3.01
C Scaling Down: Experiments with Small LMs
addition to the experiments and results presented in Section 4.1.2 we explored the generalization potential of relatively small LMs, specifically Pythia-2.8B (Biderman al. 2023 for the Helpful-Harmless LM scenario and Gemma 2-2B [Riviere et al. 2024 for the Science LM scenario. Table summarizes the performance of various LMS across these scenarios. The results indicate that methods consistently outperformed the baseline methods, demonstrating superior performance for both small and large LMs. Note that, in the Culture-Aware LM scenario, we opted not to use large LM due to the limited size of the training dataset.
Furthermore, in addition to the results presented in Section 4.2.2 and Section 4.2.3 we assessed the perfor- mance Pythia-2.SB (Biderman et al. 2023) concerning the impact of weight normalization, as shown Table 10 and analyzed variations distribution shift. levels, detailed in Figure 5 The results indicate a consistent performance between Llama J.1-BB-Instruet (Grattafiori et al., (2244) and Pythia-2.8B (Biderman
al. 2023 underscoring the robustness of our methods rogardless of model size.
D Further Related Work
D.1 Motivation for Importance Weighting under Distribution Shift
A key motivation for using importance weighting methods (such as KMM, KLIEP, or RuLSIF) is their ability explicitly reweight training examples better reflect the test distribution. This is particularly effective in our setting, where small amount of validation data from the test distribution is available to guide the
The URLs httpps://hugiiiggaace o//ELeetherr///yyyhaa-2...0 for Pythia-2.8D https::/huggingtaatt httph//ggiggaaaccccccaaaaaaaaeee co/neta-llamaa
L 3. 8- Instruct for Ilama 3.1 Instrucst http:://hugginggacc.coo. ca/ http://fwww.c.oogo/////////////e 2- 2b for Gemma-22 2-22BB and https: for Gemma-2-9R.
23

---

Published in Transactions on Machine Learning Research (07/2025)
estimation of importance weights. Such reweighting enables unbiased risk estimation and improves model alignment with the test distribution (see Proposition D.
For instance, when the training data includes multiple subpopulations (e.g, helpful-harmful and helpful- harmless), but the test distribution contains only one (e.g., helpful-harmless). importance weighting allows us to increase the influence of relevant training examples and reduce the impact of irrelevant ones based on the estimated weights.
In contrast, alternative approaches such as distributionally robust optimization (DRO) (Sagawa et al. 2020) focuses on robustness to worst-case shifts, which can be conservative and may sacrifice performance on the actual test distribution, especially if the test distribution is well characterized (e.g., helpful-harmless) Approaches like invariant risk minimization (IRM) (Arjovsky et al. 2020) and deep domain confusion (DDC) (Tzeng et al.) 2014) aim for models that are invariant across groups or domains, which could in- advertently enforee invariance for distributional aspects that are not actually present in the test set (c.g.. helpful-harmful examples), potentially reducing performance on the test distribution.
In practice, we often want to align the model with specific distribution, such that of a particular country or culture, as demonstrated in our Culturc-Aware LM scenario. In such cases, we would not want to consider worst-case scenarios or disregard country-specifie information. Instead, importance weighting offers a principled way to focus the model on the target distribution, making it. particularly well-suited for our setting.
D.2 Relation to Pluralistic Alignment
Pluralistic aligument in Al systems refers to the design of models that can accommodate and reflect a wide arryy of human values and perspectives, rather than adhering to a singular notion of correctness or preference (Sorensen et al., 2024). In the context. of language modeling, this approach aims to ensure that LMS can generate reasonable responses that encompass multiple viewpoints, thereby addressing diverse user needs and socictal norms (Sorensen et al., 2024). Ultimately, pluralistic alignment challenges traditional approaches by emphasizing inclusivity and diversity in the design and behavior of Al systems. Our Aulture-Aaare LM sccnario may excmplify stecrable pluralistic alignment defined in Sorensen et al. (2024) which entails that an LM faithfully steers (or aligns) its responses according to specified attribute or perspective, such as a particular value, framework, or population, as it aims to align the LM to accurately reflect a specific culture.
Rescarch has begun to explore RLHF approaches aimed at achioving pluralistic alignment in LMs. In order Pocieeeeeeeeeddaarrrrr to handle diverse human preferences, Poddar el al. 2024) formulated RLHF as a lalent variable problem and subsequently developed multi-modal reward modeling framework based on variational inference techniques, termed variational preference learning. They assume that diverse (or mixed) preferences exist in the training dataset, and they aim to develop an LM that can recognize all sets of preferences and respond appropriately to each individual user at test time. In contrast, our work focuses on training an LM that is attuned to specific sets of preferences within the training dataset. [Cenn et al. (2025) enhanced DPO to address its limitations in characterizing the diversity of human preferences, drawing inspiration from Mallows' theory of preference ranking to better capture the dispersion of human preferences in response to prompts. They demonstrate the robustness to out-of-distribution scenarios, but their method does not steer towards particular distribution of interest.
24

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Table 10: Performance of various methods utilizing different LMs, both with and without normalization. Best performances are indicated in bold, and an asterisk (*) denotes the methods equivalent to the best method basod on a 5% t-test.
Model
Method
Normalization
Win rate (%)
IW-DPO-L TW-DPO-L
44.29 + 5.40"
44.83 + 4.76
Pythia-2.8B
IW-DPO-R
47.76 + 5.30"
IW-DPO-R
49.70 + 4.10
IW-DPO-L
69.35 + 3.38*
IW-DPO-L
70.50 = 3.46
Llama 3.1-8BBIInstruct
IW-DPO-R
X
69.19 + 3.43
IW-DPO-R
72.28 = 4.62
SFT
SFT
75
IW-DPO-L
IW-DPO-L
IVADPO-R
65
Winn rata
30
Win rate
60
55
10
45
(255%)
lum
Complete
(25%)
High Complete
Shift level (amount relevant data)
Shift level (amount relevant data)
(a) Pythia-2.8BB
(b) Llama 3.1-8B--nstruuct
Figure 5: Analysis of the win rate function of the amount of data from the test distribution included in the training datasct. The plots illustrate how variations in distribution shift levels impact the performance of different. methods, employing Pythia-2.8B (a) and Llama 3. Instruuct (b).
25


Reviews:
[{"review_id": "4cALPu0z7O", "reviewer": "Reviewer_CaZY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper addresses distribution shift in LLM alignment, which is practically relevant and important for real-world deployment.\n- The overall approach is conceptually intuitive and clean.\n- The proposed approach can handle various types of distribution shift, including those in prompts, responses and preference labels, without knowledge on the contributing factors.\n- The authors evaluated the proposed method on three scenarios, each constructed with a different dataset.\n\n**Weaknesses**:\n- If my understanding is correct, the paper assumes that the support of the test distribution is contained within that of the training distribution. This seems quite restrictive -- what if there are some unseen prompts or responses? In addition, what if the learner cannot obtain a representative validation dataset from the test distribution -- can anything still be done here?\n\n- The experiments are in very controlled settings that involve simple mixtures of two distributions. From the results in Table 4, the improvement in win/match rates from using importance weighting seems relatively modest (3 - 8%) -- it is unclear how much benefits they bring in more complex real-world settings. In addition, It is unclear what win/match rates should be considered as meaningful targets, as there is no baseline evaluation using, e.g., DPO on the test distribution. It would also be beneficial to include comparisons with approaches such as [Zhou et al. (2024] or other domain adaptation techniques that do not use importance weighting.\n\n- The current setting assumes a single training distribution and a single test distribution, but in real-world applications, distributions may shift and evolve over time. It would be nice if the proposed approach can be extended to handle dynamic changes.\n\n- This is not a major concern, as novelty is not a necessary criterion. The proposed method is largely built on the techniques of nonlinear transformation of data and kernel mean matching for weight estimation, as introduced in [Fang et al., 2020] and [Huang et al., 2006]. Having said that, the paper does extend these ideas to LLM alignment and introduce the use of implicit reward functions for data transformation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of aligning large language models (LLMs) under deployment distribution shift, where training and test preference data used for alignment are from two different distributions. The authors introduce an importance-weighted direct policy optimization (DPO) method, which uses a small amount of validation data from the test distribution to perform distribution matching. This is done by first transforming training and validation data and then using kernel mean matching for weight estimation. The paper evaluates two variants of the proposed method on three semi-synthetic scenarios, which shows better performance in comparison with DPO without importance weighting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. Additionally,\n- Could you provide more details on the experiments? For example, what are the sizes of the datasets used for training and testing in each setting? \n- Why does the third setting use 45% test distribution data for training, whereas the first two only incorporate 25%?\n- The paper is generally easy to follow. For clarity, in Section 3.2, it would be nice if you could briefly explain the motivation for applying a transformation function $t$. In Section 3.3.2, it would be helpful to refer to equation (5) when discussing reward values. In Section 4.1.1, for evaluation metrics, it is not entirely clear what chosen responses and generated responses mean.\n- While related work is discussed in Section 1, a more thorough review would be beneficial, given the extensive research on importance weighting, domain adaptation, and distribution shift in both traditional settings and LLM domains. In addition, how is this problem related to pluralistic alignment [https://pluralistic-alignment.github.io/] -- are there techniques there that can be applied to handle distribution shift?", "broader_impact_concerns": "No particular concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RF0CLLGkpl", "reviewer": "Reviewer_KNsG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. The analysis of distributions is comprehensive and inspiring. \n3. Beyond performance enhancements in Table4, the results shown in Figure1 showing the importance weight does identify the relevant samples further provides evidence about the method strength and insights.  \n\nWeakness:\n1. The literature review about exactly \"distribution shift\" in LLM alignment training is missing[1]. And there is no comparison baseline.\n2. This targeted area is broadly also related to \"how to adjust training sample weight\", in which there are lots of existing works[2,3,4]. \n\n\nReferences:\n\n[1] \tImportance weighting can help large language models self-improve\n\n[2]    DYNAMIC LOSS-BASED SAMPLE REWEIGHTING FOR IMPROVED LARGE LANGUAGE MODEL PRETRAINING\n\n[3]    Adaptive training distributions with scalable online bilevel optimization", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to alleviate the effects of distribution shifts in DPO (direct preference optimisation), i,e., the training distribution is significantly different from deployment environment.  Specially, it proposes an important weight term considering a small set of test samples within the DPO training objective. To facilitate the weight estimation, it uses the radial kernel and utilises the reward value for the kernel transformation function. The framework is evaluated under three scenario, i,e., harmful, science and culture, based on two model types, i..e, pythia and gemma, across eight distribution shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add experimental comparison results with [1]\n2. Discuss the relatedness and novelty of your work by comparing with papers in the broader topic \"how to adjust training sample weight\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cj32wpEOat", "reviewer": "Reviewer_NepN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper tackles a crucial problem in LLM alignment, deployment distribution shift, which is often overlooked in current RLHF and DPO literature.\n2. Introducing importance weighting to reweight preference data in DPO is a valuable extension.\n3. Unlike prior methods (e.g., Weighted Preference Optimization, WPO), IW-DPO is more general and does not assume a specific type of distribution shift.\n4. Use of multiple datasets (SafeRLHF, SHP, CALI) and multiple LMs (Pythia-2.8B, Gemma 2-2B, Pythia-1.4B) ensures broad applicability.\n5. Well-designed ablation studies examining weight normalization, choice of transformation function, and robustness to shift severity.\n6. Consistently outperforms standard DPO baselines across all scenarios.\n\nWeakness:\n1. The paper should extend the theoretical discussion on how importance weighting affects preference learning dynamics.\n2. Kernel Mean Matching (KMM) might not be optimal for estimating density ratios in high-dimensional preference data.\n3. Weight normalization improves performance, but how sensitive is IW-DPO to poor weight estimation? Could erroneous weight estimation lead to overfitting or degradation in preference alignment?\n4. The paper compares IW-DPO only against standard DPO baselines, but not against other domain adaptation or robust optimization approaches.\n5. The paper claims that IW-DPO and WPO are different, but a direct empirical comparison is missing.\n6. The study uses relatively small-scale models (Pythia-2.8B, Gemma 2-2B).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Importance-Weighted Direct Preference Optimization (IW-DPO) to tackle deployment distribution shift in aligning large language models (LLMs) to human preferences. Unlike previous work focusing on model distribution shift or reward modeling, this paper proposes an importance weighting approach to reweight training data based on a small set of validation examples sampled from the test distribution. The authors conduct extensive experiments on different distribution shift scenarios and demonstrate IW-DPO’s superiority over standard DPO baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide deeper theoretical justification for why importance weighting benefits DPO.\n2. Investigate robustness of weight estimation under different density ratio estimators.\n3. Compare IW-DPO against alternative robust optimization baselines, not just standard DPO.\n4. Evaluate scaling to larger LMs to understand generalization potential.", "broader_impact_concerns": "- **Bias amplification risk:** If the validation dataset used for importance weighting is biased or unrepresentative, IW-DPO may reinforce and propagate these biases rather than mitigate them.  \n- **Security vulnerability:** IW-DPO’s reliance on test distribution data makes it susceptible to adversarial manipulation, where malicious actors could inject biased validation examples to steer model behavior.  \n- **Computational overhead:** The additional reweighting process increases training complexity and environmental costs, raising concerns about scalability and energy efficiency for large-scale language models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)