Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Authors: Seongho Son
William Bankes
Sayak Ray Chowdhury
Brooks Paige
Ilija Bogunovic
Venue: ICML 2025 poster (2025)
Keywords: LLM
fine-tuning
DPO
non-stationarity
preference drift
RLHF
Primary area: deep_learning->large_language_models
TL;DR: We address the non-stationarity preference drift using exponential reweighting strategy for LLMs.

Abstract: Current Large Language Model (LLM) preference optimization algorithms do not account for temporal preference drift, which can lead to severe misalignment. To address this limitation, we propose **Non-Stationary Direct Preference Optimisation (NS-DPO)** that models time-dependent reward functions with a Dynamic Bradley-Terry model. NS-DPO proposes a computationally efficient solution by introducing only a single discount parameter in the loss function, which is used for exponential weighting that proportionally focuses learning on more time-relevant datapoints. We theoretically analyze the convergence of NS-DPO in a general setting where the exact nature of the preference drift is not known, providing upper bounds on the estimation error and regret caused by non-stationary preferences. Finally, we demonstrate the effectiveness of NS-DPO for fine-tuning LLMs under drifting preferences. Using scenarios where various levels of preference drift is introduced, with popular LLM reward models and datasets, we show that NS-DPO fine-tuned LLMs remain robust under non-stationarity, significantly outperforming baseline algorithms that ignore temporal preference changes, without sacrificing performance in stationary cases.

Right Now, Wrong Then:
Non-Stationary Direct Preference Optimization under Preference Drift
Seongho Son
|
William Bankes"
Sayak Ray Chowdhury? 2 Brooks Paige' Ilija Bogunovic"
Abstract
has led to more precise control over the behavior they ex- hibit. This control is crucial when looking to safely deploy models in the real world (Amodei et al.) 2016: Hendrycks & Mazeika 2022). Human preference datasets enable the training of proxy reward models (see, e.g., RewardBench (Lambert et al. 2024)) that can evaluate complex human behaviour. These reward models
Current Large Language Model (LLM) prefer- ence optimization algorithms do not account for temporal preference drift, which can lead to severe misalignment. To address this limitation, we propose Non-Stationary Direct Preference Op- timisation (NS-DPO) that models time-dependent reward functions with a Dynamic Bradley- Terry model. NS-DPO proposes a computationally efficient solution by introducing only a single discount parameter in the loss function, which used for exponential weighting that proportionally focuses learning on more time-relevant datapoints. We theoretically analyze the convergence of NS- DPO in general setting where the exact nature of the preference drift is not known, providing upper bounds on the estimation error and regret caused by non-stationary preferences. Finally, we demonstrate the effectiveness of NS-DPO for fine- tuning LLMs under drifting preferences. Using scenarios where various levels of preference drift is introduced, with popular LLM reward models and datasets, we show that NS-DPO fine-tuned LLMs remain robust under non-stationarity, significantly outperforming baseline algorithms that ignore temporal preference changes, without sacrificing performance in stationary cases.
used in conjunction with RL to fine-tune the LLM. Recent works (Rafailov et al., 2024 Azar et al.) 2024 Hong et al. (024) seek to improve the efficiency and stability of these approaches (Chaudhari et al. 2024) by training the LLM straight from human prefer- ence data, avoiding the need to learn a proxy reward model.
A key assumption made in these preference optimization algorithms is that human preferences stationary, i.e., they do not change over time. However, a shift in prefer- ences can occur due to new information becoming available (Zafari et al., 2019 Johnson & Mayorga 2020), social influences, and cultural trends. As more preference datasets are gathered over long periods of time, the chance of the data containing varying preferences increases. In such cases, algorithms that do not account for these changes, view them noise and treat outdated data equally important as fresh data, often leading to deteriorated performance. An increasing body of evidence (Zhou et al., 2024 Chen iet al. 2024a) points to data quality as being key factor in fine- seee-tunig performance, thus preference drift can greatly affect the alignment of models (Carroll al. 2024). The development of preference optimization algorithms and theory to handle preference drifts are therefore crucial.
1. Introduction
In this work, we propose Non-Stationan Direct Preference Optimization (NS-DPO), a novel approach that uses a probabilistic Dynamic Bradlcy-Teme model (Cattelan et al., 2013 Bong et al., 2020; Tian et al., 2024) to account for non- stationary drift in human preferences with just single additional parameter. As we only assume knowledge of the total preference drift and not which specific preferences change over time, NS-DPO re-weights each training datapoint, down-weighting older data with potentially stale preferences and up-weighting more recent ones. We empirically show the effectiveness and robustness of NS-DPO compared to stationary approaches. Our approach is summarized in Figure
The application of Reinforcement Learning from Human Feedback (RLHF) to fine-tune Large Language Models (LLMs) (Christiano et al., 2017, Stiennon et al. 2020
Ziegler et al. 2019 Ouyang et al. 2022 Bai et al. 2022b
"Equal contribution 'Department of Computer Science, Uni- versity College London, London, United Kingdom 'Department of Computer Science and Engineering, IIT Kanpur, India "Department of Electronic and Electrical Engineering, Univer- sity College London, London, United Kingdom. Correspondence to: Ilija Bogunovic <i. ci.bogunovic@ ucl..e..kk........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Uniformly weighted gradient updotes
Predicted preference
Training
Pref. optimization DPO / IPO
Test
Yoar 2011
Yoar 2023
"We cannot be sure about this, CPUs still their
Prompt
Are 6PUs necessary for AI research?
How GPUs considered the AI community?
Year 2024
advantages over GPUs."
Will GPUs particularly be in high demand?
Preferred response
"Net really, CPUs are more "6PUs play crucial role for deep learnina, due their special design."
flexible accessible compared to GPUs."
"Because its design suits deep learning research, the demand for 6PUs will keep rising"
NS-DPO (Nlon-Staiiorar)
Farther: less relevant
Recent: relevant
Dynamic Brasley- Tarry Model
Gradient updates weighted by time distance
Figure 1: Human preferences are dynamic and influenced by a variety of factors (e.g. environment change and socictal influence). However, standard preference optimization approaches (e.g.. DPO and IPO (Rafailov al.) [0224] Azar et al.] 2024)) do not account for this non-stationarity. In contrast, NS-DPO robustly learns non-stationary data by using Dynamic Bradley-Terry model, and adjusts the loss to discount older datapoints and concentrate learning on the latest data.
Related work. One of the primary applications of the RLHF
current timestep, whilst (Bogunovic et al., 2016 Zhao et al. 2020) investigate a restarting strategy. Similarly to the strategy of (Russac et al., 2019). we use an exponentially weighted discounting term to re-weight points close the current timestep. (Faury et al.
framework is fine-tuning large language models (LLMs)
(Christiano et al. 2017 [Stiennon et al., 2020 Ziegler
(et al. 2019 Ouyang et al.
2022 Bai et al. 2022b>. key
component of this is the Bradley-Terry model (Bradley
2021: Wang et al. 2023) apply this approach to the case of generalised linear bandits first proposed by (Filippi et al.
& Terry| 1952) which learns a reward signal from paired human preferences. (Rafailov et al. 2024) propose Direct Preference Optimization (DPO), which implicitly uses the Bradley-Terry model, to fine-tune an L.M directly from a preference dataset. A variety of alternatives to DPO have been proposed which adapt or do not use the Bradley- -erry
2010). (Pacchiano et al.)
2021 Saha 2021; Mehta et al. 2023) focus on the duelling bandit setting, where only preference feedback between two actions is provided by the environment. In this work, we provide the first theoretical guarantees for the popular offline setting where the true reward parameter (used to label training data) is allowed to change over time.
model (Azar et al. 20244] Amini et al.
2024, Meng al.
2024 Cen et al. 2024 Xu et al., 2023).
Other approaches
such as maximizing utility function (Ethayarajh et al.
Main contributions. We propose NS-DPO, a direct prefer- ence optimization method that accounts for non-stationary preferences in the dataset via a Dynamic Bradley-Terry model. NS-DPO modifies the training loss with a single exponential weighting parameter 7, and thus represents a simple and computationally efficient extension of the popu- lar DPO algorithm. We provide an upper bound on the regret of NS-DPO for log-linear policies given standard data cov- erage assumptions used in offline learning. To explore the performance of NS-DPO, we construct non-stationary pref- erence datasets from a variety of existing popular datasets; including GlobalOpinionsQA (Durmus et al. 2024). Help- ful & Harmless (Dai et al., (023), and UltraFeddbcck (Cui) et al. 2023). We demonstrate that NS-DPO significantly outperforms stationary DPO and other relevant baselines on these non-stationary datasets with varying degrees of pref- erence drift on Llama LLM models (Touvron et al. 2023 Dubey et al. 2024).
2024). improving from the policy in the previous iteration
(Munos et al. 2024 Rosset et al.; 2024) Tang et al. 2025). and improving the response at inference-time (Mudgal
jet al.) 2024; Son et al.) 2025) have been investigated. Our work is the first consider direct preference algorithm
using a Dynamic Bradley-Terry model.
A variety of work has analyzed the RLHF problem from theoretical standpoint. (Xiong et al. 2024) provide subopti- miality bounds of policies in the offline, online and hybrid settings under linear rewards. They do not directly analyze the performance of DPO, but propose as a practical imple- mentation of the oracle. (Zhu al. 2023] Chowdhury et al. 2024) analyse the offline preference learning and DPO set- tings, respectively. (Chowdhury et al. 2024) address noisy preferences with modified version of the DPO algorithm, presenting confidence bounds for neural policy classes and suboptimality bounds for the setting with log-linear policies.
Parameter drift has been widely studied in the bandit literature. (Cheung al. 2019) propose using a sliding window to estimate parameters with data points close to the
2

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
2. Preliminaries
x (2) x(g]=) log Tyer ta lz) All the Where h (2,, a, a') = log )erl(a m) methods introduced in this section, including DPO, are all stationary as they assume the reward model does not change with time. However, this assumption does not hold when training on real-world data. The changes in preferences over time, captured in the dataset, appear as label noise to the stationary methods.
Stationary RLHE In the stationary RLHF setting ZZieglrr jtt al., 2019] Ouyang et al. 2022). the goal is to find a suitable LLM policy π. whose response a, to a prompt x, maximise reward function r(z, i...
[nt = Ba a)) -rr--rDnn (Tr,a (-1x)]]
(1)
3. Learning Under Preference Drift
Here, the KL-divergence prevents the learnt policy from deviating too far from some reference policy Teff that has characteristics we wish to preserve in the final model. This is controlled by the parameter т O. In practical settings, human feedback is too complex to capture in hand designed reward model, and we resort to learning model from human preference data.
To address the problem of preference drift, in datasets col- lected over period of time, we propose Non-Stationary Direct Preference Optimization (NS-DPO), NS-DPO in- corporates the Dynamic Bradley-Terry model, which in- cludes a non-stationary reward model r(z, a, t). Here E {1...., - 1] denotes a time step in the past, and Te N+ denotes the current time step, where we are evalu- ating the trained policy. Under the Dynamic Bradley-Terry model, the probability of response a; being preferred to a² is
Bradley-Terry Model. A human preference dataset consists of prompts and two possible responses D = {(a,, as, af) Fe(n) where a, is the response preferred to d', and n is the number of datapoints. To learn reward model from this dataset, we assume the preferences are gen- erated by Bradley-Terry (BT) model (Bradley& & Terry 1952) where the probability that a; is preferred to af is
p(a; > aalls, (:) = olt ais de ki))
(5)
where (x, a, a', ()) r(z, a, () r(x, a', f). We also as- sume the dataset has temporal information about when the human preference between the two responses is expressed, D = { ag a', (elveln) For the ease of indexing data- points, we assume I, S L ifi
pla; > df|l:)) = (r((s, az) - r();,
In Equation 2 o() is the logistic sigmoid function and r(z;, a) is the reward model of human preferences we do not have access to and wish to learn. We parameterise the re- ward, typically as a single layer MLP on the last layer of the reference policy model Tref (Ziegler et al., 2019), and then learn the parameters using a maximum likelihood estimator. An LLM can then be fine-tuned on the objective Equa- tion I using Reinforcement Learning (RL). is important to note that the BT model captures many of the inherent assumptions we make about our data, which include the stationary nature of the underlying data generating process.
Rather than making explicit assumption how the reward function varies over time, consider setting in which the degree the reward can change is upper bounded. We denote this upper bound as Br. This is mild assumption on the temporal variation, and allows the reward to vary drastically at any point in time over all 7 - 1 steps in the training data. We formalise this in Assumption [ (Section 4) and use to show that the convergence of NS- NPO depends upon By. To learn given this drift, employ an exponentially weighted maximum likelihood estimator (Faury et al. 2021 Russac et al. 2019 Wang el al. 2023). where the datapoints are re-weighted such that losses incurred at the most recent datapoints are prioritised.
Direct Preference Optimization. Recent work by (Rafailov ct al. 2024) avoids the training of an explicit reward model in the stationary RLHF process by optimizing the LLM policy directly from human preference data. To do this, the analytical solution to the stationary RLHF objective is rearranged into Equation
To learn a suitable reward model in this setting, we define the reward at time step T as r(x,, a, T) R, where R is the space of reward values. We estimate the reward function at timestep T, by maximising the exponentially weighted neg- ative log- likeelikliooo of the Dynamic Bradley-Tere Terry model:
|
to derive an implicit reward
r(x, a) = r log T(aax) Zzer (a(r)
+ T log Z(z),
(3)
(DBT(F)
where Z(x) is normalisation constant. This is substituted into the negative log likelihood of the Bradley- Terry model (see Equation (2) resulting in the direct preference optimization (DPO) objective
Σ
-T log (hr (1, ai, al, T)). (6)
In Equation 6 7 E (0, controls the rate at which older
[(T)= Σ (a,
- log (1h
(4)
datapoints are discounted. The loss recovers the stationary Bradley-Terry model as 7 - 1. We show in Theorem 3 that
3

---

Performance measure and Optimal Policy. Let 0r E O denote the parameter that minimises the (regularised) NS-DPO loss defined in Equation (12). We assess the performance of the policy TH,- using the difference of Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
the optimal value of y can be obtained when we know Br. the details of which provide in Appendix DD2.33
Policy Class. We use the policies parameterised by 0 θ Rd of the following form
Remark 1. We do not assume any knowledge of which preferences change and which remain fixed as part of the total preference drift. As result, we down- down-weight all older points in the dataset with the exponential scaling term, 7,
exp(fe(x, o))
I= {To(aa))) = Dn'EA exp(folz,a'))
(II)
where So(2; a) E R is differentiable function. For our analysis, we consider the case of log-linear policies where fo is linear: az = o(z, TTT and the feature map δ(x, a) is a d-dimensional vector. This is motivated by the reward model introduced in (Ziegler et al., 2019) where the last hidden layer of the LLM is used as the feature embedding function d(z,a).
Offiine Non-Stationary Direct Preference Optimization. The derivation of NS-DPO follows as previously shown in Section 2 for the stationary case. We first define the RLHF objective at timestep T as
[T) = Bawxxarn
(7) where we are interested in maximising the reward function r(z, a, T)) that reflects human preferences in the present (i.e., the current time step). We note the prompt distribution X' and the reference model Tref do not vary with time. As we consider the reward model at T, we derive an implicit reward of the same form as Equation (3). This relates the optimal policy and reward function of Equation
Loss Function with lz regulariser. For the analysis of log-linear policies, we regularise the NS-DPO loss with squared fg-norm of 8, 7² and non-linearity coefficient Ca,r (explained in Appendix D
ACo,7
F (0) NS (0) CN)(
11012.
(12)
2
(alx) log (x), Zyr (a
r(ar, a, T) T log
where क is the optimal policy that optimises Equation and Z denotes the normalisation constant of - We then parameterise the policy in Equation 7 using the parameter Or, which enables expressing the implicit reward with respect to the parameter as
non-stationary RLHF objectives between T6r and π in
Equation
17T = JT(T)) - TI (T
Exax Bawntt (r a, T)]
TT, + T log Ze, (I), (9) Tref (alz)
DDKi. [m} (- (=
T8, (z, T) = rlogg....
-E ( (r T)]
where Ze, denotes the normalisation constant of Te.. We apply Equation (
+ +DKII.(TT (- Fe (12))
(13)
into the exponentially weighted negative log likelihood Equation (6) to derive the NS-DPO objective
where r(., T)) denotes the true reward function at time T, and z/. denotes the optimal policy against which we compare the performance of our algorithm. Given a reference policy Treef the optimal policy अं defined as the policy which optimises the RLHF objective at time step T
CNS (O0(
= Σ
I T- log (Titoy, (xi, ai, di))
arg max Benx [((x,a,)) 7DKi [T((T Tret( (1>0)] #ell
(10)
(14)
4. Theoretical Analysis of Offline Non-stationary DPO
Similarly, we can define the parameter 07 of the optimal policy in each time step t [T]
In this section, we analyse the performance of NS-DPO in the offline setting. We assume the use of lig-linear policies, and present how the preference drift affects the estimation error and regret bound of the algorithm. We provide the sample complexity of the algorithm, which recovers O(n -1/2) when the preferences are stationary. See Appendix for further details.
[r(s, ()-,,a,))-DKullee (-xx) (ee()
arg max BawN 6
(15)
We now introduce assumptions on our setting. The pref- erence drift is defined as the change in the true underlying parameter 07 O, E [T] of the optimal policy x

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
at each time step. We do not constrain how the optimal parameter changes, but instead upper bound the total possible preference drift in the environment before time step T. This upper bound is known as the variation budget.
is motivated by the fact that we are assuming no knowledge of the dynamics of the actual preference drift. Note that (T)) in the assumption is the notation for the complexity, which is different from the parameter set θ in Assumption
Assumption 1. (Variation Budget Bound) The parameter drift of 0z E + across I" timesteps is upper bounded as Σ ||0 0 l22 s Br where Br O is known constant.
Assumption 5. (Temporal Coverage) For each time step t E [T - 1I. the number of datapoints in the training set is between m and m, where m > 0 and m.. > m are constants (i.e., n = (T)).
We move to general assumptions for the learning process. We bound the 2-norm of the feature and parameter spaces.
4.1. Theoretical Results
A Assumption 2. (Boundedness) The parameters and features are bounded: 0 E @ where θ = (0 E Rd |101|2 S W) and 4 = { a) E Rd || l2 S L}.
Estimation Error. To bound the expected regret of the pol- jey trained with NS-DPO, bounding the difference between the optimal and the learnt parameter is required. To analyse the parameter estimation error, we define the discounted covariance matrix of the offline dataset as
is known that an equivalence class of reward models leads to the same preferences under the Bradley-Terry model (Rafailov et al., 2024). This is similarly true in the case of the Dynamic Bradley-Terry model, because the implicit reward of NS-DPO, shown in Equation (8) relates the reward to the policy parameters 0. We thus construct the following constraint on the policy class to properly specify the problem (Chowdhury et al. (2024).
1 Σ
- 11 i=1
n
(17)
Assumption 3. (Identifiability) The optimal policy in each time step t corresponds to single parameter in θ, which satisfies Equation (55 10 = 0 vt E [2]. where 1 E IRd is a vector of 1s.
where di = H( (:) (xF; a() is also introduced for brevity. Under the assumptions from Section we introduce bounds on the estimation error of the parameter 0r. which
minimises the NS-DPO loss in Equation (12), with respect to the true parameter 07 and Σ:
In the offline setting, our learning is constrained by the available dataset D. A standard assumption in the of- fline learning literature is that of data coverage (Chowd) hury et al. 2024 Zhu et al. 2023). The data coverage assumption ensures that the reference policy Tref suitably explores the space of plausible responses of the optimal policy. We define the population covariance matrix as Σ, B[ő(,, ())(x, a] - [[, B(o(,, T))T, where the ex- pectation is calculated over samples æ ~ X, a ~ (-11x). The condition number K6, compares the coverage of the two policies x and Fref
100 0TIlS
(18)
where À >> 0 introduced to guarantee the inversion of the matrix Σ + Al. The upper bound on the estimation error is shown in Theorem 2 and a detailed proof of the result is provided in Appendix D.I Our analysis differs from the stationary case (Chowdhury et al., (024). as we consider the temporally discounted datapoints in the NS-DPO loss. This is reflected in the covariance matrix Σ by the inclu- sion of the TT- term, which decreases the influence of observations that happened further in the past. As part of our analysis, we separate the estimation error into a learn- ing term and tracking term. This tracking term accounts for the error introduced by the non -ta-ioatinn nature of the environment, depending upon By and the choice of the algorithm to upper bound it. We outline suitable choice for below.
U E HERd ETTeee Amin (Diree)
Amax (Ex))
VT E I1: Rgg sup
(16)
max, Kú denote the maximum value of Kgy The definition of K2 requires that the reference policy sufficiently explores the feature space, which leads to the following assumption.
while
Assumption 4. (Feature Coverage) The reference policy Teel satisfies Amia (Ex,.)) > 0.
In a varying setting, the quality of the dataset D also depends upon its temporal coverage. We use the following assumptionm which also guarantees a minimal amount of data in each time step. Having enough data in each time step
Theorem (Estimation error of ÔT-) Let 8 E (0, 1), 0, > 0. Let dr denote the minimiser of the NS-DPO loss defined in Equation (12). Let 0r E e denote the parameter obtained by performing the parameter projection procedure
5

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Reward Accuracy.
2.7
Reward Accuracy
0.7
Rewardd Accuracies
0.7.
Reward Accuracy
0.7
0.5
0.8
0.
0.3
6.5
0.5
93
84
0.4
04
52
06 02
52
66
63 01
ii
Eep
Change Point
Change Point
top Change Paint
-Change Paint
Figure Experiment results conducted UtraFcedbbck-RM dataset with preference drift.[Left] pair = 0.7. [Center Left] Pair 0.9. [Center Right] paitr = 0.95. [Right] Pairr = 1.0. As paist. the percentage of training datapoints with flipped preference increases, DPO fails to Iearn the preference distribution at T = 101. Mcanwhile, NS-DPO shows robust performance under various values of Pait maintaining reward accuracies above 50%. As tep. the change point of the reward model happens later in time, the gap between stationary approaches and NS-DPO gets larger. The experiments are run under reward model shift from PAIRRM ARMORM. Llama 70- chathhhtthhhhe used, and training dataset consists 100 time steps.
on 0r. Then with probability at least - 5:
NS-DPO uses the discount weight y = -1- (
3/4 to address the non-stationarity the dataset, which results in the regret bound above. However, our approach is general enough capture the stationary setting, which corresponds to BT 0. By setting 7
201
+ log(1/0)
| 1st S2VAW
TCo,
n
( with 0 <
learring
we show that the tracking term in the estimation error bound goes to zero. Corollary 4 shows that the widely considered stationary setting is special case of NS-DPO. We provide the detailed proof Appendix D.3
16L m
dm
BT
(19)
T(1
n
tracking
Corollary 4. (Regret bound under stationary preferences) Let By - 0, ô E (0, $1. > Let 0r E denote the min- imiser of the NS-DPO loss (Equation (12). Then, for X > (oggg// / n,, some constant C₁ 0, 7 1- Br and 0 2/3, we have with probability least 1 28:
where C1 > 0 is constant.
Expected Regret Bound. Starting from the definition of the expected regret in Equation (13), the regret can be expressed with the estimation error in Equation (19). We then use our results in Theorem 2 to complete the analysis. The details of the regret analysis are deferred to Appendix D.2]
4TKm
VVW++
C1
log (18)
lim Roft By
Theorem 3. (Regret bound of ÔT) Let (0, f O.
TCo,r
Let 0r denote the parameter in θ which minimises the
O(n 1) under
NS-DPO loss (Equation (12)) on an offline dataset. The
and recover the complexity of Reft stationary preferences.
following bound holds with probability at least 28 and
when X= 2 Clll (4d/ /n:
5. Experiments
TRRM] 7) Reaft <
2m - -T 10T -07 2tAA
In this section, we empirically evaluate NS-DPO's ability learn under preference drift. We analyse how NS-DPO performs under different types of preference drift and different strengths of preference change using Llama 22 7ww-ccht-aaee (Touvron et al., (023) and
where C1 denotes constant. When 7 11
Br
3/4
Reff satisfies:
Roft = Ō [ =d n 11/4)
Llama 3. 2- (-ubby et al.] 2024). We provide experiments in Appendix C,5 to further support our
theoretical results in Section 4.1 We provide code for our experiments.
Standard offline bandits and RL algorithms assuming the stationarity of the underlying scalar-valued reward achieve 1/21 regret (Wang et al. 2021 Zhan et al. 2024 Qiao & Wangl 2024 Cen et al. 2024). For stationary preference-basee rewards, (Chowdhury et al. 2024) show gap for DPO algorithm, whereas Nika et al. 2024) obtain an O(n -1/2) regret. Unlike these prior work assuming stationary preferences,
5.1. Experimental Setup
test NS-DPO in an LLM setting, we create three pref- erence datasets with known and controlled preference drift.
an O(n-111n1 1/4) regret/sub-optimalle
http:///ghhaaaccom/grrresssssssp

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
DPO
NS-DPO
NS.DPO
Roward Accuracy
Roward Accurncy
Reward Accuracy
0.6
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
0.2
0.2
0.2
50
70
50
70
30
50
70
Eep Change Point
Pep Change Point
Tep Change Point
Figure 3: NS-DPO consistently outperforms DPO and IPO the change point. bep the present T 101 for varying strengths of preference shift on the TV-HH dataset using the Pait = 0.9. We note that as the value tep increases, the performance difference between NS-DPO and the baselines increases. This is because as the change point moves closer to the present time step, the number of samples available from the updated preference distribution decreases. NS-DPO discounts samples with old preferences, focusing learning upon the small number of samples with up-to-date preference labels.
model. [Left]] paia = 0.7. [Middle] paif 0.8. [Right]
Creating Non-Stationary Preference Datasets. To create datasets with varying preference drift, we select two reward models r1, 72 that result in different preferences for the re- sponses a and a'. We assign each datapoint an arbitrary time across 100 timesteps & [100] and adjust the response pref- erence according two main modes of preference change, sudden gradual. For sudden preference change, we select a change point tep [100] for datapoints with a time before tep we assign preferences based on r1 and for points after tep we assign preferences based on r'2- For gradual pref- erence change, we linearly interpolate the reward of each prompt response pair (x, a) between F1 and 72 over some subset of the timesteps grad C [100] (see Appendix C.2).
and Llama-3, -.maa..--lb-it. (Tovrron et al. 2023 Dubey et al. 2024) for both fine-tuning and the reference model. To reduce the compute demands fine-tuning Ll 1aaa-m-7b-chat-222
we train LoRA weights (Hu) et al. 2022) (see Appendix C.4 for further details). We fine-tune all parameters of Llama-3. 2.ama-3.2-186 2-1b-it.
Evaluation Metrics. To compare the performance of NS- DPO with the baseline algorithms, we evalute the Reward Accuracy and Win Rate. The reward accuracy the pro- portion of examples for which preferred response is greater than the implicit reward of the least preferred response in the test split of dataset. The win rate of NS-DPO versus baseline algorithm in the TV-HH dataset is the portion of responses generated by the NS- DPO trained policy that score higher under the true reward model at time T, than those generated by the baseline policy. For each sample in the test set of the TV-HH dataset, we limit the maximum length of generated responses to 2048 tokens. For the UltraFeedback-LN. experiment, we evaluate the performance of each algorithm by using AlpacaEval2 DDuoos et al.) 2024). We report the Length- Congth-oodeeleedd Win Rate (LCWR), which compares the responses generated by given model to responses generated by GPT-4.
implicit reward of the
Preference Change Strength. Finally, we also adjust how the strength of preference change affects the performance of NS-DPO. We introduce Paifl. which is the portion of datapoints included in the dataset whose preferences change when assigning preferences according to r2 instead of r1- We provide further details in Appendix C.I
Datasets. We create non-stationary preference datasets for the GlobalOpinionsQA dataset (Durmus et al. 2024) and Helpful-Harmless dataset (Bai et al. 2022a) using the helpsteer-helpfulnnss and beavertails-is safe outputs of the ARMORM model. We use the Ultrafcedback dataset (Cui al. 2023) to create UltraFeedback- -MM, which contains the preferences of the PAIRRM (Jiang et al.) 2023) and AR- MORM (Wang al. 2024 reward models. We also create StrraFeedbackLLM, which uses the responses provided in the original dataset while preferences are determined based on the language model used to generate the response. We provide further details of how the non-stationary preference datasets are created Appendix C.2
LLM Baselines. We compare NS-DPO against station- ary DPO and Identity Preference Optimization (IPO) (Azar et al. 2024). We also construct In-Context Learning (ICL) algorithm referred to as tDPO, in which information about the time step is appended to the prompts of the data. All algorithms use the same supervised fine-tuned (SFT) model as the reference model. We use the SFT procedure from (Rafailov et al. 2024), training the model on the pre- ferred responses in the dataset. NS-DPO uses T 0.1 and y 0.95 for fine-tuning T.1
Language Models. We Llama -1-mm-2-7b--- 2-7b-chat-hf
with 2C NSGO dataset and UltraFeedback dataset. For
hhtpss//huggiingfacccoommm// 2--h----haa
https://huggingffaecc..omttte http::/gwwg.e..oe/oooooeeeeeeeee hww.:e/eooooooeeaaaaaaaaaaaaaaaa In-trrue

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
0.7
DPO
NS DPD
0.7
DPO
NS-DPD
0.65
Reward Accuracy
0.60
Reward Accuracy
Reward Accuracy
0.55
0.5
0.5
0.50
0.45
0.4
0.4
0.40
0.35
66
66 Egr Change Point
Fo Change Point
Figure 4: [Left Middle] NS- NS-DPO outperforms DPO as the change point, tep nears the present time. 101, for pai and
Paiff = 1.0 respectively on the TV-HH dataset finetuned on aama 13-1--1-- model. [Right] NS-DPO outperforms DPO in settings where preference drift is gradual across multiple timesteps on the TV-HH dataset.
LCWR
the Time Varying Helpful-Harmless (TV-HH) dataset, we adjust the value of 7 as = 1 - ( T00-1.p log(100). For Llama-3. 3. 1b bit..tittte..... we
Pdiff
tep
NS-DPO
SW-DPO
DPO
0.7
21
8.93
6.09
7.29
1.0 and 7 0.85.
0.7
51
8.38
4.93
7.85
0.7
81
7.85
4.63
7.17
1.0
21
9.00
6.71
8,23
Autr 0.9
0.70
Pei 0.8
1.0
51
7.41
5,59
6.99
0.65
Aur 0.7
1.0
81
7.36
4.83
6.49
0
9.12
8.81
8.81
Rato (NS DRD
Length-Controlles Win Rates (LCWRs) of 1.lama33.2-10 models, evaluated by AlpacaEvva22. The models are trained with UttraFeddack-LLM dataset (See Appendix C..)p NS-DPO outperforms stationary DPO under various types sudden preference drift, with higher preference by GPT-4 evaluator.
Table
0.55
0.50
0.45
40
50
60
70
80
90
Exp Changepoint
consistently achieves a win rate of 0.5 beating DPO at values of Pait- The UlrraFedback-LMM experiment in Section 5.2 also shows that NS-DPO effectively addresses the preference drift, having consistently higher LCWR than the baselines including DPO. We observe similar trend in Figure 2 Figure 3 and Figure [Left Middle], where the reward accuracy of NS-DPO matches that of IPO and DPO for early change points and outperforms the stationary base- lines later change points on 1 1atiaa---b-catt-t-- and ama 3-10------ respectively.
Figure 5: NS-DPO returns more aligned responses than DPO, according to the reward model at T = 101, when sudden preference shift occurs at later change points. We finetune 1asma-3-1b-it
the TV-HI dataset change points and pair, record the mean rate across 600 samples from the test split over runs.
range of of the win
5.2. Experiment Results
How robust and effective is NS-DPO under varying strengths of sudden preference drift? We investigate how different strengths of sudden preference drift affect the performance of NS-DPO. In Figure 2 Figure 3 Figure 4 [Left Middle], and Figure 5 we compare how different values of Paiff, the portion datapoints with preferences that change, and the change point tep, the time at which the preference change occurs, affects the alignment of models trained with NS-DPO versus those trained with stationary preference algorithms. In Figure we finetune Llama-3. 2 -b-itt on the TV-HH dataset and report the win rate against DPO. NS-DPO and DPO responses perform broadly the same when changes occur early tep E (10,30), however, when the change point occurs later, NS-DPO
Stationary algorithms treat the non-stationary preferences within the data as label noise. Thus, for early change points, the stationary algorithms see broadly correct preference labels as the majority of the dataset consists of data after the change point. As the change point occurs later, more of the dataset likely to have the incorrect preference, which is then learned by the stationary baseline, whilst NS-DPO focuses learning only the later correctly labeled samples. It is also important to consider how many data points are affected by the preference change. We capture this the Paitt parameter. As more of the data experiences preference drift (higher values of Pai)) the more beneficial nta-stationay algorithm like NS-DPO is. We can see this
8

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Reward Accuracy
0.7
Reward Accuracy
0.
Reward Accuracy
0.7
0.6
0.6
9.6
0.5
0.5
0.5
(y- 0.95)
MS-DPD ( 0.95]
DPD Iy 0.95)
0.4
DPC
0.4
DPO
0.4
DPO
IDPO
IDPO
IDPO
0.3
2000
.000
6200
0003
020
2000
2253
6000
8002
0.0
1500
2:00
3:03 4500
5002
5003
7002
0000
Trained Examples
Trained Examples
Trained Examples
Figure 6: Llama-2- 7b--hhtt-htb experiment results using 2C NSGO dataset. [Left] Opinion drift from the US to Germany. [Middle] Opinion drift from the US to Japan. [Right] Opinion drift from the US to Brazil. NS DPO stays robust to the statiioonna present in the dataset and achieves reward accuracies above 60%, while stationary methods show dropped reward accuracies of around 55% Including the time steps in the prompt (1DPO) does help meaningfully improve the performance of stationary DPO.
in Figure 3 where NS-DPO achieves the strongest win rate at later change points when Pditt 0.9.
that based on the definition of tep stationary datasets are equivalent to datasets with lep = 0. As demonstrated in Figure 7 the model trained with NS-DPO using the station- ary UltraFeedback dataset shows almost identical reward accuracy as the model trained with stationary DPO. The AlpacaEval2 results in Section 5.2 also support this, where NS-DPO actually shows higher LCWR than DPO even with- out any preference drift in the dataset.
How does NS-DPO perform under gradual preference drifts? Here we investigate how LLMs trained with NS- DPO perform when preference drift happens gradually over time. In Figure 4 [Right], we see that NS-DPO outper- forms the DPO reward accuracy by over 10% on the TV-HH dataset with gradual preference drift. We note that the perfor- mance of NS-DPO is dependent upon the value of y chosen, however both approaches outperform the stationary base- line. The experiment results on the 2C NSGO dataset, which given in NS-DPO shows significantly better performance compared to stationary DPO, showing performance gap of nearly 10% in reward accuracy. This difference is mainly caused by stationary methods failing to efficiently learn from datapoints at later time steps. IDPO, which trains the policy with time step information appended to the prompt, does not show a significant difference from stationary DPO.
6. Conclusion
In this work we propose NS-DPO, a practical and prov- ably efficient approach for preference optimization on non- stationary offline datasets. With standard assumptions the offiin learning setting and minimal assumption of having only a upper bound BT on the preference drift, we provide a theoretical analysis on the performance of NS-DPO in the case log-linear policies. NS-DPO achieves a sample com- plexity of O(n Onn11)). We also show that as the preference drift in the environment diminishes, B3 - 0, the complex- ity of the regret recovers O(n 0nn=, found in the stationary setting. We further support this result with a suit of empirical results on a synthetic setting. We also investigate the appli- cation of NS-DPO to LLMs, creating several non-stationary preference datasets with varied levels of preference drift, and show that NS-DPO shows superior performance to standard preference optimization algorithms and In Context Learning approaches on these datasets. Even in stationary settings, NS-DPO matches the performance of stationary algorithms. This motivates the usefulness of our approach when the exis- tence of preference drift in a dataset is unknown, as applying NS- NSO will not hurt performance even if the preference drift is too small to matter. NS-DPO introduces only a single parameter y in the stationary DPO loss, which is notably easy implement and test with. Our approach can be easily extended to the online setting where data is sequentially pro- vided as time passes. NS DPO can also be adapted to learn at time step that is not the present by discounting both past and future preference as a function of their distance from the time step of interest. We leave these ideas for future work.
also simulates a gradual drift of preferences, Figure 6
0.7
Reward Accuracy
0.6
0.5
0.4
NS-DPO 0,95)
DPD
0.3
4000
6000
8000
Trained Examples
Figure 7: Training curves of NS-DPO and DPO trained with the UltraFeedback dataset without preference drift (tep 0)e trained model is initialised from Llama-2- 2-7b-chat-hrf NS- DPO matches the performance of DPO even in stationary settings.
How does NS-DPO perform under no preference drift? While NS-DPO is proposed to address scenarios where preference drift happens, one natural question that follows is whether NS-DPO performs worse than stationary DPO when the dataset is stationary, containing no preference drift. In order to answer this question, we train NS-DPO and sta- tionary DPO on the UltraFeedback- RM and Ultraeeddacck- RM datasets without introducing any preference drift. Note

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Acknowledgements
Bogunovic, I., Scarlett, J., and Cevher, V. Time-varying
gaussian process bandit optimization. 2016.
IB was supported by the EPSRC New Investigator Award EP/X03917X/1; the Engineering and Physical Sciences Research Council EP/S021566/1; and Google Research Scholar award. WB was supported by the Engineering and Physical Sciences Research Council EP/SO21566/1 and Fluidstack AI Cloud Platform Research Funding.
Bong. H., Li, W., Shrotriya, S., and Rinaldo, A. Nonpara- metric estimation in the dynamic bradley-terry model. In International Conference on Artificial Intelligence and Statistics, pp. 3317-3326. PMLR, 2020.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: L. the method of paired comparisons. Biometrika, pp. 324-345, 1952.
Impact Statement
The development of NS-DPO provides significant ad- vancement in the field of preference optimization for non- stationary scenarios, addressing a critical challenge in adapt- ing to environments with temporal preference drift. By intro- ducing a practical and theoretically rigorous approach, NS- DPO enables robust optimization with minimal assumptions, achieving competitive sample complexity and empirical per- formance in both stationary and -ta-statonnary settings.
Carroll, M., Foote, D., Siththaranjan, A., Russell, S., and Dragan, A. Ai alignment with changing and influence- able reward functions. In International Conference on Machine Learning, pp. 5706-5756, PMLR, 2024.
Cattelan, M., Varin, C.,, and Firth, D. Dynamic bradley- terry modelling of sports tournaments. Journal of the Royal Statistical Society Series C: Applied Statistics, pp. 135-150, 2013.
Whilst NS-DPO offers clear approach address temporal drift within offline dataset, it does not take the qualitative aspects of the inputs into consideration. Aligning LLMs can be a complex task and great care should be taken to avoid introducing sources of bias during training, which lead to misaligned harmful models. When using NS-DPO practitioners should be cautious of newer data that may contain biases as such biases may be amplified in training by the NS-DPO discount parameter.
Cen, S., Mci, J., Goshvadi, K., Dai, H., Yang, T., Yang, S., Schuurmans, D., Chi, Y., and Dai, B. Value-incentivized preference optimization: A unified approach to online and offline rlhf. CORR, 2024.
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., and Silva, B. C. Rlhf deciphered: A critical analysis of reinforce- ment learning from human feedback for Ilms. arXiv preprint arXiv:2404.08553 2024.
References
Chen, L., Li. S., Yan, J.. Wang, H., Gunaratna. K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. Al- pagasus: Training better alpaca with fewer data. 2024a.
Amini, A., Vieira, T., and Cotterell, R. Direct preference optimization with an offset. pp. 9954-9972. Association for Computational Linguistics, 2024.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning convertsweak language models to strong lan- guage models. In International Conference on Machine Learning, pp. 6621-6642, 2024b.
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in ai safety. 2016.
arXiv preprint 1606
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M.,, Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447 4455. PMLR, 2024.
Cheung, W. C., Simchi-Levis D.,, and Zhu, R. Learning to optimize under non-stationarity. In International Confer- ence on Artificial Intelligence and Statistics, pp. 1079- 1087. PMLR, 2019.
Chowdhury, S. R., Kini, A., and Natarajan, N. Provably ro- bust dpo: Aligning language models with noisy feedback. International Conference on Machine Learning, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.058822 2022a.
Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. In International Conference on Neural Information Processing Systems, pp.
2017.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 02,0073, 2022b.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, 2023.
10

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning from human feedback. CoRR, 2023.
Hu, E. Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models. International Conference Learning Representations, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.2 22.0224477.178,, 2024.
Hua, E., Qi, B., Zhang, K., Yu, Y., Ding, N., Lv, X., Tian, K., and Zhou, B. Intuitive fine-tuning: Towards simplifying alignment into single process. CORR, 2024.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0mm 04445, 2024.
Jiang, D., Ren, X., and Lin, B. Y. Lim-blender: Ensembling large language models with pairwise ranking and gener- ative fusion. In Annual Meeting of the Association for Computational Zinguistics-Volumg 1, pp. 14165-14178, 2023.
Durmus, E., Nguyen, K., Liao, T., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds Z., Hernandez, D., Joseph, N., et al. Towards measuring the represen- tation of subjective global opinions in language models. Conference on Language Modeling, 2024.
Johnson, B. and Mayorga, M. Temporal shifts ameri- cans' risk perceptions of the zika outbreak. Human and Ecological Risk Assessment: An International Journal, 27(5):1242-1257, 2020.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic opti- mization. International Conference on Machine Learning, 2024.
Khaki, S., Li, J., Ma, L.. Yang, L., and Ramachandra, P. Rs-dpo: A hybrid rejection sampling and direct prefer- ence optimization method for alignment of large language models. In Findings of the Association for Computational Linguistics, pp. 1665-1680 2024.
Faury, I.., Abeille, M., Calauzènes, C., and Fercoq, O. Im- proved optimistic algorithms for logistic bandits. In In- ternational Conference on Machine Learning, pp. 3052- 3060. PMLR, 2020.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K. R., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. CoRR, 2024.
Faury, L., Russac, Y., Abeille, M., and Calauzenes, C. Re- gret bounds for generalized linear bandits under parame- ter drift. arXiv preprint arXiv:2103.05750, 2021.
Li, Z.., Xu, T., and Yu, Y. Policy optimization in rlhf: The impact of out-of--referenc data. arXiv preprint arXiv:2312. 2023.
Filippi, S., Cappé, O., Garivier, A., and Szepesvári, C. Para- metric bandits: the generalized linear case. In Inter- national Conference on Neural Information Processing Systems-Volunee 1, pp. 586-594, 2010.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. International Conference on Learning Representations, 2024.
Garivier, A. and Moulines, E. On upper-contidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805. 2008.
Lu, C., Holt, S., Fanconi, C., Chan, A.. Foerster, J., van der Schaar, M., and Lange, R. Discovering preference opti- mization algorithms with and for large language models. Advances in Neural Information Processing Systems, 37: 2024.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F,, Ramé, A., Mesnard, T., Zhao, Y., Piot, B., et al. Di- rect language model alignment from online ai feedback. CoRR, 2024.
Mehta, V., Das, V., Neopane, O., Dai, Y., Bogunovic, I., Schneider, J. G., and Neiswanger, W. Sample efficient reinforcement learning from human feedback via active exploration. CoRR, 2023.
Hendrycks, D. and Mazeika, M. X-risk analysis for ai research. arXiv preprint arXiv:2206.080 058622 2022.
Hong, J., Lee, N., and Thorne, Reference-free monolithic preference optimization with odds ratio. arXiv preprint a7Xiv:2403.07661, 2024.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 2024.
Hsu, D., Kakade, S. M., and Zhang, T. A tail inequality for quadratic forms of subgaussian random vectors. Elec- tronic Communications in Probability, 17, 2012.
Mudgal, S., Lee, J., Ganapathy, H., Li, Y., Wang, T., Huang, Y., Chen, Z.., Cheng, H.-T., Collins, M., Strohman, T.

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
et al. Controlled decoding from language models. In In- ternational Conference on Machine Learning, pp. 36486- 36503, PMLR, 2024.
Russac, Y., Vernade, C., and Cappé, O. Weighted linear bandits for non-stationary environments. Advances in Neural Information Processing Systems, 32, 2019.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. International Conference on Machine Learning, 2024.
Saha, A. Optimal algorithms for stochastic contextual prefer- ence bandits. Advances in Neural Information Processing Systems, 14:30050-30065 2021.
Son, S., Bankes, W., Yoon, S., Ramesh, S. S., Tang, X., and Bogunovic, I. Robust multi-objective controlled decoding of large language models. arXiv preprint arXivv2503.08796, 2025.
Nika, A., Mandal, D., Kamalaruban, P., Tzannetos, G., Radanović, G., and Singla, A. Reward model learning vs. direct policy optimization: a comparative analysis of learning from human preferences. In International Con- ference Machine Learning, volume 235. Proceedings of Machine Learning Research, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3024 2020.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learn- ing from human feedback. International Conference on Machine Learning, 2024.
Pacchiano, A., Saha, A., and Lee, J. Dueling rl: reinforce- ment learning with trajectory preferences. arXiv preprint arXiv:2111.04850 2021.
Tang, X., Yoon, S., Son, S., Yuan, H., Gu, Q., and Bogunovic, I. Game-theoretic regularized self-play alignment of large language models. arXiv preprint ASh::250330003000 2025.
Pang, R. Y., Yuan, W., He, H., Cho, K., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 116617-116637, 2024.
Tian, X., Shi, J., Shen, X., and Song, K. A spectral approach for the dynamic bradley- terry model. Stat. 13(3):e722, 2024.
Pentyala, S. K., Wang, Z., Bi, B., Ramnath, K., Mao, X.- B., Radhakrishnan, R., Asur, S., al. Paft: A parallel training paradigm for effective Ilm fine-tuning. arXiv preprint arXiv:2406. 1at.22446..7923, 17923, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Qi, B., Li, P., Li, F., Gao, J., Zhang, K., and Zhou, B. Online dpo: Online direct preference optimization with fast-slow chasing, CORR, 2024.
Qiao, D. and Wang, Y.X. Offline reinforcement learning with differential privacy. Advances in Neural Information Processing Systems, 36, 2024.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts CORR, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang. J., Zhao, P., and Zhou, Z.-H. Revisiting weighted strategy for non-stationary parametric bandits. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 7913-7942. PMLR, 2023.
Ramesh, S. S., Hu, Y., Chaimalas, I., Mehta, V., Sessa, P. G., Ammar, H. B., and Bogunovic, I. Group robust preference optimization in reward-free rlhf. Advances in Neural Information Processing Systems, 2024.
Wang. R., Foster, P., and Kakade, S. M. What are the statistical limits of offline rl with linear function approx- imation? International Conference on Learning Repre- sentations, 2021.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-im seff-impovve with general preferences. CORR, 2024.
Wu, J., Xie, Y., Yang, Z., Wu, J., Chen, J., Gao, J., Ding, B., Wang, X., and He, X. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. CoRR, 2024a.
12

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q- Self-play preference optimization for language model alignment. Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024b.
language models from human preferences. arXiv preprint arXiv: 2019.
Xie, T., Foster, D. Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit q apappoxximation for sample- efficient rlhf. CoRR, 2024.
Xiong, W., Dong, H., Ye, C., Wang, Z.., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. International Conference on Machine Learning, 2024.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint 1rXiv:2312.16683 2023.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401. 10020, 2024.
Zafari, F., Moser, I., and Baarslag, Modelling and analy- sis of temporal preference drifts using component-bis -based factorised latent approach. Expert systems with applica- lions, 116: 186-208, 2019.
Zhan, W., Uehara, M., Kallus, N., Lee, D., and Sun, W. Provable offline preference-be based reinforcement learning. International Conference Learning Representations, 2024.
Zhang. S., Yu, D., Sharma, H., Yang, Z., Wang, S., Awadalla, H. H., and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. In Automated Reinforcement Learning: Exploring Meta- Learning, AutoML, and LLMs, 2024.
Zhao, P., Zhang, L., Jiang, Y., and Zhou, Z.-H. simple ap- proach for non-stationary linear bandits. In International Conference Artificial Intelligence and Statistics, pp- 746-755. PMLR, 2020.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforce- ment learning with human feedback from pairwise or k-wise comparisons. International Conference Ma- chine Learning, pp. 43037-43067, 2023.
Ziegler, D. M., Stiennon, N., Wu, J.. Brown, T. B., Radford, A., Amodei, D., Christiano, and Irving, G. Fine-tuning
13

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Appendix Contents
In Appendix A
we provide further related works on DPO algorithms, different alignment settings, and a discussion of works that consider time varying alignment problems. Appendix B analyses the gradient of the NS-DPO objective. Appendix C
explains the details of experiments conducted, including the creation of non-stationary datasets for LLM experiments and the behaviour of NS-DPO and SW-DPO in the synthetic setting. We provide proofs of our theoretical analysis in Appendix D
step by step. In-depth derivations necessary for deriving the learning error are separately presented in Appendix D.4
A. Further Related Works
Recent interest in the alignment of L.LSs has lead to a wide variety of works. We briefly discuss further works that focus upon direct preference alignment algorithms.
Several approaches examine preference optimisation from game theory perspective, avoiding the implicit assumptions of the BT model. In these settings the current policy plays against previous versions to further improve performance (Swamy
et al.
2024 Rosset et al. 2024 Wu et al. [024b] Yuan et al. 2024; Chen et al., 2024b Pang et al.)
2024 Munos et al.
2024).
(Xu et al. 2023) propose a cringe loss based objective whilst (Hong et al., 2024 Pentyala et al. 2024 Hua et al.
2024) try to combine the supervised fine-tuning and preference optimization steps. (Hong et al. 2024
Hua ct al. 2024)
propose single training objective to do this and (Pentyala et al.
2024) examine combining two different models trained
on an SFT and direct preference objective respectively. Finally, (Lu al.. 2022) propose a meta algorithm which uses
an L.M to optimize the form of the direct preference learning objective itself.
Guo et al. 2024 Xie et al. 2024), where feedback is returned by human labeler or superior model. Khaki et al. 2024 Liu et al. 2024) adapt the offline settings using techniques such as rejection sampling to approximate an online setting. In this work we only consider the offline setting for simplicity, however the approach we propose can easily be adapted to the online setting. Other important directions of research include safety and robustness. (Dai et al. 2023] Ramesh et al. 2024, Wu et al., (2024a) consider robust settings where safety or group information known training time and (Dai et al.) 2023) analyse a constrained
An orthogonal direction of work is the online setting (Qi et al. 2024 Zhang al.
2024
optimization problem through the lens of safety in LLMs. Whilst these approaches look to address wide range of settings,
our work is the first to provide a solution to the case of non-stationary preferences.
(Carroll et al. 2024) consider how to correctly align LLMs under preference drift, showing several possible goals for alignment in an online setting. Whilst in the online non non-stationary setting the LLM can adapt to the changing preferences of the user, our setting considers aligning the model on an offline dataset before deploying the static model to users at test time. As such our approach is most similar to the Privileged Reward and Initial Reward settings (Carroll et al.) 2024) proposes, as we determine that the preferences exhibited in the present are the most important (Privileged Reward) and future users will interact with model aligned to preferences from their past (Initial Reward).
B. Analysis of NS-DPO Gradient
Here we analyse the gradient of the NS-DPO loss objective. The gradient of Equation parameters f is as follows:
II0 with respect to the model
VOLNS (0)
Σ
Ty T- (- as a()) (Vo log (a, - V To (c(lxx))
(20)
Gradicnt scaling
Gradient Direction
The gradient of the NS-DPO objective consists of two terms. The first term a (-ho(x; a; a()) scales the gradient update, which increases when the model incorrectly prefers response at to a and decreases when the model correctly predicts the response preference. NS-DPO only adjusts the sealing term of the gradient by discounting the scaling term further when points are temporally far away from T. The second term, Ve log Th (a, Ve log Fe (a(((xz)) controls the direction of the gradient update.
In the case of stationary preferences in the dataset (points whose preference does not change at any time 4.), the gradient of these points is still applied to the parameters 0 by the NS-DPO Loss with scaling by the term TT----1 Whilst this downweights these gradients this is price of not knowing which points have changing preferences and which points have fixed preferences within our setting. When we know that there is no preference drift, we set the value of y to 1 to remove discounts (see Appendix D.3)
14

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
C. Further Experiment Details
C.1. Controlling the Strength of Preference Drift
In this section, we give more details on how paint calculated, which is used to control the degree of preference drift as reward models are changed in the experiments. We first note that when < tep old reward model is used to evaluate the preference of the given prompt-response pair, while new reward model evaluate datapoints with 2 lepi
rold (2. a), new n (z, (I,a), a),
ift < tep if 2 tep.
a,
We then use of and "!TnW to denote the preference given by old and new reward model respectively, on the response pairs (0z, al) of prompt z;:
of ~ (f (2re, (Ii;,aa) ai) - y (ri, d()): new o(r new (Ii, -r now (zi, a,)).
Using ogld and opew, we calculate the portion of datapoints whose preferences differ between the old and new reward models:
Paif Σ 1 # o.eW).
(21)
the value of Paif large, it means that the preference drift from the old reward model to the new reward model happening stronger in the dataset. When lep is fixed for the dataset, which means that the number of datapoints from each reward model is fixed, datasets with higher Pai will result worse performance of the algorithms. This is because more datapoints evaluated with the old reward model will have conflicting preference with the new reward model, causing harm learning the true preference.
C.2. Non-Stationary Preference Dataset Creation
1) NSGO Datasets. We modify the GlobalOpinionQA datase Durmus et 2024) create time varying dataset. GlobalOpinionQA consists of questions regarding global issues, different responses, and preferences from several countries represented as a probability vector. We copy the questions and responses to create multiple time steps E [100]. We then vary the preferences with time by linearly interpolating between the preferences of two different countries. This simulates gradual preference drifts that can caused by demographic shift or a series of external events. We generate preference drift using three pairs of countries. In each pair the starting country is the US, and the ending country is either Brazil, Japan or Germany. The preferences at the first and last time step correspond to either country in the pair. The last time step is held out test dataset and treated as the current time 7' = 101. We divide the prompt-response pairs so that training and test data do not share any prompts.
helpsseee---eeeeeeeee--------e-- postter-coorccties.
helpsteer pepsteer-cchheeeee coherence
beavertal veetaalss talls-is safe
helpsteer-e
1.00
0.75
Correlation
0.50
0.25
0.00
2) UltraFeedback-B RM Datasets. Using the prompts and response candidates of UltraFeeddacck Cui al. 2023), we obtain preferences from two different reward models, PAIRRM Jiang al. 2023) and ARMORM Wang et al. 2024).
Figure The correlation of diffcrent preference labels generated by rewards from the ARMORM reward model on the Helpful Harmless harmless-base dataset (Bai et al.) 2022a). We observed that concepts such as safety and helpfulness have more correlated preferences, whilst the helpsieer-coherence reward model un un-correlated with the other models we analysod.
The datapoints in the training set are randomly assigned to one of [100] time steps, and assigned preferences of PAIRRM if the time step is earlier than the change point lep E (51, 66, 81). We assign the preferences of ARMORM for the datapoints with time steps 2 top and datapoints in the test set with = 101.
https://uugiigffae http:://wuggngggaa....aaatssss/s modify the binarized version UtraFeedbackk
global Lppinions
httph/gginnacc...ollmmmeeeeeeeee cw/llm-blenderr/ainRRM hhtp::/huggingieee http://hwggnnnaaeec.oo//////////
15

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
To test the effect of varied degrees of preference drift, we also vary the portion of datapoints whose preferences flip reward model changes. We denote this portion as Paiff and use Paiti E {0.7, 0.9, 0.95, 1.0} to create both training and test data. We use 10k datapoints for training and 500 datapoints for testing.
3) UltraFeedback-LA Datasets. Using the same UttraFedbacck dataset as above, we construct another dataset with the information of language models used for generations. The datapoints in the training set are randomly assigned to one of E [100] time steps. Among the datapoints whose time step is earlier than the change point tcp E {21, 51, 81), Pdiff E {0.7, 1.0} of the datapoints have responses that are generated by smaller language models as preferred responses. The other datapoints have responses generated by gpt-4 as preferred. We use 23.3k datapoints for training. We use the gen- erations of starchat, 1lama-2 2-7b-chat, wi maizarddmm-- 7b, pythia- 12b, a a aalpcca770 1lama-2 -----2-chabb-a- wizardlm-D 1 1b, ultralm- 13b for smaller language models in the dataset.
4) Time Varying Helpful Harmless Datasets. Using the harmless-bar subset of the Helpful Harmless datase $ Bai et al. 2022a), we create a time varying preference dataset. To do so, we use two reward models, the helpstter-heelpjuuess and beavertails-is. safe outputs from the ARMORM model (Wang et al. 2024). Figure X shows that these rewards result in different preferences on the harmless-bass dataset. We then assign each datapoint in the dataset a random time value from t E [100]. We construct two methods to assign preferences using the time step information: change point preference shift and gradual variation. Under the change point preference shift, datapoints are assigned preferences according to helpsteer-hellffulness before the change point top and beavertails-is_sae ssae after the change point. Under gradual variation, we use the following reward model
ro((z,)) y) ro(o; y) ((-33)
t < 33
r(z, v.
+ (n, - 1-33
33 < 1 < 66
33
ri (25, y)
66
where ro is the celpsteer-helpfulness reward and Ti is the beavertails-is safe reward. We use this type of schedule for gradual change to simulate preference drifts that happens gradually over a finite time horizon. We use 15k points for training and 2k for testing. We use reward models for helpfulness and safety, as these are both desired properties of an LLM but often result in differing preferences; for example, rewarding helpfulness can often lead to unsafe outputs when an L.M is asked a dubious question, like how to best rob a store.
C.3. The Two Countries (2C) Non-Stationary Global Opinions Dataset
To test NS-DPO, we create a synthetic non- -tationiion dataset in which the temporal trends are known. To do this, we use the GlobalOpinionsQA dataset (Durmus et al. 2024). We preprocess the dataset in three major ways.
Binary Preferences. We convert the dataset to a dataset of binary preferences. For each set of prompt and responses, we create row for each possible combination of prompt and binary response pairs. We calculate the preference probabílity for these response pairs as follows. Assuming the non-binary responses follow Plackett-Luce preference framework, we can find the reward associated with responses (up an additive constant) by taking the log of the preference probability. We can then take the sigmoid of these responses to find a normalised binary preference.
Country Filter. We filter the dataset down to the following countries: Nigeria, Egypt, India, China, Japan, Germany, France, Spain, United States, Canada, Brazil, Argentina, Australia and New Zealand.
Country Level Prompts. We filter the dataset such that each row of the dataset is the prompt, response, preference probability of single country.
After the preprocessing, we copy the dataset and assign a different timestep to each unique instance of (prompt, response, preference). We simulate the drift in preferences by using preference probabilities of two countries, shifting from one to another over time. Out of 100 time steps in the training dataset, the first 33 time steps consisted of preference probabilities from the US Preference labels sampled from the last 33 time steps are from probabilities of the target country. We use Germany, Japan and Brazil as target countries, creating three different datasets. In the intermediate 33 time steps, preference labels are sampled from interpolated probabilities between these two countries. To introduce sufficient shift in preferences, we selected responses in which probabilities for the same response from two countries differed at least by 0.2. We subsample
16

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
prompt-response pairs down 10,000 datapoints, allowing each time step to consist of different prompts and responses. For evaluation, we use prompts and response candidates that are not present in the training data.
C.4. Compute Resources Uses
To run the LLM experiments, we use A10 GPUs with 40GB VRAM. The synthetic experiments are run locally on a laptop without using GPUs.
C.5. Synthetic Experiments
To analyse the performance of NS-DPO in the log-linear policy class, we construct a synthetic environment with a known feature space and preference drift. We use the feature space from (Li et al. 2023), where X' [0, 1]
A= = [na]
and (22, a) is computed as
1 a+1
(xx,,) a)
(a 1) 1) cos(20 T),
sin(zo ..,... (a + 1) cos( -1 T), a a+1
sin(zdd-1
(22)
The dimensions of the feature space and the policy parameter are both 2 dg. We use dN = 4, = =8, |A| = 16 for all synthetic experiments.
Non-stationary Dataset. To construct a dataset D = {zr a, a', t-- we randomly sample ~ X and 01 A. We assign 20 datapoints per time step A4 E [100). We sample 100 datapoints for evaluation at T = 101. To introduce preference drift, we follow an approach similar to (Faury et al. 202T). We sample the preferences over d₁ and d2 from the class of log-linear policies given in Equation (11), parameterised by 02. We denote preferred response as a and the rejected response as a'. When S 33, we set the optimal parameter as 07- (1,0, 1, 0, 0, 1,T. Between 34 S t < 66, the parameter 0: varies as
0:-
[cos(
(-33 33
3) sin(
2 ........................
cos( 1-33 33
2),
x
sin
2-33
(23)
33
33
For the remaining time steps 67 S < 100, we use 0 (0,1,,0,,,,,,,0,10)
Algorithms for Synthetic Experiments. We compare NS-DPO with DPO and SW-DPO in synthetic experiments. SW-DPO uses "stiding" window only consider points close to the current timestep T,, which is commonly used in the non-stationary bandit literature (Garivier& & Moulines 2008). We test the performance of NS-DPO and SW-DPO over several values of n E {0.7, 0.9) and window size w E (33, 50}. The regularisation coefficient is = 1.0 for all algorithms. We normalise the scale of the gradient for each method to address the differences caused by the application of exponential weighting and sliding window. For the reference policies, we use a uniform policy, whose parameter 0re E Rd is a zero vector.
Evaluation Metrics. To analyse the performance the algorithms, we use the reward accuracy of the trained policies. The reward accuracy computed by the portion of test response pairs with correctly estimated preferences, using the implicit rewards defined in Equation (8). For each tested algorithm, we report averaged results of the experiments across 10 different random seeds.
How does NS-DPO perform when specialized to log-linear policy classes? As shown in the left image of Figure 9 when compared to NS-DPO and SW-DPO, DPO shows the worst performance with respect to the test data. Both NS-DPO and SW-DPO, which account for the preference drift present in the data, show significantly better performance. SW-DPO achieves similar performance to NS-DPO in the later stages of training, but NS-DPO achieves this performance in fewer training steps. As NS-DPO only varies the weights of datapoints, rather than removing them entirely, it can still leverage the information of datapoints in the earlier time steps. The right image of Figure 9 shows a comparison of different values of y, ranging from 0.3 to 0.9. The results show that the performance of NS-DPO is stable in terms of the final test accuracy across wide range of values 7 E [0.5, 0.9]. As the value of y is reduced, only the points closest to the current time step contribute significantly to the gradient update of the model. Thus, as y decreases, NS-DPO requires more training steps converge the reward accuracy on the test set.
Further Results of NS-DPO and SW-DPO. We present the experiment results of NS-DPO and SW-DPO on the synthetic dataset with varied values of hyperparameters y and w. As shown in Figure 10 The performance of NS-DPO is robust across varied values of y, maintaining its reward accuracy over 80% when 0.5 < 7 S 0.97. In the case of SW-DPO, the performance more sensitive to the change of the window size w. When w = 10, it shows similar test performance the
17

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Reward Accuracy
0.875
NS DS- (y= 0.7)
SW-DPO (w 50)
Reward Accuracy
00.875
NS DPO (y30.3) (y- 0.5)
NS DPD ( 0.7] NS-DPD (y- 0.9)
NS-DPO (y- 0.9)
0.850
SW-DPO
0.850
0.825
0.825
0.775
0.775
0.750
0.750
0.725
0.725
0.700
0.700
200
400
600
800
200
400
600
800
Training Steps
Training Steps
Figure 9: Synthetic experiment results with d, |A| 16. The shaded area represents the standard deviation of cach algorithm. [Left] NS-DPO and SW-DPO successfully addresses the non-stationarity present in the dataset, while stationary DPO fails do NS-DPO shows faster training than SW-DPO, even compared to the case where set to the optimal value of 33. [Right] An ablation study on how different values of the discount factor ry affect the training of NS-DPO. As the value of becomes larger, the final test accuracy of the policy is achieved in fewer training steps.
value
window parameter wr for SW-DPO
0.900
0,900
Reward Accuracy
0.875
NS-DPO (y
NS-OPO (y= 0.95)
NS-DPO (y 0.99)
Reward Accuracy
0.875
SW-DPO (w = 331
(wr
SW-DPO (w [ww666 66)
SW-DPO
0.850
0.825
0.825
0.800
0.800
0.775
0.775
0.750
0.725
0.700
800
0.700
200
460
Training Steps
Training Steps
Figure 10: [Left] Performance of NS-DPO with values of > 0.9. NS-DPO shows robust performance with respect to the value of t while it starts resembling the performance of stationary DPO the value approaches objective gap of SW-DPO in the same experiments. The performance of SW-DPO improves the value of w gets closer to 33, when the algorithm is only learning from datapoints where the preference distribution stays stationary in the given setting. The setting with 10 also shows final performance similar to the case of w 33, but shows slower training because of the reduced amount of data used for training.
close to 1. > 0.97. [Right] Expected RLHF
later stage of the training, while the process is visibly slowed down due to the reduced amount of datapoints actually being used. On the other hand, the window size gets bigger and starts including datapoints where parameter shift introduces conflicting preferences, SW-DPO also shows degrading performance. These results provide further support the advantages of using DSO over SW-DPO, as it shows faster training and less sensitivity to the hyperparameter.
18

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
D. Offline Learning Analysis
In this section, we provide the remaining details the analysis on offline learning of non-stationary dataset.
Non-Linearity Coefficients. Following the analysis from (Filippi et al. 2010 Faury et al., 2021). we capture the non- linearity of the sigmoid function in the NS-DPO loss. We use the coefficients ka.r, Ca,r, which are the supremum and infimum of o ( a) ('),)) over x E X, a') Aº,0 0 e θ respectively;
ko, (E..(
sup
((T(0(z, a) 6(x, a(),,a)),0 81),
(24)
inf Ca,7 =EX.(a"')EA'..EG
o( ( a) o(2, a)), 0)),
(25)
while we use RO, ka,7/Ca, to denote the ratio between k,, and Ca.T.
Loss and gradient. We recap the loss of NS-DPO with & regularisation term:
T- for logo(ho( a()) (1 0x) log (ho oe a:)) ACor 1101..
AN (0)
(26)
n
We use Equation 26) to draw parallels between the NS-DPO objective in Equation I0p and the logistic regression objective used in the generalised linear bandit setting of (Faury et al. 2021). We assume the preference label o, is sampled from a Dynamic Bradley-Terry model with the true unknown environment parameter 0r Under this assumption, the mean of the preference label is E[o.] (Is, Os, al, [ = o (hoz (Iz, 0z af)). When there only a unilateral preference sampled for a given prompt-response pairs, the sigmoid function forces the implicit rewards of DPO to have infinitely large scale, driving p(a a') to either (Azar et al. 2024). The & regularisation term our analysis mitigates this problem, by controlling the parameter
Differentiating Equation 12 with respect the parameter 0 results in
Co (0) =
Σ i=1
T7 T-
0.0:
m
T
(hoo (I; ar dz)) + >2a,,
(27)
:=
where 6, = 0(I; a:) - Φ(x; af) is also introduced for brevity. We denote the parameter-dependent part of the gradient as g" (0) = Di rq T to- (hg(2,, as o + 7 which we will use to analyse the parameter estimation error.
Parameter Projection. Let 0 denote the parameter minimising the NS-DPO loss defined in Equation 12 Gy
arg minlgrr (0). Due to both learning and tracking aspects of the estimation error, we cannot guarantee that 0r is within the boundary of the parameter presented in Assumption 2 0r θ. This motivates a parameter projection method. which enables finding an admissible parameter 0r θ while minimising its deviation from Or (Faury et al. 2021 Wang et al. 2023). Using 2r in the performance analysis of NS-DPO allows preventing the potential violation of Assumption when Or is used. We perform parameter projection by calculating 0 by
0r = arg min l|g (07) - (0)) ego
(28)
using E defined in Equation 17 and (0) defined in Equation
27
Covariance matrices. In addition to E defined in Equation 17 we also define Σ, to which squared discount weights are applied:
2T 21, ((0(I,,a)) Ma,, (()) 6xx, d.))
(29)
Due to its squared application of the exponential weighting, Σ St.
19

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
D.I. Estimation Error
Theorem 2. (Estimation error of ÔT.) Let Let
0,7 O. Let Ör denote the minimiser of the NS-DPO loss defined in Equation (1) offline dataset. Let 0r denote the parameter obtained by performing the parameter projection procedure on 0r. Then with probability least 6:
0 071 22AW + 2C1
+ log(1/6)
16LRo,7 m
dm
BT
(30)
TCO.T
n2
T(1
learning
tracking
where C > 0 is constant.
Estimation errors in typical stationary settings can be considered as learning emrrrs, which are caused by having finite data sampled stochastically. caused by the drift of the underlying true parameter along the time steps (Faury al., [202]] Wang al. (023). In this section, we show how these emoos can be disentangled and bounded separately. To do this, we apply the approach of (Wang et al. 2023 in contextual bandit setting to our setting of offline preference learning.
time-varying settings, the parameter estimation suffers from tracking error as well, which is
D.1.1. BOUND DECOMPOSITION
We begin with the deviation between the optimal parameter 07 and 0r, the projected parameter of the NS-DPO estimator 0y:
1 n n
g" (0 9" (87)
Σ
Ty----t [o(ha, (z, a, o (hoz (I,, a; ab))] o, Ac 07).
(31)
Applying the mean value theorem to the difference of sigmoid functions in Equation 31 we get
1 2227-1- 11- [ ((T(di, (1 - 2)07 + ( d. (0r 07)
g ( - 9- (87)
n
2 07).
We can now define a matrix GT to define the relation between 9" (0T) - g" (07) and Ør - 07:
[
Gy = n
T-1-1
((T - (1-v00 + v à.0! + ACo,r
(32)
0=
a(b.07,br)
g ( - 9" (07) (07)= 72. Gr (0r 07))
-²
(33)
We make a brief aside to show GT ž Co.T ( AT) 2 Faury et al. 2020 Filippi et al. 2010 as this is an important
property of GT and one we will use later in the main proof. To prove this, we first show that oi. 07. 0r) > Ca aii, 0,, 02) is the mean value of ō along the path between some points (Ø) 01) and (0, 02). (2).
This is greater than the infimum of ô at point along that path, which is in turn greater than the infimum of in the space of parameters E θ The last infimum is the definition of Ca,r Equation 25
Then
ali,0,,02)
a(T(VOT0, -(1 (1- (1-v) -)) inf 0T
aeeeeeeeeelle (ileseo,eee@((oo)) (To Co.r 0.
(34)
a(i, (1,00) > 0 comes from the fact that the logistic sigmoid function strictly increasing and has gradient greater than zero at every point. Because of this incquality, each element of GT denoted by [Gr]uVl, & E [d], is strictly larger than each element of Ca,r [E]n.. We use this to prove that GT 2 Ca,7 (S + XI) for any 01 0z. We first remind the reader of the definition of Σ:
1
Σ
T (0(1s, 0:) - H(x;, ()) a:) d(x; d.)
20

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
We then prove the inequality, using the fact that and y do not depend upon the indices I, k of the vector o to move the across indices within the over the datapoints
8 GTU=
Σ
aaai,, 0(1,01,0))):0 + ACo
(L.))E(d
1=1
12
11 (1,101000)))
Σ
urik
+ AGaT
Σ Vi
11 1=1
(1A))
[E(d]
1 72
Ca.T
upuk
ACo
(35)
71
11
(1K))
LE(d
Co.e
Σ
T
2.0T + xl
Urth
Co, (E + Al)e.
(36)
(L.))
We now continue applying Equation
33 to bound the estimation error term:
||0 8 1S+
G (01) - g" (0+))
(37)
72
We use Equation (36 apply GT
(E+AA))
6,,7
マ G (g (ÔT) 5" (07))
110 110 (0r) 5" (0
(38)
72c,,
We add and subtract 9" (0r) inside Equation (38 and apply triangle inequality to derive
1 C2CTT
11g" (Ōr)  (0)))((+>>)-
|105 ((=)( 9- (Ôr) + + - (
-cat
(lo0 g" (0r) + 110 (Ôr) g (0+)))(++))) .).
< 7Coor
(39)
We use the definition of dr from Equation get
28 to derive 11g (0r) -g0 11g g (0 and
(10 - و
72C,,
I P2co.
(0)) g" (87))
(40)
We remind the definition of θr, which minimises the gradient of the loss defined in Equation 27 making V V (0) 0:
L NS
(0)
1
( Or Pre))) 0z} 3. + Aco.- 20 0.
(41)
n
We rearrange the terms in Equation
41 to derive g² (Ôr) one side of the equation:
T - (r dr (ra)) + Aco. = Σ
rg T- 1 11
(42)
1=1
(6r)
21

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
We apply the result of Equation 42
obtain
g (0T) - 9 (07)
T- 1- [or othoz (Zi, ai d'))] - ACo,
(43)
1=1
Using the fact that the preference label o; is obtained from the optimal parameter at time step ti we define E; = Oj aT ff Oeet)) ) and use o = 4, + a(T(O,, ff, Oeet)) to get
T 1 1o; ohoo (ot, a, d'))): Ac,
7
T-1 T-1-tt [es + ( 0% 0ret)) o ohho; (2i, as d; T90G
4 (T 07, - Orot)) o(bo; (2is, az d,)]]a
i=1
tracking
11
C Aoo -0
(44)
71
leaming
We terms in Equation
444
with Equation
40 to define learning error and tracking error:
clearn
:2C,T
n
ry
>co,TT20 ((
(45)
i=1
2
12
gtrack
(T(66 07. -ree)) o aho; (24, a, df)
(46)
72
Bounding each of Equation terms are provided in Appendix D.1.2 and Appendix D.I.3 respectively.
(45) and Equation 446 results in Theorem 2
The detailed bounds for the tracking and learning
1.1.2. CONFIDENCE SETS: LEARNING
We begin with the definition of the learning error:
Jearn
T
2c, 2001:
(47)
Ca,
Ti t=1
We bound the norm of Equation
47
with respect to + AI, using the fact that Σ > E and Σ + AI = XI:
1
T
cd: 2c,
ry
72
($+2))1
T 1-1--111 -1 - 220 1
T
1=1
< |Cco 2007
1 +
1 Σ
rq
T c. (EEN--
i=1
<22 W <
1
T
-
(48)
-+
22

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
We the e 's property of being sub-Gaussian random variable, sampled i.i.d. during the creation of the dataset. We apply Theorem 2.1 of (Hsu al. 2012) Equation (48) resulting in bound holding with probability at least å:
log(1/5) Br(o).
c
(S+N)-1 S S
(49)
where C denotes a constant introduced for bounding purpose. We provide the details of applying (Hsu et al. 2012 s
theorem in Appendix D.4
We now go back the original definition of learning error term glearn and bound it. We use the result in Equation (48) and
Equation 449
derive
2
glearn
T 1 E - 2
Ca,T
t=1
2
*2VAco + TCI
d log((18)
T2
2C1
d + log(1/5)
2/WW
+
(50)
TCaTT
nh
which finishes the bounding of the learning error.
D.1.3. ESTIMATION ERROR: TRACKING
We begin with the definition of the tracking error:
strack
2
n
[0 -(ee)) oho; (x,, as d;))los 1S+N0-
Ty
T2COT
n
i=1
2
79
( 07. -ret)) ( 07 0 (E+2N)1
(51)
Ca,
We remind that using Equation 334). aii, 0i,00 07) is
ali, 07, = f v=D
((r( (1 w)67 + -00,)))
(52)
Applying the man value theorem Equation
we obtain
2
T
lo (r (0i, 02 Oret)) o(( 07 0na))lde (+2N)
|
CC,T
i=1
2
22n T- a(i, 02 07) 07) 0
(53)
C2,,
We apply telescopic sum, which separates 0z - 07 into differences of optimal parameters between each datapoint:
n 1
22n T to aii, 07 07 0. (07
07)
n
1 1
Σ 2 T- aili, 07 07 DZ
(Oor, (0r, 0r
(54)
where we use 11 to denote T.
23

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
Then we Ei=k Ej= aij = Ej=k E 0t, to rearrange the terms inside the summation:
n
" a(i, 01 05)
Σ (0 (g ( any- p=
11
i=1
(1,, 0ū, 02) (0i, - 0i oi 1
(55)
We use a(i, 0%, 07) S ka,r using the definition of a; in Equation 34 to get
2 T a (i,, 07 07 07) (0 0% oz 0 Ieetnn
أ. (0i, - 0f ) 1
S
2
KinT
(56)
n2
pml
We then apply triangle inequality and Cauchy-Schward inequality to get
2000 (0;,
- (irr)/lss 1
S 2 ka,t
3. 11 12111,,
(57)
We use ||ô||| S 2L and arrange terms to obtain
2.
110 112100
0f 1 1SAA1
p=1
S 2T 2
100 - 02p 112. |100
|1
(58)
p=1
Here we bound the term 01. We first apply Jensen's 's inequality to derive
11 <
115
1
1
-4
-- 11 11 133
(59)
1=1
We then use the property of trace operation and Σ
D
nT T1--
t, Φ8
from Equation 17) get
1
T
T-1-1
1
T T-1- tr (OT(S A1) E.)
(S+A))
n
n i=1
i=1
= tr
( AI) -
1 Σ
7 T- 1- 2:07
i=1
< tr (la) d.
(60)
24

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
We apply Assumption here. Because each time step can have at maximum datapoints, we can upper bound E E/=17
with
my(7- - 1) n(1 -7)
171
(61)
77
1=
[pl] m
where
We combine Equation (60)
and Equation obtain
2 2
ko,5
Ary-1 - Obpps ll2
p=1
1=1
dm -1)
2L kar
n11 -7) 100 - 0ipt. 1l2.
(62)
pml
We apply Assumption aggain to upper bound the summation
Ep Up m Et= Uz getting
dm (t++1) 11) (
2L 2
110%, - -00 -0 ll2
n(1 7)
p=1
T-1
d (c+ (xt+1) -1) 110 01+11l2.
S 2T lio
m
(63)
n11 -7)
We apply v = = + v to introduce another summation:
T
dm
2T ka m
T-1
100 02+1112
n(1 71)
2LT kaa m
dinalyy (1+1) - 1)
n11 -7) 100 0.+1
(64)
mm1
Because 7 < we can bound
T T-1
(m+((-(1+1) 1)
T-1 dmn - 1)
2
(65)
n(1 7)
n(1 7)
A=
1=1
1=1 k=t+1
and apply geometric sum to obtain
T-I Σ
dmy(y-(1+)) (1+1) - 1)
T-1 T
72 2
dm (t+1) -
y
(66)
n01 71
n(1 77
k=t+1
t=1 1 - 7/2
14. 2
We use again to derive
1, and get
T
7
7
T
dm-(y-(041) -
T-1
7/2 - 2
dm (t+1)
<
n(1 77)
n11 -7)
1 -
1+
1 7
2
T-1
42
1
4
T
dm (1+1) 1)
(67)
1-7
n( - 77)
25

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
We then use (12 -g -1
V7 (1+1) - 5)1577771 1 = 1 to derive
T 1
22 Y
dm (1+1) I 41 n01 -7)
dm
(68)
1-9
-=1
t=1 (1 7)
We use the result from Equation get
68
to Equation
64
and use the definition variation budget By from Assumption to
2T Ko, m
T T
T-1 2
dm 1)
100 0.+11l2
T
n(1 -7)
8LT 2 hio m
dm
T-1
|10 0 l1
72
(1 7)
8LT m
dm
BT.
(69)
T(1 -
We now combine Equation (69) with Equation (46) derive the full bound the tracking error:
strack 16L
dm
Br-
(70)
T(1
We now use Equation
70 with Equation 50 to obtain the full estimation error:
10T 6 ll S elearn strack
2C1
d log (1/8)
16LR,,
dm
S 2VA
BT,
(71)
3
TCa.
T(1 7)
which concludes the analysis for Theorem
D.2. Regret Bound
Let ÖT denote the parameter in θ which minimises the NS-DPO loss (Equation 12 offline dataset. The following bound holds with probability least 1 - -62 and when
Theorem (Regret bound of θT) Let 8
(0,
2 C / dllog( (dd/8)//
[1 -7)
2VAW
201 TCg.r
+ log(1/8)
16L m
dm
BT
2m(1
-1-
T(1 7
where C > 0 denotes constant. When 7 = BB))
3/4
Reff satisfies:
Ô (d B2 n 1/44
1.2.1. POPULATION COVARIANCE OF FEATURE DIFFERENCES
Let diff define the population covariance matrix of the feature differences:
Ex diff E[
(72)
where J δ(x, a) :(0, a') denotes the feature difference vector, and the expectation is computed with respect to X,t T, a' (1x)). We also define the discounted population covariance matrix Σ? diff"
Σ7
T11-1 (6)
(73)
dift.dif
26

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
where the expectation is computed with respect the distributions E diff We then define w (T,, r)):
U "TEn Meat.difft
(T, ( DERd ut Σ> Midt
sup
(74)
Without any assumptions on the time distribution, w (T, S (T-()-)), which happens when all the datapoints come from the oldest time step. We use Assumption 5 to obtain a tighter upper bound of upp, Using ( - 1) S n S m 1), we can get
T-1
m
Σ
T-1-
(75)
We note that the prompt distribution X' and the reference policy Thef independent from the time step distribution T. Using Equation
75 we obtain
T-2
mi 1 - ---11111 T dine))))))) m(T - 1)(1 -)
UT
dier 2
m
Σ 7p
(2
Tint ditt
(76)
mT
1)
km0
which implies uPP (T,7)
mIT 11(1))
m
D.2.2. DECOMPOSING REGRET BOUND
In order decompose and bound the detailed elements of the regret bound, first show the relation between the regret and the estimation error of the model parameters.
Let E [0, 1]. Let Ôr denote the parameter obtained by performing the parameter projection in Appendix D after training with the NS-DPO loss defined Equation (12) on an offline dataset. When X N (o/llog(4/8))) with probability least 8:
Theorem 5.
TKM - 7) 2m T-1) 100 0r
R
(77)
Let T. denote the policy we obtained by training with NS-DPO and performing parameter projection. We use Mgg denote the population covariance matrix, whose expectation taken with respect to TØy We assess the performance of T using the difference in expected non-stationary RLHF objective (r(m) defined in Equation (7), which
(m) Ex aaw (((,,,T)) (DLa
Ref (r(Tt) Ir((TTT)
Exex Ba (1) (r(2
T"] -
B Be' ( (19 7 7Duu (1a
(78)
We plug Equation 8 in Equation (8) to obtain
ア (a(ar) DKLL (-ea)
R
Exox E. Eawny
(Mar) (T log
Trer (a(z))
क (ala))
E Ea'n ( 2)) [) log
+ DKL, (1ax)))
(79)
Wer (a(r))))
where terms with normalisation constant 77 (x) are cancelled out. By using the definition of KI. divergence in Equation 79
27

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
again, we obtain
T Flog m
log or (a)zr)
129 Exnx
Bo' (12)
Eegnaar (110
Tret (0/2))
Traef (a (3)
T8 (a|xx)
Exxx
TB / Ka
(2)) m)
log
क (7)(((22))
Exex
7DKL
(+12))|1 (11))
(80)
Here, we borrow the analysis in Appendix A.5. of (Chowdhury et al. 2024). We use the property of the Bergman divergence Be, with its potential function L2 (0) = log Ex'ed (0, 6(x d))):
DkL. TT. (-1=)] = 207 - - 0r)
(81)
for parameter 0 E (1-+ E [0, 1]} using Taylor's approximation. With log-linear policies, [x>x(V=LL((0 Egg. We use this derive the upper bound of Equation (80)
17
Exnx [ [T (-1x)
166 -0r13...
(07 (0) T2) (07 θT) -1107- Ör 0T1 (6) - - (T) 0r)
(82)
We now use the following lemma from (Chowdhury et al. explain the difference between E and -
2024), which relies on the matrix concentration inequality to
Lemma 6. (Lemma A.I. of (Chowdhury et al. have
2024g)
With probability at least 1 8. for some universal constant C, we
Σ Fur dif ll2 S CV dlgooo((1i (4d/666))
(83)
Lemma implies that with probability at least - 8 and
X 2 C. /l/gg(d///)))
+ AI - dlog( (0ogg(4//8)) 4d/8)/n
D2 diff
(84)
We use Equation (84) to derive
(07 1T, (07 00)
T|00 0r
(07 - TT) (8+ + A1)(07 θT)
(07 TZ (07 - -7)
< - - 01 + (07 T22 Asst, difl (07 - 0T)
(85)
We then apply the result from Equation 74 which implics (1ll|ll for )-1 (T, m aies) aitt) -1:
(07 0r)) En. (07 0x))
-1007 0r
(07 T Σ Feet ditf 107 θr)
(07 TE (07 -0,)
< Tw "PP((T (T, 1) 0
(86)
(07 θT) Ex ditl (07 0T)
28

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
72 a, a' are independently sampled. We combine this fact with the population covariance matrix maer deriving dife 2
From the definition of E Smae dift in Equation
We use this to get
(07 11 07 rr)
Tw TP (T, (7 0 (07 - TT diff (6) Br)
(TuPP(T.1)
(07 T 107 02)
107 Br Brll+
(87)
2
(07 (T)T (07 OT)
We use max Kn with the definition of K2 in Equation (I..(I'>) 7) (T-1)(1-) T(1-)).
(16.. along with the result obtained in Equation 776 to
Tw (T.
(0z- 0T) (0) 0T)
107 01
2
(07 0r TTire neat (07- 0r)
THw "PP (T, 7)
110 0r 11 S
TKmil 7)
2m - T1) 100 0r 0T13
(88)
D.2.3. COMPLEXITY ANALYSIS
In order to investigate the complexity of the regret bound, we set the value of using T, Br. We first set 7 as
3/4
BT T
7 =
(89)
We apply Equation (89 in the estimation error 110 0rl1s442 with assumption of X while ignoring the logarithmic factor:
from Lemma
2v VWW
(= d n
201
+ log 1/8)
(= d2 n
TCa,7
n
16LR,,
dr BT
(= d By T
(90)
T(1 7)
n
Here, we note that from Assumption 5 = (T)) This allows us to consider the complexity with respect to the dataset
size n and T' together. We can conclude from Equation (90) that the complexity bound of the entire estimation error is
(d 2). By setting the value T to sufficiently large one, making 1 - -- 2 $1 then the complexity of uP (T, 7)
T(1 7)
(= BJ T
(91)
Finally present the total complexity bound of the algorithm, by applying the complexity of (P in Equation 991 to the squared estimation error |10 - 0 07 x
3 Reff Odd B T 4)
= Odd B7J n 1).
(92)
29

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
D.3. Theoretical Analysis of NS-DPO under stationary preferences
Corollary 4. (Regret bound under stationary preferences) Let BT - 0, (0, 21 O. Let OT O denote the
minimiser of the NS-DPO loss (Equation (12)). Then, for X 2 Cv dloo(Ad/0)/n, some constant C₁ > 0, 1- (47 and & E (0, 3/,
2), we have with probability ar least 1 25:
2 d+ (1/0)
4Tkm
Ci TCo.7
lim Re Br
VAW +
m
n
Pre-fuctor
and recover the complexity of RCj = O(n 2) under stationary preferences.
We show that under certain conditions, NS-DPO's regret bound recovers O(n 2)). We first analyse the estimation error in the limit By + 0. Consider the estimation error bound in Theorem
2C1 TCO,T
d log log(1/0) (1/8)
16LR
dm
||0 - 07.1S4M S SW AW
BT,
(93)
T(1 7 n
learning
tracking
in which the tracking term depends upon y and Br. the regret bound, we write y in terms of Br the form of
BT
1=1-
(94)
where R. We obtain
Br (T)
by rearranging terms. Substituting Br back into the estimation error bound, 3
3
we find that the tracking term reduces to 16L m 2" tends to 0 as Br + O. Thus we conclude that
Br
By inspection, for 0 < a the tracking term
d (1/8) 201 TCa.7
16L m
dm
201 TCo,
d + log(1/6)
lim Br-+0
2VAW
BT 2VAW
(95)
3
n
T(1 7)
n
n
We now consider the regret bound in Theorem
2 Br
12
TR 7) 2m
2VAW +
2C1
+ log log(1/8)
16LR m
dm
(96)
T-1)
TCa.
n
T(1 7)
n
Pre Pre-factor
Tracking
Here we note that the tracking term and the pre-factor term are dependent upon Using the product rule of limits, we analyse the limit of the pre-factor and tracking terms independently and then multiply them together. Using L L'Hopital's rule, the fre-factor term Equation (96) in the limit By + becomes
lim TR - (BT)) lim
TRMI( BB )a
21 7(Br)) BT 2m(1 (1 BT JT)
(97)
We remove terms that do not depend upon By for simplicity and then apply L' L'Hopital's rule:
)a
1
lim
lim
(98)
Br-20 (1 (1
T-
Br
1)
J
(99)
T-1
30

---

Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift
analysis from the estimation bound and set 0 < 3. such that the limit of the tracking term is 0 as expected in stationary scenarios. We can now write the regret bound as
thus finding the limit of the pre-factor term. As T
1, 2m
Z
lim Br 0
RT
4rkm
VAW+
Ci
+ log (1/0)
(100)
TCo,
n
and recover the result (0(n1// in Corollary 4
D.4. Details of applying Bernstein's 's inequality
We restate the norm to investigate:
(101)
We then define two vectors and Z, followed by a matrix M:
V [e,,.., Cn].
(102)
Z hT 1--1-6 .1,..., x11--im
(103)
1
M = Z(Σ + 11) n²
(104)
We then express Equation [001 using V, V:
:e::: -VPVTMV VVVVMV
(105)
11
We here recall the definition of q, which is A.sub-Gaussan random variable:
E o, (T 0% (ror))
Borspa (a/-0 = 0,
(106)
Varo (a.00/1e.) Eo, pr, (a. af (+) (Borp,, (o:o)))))) s
(107)
As stated in (Hsu al. 2012), the Bernstein's 's inequality for sub-Gaussian random variables in quadratic form implies
PVMM -² (I(M) + (1 M|| log(1/6)
S² C d + log(1/0)
(108)
n
for some C1 > 0, while ||M]] Amax (M). Here used the definition of Σ in Equation (17 to show Σ = S, and
derive for À > 0
1 T
M < - n2 (r(M) < d/n,
2t
(109)
(110)
M) S d/n3.
(111)
|M < 1/n.
(112)
31


Reviews:
[{"review_id": "FVnlweIMaJ", "reviewer": "Reviewer_cPK3", "summary": "This paper addresses the issue of temporal preference drift in training large language models (LLMs) using human feedback. The authors propose Non-Stationary Direct Preference Optimization (NS-DPO), which incorporates a Dynamic Bradley-Terry model to account for time-dependent reward functions. This method introduces an exponential weighting parameter to focus learning on more recent data points, improving model robustness under preference changes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their response and recognising that our conclusions are well supported by both theoretical evidence and extensive experiments that show NS-DPO’s effectiveness in handling non-stationary preferences. We think this is a super important problem, unaddressed within the literature, and that it will become more prevalent as companies build proprietary datasets over longer time frames. \n\n## Response to Limitations\n\n__(1)__ To the best of our knowledge, we are the first to address the problem of non-stationary preferences within the LLM fine-tuning literature. As such we were __unable to find any publicly available datasets__ that exhibit this property, as most are well curated for specific goals e.g. Helpful Harmless, and not preference drift.\n\n__(2)__ To account for this we created datasets by introducing preference drift into pre-existing, realistic preference datasets. We specifically tested challenging preference drifts in our experiments i.e. the preference change is strong and occurs suddenly. In Figures 3a and 4 we explicitly test NS-DPO on such preference drifts, a sudden shift from one reward model to another over varying times and with increasing strengths, and show that NS-DPO is robust to such changes even in the most challenging settings.\n\n__(3)__ By drawing attention to this problem and providing a simple and effective solution, we hope to attract the attention of the wider research community and collaborate to create more realistic datasets, however that is beyond the scope of this initial work which clearly demonstrates the problems that non-stationary preference drift can have on algorithms designed from a stationary perspective.\n\nOnce again we thank the reviewer for their time and ask that they consider raising their score further in light of these arguments. This problem is exciting and underexplored though it will be of great interest in the future, and we believe that our work addressed this in a simple and effective way.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence.\nThe authors provide theoretical analysis with proofs for the convergence and regret bounds of NS-DPO, showing its effectiveness in handling non-stationary preferences. They also conduct extensive experiments on various datasets with controlled preference drift, demonstrating that NS-DPO outperforms existing methods like DPO and IPO in different scenarios, especially when significant preference changes occur.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are well-suited for addressing non-stationary preference optimization in LLM training.", "theoretical_claims": "While I did not verify the detailed proofs, the derivations and arguments presented are clear and logically consistent.", "experimental_designs_or_analyses": "The experimental designs demonstrate both merits and drawbacks.\nOn the positive side:\n1. The synthetic experiments offer controlled settings to assess NS-DPO under known preference shifts\n2. Testing across multiple datasets showcases the method's broad applicability\n3. Ablation studies effectively clarify the impact of key parameters\n\nHowever, limitations exist:\n1. The paper primarily uses synthetic data for main experiments\n2. Real-world datasets have artificially induced preference drift rather than naturally occurring shifts", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "see `Experimental Designs Or Analyses`", "other_comments_or_suggestions": "see `Experimental Designs Or Analyses`", "questions_for_authors": "see `Experimental Designs Or Analyses`", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "squmyLQJzl", "reviewer": "Reviewer_izAu", "summary": "This paper discusses a new approach called NS-DPO to address the issue of temporal preference drift in LLMs. Current LLM preference optimization algorithms do not account for changes in preferences over time, leading to misalignment. NS-DPO models time-dependent reward functions using a Dynamic Bradley-Terry model and introduces a computationally efficient solution by adding a single discount parameter in the loss function. This parameter enables exponential weighting, focusing learning on more recent and relevant data. The paper provides theoretical analysis of NS-DPO's convergence, offering upper bounds on estimation error and regret caused by non-stationary preferences. Experimental results demonstrate that NS-DPO fine-tuned LLMs remain robust under drifting preferences, outperforming baseline algorithms that ignore temporal changes, while maintaining performance in stationary scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and recognising the intriguing nature of this problem. We believe this will become a far more prevalent problem as LLM providers gather bespoke datasets over longer time frames and are keen to raise this within the LLM research community. As such, we are glad the reviewer agrees with us that this is aligned with the evolving direction of LLM research. \n\n__Response to the Concerns: *“Adopting decay factor $\\gamma$ may fail to account for sudden changes in user preferences.”*__\n\nWe address this at two points in our manuscript: \n\n__(a)__ Firstly we direct the reviewer to Assumption 1 [Section 4]. __This assumption places no constraint on how the preference changes__, as such our theoretical analysis addresses the case the reviewer raises where preferences change suddenly. As seen in Theorem 3, the value of $\\gamma$ is set by $B_T$ which can be thought of as the ‘size’ of the preference change. A large sudden change will result in a smaller value of $\\gamma$ focusing the algorithm’s learning on points closer to the present. We also note that this setting and exponential weighting technique are thoroughly analyzed in the multi-armed bandits literature. We point the reviewer to line 99-105 of page 2, left column where we discuss the references.\n\n__(b)__ Secondly, __we explicitly test the setting where preferences change suddenly__ in our experiment section. We direct the reviewer to Figure 3 a) and Figure 4, in which we test NS-DPO when preferences change suddenly at a specific changepoint for a variety of different strengths of preference change (controlled by $\\rho_\\textrm{diff}$). We note that whilst DPO and IPO fail as the reviewer predicted, NS-DPO maintains its strong performance, even when we fixed the value of $\\gamma$ for all sudden preference drifts!\n\n\n__Response to the Question: *“Can NS-DPO be aware of the preference change of users?”*__\n\nYes, while the information is limited. The setting we consider in our work only allows the knowledge of the __total amount of parameter drift, which is $B_T$__. NS-DPO requires the knowledge of $B_T$ to set its exponential weight $\\gamma$ for the best possible performance. In practice, we additionally provide experiment results where we test NS-DPO under various strengths of preference drift with a __fixed value of $\\gamma$__, which still showed significant performance improvement over stationary baselines.\n\nWe hope our responses have addressed all the questions the reviewer had about our work and ask that they raise their score. We are eager to help clarify any further questions the reviewer might have as we believe in the importance of the problem this paper seeks to address.", "claims_and_evidence": "The problem setting and direction of this research are intriguing, as current studies in LLMs are increasingly focusing on more complex and challenging scenarios. The overall claims made in the paper are valid, and the identified challenges, as well as the discussion of related works, are well-aligned with the evolving landscape of LLM research.", "methods_and_evaluation_criteria": "However, the proposed method does not fully make sense to me for this challenging problem setting. The main difference between NS-DPO and DPO is the introduction of the decaying factor $\\gamma$, which is sensitive to time $t$. While this design allows the LLM to focus more on recently collected data, it may fail to account for sudden changes in user preferences. This is because the decaying factor cannot detect or adapt to abrupt shifts in user preferences. If the LLM places more weight on learning recent data, historical data may become irrelevant; conversely, sudden changes in preferences could negatively impact the model's alignment. As a result, it is challenging to determine an appropriate value for $\\gamma$ that effectively resolves this dilemma.", "theoretical_claims": "I appreciate this analysis of this paper, especially for leveraging the regret analysis in multi-armed bandits. Provide new insights for this RLHF community.", "experimental_designs_or_analyses": "The experimental design make sense and may create some dataset with different preference change strength.", "relation_to_broader_scientific_literature": "LLM, RLHF, and DPO.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": ".", "questions_for_authors": "Can NS-DPO be aware of the preference change of users?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "canRXZIFhw", "reviewer": "Reviewer_Z8Hb", "summary": "The paper introduces Non-stationary Direct Preference Optimization (NS-DPO) for preference learning on non-stationary offline datasets. With an assumption on the upper bound of preference drift and log-linear policies, the paper also shows that the method achieves a sample complexity of $O(n^{-1/4})$, or  $O(n^{-1/2})$ when the upper bound of drift goes to zero. In experiments on synthetically-created temporally-varying preference datasets, the paper shows that NS-DPO achieves a better reward accuracy than vanilla DPO. In experiments on stationary datasets, experiments show that the two achieve comparable reward accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and recognising the precision of our theoretical analysis. We address the questions and suggestions of the reviewer as follows.\n\n## Response to the questions\n\n__*(1) “For the gradual drift experiments, were the datasets constructed such that the model sees increasing new data during training? ”*__\n\nPlease note that our setting considers __offline learning__ scenarios, so __all the datapoints with time step information are provided before the training starts__. Because the reward for determining preference is changing gradually, the preferences between responses are more likely to change at datapoints with later timesteps. It is correct that the evaluation is done with the preferences at a time step that is later than all the training datapoints.\n\n__*(2) “Does the evaluation occur on a held-out timestep that's later than all the training data?”*__\n\nYes, the evaluation is done with the preferences at a time step that is later than all the training datapoints.\n\n__*(3) “The conclusion mentions ICL but I do not see any mention elsewhere”*__\n\nIn page 7, left column, line 369-372, we explained that tDPO refers to the in-context learning baseline. tDPO is a variant of DPO where the time step information is explicitly mentioned in the prompt. As demonstrated in the experiment result (Figure 2, Figure 9), tDPO did not significantly improve from stationary DPO.\n\n## Response to the suggestions\n\n__*(1) “Could the authors provide more discussion on the implications of the theoretical results?”*__\n\nWe believe the core implications of the theoretical results are sufficiently discussed in the paper. To recap and summarize:\n\n__(1-a)__ Preference drift negatively affects the performance of policies. (in the form of tracking error, Theorem 2 in page 5)\n\n__(1-b)__ When we only know the total amount of preference drift ($B_T$ in the paper) and the time steps in the paper, we can use exponential weighting to address the preference drift. The optimal value of the discounting parameter $\\gamma$ can be computed when we know $B_T$. (line 293-294, page 6)\n\n__(1-c)__ As $B_T$ approaches zero, NS-DPO with optimally defined $\\gamma$ becomes identical to stationary DPO. (Corollary 4 in page 6)\n\n__*(2) “The experimental analysis could be strengthened with the use of other evaluation metrics (e.g., win rate)”*__\n\n__Win rate evaluation is in the appendix, where we find it correlates well with reward accuracy__. We point the reviewer to the UltraFeedback-LM dataset explained in Appendix C.2 and Appendix C.6 where we presented the results of win rate experiments. We used `llama-3.2-1b-it` for fully fine-tuning the policy within our compute limit. NS-DPO outperforms stationary DPO in terms of Length-Controlled Win rate (LCWR), similar to the reward accuracy comparisons.\n\n__*(3) \"inclusion of the SW-DPO baseline in the main experiments”*__\n\nWe point the reviewer to the additional experiment we conducted with SW-DPO [__(link 1)__](https://anonymous.4open.science/r/nsdpo_icml2025-7DCE/imgs/swdpo_ultrafeedback_lm_optimalw.jpg), where we added SW-DPO results to Table 1 in page 18 of the paper. We trained policies with SW-DPO using the __optimal window sizes__ to only allow the datapoints after the preference change. __Despite using information not allowed in the setting of our work__, SW-DPO significantly underperforms NS-DPO in Length-Controlled Win Rate (LCWR) of AlpacaEval. We attribute this to SW-DPO __ignoring all the datapoints outside the window__. We also present the results for UltraFeedback-RM datasets extended from Figure 9, where we show that SW-DPO shows worse reward accuracy when window size is not optimal [__(link 2)__](https://anonymous.4open.science/r/nsdpo_icml2025-7DCE/imgs/swdpo_exp_w51_l27b_nbc_ufb_rho1.0.pdf) and similar accuracy when we assign optimal window sizes [__(link 3)__](https://anonymous.4open.science/r/nsdpo_icml2025-7DCE/imgs/swdpo_exp_optimalw_l27b_nbc_ufb_rho1.0.pdf).\n\nWe hope our responses have addressed all the questions the reviewer had about our work. We respectfully ask the reviewer to reconsider their evaluation of our paper in light of our answers and additional experimental results. We are eager to help clarify any further questions the reviewer might have as we believe in the importance of the problem this paper seeks to address.", "claims_and_evidence": "- The theoretical claims are well-supported, with careful enumeration of assumptions.\n- The claim that NS-DPO achieves comparable performance empirically is based on reward accuracy, which doesn't necessarily reflect how the policy model is used in practice. For instance, [Rafailov et al. 2024](https://arxiv.org/pdf/2406.02900) show that reward accuracy correlates poorly with win rate (Fig 4).", "methods_and_evaluation_criteria": "- See above for note on the use of reward accuracy alone to assess performance in the experiments.\n- Even after reading the supplement, I was still a bit confused about the dataset construction and setup. Just double checking, for the gradual drift experiments, the datasets were constructed such that the model sees increasing new data during training, and evaluation occurs on a held-out timestep that's later than all the training data?", "theoretical_claims": "I skimmed but did not carefully check the proofs in the appendix.", "experimental_designs_or_analyses": "- See above for extending the evaluation beyond reward accuracy.\n- The sliding window DPO (SW-DPO) method seems like an important one to mention in the main paper. I appreciate the synthetic experiment comparing with it, but a further comparison seems important to justify the use of all the data and NS-DPO over a subset of the data and vanilla DPO. For instance, a characterization of when NS-DPO is preferred would improve this paper substantially.", "relation_to_broader_scientific_literature": "I appreciate the comparison to the literature on parameter drift in bandits, as well as other theoretical literature on DPO.", "essential_references_not_discussed": "To my knowledge, the related work does a good job comparing to recent work studying and extending DPO.", "other_strengths_and_weaknesses": "Strengths:\n- The theoretical analysis is precise and is well-situated within the existing literature (e.g., comparable setup to Chowdhury et al. 2024).\nWeaknesses:\n- More discussion about the practical implications of the theorems would be helpful.\n- The experimental analysis could be strengthened with the use of other evaluation metrics (e.g., win rate) and inclusion of the SW-DPO baseline in the main experiments.", "other_comments_or_suggestions": "The conclusion mentions ICL but I do not see any mention elsewhere. Is this perhaps a typo?", "questions_for_authors": "1. Could the authors provide more discussion on the implications of the theoretical results?\n2. Could the authors additionally evaluate using win rate?\n3. Could the authors include SW-DPO for further comparison?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that this paper examines a relevant problem in the area of preference optimization for LLMs (in particular, the problem that human preferences can change over time, and standard RLHF methods may be slow to react to shifts in these preferences) and were generally convinced by both the theoretical arguments and the empirical evidence in favor of the proposed solution (DP-PPO). All reviewers were in favor of acceptance.
Decision: Accept (poster)