RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Authors: Yuhao Du
Zhuo Li
Pengyu Cheng
Zhihong Chen
Yuejiao XIE
Xiang Wan
Anningzhe Gao
Venue: Accepted by TMLR ()

Abstract: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning Large Language Models (LLMs) with human values. However, RLHF has been continuously challenged by its high complexity in implementation and computation consumption, specifically for online sampling-based methods like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). Even with recent simplifications, such as Direct Preference Optimization (DPO) that designs an offline implicit reward learning objective relying on pre-collected preference datasets, the problems of over-fitting and training instability remain hindering the alignment process from the expected optimal performance. To address the existing challenges, we propose a novel simplification of RLHF from the perspective of variational inference, called **V**ariational **A**lignment with **R**e-weighting (**VAR**). Specifically, by directly minimizing the distribution gap between the learning LLM policy and the optimal solution of RLHF, we transform the alignment objective into an offline reward-driven re-weighted supervised fine-tuning (SFT) form, which only requires minor adjustment on the SFT loss to obtain noticeable improvement on training stability and effectiveness. In comprehensive evaluation benchmarks, our objective empowers LLMs to outperform offline alignments, demonstrating superior performance in both helpfulness and harmlessness metrics (avg. $\uparrow7.16\%$ than DPO). Meanwhile, when compared to online sampling methods, our method is also comparable even better while significantly reducing computational overhead and accelerating convergence speed (over $5\times$ faster than GRPO), suggesting our approach as an efficient and effective solution in bridging the gap between efficiency and performance in LLM alignment.

Published in Transactions on Machine Learning Research (01/2026)
RLHF in an SFT Way: From Optimal Solution to Reward- Weighted Alignment
Yuhao Du'
yuhaodu@link.cukk cuh..
Shenzhen Research Institute of Big Data The Chinese University of Hong Kong, Shenzken
Zhuo Li
shwokt@@ink.cchkkccco edu.ccm
Shenzhen Research Institute of Big Data
The Chinese University of Hong Kong, Shenzhen
Pengyu Cheng"
pengguchengg55@@gaill.cm
Qwen LLM Application Team, Alibaba
Zhihong Chen Stanford University
shibongeSsttnfoord edu
Yuejiao Xie The Chinese University of Hong Kong, Shenzhen
puejjoocie@@inn...ukk.......
Xiang Wan
wwncianggsrrbdd.cr
Shenchen Research Institute of Big Data
Anningzhe Gao
aminnghhegoo@gmmillcom
Shenzhen Research Institute of Big Data
Reviewed OpenReview: https: http://veeeee.e..../ net/ forum? id= jewBOUhFu)
Abstract
Reinforcement Learning from Human Feedback (RLHF) crucial for aligning Large Lan- guage Models (LLMs) with human values. However, RLHF has boen continuously challenged by its high complexity in implementation and computation consumption, specifically for on- line sampling-basee methods like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). Even with recent simplifications, such as Direct Preference Optimization (DPO) that designs an offine implicit reward learning objective relying on pre-collected preference datasets, the problems over-fitting and training instability re- main hindering the aligument process from the expected optimal performance. To address the existing challenges, we propose novel simplification of RLHF from the perspective of variational inference, called Variational Alignment with Re-weighting (VAR). Specifically, by directly minimizing the distribution gap between the learning LLM policy and the opti- mal solution of RLHF, we transform the aligument objective into an offinne reward-driven re-weighted supervised fine-tuning (SFT) form, which only requires minor adjustment on the SFT loss to obtain noticeable improvement on training stability and offectiveness. In comprehensive evaluation benchmarks, our objective empowers LLMs to outperform offline alignments, demonstrating superior performance both helpfulness and harmlessness met- rics (avg. 7.16% than DPO). Meanwhile, when compared to online sampling methods, our method is also comparable even better while significantly reducing computational overhead and accelerating convergence speed (over 5x faster than GRPO), suggesting our approach
"Equal Contribution.
i "Correpponding Author. Cade available httpp:://gsthab..cmm//naaaaatn

---

Published in Transactions on Machine Learning Research (01/2026)
as an efficient and effective solution in bridging the gap between efficiency and performance in LLM alignment.
1 Introduction
Large Language Models (LLMs) (T
2024
Touvron et al.)
2023a; Yang et al. 2024) have achieved
remarkable success in extensive applications of artificial intelligence (AI), including dialogue generation (Ab-
dullin ct al.] 2024), coding (Cobbe et al. 2021 Shao et al. 2024). logical reasoning (Suzgun ct al.
2022
AI agents Wu et al. 2023 Hu et al.) 2025a b). and data curation Duu et al. 2025 Li et al.
2024a
2025a
Among the diverse L.LM training techniques, Reinforcement Learning from Human Feedback (RIHFF plays
a core role in ensuring the LLM generation is helpful and harmless (Ouyang et al. 2022 2024
Rafailov et al.
In particular, RLHF first learns a reward model (RM) (Li et al. 2025b) from annotated human prefer- ences, then trains LLMs within reinforcement learning (RL) scheme via Proximal Policy Optimization (PPO) (Schulman et al.
2017 to optimize the expected rewards from the learned RM (Ouyang et al.
2022
Although recognized as the mainstream solution to LLM alignment (Ouyang et al.
2022 Shao et al,
2024
Touvron et al. 2023b OpenAl 2024 Yang et al., 2024), RLHF remains being challenged because of
its expensive computational resource consumption (Cheng et al. 2023 Yuan et al. 2023) and complicated
implementation (Ouyang et al. 2022 Shao et al. 2024 Yuan et al.) 2023) in which multiple models (e.g. the lcarning policy, the reference, the critic model, and the reward model) are required to cooperate the online RL training scheme. Moreover, incorporating such a complicated pipeline significantly induces training complexity and instability, leading to the difficulty of training convergence and the high risk of collapse (Song al. 22023 Go et al. 2023). Although methods like GRPO (Shao et al.) 2024) forego the critic model, instcad estimating the bascline from group scores and significantly reducing training resources, its online sampling strategy still challenges the practical implementation and training speed.
Towards more stable training than online alignment, several ranking- based offline alternatives are proposed, primarily from the perspective of enlarging the likelihood margin between preferred and rejected response pairs in a contrastive approach. Direct Preference Optimization (DPO) (Rafailov et al.) 2024) implicitly maximizes the difference in sampling probabilities between good and bad answers. Ethayarajh et al. (Etha- yarajh et al. 2024) introduces Kahneman-- Tversyy Optimization (KTO) to directly maximize the utility of generations instead of maximizing the log- likelihoooo of preferences. While effective, these methods usually rely on the collection of preferred / rejected response pairs with high quality, which introduces a substitution data collection consumption. Instead, Advantage Leftover Launch (A-LoL) (Baheti et al., 2024) formulates the RL process at the sequence level and derives an advantage-based oflline objective that exclusively uti- lizes preferred responses to achieve hunan-aligned results. However, it still relies on clipping the importance weights to ensure training stability, which prevents the optimization from reaching the true RLHF optima. Furthermore, approaches like DPO and ALoL could employ negative weights for potential dis- dis-preferred re- sponses, leading to an unstable training process due to the unbounded nature of loss landscape when negative weights are applied (Pal et al. 2024 Yan et al. 2024).
In this paper, we address these limitations by proposing a reward-driven variational offline alignment frame- work that eliminates the need for online sampling and clipping, and avoids the instability introduced by negative weights. Our approach reformulates RLHF as variational inference problem over positive mea- sures, ensuring stable and well-defined optimization landscape. Specifically, starting from the closed-form optimal solution of RLHF, we directly minimize the Heilbaack-eibbbee (KL) divergence (Kullback & Leibler 1951] between the to-be-learned LLM and its optimal solution. The resulting loss function takes the form of an oflline reward-driven weighted supervised fine-tuning (SFT) loss, where non-negative weights are derived through an exponential reward transformation. Furthermore, we introduce an efficient in-batch normal- ization tochnique to approximate the normalization term, enabling scalable and practical implementation. Experimental results demonstrate that our framework outperforms existing methods in both stability and alignment performance, providing robust alternative to the challenges of RLHF
2

---

Published in Transactions on Machine Learning Research (01/2026)
2 Preliminary
Reinforcement Learning from Human Feedback is an essential approach to alignment LLMs with human values, especially from the perspectives of helpfulness and harmlessness (Ouyang et al. 2022) We denote the input query to LLMs as a E X, and the model response E y where X, are discrete enumcrable sample spaces containing natural language sequences. RLHF first learns a reward model r(æ; y) from a given collection of human preference data 2, {(0; ywu y where æ a user input prompt, yo Vi are the preferred and rejected responses selected by annotators, respectively, To learn a representative RM, objective is usually utilized:
following Bradley-Terry (Bradley & Terry 1952
E (e DD
loga(r(z,, yw)) - =))]
(1)
where a(-) is the Sigmoid function. Intuitively, Equation induces r(x, y) to assign a higher reward score to the preferred response yo than the rejected response yi with respect to input æ.
With a learned RM r(x, v), RLHF optimizes the target LLM policy no (y((a))) by maximizing the expected reward:
E Prw-Daa-mmaalaallttaa y) [
(2)
where [xrrd is the KL divergence (Kullback & Leibler 1951) between the training policy (e(y) with reference model ())(() to prevent ())()) from the degencration and preserve the generation diversity. B >> 0 is a hyper-parameter to re-weight the expected reward and the KL regularization term
To solve the RLHF objective in Equation 2 Proximal Policy Optimization (PPO) (Schulman et al. 2017 has been rocognized as the mainstream optimization algorithm (Rafailov et al.] 2024]. However, as mentioned in Section [ PPO suffers from training instability and high complexity in computation and implementa- tion (Yuan et al. 2023 Cheng et al. 2023 Therefore, many of recent works have been proposed to simplify
and improve the original PPO algorithm. Rafailov et al. [Rafailov et al. 2024) theoretically demonstrate
that Equation has a closed-form solution:
exp Z(x) (y/e) exp 1 (r(a.1))
n" (y((x²
(3)
where Zxx) (expe y))l is the denominator that normalizes the conditional distribution. Based on the relation between the optimal policy x *" ()) and the RM r(x, y), Rafailov et al. (Rafailov) et al.] 2024) convert the RM learning objective Equation to an optimal policy learning loss named Direct
Preference Optimization (DPO):
-E, oogo o (8 log
no (yrl ar) Traf you x)) Feat y (x)
3 log
(4)
Baheti et al. (Baheti et al.] 2024) adopt the PPO objective into an offline scheme by using importance sampling and converting the expectation of NH (y)(z) to the expectation of (re((y) then propose Advantage- Leftover-Lunch (A-LoL) gradient cstimation:
To ()((xx)
A
Vo log mo(g)ar))
(5)
Tref y a.)
where A is the estimated advantage value Schulman et al. 2016) with respect to Tg. also calculated offline.
Variational Methods provide a principled framework for approximating unknown probability distribu- tions by leveraging optimization over a family of tractable parameterized distributions [Kingma & Welling 2022). The fundamental idca of variational methods is to reformulate probabilistic inference as a functional optimization problem. More specifically, in the domain of machine learning, the goal to find a surrogate distribution go(y) from parameterized family Q {golo O), so that g) can best approximates the
3

---

Published in Transactions on Machine Learning Research (01/2026)
target unknown distribution ply). For example, in variational inference (Blei et al. 2017 the unknown distribution p(y) is approximated by minimizing the inverse KT. divergence between qo ( and p(y):
qo (0(
min ((L((()))) = min E. Ey (v) Dlog 9
(6)
dge
p(y)
Variational inference has been widely applied in various models such as variational autoencoders (VAEs) (Do-
ersch
2016 Bayesian Neural Networks (BNNs) (Graves 2011 and Latent Dirichlet Allocation (LDA) Blei
jet al. 2003 Besides, the forward KL divergence KL[pl|go - EEy~p(y) llog ply) has also been recognized effective objective for variational methods, and has been applied in multiple machine learning scenar- ios such as f-divergence optimization (Namkoong & Duchi 2016), reinforcement learning (Schulman et al.
go(y)
2017), and Markov Chain Monte Carlo (MCMC) (Salimans et. al. 2015).
3 Methodology
3.1
Variational LLM Alignment
We consider the RLHF task from a novel perspective of variational methods. Instead of optimizing the learning policy To (y]az) using the objective in Equation we can minimize the forward KL[TT" between the optimal policy n" (y (a)) and the learning policy Te (y(()))
x <" x)) log TT (3 xx)
KL|x" [[Te] E,,
H(TT' E,, [log To (y]=))]
where H-) is the entropy of .t and is constant with respect to the model parameters 0. For conciseness, we adopt the expectation form of the KL divergence and let E, denote Ey-r(y)a)- As a result, the approximation of n² under minimizing KL. divergence can be achieved by:
min KL[[="]| = max E. [log To a=)]
(8)
0
Using importance sampling [Goertzel 1950 approximales an unknown distribution with a known one, we can rearrange Equation to obtain the following objective by incorporating the closed-fore solution of л² in Equation
Kahn & Harris
1951 Klock & Van Dijk 1978), which effectively
x (y((z)
exp( (0. y)) Z(x)
E,- [log To (y|=)) = E
log (o())))) x))
E,,.r
log no (3)2z)
LVAR.
Log-likelihodd
Reward-driven Weight
This overall objective contains two parts: reward-driven weight and the log-likelihone of on the samples from the reference Tret, which reform the loss as re-weight SFT loss. We call this novel alignment objective as Variational Alignment with Re-weighting (VAR). However, 7(x) = (yy (exp( r (x)) as the partition function remains unknown. Therefore, the following subsection, we discuss the estimation of (zx) within each training micro-batch.
3.2 Batch-wise Partition Estimation
Equation 9 iimples that the key challenge in effectively approximating x² through a parameterized model mo lies in the computation of ((x). However, estimating Z(x) involves summing over all possible outputs for a given x, which can be computationally expensive, as mentioned in previous work (Rafailov et al., 2024). To avoid directly computing Z (xx,, some alignment methods adopt policy gradient algorithms (c.g., REIN- FORCE (Williams) 1992) and PPO (Schulman et al., (017))) that optimize no without explicitly normalizing over all outputs. Hence, with batch of sample pairs {æ, y: BB
1=1: Vi (2;)) approximate each

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1: VAR training within batch
Input Batch B = {(m, yi))= policy To(y)=), reference Tre)(y(a)) reward model r(x y).
for 1 to B do
Compute policy logit lo (v.)e)) for the given input-output pair.
Evaluate reference logits (u)))(s))))))== over all candidates in the batch.
4 Obtain rewards (r(a., ) for each candidate given æ;.
Calculate ((x:) - 2== exp(gr(as vy)) (erly (a.) as the reward-weighttd normalization teri.
Compute &. 2(2,) exp( 3 rav, y:)) log (x:)))) the per-sample loss term for the i-th example.
end
Update No with ivva(() (B) # EE B 2 via gradient descent.
(x;)) using Monte Carlo estimation Salimans et al. 2015
Z(x,) = E (a mm (exp( ( (z v))l v))l = Σ expl 3 g(( v]) Tee((y)e,)
1
DEy
B
1 yj)) ) [ret(y)) r.x, =: ((x)).
exp(
(9)
B
3
When the batch size B is large enough, are uniformly distributed in sample space V. With the above estimation, we can calculate the objective in equation 9 with batch LVAR(B)
collect diverse yj Tess(-la;)) and assume that {u,)
B expl (ri
log To (8,12.)))
(10)
B
((x.))
In conclusion, we summarize the algorithm process of our in-batch estimation and loss computation in Algorithm 1
3.3 Comparison with Previous Methods
Existing preference alignment methods exhibit two fundamental limitations. First, clipping-based approaches like PPO (Schulman et al. 2017):
o
(o, x oct) (ord (x, oce)
T x Oce) (oee [x, (or)
e, + A Ar..
Eg Dow.D.Dooooolllla
min
Ar, clip
(11)
2024 (ref equation 5) bound the importance ratio Ty (g)(=)//TTr((()) within the interval e, e]. This flattens the reward distinctions between responses with similar values. For instance, when two responses have rewards R₁ 100 and R2 99, clipped methods assign nearly identical probabilities (~ 1/2), failing to resolve fine-grained preferences (detailed analysis in Appendix A.T
and A-LoL (Baheti et al.
Second, existing methods such as DPO and A-LoL may encounter optimization instabilities caused by imbalanced or incorrect gradient dynamics. In particular, the gradient descent on dis-preferred responses can become pathologically strong, leading to an unstable optimization landscape. Consider the general weighted SFT objective:
LW-SFT = -E w(x y) log To(
(12)
When w( y) takes negative values for dis-preferred responses, the loss bocomes unbounded below. Min- imizing the loss corresponds maximizing w(x, 8) log To( 2)). For negative weights (w < 0), this re- duces to minimizing log no (g) a)), creating non-compact optimization landscape. While perfect performance
5

---

Published in Transactions on Machine Learning Research (01/2026)
(log ( [)) - 0) is theoretically achievable, it is practically unreachable Gao et al. 2023 (detailed analysis in Appendix A.2
Our key insight is that reward-driven alignment should operate in the space of positive measures. We there- fore propose a variational method that naturally induces non-negative weights through exponential reward transformation:
w(x y) Terr(ylæ) exp(r(x, y)/3) 0.
(13)
This construction guarantees that the loss landscape has well-defined minima bounded by the reference policy's support. By reformulating RLHF as variational inference over positive measures, we achieve stable optimization without artificial clipping or negative weighting.
4
Experiments
To evaluate the effectiveness of our proposed approach, we conducted experiments under two primary set-
tings: (1) the Helpful and Harmless Assistant Task (HHA) (Bai et al. 2022 Ganguli et al.] (022);; and
(2) generative benchmarks, including MMLU (Hendrycks et al., 2020).
HunanEval
(Chen et al. 2021
BigBench- Hard (Srivastava et al. 2023), and GSM8k (Cobbe al. 2021
4.1 HHA Settings
Dataset Our primary experiment utilizes the HHA dataset, which consists of user-assistant conversations paired with model-generated responses labeled as "chosen" or "rejected" based on human preferences. This dataset divided into four subsets: (1) Harmless-base, containing red-teaming conversations designed to elicit harmful responses; (2) Helpful-base, (3) Helpful-online, and (4) Helpful-rejection, which focus on advice- and assistance-secking interactions. We evaluate our method using the test sets of these subsets, comprising total of 8.2K test conversations annotated with human preference labels.
For model training, we employ the OffsetBias (Park et al., 2024) dataset, a. preference dataset similar to HHA. We utilize OffsetBias because our method explicitly relies on reward scores during training. We obsorved that directly applying the HHA training set and its corresponding reward models often results inappropriate reward scores, such as instances where the chosen response receives a lower reward score than the rejected response. This issue significantly compromises the effectiveness of our method. In contrast. Oisset addresses six common types of biases, including length bias-where models tend to assign higher scores to longer scquences- can mislcad reward models into assigning inaccurate reward scores. By mitigating these biases, OffsetBias provides more robust and reliable reward scores, making it better suited for training our model effectively. For all HHA experiments, we use the full training set of Offsett which consists of 8.5K samples.
Models To evaluate the scalability of our method, we conducted experiments on two model collections: Llama-(1B, 3B, 8B, 13B I (Dubey al.] (2224) and Qwen2.5-(0.5B, 1.5B, 3B, 7B, 14B, 32B} (Yang et al. 2024h. Specifically, we benchmark our method against DPO across all models and consider two RL training settings: 1) starting from the pre-trained (base) model; (2) starting from the SFT model. For the reward model, we employ a popular BiaSRM (Park et al. 2024) that is trained on the Ofiset preference dataset. Bia is designed to provide more accurate reward scores by addressing common biases, making it more suitable for our experiments.
Evaluation By following previous work (Rafailov et al. 2024 Baheti et al. 2024). we adopt two popular evaluation strategies: 1) Reward Score: À higher reward score usually indicates more useful and helpful response with respective to the input. Specifically, to calculate reward scores for sequences generated by the aligned models on the HHA test set. Additionally, we also evaluate reward scores on the split Offset evaluation set to assess the in-distribution ability of the models. 2) Pairwise Winrate Score: Following the common practice of LLM-as-a-judgee we evaluate model outputs from the HHA test set using the GPT-4 judge with its standardized prompt template from
the OffsctBinsRM reward model (Park et al.) 2024
ITo ensure the use of the most updated models, we selected Ilamm3.2- 3R), Llama3. 1-8B, and IJama2-13B.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Reward scores obtained by aligning Llama model series using the Oiis training set and evaluating on the four subsets of the HHA benchmark. Additionally, we report the lest. reward the split test set of Bias, "Avg. Helpful" denotes the average reward across Helpful-base Helpful-online, and Helpful-rejection, while "Avg. All" represents the average reward across all four subsets of HHA.
Harmless 1
Helpful f
Method
Avg. Helpful f
Avg. All
OffsetBlas f
online
rejection
Base
37.03
20.51
24.04
21.93
22.16
25.8811.59
21.00
DPO
45.50
44.45
47.07
45.61
45.71
45,6610.
37.31
Jlama3.2-12
VAR
52.48
57.35
60.58
59.38
59.10=0.24
57.4440.14
56.81
SPT+
64.65
64.95
65.90
65.1620.43
62.98.1.330
SPT+VAR
60.19
65.96
68.94
68.27
67.72
65.84..007
61.97
Base
35.05
26.50
31.15
28.60
28.75
30.331..11 14
26.61
DPO
53.71
59.38
60.04
60.55
59.90 20.99=0.000
58.4240.00
53.94
Llama3.2-3B
VAR
57.97
00.23
64.92
62.92
62.69+0.04
61.5 61.51+0.00 +0.00
60.88
DPO+
64.00
69.44
71.01
71.56
70.67 10.00
69.00
63.81
SAT+VAR
71.32
70.83
70.02
68.52.10.12
Base
38.73
34.74
39.96
37.30
37.33
37.65 37.68--0.66 2-0.4
30.42
DPQ
56.17
50.89
61.02
60.86
60.59
59.48
51.38
Llama3.1-4 1-BB
VAR
57.18
61.13
65.57
64.33
63.68 2
62, 12.06++0.11
63.91
SPO+DPO
70.61
69.93 07
68. 18..041.0.11
S
63.24
68.13
71.13
70.78
70.01 10.00....00
68.32 20.51
65.75
Base
33.06
27.39
29.53
28.36
28.43
29.59
27.05
DPO
50.32
50.01
53.68
52.24
51.98 1..15
51.61 10
51.75
Llama2-13B
VAR
58.45
58.94
62.94
61.89
61.26=0.22
60.56+0.26
61.44
SFT+DPO
53.19
59.90
60.61
61.26
60.59 5..59--..2
59.24
59.09
SPT+ VAR
66.07
65.75
65.03 20.15
64.09 20.00
MT-Bench [Zhnng et al. 2023 To alleviate potential positional bias, we present the responses of two models to the judge in two different orders and compare their scores. A model is considered to win only if it does not lose in both orderings. Specifically, we define: Wins: Outperforms in both orderings or wins in one and ties in the other. Tie: Ties in both orderings or wins in one and loses in the other. Loses: Lags in both orderings or tics in one and loses in the other.
Implementation Details For all comparisons, we ensure consistent settings between our method and DPO. We select learning rales of le-5 and 5e-6, employing the AdamW optimizer with a cosine learning rate scheduler. For the Winrate evaluation, we randomly sample 99 instances from each of the four subsets of the HHA test set and conduct experiments using three different random soeds to compute the mean and standard error. To mitigale potential positional bias in GPT-4's preferences, we randomly shuffle the positions of model-generated sequences and the SFT target during evaluation using gpt-4o-2024-11-20 For the Reward evaluation, use the entire ННA test set, evaluating across three different random seeds to compute the mean and standard error. We employ generation parameters = 0.8, top_p 0.9 and top k = 50 for all the generations. All models are trained on 4xA100-80GB GPUs, with Llama3-8B and Qwen2.5-14B utilizing 8xA100-80GH Due to resource constraints, Qwen2.5-32B trained using 4-bit quantization (bnb-4bit)) across nodes, each with 8xA100-80GB GPUs.
4.2 HHA Results
Reward Evaluation Table present the reward scores the HHA test sets. DPO and VAR denote models trained directly from the Base model (pre-trained only), while SFT+ refers to models first fine- tuned via SFT and then further fine-tuned on the SFT model. From Table [ we observe that our method outperforms DPO in both Aug. Helpful and Aug. All across all Llama models for the base version, as well as for the SFT+ version, except. for IJama3.2-3B, where shows a marginal decrease of around 0.5% compared to DPO. Additionally, our method achieves comparable results whether trained directly from the base model or the SFT model, particularly for larger LLMs such Llama3.1-8B and Llama2-13B, whereas DPO struggles to achieve strong results when starting from the base model. Moreover, method achieves performance comparable to the RLHF objective in a single training step, resembling the simplicity and
Note that we only adopt the AT-BBend judging protocol but htttps://uuginggfac.. wwt/oo////ttteeeeeeeeeeeeeeeoeee bitsandbytes
the MT-Bendd benchmark itself.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Reward scores obtained by aligning Qwen2.5 model series using the Offset training and evaluating on the four subsets of the HHA benchmark. Additionally, we report the lest. reward the split test set of Bias, "Avg. Helpful" denotes the average reward across Helpful-base Helpful-online, and Helpful-rejection, while "Avg. All" represents the average reward across all four subsets of HHA.
Method
Harmless
Helpful online
Avg. Helpful
Avg. All f OffsetBies f
rejection
Base
33.03
25.44
30.86
26.94
27.7520.30 30
29.0610.11
40.38
DPO
55.21
55.50
56.24
56.75
56.1710.55
55.9310.4t
53.44
Qwen2.5-0.5B
VAR
55. 22
58
62.32
60.38
60.26=0.21
59.00=0.07
59.50
SPO+
59.91
59.44.0000
58.6810.02
SFT+
58.22
61.72
63.58
63.56
62.95
61.77
60.63
Base
35.01
26.18
32.11
28.13
28.81 0.0.1.
30.36 0..3....2
26.52
DPO
53.40
57.00
57.38
57.85
57.41 4:017
56.41 1.0.3
56.63
Qwen2.5-1.5B
VAR
61.33
64.72
68.87
68.01
67.20+0.00
65.73+0.08
64.75
SPT+
51.51
61.13
61.51
62.26
61.61:0032
59.86g0.22
57.65 07.63
SAR+VAR
67.9910.13
66.691.0 03
Base
47.07
34.27
41.86
36.61
37.58
39.96
45.06
DPO
60.58
63.37
63.84
64.90
64.03 0.0..... 0.53
63.171.0.0
58.97
Qwen2.5-3B
VAR
63.30
G6 29
70.07
69.32
68. 58.56=
67. 17.244:...os
66.06
SPO
59.11:0.17
57.98 0.11
52.66
SFT+VAR
65.15
67.86
71.24
70.78
69.96
68.75
65.63
Base
42.49
42.02
47.96
44:21
41.7320.05
41.17
54.09
DPO
61.51
66.09
66.50
67.28
66.62 6..22.0.11
65.35 0.000
65.81
Qwen2.5-7B
VAR
64.86
66.41
70.87
69.78
69.02 +0.00
67.98 +-0
65.38
SFT SFT+DPO
60.78
62.02
61.62
62.67
62.10+0.22
61.77 11.77++0.330
56.91
SAT+VAR
68.55
67.65
Base
39.29
32.73
39.03
35.16
35.64+
36.55++0.27
45.16
DPO
55.83
55.63
59.04
57.61
57.43 0.2
57.03....0
62.94
Qwen2.5-14B
VAR
64.24
65.23
69. 79
68. 50
67.84+00.14
66.94 66.94440.09
65.65 63
SPO
67.83
68.91
68.31 20.42
67.97 20.38
67.94
SFT VAR
66.37
67.74
71.70
71.08
70.17 20..1
69.22
66.44
Base
38.80
34.36
39.13
36.78
36.77
37.27
38.97
DPO
37.09
31.38
34.58
33.04
33.00
34.02
32.38
Qwen2.5-32B-Int
VAR
50.03
45.36
51.77
17.17
48.30
48.73
69
DPO
37.95
27.70
28.63
28.16
28.16
30.61
30.23
VAR
52.21
52.45
Table 3: Winrate results for the Llama3 series instruct versions (IB, 3B, and 8B) the HHA benchmark.
Harmlese
Helppul
Helpful
Helpffll zejection
A win
win
Tie
A wrin
B win
Tie
A win
win
Tic
A win
win
Tie
All
Helpful
L.uma3.2-1B
DPO
VAR
55.58
20.2
34.34
59.6
10.10
20.2
72.73
49.49
32.32
16.50
18.18
10.77
48.48
12.46
71.38
42.12
16.16
9.09
69.36
34.68
47.47
19.88
17.84
10.77
61.53
29.21:0
63.5340.30
32 32.21
Llama3.2-3D
VAR
76.10
63.30
26.26
10:44
3.37
89.90
78.77
16.81
3.70
6.40
61.96
32.86
5.39
70.13
62.87
12:13
11:45
8.75
73.5750.95
08.4229 ta
76.99
19
Llama3.1-8U
DPO
42.42
42.42
15.15
88.55
88 08.69
19.19
3.37
33.33
34.34
54.21
12.46
85.86
$3.20
10.44
65
15:15
3.70
49.41
74
efficiency of SFT. Table 2 further demonstrates the scalability of our method across different model sizes. Our approach consistently outperforms DPO across all average reward scores on both HHA and OffsctBias, even when starting from the base model. For Qwen2.5-32B, due to limited resources, we employ 4-bit quantization for training. Nevertheless, our method maintains its advantage over DPO.
Winrate Evaluation Figure presents the win rates evaluated by GPT-4o for answers generated by aligned models compared to the SFT targets (chosen answers in the test set) for the Llama and Qwen series. The Llama series show results consistent with the reward scores, where DPO across all models except Llama2-13B, which achieves comparable results. For the Qwen series, our method outperforms DPO across all Qwen models except Qwen2.5-14B, where it shows a slight decrease. The Qwen series exhibit slightly different trends compared to the reward scores, with our method starting from the base version outperforming the SFT+ version across scales from 0.5B 14B and achicving comparable results 32B. These findings further demonstrate that our method can achieve the RLHF objective in a single Si'--ikke step without the need for resource-intensiee reinforcement learning.
method outperforms

---

Published in Transactions on Machine Learning Research (01/2026)
Aligned Model v.s. SFT Target
80
BASE
63.4
DPO
14.1
60
SET+DPO
57.1
Wineste (%)
VAR
466
49.7
51.8
$1.9
40-
100000
SFT+VAR
30.4
19.1
66
Llama3.2-1B
Llama3.2-3B
LIama3.1-BB
Llama2-138
Aligned Model v.s. SFT Target
80
BASF
2011
166
DPO
10:4
60
SFT+DPO
Winrate (%)
VAR
$2.9
SFT+VAR
4500
42.3
75.
40-
36.7
34.0
37.5
26,5
10.9
20-
152
4.5
Qwen2.5-0.5B
Qwen2.5-1.5B
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-14B
Qwen2.5-32B
Figure GPT-4 evaluation results on the HHA test set for the Llama (top) and Qwen (bottom) series, reporting average win rates, where error bars are calculated across three different random seeds.
DPO vs VAR Validation Reward
DPO vs VAR Validation Reward
DPO vs VAR Validation Reward
Validation Reward
Validation Reward
Volidaionn Reward
Vall
IIO VAR
DPO VAR
DPO
VAR
Training Steps (%))
Training Steps (%)
Training Steps (%)
(a) Llama3.2-1B
(b) Llama3.2-3B
(c) Llama3.
Figure 2: Average validation reward during the training process for (a) Llama3.2-1B, (b) Llama3.2-3B, and (c) Llama3.1-8B the OffsetBias dataset, comparing DPO and our method.
Table shows results on the Llama series instruct versions (i.c., models after RLHF). Our method outper- forms DPO by a large margin across three scales (1B, 3B, and 8B) and consistently achieves higher win rates across all subsets of the HHA benchmark. This demonstrates the robustness of our method across models at different training stages, including those that have already undergone RLHF alignment. The results highlight the effectiveness of our approach in further refining and aligning models with human preferences, even when starting from pre-aligned instruct versions.
Training Stability Figure 2 illustrates the average validation reward during the training process for three Llama model collections. Comparing our method with DPO, we observe that DPO exhibits greater volatility and tends to decline from the carly training steps. In contrast, our mothod demonstrates gradual increase in validation reward, ultimately reaching a consistent level. This indicates that our approach is more robust over longer training steps and maintains more stable training process compared to DPO.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Comparison with Online
Table 4: Comparison between offline VAR and online GRPO methods on
Methods We evaluate
Qwen2.5-3B and Mlama3.2-3B. including training time per epoch.
our offline VAR method
against the online GRPO
Method
Harmless
base
base
Helpful
online rejection
Avg. Helpful 1 Avg. All
Train. Time
approach (Shao et al. using the OffsettBias training set, with results measured
2
Qwen2.5-3R
47.07
34.27
41.86
36.61
37.58
39.95
+GRPO +VAR
61.75
68.78
70.73
71.64
70.38
68.23
2h544min 42min
63.30
66.29
70.07
68.56
the HHA test sets (Ta-
Llama3.2-30
33.03
25.44
30.86
26.94
27.75
29.06
GRFT+GRPO
56.31
59.98
62.32
62.72
61.67
60.34
3h8min
ble 4. Despite being an offline method, VAR achieves comparable performance to GRPO on Qwen2.5-3B and even outperforms it on Camaa.2-3BB (where GRPO requires SFT initialization for stable training). Notably, VAR trains approximately 5x faster per epoch than GRPO, demonstrating its efficiency while maintaining competitive performance with significantly fewer computational resources.
+VAR
57.97
60.23
64.92
62.92
62.69
61.51
37min
4.3 Generative Benchmark
Settings Following prior settings (Tunstall al. 2023 Ethayarajh et al., 2024), ; we utilize UltraFoed- back (Cui et al., 2023) the training dataset. Uterbbae is large-scale preference dataset collected from diverse sources, where multiple LLMs generate four distinct responses for cach prompt. The dataset comprises 64k prompls, resulting in 256k samples. Additionally, it includes GPT-4-evaluated scores for mnstruction-follwiin truthfulness, honesty, and helpfulness. For our experiments, we sampled 10k prompts, selecting the highest average-scored samples for training SFT, OURS, and ALoL, and using the highest- worst score pairs for training DPO. For training, we utilize Llama2-7B and Qwen2.5-7B as the base models. For comparison, we benehmark our method against. ALoL and DPO. As for the reward model, we employ Biase for both ALoL and our method, with the same setting in HHA experiments.
Implementation Details For all experiments, use 4 x 8 A100-80GB GPUs, training with learning rate of 56-6 and the AdamW optimizer combined with cosine learning rate scheduler for exactly two epochs. We evaluate the aligned models using the OpenCompass (Contributors 2023 toolkit, with the following benchmarks: GSM8K (4-shot), MMLU (0-shot), HumanEval (0-shot), and BBH (3-shot chain-of- thought), following the default settings from OpenCompass. For GSM8K, MMLU, and BBH, we use exact match (EM) as the evaluation metric, while for HumanEval, we use pass@1.
Generative Benchmark Results Table 5 presents the results of different. methods on Llama2-7B and Qwen2.5-7B models across four benchmarks: MMLU, GSM8k, HumanEvval, and BBH. For Llama2-7B, our method consistently outperforms both DPO and ALoL across most benchmarks, achieving the highest aver- age results. On Qwen2.5-7B, our method also demonstrates strong results, achieving the best performance on multiple benchmarks while maintaining competitive results on others, highlighting the robustness and effectiveness across various settings.
To further assess general conversational ability, we also evaluate the models AlpacaEval 2.0 and Arena-Hard 0.1. The results are pro-
Table 6: results on AlpacaEval 2.0 and Arena- Hard 0.1.
Conversational benchmark
sented in Table 6 On the AlpacaEval 2.0 benchmark, DPO shows
slight advantage, achieving 4.78% win rate (WR) and an 8.73% length-controoled (LC') win rate, compared to 4.29% and 8.26% for VAR, respectively. However, on the more challenging Arena-Hard 0.1 benchmark, our VAR method clearly outperforms DPO, scoring 10.8% DPO's 8.6%. This suggests that the two methods may have different strengths across various conversational benchmarks.
AlpacaEval 2.0
Arena-Hard 0.1
Method
(%)
WR (%)
Wit (%)
Base
4.01
3.73
8.1
DPO
8.73
4.78
8.6
VAR
8.26
4.29
10.8
4.4 Output Sequence Length Analysis
We calculate the average output sequence lengths for models aligned on Llama3.1-8B across the four subsets of HHA, shown in Figure 3 Starting from the base models, DPO generates longer sequences than our
10

---

Published in Transactions on Machine Learning Research (01/2020)
Table 5: Benchmark comparison of different aligument methods on Llama2-7B and Qwen2.5-7B models. Re- sults are shown for MMLU, GSM8K, HumanEval, and BBH, along with the average score across all bench-
176.9
Avarage Output Sequuence Langth
127.5
116.7
marks. Best results for each model are bolded.
103.8
Method
MMLU EM
GSM8k
HvmanEval passt1
BBH
Avg.
EM
EM 1
Base
37.46
1.90
3.05
12.77
13.79
Llama2-7B
DPO
32.45
4.55
7.32
39.10
20.86
ALoL
35.78
4.09
12.80
38.16
22.71
VAR
38.57
6.67
14.02
37.56
24.20
Base
67.13
86.13
64.63
29.30
61.80
DPO
68.64
74.53
33.54
53.06
57.44
Qwen2.5-7B
ALoL 68.62
64.97
54.88
61.14
62.40
Figure 3: Distribution of output sequence lengths for aligned Llama3.1-8B on the HHA Lest set.
VAR 69.11
74.30
68.90
61.35
68.42
method. When starting from SFT, models maintain output sequence lengths similar to the SFT version, while DPO produces sequences approximately twice as long ours and the SFT version. Longer sequences tend to achieve higher reward scores and GPT-based scores (Baheti et al.) 2024 Ethayarajh et al. 2024) Despite this, our method outperforms DPO in most reward evaluations and winrate evaluations.
4.5 Ablation Study
we estimate Z(x;) with a micro batch, making batch size B crucial for training. We conduct an ablation study using batch sizes 2, 4, and under the settings in Section 4.3 with UltraFeed- back as the training set. Table shows the impact model performance. The
As in Equation P
Table 7: Ablation study of different batch sizes B on Q Qwen2.5-7B model and Tyol sampled alternative (8*).
MMLU EM
GSM8k EM
HumanEval passe1 1
BBH EM
Avg.
2
69.59
72.55
65.24
62.57
67.49
of different B values model performs best at B = 8, but the improve- ment over B = 4 is just 0.14%, suggesting that a larger B can slightly boost performance. Thus, we use B = 8 for all experiments. However, our method is also robust to batch-size changes, which gives satisfactory results even at B = 2, showing stability and suitability for resource-constrained situations! I
69.09
75.44
68,29
60.32
68.
69.
74.30
68.90
61.35
68.42
07.83
75.97
65.85
63.66
68.32
Furthermore, we compare our in-batch (offline) estimator to an alternative strategy, Tee sampled (B 8), which estimates 2(z) using dedicated candidates pre-sampled from nee for each prompt. As shown in the last row of Table 7 this alternative approach is not only significantly more costly (requiring a separate, large-scalc pre-sampling process) but also results in slightly lower performance (68.32 Avg. vs. 68.42 Avg.). This result validates our choice of the more efficient and effective in-batch estimation strategy.
4.6 Analyses of (zx) Estimation
The stability of our VAR method depends heavily on getting good estimate for the partition function Z(x). Our approach uses an efficient Monte Carlo estimation based on the samples already available in the training micro-batch (as described Equation and Algorithm D A fair question is how reliable this estimate is, and specifically, how the micro-batch size affects its variance.
To investigate this, ran analysis, shown in Figure 4 We compare the variance of (xx under two different estimation strategies: (1) Offline (In-Batch) Estimation (Figure 4 (a)): This is the method we use in this work; it leverages the B samples already in the micro-batch to calculate 2(x)) for each x; in that batch. The plot clearly shows that the variance drops sharply as B increases from 2 to 8, This confirms that even a modest batch size (like B = 8, which we used in our experiments ) provides a significantly more stable estimate than a very small one. (2) Online Sample Estimation (Figure (b)): For comparison,
We provide study in Appendix C from which we can observe our method effectively produce more comprehensive and helpful response with higher quality compared the baseline methods.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Variance of Z(x) v
Variance 2(x) v
Varlance ofrlanceof211 of
0.0125
Vf
Variance ((())
0.0100
21x)
0.0009
0.0075
0.0050
0.002
0.0000
0.0007
(a) Offline (In-Batch) Estimation
(b) Online Sample Estimation
Figure 4: The variance of the partition function estimate (x) versus the micro-batch size B.
we simulated an "online" approach. Here, for each Fy; we would need to actively sample B new responses from the reference policy wref to compute (z;).
This analysis confirms our design choice. The online method is not only computationally expensive (requiring B extra sampling steps per instance) but also shows higher and less consistent variance at small batch sizes. Our in-batch (offlinc) method is significantly more efficient and yields stable, low-variance estimate as the batch size increases.
4.7 Computational Overhead Analysis
Our method is structured to avoid adding significant over- head to the main training loop. We achieve this by inte- grating the costliest parts of the (xx estimation into the data preprocessing step: (1) Preparation: Before the main training loop starts, we pre-shuffle the train- ing data and split it into the exact micro-batches (c.g., B 8) that will be used for training. (2) Reward Cal- culation: We then calculate the reward r(x4, y5) for cach in-batch pair and store these values on disk.
Table Computational profile comparison for single training epoch (Llama3.2-31, B 8, 4x A100-80GB).
Wall Clock Throughput Tokens/s 1
Peak GPU
Method
Time/Epoch
Mem. /PPU
SET DPO
28mia
29115
<31 GB
322ma
2577
GB
VAR
<-mmin
~22288
388 GB
Therefore, during training, only need to perform one major extra step compared to standard SFT: forward pass through the reference model (Tret) to obtain the logits necessary to compute ((x) (using the pre-calculated rewards). However, the most time-consuming part of training LLMs is the backward pass (gradient computation). Since our training loop does not involve running the backward pass on the reference model (Trer is frozen), the minimal extra cost incurred by the single forward pass is negligible when amortized over the entire training process.
We provide direct comparison of SFT, DPO, and VAR in Table run under identical settings (Tlama3.2. 2- 3B, 4x A100-80GBB GPUs, B 8). The results confirm approach is highly efficient. As shown in the table, the wall clock time per epoch for VAR (37min) is close to SFT (28min) and DPO (32min). This minimal time difference is reflected in a slight reduction in throughput. For peak memory, VAR is comparable to DPO, as both must load the Tref-
5 Related Work
Aligning LLMs with human preferences has evolved from studies on RLHF, aiming to achieve human-aligned outcomes (Stiennon et al. 2020 Ouyang et al. 2022 Bai et al. 2022 Lee et al. 2023). The RLHF process typically begins with SFT, followed by further fine-tuning to maximize expected reward scores. This requires the runston of a reward model based on the Maximum Likelihood Estimatioo (MLE) of the BT model to provide such reward scores. This fine-tuning process referred to as RLHF, with the PPO algorithm being the most widely applicd (Schulman et al., (017). A series of works focus on self-training, where the workflow involves sampling online data from the model and training it using a two-player min-max game between two policies (Rosset et al. 2024 Swamy et al. 2024 Chen et al. 2024). However, the of online data in
12

---

Published in Transactions on Machine Learning Research (01/2026)
the learning process presents significant challenges, it requires substantial computational resources and limits training efficiency. To address these challenges, researchers have shifted their focus to offline preference alignment learning algorithms. These methods operate in single stage and directly optimize a designed loss function to achiove optimal preference alignment based on pairwise datasets (Zhao et al. 2023 Rafailov et al.
2024 Azar et al. 2024 Ethayarajh et al. 2024 Xu et al. 2024). However, standard offline methods like DPO often rely on a fixed reference model, which can lead to optimization instability. To mitigate this, recent alternatives have proposed eliminating the reference-policy mismatch via reference-free objectives (Meng
et al. 2024) minimizing the divergence by learning better reference model (Gorbatovski et al. 2024
Separately, to alleviate the resource-intensive nature of training, Remax (Li et al. 2024b) introduces a variance reduction method for LLMs. Most closely related to our work is ALoL (Baheti et al. 2024). which
formulates the reinforcement. learning process at the sequence level and derives its advantage-baes based offline objective. This sequence-level perspective is further supported by recent theoretical analysis linking SFT gencralization to reward rectification (Wu et al.) (025), reinforcing the motivation for formulating RLHF within a supervised framework. However, unlike ALoL which relies on heuristic clipping, or methods like SimPO that discard the reference model, we formulate our approach by directly approximating the optimal solution of RLHF via variational inference, thereby achieving a more precise and stable solution.
6 Conclusion
This paper proposed reward-driven variational alignment framework to address the limitations of existing RLHF methods, such instability from negative weights and suboptimal performance due to clipping. By reformulating RLHF optimization landscape and derives reward-driven weighted SFT loss through KL divergence minimization. The introduction of an efficient in-batch normalization technique further cnables scalable and practical implementation. Experimental results demonstrate improved aligmment performance and training stability, offering a effective solution for preference alignment in LLM.
variational problem over positive measures, our approach ensures stable
Impact Statement
The Variational Alignment with Re-weighting (VAR) method has a significant impact on LLM alignment, providing an efficient and stable alternative complex RLHF pipelines. While our focus is on improving alignment performance and training stability, we recognize the broader socictal implications, including the risks of bias amplification, misuse of generative capabilities, and ethical concerns surrounding the automation of human-like decision-making We emphasize the need for ongoing research to address these challenges and ensure responsible deployment of RLHF advancements.
Acknowledgment
This work supported by the Guangxi Key R&D Project (No. AB24010167), Guangxi Seience and Tech- nology Program under Grant No. FN2504240022, the Project (No. 20232ABC03A25), Shenzhen Longgang District Science and Technology Innovation Special Fund (No. LGKCYLNS20230188, Futian Healthcare Re- search Project (NW.F2)))02), and Central Funds Guiding the Local Seience and Technology Development Project (No. 20252YDF106).
References
Yelaman Abdullin, Diego Molla-Aliod, Bahadorreza Ofoghi, John Ycarwood, and Qingyang Li. Synthetic dialogue dataset generation using Ilm agents, 2024. URL httpp://aarxiv.or/abbbs
17461
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. Tn International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PML.R, 2024.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, and Mark Riedl. Leftover lunch: Advaniage-based offline reinforcement learning for language models. In The Twelfth International Conference on Learning Representations, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprinl arXiv:2204.0055 05862, 2022.
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022 2003.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.
Journal of the American Statistical Association, 112(518):859-877 877, April 2017. ISSN 1537-274X. doi:
10.1080/01621458
URL http:h/tt////.../// hwtp:/xxxxoii.oooo org. 10000///00000000001114519999999 2017 1285773
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 32: 1952.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. erXiv preprint ArXXc:2107.03374 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du. Adversarial preference optimization. anXiv preprint arXiv:2311. 0tt.22111.880.. 1.0004,, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: http:///ppp..cccoap/////ocoooooa
2023.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv e-prints, pp. arXiv-2310, 2023.
Carl Doersch. Tutorial variational autoencoders. arXiv preprint 3rXiv:1606.05900 2016.
Yuhao Du, Zhuo Li, Pengyu Cheng, Xiang Wan, and Anningzhe Gao. Atoxia: Red- readiig large language models with target toxic answers, 2025. URL ntttp:://arxiv.org/abb////
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandoy, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXin:2407221783 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment prospect theoretic optimization. arXiv preprint or 3rXiv:2402.01306 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint or (xXi:2229.078855 2022.
Loo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835 10866. PMLR, 2023.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahycon Ryu, and Marc Dymetman. Align- ing language models with preferences through f-divergence minimization. arXiv preprint or 15 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Gerald Goertzel. Quota sampling and importance functions in stochastic solution of particle problems, volume 2793. US Atomic Energy Commission, Technical Information Division, 1950.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Mak- simov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint ar arXix:2404.0000 09666, 2024.
Alex Graves, Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Jinpeng Hu, Hongchang Shi, Chongyuan Dai, Zhuo Li, Peipei Song, and Meng Wang. Beyond emotion recognition: A multi-turn multimodal emotion understanding and reasoning benchmark, 2025a. URL
https://arxiv www.aarwiv..ooggbb.ee.a arg/aba/2508.. 16859
Jinpeng Hu, Ao Wang, Qianqian Xie, Hui Ma, Zhuo Li, and Dan Guo. Agentmental: An interactive multi- agent framework for explainable and adaptive mental health assessment, 2025b. URL
org/ abs/2b08 11567
Herman Kahn and Theodore E Harris. Estimation of particle transmission by random sampling. National
Bureau of Standards applied mathematics series, 12:27-30, 1951.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https: //txxx.xvvvorr//////e abs/ Abs//332.6111
Teun Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pp. 1-19, 1978.
Solomon Kullback and Richard Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-88 86, 1951.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellic Lu, Thomas Mcsnard, Colton Bishop, Victor Car- bune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. aiviv preprint XxXo:2209.00267, 2023.
Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, and Anningzhe Gao. Self-instructed derived prompt generation meets in-context learning: Unlocking new potential of black-box 1lms, 2024a. URL httpps//arxxiv..rr
Abs/2409.0155.
Zhuo Li, Yuhao Du, Xiaoqi Jiao, Yiwen Guo, Yuege Feng, Xiang Wan, Anningzhe Gao, and Jinpeng Hu. Add-one-in: Incremental sample selection for large language models via a choice-basee based greedy paradigm, 2025a. URL httts://aaxxx http:/:///xwvvwoaaaaaa/aa org/abs 2
Zhuo Li, Yuege Feng, Dandan Guo, Jinpeng Hu, Anningzhe Gao, and Xiang Wan. Aplot: Robust reward modcling via adaptive preference learning with optimal transport, 2025b. httt::ttp////aat owgpxxiiiioor/oooooooooo abs/
2510.10965
Ziniu Li, Tian Xu, Yushum Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models, 2024b.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances Neural Information Processing Systems, 37:124198 -24225,,, 2024.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with Edivergences. Advances in neural information processing systems, 29, 2016.
OpenAL. Gpt-4 technical report, 2024.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feodback, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. ar preprint arXtv:2402. 13228, 2024.
Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: FMNLP 2024, pp. 1043 1067, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36, 2024.
Corby Rosset, Ching An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences, 2024.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In International conference on machine learning, pp. 1218 1226. PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional con- tinuous control using generalized advantage estimation. In International Conference on Learning Repre- sentations, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepsockmath: Pushing the limits of mathematical reasoning open language models, 2024.
Ziang Song, Tianle Cai, Jason D Lee, and Weijie J Su. Reward collapse in aligning large language models. arXiv preprint arXiv:2305. 17608, 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Marriga-Alnnoo et al. Beyond the imitation game: Quantifying and extrapolating the capabilitics of language models. Transactions on Machine Learning Research, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Newral Information Processing Systems, 33:3008-200 3021, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Lo, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-Chought can solve them. Xr preprinl ar Xii:22200.0000 0..0961000110 2022.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alckh Agarwal. A minimaximalist approach to reinforcement learning from human feedback, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint arXiv:2307.09888 2023b.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXi preprint arXiv:2310.16944, 2023.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256 256, 1992.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint (rXiv:2308.08155 2023.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming- Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.050m 2025.
Haoran Xu, Amr Sharaf, Yummo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation, 2024.
Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, and Dong Yan. 3d-properties: Identi- fying challenges in dpo and charting a path forward. arXiv preprint arXiv:2406.0755 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint 4r 12. 15115, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint XXic:230400530, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slie-hf: Sequence likelihood calibration with human feedback. CoRR, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging lln-as- a-judge with mt- bench and chatbot arena, 2023.
A Theoretical Analysis
A.1 Analysis of Clip Operator and Policy Distinction
The clip operator in PPO is:
1ip(
commonly used in methods like ALoL to stailize training, bounds the importance ratio r(0, ref) = no (x)
within [1 - e, e]. While effective in controlling gradient variance, clipping introduces bias by flattening reward distinctions between responses with similar values.
For instance, suppose for a given instruction x, we have a set of answers {vr, 32. .... b and the loss function of R-LoL (R-LoL used here as a simple example; the only difference between A-LoL and R-LoL is replacing r(z, y) in R-Loo to A(x, y) r(x, y) - V(x)) with clipping is:
n
CR-LOL
(R(z, w) clip(r(), e, log - B No (v)lz))
(14)
dipp:...eep.......11------ B) log (v.l2.).
(15)
17

---

Published in Transactions on Machine Learning Research (01/2026)
When is small, we assume that the parameter 0 in the function r(0,ref) is frozen when do the update of the policy model no, i.e, the loss function becomes:
Σ ()) reif - log no (Walzr),
(16)
and we first update 0, then when we do the next iteration we set 01 = 0. We denote Y = R(a, yı) clip(r(01, ref), e, e,1+e) 3 and we can see that ~ R(x, y4) when e, 3 are small.
We write a; = Ei n
and since the denominator independent of æ, we just neod optimize
a LR.LoL. Σ au log Te (Va
(17)
and aa; = 1. By using the Lagrange's method, construct (for simplicity, write 2, = (b((()))
n
F(21, 22; m. 2n
ar log 2,
3
2i - 1),
(18)
and the partial derivatives are
OF
0i
OF
3,
2/ 1).
(19)
0z 24
03
Hence we can see that the optimal solution of the A-LoL loss is: (BY solving the partial derivatives the optimal solution must have the same distribution as a;)
("((/)//> (()),)) = R(x, ui) -11 [ ciip(r(0,,ef),
(20)
So we can see that for close rewards responses, this optimal solution will not distinguish their distributions. For example, if we have two responses VI, 3/ with R1, R2 as their rewards, then the A-LoL method will give n2 ti+ and Rg t the optimal solution and could be closed to 1/2 if R₁ Rg is closed to 1 (e.g. Rı 100, R2 = 99). However, for our method, it will distinguished by exp RR R2), i.c for R = 100, Rg 99.
our method gives us etI and I
e+T, which seems better.
A.2 Loss Bound Analysis
Lower Bound of Positive Weighted Loss
Theorem 1. For any policy no (y)zz) satisfying E, Nh (g((a))) 1 and weights w(x, y) 0, the weighted SFT loss satisfies:
(())- - -En,y w(x, v) log To(y)ar) 0,
(21)
with equality if and only if no [o(y)æ)= = 8y=p- where 8y=v- is the optimal policy when y" = arg max, w(x, y)-
Proof. Using the incquality logz < 2 for 2 >
((0) = -E E log nel
(22)
E w (To 1)] (since log me S To 1)
(23)
E|w(1 π
(24)
0 (since and no < 1).
(25)
The equality holds when To(y for the y maximizing w(< y).
18

---

Published in Transactions on Machine Learning Research (01/2026)
B Limitations and Future Work
Limited by training resources, the effectiveness of our method can only be validated on Qwen2.5-32B-nnd at maximum, rather than on LLMs with higher precision or larger scales. Additionally, during experiments, we observed some inconsistent performance in LLMs after alignment with our method.
In the future, we will explore several promising directions to further enhance our framework. First, we aim develop an online version of our method, enabling real-time interaction for calculating Z(z) and updating the policy Ne dynamically. Second, we plan to conduct extensive experiments across broader range of tasks, such multi-turn dialogue and long-form text generation, to validate the generalizability of our approach. Finally, scaling our framework to larger models and testing on more diverse and noisy preference datasets will provide deeper insights into its scalability and robustness.
C
Case Study
compares responses from SFT, SFT+ SFT+DPO, Base+VAR, and Base+DPO, along with their corresponding reward scores. Notably, the SFT+DPO response is excessively verbose and includes irrelevant details, yet it receives the highest reward score. This highlights a tendency the reward model to favor longer sequences, even when the content less helpful or accurate. In contrast, SFT+VAR provides a concise and accurate answer, demonstrating the effectiveness of our method in balancing response quality and length. Additionally, Base+tDPO generates highly structured but overly complex and repetitive response, while Base+ VAR delivers clear and straightforward answer. Importantly, Base+ VAR achieves performance comparable to SFT+VAR, showing that our method can effectively align models even when starting from the base (pre-trained only) version, without the need for SFT.
Table 9
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Sample generations from different aligned versions of Llama3.1-8B for prompt about checking a car's gas tank capacity.
Prompt
How I look up how much gas my car hold?
Reward
SFT
You can find out how much fuel your vehicle holds by checking the owner's manual or looking on the door jamb (where some vehicles display this information). You may also able use an online resource, such as Edmunds..com which has detailed specifications for most cars.
31.36
SAT+VAR
You can find out how much gas your mannal or contacting the manufacturer. The amount of gas your car can hold will vary depending the make and model, well as any modifications that have been made to it.
hold by looking at the vehicle's owner's
31.69
SFT+DPO
The total amount of gas that car can hold will depend on the type and size of the car, as well as local environmental conditions and safely regulations. To find out exactly how much gas specific car can hold, it is best to check with the car manufacturer or local fuel suppliers.
60.75
In general, larger cars such 4x4s and vans can store more gas than smaller cars like compact and economy cars. Most modern vehicles have a fuel tank capacity of between 50-120 litres (depending on vehicle size), but some heavy-duty vehicles may have larger tanks of 120-130L or more.
note that many countries have limits on how much LPG CNG vehicle can store due to safety concerns for example, most countries limit the storage of LPG to no more than 20kg, while CNG usually limited to 70m3 (for light duty vehicles) and 100 m3 (for heavy duly vehicles). If you're wu're not sure what these limits are in your country, please check with the relevant authorities or consult the car manufacturer. In addition, certain areas may restrict the use of alternatiee fuels such as LPG and CNG during air pollution season or other times when air quality is deemed unhealthy
It's also important
VARR
You can lookup the capacity of your car's fuel tank by consulting your vehicle's manual or by contacting the manufacturer. Alternatively, you can estimate the capacity based on the size and shape of the fuel tank.
28.58
Base-+DPO
Sure, there is an easy way to look this up:
52.33
1. Look up the cars make and model This will give us its total capacity
2. Find its fuel tank size This will tell us what type vehicle
3. Check if it has any additional tanks containers that may be used for gas
4. Calculate the vehicles maximum capacity
5. Add up any additional tanks or containers that may be used for gas
6. Subtract from the maxiumum capacity anything not yet accounted for
7. Finally, we'll add back any additional tanks containers that were previously
subtracted from
8. Finally, we'll add back any additional tanks containers that were previously
subtracted from
9. Finally, we'll calculate the final maximum capacity
10. We'll now want to check that our calculated maximum capacity makes sense.
Let's go through each step:
1: Make and model To find out the vehicle's make and model (and year), let's run
a Google search.
The results should include the following information about the vehicle: Make/model| ttrim level
Fuel economy
Transmission
Cargo volume
Total width
Wheelbase
Ground clearance
20


Reviews:
[{"review_id": "AYDRExRuGL", "reviewer": "Reviewer_9EjE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Strength**\n\nProposes Variational Alignment with Re weighting (VAR), an offline, reward weighted SFT objective derived by minimizing the forward KL between the RLHF closed form optimum ( $\\pi^* $ ) and the learned policy ($ \\pi_\\theta$ ). The paper derives ( $\\pi^* $ ) (Eq. 3) and shows that maximizing ($ \\mathbb{E}{\\pi^*}[\\log \\pi\\theta] $) yields a reward weighted SFT loss (Eq. 8–10), implemented via Algorithm 1 with an in batch estimator for the partition function (Z(x)). \n\nArgues theoretically that VAR avoids (i) clipping bias in PPO/ALoL and (ii) instability from negative weights (e.g., DPO/ALoL), by operating over positive measures ($w(x,y)\\propto \\pi_\\mathrm{ref}(y|x)\\exp(r/\\beta)>0$) (Sec. 3.3; App. A.1–A.2). \n\n**Weakness**\n\nPartition Estimation is not accurate. The current in batch estimator (Sec. 3.2; Algorithm 1) reuses candidate responses across prompts; without a bias analysis or ground truth comparison, it is unclear how accurate ( $\\hat Z(x)$ ) is, and cross prompt reuse may systematically mis estimate normalization for some prompts. \n\nTime and memory comsumption is not addressed. Beyond Table 4’s GRPO comparison, the paper lacks a direct VAR vs. DPO compute/memory analysis. Given the ($O(B^2)$) cross prompt scoring in Algorithm 1, clearer profiling (tokens/s, peak memory, throughput vs. sequence length) is needed to substantiate efficiency claims and guide practical adoption.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Effectiveness vs. DPO: On HHA, VAR outperforms DPO in average reward on nearly all model sizes and initializations (Tables 1–2), and achieves higher GPT 4o win rates on most Llama and Qwen models (Fig. 1; Table 3). These address the paper’s central claim that VAR is a stronger offline alternative. \n\nStability: Validation reward curves show DPO’s volatility versus VAR’s steady increase and convergence (Fig. 2), directly supporting the “training stability” claim. \n\nEfficiency vs. online RL: The head to head with GRPO includes wall clock times per epoch and shows VAR achieves comparable rewards while being ~4–5× faster per epoch, depending on the model (Table 4). This supports the “significant computational savings” message in the abstract with concrete numbers.", "audience_yn": "Yes", "audience_explanation": "RLHF simplifications and offline preference optimization (e.g., DPO/ALoL/KTO) is a heated topic. A method that (i) ties directly to the RLHF optimum via a variational view, (ii) works as a drop in, reward weighted SFT loss, and (iii) demonstrates competitive performance and stability across widely used model families is highly relevant to both researchers and practitioners concerned with scalability and engineering simplicity of alignment. The paper’s clear comparisons on HHA and standard reasoning/code benchmarks (Tables 1–5; Fig. 1–2) make the findings immediately actionable.", "requested_changes": "1.Quantify the accuracy of the partition estimator\n\nProvide a bias/variance analysis of the in batch Monte Carlo estimator in Sec. 3.2, especially because Algorithm 1 forms ($ \\hat Z(x_i) $) using responses ($y_j$) drawn under other prompts ($x_j$) but re weighted under ( $\\pi_\\mathrm{ref}(y_j|x_i)$ ). This cross prompt reuse is convenient but may be distribution mismatched relative to the ideal (y\\sim\\pi_\\mathrm{ref}(\\cdot|x_i)). Include a sanity study: for a subset of prompts, estimate ($Z(x)$) with many samples ($y\\sim\\pi_\\mathrm{ref}(\\cdot|x)$) and report the relative error of ( $\\hat Z(x) $) as a function of (B). (Sec. 3.2; Eq. 9–10; Algorithm 1.) \n\n2.Report full compute profiles and complexity vs. DPO\nWhile Table 4 compares against GRPO, please also report wall clock time/epoch, tokens/s, and peak GPU memory for VAR vs. DPO under identical hardware, batch size, and sequence lengths. Given Algorithm 1’s ($O(B^2)$) cost for cross prompt reference logits and reward evaluations (steps 3–5), quantifying actual overheads will clarify when VAR is faster/slower than DPO in practice. (Algorithm 1.) \n\n3.Ablations that isolate design choices of ( $\\hat Z(x)$ )\nCompare the current cross prompt estimator to (a) within prompt candidates sampled from ( $\\pi_\\mathrm{ref}(\\cdot|x_i)$), (b) top k candidate subsets, or (c) cached candidate pools built per prompt across epochs. Report effects on accuracy, speed, and memory. (Sec. 3.2; Algorithm 1.)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f3D2s6RYX5", "reviewer": "Reviewer_Di1a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel framework that recasts Reinforcement Learning from Human Feedback (RLHF) as a variational inference problem. The core objective is to directly minimize the KL divergence between the learning policy and the known closed-form optimal solution of the RLHF objective. This reformulation yields a new alignment algorithm, Variational Alignment with Re-weighting (VAR), which manifests as a simple, reward-driven weighted Supervised Fine-Tuning (SFT) loss. The weights are derived via an exponential transformation of the reward, guaranteeing them to be non-negative. This design resolves the training instabilities that can arise in methods like DPO.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors have conducted detailed comparisons of the proposed VAR method against baselines like DPO across two model families (Llama and Qwen (why we are still using Qwen2.5? which is commonly known as a questiable model)) of various sizes. The reward score and win-rate evaluations on the HHA benchmark, in particular, provide strong support for the paper's core claims regarding VAR's stability and effectiveness as an optimization algorithm.\n\nNevertheless, there are two areas in the evaluation section where significant improvements could be made:\n\nThe current evaluation focuses predominantly on the HHA benchmark. While this setup is well-suited for demonstrating the algorithm's effectiveness in optimizing a given reward function (an in-domain evaluation), it does not capture the general conversational and instruction-following capabilities. To substantially strengthen the claims about VAR's practical utility, I strongly recommend including results on at least one widely-recognized, automated conversational benchmark, such as AlpacaEval 2.0 or Arena-Hard 0.1. \n\nIn the win-rate evaluation, the paper states it \"adopt the evaluation template from MT-Bench\" while reporting results \"on the HHA benchmark.\" This phrasing could be confusing to readers, potentially leading them to believe the standard MT-Bench benchmark was run. I suggest the authors explicitly clarify in the experimental setup that they are applying the judging protocol from MT-Bench (i.e., the GPT-4 judge with its standardized prompt template) to evaluate model outputs generated from prompts from the HHA test set. (Why not judged by MT-Bench benchmark?)", "audience_yn": "Yes", "audience_explanation": "This is a good read. The motivation is solid, the proposed solution is elegant, and the writing is exceptionally clear. The VAR method, which reframes the complex RLHF problem into a stable, weighted SFT objective, is a commendable contribution. I must miss something or this paper should be accepted, though I have one primary concern that I would like the authors to address.\n\nMy main reservation regards the given SFT-based formulation. Recent literature (Sft memorizes, rl generalizes: A comparative study of foundation model post-training; Reinforcement Learning Finetunes Small Subnetworks in Large Language Models) has highlighted the critical role of reinforcement learning paradigms for better generalization within alignment. Since the proposed VAR objective is ultimately a weighted SFT loss, it is crucial to assess whether it inherits the potential generalization limitations of SFT, even if it outperforms in-domain.\n\nTo this end, I would like to see a more convincing evaluation of the model's out-of-domain, conversational abilities. The paper would be significantly strengthened by the inclusion of results on widely-recognized, general-purpose benchmarks such as AlpacaEval 2.0 and Arena-Hard 0.1. To be clear, my expectation is not that VAR must outperform DPO on these benchmarks; rather, demonstrating that it achieves competitive or reasonably close performance would be sufficient to alleviate this concern.", "requested_changes": "The paper's motivation for avoiding instabilities present in methods like DPO is well-founded. However, the description of DPO's failure mode as ``employing negative weights'' is an over-simplification. The cited works (3D-Properties: Identifying Challenges in DPO and Charting a Path Forward; Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive) offer a more nuanced explanation based on imbalanced or incorrect gradient dynamics, rather than an explicit negative weighting scheme. \n\nFor example, stating like (just for toy) ``the gradient descent on dis-preferred responses can become pathologically strong, leading to an unstable optimization landscape'' would be more accurate.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "s92NIFi7au", "reviewer": "Reviewer_qm74", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to use a variational approach to solve reinforcement learning with human feedback (RLHF) problem by minimizing the distribution difference between an optimized policy and the optimal solution to RLHF. The method named Variational Alignment with Re-weighting (VAR) provides a solution  takes the form of a reweighed supervised fine-tuning (SFT) . The paper claims that this solution is superior to DPO (an off-line baseline) and GRPO (online baseline) and provides empirical evidence to support the claim. The paper proposes to use a Monte Carlo-based estimator for the partition function to implement  VAR. Another interesting claim based on the analysis is that the reweighing values should be non-negative to ensure stability during optimization. Experimental evidence is provided to show the merits of this method\n\n(EDIT: updated claims made to Yes after rebuttal)", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The paper claims to be an offline alignment method like DPO and uses DPO as the baseline for comparison. However, Algorithm 1 listed in the paper calls for a reward model. Is this reward model explicit? If it's explicit then is the comparison to DPO fair? \n\n- The method, VAR, does not outperform GRPO which is online and has an explicit reward function. If so, one conclusion is that the current method has limitations and online methods like GRPO/PPO should be preferred. This point needs to be addressed as a central claim is that regularized methods like clipping prevent the optimization process to reach RLHF minimum.\n\n- The paper provides a Monte Carlo-like estimate for the partition function. The paper briefly studies the nature of the estimate in Section 4.5. Would it be possible to get more details on the variance of the estimates with batch sizes. Related to this point, the paper defines a term \"micro-batch\". What does this term mean? \n\n- Another claim made in the paper is that the space of reward values should be non-negative. An exponential function is used as this follows from substituting the optimal value from DPO paper [Rafailov et al.]. However, values can be made non-negative by using other functions such as $f(x) = x^2$. Would it make sense to study this ablation to test the claim? \n\n(I have some questions about the claim that may stem from my confusion due to being less familiar with some of the approaches in RLHF.  My  hope is that my questions will be clarified during the authors-reviewers process. Looking forward to authors response.)", "audience_yn": "Yes", "audience_explanation": "RLHF-related methods is an active field so many folks in TMLR community will be interested in the work", "requested_changes": "- In Table 1 and 2, the paper uses several metrics. It would be helpful to the reader to know whether \"higher is better\" or \"lower is better\" to improve readability and interpretability \n\n- Due to batching (micro-batching), does the run-time of the method increase over other methods?\n\n- I was wondering about the claim on non-negative weights which happens naturally with the use of exponential(). Are there other functions that can work just as well? The reader is trying to figure out whether this argument is critical or just a consequence of using closed form solution to DPO", "broader_impact_concerns": "Since this work uses RLHF, a statement similar to what other papers say on the impact of their method would be nice", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)