PIPA: Preference Alignment as Prior-Informed Statistical Estimation
Authors: Junbo Li
Zhangyang Wang
qiang liu
Venue: ICML 2025 poster (2025)
Keywords: Preference alignment
Prior
Probabilistic estimation
Reasoning
Primary area: deep_learning->large_language_models
TL;DR: We propose a unified, RL-free probabilistic framework for language model preference alignment achieving state-of-the-art performance in math reasoning tasks.

Abstract: Offline preference alignment for language models such as Direct Preference Optimization (DPO) is favored for its effectiveness and simplicity, eliminating the need for costly reinforcement learning. Various offline algorithms have been developed for different data settings, yet they lack a unified understanding. In this study, we introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework that formulates language model preference alignment as a Maximum Likelihood Estimation (MLE) problem with prior constraints. This method effectively accommodates both paired and unpaired data, as well as answer and step-level annotations. We illustrate that DPO and KTO are special cases with different prior constraints within our framework. By integrating different types of prior information, we developed two variations of PIPA: PIPA-M and PIPA-N. Both algorithms demonstrate a $3\sim10\%$ performance enhancement on the GSM8K and MATH benchmarks across all configurations, achieving these gains without additional training or computational costs compared to existing algorithms.

PIPA: Preference Alignment as Prior-Informed Statistical Estimation
Junbo Li
Zhangyang Wang' Qiang Liu'
Abstract
web agent (Qin et al., 2025), and more. The fundamental ap- proach to alignment involves supervised fine-tuning (SFT) on the target domain, which essentially maximizes the like- lihood of predicting the next token. However, numerous empirical studies have shown that simple SFT on preferred samples is inadequate for attaining optimal performance (Shao et al., 2024; Ouyang et al., 2022).
Offline preference alignment for language models such as Direct Preference Optimization (DPO) favored for its effectiveness and simplicity, elimi- nating the need for costly reinforcement learning. Various offline algorithms have been developed for different data settings, yet they lack a uni- fied understanding. In this study, we introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework that for- mulates language model preference alignment as Maximum Likelihood Estimation (MLE) prob- lem with prior constraints. This method effec- tively accommodates both paired and unpaired data, as well as answer and step-level annotations. We illustrate that DPO and KTO are special cases with different prior constraints within our frame- work. By integrating different types of prior in- formation, we developed two variations of PIPA: PIPA-M and PIPA-N. Both algorithms demon- strate a 3 ~ 10% performance enhancement on the GSM8K and MATH benchmarks across all configurations, achieving these gains without ad- ditional training or computational costs compared to existing algorithms.
Moving beyond basic imitation learning in SFT, it is sug- gested to learn from both positive and negative samples. Sample quality can be measured by training reward models to capture general preferences (Dong et al., 2024) or lever- aging accurate rule-based rewards (Guo al., 2025) for specific tasks like math and coding. By treating the autore- gressive generation of LLMs as a Markov decision process (MDP), traditional reinforcement learning (RL) algorithms can be effectively applied, such as PPO (Ouyang et al., 2022), SAC (Liu et al., 2024c), REINFORCE, (Ahmadian et al., 2024), etc.
While online RL-based methods deliver strong performance, they face challenges such as high training costs, instability, and the need for strong base model as the initial pol- icy. As result, offline algorithms like direct preference optimization (DPO) (Rafailov et al., 2024) are often pre- ferred, thanks to their effectiveness and simplicity, particu- larly when high-quality datasets are accessible. The original DPO algorithm has several limitations. It relies on paired preference data, which is not essential for tasks with ground truth such as math and coding. Additionally, it is unable to accommodate step-level annotations. Furthermore, it treats all tokens equally, lacking token-level credit assignment. To address these issues, a series of approaches have been developed, such as Kahneman- Tvversky Optimization (KTO) (Ethayarajh et al., 2024) for unpaired data, Step-DPO (Lai et al., 2024; Lu et al., 2024) and Step-KTO (Lin et al., 2025) for step-level annotations, and RTO (Zhong et al., 2024), TDPO (Zeng et al., 2024), and OREO (Wang et al., 2024) for fine- ine-grained token-level DPO. However, these methods are designed from specific perspectives, each addressing only particular challenges, and they lack a unified understanding to integrate their solutions.
1. Introduction
Pre-training large language models (LLMs) from scratch on trillions of text tokens allows for accurate prediction next to- kens in natural language (Achiam et al., 2023; Dubey et al., 2024; Liu et al., 2024b). Following this, alignment, achieved through fine-tuning on smaller, high-quality datasets de- signed for specific tasks, becomes critical for enabling the model to develop specialized skills, such as engaging in conversation (Ouyang al., 2022), math reasoning (Shao al., 2024; Yang et al., 2024), coding (Zhu et al., 2024),
'The University of Texas Austin, US. Correspondence to: Qiang Liu <lqiang uu.u.........u.........
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this work, we introduce unified framework designed to address all the aforementioned challenges in offline ap-

---

Prior-Informed Preference Alignment
proaches. Rather than framing the alignment problem within offline RI., we reformulate it as maximum likelihood estimation (MLE) problem with prior constraints, operat- ing within a purely probabilistic framework called Prior- Informed Preference Alignment (PIPA). From a statistical estimation perspective, we analyze the suboptimality of su- pervised fine-tuning (SFT). We demonstrate that both the original DPO and KTO algorithms can be interpreted as special cases within our framework, differing in the prior information they incorporate and the loss used. Building on the PIPA framework, we propose two variants, PIPA-M and PIPA-N, that incorporate prior information in different fashions. The probabilistic formulation naturally accommo- dates unpaired data and extends to step-level annotations. Our PIPA functions as a versatile plug-in loss design that seamlessly integrates with any (iterative) data generation pipeline in existing alignment framework. Furthermore, we show that PIPA training effectively learns token-level credit assignment, yielding precise per-token value estimations that may enable search during test-time inference.
like math and coding, KTO (Ethayarajh et al., 2024), Step- KTO (Lin al., 2025), and OREO (Wang et al., 2024) have been proposed. Our PIPA framework addresses all these challenges within a unified paradigm, demonstrating that existing algorithms like DPO and KTO can be interpreted as special cases within our approach.
Probahilistic alignment In addition to reward maximiza- tion, some research approaches alignment from a probabilis- tic perspective. (Abdolmaleki et al., 2024) decompose label likelihood into target distribution and a hidden distribution, solving it using the EM algorithm. Other works leverage importance sampling to train a policy parameterized by an energy- based model that aligns with the target distribution including DPG (Parshakova et al., 2019), GDC (Khalifa et al., 2020), GDC++ (Korbak et al., 2022), BRAIn (Pandey et al., 2024). (Dumoulin et al., 2023) uses a density estima- tion formulation and recovers DPO. Unlike these methods, our PIPA framework maximizes label likelihood while en- forcing prior constraints by transforming it into the target distribution using Bayes' Theorem. We directly Iearn the distributions without relying on complex sampling and esti- mation procedures. PIPA is scalable, incurs no additional training cost, and remains flexible across any preference data.
Our contributions can be summarized as follows:
We formulate preference alignment as a prior-informed conditional probability estimation problem that is RL-free and provides clear theoretical insight.
Our approach does not need paired preference data, and seamlessly unifies both answer-wise and step-wise settings under a single, theoretically grounded framework.
2. PIPA
We introduce the prior-informed preference alignment (PIPA) framework in Section 2.1, followed by the first ver- sion, PIPA-M, in Section 2.2. Next, we explore its connec- tion to DPO (Rafailov et al., 2024) and KTT (Ethayarajh et al., 2024) in Section 2.3. Drawing inspiration from DPO and KTO, we develop the second version, PIPA-N, detailed in Section 2.4. In Section 2.5, we extend PIPA-M M and PIPA- N naturally to incorporate step-level annotations. Finally, Section 2.6 presents the refined algorithms for PIPA-M and - and compares them with prior methods like KTO.
Compared to existing approaches such as DPO (Rafailov et al., 2024) and KTO (Ethayarajh et al., 2024), our algo- rithm achieves improved performance without additional computational overhead.
1.1. Related Work
Learning from preference data RL has become a key framework for leveraging preference data for LLM align- ment, with early methods like PPO (Schulman et al., 2017), which first trains a reward model on pairwise human feed- back (Ouyang et al., 2022). Due to PPO's high training cost, direct policy optimization methods without online RL have been explored, integrating policy and reward learning into single stage. Notable works include DPO (Rafailov et al., 2024), SLiC (Zhao et al., 2023), IPO (Azar et al., 2024), GPO (Tang et al., 2024), and SimPO (Meng et al., 2024). For fine-grained token-level optimization, DPO variants like TDPO (Zeng et al., 2024), TIS-DPO (Liu et al., 2024a), RTO (Zhong et al., 2024), and OREO (Wang et al., 2024) have been introduced. To address step-level annotation in- spired by PRM (Lightman et al., 2023), methods such Step-DPO (Lai et al., 2024), SCDPO (Lu et al., 2024), and SVPO (Chen et al., 2024b) have emerged. To relax pairwise data constraints, particularly for tasks with ground truth
2.1. Prior- Informed Preference Alignment
We define the preference alignment problem as probabilistic estimation. Assume that we are given a preference dataset
Problem
Lo J A Fi=1 patta
where x² is the instruction input, y² is the answer, and c¹ [0, 1} represents the preference or correctness of the answer. We are interested in predicting y given in the correct case (c = 1). This amounts to estimating conditional probability:
ply x., c 1).
The canonical approach for estimating p(y | x, c = 1) is, of course, the maximum likelihood estimation (MI.E), which
2

---

Prior-Informed Preference Alignment
yields supervised finetuning (SFT) on the positive examples with c= 1.
Parameterization of PIPA-M Recall that our final goal is estimate p'(y | I = 1), which is expected to be parameterized with an autoregressive transformer model. Hence, we are going to parameterize the target p (c z, y) using P'(y | x;, c = 1) and constraint p0 x). This can be obtained by Bayes' rule:
However, SFT only uses the positive samples, rendering the negative samples with = 0 unusable. Preference alignment methods, on the other hand, aims to use both positive and negative data to get better estimation. But how this possible while adhering to statistical principles, given that MLE statistically optimal and p(y 2,C 1). by definition, involves only the positive data (c 1)?
max Eponup (cl x. = Eyaus p'(v x. c)p (c) 8
(3)
pd x)
which includes the two terms we are interested with an additional term pa(c z). We set the two terms in the numerator to be learnable and the denominator to be fixed by prior constraint. Denote
The idea is that the estimation should incorporate impor- tant prior information involving the negative data, thereby introducing a "coupling" between the estimations of the positive and negative probabilities, p(y I, c = 1) and ply c = 0). Generally, this prior-informed estimation can be formulated as a constrained optimization problem that minimizes loss on data fitness subject to prior infor- mation constraint. In this work, we always set the data loss to be the log-likelihood of the label c:
I'(w x x, 1)= - f"
(4)
to be two learnable networks. The likelihood for positive and negative samples are:
10 x)g²(()
p³ = x. v) pprior (y x))
(5)
max Eypans log (cl x. v) s.l. p² PriorInfo. (1)
9
(c 0 x. y) = - (c= = 1
By assuming different prior information, we can derive various algorithms a principled manner, with a transparent understanding of the underlying priors and preferences. The problem formulation in (1) naturally does MLE for each sample (x, y, c) without the need of paired data as in DPO (Rafailov et al., 2024). If only N pairs are available, we can decouple them into 2NV samples as in KTO (Ethayarajh et al., 2024).
Therefore, we reformulate PIPA-M (2) as unconstrained problem by directly maximizing the log-likelihood p(c x, y) via (5). The resulting loss is similar to KTO (Etha- yarajh et al., 2024) that does not need preference pairs, but differs notably in the loss formulation. A detailed compar- ison is provided in the next section.
In addition, in PIPA-M we notice that:
2.2. PIPA-M: Enforcing Prior on Marginal Distribution
Since p²(y. c x) = pc x. v)p" 2)) PIPA-M problem (2) is equivalent to directly maximizing the joint probability p²(y e x)) with p (y () being a fixed prior.
We first consider a straightforward case when we know that the marginal prediction p(y z) should match a prior distribution p (y | æ) defined by the pretrained LLM. Because the marginal distribution is sum of the positive and negative probabilities, that is,
It's possible that the parameterization with f and g makes (C = x, y) large than 1. Theoretically, we can first use network of and than set g (x) min(g p pprior term to be well-defined. In practice, we observe that such cases are rare and we just apply clipping outsides of 1 m)) (z) presos (g (a))
p(y x) p(y x, 1)p(1122) + p(y 2, 0)p(0)z),
x))f"(y x)) to ensure the
where we abbreviate p(c- = ilzz) as pillr)). The estimation of the positive and negative probabilities are coupled.
to make it smaller than
In this case, the estimation problem is in principle for- mulated as the constrained maximum likelihood problem (PIPA-M):
2.3. DPO and KTO: Prior-Informed Views
We analyze existing methods, such as DPO and KTO, through the lens of the prior-informed estimation frame- work (I). Our analysis demonstrates that both DPO and KTO can be interpreted as enforcing prior assumptions on the negative probability, p'(y) x, c = 0), rather than the marginal probability, p(y x:). However, these methods differ in their choice of loss functions, the prior assumptions made about p(e x),, and the parameterization employed.
max Eyeana logr x. x. s.f. x)(1x)== = persor (y) x). Vz, y-
This is by definition the best way statistically to estimate p provided the data information pdata and the prior constraint prrior.
3

---

Prior-Informed Preference Alignment
Algorithm 1 PIPA: Prior-Informed Preference Alignment
1: Input: A dataset ((x, y/ } of questions æ, answers y, and preference with E {0, 1)"; fixed prior model perior, and trainable models fo and g" initialized from an SFT model fo. Choice: either PIPA- -M or PIPA-N.
2: for each batch do
3: PIPA-M: Compute
PIPA-N: Compute
F (r. = f0e x. ye2)) (x,
(ye 2, 22) g M<r))
(2, == T
pprior (ye 2. (21)
pprior (Ve a, yke) (1 - (z.
where T(x) == x+1
x
4: Minimize the loss:
Mx, = Σ to
log Fo (x. vgt) - E
log(1 F6 (s))))))
t:
5: end for
Direct preference optimization (DPO) and related methods are often cast as estimating the log density ratio log(p(y c,c- = 1)/p(y c = 0)) of the positive and negative generation probabilities (Dumoulin et al., 2023). Related, it may not be of suprise that these models make the implicit assumption the negative (reference) probability p(y 2, 0) = perior (y æ)) In particular, DPO can be formulated as
DPO
addition, it makes a particular assumption on the class ratio p (c= 0 x)/p = 1 x)) which is consistent with the fact that the class percentage is no longer guaranteed to be balanced without paired data. In particular, KTO uses
2 (x) = (L (y) x.
= 1) pprior (y z)).
Here 2 depends on parameter 6 but the gradient is stopped through 2 the KTO algorithm. In practice, 2² is esti- mated with empirical samples in each batch. Details and proof are shown in Appendix A.2.
max LoFa(p" pdata)
(6)
s.t. pd (y] x, c = 0) = pd pprior(y) (y/ x)),
1
2.4. PIPA-N: Enforcing Prior on Negative Condition Distribution
cc=c=1 x) = p((c (c-cc-00 0 [))
-) Vx,
where the loss T.po is a special pairwise comparison loss related to Bradley-Terry model provided paired data fæ, "t.""""")" }. of both positive answer y* p(y 2x, c 1) and negative answer y ~ ply x, c = 0) for each a; the assumption of (c = 1 r) = 0.5 is due to the balanced sampling of positive and negative weights. See Appendix A.1 for more discussion for Lopo and proof of equivalence.
Knowing the prior informed formulation of DPO and KTO, we can propose changes to make them simpler and more natural. One option is to keep the conditional log- likeliiel loss function of KTO, but seeks to learn p²(c = 1|z) as a neural network g" (zz) from data, rather than making the heuristic assumption. This yields
max Eypasa [logr' (c 2r, v)] 9
KTO
One key limitation of DPO is that it requires to use paired data. KTO has been proposed as an approach that relaxes the requirement. In the informef framework, it can be viewed as solving:
s.f. p'(y x. x= 0) = prrior (y) x). Vz, y.
The only difference with PIPA-M (2) is in the prior con- straint. We call this PIPA-N, for it places prior the negative probability, As in PIPA-M, we apply Bayes' rule to pC(c x, v), but additionally expand the denominator 1d (y x) using:
max Eyton (2(cc. 0 s.1. x, -= 0) = py x:)
log (c 0 x)
== 2x)x
Vx, y.
2z)) =p (p (==g 1>p xx)
(=== x))
+ +p x, (c=
where it changes the loss function to the standard conditional likelihood without log, which holds for unpaired data'. In
This allows us to incorporate prior information on p²(y | x, c = 0). With the parameterizations p²(y I, = 1) =
In the KTO paper, an importance weight placed the posi- tive and negative data (Ap, Ay in their notation), but the default
settings the code is balanced weight (Ap = Xu = 1).
4

---

Prior-Inforie Preference Alignment
Σ
g:)
Σ
log
prior
fe
g
max
0
correct
incorrect
Head
Head
Head
Head
Autoregressive transformer
Input tokens:
1
1
1
1
1
Step (correct)
Step (incorrect)
Figure 1. The figure illustrates PIPA-M, with PIPA-N following a different loss function as outlined in Algorithm We denote f2 : f y<c) p" (3) x, Mes) be the target cext-token-predicttee probability, of 9 yct) p" (c. 2, V<t) be the value model, which differ only in their output hcads. And pf := preior (ye Vicc) is another frozen language model. When only answer-level annotations are available, objective contain only one of correct and incorrect parts.
(y (yyl x)) and p (c= x) = (x), we obtain:
can still define C1 = =.. E {0, 1]. For any 1 S T, denote Vgr == (y), ye) and the same for Cgt- Figure presents a visualization comparing token-level representa- tion with the previous sequence-level representation.
( (x)
p-=c=11 x, y) = pprior (y x²(1 (z)))
0== x, = (c |1,,)),)) v).
Representation
Annotation
Label
where 7(x) </(x + 1). This allows us to reduce the problem an unconstrained optimization of maximizing Eyana [log 2 (cl x.,) x.,)].
Sequence
Answer
Token
Answer
0
0
0
0
0
0
Token
Step
1
1
0
2.5. Extension to Step-level Settings
The introduction above about the answer-level setting, which uses single label for entire solution-even if multiple steps may vary in correctness. In contrast, the step-level setting assigns separate labels to each step. The advantage of our probabilistic framework is that it seam- lessly adapts to a step-level setting for both PIPA-M and PIPA-N algorithms. The core idea is intuitive. We use token-level labels rather than answer-level labels, followed by decomposing the joint probability autoregressively.
Figure 2. We show visualization for the label of negative answer under different circumstances.
Parameterization Same as the answer-level setting, the objective still (2) PIPA-M or (7) for PIPA-N. We fac- torize p (c xr y) in an autoregressive manner for c. Since Ce can be determined by (x, Vet), so is conditionally independent of both y>t and eat given (z, Vet). We have
p x. v)
[[ x,
Il p(ce x. x.y)))
Problem formulation In the step-level setting, decom- pose answer y and label to tokens. Specifically, for each data (z, c) with & steps and I' tokens in the answer, we have = (V). ... yr) and c = (c). ... ..r) E {0. 1)T, where e, is if the corresponding step is correct otherwise Notice that only answer-level annotation available, we
By Bayes' Theorem, for each have:
p(c xx p(ve xr, Det, (p)p(e xr, vet)) p(y Ie. Vee)
5

---

Prior-Informed Preference Alignment
Now similar to the answer-level PIPA, we introduce neu- ral networks parameterize (y: x. M<+)) (v: T, yict, Ct 1) and g (r, (r,)<<r)= =- p (c+ = x. Mcx.)
3. Experiments
3.1. Settings
Then we solve the same unconstrained optimization problem as answer-level setting, iee, maximizing Eggeoalogg p8 (c r, y)]. See Algorithm I for details.
Our PIPA framework is capable of handling scenarios both with and without preference pairs, as well as with or with- out step-level annotations. Consequently, we evaluate it across four distinct experimental setups determined by (pair, unpair) x (answer, step). In this work, we primar- ily focus on math reasoning tasks, as they serve as a strong testhed for the scenarios under consideration. We leave systematic exploration of general tasks for future work.
2.6. Practical Implementation
As shown in the formulation of Section 2.5, we always use PIPA-M and PIPA-N with token-level label representation in practice which provides fine- greingd information.
We have three models in total: f ".ggppiior. In practice, f is the target language model initialized from the one obtained after Supervised Fine- Tuning (SFT) and will be used for inference after training. g" shares exactly the same base model with 1 differing only in the output head. perior is also a language model but frozen. We show our PIPA- M and PIPA-N in Algorithm I, and figure illustration in Figure 1. For the negative samples in PIPA-M, we apply clipping function to constrain the term g² /prrior within [0, 1 - e). where 10-6.
Baseline algorithms For paired preference data, we use DPO (Rafailov et al., 2024) and its variant IPO (Azar et al., 2024), and KTO (Ethayarajh et al., 2024) as baselines. Both KTO and PIPA decouple the paired data. For data without preference pairs, we compare PIPA with KTO. In answer- wise settings, we benchmark PIPA against DPO, IPO, and KTO. In step-wise settings, we compare PIPA with Step- DPO (Lai et al., 2024) and Step-KTO (Lin et al., 2025). The original Step-DPO and Step-KTO methods involve ad- ditional data generation phase. For fair comparison on an offline dataset, we extract only their loss functions. See Appendix B for detailed descriptions of the baseline algo- rithms.
Comparison with KTO Although it stems from completely different derivation, KTO (Ethayarajh et al., 2024)-which also targets pair-free alignment-is the clos- est prior work to PIPA in terms of its algorithm. A detailed analysis of their differences and PIPA's advantages can be found in Appendix A.2.
Data We use the unpaired preference dataset for math rea- soning released by AlphaMath (Chen et al., 2024a), which includes training problems from GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) with both CoT (Wei et al., 2022) and TIR (Gou et al., 2023)- solutions, along with step-level label annotations. There are more in- correct answers in this dataset, so we construct the paired subset by matching each correct solution to a corresponding incorrect solution from the same problem and discarding the remaining incorrect solutions. We use the entire original dataset for the unpaired setting.
Credit assignment A key issue with traditional DPO and KTO loss is that all tokens receive equal treatment, since only the answer-level reward is considered. PIPA offers natural way to weight f" x, Vet) differently by jointly optimizing it with g Vet) = pc( = x, v<e). The optimized gº, with its clear probabilistic interpretation, can be viewed as a value function and may be used for inference- time search in future work. We present its learning trajectory in Section 3.
For the answer-level setting, we only keep the final label for the answer. In the step-level setting, steps of a correct answer are always correct. For incorrect answer, label steps whose Q values fall within [-1, as incorrect, and those in [0.5, 1| as correct with a threshold 0.5. Instead of threshold 0, the intuition of this shifted threshold is that it's better to be conservative for the correct steps in the wrong answer. Despite being correct, some steps may still contribute to an incorrect overall analysis. Therefore, minimizing the likelihood of such steps is also crucial. The efficacy of this choice is further explored in the ablation study of Section 3.3.2.
Compatibility and flexibility PIPA can be applied when- ever answer- or step-level annotations are available, requir- ing no additional training stage. These step-level annota- tions can be derived via MCTS (Chen al., 2024a; Zhang et al., 2024b; Guan et al., 2025) or by LLM-as-a-judge (Lai et al., 2024; Lin et al., 2025). Furthermore, PIPA easily gen- cralizes to an iterative version, similar to other works (Xiong et al., 2024; Pang et al., 2024; Wang et al., 2024). In this paper, we focus on statistical estimation using a static offline dataset, leaving the online version for future exploration.
For our evaluation, we use the standard GSMSK and MATH benchmarks. We adopt the MARIO evaluation toolkit (Zhang et al., 2024a), configuring the beam search width and number of generated samples, i.e., (B1, B2) in
6

---

Prior-Informed Preference Alignment
their notation, to be (3, 1) for GSM8K and (1, 1) for MATH.
self. Section 3.3.2 examines the step-level and answer-level setrings.
Model The AlphaMath dataset is generated using Deepseek-based models, so we use Deepseek-Math-7BB Instruct (Shao et al., 2024) as the pre-trained model for
3.3.1. ALGORITIMS
PIPA-M vs. PIPA-N PIPA-M and PIPA-N are two ver- sions of our framework that incorporate distinct prior con- straints. As shown in Table I, neither variant consistently outperforms the other. Notably, PIPA-N tends to perform better with the Deepseek model, while PIPA-M shows su- perior results with the Qwen model. This may suggest that PIPA-N is better suited for self-alignment scenarios, while PIPA-M is more effective for alignment tasks where there is a distribution shift between the model and the data. In practice, we recommend experimenting with both variants to determine the optimal choice for your specific use case.
self-alignment. Additionally, to evaluate generalization ca- pabilities, we test Qwen2.5-Math MMath--B-Insae BB-Instruct (Yang et al., 2024) as the base model on the same dataset. For PIPA, we set the head of gd be two-layer MLP followed by Sigmoid function, with hidden dimension 4096 same as the base model.
Training All experiments are conducted based on Open- RLHF (Hu et al., 2024). For all training. we use LoRA (Hu et al., 2021) with rank 64 and a = 16. All alignment algorithms are conducted for epoch after the SFT stage. Denote bs be the batch size and Ir to be the learning rate. we do grid search for Ir (5 10-7,5 5 x 10-6, 5 x 10-5) for all experiments and present the best one.
Value model p(c, | 2, 2.8<t) Our framework employs two components: the target model p'yy | x.C = 1) and value model p (c+ trained through optimization of their combined probabilis- tic objective, rather than being learned separately. Down- stream task evaluations confirm that p²(y F.C = 1) is well optimized. To assess the impact of the value model, we present the results of removing it in Table 2, where the performance decline highlights its importance. To examine the value model's learning behavior, we plot the training trajectory of II. (ce x. x,,<()) " using Deepseek model in the step-wise setting for the first 300 steps in Figure 3. The implicit optimiza- tion process yields continuous improvement in likelihood estimation, with the model's predictions showing a steady increase from the initial random-guee -uesss baseline of 0.5. This empirical validation establishes foundation for using the optimized value model p (c+ xr, U<ce) to implement search during inference, presenting clear direction for future research.
2, V<t)- These are jointly
SFT Before all alignment algorithms, we first fine- tune the pre-traing pre-trained Deepseek and Qwen models the positive samples for 3 epochs with bs = 1024 and Ir = x 10-5. The model obtained after SFT is then used as the initilization for the target model s in alignment procedures, as well as the fixed reference model in DPO and KTO, Furthermore, to avoid extra computation, this same model serves as the prior in both PIPA-M and PIPA-N, ensuring that PIPA does not require an additional training phase compared to DPO and KTO.
GSM8K
MATH
78.24
51.82
73.54
47.92
DPO For DPO-based algorithms including DPO, IPO, Step-DPO, we train 1 epoch after the SFT stage, with bs= = 226,6 (r = 5 x 10-7 and 8 = 0.1.
Table 2. In the second row, p" (cc x. is fixed at 0.5.
KTO For KTO, we set Ir = 5 x 10-5 for Deepseek model and Ir = 5 x 10-7 for Qwen model. For both, bs = 256,55 3 = 0.1. KtOp shares exactly the same recipe with KTO.
PIPA We set bs = 256 for all four settings, Ir 5 x 10-5 for Deepseek and 5 x 10-7 for Qwen. All settings are the same as KTO and Step-KTO, without additional hyperparameters to be tuned.
Prior choices To ensure fair comparison with baseline methods such as DPO and KTO, we utilize the same SFT model for initializing y (y | x. == 1) and setting priors. In our PIPA model, however, the priors should ideally be p x)) for PIPA-M and p(y | x, = 0) for PIPA-N, which differ from the SFT model's p (1)ax, further investigate PIPA with accurate priors, we began with the released Deepseek model, training it on both positive and negative samples for three epochs to derive p (y I). and solely on negative samples for three epochs to obtain pd (yy 2, 0). Unfortunately, these adjustments did not yield improvements. Results are shown in Table 3. This
3.2. Main Results
We show our main results in Table 1. We can see that for all four settings and two models, PIPA achieves the best performance without additional computation cost.
1). To
3.3. Additional Analysis and Ablation Studies
We conduct a more detailed analysis from two perspectives. Section 3 delves into further studies on our PIPA it-
7

---

Prior-Informed
Preference Alignment
GSM8K
MATH
Data
Annotation
Algorithm
DS
QW
DS
QW
DPO (Rafailov al., 2024)
68.39
67.17
46.94
47.78
IPO (Azar al., 2024)
69.14
72.33
46.94
49.96
Answer-wise
KTO (Ethayarajh et al., 2024)
76.72
62.47
47.38
46.53
PIPA-M
79.08
73.77
50.82
51.60
Paired
PIPA-N
80.29
70.89
52.32
47.26
Step-DPO (Lai et al., 2024)
68.54
66.11
46.96
48.38
Step-KTO (Lin et al., 2025)
75.44
62.47
47.38
45.64
Step-wise
PIPA-M
79.15
74.91
51.94
53.26
PIPA-N
78.70
73.84
52.54
49.06
KTO (Ethayarajh et al., 2024)
76.04
64.44
46.72
47.08
Answer-wise
PIPA-M
79.08
74.75
51.04
52.78
Unpaired
PIPA-N
80.97
74.22
52.22
52.00
Step-KTO (Lin et al., 2025)
76.81
64.14
46.98
45.64
Step-wise
PIPA-M
78.24
74.22
51.82
53.10
PIPA-N
79.98
72.86
52.78
52.52
Table 1. Results GSM8K and MATH. DS means Deepseck-based models, and QW means Qwen-based models.
to the loss function.
0.625
P (Answer-lovs)) lovel)
PIPA N (nnwerreeeel)
0.600
PIPA.M (Stop-leve.)
Likellhood
PIPA N (Step level)
GSM8K
MATH
0.575
PIPA-M PIPA-M(T)
78.24
$1.82
GSMSK
MATH
0.550
77.86
46,72
0.525
PIPA-N
79.98
52.78
KTO+SFT
76.27
47.96
PIPA-N(T)
79.83
50.84
PIPA M PIPA-M+SFT
79.08
50.82
0.500
78.24
50.12
50
100
150
200
250
300
Table 3. PIPA with different pri- ors. (T) denotes further fine- tuning of the SFT model on all or negative samples.
Training Step
Table 4. Effect of additional SFT loss on KTO and PIPA-M.
Figure 3. We plot the geometrically averaged likelihood (C | x. vce)) + for PIPA-M and PIPA-N with answer-level annotation and step-level annotation respectively during training. showing consistent increase.
3.3.2. STEP-LEVEL SETTING
As our research is the first to systematically explore the performance of alignment algorithms across various settings, we provide in-depth analysis of the step-level setting in this section. We aim understand the advantages of step- level annotation and how it influences the training.
may be due to the lack of training on marginal or negative samples the initial pre-training and fine-tuning stages of model development, meaning that a few epochs fine- tuning insufficient to establish accurate priors for these distributions.
Additional SFT loss Previous studies (Pang et al., 2024; Dubey et al., 2024) have shown that incorporating an addi- tional SFT loss in DPO enhances its stability. We extend the unpaired setting, applying
Influence of step-level annotation From Table 1, we observe that step-level annotation does not consistently improve performance when comparing answer-wise and step-wise annotation within the same algorithm and dataset. Specifically, step-level annotation proves beneficial for MATH but can sometimes negatively impact GSM8K. This finding aligns with previous studies (Chen et al., 2024a), suggesting that step-level amnotation is more advantageous for challenging reasoning tasks like MATH but may be un- necessary even harmful for simpler tasks like GSMSK.
our PIPA-M and KTO algorithms in the answer-win wise case, with SFT loss coefficient set at 1.0. As shown in Table 4, incorporating additional SFT loss provides more advantages for KTO compared to our PIPA, yet it remains less effective than our PIPA. This is because our PIPA is theoretically grounded for general case and does not require further modifications
8

---

Preference Alignment
answers while retaining sufficient high-quality positive steps maintain learning efficacy.
0.-
0.0
Chosen Reward
Rejeett Reeward
0.0-
-02
4. Conclusion
-0.1-
-22
Step DPO
-44
Stsp-DPO
In this paper, we introduce Prior-Informed Preference Align- ment (PIPA), a fully probabilistic framework grounded in statistical estimation. We analyze the limitations of pure SFT within this framework and demonstrate that DPO and KTO emerge special cases with distinct prior constraints, We propose two variants, PIPA-M and PIPA-N, each incor- porating different prior constraints. Through comprehensive evaluation across four distinct data settings, we system- atically highlight PIPA's advantages over previous meth- ods. Additionally, ablation studies reveal the optimal design of PIPA, while empirical analysis explores the impact of step-level annotation from multiple perspectives, leveraging PIPA's flexibility.
DPQ
50
1:00 Traininn Step
150
200
50
100 150
200
Training Step
2 zx._m11) for both correct and Figure 4. We present plots of log (orler(y #))
incorrect samples, shown in the left and right tigures respectively, for both the original DPO and Step-DPO. While the term for cor-
samples decreases as observed in previous studies, it increases in Step-DPO.
Reward curve As shown in previous works (Pang al., 2024; Dubey al., 2024; Razin et al., 2024; Liu et al., 2024d), one problem in DPO is that the implicit reward for
positive samples log yasase [y]] ==(x)=1e=11 can also decrease during preference
since this term is pre- cisely what DPO aims to optimize. Their approach ad- dresses this problem by adding extra SFT loss during DPO training. We have noticed similar patterns in our DPO ex- periments. Notably, we found that employing step-level annotation can effectively address this issue. This observa- tion offers an alternative angle for tackling the problem in DPO, stemming from the absence of step-level annotation. It's possible that some steps in incorrect answers are actually correct and share similarities with the distribution of cor- rect answers. Consequently, minimizing these correct steps in incorrect answers with the original answer-level DPO could also reduce the likelihood of correct answers. Our results highlight the importance of fine-grained, step-level annotation alignment from new perspective.
A Acknowledgment
We thank the anonymous reviewers for their helpful sugges- tions. Z. Wang in part supported by NSF Award 2145346 (CAREER) and 2212176. This research has been supported by computing support on the Vista GPU Cluster through the Center for Generative AI (CGAI) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin.
Impact Statement
This paper introduces a principled statistical framework for LLM alignment that allows us to both understand existing methods and deriving new efficient ones. Experimental results highlight its effectiveness in improving LLMs' math- ematical reasoning, while its applicability extends to broader social implications, such as enhancing L.LM safety, line with previous research in the field.
Threshold for positive steps The step-level annotation, specifically the Q value obtained by MCTS in our Al- phaMath dataset (Chen et al., 2024a), is presented in a continuous format ranging from [-1, The simi- lar continuous format is employed in other annotation pipelines such I.LM-as-a-judge (Lai et al., 2024).
References
Abdolmaleki, A., Piot, B., Shahriari, B., Springenberg, J. T., Hertweck, T., Joshi, R., Oh, J., Bloesch, M., Lampe, T., Heess, N., et al. Preference optimization probabilistic inference. arXiv preprint 1rXiv:2410.04/660 2024.
In our main experiments, we employ a default threshold of 0.5 for labeling correct steps in incorrect answers. As analyzed in Figure 5, which evaluates the im- pact of varying threshold values, we observe that an intermediate threshold achieves optimal perfor- mance. This balance ensures cautious filtering of positive steps in negative
MATH
52.0
Accuracy
51.0
50.5
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08741 2023.
49.5
0.0
0.2
0.4
0.6
0.8
1.0
Thereshold for Positive Steps
Figure 5. Accuracy MATH with different threshold for the positive steps in wrong answers.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basies: Revisiting reinforce style optimization for

---

Prior-Informed Preference Alignment
learning from human feedback in llms, arXiv preprint arXiv:2402. 14740, 2024.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
Anonymous. Mask-DPO: Generalizable fine-grained fac- tuality alignment of LLMs. In The Thirteenth In- ternational Conference on Learning Representations, wwweeevvwwweeeeeeeeeeeeeeeeeeeee
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.0381 2021.
2025. URL
id=d2H1oTNITh
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455, PMLR, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhi. Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Hu, J., Wu, X., Zhu, Z.,, Xianyu, Wang, W., Zhang, D., and Cao, Y. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv: 2405. 10.0011113.. //13, 2024.
Chen, G., Liao, M., Li, C., and Fan, K. Alphamath almost zero: process supervision without process. arXiv preprint MrXiv:2405.035533 2024a.
Chen, G., Liao, M., Li, C., and Fan, K. Step-level value pref- erence optimization for mathematical reasoning. arXiv preprint arXiv:2406.1066
Khalifa, M., Elsahar, H., and Dymetman, M. distri- butional approach to controlled text generation. arXiv preprint arXiv:2012.1166 2020.
2024b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution match- ing for fine-tuning language models with no catastrophic forgetting. Advances in Neural Information Processing Systems, 35:16203-16222 2022.
Dong, H., Xiong, W., Pang. B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint XXXx:2405.00863. 2024.
Lai, X., Tian, Z., Chen, Y., Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimiza- tion for long-chain reasoning of llms. arXiv preprint 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint MXXi::305.200555 2023.
Dumoulin, V., Johnson, D. D., Castro, S., Larochelle, H., and Dauphin, Y. density estimation perspective on learning from pairwise human preferences. arXiv preprint arXiv:2311.14115. 2023.
Lin, Y.-T., Jin, D., Xu, T., Wu, T., Sukhbaatar, S., Zhu, C., He, Y., Chen, Y.-N., Weston, J., Tian, Y., et al. Step-kto: Optimizing mathematical reasoning through stepwise bi- nary feedback. arXiv preprint arXiv:2501.100 10799, 2025.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Liu, A., Bai, H., Lu, Z., Sun, Y., Kong, X., Wang, S., Shan, J., Jose, A. M., Liu, X., Wen, L., et al. Tis- dpo: Token-level importance sampling for direct prefer- ence optimization with estimated weights. arXiv preprint arXiv:2410.04350. 2024a.
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang. Y., Huang, M., Duan, N., and Chen, W. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint 2309 /7452, 2023.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C.,, Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint 4rXiv:2412.19437. 2024b.
Guan, X., Zhang, L. L., Liu, Y., Shang. N., Sun, Y., Zhu, Y., Yang, F., and Yang. M. rstar-math: Small llms can master math reasoning with evolled deep thinking. arXiv preprint Krxx:25001.45 04559, 2025.
Liu, G., Ji, K., Zheng, R., Wu, Z., Dun, C., Gu, Q., and Yan, Enhancing multi-step reasoning abilities of language
10

---

Prior-Informed Preference Alignment
models through direct q-function optimization. arXiv preprint 0rXhv:2410.09302, 2024c.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song. J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. /6436, 2024d.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H.,, Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Lu, Z., Zhou, A., Wang, K., Ren, H., Shi, W., Pan, J., Zhan, M.. and Li, H. Step-controlled dpo: Leveraging step- wise error for enhanced mathematical reasoning, arXiv preprint ar Xiv::2407000782 2024.
Wang, H., Hao, S., Dong, H., Zhang, S., Bao, Y., Yang, Z., and Wu, Y. Offline reinforcement learning for Ilm multi-step reasoning. arXiv preprint 2...::4422.161455 2024.
Meng. Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint 2rXiv:2405.1473: 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. Zhou, D., et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248377, 2022.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems,
2022.
Xiong, W., Dong, H., Ye, C.,, Wang, Z., Zhong, H., Ji, H., Jiang. N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kllconntraint. In Forty-first International Con- ference on Machine Learning, 2024.
Pandey, G., Nandwani, Y., Naseem, T., Mishra, M., Xu, G., Raghu, D., Joshi, S., Munawar, A., and Astudillo, R. F. Brain: Bayesian reward-conditioned amortized inference for natural language generation from feedback. arXiv preprint ar arXiv:2402.02479 2024.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409.12122 2024.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.197333 2024.
Zeng, Y., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization. arXiv preprint arXiv:2404.) 1999, 2024.
Parshakova, T., Andreoli, J.-M., and Dymetman, M. Distri- butional reinforcement learning for energy- based sequen- tial models. arXiv preprint 19//:v:9/220855 2019.
Zhang, B., Li, C., and Fan, K. Mario eval: Evaluate your math Ilm with your math llm-a mathematical dataset evaluation toolkit, 2024a.
Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., et al. Ui-tars: Pioneer- ing automated gui interaction with native agents, arXiv preprint ar arXiv:2501.12326, 2025.
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, Rest-mets*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024b.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Salch, M., and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. 2023.
Zhong, H., Feng, G., Xiong, W., Cheng, X., Zhao, L.,, He, D., Bian, J., and Wang, L. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404. 18922, 2024.
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.008: 08847, 2024.
Zhu, Q. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek- coder- v2: Breaking the barrier of clored-source models in code intelligence. arXiv preprint arXiv:2406. 1931, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. 2017.
arXiv preprint

---

Prior-Informed Preference Alignment
A. Detailed Discussion about Connection to DPO and KTO
A.1. The DPO Loss
DPO uses pairwise comparison loss on paired positive and negative data. Denote the pair data to be {xi, y² yy }. where
y" is the chosen answer sampled from p(y zz 1) and y² sampled from ply = 0) is the rejected answer. Using our notation, the DPO objective (Rafailov et al., 2024) is
max Ellog (r" , ,))],,
(8)
where
1 Ny, = -log p'(y
Bi e = 1) 1)pprior (y x)) I. 1 1)prror (y+ x))
pd (y
From our perspective, DPO can be viewed as minimizing a pairwise comparison loss, subject to an prior assumption on the negative probability ply æ, c = 0). rather than the marginal probability p(y æ). We show in the following Theorem.
Theorem A.1. Draw (I. c) from joint distribution p. for which p(c = 1 = 1/2. Further, set the sample yf y, and draw an independent contrastive sample via ~ ~p(- x, -c). c). Then maximizing the original DPO objective (8) is equivalent to solving the following problem:
Ep llggp (c x, (y¹ [)]
max 0
s.l.
7'(1 0 x1 0) prior (y) xr),
(9)
p= = (c = x) = 0.5, Vx,y
Proof. Using Bayes' Theorem, we have:
(2 , x c)p (VV" y () x)
(c x. (v.
(10)
Denote
11 (y .y"""""" ",,",,, xr,,, =)=;)==11 = == (3 xr c)p (y x, c)2
Define (y+,y") = |1=11 () + |c=0 (y 1 ".)) We have
2,, c)p (cl x))
pc x, ()) = h" h" y c), x) + hy y " ", c)p cla0
p' c)p (1 (3) xr, - c)
p' x, x,c)p"(
x, c) + pd (y x, 1 c)p²(y xxC)
x)) 1 (y+ xr, x)) + parior (yt x))²(1 [x,==
1 prior (y-
(11)
p" (y+ 2, 1) 1)) (y
On the other hand, notice that log (x) - log(1 + exp( -x)). So the original DPO loss is
max log (r" (,,,,, "y,, ,
= log(1 + exp(- -r "0yy* (1y+.90 xz))
p'(1" (y I, = 1) 1)p (y+ x)
log 1 pd (y+ I 1)pprior (y-
From this, it's straightforward to see that DPO loss (8) is equivalent to applying - log() to (11).
Therefore, Theorem A.1 shows that DPO is well recovered by our framework. In DPO, besides injecting prior for p(yl x, c= 0) instead of ply x), it has additional prior (c = (c 0 z). A direct idea is to remove this prior, and set pd (c = æ) be a learnable model for similar to PIPA-M.
12

---

Prior-Informed Preference Alignment
A.2. Connection to KTO
The KTO objective is
max you
Ico (ha(a,y) - z(x)) + (1 cJoor (-ho(z, + +(x)))
(12)
where
p (y)
x, =
ho = log perior y x
and
=(x) : KL(p"(v (v x (===))lle 1) a;))) We show the following equivalence. Theorem A.2. Maximizing the original KTO objective (12) is equivalent solving the following problem:
max E. Ega (c(2r,
(13)
s.t. Vz, y p0 ( I,C= = 0) = -prrioo (y |x)
log (' 0 x) KL(p'(v> (v/ x; 1) prio x)).
p¹ = x))
Proof. For the original KTO objective, we have:
(ha(2, u) - =(x)) = (log p'(y x, pprior (y x))ez(0))
p x c ==
pn(v x c = 1) + pprior x))=(0)>
a--ho(r,y) + x(x)) = (ha(v.y) - =(x)).
And for (13), we have:
p'(y x, c= = 1)p² (c = 11
p² 12,,) p"(y x; 1)p (c= x) + persor (y x)p
p -r 1)
pl x +) pprior (y x) -
(c=1(p)
Hence the equivalence is straightforward.
A.2.1. COMPARISON WITH KTO
Algorithm 2 KTO (Ethayarajh et al., 2024)
1: Input: data ((x, y, c where e {0, 1}, a fixed reference model prrior, trainable models fo initialized with Pu-
2: for every batch do
3: Compute:
("Vll Vce)) pprior (ye 2 yes) (es))
Fo(x.y) Σ log
4: Estimate z(x) == (L(f((y 2)]lprrior (y x)).
5: Denote (Ix) = 1+exp(-2)* Minimize loss:
L(z,v,c) -co (Fo 2(x)) - (1 cjo (-Fa + x(x)).
6: end for
We present KTO in practice in Algorithm using our notation for better comparison. The prior assumption on p(c x)) does not seem to be natural and removing yields PIPA-N. In terms the algorithms, PIPA has the following key differences with KTO:
13

---

Preference Alignment
PIPA has additional learnable head of capture p (c V<r). Unlike the language model head, which matches the vocabulary in its output dimension, g has output dimension of only 1. Consequently, the extra parameters amount to fewer than 1% of the total, resulting in no overhead in both memory and speed.
KTT does not extend the step-level setting, whereas PIPA seamlessly accommodates both the answer-level and step-level settings within single framework, supported by clear theoretical guidance.
KTO needs to additionally estimate the KL term c(x) by pairing random question and answers, which adds an extra step and slows its overall process. In contrast, even with its additional learnable head, PIPA remains faster in practice.
KTO uses the SFT model for both fixed reference model and f initialization, and this the only choice. PIPA framework allows arbitrary selection of the fixed prior model perior, For simplicity, we can choose perior fo which is the same choice KTO. But in PIPA, since the prior pertor is unrelated to f", we can also set perior to be the fine-tuned version of fo on both positive and negative data to get better estimation.
Following DPO, KTO views the log ratio log(f" / /perior) rewards, which is directly maximized or minimized. In PIPA-M, the ratio (fg(f" / is the log likelihood, and we need to compute log(1 gd / for the negative steps, instead of things like log(f" g² /prrior) as KTO and other works.
gog(f"""/pProo
B. Baseline Algorithms
The original Step-DPO (Lai et al., 2024) requires preference data generated from a tree structure and maximizes the standard DPO loss on the diverging nodes. Here, we propose generalization of the algorithm that works with more generic paired data, without requiring tree structure.
Step-DPO
We are given pairwise data y V/" contains a mix of ones and zeros. First, we define
c=, with token-level representation, where cf consists entirely of ones, while c
p (yt 2, Vete =) ri (x v) == log perior (yt x, Mc())
Treating the sequences as a whole, the original DPO loss given by
Σ (e( - E Entov))
Loeo (x, yt y ,,, .c, = - log o
Given that the positive steps in yr negatively impact model performance if minimized, straightforward approach when providing step-level annotations is to exclude these steps (see e.g., Anonymous, 2025), which yields the following loss function:
La(x,y yy = llog Σ En((rtut)
)n((a, tse, =0
where we remove the positive steps y from the log ( term. However, a potential issue arises when the number of steps varies, the magnitude of the term inside a() may differ, affecting the optimization due the a() function applied externally. To address this, propose intermediate solution between the original DPO and the above loss. Specifically, we apply the stop-grade gradientt operation to the positive steps in y and get l,:
( En(antt)
Σ re(2, v)) sg Σ ro(a.8 (n(as)))) 1o0
Li n,,,y,,e -
Σ
Ato m1
where sg(.) denotes stop gradient.
In essence, Lo masks the implicit reward of positive steps within negative answer in the objective, while L1 masks these positive steps only during the backpropagation step when computing gradients. The loss function Lo corresponds to the approach introduced Anonymous (2025), whereas L1 is equivalent to the Step-DPO formulation when using tree-structured pairwise data. Our experiments adopt I,,, as it demonstrates better performance.
14

---

Prior-Informed Preference Alignment
Step-KTO Very recently, Step-KTO (Lin al., 2025) introduced loss function designed for data with step-level annotations. They partition the answer into groups corresponding steps, where each a() contains only group. For unpaired data (z, V/ c) represented at the token level, let < 3K 'I' denote the starting tokens of all K steps. Here, C remains constant for & E [sk, Sk+1) for I k I K. The function rr(z, y) follows the same definition as in Step-DPO. The original KTO loss is
Lgro(Z, 8, = cro
En( 20 + (1 crjo - Σ re(x, + 20
The Step-KTO loss is given by
)
( v, =
Csp o
Σ St<ce44
(1( 20 (1 cs
Σ SKSt
re v) + 20
However, our experiments revealed that Step-Kiy KTO loss does not improve performance. Inspired by the Step-DPO loss proposed earlier, we adopt the original KTO for positive answers while applying different approach for negative answers by masking the gradient of positive steps:
(
(z =
Σ =:0==0
re(a, y)
ag
Σ re(2,)) y) + 20 ::0==1
The key idea is to retain the forward pass of all steps in o() for normalization while excluding positive steps in the backward pass to prevent their probabilities from being minimized.
15


Reviews:
[{"review_id": "FRWwZcxGzV", "reviewer": "Reviewer_3B8U", "summary": "The paper presents Prior-Informed Preference Alignment (PIPA), a unifying probabilistic framework for offline preference tuning of language models. It views alignment as a maximum likelihood estimation with constraints that tie the “correct” and “incorrect” output distributions to a reference prior. Within this perspective, the authors explain how existing algorithms like DPO and KTO are special cases with different ways of imposing prior constraints. They also propose two PIPA variants (PIPA-M and PIPA-N), each incorporating the prior differently, and how they can handle both answer-level and step-level annotations. Experimental evaluations on GSM8K and MATH confirm that PIPA consistently outperforms baselines under multiple conditions while retaining a simple, efficient training procedure.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the recognition of our theoretical contribution in unifying existing preference alignment methods under a single probabilistic framework, and our experiments demonstrating PIPA's effectiveness on math reasoning tasks with stepwise feedback.\n\n## 1. Hyperparameter validation  \n\n**Validation via learning trajectory**  \nFollowing the paper referenced by the reviewer, we plot the trajectory of the forward KL divergence and the win rate computed by the implicit reward during training on Alphamath with Deepseek (https://anonymous.4open.science/r/PIPA-DB7B). Both metrics indicate that DPO effectively learns throughout the training process without exhibiting the over-parameterization issues highlighted in the referenced paper.  \n\n**Validation via additional results**  \nOur hyperparameters were carefully selected through rigorous search, ensuring that they were not cherry-picked to favor our results. To illustrate this, we present additional results for batch size, $\\beta$, and learning rate—three key hyperparameters in DPO. The current settings in the paper are (256, 0.1, 5e-7). We here find a slightly larger $\\beta$ than 0.1 allows for a higher learning rate and improves performance, leading to better results. However, DPO still falls short of PIPA. Specifically:\n\n- **Batch size:** Consistent with previous studies, we use a fixed batch size across all algorithms to ensure a fair comparison by maintaining the same number of updates. We evaluate batch sizes of {64, 256, 1024} for DPO, IPO, and PIPA, and the results consistently show PIPA’s advantage at all batch sizes.  We use 5e-7 for DPO and IPO here because larger LR will crash.(https://anonymous.4open.science/r/PIPA-DB7B/table1.md)\n\n- **$\\beta$:** Initially, we set $\\beta$ to 0.1 following the original Alphamath paper [1]. We then explored values from {0.5, 1.0, 2.0, 5.0} and found that $\\beta$ = 1.0 yielded the best results. However, DPO still lags behind PIPA. (https://anonymous.4open.science/r/PIPA-DB7B/table2.md)\n\n\n- **Learning rate:** With optimal batch size and $\\beta$, we try learning rates from {5e-7, 1e-6, 5e-6, 1e-5, 5e-5}. We find increasing $\\beta$ to 1.0 allows for a higher optimal learning rate of 1e-5. This adjustment improves DPO's performance on the GSM8K dataset, reaching 0.74 and narrowing the gap with PIPA to 5%. However, it does not lead to any improvements on the more challenging MATH dataset. (https://anonymous.4open.science/r/PIPA-DB7B/table3.md)\n\n\n## 2. Non-math preference tasks\nWe employed the paired version of PIPA-N as outlined in Theorem A.1, with further algorithmic details provided in Section 2 of response to Reviewer Zrno. Following the setup proposed in SimPO, we conducted comparisons between DPO and SimPO by fine-tuning Llama-3-8B-instruct on the Ultrafeedback dataset. We reproduced the DPO and SimPO baselines in our codebase using OpenRLHF, maintaining identical hyperparameters from the SimPO paper (e.g., batch size, epochs, $\\beta$, $\\gamma$), except for using LoRA (r=64, $\\alpha$=16) with a higher learning rate, as the original lower learning rate is ineffective with LoRA. Our reproduction produced results matching those reported by SimPO.\nFor our PIPA-N, current results are better than SimPO. It’s noteworthy that due to the limited rebuttal period, we did not have the resources to thoroughly optimize the hyperparameters for PIPA-N. In contrast, the SimPO results benefited from careful tuning (e.g., using $\\beta$=2.5 and $\\gamma$=1.375 for Llama-3-8B-instruct). This suggests that PIPA-N has the potential for further improvement with more extensive parameter tuning. More results will be provided later given the limited response period.\n\n| Algorithm | AE2LC | AE2WR|  AH |\n|-----------|-------|---------|----------|\n| DPO | 41.17 | 36.25 | 30.47 |\n| SimPO | 43.28 | 39.86 | 32.49 |\n| PIPA-N | **44.79** | **40.53** | **33.08** |\n*Table 4: Non-math tasks.*\n\n## 3. Comparison to SimPO\nTable 2 shows that SimPO is similar to DPO in Alphamath which lags behind PIPA, and Table 4 shows that PIPA-N is also better than SimPO.\n\n## 4. Missing citation\nWe will cite the paper by Zhihan Liu et al. in Line 421.\n\n[1] Chen Guoxin, et al. AlphaMath Almost Zero: Process Supervision without Process, NeurIPS 2024.", "claims_and_evidence": "Overall, most of the paper’s claims are supported by evidence in the form of clear theoretical derivations (showing how DPO and KTO emerge as special cases of PIPA). However, one possible limitation is the scope of experimentation: all empirical results come from math-focused datasets (GSM8K and MATH). While these are strong tasks for stepwise feedback, it’s less clear how PIPA behaves for broader preference-alignment scenarios.", "methods_and_evaluation_criteria": "While the math-domain experiments are systematic and informative, a key limitation is that both the tasks (GSM8K, MATH) and the models (Deepseek-Math-7B, Qwen2.5-Math-7B) are math-focused, which may not fully generalize to broader open-ended preference scenarios (e.g., creative writing, summarization, or subjective opinion tasks). In particular, it is unclear whether step-level preference alignment would improve performance on non-math benchmarks where correctness is less binary. For example, I would suggest evaluating the proposed methods on widely used preference alignment benchmarks, such as AlpacaEval and ArenaHard.\nMoreover, while the authors compare PIPA against DPO and KTO variants, they omit more recent or higher-performing preference-alignment methods like SimPO, which reportedly achieves strong results on open-ended generation tasks.", "theoretical_claims": "The main theoretical claims center on \n1.\tShowing that DPO and KTO emerges as special cases by imposing different prior constraints on the model distribution (theorem A.1, theorem A.2)\n2.\tConstructing a parameterization that satisfies a marginal or negative-prior constraint (theorem 2.1)\nThe derivations are consistent with standard probabilistic modeling arguments, and there do not appear to be any major errors", "experimental_designs_or_analyses": "From an experimental design perspective, one area that could be strengthened is hyperparameter validation. While the paper mentions a grid search, it isn’t entirely clear whether the proposed methods and baselines are treated equally or whether the chosen hyperparameters reflect their optimal settings. From my experience, simple hyperparameter tuning can make it seem like a proposed method shows better performance than the baseline.\nA helpful way to address potential “hyperparameter hacking” is to plot a “GT win rate” (or accuracy/performance) against the KL-divergence across a dense grid of hyperparameter values for both the new methods and the baselines (similar to approaches in other RLHF or offline alignment work (https://arxiv.org/abs/2406.02900) which uses a similar plot for observing overoptimization). Such plots would demonstrate (1) whether the selected hyperparameters lie near a reasonable optimum and (2) that the same thoroughness is applied to all methods. Without this, one might worry that the authors’ methods have undergone more meticulous tuning than the baselines, potentially skewing the comparison.", "relation_to_broader_scientific_literature": "A number of recent works (e.g., DPO, KTO) already eschew traditional RL algorithms in favor of more direct, offline approaches. However, these methods typically target either paired preference data (DPO) or unpaired data (KTO) without a unifying outlook. The paper’s “prior-informed” lens systematically shows how such offline algorithms can be viewed as special cases of one probabilistic formulation.", "essential_references_not_discussed": "In line 421, the authors discuss previous works showing that DPO decreases the probability of positive samples during training. They, however, left out one work that also shows this problem (“Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer” https://arxiv.org/abs/2405.16436).", "other_strengths_and_weaknesses": "The most original aspect is the unified viewpoint that treats offline preference alignment as a constrained MLE problem. This clarifies perspectives, bridging various existing methods—DPO, KTO, Step-DPO, etc.—under one probabilistic framework. The theoretical consolidation is meaningful: clarifying how each alignment technique differs only by the choice of prior constraint (marginal vs. negative) can help practitioners see how new or existing approaches fit in.", "other_comments_or_suggestions": "It would be nice to see how the recently proposed SimPO, which shows significant performance gain compared to DPO or other DPO variants, fits into the proposed framework.\nMoreover, as discussed earlier in my review under [Experimental Designs Or Analyses], it is essential to perform experiments on non-math-related tasks and validate the choice of hyperparameter by showing the Performance vs. KL-Divergence curve.", "questions_for_authors": "1.\tAs discussed earlier, could the authors show the Performance vs. KL-Divergence to validate their hyperparameter searching?\n2.\tCould the authors show their method on non-math related tasks (Alpaca Eval, Arena Hard)\n3.\tAlso, could the authors compare their method to SimPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "ngz7eHRQlp", "reviewer": "Reviewer_591D", "summary": "**Post-rebuttal Update**: I thank the authors for their detailed explanation on the unified framework. I have adjusted my score to 3.\n\n---\n\nThe paper presents a novel framework called Prior-Informed Preference Alignment (PIPA), which unifies various preference optimization objectives for language model alignment. The authors propose two variants, PIPA-M and PIPA-N, both formulated as constrained maximum likelihood estimation (MLE) problems with prior constraints. The PIPA framework effectively generalizes existing methods like Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) by incorporating prior-informed constraints. Empirical results on GSM8K and MATH benchmarks demonstrate that PIPA achieves consistent performance gains without additional computational costs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for acknowledging our innovative framework and thorough experiments for mathematical tasks.\n\n## 1. Unified derivation of PIPA-M and PIPA-N\nBoth PIPA-M and PIPA-N derive from the same MLE target, differing only in their prior assumptions. For data (x, y, c) where:\n- x: question\n- y: answer\n- c: label (1=pos, 0=neg)\n\nWe want to use MLE to solve $p^{\\theta}(c | x,y)$. By Bayes’ rule:\n$$p^{\\\\theta}(c=1 | x, y) = \\\\frac{p^{\\\\theta}(y|x, c=1) \\\\cdot p^{\\\\theta}(c=1|x)}{p^{\\\\theta}(y|x)} \\quad \\\\text{and} \\quad p^{\\\\theta}(c=0 |x, y) = 1-p^{\\\\theta}(c=1 | x,y)$$\nNow how to parameterize the 3 terms in $p^{\\\\theta}(c=1 | x, y)$? In both PIPA-M and N, the 2 terms in numerator are learnable:\n- $p^{\\theta}(y|x, c=1)$ corresponds to $f^{\\theta}(y|x)$ in our paper\n- $p^{\\theta}(c=1|x)$ corresponds to $g^{\\theta}(x)$ in our paper\n\nThe key difference lies in how we handle the denominator $p^{\\theta}(y|x)$:\n- *PIPA-M*: Uses a fixed prior for the margin distribution: $p^{\\theta}(y|x) = p^{prior}(y|x)$\n- *PIPA-N*: Uses a fixed prior for the negative conditional distribution. So we expand the marginal distribution to conditional distribution:\n$$\\\\begin{align}\np^{\\\\theta}(y|x) &= p^{\\\\theta}(y|x,c=1)p^{\\\\theta}(c=1|x) + p^{\\\\theta}(y|x,c=0)p^{\\\\theta}(c=0|x) \\\\\\\\\n&= f^{\\\\theta}(y|x) \\\\cdot g^{\\\\theta}(x) + p^{prior}(y|x) \\\\cdot (1-g^{\\\\theta}(x))\n\\\\end{align}$$\n\nThis concise derivation illustrates the relationship between PIPA-M and PIPA-N. **The reviewer questions the relationship between (2) and (5). We point out here that they are the same problem with different priors.**\nEquation (5) for PIPA-N is exactly the MLE above. The equivalence between the above MLE for PIPA-M and the KL objective (2) is from the marginal prior and is demonstrated in Lines 144-147 of our paper. In our next revision, we will improve the presentation to provide a more unified description of both approaches.\n## 2. Empirical understanding about performance difference\nAs mentioned in Lines 318-329, our empirical findings suggest different optimal scenarios for each approach. Below, we provide additional understanding for these observations:\n### 2.1 From the prior\n- **PIPA-N performs better for self-alignment**: When data is generated on-policy from previous model versions, our goal is self-improvement. In this context, it's reasonable to assume the current model $p^{prior}$ better resembles the negative-condition distribution.\n- **PIPA-M performs better under distribution shift**: When using data from external sources, the distribution shift makes it more appropriate to use the current model as a prior for the overall distribution rather than specifically for the negative class, which would be an aggressive assumption.\n### 2.2 From the loss\nA deeper analysis of the loss functions of positive samples also show the differences:\n\n**PIPA-M Loss:**\n$$-\\\\log\\\\frac{p^{\\\\theta}(y|x,c=1)p^{\\\\theta}(c=1|x)}{p^{prior}(y|x)}$$\n\nThis effectively maximizes $p^{\\\\theta}(y|x,c=1)$ and $p^{\\\\theta}(c=1|x)$ directly.\n\n**PIPA-N Loss:**\n$$-\\\\log \\left(p^{\\\\theta}(y|x,c=1)p^{\\\\theta}(c=1|x)\\right) + \\\\log \\left(p^{\\\\theta}(y|x,c=1)p^{\\\\theta}(c=1|x) + p^{prior}(y|x)(1-p^{\\\\theta}(c=1|x)\\right)$$\n\nPIPA-N incorporates an **additional regularization term** that prevents $p^{\\\\theta}(y|x,c=1)$ from overfitting to the training data. This regularization helps the model avoid getting stuck at the checkpoint that generated correct but suboptimal examples, which is particularly beneficial for self-improvement scenarios.\n## 3. General preference tasks\nPlease see Table 4, in Seciton 2 of the response to 3B8U.\n## 4. Why math tasks?\nSee Section 6 in the reply to Zrno.\n## 5. Why use DPO/KTO for baselines?\nDPO/KTO and their variants (iterative, stepwise, etc.) are widely acknowledged as **the primary—and often the only—baselines** in most alignment research on mathematical reasoning [1,2,3,4…]. We respectfully disagree with the reviewer’s suggestion to use RL methods like PPO/GRPO as baselines. These are **on-policy** algorithms, whereas the original DPO/KTO and our PIPA are all **off-policy** methods using a fixed dataset that differ only in their loss functions. Comparing off-policy methods with RL approaches would be inherently unfair. When comparing DPO/KTO in our work, we keep everything the same except the loss functions. A fair comparison with RL would require considering iterative/online DPO and developing an online version of PIPA, which extends far beyond the scope of this study.\n\n## 6.Typos\nWe will fix the typos in Line 140, 151.\n\n[1] Pang, Richard Yuanzhe, et al. \"Iterative reasoning preference optimization.\" NeurIPS 2023.\n\n[2] Chen, Guoxin, et al. “Step-level Value Preference Optimization for Mathematical Reasoning.” EMNLP 2024.\n\n[3] Wang, Huaijie, et al. \"Offline Reinforcement Learning for LLM Multi-Step Reasoning.\" \n\n[4] Xiong, Wei, et al. “Building Math Agents with Multi-Turn Iterative Preference Learning.” ICLR 2025.", "claims_and_evidence": "The authors claim that PIPA unifies multiple preference optimization approaches, generalizes existing methods, and improves performance. Theoretical analysis justifies the framework's design, while extensive experiments across answer-level and step-level tasks validate these claims.\n\nHowever, the observed performance difference between PIPA-M and PIPA-N lacks a consistent explanation. Neither can consistently surpass the other. This raises questions about the underlying motivations for these two variants.", "methods_and_evaluation_criteria": "The proposed methods are sound and align well with established evaluation criteria for preference optimization in language models. The authors provide a clear derivation of their approach and conduct comprehensive experiments on relevant benchmarks (GSM8K and MATH). \n\nThe choice to test on math benchmarks may limit the broader applicability of the proposed method. I wonder why the authors chose to test on math benchmarks, where a rule-based reward signal is available and there is no pressing need to construct preference signals. Alternatively, have the authors tested PIPA on alignment tasks, where the pairwise preference makes more sense?", "theoretical_claims": "The derivations appear correct to me.", "experimental_designs_or_analyses": "The experiments are comprehensive and address key aspects of preference optimization across different data types (paired/unpaired and answer/step-level). However, the comparison to DPO, IPO, and KTO in the step-level setting may be somewhat unfair since these methods are not inherently designed for multi-step math/reasoning tasks. A comparison to reinforcement learning (RL) methods would be more appropriate given the rule-based reward signals available for math tasks.", "relation_to_broader_scientific_literature": "None that I am aware of.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths:\n\n+ The proposed framework has the potential to offer a unified understanding of preference optimization techniques, which might be of independent interest.\n\nWeaknesses:\n\n+ The empirical distinction between PIPA-M and PIPA-N lacks a strong theoretical grounding or unifying motivation.\n\n+ Testing only on math benchmarks underutilizes the potential impact of PIPA on alignment tasks with human preferences.", "other_comments_or_suggestions": "typos:\n1. Line 140, left column: This is by definition is...\n2. Line 151, left column: we are going to parameter (parameterize?)", "questions_for_authors": "1. Can the authors provide a clearer explanation for the differing motivations behind PIPA-M and PIPA-N? How can these methods be unified under a common perspective?\n\n2.  There is an impression that PIPA-M and PIPA-N are quite different algorithms, and empirically there is no consistent advantage for one to be better than the other. Is there any more unified motivation so that PIPA-M and PIPA-N are motivated from the same starting point? Specifically, eqn (2) and (5) are not quite related to each other, except that they can fall under (1) as a constrained optimization problem.\n\n3. The answer-level setting is reasonable to me. I wonder why the authors chose to test on math benchmarks, where a rule-based reward signal is available and there is no pressing need to construct preference signals. Alternatively, have the authors tested PIPA on alignment tasks, where the pairwise preference makes more sense?\n\n4. For the step-level setting, the comparison to DPO, IPO or KTO might be slightly unfair as they are not originally designed for math or reasoning tasks. Instead, given the rule-based reward, comparison to RL algorithms is more suitable, suppose a process reward signal is available. And even if there is no process reward signal, the RL algorithms can still optimize on the rule-based outcome reward without any computation overhead.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UFqJ8FS6Sr", "reviewer": "Reviewer_Zrno", "summary": "The paper proposes a prior-oriented perspective on effectively leveraging negative samples in preference learning, given that MLE (SFT) is the optimal solution with positive samples. Within this perspective, DPO and KTO are unified by their use of different priors and loss functions for negative samples.\n\nBuilding on this framework, the authors introduce PIPA-N and PIPA-M as two design approaches, which naturally incorporate learning a value model for token-level valuation. Experimental results on paired/non-paired data, as well as outcome-level and step-level labels, demonstrate the effectiveness of the proposed methods\n\n---\n\nPost-rebuttal update:\n\nI thank the authors for their rebuttal, which addresses most of my concerns. I continue to recommend acceptance of this paper, as I find the proposed framework to be a valuable contribution, and the experimental results adequately support the claims. My remaining concern lies primarily in the clarity and writing quality of the paper. With significant improvements in presentation, I would give a score of 4. As it stands, please interpret my final score as a 3.5, if such granularity is permissible.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for acknowledging our novel well-supported framework and comprehensive experiments in math. We will improve writing to make presentation more clear.\n## 1. Why learnable $p^\\theta(c=1|x)$ is better?\nUnlike DPO (which uses a simple 0.5 prior) and KTO (which uses a complex KL-based prior), our approach makes this term learnable—a more elegant solution. As explained in our response to reviewer 4Kmz (section 1.3), this term can be interpreted as problem difficulty and is better learned during training rather than using a predetermined fixed value.\n## 2. Meaning of $y_{<t}$\nWe agree that conditional independence only works for math-like tasks with clear good/bad examples. However, for scenarios involving good/better comparisons you mentioned, a DPO-style paired loss function is necessary rather than the current KTO-style unpaired approach presented in our work, and it also means there is no step-level signals due to the uncertainty. \nTo get the new algorithm in the new setting, we can follow Theorem A.1's derivation and design a paired PIPA variant (not explored in our current version which focuses on KTO-style unpaired PIPA). Starting from equation (9), we can expand terms in $h_\\theta$ autoregressively while directly learning $p^{\\theta}(c=1|x)$. \nThis deriviation does not need token-level labels, avoiding the concerns you mentioned.\n## 3. Writing improvements and clarifications\nThank the reviewer for the careful reading and suggestions to improve our work. We will\n- Explain $g_0^\\theta$ more clearly in Thm 2.1.\n- Clarify the setting in Table 2.\n- Change Parameter to parameterize in Line 152.\n- Make notation with $\\theta$ more clear.\n- Reorganize the sections and move some experiments to appendix to make presentation more clear.\n\nDataset used in Figure 3: The likelihood is computed on Alphamath during training.\n## 4. Experiments\n### 4.1 Source of advantages\nOur approach gains benefits from both formula and token-level rewards—these components work effectively and neither functions effectively without the other. Importantly, integrating token-level rewards into DPO/KTO frameworks is non-trivial and it’s not straightforward to design an algorithm that only employs token level rewards without appropriate changes to the formula.\n### 4.2 Baselines\nWe don't include comparisons with RTO and OREO because they require an additional learnable model, unlike our approach which only needs an extra prediction head. This makes our method more directly comparable to the original DPO and KTO implementations with similar computational costs. We add the comparison with TDPO below using Deepseek model, and PIPA is better. More comparison will be included later due to the limited response period.\n\n| Algorithm | Learning Rate | $\\beta$ | Batch Size | GSM8K Accuracy | MATH Accuracy |\n|-----------|---------------|------|------------|----------------|---------------|\n| TDPO | 1e-5 | 1.0 | 256 | 76.15 | 48.84 |\n| PIPA-M | 5e-5 | NA | 256 | 79.08 | 50.82 |\n\n## 5. Encompassing other methods\nThe answer is yes. In paired PIPA, equation (9) in Theorem A.1 can be formulated as DPO plus a margin term structured as $\\log\\frac{p^\\theta(c=0|x)}{p^\\theta(c=1|x)}$, which enables us to cover margin-based approaches similar to SimPO. Additionally, by introducing a power prior into PIPA, we can encompass DPO variants with $\\beta\\neq 1$. See section 1.1 in the response to reviewer 4Kmz for details.\n## 6. Why math tasks?\nHere we provide further explanation on our choices of using math tasks in this paper.\n### 6.1 Math is a natural fit\nGiven that our PIPA framework provides key advantages—such as removing the requirement for paired data and naturally extending to step-wise labels—we prioritize math tasks as the optimal testbed for validation. This enables us to benchmark PIPA against general baselines like DPO and more specialized approaches such as step-DPO and step-KTO, which are designed to tackle these challenges.\n### 6.2 Easy generalization to other tasks\nWe appreciate the suggestion to expand the scope of our study. Our PIPA framework is applicable to general preference tasks, particularly through the paired PIPA formulation outlined in Theorem A.1 of our paper. Preliminary results are presented in Table 4 in response to Review 3B8U. Further exploration about paired PIPA and other tasks will be explored more in future work.\n### 6.3 The importance of paired data in math\nWhile some math benchmarks use rule-based rewards, a preference signal that distinguishes correct from incorrect answers remains crucial. It helps the model recognize and learn from flawed reasoning or incorrect steps within the same problem [1, 2, 3…]. The impact of paired data is more pronounced when high-quality paired data is available. Thus, when designing alignment algorithms, it is essential to consider both paired and unpaired data. PIPA serves as a promising step in exploring this direction. See more discussions in Section 4.2 in the reply to 4Kmz.", "claims_and_evidence": "The claims are generally well-supported. However, there are concerns regarding which specific design choices contribute to the observed experimental results and the comparison to other baselines. Please see the #2 and #4 question on the experimental analysis.", "methods_and_evaluation_criteria": "The methods are generally well-founded. However, the reviewer has the following question on design choice in PIPA-N and the token-level reward:\n\nIn PIPA-N, why is learning $p^{\\theta}(c=1|x)$ considered a simpler and more natural design compared to DPO and KTO? A more detailed explanation would help clarify the advantages of this approach over existing methods.\n\nThe token-level reward is defined by the equation at line 235. If $c$ epresents correctness, then the definition is well-defined. However, when $c$ represents preference, what does it mean in the context of $y_{<t}$?", "theoretical_claims": "It would be helpful to provide a more detailed explanation of the reduction after Theorem 2.1, particularly why $g_{0}^{\\theta}$ disappears from the objective. The reviewer understands that $g^{\\theta}$ is introduced to satisfy the constraint and is inherently not a variable to optimize, but this could be made clearer to the reader.", "experimental_designs_or_analyses": "1. Table 2 lacks a clear explanation of the experimental settings. The reviewer has to guess that this is the DeepSeek model and the step-wise approach for unpaired data. \n\n2. Table 2 raises a question about the source of performance improvement. Does the observed improvement primarily stem from the learned token-level values, rather than modifications in the PIPA formula? The results with a fixed $p^{\\theta}(c_t|x,y_t)$ appear similar to step-KTO, suggesting that the gains might not be attributed to changes in the formula itself. \n\n3. Figure 3: On which dataset was the likelihood calculated?\n\n4. The PIPA methods apply token-level rewards, whereas the baseline methods rely on outcome-level or step-level rewards. How does PIPA compare to other approaches that also utilize token-level rewards within the DPO framework (e.g., RTO, TDPO, OREO)?", "relation_to_broader_scientific_literature": "The paper aims to unify offline preference alignment algorithms from a prior-oriented perspective. It introduces a general framework that encompasses DPO and KTO, simplifying them into practical algorithms that outperform previous methods.\n\nAdditionally, the paper provides intuitions on the distinction between SFT and preference alignment, framing it as the incorporation of a prior on negative responses.", "essential_references_not_discussed": "To the reviewer’s knowledge, all significant references have been appropriately discussed.", "other_strengths_and_weaknesses": "The prior-oriented perspective is novel and effectively illustrates the distinction between SFT and offline preference alignment methods.\n\nThe definition and introduction of the token-level reward is novel.", "other_comments_or_suggestions": "1. Line 152 (left): The sentence is not fully clear. Should \"Paramter\" be \"parameterize\"? What does this sentence meant to say?\n\n2. The notation $p^{\\theta}$ and $g^{\\theta}$  both include $\\theta$, which is confusing before reaching Section 2.6.", "questions_for_authors": "After reading the paper, the reviewer feels that the authors attempt to cover too much content within the main text. The paper would benefit from moving some ablation studies to the appendix while expanding the explanations in Sections 2.2, 2.3, and 2.4 for better clarity.\n\nAdditionally, could the proposed framework also encompass other offline preference learning methods discussed in the related work section?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GG4r7sgEDw", "reviewer": "Reviewer_4Kmz", "summary": "The submission presents an approach called Prior-Informed Preference Alignment (PIPA) that accommodates paired and unpaired preference data, as well as answer and step-level annotations, for the purpose of doing offline preference alignment in language models. The approach is claimed to unify recent approaches like DPO and KTO into an RL-free probabilistic framework.\n\nPIPA starts by assuming an (unpaired) preference dataset made of triplets $(x, y, c)$ sampled from $p^{data}(x, y, c)$, where $x$ is the instruction input, $y$ is the model response, and $c$ represents whether the response was desirable or not. It then formulates the preference learning problem as minimizing the discrepancy between the model's distribution and the data distribution subject to some prior constraint on the model's distribution. The nature of the constraint gives rise to different PIPA variants.\n\nPIPA-M attempts to model the conditional distribution $p^\\mathrm{data}(y, c \\mid x)$ under the constraint that when marginalizing over $c$ the equality $p(y \\mid x) = p^\\mathrm{prior}(y \\mid x)$ (the latter being defined through a pretrained LLM) holds. It introduces two parameterized functions $f^\\theta(y \\mid x)$ and $g^\\theta(x)$ to represent the conditionals $p(y \\mid x, c = 1)$ and $p(c = 1 \\mid x)$, respectively. The constraint is enforced by construction by i) defining $p^\\theta(y \\mid x, c = 1) = f^\\theta(y \\mid x) g^\\theta(x)$ and $p^\\theta(y \\mid x, c = 0) = p^{prior}(y \\mid x) - f^\\theta(y \\mid x) g^\\theta(x)$ and ii) enforcing that $p^\\theta(y \\mid x, c = 0) > 0$  via $g^\\theta(x) = \\min(g^\\theta_0(x), p^\\mathrm{prior}(y \\mid x) / f^\\theta(y \\mid x))$.\n\nPIPA-N instead attempts to model the conditional distribution $p^\\mathrm{data}(c \\mid x, y)$ under the constraint that $p^\\theta(y | x, c = 0) = p^\\mathrm{prior}(y \\mid x)$. The parameterized $p^\\theta(c = 1 \\mid x, y)$ is constructed using this equality constraint and conditional probability identities, and the counterpart for $c = 0$ is obtained by noting that the two must sum up to 1 since $c$ is binary.\n\nThe step-level variants for PIPA-M and PIPA-N are defined analogously, with the difference that the functions $f^\\theta$ and $g^\\theta$ are defined autoregressively.\n\nExperiments are presented for all combinations of {paired,unpaired} preferences and {answer,step}-level annotations using an unpaired preference dataset for math reasoning that includes problems from GSM8K and MATH. PIPA is compared against DPO, KTO, and IPO as well as the step-level variants of DPO and KTO and is shown to outperform competing approaches in all settings. Ablations are presented to compare PIPA-M and PIPA-N, examine the effect of several design decisions, and investigate the effect of step-level annotation.\n\n## Update after rebuttal\n\nThe discussion with the authors addressed my concerns regarding the soundness of the proposed approach, and I feel more inclined to recommend acceptance. I have adjusted my score accordingly. I remain worried about clarity, but the authors' efforts in the discussion alleviate that to some extent.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the detailed summary of the PIPA-M and PIPA-N derivations, including their extension to the step-level setting, as well as for recognizing our comprehensive experiments on math tasks.\n## 1. Connectio between PIPA and DPO\n### 1.1 $\\beta$ \nThe reviewer mentions that our current PIPA-N only reduces to DPO when $\\beta=1$. We emphasize that DPO with $\\beta \\neq 1$ directly corresponds to PIPA-N with a power prior [1] in Bayesian analysis, meaning:\n$$p(A|B) \\propto p(B|A)^{\\beta} p(A).$$\nSpecifically, introducing $\\beta$ modifies the original DPO loss as follows:  \n$$\\max_\\theta  \\quad \\log\\sigma\\left(\\beta\\log \\frac{p^{\\theta}(y^+ | x, c = 1)p^{\\text{prior}}(y^- | x)}{p^{\\theta}(y^- | x, c = 1)p^{\\text{prior}}(y^+ | x)}\\right)\n= -\\log\\left(1+\\left(\\frac{p^{\\theta}(y^- | x, c = 1)p^{\\text{prior}}(y^+ | x)}{p^{\\theta}(y^+ | x, c = 1)p^{\\text{prior}}(y^- | x)}\\right)^\\beta\\right).$$\nFurthermore, when incorporating the power prior $\\beta$ in PIPA-N, all terms of $h_\\theta(y_1,y_2,x)$ in equation (9) transform into $h_\\theta^{\\beta}(y_1,y_2,x)$. It is straightforward to verify that the updated PIPA-N loss with $\\beta$ precisely matches the DPO loss with $\\beta$.\n### 1.2 Sampling\nWe respectfully disagree with the reviewer's interpretation and emphasize that our PIPA process aligns with practice. As the reviewer noted, an answer $y$ can be classified as positive ($c=1$) or negative ($c=0$) based on its comparison with the paired answer. This implies the existence of a **non-trivial joint distribution** over $(y,c|x)$. For a high-quality response, $p(y, c=1|x)$ will be greater, while $p(y,c=0|x)$ will be higher for a less favorable response. In practice, we can model positive samples as being drawn from $p(y|x,c=1)$ and negative samples from $p(y|x,c=0)$, allowing us to apply MLE to address the problem.\n### 1.3 Meaning of $g^{\\theta}(x)$\nThis represents the marginal distribution $p^{\\theta}(c=1|x)$, which computes integral over all possible answers $y$. Empirically, it can be interpreted as **the difficulty of a question**. For example, if the question $x$ is \"What is 2+2?\", then $g^{\\theta}(x)$ should be expected to be close to 1 since there's clear consensus on the correct answer. However, if the question is \"What is the most beautiful color?\", then $g^{\\theta}(x)$ would be closer to 0.5 since this is highly subjective with no definitive answer.\n## 2. Experiments\n### 2.1 Paired data construction\nWe respectively disagree with the reviewer’s concern about pairing correct and incorrect answer together does not help learn the preference. \n- Paired data is standard and widely used in SOTA math models training, e.g., [2]. These paired data may start from similar analysis but suddenly diverge in certain steps, which help the model learn the key steps from comparison. \n- Our experiments about win rate and KL trajectory (https://anonymous.4open.science/r/PIPA-DB7B) also clearly shows that our DPO training is effective. \n- Practical paired data construction should rely on the goal. For math, if the goal is to enhance reasoning ability, pairing correct and incorrect answers should be effective. And if the aim is to achieve an efficient and short reasoning chain, pairing a longer correct answer with a shorter correct answer is preferable. While pairing two incorrect answers is theoretically possible, it serves limited practical utility in most training scenarios. For general preference alignment, we also don’t think pairing two bad answers (bad versus worse) is helpful. We believe robust filtering mechanisms should be implemented to avoid such non-productive training signals.\n### 2.2 Hyperparameters\nOur hyperparameter choice is based on rigorous selection to ensure fair comparison across all methods. \n- **Batch size**: Since PIPA only modifies the loss function compared to baselines, **a fair comparison would be keeping the batch size and training steps to be the same across all algorithms.** And 256 is a standard choice  for datasets containing tens of thousands of examples.\n- **$\\beta$**: See the results in Section 1 of response to 3B8U.\n### 2.3 Data split\nWe use standard benchmarks (GSM8K, MATH) for testing and Alphamath for training (10% for validation). LR search follows standard paradigm using train loss and validation loss.\n### 2.4 Results differences\nThey are from differences in models and training data.\n## 3. Writing Improvements and Clarifications\nWe thank reviewers for their careful reading and suggestions. We will:\n- Add Dumoulin et al. (2024) to the related work section.\n- Clarify PIPA's approach for paired data earlier.\n- Clarify the training objective explanation.\n- Clarify substitutions in Equation 3 and step 3.\n- Change minimizing to maximizing.\n- Remove the 0 in Equation 5.\n\n[1] Ibrahim J G, et al. The power prior: theory and applications[J]. Statistics in medicine, 2015.\n\n[2] Xiong, Wei, et al. “Building Math Agents with Multi-Turn Iterative Preference Learning.” ICLR 2025.", "claims_and_evidence": "I am not entirely convinced by the connection drawn between PIPA and DPO. Firstly, Equation 6 in Appendix A.1 represents the DPO objective only in the special case where $\\beta = 1$. Secondly, and more importantly, the chosen and rejected responses are characterized as having been sampled from $p(y \\mid x, c = 1)$ and $p(y \\mid x, c = 0)$, respectively, but this is not how the data generating process works in practice for paired preferences: two responses are sampled, and the annotator makes a relative judgement on the two responses. This means that response $y$'s categorization into $c = 0$ or $c = 1$ is intrinsically linked to the second response $y'$ in the pair that was presented to the user: response $y$ could be bad, but so long as response $y'$ is _worse_, $y$ would be associated with $c = 1$.", "methods_and_evaluation_criteria": "The proposed approach makes sense in how it is constructed: having verified the derivations on my own for Sections 2.2 and 2.4, the objectives are sound and follow naturally from the definition of conditional probabilities.\n\nOne aspect of PIPA that makes less sense to me is what $g^\\theta(x)$ is meant to represent. A literal interpretation would be that it is the probability of a correct answer given an instruction $x$, but that has to depend on how answers are generated from the instruction, right? If so, what generative process do we assume?", "theoretical_claims": "I have not checked the proofs in the Appendix, but I did verify for myself that the objectives constructed in the main paper correctly follow from first principles of conditional probabilities.", "experimental_designs_or_analyses": "The dataset used to compare approaches makes sense in the unpaired case, but I don't think creating paired data out of correct and incorrect responses to problems is a good stand-in for a paired preference dataset (see my point about the paired/unpaired correspondence in Claims and Evidence). Given the clear dichotomy between \"correct\" and \"incorrect\" in the mathematical domain, such a procedure cannot create paired data where two responses are satisfying but one is better than the other (e.g., both are factually correct, but one is more detailed than the other), or two responses are bad, but one is worse than the other (e.g., both are factually incorrect, but one uses toxic language). It's unclear to me how PIPA's way of adapting to paired data would fare in that case in comparison to approaches specifically designed for paired data. A comparison on a paired dataset would be necessary to determine that.\n\nI am also concerned that the model selection procedure is not sufficient to ensure that each method is presented in the best light possible. Are we sure that some methods wouldn't benefit from smaller or larger batch sizes than 256? Are we sure that some methods wouldn't benefit from a smaller or larger number of parameter updates? Why was beta chosen to be 0.1 for DPO, IPO, and Step-DPO? Do we know that this is a good hyperparameter choice for AlphaMath?", "relation_to_broader_scientific_literature": "In general the submission does a good job of relating to the broader scientific literature. Dumoulin et al. (2024) is cited when discussing DPO's probabilistic interpretation, but is not mentioned among the works that \"[approach] alignment from a probabilistic perspective\"; why is that?", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "The submission presents interesting ideas in its proposed approach.\n\nOne major weakness is its clarity. I think I now understand the ideas it tries to convey, but I had to spend a lot of time connecting the dots and navigating between sections before reaching that point. A few notable examples:\n\n* The introduction sells the proposed approach as being applicable to both paired and unpaired data, but the preference alignment problem formulation in Section 2.1 describes the unpaired case specifically without going over the paired case. How PIPA handles paired data is only made clear in Section 3.1: \"Both KTO and PIPA decouple the paired data\"; \"[...] we construct the paired subset by matching each correct solution to a corresponding incorrect solution from the same problem [...]\". From this I infer that adapting PIPA to work on a paired dataset amounts to treating all chosen responses in the comparison pairs as having been classified as \"correct/desirable\" and all rejected responses as \"incorrect/undesirable\". This should be clarified sooner in the paper. (I also have my doubts about the validity of this adaptation; see my point in Claims and Evidence, but essentially if two responses are bad but one is less so than another, does it even make sense to say that it is \"correct/desirable\"?)\n* The notation also contributes to the confusion: Section 2.1 talks about \"estimating $p(y | x, c = 1)$\", which suggests that this is the target distribution we are trying to approximate, yet the preference dataset is sampled from $p^\\mathrm{data}$, and MLE usually attempts to approximate the distribution from which the empirical data is sampled. Do the authors mean that $p(y | x, c = 1)$ is the distribution with which we try to approximate $p^\\mathrm{data}$? This is what Equation 1 suggests, but it should be clarified.\n* The connection between PIPA-M's Equation 3 and Step 3 in Algorithm 1 is not immediately obvious. If I understand correctly, since $p^\\mathrm{prior}$ does not depend on $\\theta$, the derivative with respect to $\\theta$ for $c = 0$ and $c = 1$ in Equation 3 are identical to that of step 3 in Algorithm 1, which is why the latter is substituted for the former to unify PIPA-M and PIPA-N in Algorithm 1. Is this correct? In any case, the substitution should be explained more clearly.", "other_comments_or_suggestions": "* In Section 2.2's last unnumbered equation, doesn't the optimization reduce to _maximizing_ the expected log-likelihood rather than minimizing it?\n* What does the subscript 0 in $p^\\mathrm{prior}_0(y \\mid x)$ mean in Equation 5?", "questions_for_authors": "1. How was the dataset partitioned for training, validation, and testing? What criterion (and what dataset split) was used to determine the best learning rate in the grid search?\n2. The KTO paper also presents results on GSM8k which appear to be different from the ones presented in the submission. What explains this discrepancy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Prior-Informed Preference Alignment (PIPA), a probabilistic framework for offline preference tuning of language models. PIPA is an intriguing concept, and most claims are well-supported by clear theoretical derivations. Experimental evaluations on GSM8K and MATH show that PIPA consistently outperforms baseline models. Overall, this is a good work. To enhance the final paper, the authors should improve the presentation clarity, include results beyond math benchmarks, and provide more discussion on the distinctions between the two model variants.
Decision: Accept (poster)