Understanding the Logic of Direct Preference Alignment through Logic
Authors: Kyle Richardson
Vivek Srikumar
Ashish Sabharwal
Venue: ICML 2025 poster (2025)
Keywords: preference learning
neuro-symbolic
logic
reasoning
model alignment
Primary area: deep_learning->algorithms
TL;DR: Formal symbolic characterizations of DPO and preference learning.

Abstract: Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic program that characterizes its semantics? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.

Understanding the Logic of Direct Preference Alignment through Logic
Kyle Richardson
Vivek Srikumar
Ashish Sabharwal
Abstract
Symbolic Programs
Loss functions
Decompilation
Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human prefer- ences. While this has motivated the development of many new variants of the original DPC loss, understanding the differences between these re- cent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for rea- soning about the underlying semantics of these algorithms. In this paper, we attempt remedy this by formalizing DPA losses in terms of dis- crete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive symbolic program that characterizes its semantics? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and iden- tify symbolic forms for number of commonly used DPA variants. Further, we show how this for- mal view of preference learning sheds new light on both the size and structure of the DPA loss land- scape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those work- ing on human AI alignment.
DPO loss
Impliea(
And0Mt-ow AnndM-szhRRetlgl
log
log,
log, solatz
Modify
DPO suriant
Impliest And Re 2h 8057))
llog
log calou bradee seluda) iolal (255-1))
Compilation
Semauties
Puro F=
Poecz
Figure I. Can we uncover the hidden logic of DPO? Here show the decompilation of the DPO loss into a symbolic expression that expresses its high-level model behavior, along with semantically modified version that we can compile into a novel DPO variant. We study how translate between these two spaces better un- derstand the semantics of existing preference lcarning algorithms and to derive new ones from first principles.
reasoning that are driven by declarative knowledge, in sharp contrast to purely machine learning-base approaches, have the advantage of allowing us to reason transparently about the behavior and correctness of the resulting systems. In this paper we focus on the broad question: Can the declarative approuch be leveraged to better understand and formally specify algorithms for large language models (L.Ms)?
We specifically investigate direct preference alignment (DPA) algorithms, such as direct preference optimiza- tion (DPO, Rafailov et al., 2023), for pairwise preference learning, which are currently at the forefront of research on LLM alignment and learning from human preferences (Ouyang al., 2022; Wang et al., 2023). While there has been much recent work algorithmic variations of DPO (Azar et al., 2024; Hong et al., 2024; Meng et al., 2024) that modify or add new terms to the original loss, understanding the differences between these new proposals, as well as com- ing up with new variants, remains a formidable challenge due to the lack of conceptual and technical framework for reasoning about their underlying semantics.
1. Introduction
Symbolic logic has long served as the de-facto language for expressing complex knowledge throughout computer sci- ence (Halpern et al., 2001), including in AI (McCarthy et al., 1960; Nilsson, 1991) and early ML (McCulloch Pitts, 1943), owing to its clean semantics. Symbolic approaches to
Our study attempts remedy this problem by formalizing the corresponding loss functions in terms of logic, trying to answer the question: Given an existing loss function, such as DPO (see Figure 1), can we derive a symbolic expres- sion that captures the core semantics of that loss function (i.e., one that we can then systematically compile back into
1 Allenn Institute for Al University of Utah. Correspondence to: Kyle Richardson Shyler@allenaa.orgg.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Understanding the Logic of Direct Preference Alignment through Logic
exactly that same loss}? By mapping loss functions dis- crete reasoning problems ones that abstract away from lower-level optimization details and reveal high-level model behavior - we can study them using conventional semantic notions from logic (e.g., entailment), relate them semanti- cally to other algorithms, or even modify their underlying logical semantics to derive entirely new algorithms. For this formalization, we devise a novel probabilistic logic based on generalization of the notion of semantic loss (SL., Xu et al., 2018) coupled with provably correct mechanical procedure for translating DPA losses into programs in our logic. As in SL, losses are produced from symbolic pro- grams by counting the weighted propositional models of those programs, reducing the problem to one of probabilistic inference (Chavira & Darwiche, 2008). In contrast to the kinds of symbolic programs commonly used with SL, how- ever, empirically successful DPA losses impose systematic conditional constraints on the types of models that should be counted, which shape the structure of the underlying proba- bility distribution. We express these constraints through a new primitive called a preference structure that addresses the technical issues involved with modeling pairwise pref- erence symbolically. Via such constraints, certain semantic relationships between existing losses can be casily observed and new losses can be derived.
2023) and slic (Zhao et al., 2023) that use closed-form loss functions tune models directly offline preferences.
We touch on two recent areas: formal characterizations of DPA losses (Azar et al., 2024; Tang et al., 2024; Hu et al., 2024) and work on devising algorithmically enhanced variants of DPC (Amini et al., 2024; Ethayarajh et al., 2024; Park et al., 2024). In contrast to the former, which focuses on the optimization properties of DPA losses, we attempt to formally characterize the semantic relationships between these variants of DPO in optimization agnostic way to better understand the DPA loss landscape.
Neuro-symbolic modeling, We take inspiration from work on compiling symbolic formulas into novel loss functions (Li et al., 2019; Fischer et al., 2019; Marra et al., 2019; Asai & Hajishirzi, 2020, inter alia). We focus particu- larly on approaches based on probabilistic logic (Manhaeve et al., 2018; Ahmed et al., 2022; 2023a;b; van Krieken et al., 2024b; Calanzone et al., 2025), yet our study differs in focus- ing on the inverse problem of decompilation (see Friedman et al. (2024)), or deriving symbolic expressions from known loss functions (see Appendix A for more related work).
3. Direct Preference Alignment
We study offline preference alignment, defined as follows: given data Dp {(x(0) yiu Vi ) M model input and two possible generation outputs, a pre- ferred output you (the winner w) and a dispreferred output Ii (the loser 1), the goal is to optimize a policy model (e.g.. an LLM) y ~ TT(- x)) to learn such preferences.
Our formal view of preference learning sheds new light on the size and structure of the DPA loss landscape. Under modest assumptions motivated by the structure of existing DPA losses, we find that the number of definable preference structures is doubly exponential in the number (n) of unique predictions (i.e., forward model calls) made in loss func- tion, or 42". This results in an upper bound of 4.3 billion definable DPA losses that are variations of the original DPO loss, leaving much room for exploration. While huge, our semantic characterization of the losses in this space also reveals an interesting lattice structure: losses are connected via semantic relations ge. logical entailment and equiva- lence) well as monotonicity properties in the loss space.
consisting of a
We focus on direct preference alignment (DPA) ap- proaches that all take the form of a closed-form loss function that we can use to directly train a model Ty on Dp. Since our study focuses the formal properties of DPA losses, it is important to understand their general structure, which will take the following form from Tang et al. (2024):
[s(Oo(s. yur u).
These formal results also provide practical insights into effectively searching for new DPA losses. For example, one can start with empirically successful loss functions, use the formalization to understand their semanties, then modify their semantics to arrive at novel variants (e.g., more constrained ones), then evaluate. We report on small-scale case study demonstrating the fcasibility of this approach, motivating an exciting avenue for future work.
fDD D) =
E (st (R. )D,
consisting of convex loss function f R x R+ + R. a differentiable model quantity Pe(x: Bw- Vt), which we ab- breviate to pa, and a parameter 3 for scaling terms in Pa.
Table 1. Examples of some DPA loss functions (Eq 1) with differ- ent choices of convex function f and model quantity pa-
f(po, 19)
Pg
2. Related work
DPO
log 0 (3po)
log "a(uula) Thel
log
de (wr [ae]
IPO (po )12²
( [x)
Language model alignment. While traditional approaches to language model alignment have employed reinforcement learning (Ziegler et al., 2019; Christiano et al., 2017), we focus on DPA approaches such DPO (Rafailov et al.,
Slic
max(0, 13 3 ppo)
log To
xa [=)
RREF
max(0, -pa)
log Se( (m) lsw
(v (z)
2

---

Understanding the Logic of Direct Preference Alignment through Logic
$4
Table How are variants of DFO structured? We define popular variants in terms of their core loss equation ps and the helper fune- tion mug (y1; 32) (last column) that rewrites each pe in a way that brings out general shared structural patterns and added terms compared with the log win/loss ratio se (yw,,)) All original losses are implemented via the logistic log loss: &x log or (3po).
compilation
decompilation
Input Loss LoRPo ($3)
Preference structure P ($5)
Pr implies(
logg log (3ull)))
Mix,y Mixy
Odd5a (yu 13
Pa :- (0g0 Myy)))
Pg Loss Pe log
8m (mma) (1), := log P'm, (wn Pmn (x)
Algorithm ($5.2)
Pg
Baselines po
Exx log 1- Paliu [n]
(oxun log
Palaw Bow a1 Pa an (a))
(r Pofale)) -M
SuM(pL) M(x.ya) A -M0x SEM(pL) M(x. A A-M(x.yw)
PPlPa [=111 Palon w]]]]
Single model approaches (no reference) Po
Esco
log Polzules (+)
Potugla)
so (yu, 3c)
Core loss equation (Table 2) Semantic translation (Table 7)
lowrs log (2) Pol-PPPpo Pa (2))
Patsu Pa
so (tw,, 36)
(SINOO log
Paltu Puarrrrl [=)
80 80( 9x)
-mareel (Nwr: U1)
Figure 2. How do we map losses to discrete reasoning problems? Our approach and key results (by paper section $). First an input loss (upper left) is stripped down to its core loss equation (lower left), then semantically translated (lower right) and mapped into a semantic structure (upper right) that is provably compilable (under novel logic, $5) back into the original loss (Thm 1).
Fuorlyu |a Pay
with reference model Piet
Esso log Folpu Paa
so (yer, Bc)
SSee( (Uw,, Me)
tTFO log Palw 2 Pa)lll))
8a sa(yer, 9t)
- Seer(B Mit)
FarlDr/))) (w) an (n)
S102,,22 (Nor Mex)
Table 1 lists four specific DPA losses: DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), Slic (Zhao et al., 2022; 2023), and RRHF (Yuan et al., 2023) that differ in f or pa- Here the logistic log loss (with a (a)) = I=exp(-a)) sqqure loss, hinge loss, and perceptron loss are used for f. slic ic and RRHF are examples of single model approaches that define P9 in terms of the log ratio of the winner and loser given prediction probabilities of the model being trained To. The prediction probabilities are sometimes computed using length normalization (i.e., taking a geometric mean of token probabilities) as shown for RREF, Single model losses are usually regularized using an added cross-entropy term, which we exclude from our formal analysis.' For DPC and IPO, in contrast, the model quantity po is the log ratio difference (of the winner and the loser) between the predictions of the model being trained and frozen LLM called reference model, Tee.- These two approaches constitute a two model approach, where the role of the reference model is to avoid overfitting (controlled by 3).
tails about f. 3, length normalization). Next we discuss these core loss equations and the general structure of the DPA losses (Table 2) that we aim to derive formally.
Core loss equations. Table 2 shows the different variants of DPO we investigate and two common baseline losses from Rafailov et al. (2023) - the cross-entropy loss tas and a variant that uses an unlikelihood term (Welleck et al., 2019) fCEual - all using a uniform notation for pg. We use Pmn (y x)) in place of am (y x:) denote the probability assigned by model to output y in way that is agnostic to length normalization." 2 Importantly, we express each pa as a single log ratio log pà/ pp called the core loss equation, which is the starting point of our analysis.
Under these core loss equations, we see that DPO variants share much structure, which is further brought out via the log ratio function sm (y1, 92) defined in Table 2 (using y to denote the negation of y. or 1 - Pm x)). Specifically, we see that all losses are derivable from the log ratio of winner and loser se( M1) used in SliC either exactly, as in CPO (Xu et al., 2024), or with additional log terms. DPO, for example, is expressible as this ratio minus an additional log ratio term (y M1) that contains information about the reference model. Many variations of DPC specifically involve making the following two modifications:
Conceptually, preference losses involve making predictions about winners and losers across models and reasoning about the relationships between predictions. Our main question is: If we view this process a discrete reasoning prob- lem, what is the nature of the reasoning that underlies these different losses and each model quantity po? As shown in Figure 2, our analysis operates at the level of po and starts by rewriting each loss into a core loss equation that strips away oppimization/mmpeeennttioo details (e.g., de-
1. Adding additional terms. Approaches like ESpO (Pal et al., 2024) (see also Amini et al. (2024); Park al. (2024)) incorporate additional terms into DPO (Sref2, (Nos Bro see Appendix H) that address specific failure cases.
When referring to the CPO, ORPO, and S1iC losses, we re- fer to the losses without their original cross coossyy terms. For example, what we call $l C and ORPC refers to the cal 1 and OR losses, respectively, in the original papers. See Appendix B for details of the original losses and our generalization.
AA in Zhao al. (2025), can define (y 2) := Tm x) 1i where 7 E {0, 1} (with r = employing length norm.)
3

---

Understanding the Logic of Direct Preference Alignment through Logic
(A) Example symbolic formulas
(B) Model output distribution
Model predicts loser
Model predicts winner
Po(y (x)
Whenever the model deems the loser (21) to be valid generation (i.e., above e), it should deem the winner (yw) to be yalid too.
Implies(
valid
winner
uucco
Mix,1 h M(xy)>
loser x
not vid
And( M(x,yx) )
The model should deem the win- ner to be valid and the loser to be not valid (i.e., below e).
Rexun1
Not(Mox)))) ))
y e Σ"
Figure 3. What do symbolic representations of loss functions tell us? (A) shows two symbolic formulas related to single model preference learning (involving an input (2, You (w,,k)) ()) with their semantics paraphrased in informal English. When grounded in model behavior, they tell us about the structure of the model's output probability distribution (B) and where predictions belong in that distribution (relative to some threshold e). We later show that these formulas correspond to luncpo (Figure 4-5) and the common baseline lewwn (Table 2).
2. Changing the reference ratio. No reference ap- proaches, such as LoRPo (Hong et al., 2024) and lainso (Meng et al., 2024), instead reparameterize the reference ratio Ser(YW 31) either in terms of some quantity from the policy model as in ORPO (so(Uw: Ui)) or a heuristic penalty term 7 as in SimPO. For SimPO we rewrite the y penalty term in terms of the ratio y = (y 31) (where 'mref' refers to a manually defined reference model simulating 7) in order to align its form with that of DPO as done by Zhao et al. (2025). For example, given any Y 0, ? Smert (Jw, 31) can be satisfied by setting P P3t .r)= = P (w / exp(1) as long as the prefer- ence pairs data does not contain transitive triples or cycles.
tic predictions XI,..., X,, that underlie symbolic formulas.
Relationships between predictions are expressed as sym- bolic formulas. Relationships between model predictions take the form of symbolic constraints expressed as formulas of propositional logic P defined by applying zero or more Boolean operators over probabilistic predictions. For exam- plc, in Figure 3 (A), the top formula, which we later show is fundamental to the semanties of many DPA approaches, uses the implication operator (Implies) to express the constraint that model M should never deem the loser y to be valid generation (M(x, yr)) without deeming the winner the to also be valid (M(x, yw.)). The bottom formula tells us that only the winner yho should be deemed valid, using the conjunction and negation operators (And, Not).²
While these approaches share a similar structure, understand- ing what these log ratios and extra terms in Pe semantically mean remains unclear, which is the topic we discuss next. While our techniques will cover both reference and no refer- ence approaches, due to their simplicity we use no reference losses such as Egmin) fopo. larpa and a novel loss Eyncro (defined later) as running examples throughout.
When grounded to model behavior via compilation (Sec- tion 4.1), such constraints tell us about the structure of model's output probability distribution, visualized in Fig- ure 3 (B). Semantically, we assume that a valid generation is any probabilistic prediction whose weight exceeds some threshold e in that distribution, similar to c-truncated sup- port in Hewitt et al. (2020). While our results later will not depend on making any direct assumptions about e, such a definition is merely meant to provide intuitions for how to understand our formulas. For example, M(x, yt) M(X, Yw)) dictates that whenever the loser (i.e., a point in Figure 3B) is found to be above the threshold the winner (another point) should also be above e. In other words, if the loser is deemed to be a valid generation, the winner should be too.
4. Preference modeling as a reasoning problem
To better understand the DPA loss space, we will formalize the preference losses and the model quantities/log ratios po in terms of symbolic reasoning problems. Conceptually this will involve the following core ideas and assumptions.
Model predictions are symbolic objects. The declara- tive approach involves treating LLM predictions (e.g., in Po) as logical propositions. For example, when a model M generates an output you for x, we will use M(x, ym) to ex- press the logical proposition that yo is valid generation for x. Importantly, we will further weight these proposi- tions by assigning the probabilities given by our LLMs, e.g., (M(M = Po(y x). We call these our probabilis-
Existing loss functions are expressíble as symbolic formu- las. We assume that the preference loss functions in Table 2 all have an internal logic that can be expressed in the form described above. Our goal is to uncover that internal logic.
We will switch between using conventional logical notation (e.g.. A, 7. -,,000 @) and operator notation (e.g.. And, Or, Not, Implies, XOR) depending on the context.
4

---

Understanding the Logic of Direct Preference Alignment through Logic
we consider the conditional semantic loss ((P) Pc, and show its full objective below:
M(X,Yol
M(X, yi)
ELaCRO
Eanro
Ecrc
T
T
V
X
PA
T
F
V
min I 8 di
log po (Pa Pc.))
(2)
T
X
X
F
WMC[PAPE:0)
with pp |
Pc)
WMC(PAPP::) +WMC -WMC (PPAPc:0) which
P
P
Pc
Implies(
M(x,y)) ) M(Xyw)
L = logo a
log Σ
follows from standard conditional probability.
Σ
As an important technical point, it is easy to see that we can rewrite the formula probability (for P # T) as WMC(P::0
Figure 4. Loss functions as truth tables, The Boolean semantics (top) our logic: V s correspond to propositional models of formulas P (or Py. Eq. 4 left), x s to tP-P (or y, Eq. right), blank cells to conditioning constraints Pc (and cells with multiple marks to PA in Eq. 4). Losses & (columns) are created/recov- ered by assigning marks (denoting formulas) then taking weighted model counts (WMC) of v and x (i.c., Σ v and Σ and computing the bottom equation (generalized from Eq. 3).
Po(P) = ( wMc(-P:0) ). yielding logistic log form of the semantic loss below that aligns with the structure of the DPA losses in Section 3. This relationship is key when translating, decompiling, DPA losses to symbolic forms:
WMC (P2:0)
}
E(P, 0, D) == E d=
log a
log WMC P.550)
(3)
sem.
loss ratio (am
4.1. Compilation and Decompilation
As an analog to pa (Table 2), we call the inner log ratio in at)) above the semantic loss ratio of P, or Psem (P)-
Deriving the formal semantics of losses involves understand- ing how to robustly translate between the loss space and the symbolic space. This is reducible the two problems we address in turn: compilation and decompilation (Fig. 1).
Decompilation into semantic loss. The input in set- ting is not formula P but particular DPA loss é,. The goal of decompilation is to find a P that characterizes the seman- tics e. which we treat as the inverse of compilation, i.c., P characterizes Ey whenever its semantic loss equals lg that is, {(P, 8, D) = (6, D). Given the symmetry between the DPA loss and po == log L (Table 2) and the semantic loss P
Compilation and semantic loss. Given a symbolic for- mula P specifying model behavior (e.g., those in Figure 3), to compile this into a loss we interpret in some differen- tiable logic. Our approach is based on probabilistic logic and the semantics of weighted model counting (WMC) (Chavira & Darwiche, 2008; Fierens al., 2015). Accordingly. loss for formula is computed the negative logarithm of the probability of that formula pa (P)- WMC(P;0) 0) given as:
and ratio Psem(P) == log WMC(P) /WMC(-P). we define decompiling into the standard semantic loss (Section 5.2) as translating the equations phB and p into logical formu- las P.. and P1 s.t. P = WMC(P,) w P3 = WMC(P;), and there exists a single formula P where P.. = P and P, = P
WMC(P;;) = Σ II (x.)
II Po())) w
W W
We pursue this loss equation to logic translation approach to decompilation in Section 5.2, later using the translation rules in Table 7 for translating Pe to P... and P,. To make the translation direct and transparent, impose the following compositionality constraint familiar from programming language semantics (Stoy, 1977):
This is the weighted sum over all the propositional models or truth assignments w E {0. 1]" of P (i.e., rows in Fig. 4) where P is satisfied (i.e., w = P or rows in Fig. 4 marked with ). Each is weighted via an independent prod- uct of all the probabilistic predictions X, in w (Po (Xx) or 1 (X,, based on the truth value of X; in w), which induces a probability distribution over all truth assignments (De Raedt& Kimmig, 2015).
Assumption (compositionality)) When translating the preference log ratios pe from Table 2 to propositional for- mulas Ps,, and Pt, every unique model prediction Ps(()) in p and P is treated as a unique weighted proposition form- ing atomic variable, and the propositional formulas P and Pt are built independently and compositionally by re- peated application of Boolean operators over these atomic variables and none others.
Formally, the standard semantic loss of Xu al. (2018) takes the form e(P, 0, D) Ea-Di- log po po (Pa)), where we use the notation P. throughout to refer to the substitution of variables in our formulas P (e.g., x, Yw, Yi) with specific values from d ~ D. Since we will later compute the proba- bility of P conditioned (optionally) on some conditioning constraints Pc (i.e., an additional propositional formula),
The following establishes that not all DPA losses can be compositionally decompiled using the standard semantic
5

---

Understanding the Logic of Direct Preference Alignment through Logic
loss (see proof in Appendix C involving the simplest DPA loss (CPc) and motivates the need for more expressive logic and semantic encoding of DPA, which we investigate next.
Table 3. Different forms of the generalized semantic loss that
match the structure of DPA losses in Table 1.
Variant Edg
M B)
Semantic loss ratio
Proposition 1 (decompilation and standard semantic loss). Under Assumption 1, not all of the losses in Table 2 can be decompiled into the standard semantic loss (Eq 3).
Cablig
log (Pppan))
WMC
Py:0
Exhapured
(prm 28
Pum (P) = log
max(0. 1 [(emm)
WMC
5. A logic for preference modeling
With full preference structures containing PA. any two propositional formulas (e.g., any Pw and Pt) can be ex- pressed as a preference structure based on a particular con- struction, called the implication form, which will play a central role when doing decompilation in Section 5.2.
In the standard semantic loss, loss functions ly are express- ible as single propositional formulas P interpreted via probabilistic logic, with la = - Pog))(P) Proposition 1, however, reveals issues with trying to perform a composi- tional translation of preference losses into a single formula. Indeed, in logical accounts of pairwise preference (Jeffrey, 1965; Rescher, 1967). it is common to model preferences not as a single propositional formula but as an inequality between the scores j (computed e.g., by WMC) of two independent propositional formulas μ(Pw.) > p(Pi).
formulla Given any two propositional formulas Par and Pt, there exists a preference structure P = (P, Pc. PA) such that P u p (Eq 4 left) and P1 = -P, (Eq 4 right).
Proposition 2.
Proof. We provide specific construction called the impli- cation form of P w and Pt, based on the following logical equivalences, which can be checked manually:
To bridge this gap, we define a preference structure, a relational structure and semantic encoding, that allows us to capture the semantics of DPA losses in a modular fashion using a single propositional formula coupled with auxiliary constraints. This structure, based on a novel construction in propositional logic (Prop. 2), makes it easy to cleanly char- acterize different DPA losses. We will use it to generalize the semantic loss and create a novel logic for DPA.
Pw =
(Pr - Pw))
V (Pw A Pt)
^
(P VP)
PA
Pc
P =
(Pr - P m
V
(Pw A Pi)
Vt)
PP
P
Pc
As noted above, this construction corresponds exactly to the preference structure (P, Pc, Pc,,PA Px) with P := Pt + P Pc := P.. V Pt. and PA = P., A Pz- (As a special case, when P. = = w. this simplifies to P = (Pw, T))).
Preference structure. A preference structure is a tuple = (P, Pc, PA) that, as will become clear shortly from Prop 2, captures the semantics of a winner and a loser. It consists of three propositional formulas: a core semantic formula coupled with conditioning constraints Pc (as in Eq 2, which restrict the propositional models that can be counted). and additive constraints PA that tell us which propositional models must always be counted. As we will show, all DPA losses in Table 2 are representable as prefer- ence structures, often ones where the same core formula P shared (e.g., the formulas in Figure 3), differing only in their constraints (Pc and P.).
Figure 4 shows a natural encoding of preference structures as Boolean truth tables where rows with v denote the propositional models of P., and rows with x denote the propositional models of P. (or, equivalently, P, and -P, using the implication form of P... and P; just introduced).
5.1. Semantic loss based on preference structures
In our generalization of the semantic loss, formulas P will be replaced with preference structures P. For example, can modify the logistic log form of SL in Eq 3 to be e(P, A, D) and change the semantic loss ratio Psem accordingly to oper- ate over the formula forms P in Eq 4. By analogy to the generalized DPA in Eq 1, we can view this logistic log form as a particular instance of generalized semantic loss:
Each preference structure has a formula form py and negated formula form -p,, defined follows:
Py : (PPPP. ^ Pc, Pp (-PPP.) ^ Pc- (4)
Intuitively, Py and -Py correspond to the semantics of the winner (Pw) and the loser (Pt). Preference structures and their corresponding formula forms are designed to give us a way to express the original semantic loss, the conditional semantic loss, and arbitrary pairwise preferences. For exam- ple, making PA equal to 1 makes the semantic loss of P, equivalent to the conditional semantic loss from Eq
[(peom ((pemm(Pa).))) (Pa).
(a 0, D)
E dn
(5)
where, like in DPA, different choices can be made about f each giving rise to novel loss - and the model quantities encoded in each P. To prove our main formal results, we
6

---

Understanding the Logic of Direct Preference Alignment through Logic
focus the losses in Table 3, each defined with added 3 sealing parameter to match the structure of DPA losses. We note that the original semantic loss is special case of Catllog (specifically in cases where the input preference structures have Pc = T and PA = I).
Algorithm Translation of loss to logic (decompilation)
input Disjoint polynomial Po = log & output P
P. + (EM (Translation to logic, Table 7}
P. K SEM
P < SIMPLIFY SIMPLIFY(ImpIIn (Implies(pp: Pc)) (Implication form]
Pc < SIMPLIFY(OZ(PP, Pa)) (via Proposition 2}
How many loss functions are there? Under this formula- tion, we can view loss creation as a generative procedure: select an f then sample two formulas P... and P. (each de- noting unique Boolean function in variables) to create P via Prop 2 (see also Figure 4). Absent any constraints, the total number of definable preference structures is doubly exponential in the number of probabilistic predictions n, specifically 42" (i.c., all unique pairs of Boolean functions). While not all such preference structures will lead to mean- ingful or unique losses, for DPO (n = 4), this results in an upper bound of about 4.3 billion definable losses.
PA AMPLIPY(And((P) Pb))
WMC pr:0)
return P PA) fpo=log
Lem. 1}
WMC -
Lal whenever the condition in Eq 6 holds. Given the struc- ture of the DPA loss (Eq 1) and the generalized semantic loss (Eq. 5), for any fixed f. this can be reduced to finding a P whose semantic loss ratio Prom (P) is equal to lg 's core loss equation pa. i.e.:
p log
log WMC WMCC(Pf;:0
How is the loss space structured? While the space is large, one can structure this space using the semantics of the corresponding formulas. Below we define preference struc- ture entuilment and equivalence, and relate these semantic notions to the behavior of the compiled losses. These formal notions not only give us tools for structuring the DPA loss space but also inform the search for new loss functions.
P
Based on this, we define a procedure for translating the core loss equations P9 in Table 2 into preference structures.
Characterizing the DPA equation class. Motivated by the structure of real preference losses, we assume that by construction all the core equations for DPA losses Pa and Pg are expressible certain types of disjoint multilinear polynomials over binary variables {ar, Jr intuitively poly- nomials whose translation via the rules in Table 7 results in valid formulas of propositional logic. Formally, such polynomials over n variables are defined as any polynomial e of the form e = Σ, e; where (a) for all there exists J, S {1, ......,,1.. n) such that e; = He lys where l.g is ei- ther æi or (1 - xy)), and (b) for all i, i", terms e; and eg are disjoint, i.e., have no common solutions (for some k. one term has 24 and the other has 1 - Ix) (see Appendix H for discussion of a loss that doesn't 't immediately fit this format).
We define preference entailment for two preference struc- tures P
in terms of ordinary propositional entail- ment (-) between their formula forms: pT c())
E
(2)
"D) These losses are monototic w.r.t. preference entailment (proof deferred to Appendix E), as in the original SL (Xu et al., 2018):
(P, (1)
+ p
(2) A =P, =
Proposition 3 (monotonicity). If (()) 5 p(2) then (P() ,0, D) 2 (x(P")) 0, for any 0, D.
We will later use entailment to characterize the relative strength of DPA losses and visualize their relations using representation called a loss lattice (see Figure 5). We also
Translation algorithm. Our translation process is shown in Algorithm 1 along with an example in Figure 2. Given po. both p and P are independently translated into logic via a compositional translation function SEM. The translation is standard, based on the rules in Table 7: first each model pre- diction (a(-) is mapped to a probabilistic prediction vaiable M(-): then 1 P is mapped to negation, P to conjunction, and P1 + P2 to disjunction; these rules are applied repeat- edly until the full expression translated. By induction on the rules, one can establish the correctness of the translation function SEM, i.e., that for any disjoint multilinear polyno- 0). Finally, the
extend entailment to preference equivalence P (1) (2) in
the natural way, namely when (()) C _ and (2)) = (
Equivalent preference structures have identical semantic losses (see Corollary 1 in Appendix E).
5.2. Decompiling DPA losses into preference structures
The decompilation of a DPA loss Lis,, into a symbolic form can now be stated as finding a preference structure P whose partieular semantic loss la, is equal to EDPA,
mial pj. it holds that P = WMC (pt); implication construction from Prop 2 is applied to create a preference structure P. where formulas are (optionally) minimized via SIMPLIFY (see example in Fig. 9).
(DPA_ = (P,0))
(6)
Correct characterization. We say that a preference struc- ture P correctly characterizes a loss e, under a particular
The following follows from the correctness of our translation
7

---

Understanding the Logic of Direct Preference Alignment through Logic
of f. For example, by changing fa1-1eg l1 squared or l fal-marginn we immediately obtain the following results:
Table What formalized versions of standard losses look like? Formalizations of some of the losses from Table 2 shown in terms of (their core semantic formula) and conditioning constraints Pc (for succinetness, we exclude Px. which can be inferred from each Pc via Algorithm 1).
Theorem 2 (Extension other DPO). The DPO and CPO preference structures in Table 4 correctly characterize the IPO and Slic ic losses (TTblle 1) and satisfy Eq 6 under the aat -al-aqueeed and Eg)-margin semantic losses, respectively.
Loss
Representation
CR
P = M(X.y ) Pc
CEUn1
P = A.d( Not(M(xy)))
Given the ubiquity of DPO-style updates in other online vari- ants of DPA (Qi et al.. 2024; Zhang et al., 2024; Chen et al., 2024b; Guo et al., 2024), similar formal characterizations could be extended to these variants, as well as to recent reward distillation approaches such Fisch et al. (2024), which see a promising future direction of research.
Pc :c:=4. 1
CPO
P == Implies(M(xx))) M(X.y)))
Pc = Oz(M(xyr) ), M(x,yw))
ORPO
P == [mp]iee(MXxyy))Mxxx)
Pc = XOR(M(x.)) M(X.yw))
DPO
P == Implies(
(x.y)) ))
6.1. What do we learn about known losses?
And(Re ))
Pc = Or(And(Re M(x. ))
Single model approaches have intuitive semantics, but differ in conditioning constraints. One goal of our formalization is to cleanly characterize the semantic rela- tionships between losses, For example, with CPO and ORPO we see that both are derivable from the same core semantic formula P == M(x,
SimPO
P == Implies(
And(Mree(xxy..)) M(X.; )).
And(Mref(xxy)) M(Xyy. )))
Pc = ( M(x.)))
And(Mrref(x.))MMxyyy
x, yt) - M(X, from Figure Indeed, this formula appears to capture the fundamental semantics of many known DPA losses. Under our account, however, losses differ in the conditioning constraints Pc they im- pose, which structure the underlying probability distribution in different ways. CPO conditions P on a one-true con- straint that requires al least one of the winner and the loser to be deemed valid (i.e., rules out the semantic interpreta- tion where both are deemed invalid; last row in Fig. top), whereas ORPO imposes one-hot constraint where exactly one must be deemed valid. Through further semantic analy- sis of their preference structures, we can see that both losses are semantically entailed by fegeni, yet have non- con-entailing re lationn to one another or to the cross-entropy loss fcr.
rules and the implication construction (Prop 2):
Lemma 1 (correctness of translation). Given a loss equation Pa :- log pal P with disjoint multilinear polynomials Par- and d/., Algorithm 1 returns preference structure P whose semantic loss ratio Prom (P) equals Pa-
This establishes the correctness of our decompilation algo- rithm, showing specifically that Algorithm yields prefer- ence structures that satisfy Eq 7.
6. Results and Discussion
Interestingly, we see that preference losses are highly con- strained, which might explain their success. This is in con- trast to the losses typically compiled via semantic loss in neuro-symbolic modeling (Marra al., 2024), suggesting that there is much to learn by working backward from empir- ically successful loss functions to their semantic properties.
Table 4 shows the preference structures obtained from Algo- rithm 1 for the DPA losses Table 2. The following result establishes their correctness:
Theorem (Correctness of formalization). The preference structures in Table 4 correctly characterize the losses in Table 2 and satisfy Eq 6 under semantic loss lato (Table 3).
Proof. Since the original losses were all formulated using the logistic log form of DPA, the correctness of Algorithm (which follows from Lemma 1) implies that compiling the representations in Table 4 (which, as noted above, were obtained by running Algorithm 1 on the losses in Table 2) under fsoo will yield precisely the original losses, and hence satisfies Eq 6.
There are many single model losses still to explore, and we can exhaustively enumerate them. Another goal of ours is to be able to derive novel losses from first principles. This can be achieved by modifying the conditioning con- straints Pc. Figure shows (non-exhaustive) lattice repre- sentation of the loss landscape for single model preference approaches (see Fig. 8 for an exhaustive variant) created by mechanically manipulating the constraints in Eezual (the most constrained) and ordering the resulting losses by strict entailment (E), terminating in lunceo (our constraint free running example from Figs. 3-4).
By changing the version of semantic loss, we can extend our analysis to other variants of DPO, showing the generality of our semantic analysis and its invariance to the choice
8

---

Understanding the Logic of Direct Preference Alignment through Logic
case study, we implemented single model losses around the known fceo in Figure 5, treating it a baseline to improve upon. Using a model-as-judge style evaluation adapted from Hong et al. (2024) and a Owen-O. 5B LLM (details in Appendix G), we found one particular loss, fecro to be competitive with &eeus achieving a win-rate of 52.0 as shown in Table 5. We also observe that different losses have markedly different performance across different datasets subsets, suggesting that one-size- -itt--lit--ll approach isn't 't different tasks are best learned using different losses.
Fice
Frues
Fanzo vf
Feina
Pecpo
Eeso of
MNX yur
Ros
Mix MCx, yeu
most constrained
> least constrained
Figure 5.
What other losses are there? Here show the loss land- scape for single model preference approaches using a loss lattice showing losses (nodes) structured according to strict entailment (C) and their core formulas (boxes) with v being the known losses. See Appendix F for details of the individual losses and a more exhaustive lattice with DPO variants Figure 8.
Table 5. Results of feasibility study involving Qwen-0, SB
tuned on the new losses (rows) compared against the known loss Écro (second column) ultra feedback (Cui al., 2024) test in aggregate (2nd column) and on subsets (right columns). See
We structure the resulting set of losses using the core loss equations P. which reveals different semantic regions (blue boxes). In addition to novel variants of fcpo and foapo that optimize for M(x, yr) - M(x, yw), we see an entirely unex- plored region of unlikelihood losses (E120; LooNT,, Funv.. that optimize for the negation of the loser -M(x, yt). Through compilation, all new losses can be explored experimentally, which we discuss below.
details in Section G.
lons
cxrl
Ban
Junnyy
altnchit
46.1 (251:)
46. (22.29
(1:2 9)
46.2 (61.2)
(2.10
Fyree
48.9 (1005)
(51
34. (e6 93
(+:12)
(+:24)
(+:14)
5200 (5:.0)
50.2
(30.7)
572 (4:1.)
531 (41.9)
Faces
46909007)
45.0
(#5.0)
(6-2
While small scale, this study demonstrates the feasibility of using our framework to derive empirically successful losses. Appendix G reports additional experiments and findings. We include further details about the log probability behavior of different losses and how such behavior relates to the construinedness of each loss. We see further work on empirically exploring these new losses on a wider range of models and scales as a promising direction of future research, as well as using the full power of logic to derive even more complex losses from first principles.
Adding a reference model has a clear, though sometimes peculiar, semantics. The semantics of DPO, is shown in Table 4 and is logically equivalent to a conjunction of two implications: (ef((xyw) A M(X, yt) + M(x, Yw) and MM(X yi). The first says that If the reference deems the winner to be valid and the tunable model deems the loser to be valid, then that model should also deem the winner to be valid, while the second says that the tunable model should deem the loser to be not valid whenever the reference deems the winner to be valid and the loser to be not valid. While this semantics makes sense, and complements nicely the semantics of CPO by adding information about the referent model, DPO includes conditioning constraints that are hard to justify from first principles, and that make it semantically disconnected from the CE and CEUnl baselines.
Ref( A -Re((x,
7. Conclusion
Despite the routine use of variety of DPA algorithms to align LLMs with human preferences, knowing what exactly the losses underlying these algorithms capture and how they relate to each other remains largely unknown. We presented a new technique for characterizing the semanties of such losses in terms of logical formulas over boolean proposi- tions that capture model predictions. Key to our approach is decompilation procedure, allowing one to compositionally derive provably correct symbolic formulas corresponding to any loss function expressed as a ratio of disjoint multilinear polynomials. Our approach provides a fresh perspective into preference losses, identifying a rich loss landscape and open- ing up new ways for practitioners to explore new losses by systematically varying the symbolic formulas corresponding to existing successful loss functions.
We also note that variants like SimPO and DPOP when formalized maintain exactly the same strueture of DPO in Table 4, with DPOP adding repeated variables that amplify the score of the winner (see Appendix H). Giving the se- mantic similarity between these variants and DPO, any small semantic change found in one would likely be useful in these others, which motivates general exploration into varying the conditioning constraints. Several such variants of DPO and SimPO are shown in Figure 8.
Acknowledgements
Can we find empirically improved losses using our method? The ultimate goal of our analysis is to facili- tate the discovery of empirically improved DPA losses. As
We thank the following people, as well as the UtahNLP group and anonymous reviewers, for their feedback at vari-

---

Understanding the Logic of Direct Preference Alignment through Logic
ous stages of the work: Kareem Ahmed, Gregor Betz, Jun- yan Cheng, Hamish Ivison, Maryna Kavalenka, Emile van Krieken, Nathan Lambert, Robin Manhaeve, Valentina Py- atkin, Antonio Vergari, and Gijs Wijnholds. VS was sup- ported in part by NSF awards #2217154 and #2411319.
Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X.,
Lv, C., Lx, H., Lv, K., Ma, L.., Ma, R., Ma, Z., Ning, W., Ouyang, L., Qiu, J., Qu, Y., Shang. F., Shao, Y., Song, D., Song, Z., Sui, Z., Sun, P., Sun, Y., Tang, H., Wang, B., Wang. G. Wang. J., Wang, J., Wang, R., Wang, Y., Wang, Z., Wei, X., Weng, Q. Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang. X., Ye, H., Ying, H., Yu, J., Yu. J., Zang, Y., Zhang, C., Zhang, L., Zhang, P., Zhang, P., Zhang, R., Zhang, S., Zhang, S., Zhang, W. Zhang, W. Zhang, X., Zhang, X., Zhao, H., Zhao, Q. Zhao, X., Zhou, F., Zhou, 7.,, Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. InternLM2 technical report, 2024.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Calanzone, D., Teso, S., and Vergari, A. Logically Consis- tent Language Models via Neuro-Symbolic Integration. Proceedings of ICLR, 2025.
References
Ahmed, K., Teso, S., Chang, K.-W., Van den Broeck, G., and Vergari, A. Semantic Probabilistic Layers for Neuro- symbolic Learning. Advances in Neural Information Processing Systems, 35:29944 29959, 2022.
Chavira, M. and Darwiche, A. On Probabilistic Inference by Weighted Model Counting. Artificial Intelligence, 172 (6-7):772-799, 2008
Ahmed, K., Chang, K.-W., and Van den Broeck, G. A Pseudo-Semantic Loss for Deep Generative Nodels with Logical Constraints. Advances in Neural Information Processing Systems 36 (NeurIPS), 2023a.
Chen, A., Malladi, S., Zhang, L. H., Chen, X., Zhang, Q.. Ranganath, R., and Cho, K. Preference Learning Algo- rithms Do Not Learn Preference Rankings. Proceedings of NeurlPS, 2024a.
Ahmed, K., Teso, S., Morettin, P., Di Liello, L., Ardino, P., Gobbi, J., Liang, Y., Wang, E., Chang, K.-W., Passerini, A., et al. Semantic Loss Functions for Neuro-Symbolic Structured Prediction. In Compendium of Neurosymbolic Artificial Intelligence, pp. 485-505. IOS Press, 2023b.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self- play fine-tuning converts weak language models to strong language models. Proceedings of ICML, 2024b.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Amini, A., Vieira, T., and Cotterell, R. Direct Preference Optimization with an Offset. In Findings of ACL, 2024.
Asai, A. and Hajishirzi, H. Logic-guided Data Augmenta- tion and Regularization for Consistent Question Answer- ing. Proceedings of ACL, 2020.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. UltraFeedback: Boosting Language Models with High-quality Feedback. Proceedings of ICML, 2024.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. General Theoretical Paradigm Understand Learning from Human Prefer- ences. Proceedings of AISTATS, 2024.
De Raedt, and Kimmig, A. Probabilistic (logic) Program- ming Concepts. Machine Learning, 100:5-47, 2015.
De Raedt, L.., Kimmig, A., and Toivonen, H. ProbLog: probabilistic Prolog and its application in link discovery. In Proceedings of IJCAI, pp. 2462-222222222 2467, 2007.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309. 1909.,6669 2023.
Dohan, D., Xu, W., Lewkowyez, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., Murphy, K., and Sutton, C. Language Model Cascades. arXiv preprint arXiv:2207. 10342, 2022.
Beurer-Kellnee L., Fischer, M., and Vechev, M. Prompting is Programming: A Query Language for Large Language Models. Proceedings of the ACM Conference on Pro- gramming Languages, 7(PI.DI):
2023.
Eisner, J., Goldlust, E., and Smith, N. A. Dyna: A Declara- tive Language for Implementing Dynamic Programs. In Proceedings of ACL, 2004.
Cai, J., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., Fan, Q., Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C., Hu, Y., Huang. T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L.,, Li, S., Li, W.,
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization. Proceedings of ICMI., 2024.
10

---

Understanding the Logic of Direct Preference Alignment through Logic
Fierens, D., Van den Broeck, G., Renkens, J., Shterionov, D., Gutmann, B., Thon, I., Janssens, G., and De Raedt, L. Inference and learning in probabílistic logic programs using weighted boolean formulas. Theory and Practice of Logic Programming, 15(3):358-401, 2015.
language model calls into self-improving pipelines. Pro- ceedings of ICLR, 2024.
K Kwon, W. Li, Z.,, Zhuang, S., Sheng, Y., Zheng, L.,, Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Fisch, A., Eisenstein, J., Zayats, V., Agarwal, A., Beirami, A., Nagpal, C.,, Shaw, P., and Berant, J. Robust Preference Optimization through Reward Model Distillation. arXiv preprint arXiv:2405. 19316, 2024.
Li, T., Gupta, V., Mehta, M., and Srikumar, V. A Logic- Driven Framework for Consistency of Neural Models. In Pruceedings of EMNLP, 2019.
Fischer, M., Balunovic, M., Drachsler-Cohen D., Gehr, T., Zhang, C., and Vechev, M. DL2: Training and Querying Neural Networks with Logic. In Proceedings of ICML. 2019.
Li, Z., Huang, J., and Naik, M. Scallop: A Language for Neurosymbolic Programming. Proceedings of the ACM on Programming Languages, 7(PLDI): 44877187, 2023.
Friedman, D., Wettig, A., and Chen, D. Learning Trans- former Programs. Advances Neural Information Pro- cessing Systems, 36, 2024.
Liu, W., Zeng, W., He, K.,, Jiang, Y., and He, J. What makes good data for aligmment? comprehensive study of auto- matic data selection in instruction tuning. Proceedings of ICLR, 2024.
Giannini, F., Marra, G., Diligenti, M., Maggini, M., and Gori, M. On the Relation between Loss Functions and T-norms. In Inductive Logic Programming: 29th Interna- tional Conference, 2020.
Manhaeve, R., Dumaneic, S., Kimmig, A., Demeester, T, and De Raedt, L. Deepproblog: Neural probabilistic logic programming. Advances in neural information processing systems, 31, 2018.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint ar Nww:2402.04792. 2024.
Marconato, E., Teso, S., Vergari, A., and Passerini, A. Not all Neuro-Symbolic Concepts are Created Equal: Analy- sis and Mitigation of Reasoning Shortcuts. Advances in Neural Information Processing Systems, 36, 2024.
Halpern, J. Y., Harper, R., Immerman, N., Kolaitis P. G., Vardi, M. Y., and Vianu, V. On the Unusual Effectiveness of Logic in Computer Science. Bulletin of Symbolic Logic, 7(2):213-236, 2001.
Marra, G., Giannini, F., Diligenti, M., and Gori, M. Inte- grating Learning and Reasoning with Deep Logic Mod- els. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 517-532. Springer, 2019.
Hewitr, J., Hahn, M.. Ganguli, S., Liang, P., and Manning, C. D. RNNs can generate bounded hierarchical languages with optimal memory. In Proceedings of EMNLP, 2020.
Marra, G., Dumančić, S., Manhaeve, R., and De Raedt, L. From statistical relational to neurosymbolic artificial intelligence: A survey. Artificial Intelligence, pp. 104062, 2024.
Hinnerichs, T., Manhaeve, R., Marra, G., and Dumancic, S. Declarative Design of Neural Predicates in Neuro- Symbolic Systems. arXiv preprint arXiv:2405.09521, 2024.
McCarthy. J. et al. Programs with common sense. RLE and
Hong, Lee, N., and Thome, J. Reference-free Monolithic Preference Optimization with Odds Ratio. Proceedings of EMNLP, 2024.
MIT computation center Cambridge, MA, USA, 1960.
McCulloch, W. S. and Pitts, W. A logical calculus of the ideas immanent in nervous activity. The bulletin of math- ematical biophysics, 5:115-133, 1943.
Hu, X., He, T., and Wipf, D. New Desiderata for Direct Preference Optimization. Proceedings of ICML Workshop on Models of Human Feedback for Al Alignment, 2024.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple Preference Optimization with Reference-free Reward. Proceedings of Neurips, 2024.
Jeffrey, R. C. The logic of decision. University of Chicago press, 1965.
Meurer, A., Smith, C. P., Paprocki, M., Čertik, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., et al. SymPy: symbolic computing in Python. PeerJ Computer Science, 3:e103, 2017.
Khattab, O.,, Singhvi, A., Maheshwari, P., Zhang, Z., San- thanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. Dspy: Compiling declarative

---

Understanding the Logic of Direct Preference Alignment through Logic
Miranda, L. J. V., Wang, Y., Elazar, Y., Kumar, S., Py- atkin, V., Brahman, F., Smith, N. A., Hajishirzi, H., and Dasigi, P. Hybrid Preferences: Learning to Route In- stances for Human vs. Al Feedback. arXiv preprint arXiv:2410. 19/333,, 2024.
Tang, Y., Guo, Z. D., Zheng, Z.., Calandriello, D., Munos,
R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint MiXiv:240.007444 2024.
Nilsson, N. J. Logic and Artificial Intelligence. Artificial intelligence, 47(1-3):31-56 1991.
van Krieken, E., Badreddine, S., Manhaeve, R., and Giunchiglia, E. Uller: A unified language for learn- ing and reasoning. In International Conference on Neural-Symbolic Learning and Reasoning, pp. 219-239. Springer, 2024a.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., et al. Training Language Models to Follow Instructions with Human Feedback. Advances in neural information processing systems, 35:27730-2777 27744, 2022.
van Kricken, E., Minervini, P., Ponti, E. M., and Vergari, A. On the Independence Assumption in Neurosymbolic Learning, Proceedings of ICML, 2024b.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing Failure Modes of Pref- erence Optimisation with DPO-Positive, arXiv preprint UrXiv:2402.13228 2024.
Vieira, T., Francis-Landau, M., Filardo, N. W., Khorasani, F., and Eisner, J. Dyna: Toward a self-optimizing declarative language for machine learning applications. In Proceed- ings of the 1st ACM SIGPLAN International Workshop Machine Learning and Programming Languages, pp. 17,77,00 2017.
Park, R.,, Rafailov, R., Ermon, S., and Finn, C. Disentangling Length from Quality in Direct Preference Optimization. Proceedings ACL, 2024.
Von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learn- ing. https: /ttp//ggtttt....ee 2020.
Qi, B., Li, P., Li, F., Gao, J., Zhang, K., and Zhou, B. Online dpo: Online direct preference optimization with fast-slow chasing. arXiv preprint arXiv:2406.05552 2024.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct Preference Optimization: Your language model is secretly reward model. Proceedings of NeurlPS, 2023.
Wang. Y., Zhong, W., Li, L., Mi, F., Zeng. X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning Large Language Models with Human: A survey, arXiv preprint arXiv:2307. 12966, 2023.
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. Pro- ceedings of ICIR, 2025.
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, Neural text generation with unlikelihood training. In International Conference on Learning Repre- sentations, 2019.
Rescher, N. Semantic foundations for the logic of preference. The logic of decision and action, pp. 37-62, 1967.
Winata, G. I., Zhao, H., Das, A., Tang, W., Yao, D. D., Zhang, S.-X., and Sahu, S. Preference tuning with human feedback on language, speech, and vision tasks: A survey, Journal of Artificial Intelligence Research, 82:2595-2661, 2025.
Richardson, K. and Srikumar, V. Lectures For- mal Techniques for Neuro-symbolic Modeling, August 2023. URL https: / /htbbbb..ttttt/g//ggg// com/ httm//yaazimir// satllinneuaaa__yb_aaa
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, J. Contrastive Preference Op- timization: Pushing the Boundaries of LLM Performance in Machine Translation. Proceedings of ICML, 2024.
Richardson, K. and Wijnholds, G. Lectures on Lan- guage Model Programming, August 2024. URL https: /t/it/ttttuuu/e/ com/ www.aiiiireesss//ee.. 2024 hat.__poorraammnn.mm
Xu, J., Zhang, Z., Friedman, T., Liang, Y., and Broeck, G. A Semantic Loss Function for Deep Learning with Symbolic Knowledge. In Proceedings of ICML, pp. 5498- 5507, 2018
Šlusarz, N., Komendantskaya, E.,, Daggitt, M. I.., Stewart, R., and Stark, K. Logic of differentiaabee logies: Towards uniform semantics of dl. Proceedings of LPAR, 2023.
Yixing, L., Yuxian, G., Dong, L., Wang, D., Cheng, Y., and Wei, F. Direct preference knowledge distillation for large language models. In arXiv preprint 24Xiv:2466.19774 2024.
Stoy, J. E. Denotational semantics: the Scott- Strah-Strchey approach to programming language theory. MIT press, 1977.
12

---

Understanding the Logic of Direct Preference Alignment through Logic
Yuan, Z.,, Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. RRHF: Rank Responses Align Language Models with Human Feedback without Tears. Proceed- ings of NeurlPS, 2023.
B. Original losses
Further details of the original losses in Table 2. along with other variants such as R-DPO (Park et al., 2024), ODPO (Amini al., 2024), and DPKD (Yixing et al., 2024), are shown in Table 6. While our formalization abstracts over certain details such as length normalization and additional regularization terms (and hence generalizes//impliies some of the original losses), we include here such details from the original studies. In the case of regularization terms, as noted in Table 6, most no reference approaches add an additional cross-entropy term, often making the full losses in these studies equal to fx+CF = Eg + XeER (with weight A). In some cases, additional terms over in our analysis, e.g., in Corso the full loss includes an additional weight term or that is added to the main loss (in our experiments below, a is implicitly set to 1).
Zhang, S., Yu, D., Sharma, H., Zhong, H., Liu, Z., Yang. Z., Wang, S., Hassan, H., and Wang, Z. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405. 2024.
Zhao, H., Winata, G. I., Das, A., Zhang, S.-X., Yao, D. D., Tang, W., and Sahu, S. RainbowPO: A Unified Frame- work for Combining Improvements in Preference Opti- mization. Proceedings of ICLR, 2025.
assumed that we abstract
Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., and Liu, J. Calibrating Sequence likelihood improves Conditional Language Generation. In Proceedings of ICLR, 2022.
C. Compositionality constraint
Proposition 1 (decompilation and standard semantic loss). Under Assumption 1, not all of the losses in Table 2 can be decompiled into the standard semantic loss (Eq 3).
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Salch, M., and Liu, P. SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arXiv preprint arXiv:2305.10425. 2023.
Proof. Taking EcPO as an example, the loss equation is based on the ratio sp (B y1) consisting of two predictions (s I) and Po(y x). which we can translate into the propositional formulas P. == M(x, yw) and Pa == M(x, yt), consisting of total of two atomic propositions. Translating this to the standard semantic loss involves finding single P such that P = P and P:- P. To see that no such P exists, we can enumerate all 16 unique Boolean functions over variables M(x, yw) and M(x, (xxy) y:) (the only variables we are allowed under Assumption 1) and verify that none yield
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Sine-tuning language models from human preferences. arXiv preprint (tf1::1900.08999 2019.
A. Additional related work
Decompilation of losses. In contrast to work on loss com- pilation, we focus on the inverse problem of decompilation (see Friedman et al. (2024)), or deriving symbolic expres- sions from known loss functions. Work in this area has mostly been limited to symbolically deriving standard loss function such as cross-entropy (Giannini et al., 2020; Li et al., 2019), whereas we look at deriving the semantics of more complex LLM algorithms.
single formula s.t. log WMC(P:9)
WMC vMMc()P:0) so w)). The same argument be applied to each of the other non- baseline losses in the table.
Without the compositionality assumption, one can encode any Pe as a formula using additional variables and weighting schemes, as is commonly done in standard WMC encodings (Chavira & Darwiche, 2008). However, the semantics of the resulting formulas are less transparent and often hidden in the weights. We instead propose to define below a novel (unweighted) encoding for preference that doesn't 'ttttnnnn require additional variables, thereby facilitating a compositional and transparent translation from loss equations.
Declarative model programming. In addition to neuro- symbolic modeling (Richardson & Srikumar, 2023; Marra et al., 2024), we take inspiration from recent work on formal- izing LLM algorithms in terms of programming language concepts (Dohan et al., 2022; Beurer-Kelli Khattab et al., 2024), with our approach being declarative in style (see review in Richardson & Wijnholds (2024)). As such, our study takes much inspiration from the large literature on declarative programming techniques for ML (Eisner et al., 2004; De Raedt et al., 2007: Li et al., 2023; Vieira et al., 2017; Šlusarz et al., 2023; van Krieken et al., 2024a; Hinnerichs et al., 2024).
et al., 2023:
D. Semantic translation rules
In Table we show the full translation rules for Algorithm 1.
13

---

Understanding the Logic of Direct Preference Alignment through Logic
Table Details of the original losses from Table 2 and others (adapted from Meng (2024)), all of which originally implemented using the logistic log-loss, i.e., each & log a (3pe). We also include details about whether cross-entropy regularization (CE term) length normalization (length norm.) were used (yes v. no x) along specific choices about 3 or cross-entropy weight X) that either exclude or generalize in analysis and experiments (e.g.. extra loss weighting terms a). See Winata al. (2025) for comprehensive review and Zhao (2025) for an approach that further mixes the DPO and S SimPO losses.
other details (Extra details) (e.g., extra weight terms,
Loss name
core loss equation Pe
CE term
length norm.
Extra details terms
common baseline losses
Ecs
log
Pel (r)
1-Pe(Suilr (x)
Eerost (Ratalioo
Pa (ine (1 Pa (arl))) log
Unlikelihood term weighted by
(Va 2211 Pater FM
reference approaches
Epro (Rafallov
log Pol Four
X
X
Fot Pat
Eoona (Amini 2024)
log
Polyal=>) Patt
Toffer
AX
X
X
Added offset Totket
Eopop (Pal 2024)
log
C Paa (a
X
See Appendix
Pur (ao Po(an())
En. pPo (Park 2024)
log
+ 7Mm
Added length bias term "Men
EarCin
Patorla)
tunu (Yixing et all. 2024)
log
Fauken (Sa (ir) Plesser (3i lar)
Distillation, re parameterizes ref and
F Tau (in) (
single model (no reference), CE weight
Euro (Xa et al,, 2024)
log Petyulx) [x]
Removes ref
Prlgr [z)
EoRDo (Hong al, 2024)
log P [o [1]
Pa(u(l))))))
B 1, main loss weighted by o, À =
Polyila)11 Pa(wl
fiiooo (Mang
log [a(
X
Added margin term re-formalized in Table
It follows that (((P")) 00 (d)) 2 (((( ,, (.)))) Taking the expectation over d ~ D, we obtain (a())) .0 > (((2)) , D).
Table Rules for the compositional translation of loss expressions into symbolic formulas. See again example in Figure 2.
Input
SFM(-)
predictions
Puly x)
P P:=M(<> Mxx
It follows that equivalent preference structures have identical semantic losses:
formulas P
P1 Pg
P == And P1, P2)
Corollary 1 (semantic equivalence). If P P then (,(()() 0, D) = Lu(PP" A, D) for any 0, D.
1 - P
P = Not P))
P
+ P2
P ::= Or(P P2)
The next result is analogue the locality property in the original semantic loss (Xu et al., 2018), which tells us that unused logical variables in formulas do not affect loss values, which allows us to compare losses with different number of variables.
E. Proofs of other propositions
Below we state propositions discussed in Section 5.1 with their proofs.
Proposition 4 (locality). Let be a preference structure defined over probabilistic prediction variables X. with pa- rumeters 0g. Let Y be some disjoint set of variables with parameters 0y. Then (x any D.
Proposition (monotonicity). If (1) 5 p (2) then (P" 0, D) 2 (P()), ,0, D) any 0,
(P. Pz. D) (P, 10 0g), for
Proof. By the definition of preference entailment, we have
(()) - p(?) This means that for any d, Pdd()
Proof. Let w. be any world over variables X and w, be any world over (disjoint) variables Y. Let Wa,y denote the joint world. By the standard semantic loss, the probability of the world Wx.y in the (X, Y) space
(d)), which implies that for any 8, WMC(F( (d):
WMC(P (d): From the definition of preference en- tailment, also have _p(2)()) (d) } -p (d). Follow- ing a similar line of reasoning above, this implies WMC(-P 0d):0) 2 WMC(-p(() (d); 0). Thus, for any and 0, the weighted model counting ratio term in the se- mantic loss in Table is no larger for (1) than for 2(2)_
be written as IIx, Qa (X:) IIY Qo (Y₃) where Q is either P or - P. Since the parameters 0, and Oy refer to disjoint sets of variables, we can simplify this to
Po..,, a (Wa.y)
IIx, Qs. (X,) IIy, Q0, (Y,).
It follows that the marginal probability of the world
14

---

Understanding the Logic of Direct Preference Alignment through Logic
Wo in the (X space equals
Po.,0, (Wz)
As described above, the final loss that we implemented was then obtained by applying the logistic loss loss over these equations and adding 8 term and cross-entropy terms (see details below). We used the t. library for implementation from (von Werra et al., 2020), with assistance from the trainer scripts used in Meng et al. (2024).*
Ex (Ixxx (x,(X:) Iryer Qo,())))
Ixxex (Xi) Ev (Ilyse Qa, (Ki)
(1- Qo, (K3)) exxexx Qe, (X.) Pe. (Wa). This last expression is precisely the probability of the world Wa in only the X space. Thus, Po. (Wa) = Po (Wa), which implies WMC((P:0) = WMC 02: 0y) and similarly for -P. From this, the claim follows immediately.
Ix;ex (Xi)- -Ilyev
Extending the loss lattice to reference models As seen in Table 2. single model losses can be mapped DPO- style losses with reference models by subtracting the log ratio $ee( v1) from their loss equation pe, which we call the reference form of a single model loss. We note the following fact about reference loss forms. Formally, given any core loss equation Po equal to log Pb/ the reference form of that loss (i.e., Pe - (y Mt) with (ye 31 := log (ye | x) Prr/Prr(y
0
F. New losses in loss lattice
To visualize the semantics of the single model losses shown in Figure 5, we use the Boolean truth table shown in Figure 6. As already illustrated in Figure 4, each loss column can be mechanically converted into a preference structure via the following steps: 1) translate V and x into two standard propositional formulas that are logically consistent with the marks, P. for Ps. respectively, then 2) apply the rules in Algorithm to these formulas to get a preference structure P. (Note that the formulas in boxes in Figure show the core formula P in the resulting preference structure and intentionally hide details about the constraints.)
2)) is equal to the core loss equation P5 == log Pg Par(url)) which fol- lows from the application of the quotient rule for logarithms.
p Par()))))))
As an example, the reference form of Ecpo is equal lozo- given that the reference form of so (y yR) (i.e., CPO's loss equation) is 5g 6y M) Seer S M1 (DPO). Using the quo- tient rule for logarithms, we can transform this into the core loss equation pgf equal to log Po Par((lll)))) firms the observation above. In contrast, the reference form of Coxpo is novel loss se (y 31)) - sy (Fwr Wi - Sjet(yur 31) corresponding, after the same algebraic manipulation, to the new loss shown in Figure 1 (DPO variant) and the core loss equation log Po P20
Py (e(dd)))))))) which con-
With these preference structures, we can then obtain com- piled version of the loss by simply applying one of the versions of the semantic loss. simplified terms, finding the compiled loss equation directly from truth table for given version of semantic loss with convex function f (e.g.. those listed in Table 3) involves the following
Py (x) (1)(1) (al
Given how our decompilation procedure works via Algo- rithm 1, we note the following fact about how such extra reference terms affect the semantics of the original loss:
Σ Σ
f( log
Proposition 5 (semantics of reference forms). Given loss characterized by the core loss equation po equal to log pal ph. the core semantic formula P for that loss's 's refer- ence form is logically equivalent to the formula (SEM(P')) A Ref(x, (x))) > (SEM(p)) A Ref(x,,t)).
where we can replace each E. with the corresponding WMC equations for each mark, then simplify the resulting equation (i.e., the core loss equation) to arrive at a compact loss equation that can be directly used for implementation.
Figure 8 (gray boxes) show the semantics of the reference forms for more exhaustive set of single model losses from Figure 5 using the recipe above. This reveals wide range of novel variants of DPO that we leave for future exper- iments and study. Correspondingly, Figure 7 shows the Boolean semantics of DPO/S o and some novel variants based on the reference form of ORPO (CORPO-reE). qFUNL (fqtunl-ret) and 15 (fis-res).
Losses used in experiments Employing the process above, below we show the core loss equations for the losses we used in our experiments in accordance with the form in Table 2:
Loss name
Core loss equation (implementation)
lepa
log (nn
Po Po.uull [z)
Lozpo log Po(
(M.. (z) [1 Po lx))
Locra
log [1 Po (z Po Mo m)
Po
[z)
Computing preference structures Figure shows how to symbolically compute preference structure representa- tions in Python using the computer algebra library Sympy
log
11
-
(e (x))
(3 a)
LeSUND
log (1 (g (z Ta x)
[1
Po. (x)
see https:/ http://gggttuuum////gggggae https: /ttps::/giitae http://iinnoooooooonnoooooe
and
Po(yu (2) Pa1 (.. (1 (1--- (x)))
Luncpo
log
Polm [2] - Po ml)
15

---

Understanding the Logic of Direct Preference Alignment through Logic
M Xx Yw
M(xY()
EORPO
Launa
213
Rezuni
Recso
Ecp
Ece
ECe
T
T
X
X
X
V
X
T
F
V
v
v
F
T
X
X
X
X
X
X
X
X
F
F
X
X
X
X
M(X, Yw)
M(X, Yt)
Eccuni
Eru01
Parini
f120
FuncPo
f1i4
facro
f1s
T
T
X
X
X
X
T
F
v
V
F
T
X
X
X
x
X
X
X
X
F
F
X
X
v
x
X
Figure Boolean representation (in the style of Figure 4) the single model loss functions shown in Figure 5. See again Figure for how to interpret the corresponding losses.
(M) Ref (x, Yw)
yI)
(M) ReE (x, yi
M Mw)
EpPo/8La00 Rorporeut EgEumt- out 2is-cwt
F
F
F
F
F
F
T
F
F
T
F
F
F
T
F
T
T
F
F
T
F
F
T
F
T
F
T
F
T
T
V
T
T
F
F
EXP
x
T
T
T
T
T
F
X
x
X
T
T
T
T
V
X
V
X
Figure 7. Boolcan semantics of DPO and S SimPO (column 5) and some novel variants of (columns 6-8) representing the different semantic regions Figure 8.
M(x, A M(x,
MMx yt)
M(x, vi) - M(X,yw)
Re M(XYY) A M6 ve)
[ef(x yu) (M(x y)) ())((x.1) M(x. -M(s.1)
Ref(x. (M)X.))) Masye)
-Red(x. vo)
Unl
120
ЫСPC
CBUn1
Ref (X.
-M0x yt)
1Un1
MCx
M(x
cfUnl
CUNL
ORPO
CCPO
UNCPO
M(x
SCE
CPO
15
CR
Re yer) (MC), yc) -R))(xx)))) A (M- vie (R)((x))))
Re yu) (-MEx,Y)) V -Bed(x wi)
Rad yar) M(x 99) MDx y.. Ref
Yu)
Mix.Y..)
Figure 8. What are interesting variants explore? Extending the loss lattice in Figure version of the single model losses with reference models (i.e., their reference forms), showing different (largely unexplored) variants of DPO and the different semantics regions (gray boxes, corresponding to the core semantic formula for each set losses). See Appendix for details.
16

---

Understanding the Logic of Direct Preference Alignment through Logic
Below provide details of the experiment setting then discuss some results and observations.
1 from sympy import *
2 winner (W), loser (L),
3 f (ref) winner (R_),, loser (R_1)
Dataset and Model Following much of the DPA work we cite, we train models on the ult ra rafees feedback dataset (Cui et al., 2024), which contains around 60k binarized pref- erence pairs aggregated from several individual preference datasets (the different categories are listed Table 5). For tuning (detailed below) we used custom held-out develop- ment set containing around 1.3k examples taken from the train set and reserve the test set (containing 2k examples) for final evaluation.
4 W,,,R.w,RR.
(yubols("W,L,R..))
5 ## equation translation for ORPO
6 P.t An, (L))
7 P.b And (W))
8 ## pref. structure P = (P, Po.
9 P Implies(PPb,PPt).ssiimn simpliyy
10 assert P. r.quals
11 P_C Or
12 P.A Andd(P.t.P.b..simmpll
Standardly, we ran experiments starting from a instruc- tion tuned model (SFT), using Qwen-0 5B (contain- ing .5 billion parameters) base model (Bai al., 2023) that was initially tuned on 6k pairs from the deita dataset of (Liu et al., 2024). To avoid repeating the pro- cess of instruction tuning, we started from the trained Qwen model released in the TRL library". Our full code is available at https: / github. com/a allenai//////// peellrratiee_refeeeneeee anggmentt.
13 ## The reference form formula
14 P.ref = Implies(
15 (nd(D.b,R..) And(P
16 ) .simplif(()
17 assert ref. rref.equale equals(
18
(mplies(And(e)
19
Figure 9. An example showing how compute the simplified sym- holic formulas in preference structures for ORPO (see Figure 2) in Sympy (Meurer al., 2017).
Hapaeeparameters and model selection The following are the standard set of tunable hyper-parameters involved in our experiments: the 3 term for DPA losses (see again Table 1), the learning rate, number of epochs, batch size and length normalization. Following other studies, we also regularized our losses with cross-entropy terms (CE) that include tunable weight parameter À that controls their con- tribution the gradient. Specifically, we kept set / to 1, and experimented with learning rates in the range {le-6 3e-6, 8e-6) 6), number of epochs in the range of {3, 5, 8} and batches sizes in the range { 32, 128 (for efficiency reasons, most tuning with done with a batch size of 32), which follow many of the suggested ranges in Meng et al. (2024). Importantly, length normalization was used through- out make all losses comparable and given that it has been shown to improve training performance (Meng et al., 2024). We used As in the range of (0.0, 0.01, 0.1, 0.3, 1.0] (we found lower values, around 0.01 and 0.1, to be most effective).
(Meurer et al., 2017). Specifically, lines 8-12 show how to compute a preference structure in the no-reference case, and lines 14-20 show how to compute a reference form of ORPO by adding a reference ratio.
G. Experiments and Case studies
Our formal analysis reveals that the space of DPA losses is large, yet structured in systematic ways that we can now describe through symbolic encodings. Through case studies involving the new losses in Figure 5, we discuss some empir- ical results that give tips for how to better navigate this space and look for improved DPA losses using framework. Specifically, we focus on losses around the known loss ÉCPO, which we treat as a natural baseline to compare against. All experiments are performed using a 0.5 billion parameter LLM, Qwen-0 5B (Bai et al., 2023), tuned using trl (von Werra et al., 2020) on the ult ra feedback dataset; following standard practice, losses were implemented with weighted eross-entropy regularizer term.
For each loss function we searched the best hyper- parameters by performing a comprehensive search over the ranges detailed above. Final model selection was then per- formed by performing inference with each trained model on our held-out development set and scoring the resulting generating outputs using an off-the-shelf reward model, in particular, 1.8B parameter reward model from (Cai et al., 2024) We then selected the models with the highest aver-
While these experiments are small scale and limited in scope, they are merely meant to suggest possible uses our frame- work and open questions. We also share some general ob- servations and conjectures that we hope motivates future research in this area.
'https: https:///hhginngfcc..com co/tr /wwe/l 5- 5b sft
'intern wwwnt/rrnnnnnmm/ 1_8b- reward
17

---

These results suggest that understanding the way in which a loss is constrained and whether it gives rise to spurious or unintentional alignment shortcuts (e.g., making both predictions false) is an important factor when designing Understanding the Logic of Direct Preference Alignment through Logic
age reward score over the development set for comparison.
(A)
M
2, V/u)
M (2,)))
EcpO
Luncpo
Lacso LufUni
For the log probability experiments shown in Figure 10, we kept the learning rate, epoch and cross-entropy term constant (with learning rate equal to 1e-6, 3 epochs, and low cross-entropy term 0.01) to directly compare the different approaches and try to bring out their more extreme behavior.
T
T
X
F
T
X
X
X
F
(B)
Evaluation protocol and win-rate comparison We com- pare models tuned using our different losses using proce- dure similar to how model selection is performance, which also follows the setup in Hong et al. (2024). Specifically, we do a instance-level comparison of the reward score given for each generated output, compare that score with the score of our baseline lop and compute an overall win-rate, i.e., % of instances where the reward score is higher than equal to the reward score for kapo (we consider cases where items are equal given that some tasks involve generating single token output, such as the identifier of multiple choice question or yes or no). We report the average win-rate averaged over 3 runs of each models with different generation seeds using vllm (Kwon et al., 2023).
Winner logprob
Loser logprob
unEPO
uncPo
cpo
clUNL
cUNL
cCPO
200
420
600
890
1500
200
4.00
600
1002
Training step
Tr Training step
(C)
0.5
unCPO
UnCPO
0.5
cpo
cpo
crUNL
CIUNL
0.5
cCPO
CCPO
6.3-
0.2
6.2-
0,1-
G.1. Results and discussion
Winner logprob (validation)
Loser logprob (validation)
How does constrainedness relate to loss behavior? Un- intentional alignment shortcuts Moving left the right in Figure 5 yields semantically less constrained losses. For example, we see through the Boolean semantics in Figure 10 that some unconstrained losses can be satisfied by making the winner and loser both false (Cuncpo; ferow. or by mak- ing the the winner and loser both true (Luacpo, LECUNL). One natural question is: How does constrainedness contribute to a loss functions empirical success?
Figure 10. An illustration (A) of how semantically satisfy losses V and the corresponding log probability behavior during train- ing (B) and evaluation (C).
susceptible to such extreme behavior, which could explain their success.
Can we find empirically improved losses using our method? Formalize and refine Our ultimate aim to use our framework to help discover new and successful pref- erence algorithms. Given the spurious behavior of losses funcro and fEFUNL, we would expect them to be less em- pirically successful. To test this and compare against fyro, we performed a woodel-s-judggestyll experiment based on (Hong et al., 2024) that uses an off-the-shelf reward model (Cai et al., 2024) score the outputs generated by our new models using the prompts from the ultra feedback test set. We then compare these rewards scores against those of Eceo to compute win-rate, which gives an indication of improved or comparable generation quality over Ecpc- In- deed, we see in Table 5 that in aggregate, Luncro and Eesus have the lowest win-rate against fero- Interestingly, we see that lecpo has a win-rate that suggests comparable genera- tion quality to fevo, which shows the potential of using our framework to derive new and empirically successful losses.
We observe, consistent with other recent work neuro- symbolic modeling (Marconato et al., 2024; van Krieken et al., 2024b), that such unconstrainedness can yield extreme behavior as illustrated in Figure 10. For example, funcpo and Locont attempt to make both the winners and losers false by driving their probability in the direction of zero (as shown in both training (b) and evaluation (c)), whereas fETCNL keeps both probabilities high to make both true. When viewing learning as constraint satisfaction problem, such behavior makes sense and could help to better understand various spurious training behavior observed elsewhere in the DPA literature, e.g., related to likelihood displacement and unintentional unalignment studied in Razin et al. (2025) or issues with preference ranking (Chen et al., 2024a).
These experiments are an exercise in an approach we call formalize and refine, i.e., starting from empirically suc- cessful losses such as lero, one can formalize such losses then modify the semantics to be more or less constrained
new loss functions. We note that existing losses in Figure 5 are the middle of the two extreme points and seem less
18

---

Understanding the Logic of Direct Preference Alignment through Logic
based empirical findings. We think more large scale
exploration of the full loss space, especially for DPO, is promising direction of future research.
Is there a single semantics for all preference learning?
The different semantics conjecture We note that win- rate across different categories the right most columns Table 5) varies quite considerably across models and loss types. This suggests that different types of preference data rely different semantics of preference, which requires tuning approach that's 's tailored to those differences. We conjecture that such phenomenon is likely to be wide spread across different tasks and datasets, and we see more empirical work on understanding the kinds of semantics needed in different scenarios as a promising direction of future work. Such work will benefit for recent attempts as incorporating more fine- mne-grained annotation into preference such, such in Miranda et al. (2024).
(i.e.,
H. DPOP equation
The original DPOP loss (Pal et al., 2024) in Table subtracts Par(ya))))
an additional term À max(0, log Pa(mul from DPO loss, with the aim of ensuring that the log- logeliielihood of the winner for model 0 is high relative to the reference model (in their study À is set to a whole number ranging from 5 to 50). At first glance, this formulation does not fit the core loss equation format introduced in Table 2. Our approach is the following: first, we make this into a single log ratio by making À an instance-level parameter
0 whenever Per [x) is less than zero and to the original À otherwise, Pz)
resulting in log Po Expanding this out to the full DPCP loss function, this results in the core loss equation:
Prer(yu PoVur x))+1
log
Pre | x))+1 Po(n | x))
which not multilinear when multilinearity, while also maintaining compositionality from Assumption 1, treat Prr(y 1 and Pa (y z)) separate probabilistic variables, Prezz (vw x) and Peo (yw r), respectively. This results in the core loss equation shown in Tables and 6.
order to ensure
Below we show the core semantic formula for DPOP, which, as noted before, makes small adjustment to the DPO se- mantics shown in T:ble 4:
P:= Implies(
And(Ref(xy,)) Refz(xy.)) M(x,;) )), (xx()) M(xyy ), (x(Xy).))
19


Reviews:
[{"review_id": "QhZkhnoeJ6", "reviewer": "Reviewer_fMQw", "summary": "The work attempts to structure the corpus of existing and discover new optimization losses for direct preference alignment (DPA). To this end, existing DPA methods are unified and cast as a reasoning problem. Namely, each loss corresponds to a set of logic formulas that are optimized via weighted model counting (WMC) under extended preference structures. Logical entailment of the formulas leads to a lattice of losses and possibly new losses where first empirical evaluations are promising.\n\n## update after rebuttal\n\nI want to thank the authors for their insightful and comprehensive response. All major concerns have been resolved. However, the clarity is still not quite where it could be, for which reason I maintain my score of 3 (weak accept).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we are encouraged that you found our approach to be “very original and promising”. We have already taken steps to improve the presentation in the places you mention and we will address your particular points below. \n\n> To me, the start of Sec. 3 is harder to read than necessary. It might help to introduce the role of  β a bit earlier and move the information from the caption of Tab. 2 (\"All losses ...\") to the main text to make it self-contained.\n\nWe will address this in the revised draft. \n\n> Fig. 1: DOP2 could also be named as such on the right-hand side.\n \nWe have already modified Figure 1 according to your suggestions and in addition: 1) tried to make the figure easier to read and less blurry and; 2) we expanded the caption to more clearly state the problem and the goals of the paper. \n\n> Fig. 1 talks about \"compilation\" and \"derivation\"\n\nThank you for noticing this. We will fix it to be more consistent. \n\nAlso, we restructured part of Section 4 to make clearer the precise relationship between compilation and derivation (i.e., that we treat the former as the inverse of the latter), which we hope better motivates why we immediately start in Sec. 4.1 by talking about the “Compilation to semantic loss”. \n\n> Fig. 3: The entire figure is a bit unclear, and it personally confused me more than it helped. I\n\nAs with Figure 1, we did a complete revamp of this Figure, which we hope addresses your confusion and makes the modifications you suggest. \n\nTo state the role of Figure 3 more clearly (as we do in our updated draft): Preference structures, via Prop. 2, can be equivalently expressed in terms of *two* Boolean functions, $P_w$ and $P_l$, which correspond to the checkmarks and xmarks in the figure respectively (and are also equivalent to the formula forms in Eq. 4). We think that this view is helpful for: 1) understanding the corresponding model counting problem visually; 2) understanding how our generalized formulation of semantic loss, as captured by the equation in this figure, is general enough to capture things like conditional probabilities (i.e., not satisfying $P_C$, as captured by the white boxes in Fig 3; we will update the arrow from $P_C$ in Fig 3 to point to such white boxes without any checkmark or xmark).\n\n> \"We assume that all preference loss functions have an internal logic that can be expressed in the form described above.\" To what degree is this a limitation of the approach? What happens if that is not the case and we apply the translation regardless?\n\nTo clarify this claim, which is quite general, we assume that all preference losses have *some* internal logic that can be expressed in a discrete form. Whether or not they can all be expressed in the logic we propose is a separate question (e.g., it might be that some preference losses outside of our study require a more complex logical system beyond propositional logic. We will clarify this point, but we think it is an important working assumption to note).\n\nFor the preference losses under consideration in Table 2 (which cover many of the most popular DPA losses), Thrm. 2, however, does establish that our decompilation procedure and logic correctly captures the logic of these losses in the following sense (which we will make more clear): our decompilation procedure can take any of these losses as input and produce a semantic representation that can be compiled back into exactly and uniquely that loss via our logic. \n\nThe main condition that needs to be met for the decompilation procedure to be applicable is that the input loss equation is a “disjoint multilinear polynomial” as defined in line 346 (regarding your other point, we see that a non-inline version of this would be helpful here to make this point more clear). Of course, if the input loss does not follow this polynomial form the translation runs the risk of not being correct, but such losses are out of scope for this study and we could imagine making the translation more complex to expand this to other polynomial classes. \n\n>How is, intuitively, the implication of the left-hand side of, say Fig. 1, realized by the loss formula?\n\nYes, we agree that a clear intuition here would be helpful. Given the way that our implication construction works (i.e., the construction in the proof of Prop.2 which drives Algorithm 1 and our decompilation procedure), the left side of the implication in semantic representations like those in Figure 1 (or the representations in Table 4) corresponds to the lower/bottom part of the log ratios in the preference losses (Table 2, $\\rho_{\\theta}^{b}$) and the right side to the upper part of these log ratios ($\\rho_{\\theta}^{t}$). \n\n> The definitions are mostly given inline in the text\n\nWe will fix this, particularly by considering making some of the core definitions that our formal results rely on (non-inlined) formalized definitions.", "claims_and_evidence": "The main claims are that most common DPA losses can be cast as optimizing probabilistic logic formulas. This is appropriately substantiated by the presented constructions and proofs (see also \"Theoretical Claims\"). However, it is unclear which of the many DPA methods are covered by the new perspective introduced in the work.", "methods_and_evaluation_criteria": "The presented empirical findings are but a first glimpse of what can be explored, but sufficient to show that the approach is fundamentally feasible.", "theoretical_claims": "The paper contains various definitions, theorems, and proofs of them. The definitions are mostly given inline in the text, which is legitimate but slightly hinders clarity.", "experimental_designs_or_analyses": "The experiments are very minimal, yet sufficient for showing the basic feasibility of the approach. A lot of future work could extend this.", "relation_to_broader_scientific_literature": "I cannot judge the completeness, yet I did not find any missing spots.", "essential_references_not_discussed": "I am not aware of any.", "other_strengths_and_weaknesses": "The presented work uses well-known tools from different sub-disciplines (logic/neuro-symbolic reasoning) to structure the jungle of DPA losses. This very original and promising approach leads to an improved understanding of existing works and possibly new principled discoveries. It is original, yet the writing could be clearer in various places (see, e.g., the next section).\n\nThe lack of clarity regarding when the method is applicable, and the description and verification of the method itself are the main weaknesses of the work.", "other_comments_or_suggestions": "Comments on clarity and minor opportunities for improvement:\n- To me, the start of Sec. 3 is harder to read than necessary. It might help to introduce the role of $\\beta$ a bit earlier and move the information from the caption of Tab. 2 (\"All losses ...\") to the main text to make it self-contained.\n- Fig. 1: DOP2 could also be named as such on the right-hand side.\n- Fig. 1 talks about \"compilation\" and \"derivation\", while other sections, such as 4.1. (any others), talk about \"compilation\" and \"decompilation\" instead. This should be unified.\n- Fig. 3: The entire figure is a bit unclear, and it personally confused me more than it helped. It might make sense to remove it, as WMC is well-known and can be learned about from other resources. Otherwise, why does $\\checkmark$ both correspond to $P$ and tp $\\bar{P_f}$, and similarly for the inverse. The meaning of empty cells in the table is also unlcear to me.\n- Some columns in Tab. 5 are missing highlights.\n- Using page 16 for a single `)` is avoidable.", "questions_for_authors": "- **Q1**: In Sec. 4, l. 172ff, you state, \"We assume that all preference loss functions have an internal logic that can be expressed in the form described above.\" To what degree is this a limitation of the approach? What happens if that is not the case and we apply the translation regardless? Generally, can you more clearly state the conditions needed to apply the translation from loss to your symbolic representation?\n- **Q2**: A crucial piece for the understanding of the translation is missing: How is, *intuitively*, the implication of the left-hand side of, say Fig. 1, realized by the loss formula? Some discussion along the lines of \"if $\\pi_\\theta(x, y_w)$ is high, $\\sigma$ saturates, and then ...\". This would help to understand the compilation/derivation steps and serve as a good sanity check of the method/as a case study.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "14y7cyxH1u", "reviewer": "Reviewer_mbGz", "summary": "The paper proposed the decompilation of loss functions such as DPO into symbolic programs. More specifically, the authors present how to derive probabilistic propositional logic programs that can, in turn, be manipulated and compiled into potentially novel and improved losses for preference alignment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback; as noted in our response to the above reviewer, we’ve already taken steps to improve the presentation, which we hope will also address your concerns. \n\nWe are encouraged that you nonetheless found our approach to be “compelling” and an “interesting avenue to explaining, manipulating, or exploring losses”\n\nBelow we address your particular comments and questions. \n\n> the experiment mostly shows a basic usage example without providing deeper insights into the method or discussion on limitations.\n\nOur main contribution is to provide clarity into the ever-growing space of preference losses. The symbolic view of these losses allows us to make the preferences encoded by them explicit, understand relationships between them, and, via counting arguments, enumerate the space of all possible losses with or without a reference model. Moreover, it also provides a conceptual framework for inventing novel kinds of losses.\n\nWe also note that much of our experimental results were pushed to the Appendix due space issues (see please our response to JCXg). If accepted, we intend to use the additional page to incorporate such results in the main paper. \n\n> \"In particular, we focus on approaches based on probabilistic logic.\". The next sentence disagrees, \"In contrast, we focus on the inverse [...]\".\n\nSorry for the confusion, we will modify this for clarity. \n\nTo clarify: the point about this “inverse problem” is that while most people in the neuro-symbolic field have focused on the problem of “compilation” (i.e., translating symbolic formulas into loss functions by interpreting those formulas using probabilistic logic), we focus on the “inverse” and more unique problem of “decompilation” (i.e., deriving symbolic formulas for *existing* loss functions that we also interpret in terms of probabilistic logic).\n\nIn both cases, probabilistic logic is used as the ingredient that makes communicating between these symbolic forms and losses possible, which is why we say that our approach is “based on probabilistic logic” (we felt that this was important to clarify since other popular modes of translation, such as fuzzy logic, could have been used here as an alternative to the probabilistic approach). \n\n>On page 3, line 146, \"No reference\" is in bold text, this might be by mistake.\n\nWe will fix this inconsistency. \n\n> Figure 1 illustrates the core idea of the paper well, but the concepts within the symbolic program may be difficult to parse for a first-time reader on page 1\n\nAs noted in the response to **fMQw**, we completely revamped this figure to make it more clear along the lines that you suggest. \n\n>The abstract employs the abbreviation DPO without introducing it beforehand..\n\nThank you for catching this, we will change DPO to “Direct Preference Optimization”.\n\n> Figure 1 illustrates the core idea of the paper well, but the concepts within the symbolic program may be difficult to parse for a first-time reader on page 1. Perhaps additional annotation or a simplified illustration/example would be more digestible at this point in the paper.\n\nPlease see our response above. \n\n> The paper's presentation suffers from how information is distributed throughout the manuscript. For example, on page 5, \"Decompilation into semantic loss\" is described but references/requires insights from Table 2 (page 3), Section 5.2 (page 7), and Table 6 (page 11) to be understood.\n\nWe updated our draft to account for these structural issues.", "claims_and_evidence": "The paper claims to (i) introduce formal insights into and characterization of DPA losses through their decompilation into symbolic programs and (ii) practical insights into effective searches for novel DPA losses that improve over the state-of-the-art. Although compelling, the work does not convincingly support the claims, as it lacks clarity in the formal presentation and only provides minor practical insights on feasibility.", "methods_and_evaluation_criteria": "The proposed method seems plausible to improve preference alignment techniques through the interpretation and manipulation of losses on a symbolic level. However, further experiments and discussions are required to evaluate the approach fully.", "theoretical_claims": "Although the paper contains theorems (one of which is followed by a proof paragraph, the other is not), the paper overall is lacking in clarity, which hinders thorough checks on the contributions' correctness.", "experimental_designs_or_analyses": "Compared to the paper's claims, I found the experiment design lacking in its scope and discussion. Although the authors state \"While these experiments are small scale and limited in scope, they are merely meant to suggest possible uses our framework and open questions.\" (page 14, line 716), the experiment mostly shows a basic usage example without providing deeper insights into the method or discussion on limitations.", "relation_to_broader_scientific_literature": "This work explores the decompilation of losses employed in preference alignment into probabilistic propositional logic. Hence, it relates to literature aiming to guide the training of deep models such as LLMs to align with some additional human preferences. Furthermore, the chosen representation of the decompiled losses relates to probabilistic logic programming (and thereby methods of statistical relational and neuro-symbolic AI), employing weighted model counting over the automatically generated symbolic representations of the loss.", "essential_references_not_discussed": "Essential references have been discussed.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces a compelling method for the decompilation of loss functions into probabilistic propositional logic (and vice versa), opening an interesting avenue for explaining, manipulating, or exploring losses.\n- The research direction of the paper is important and approached in an interesting and original way.\n\nWeaknesses:\n- The paper contains some confusing wording that needs to be improved. For example, at the end of 'Neuro-symbolic modeling,' the authors first state, \"In particular, we focus on approaches based on probabilistic logic.\". The next sentence disagrees, \"In contrast, we focus on the inverse [...]\".\n- Some figures, e.g., Table 2, are visually challenging to parse, so I recommend reworking them for clarity. Others, like Tables 1, 3, and 4, are less problematic but may similarly be improved.\n- Some statements made in the paper need clarification. For instance, on page 3, the authors write, \"We use θ2 and ref2 to refer\nto copies of our two models, which is a decision that we address later [...]\". At this point in the paper, the statement left me somewhat puzzled in its meaning.\n- The paper's presentation suffers from how information is distributed throughout the manuscript. For example, on page 5, \"Decompilation into semantic loss\" is described but references/requires insights from Table 2 (page 3), Section 5.2 (page 7), and Table 6 (page 11) to be understood.\n- Along the lines of the previous comments, the paper could benefit from some restructuring. As an example, rather than leading into the experiments, Section 6 \"Results and Discussion\" begins with two more theorems, only one of which is followed by a proof paragraph.", "other_comments_or_suggestions": "- On page 3, line 146, \"No reference\" is in bold text, this might be by mistake.\n- Figure 1 illustrates the core idea of the paper well, but the concepts within the symbolic program may be difficult to parse for a first-time reader on page 1. Perhaps additional annotation or a simplified illustration/example would be more digestible at this point in the paper.\n- The abstract employs the abbreviation DPO without introducing it beforehand.\n- The use of bold text for both sectioning and underlining important concepts/keywords may be suboptimal.", "questions_for_authors": "I have no questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnKi8U7iUh", "reviewer": "Reviewer_pLhs", "summary": "This paper proposes a novel framework to unify various preference optimization losses as a logical program of response orders. A pairwise preference implies the logic that a rejected response in the policy shall imply that the chosen response is in the policy; A supervised loss implies that the response is favored by the policy. \n\nThe proposed framework characterizes each preference optimization loss function as such as logical program. On the other hand, they provide an mechanism to remap a logical program to a corresponding loss function. By exploring in the space of logical implications, they essentially explore in the space of optimization losses.\n\nIn addition to drawing the relationship between losses and logical constraints, their framework provides an interesting roadmap to understand the relationship between different losses.\n\nFinally, they provide a proof-of-concept experiment showing that their approach has the potential to discover better preference optimization losses.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we are pleased that you find our approach to be “a valuable addition to the relevant science literature” \n\n>many relevant papers which incorporate \"scalar reward signals\" in the training objective. With the fine-grained reward information, those methods tend to outperform methods relying on binary preference labels. \n\nThank you for these pointers. This does look like an exciting new area where we can try to apply our techniques. While we haven’t digested yet all the details of these papers, it does seem like the kind of distillation objective in Fisch et al. 2025 (Eq. 7) could fit into our framework by adding these additional reward model estimates into our logical formulas as additional predicates (we will think more about this and, if appropriate, mention this direction in an updated draft).", "claims_and_evidence": "This paper is mostly a theoretical paper that provides a mathematical framework between preference optimization loss and logical implications. I have followed most of its derivations in the paper, which look good to me.", "methods_and_evaluation_criteria": "The proposed method is a brand new viewpoint of preference optimization losses. The most interesting finding is that most of the preference optimization losses are a combination of \"semantic log ratios\". Each semantic log ratio corresponds to a certain logic constraint, which are then combined together in the overall loss function. It is a very interesting framework to shed more light on understanding preference optimization algorithms. \n\nBeyond its main mathematical contributions, the paper shows a proof-of-concept experiment demonstrating some new losses found in their framework. While the experiment is only minimal, this framework provides the potential to dynamically search for the best loss function, similar to how neural architecture search improves neural network performances.", "theoretical_claims": "I can follow the logical flows in the paper. But I didn't check the proofs carefully.", "experimental_designs_or_analyses": "This paper involves a minimal proof-of-concept experiment only. If there are more compelling experiments showing some really strong losses, it will help the paper more.", "relation_to_broader_scientific_literature": "This paper is a valuable addition to the relevant science literature. In the literature, the preference optimization landscape has been very complicated, due to the combination of many design factors like loss function and reference policy. This paper provides a novel interpretation for those algorithms and a roadmap to connect the scattered dots.", "essential_references_not_discussed": "Not necessarily essential references, but there have been many relevant papers which incorporate \"scalar reward signals\" in the training objective. With the fine-grained reward information, those methods tend to outperform methods relying on binary preference labels.\n\nIt would be a strong addition if this framework could include these methods into the roadmap as well. Some relevant papers are follows.\n\n[1] RPO. Nemotron-4 340B Technical Report; \n[2] Distill DPO. Robust Preference Optimization through Reward Model Distillation.\n[3] InfoNCA. Noise Contrastive Alignment of Language Models with Explicit Rewards.\n[4] Brain: Bayesian reward-conditioned amor- ´ tized inference for natural language generation from feedback.", "other_strengths_and_weaknesses": "This paper is well written. The logic is easy to follow.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vgYSWe9pzt", "reviewer": "Reviewer_JCXg", "summary": "This paper presents a fresh perspective on common loss functions in the rapidly growing direct preference optimization literature. In particular, by translating loss functions into symbolic expressions, the paper offers a principled way to analyze their semantics. This approach makes it easier to understand relationships between different DPO-style algorithms, in particular by revealing a hierarchy based on the amount of constraints imposed by each individual algorithm.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you, we are excited that you find our work “really exciting”, below we address your comments and concerns. \n\n>However, while the paper briefly mentions new loss variants, it lacks a strong empirical demonstration\n\nPlease see our response below. Much of our experimental results were pushed to the appendix due to space limitations.  If accepted we intend to use the extra page to include more empirical results in the main paper, including the extra experiments we mention below.   \n\n> computational complexity of WMC\n\nIndeed, WMC is a hard algorithmic problem, but we note that in our study this complexity is side-stepped since we are working with problems with a small number of variables (2-4). In such cases, one can simply write the WMC formulas in full and simplify them using algebra offline, which is how we obtained the new losses in App.D (we updated the draft to include details of how to compute this using SymPy).\n\nNonetheless, the probabilistic reasoning community has devised various “knowledge compilation” techniques that allow one to scale WMC to much larger problems by compiling these logical representations into tractable circuits, which make many problems feasible in practice (and sometimes in theory). We are confident that if we were to greatly expand the complexity of our programs, such techniques, or other advanced SAT techniques, could be leveraged for doing the involved experiments.   \n\n>Have you discovered any novel loss functions that outperform existing approaches, and more importantly, a mechanism \n\nYes, we found in the experiments reported in Table 5 that the novel loss, $\\ell_{cCPO}$ from Figure 4 outperforms the known loss $\\ell_{CPO}$ in a win-rate study adapted from Meng et al 2024. This does show evidence that our theory is able to yield competitive new losses (we have subsequently run these same experiments at larger scales using a >3x larger Smollm-1.7B model and are seeing similar trends; we intend to report these results in the updated draft pending the full results).\n\nIn terms of mechanisms, we think our experiments do give insight into this, as noted below.  \n\n– We find that the logical constrainedness of a loss function is an important contributing factor to its empirical success. For example, the highly unconstrained losses in Fig.4 tend to have spurious behavior due to the nature of their semantics and how the underlying constraints can be satisfied (see discussion in E.1), which we can see clearly in the behavior of their win/lose log probabilities and training dynamics shown in Figure 8 (such empirical behavior is something we consistently see across the many experiments we’ve run at different scales). \n\nSince DPO was introduced, there has been much puzzlement about the empirical behavior of log probabilities during training, which we think our constraint satisfaction view of the problem can help to elucidate. \n\n– We also see in Table 5 that losses perform differently on different subsets of data, which is a trend that we’ve seen persist through our later experiments. We take this as evidence that different tasks and datasets involve different semantics, requiring one to carefully tailor their losses to those semantics. \n\n> How do we then navigate such a huge space\n\nWe tried to address this in “How is the loss space structured” (the right column of line 287). While the space is indeed large, we can see through results such as Proposition 3 that loss behavior is linked in interesting ways to the logical semantics of the losses, which can be exploited for exploration. The strategy we pursued in our case study was the following: start with an empirically successful loss (e.g., CPO), formalize its semantics, then modify its systems to find new losses that are either more constrained (or that entail CPO) or less constrained (that are entailed by CPO) and experiment accordingly. \n\nWe think strategies like this are useful tools for navigating this space. \n\n> My main concern is regarding the notion of using an absolute value\n\nThe \\epsilon notion is only meant to be a tool or heuristic to conceptually think of the distribution as digitized or divided into “valid” and “invalid” outputs; it is not a value that we model explicitly or that plays any role in our formal analysis, and hence we do not make any assumptions about it (e.g., it needn’t be a fixed value). \n\nRegarding your example: if we interpret the top symbolic formula in Figure 2 in terms of this digitized distribution, this just says that “whenever we find the loser to be a valid generation (i.e., to be above this \\epsilon) we should always find the winner to also be above this line (\\epsilon) too”. Note that this semantics, which underlies many DPA approaches, does not rule out the possibility that the loser is a “valid” generation (or also above \\epsilon). So for your example of `(x, y_1, y_2) (x, y_2, y_3)` we could set this hypothetical \\epsilon to be the probability `y_3` while still satisfying this constraint.", "claims_and_evidence": "The paper makes two claims:\n\n1- the translation into discrete reasoning allows us to better understand the differences between recent proposals in DPO-style algorithms. This is well-supported.\n\n2- the second claim is that this Logic perspective allows us to derive and develop new DPO-style algorithms. While some preliminary signals are presented, this claim is not fully supported, as the evaluation for the new algorithms feel quite limited, on a single benchmark and with limited comparison with existing algorithms and also limited insights found. I think the key question about this paper remains what authors ask on the last page: Can we find empirically improved losses using this method?", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The finding about the large space of algorithms being possible is really exciting. However, while the paper briefly mentions new loss variants, it lacks a strong empirical demonstration of how these losses improve performance over existing algorithms.", "relation_to_broader_scientific_literature": "Instead of focusing on empirical improvements, this work formalizes DPA loss functions using symbolic logic.\nThe work introduces preference structures, which categorize loss functions using logical relationships rather than just their optimization properties. This builds on the trend of moving beyond black-box RLHF and into more interpretative framework.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "My main concern is regarding the notion of using an absolute value of \\epsilon for determining \"valid\" model predictions. This seems somewhat ambiguous. If a winner in one preference pair is a loser in another, how does the framework handle such inconsistencies?\n\nIn particular suppose that we have:\n\nx, y_1, y_2\nx, y_2, y_3\n\nin our dataset, which in fact can be fairly common. What would be the value of \\epsilon in this case?", "other_comments_or_suggestions": "The paper states that the number of definable preference structures is doubly exponential in the number of model predictions. While this highlights a rich space of potential loss functions, it also raises concerns about how efficiently we can explore this space. Having read the paper, it is hard to identify a practical approach for navigating this space.", "questions_for_authors": "The paper identifies that the number of definable structures is in fact doubly exponential. How do we then navigate such a huge space so that we actually get a practical benefit from this insight?\n\nYou mention that the framework could help derive new DPA losses, but the paper primarily formalizes existing ones, with some limited observations about new algorithms. Have you discovered any novel loss functions that outperform existing approaches, and more importantly, a mechanism to understand which new algorithms could be more promising?\n\nWeighted Model Counting (WMC) is a core component of the framework for translating DPA losses into logical expressions. In light of the computational complexity of WMC how does this approach scale? What is the computational bottleneck in algorithm 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "meacVfDPIE", "reviewer": "Reviewer_zia1", "summary": "This paper introduces a novel approach to describing direct preference alignment (DPA) algorithms in terms of propositional logic. By generalizing the notion of semantic loss, the authors attempt to provide a formal framework to characterize differences between DPA variants. \nSubsequently, the authors leverage the introduced framework to discover improved loss formulations. \nExperiments on a small (0.5B params) LLM show that the proposed systematic comparison can produce better loss functions for DPA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging feedback. \n\n> How well can other researchers leverage the proposed framework to identify novel loss formulations beyond the 4 shown in the paper?\n\nWhile we reported experiments on 4 losses, we derived many more novel losses, including the 16 single model losses shown in Figure 7 each (excluding 1) with a novel DPO variant. We think that this set exhaustively captures the full set of single model losses that researchers would want to experiment with (it exhaustively shows all definable losses between CEUnl and unCPO), and offers many novel DPO variants that have not yet received empirical verification. We plan to release all the associated code and loss implementations to facilitate further work in this area. \n\nGiven that our framework now allows one to define losses using a much more expressive logical language, where any valid propositional formula constitutes a valid loss function, we also believe that this makes it possible to more easily devise entirely new classes of loss functions that would otherwise be difficult to derive working solely from the mathematics of DPO (e.g., losses that involve more complex forms of feedback, non-differentiable components). \n\nWe will include a concrete example of this in the updated draft. \n\n>Does this approach generalize to other models and larger parameter counts\n\nThese approaches can be generalized to any language model of any size or number of parameters; the size of the underlying language model is usually not a critical factor in the loss computation (see below). \n\n> Is there any significant computational overhead with the new loss formulations? (especially cCPO)\n\nNo. The cCPO loss is computed according to the equation given in Appendix D, and does not require any more computation than another of the other losses in that table. As above, computing losses is a relatively easy computation once the basic forward calls have been made to the model to obtain model output probabilities, which are done independently of the loss computation. \n\n>Related to the point above, Appendix E contains many important results to validate the proposed frameworks’ capability.\n\nThank you for this suggestion. If accepted we intend to use the additional page to move into the main paper some of the details of the experiments now in the appendix (these were not included due to space limitations), as well as subsequent experiments we did to further verify these findings (see our response to `JCXg`).", "claims_and_evidence": "The central claim of the paper is that the proposed framework allows for novel insights into relationships between existing DPA formulations. Additionally, they propose to leverage that framework to create new loss formulations in a structured manner. \n\nGiven that this is more of a theoretical paper the authors provide limited experiments. \n\nHowever, they verify their first claim by deriving a loss landscape (or lattice) that provides valuable insights into the interplay between different losses. \nGiven that landscape they also derive new loss formulation that outperform existing formulations.", "methods_and_evaluation_criteria": "- the theoretical results are strong and reasonably evaluated/demonstrated \n- The subsequent empirical experiments are decent but lack some depth since only one (and very small) LLM was considered", "theoretical_claims": "The theoratical claims and proofs appear to be sound but my confidence in that part of the review is low.", "experimental_designs_or_analyses": "As discussed above the theoretical analysis is strong and well rounded. \nOnly one actual experiment is performed, but the setup and choice of evaluation datasets is sound as well.", "relation_to_broader_scientific_literature": "The idea of using logic to describe different DPO algorithms is novel.\nAlthough the scheme itself may be largely covered by the original semantic loss paper (i.e. composing logic representations based on NNs outputs), the dedicated arguments and discussions for DPO algorithms would be valuable to the community.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "# Weaknesses\nWhile the paper provides interesting insights the main impact on the field remains somewhat opaque. \nFor example,  \n- How well can other researchers leverage the proposed framework to identify novel loss formulations beyond the 4 shown in the paper? \n- Does this approach generalize to other models and larger parameter counts", "other_comments_or_suggestions": "Related to the point above, Appendix E contains many important results to validate the proposed frameworks’ capability. I would encourage authors to include key insights in the main text as well.", "questions_for_authors": "- Is there any signficant computational overhead with the new loss formulations? (especially cCPO)\n- Can the introduced semantics over loss functions contribute to efficient search of loss functions (e.g. pruning redundant ones)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9pYLI01mS", "reviewer": "Reviewer_f9d6", "summary": "This paper proposes a symbolic method to interpret direct preference alignment (DPA) loss functions. Given a DPA loss, the proposed method translates it into a preference structure consisting of three formulae, which can be further used to construct a corresponding semantic loss. The paper further shows how existing DPA variants can be converted to semantic loss forms and explores the relation between these different losses. Finally, the paper proposes a simple study about how the formulation can be used to search for new DPA loss functions.\n\n## update after rebuttal\n\nI have read authors' rebuttal and other reviews. I would like to raise my score to weak accept.\nThe paper can benefit from the promised revisions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. \n\n> The translation is based on a set of fixed rules, making it unclear how general these rules are.\n\nSee comment below about WMC. \n\n> The algorithm uses rules in table 6, which covers P1*P2, (1-P1) and P1+P2. I wondered how general this is. For example, how can  sqrt(P1×P2) be translated?\n\nGiven that our analysis is based on WMC, the losses being decompiled must be expressible as the polynomial class defined by WMC; by construction, we restrict this to the *disjoint multinear polynomials* defined on line 348. Except for DPOP (see discussion below), this naturally captures all known preference losses, including those in Table 2, which we emphasize is a fairly comprehensive set of DPA losses (we updated our appendix to include additional DPO variants that fit this analysis), as well as the common loss functions such as cross-entropy and unlikelihood. \n\nThe square root clearly does not fit this equation class, so it is not covered by our analysis. Is there a particular loss involving a square root that you have or mind, or some other salient loss that seems out of scope? \n\n> The method is limited to the \"main\" component of the loss, disregarding the regularisation terms. Regularisation terms usually play important roles in those preference optimisation losses. So while I believe this method is able to interpret part of the semantics behind the loss, it may also ignore important aspects.\n\nThis is an important point, which we will say more about in the updated draft. We note, however, that the decision to avoid regularization terms in our analysis follows other formal studies such as by Tang et al. (2024). We also found cross-entropy terms, even when removed, to have limited impact on empirical performance, which is a finding consistent with [1]. \n\n[1] Hanyang Zhao et al. RainboPO: A Unified Framework for Combining Improvements in Preference Optimization\n\n> Special treatments are required for some DPA loss variants (e.g., DPOP).\n\nThis special treatment is due to the compositional constraint from Assumption 1 (inspired from programming semantics), which stipulates that every model prediction in a loss should be directly accounted for in the semantics. *For clarity*: the DPOP loss has terms such as $p_{\\theta}(y_{w} \\mid x)^{k}$ and $p_{ref}(y_{w} \\mid x)^{k}$ with exponents $k$ that make the equation not multilinear when $k > 1$ (without loss of generality we only considered cases where $k=1,2$).  Our decision to make them multilinear when $k >1$, by creating new variables $p_{\\theta2}(y_{w} \\mid x)$ and $p_{ref2}(y_{w} \\mid x)$ is based on a common polynomial transformation for making polynomials multilinear, which in the end maintains compositionality. Importantly, as we also detail in Appendix F, the value of $k$ also varies from the instance to instance, so both $p_{\\theta2}(y_{w} \\mid x)$ and $p_{ref2}(y_{w} \\mid x)$ do not have a fix value (i.e., they either can take the value of the other variables or be equal to 1 when $k=1$), which we think justifies treating them as semantically distinct values. \n\nWe will discuss this more in the draft, including making this more clear when we first reference DPOP around lines 140 (second column). We acknowledge that such exponents could be treated differently, e.g., as parameters that are part of the probability computation for each variable similar to how length normalization is computed, which would sidestep completely these issues involving multilinearity. \n\n> Furthermore, it is unclear to me how the proposed formulation can help explain the performance of different DPA variants, although from figure 4 there are indeed relations and differences in the semantics of these variants.\n\n\nAs we discussed in our response to JCXg , Proposition 3 establishes that semantic entailment induces certain monotonicity properties w.r.t to the behavior of the compiled losses, and provides us with a notion of the relative constrainedness of losses that is grounded in loss behavior. As discussed in Sec.E.1, the relative constrainedness of a loss is a property that we think has an important impact on its empirical performance, which is explained by our framework and observable when looking at training dynamics as shown in Figure 8. \n\nWe moved this analysis into the appendix due to space issues. If accepted, we intend to include some of this empirical analysis in the main paper (including some additional experiments that scale the reported ones).", "claims_and_evidence": "**Claim: **\n\n**Claim: we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to rigorously characterize the relationships between recent loss proposals.**\n\nThis claim is supported by the theoretical analysis of the paper on existing DPA loss and their relations.\n\n**Claim: (the finding also makes it possible) to systematically explore the landscape and derive new loss functions from first principles.**\n\nThis claim is not fully supported. Although the paper proposes a limited study in table 5, the found loss term does not exhibit significant advantage over existing ones. \n\nFurthermore, it is unclear to me how the proposed formulation can help explain the performance of different DPA variants, although from figure 4 there are indeed relations and differences in the semantics of these variants.\n\nFurthermore, analysis on DPOP in Section F suggests that ad-hoc treatment is needed for DPA loss with \"non-standard\" forms. This undermines the generalisability of the proposed method as an explanation and exploration tool.", "methods_and_evaluation_criteria": "**Formalising DPA loss terms as semsntic loss**\n\nThe method is interesting. However, it seems that the method is limited to the \"main\" component of the loss, disregarding the regularisation terms. Regularisation terms usually play important roles in those preference optimisation losses. So while I believe this method is able to interpret part of the semantics behind the loss, it may also ignore important aspects.\n\n**Algorithm 1: translation of loss to logic**\n\nThe algorithm uses rules in table 6, which covers P1*P2, (1-P1) and P1+P2. I wondered how general this is. For example, how can $\\sqrt{P1 \\times P2}$ be translated?", "theoretical_claims": "I did not check all the theoretical results, but the main results in proposition 2, lemma 1, theorem 2, and Table 4, seem correct.", "experimental_designs_or_analyses": "There are not many experiments.\n\nTable 5 presents an empirical study on how a new loss, i.e., $L_{cCPO}$, performs compared with existing loss terms. The performance of the found loss is mediocre and the experiment overall does not convince me that this method is able to help with loss design.", "relation_to_broader_scientific_literature": "This paper is a symbolic interpretation of preference optimisation losses for value alignment. The topic is important.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengthes:\n\n- This paper proposes a formulation of DPA loss in terms of symbolic representation-based semantic loss. From many existing loss terms, the proposed method draws interesting insights about their semantics.\n\nWeaknesses. Please check the review above, I repeat some important points below.\n\n- The paper lacks empirical or theoretical study on the applicable extend of the proposed method. The translation is based on a set of fixed rules, making it unclear how general these rules are.\n\n- Special treatments are required for some DPA loss variants (e.g., DPOP).\n\n- The paper does not provide many insights on different performances of loss variants, nor does it provide insights on how to design new DPA loss. I wondered how this method can practically benefit preference optimisation studies.", "other_comments_or_suggestions": "Please refer to my points above.", "questions_for_authors": "Please refer to my points above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a formal framework to analyze and understand Direct Preference Alignment (DPA) algorithms, such as Direct Preference Optimization (DPO), which are used to align large language models (LLMs) with human preferences. The authors propose a novel formalism, a kind of logic for preference modeling, that characterizes preference losses for single- and reference-model-based approaches by deriving symbolic expressions that encapsulate their semantics. This enables a systematic comparison of different DPA loss functions as well as the derivation of new loss functions. By formalizing DPA losses in terms of discrete reasoning problems, the paper sheds light on the structure and relationships within the DPA loss landscape. Overall, the paper is very well written and presented. The reviews are a bit mixed with 3x weak reject, 2x weak accept, 1x accept, i.e., a borderline overall. However, two of the reviews were asked as emergency reviews, which I will factor in. One of the weak rejects says "fresh perspective on common loss functions" while it "lacks a strong empirical demonstration" of finding new loss variants. Some of them can be found in the appendix.  The other weak reject also states " The topic is important." So overall, while there are indeed some downsides, overall the reviewers agree that the work is interesting and important and lean towards the accept side. Consequently, and since many of the issues raised have been addressed, I vote for accept.
Decision: Accept (poster)