Humanline: Online Alignment as Perceptual Loss
Authors: Sijia Liu
Niklas Muennighoff
Kawin Ethayarajh
Venue: ICLR 2026 Poster (2026)
Keywords: alignment
LLM
LLM alignment
prospect theory
perceptual loss
behavioral economics
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Online alignment objectives (e.g., GRPO) mimic how humans perceive probability. By modifying offline objectives to do the same, we can match the performance of online alignment with offline off-policy data, giving us the best of both worlds.

Abstract: Online alignment (e.g., GRPO) is generally more performant than offline alignment (e.g., DPO)---but why? Drawing on prospect theory from behavioral economics, we propose a human-centric explanation. We prove that online on-policy sampling better approximates the human-perceived distribution of what the model can produce, and PPO/GRPO-style clipping---originally introduced to just stabilize training---recovers a perceptual bias in how humans perceive probability. In this sense, PPO/GRPO act as perceptual losses already. Our theory further suggests that the online/offline dichotomy is itself incidental to maximizing human utility, since we can achieve the same effect by selectively training on any data in a manner that mimics human perception, rather than restricting ourselves to online on-policy data. Doing so would allow us to post-train more quickly, cheaply, and flexibly without sacrificing performance. To this end, we propose a design pattern that explicitly incorporates perceptual distortions of probability into objectives like DPO/KTO/GRPO, creating $\textit{humanline variants}$ of them. Surprisingly, we find that these humanline variants, even when trained with offline off-policy data, can match the performance of their online counterparts on both verifiable and unverifiable tasks.

Published as a conference paper at ICLR 2026
HUMANLINE:
ONLINE ALIGNMENT AS PERCEPTUAL Loss
Sijia Liu
Niklas Muennighoff Stanford University
Princeton University
sijia. www.uss.ppicnctooo..oooo edu
muennighoff@stantee
Kawin Ethayarajh
University of Chicago
kawin@uchicago..dd
ABSTRACT
Online alignment (e.g., GRPO) is generally more performant than offiin align- ment (e.g., DPO)-but but why? Drawing on prospect theory from behavioral eco- nomics, propose human-centric explanation. We prove that online on-policy sampling better approximates the nuraa-perceived distribution of what the model can produce, and PPO/GRPO-StyL lize training Maining-recovers a perceptual bias in how humans perceive probability. In this sense, PPO/GRPO act as perceptual losses already. Our theory further sug- gests that the online/offline dichotomy is itself incidental to maximizing human utility, since a manner that mimics human perception, rather than restricting ourselves to on- line on-policy data. Doing so would allow us post-train more quickly, cheaply, and flexibly without sacrificing performance. To this end, we propose design pattern that explicitly incorporates perceptual distortions of probability into objec- tives like DPO/KTÖ/GRPO, creating humanline variants of them. Surprisingly. we find that these humanline variants, even when trained with offline off off-policy data, can match the performance of their online counterparts (on both verifiable and unverifiable tasks) while running up to 6x faster.
Soopiig--rigiinll introduced to just stabi-
achieve the same effect by selectively training on any data in
INTRODUCTION
Aligning generative models with feedback-from a human, a learned reward model, ground- truth verifier-is an increasingly important part of post-training, with methods categorized as offline off-policy (e.g., DPO, KTO) or online on-policy (c.g., GRPO). Despite flurry of initial optimism around the former, recent work concurs that the latter have higher performance ceiling. though they come the cost of more compute, training time, and instability (Xu et al.) 2024b) Ivison al.) 2024). But why are they better? Explanations range from online methods having better data coverage (Song et al., 2024). emphasizing generation over discrimination (Tang et al. 2024a). and navigating simpler search space over policies (Swamy et al. 2025).
Although all these explanations have merit, we argue that if the goal is to maximize a model's util- ity to humans, then the dichotomy itself is incidental. We start with prospect theory, framework in behavioral economics that explains why humans make decisions about random variables that do not necessarily maximize their expected value (Tversky & Kahneman 1992). Classically. the random variable would describe a monetary outcome, measured in dollars; when extended to gener- ative modeling, it describes the goodness of outputs, measured in bits/nats (Ethayarajh al. (024)) Prospect theory offers a well-defined and empirically validated model of the subjective probability distribution that humans implicitly assign outcomes. As we will show, compared random of- fline data, online on-policy sampling better approximates the prospect theoretic distribution of what the model can produce, offering centrccentric explanation for why online alignment should be more performant ($3).
"qqual contribution. Work started while the last author was at Princeton Language & Intelligence.

---

Published as a conference paper at ICLR 2026
Offline
Offline + Humanline
Online
Online Humanline
Length-Conten Ceath-Contooledd Winrate (%)
30
28
+1.4x
(GPT 4.1 4.1-as-a-Jil 1-as-a-judge
25
111 1x
22
20
18
15
before allgnment
12
10
DPO
KTO
GRPO
Figure 1: On instruction-following L1 al 8B- Instruct aligned with online on-policy data (blue) is 1.3x to 1.6x better than one aligned with offline off-policy data (red). However, when the same offline data is fed to the humanline variant of the objective (orange), the gap vanishes.
However, this also suggests that online on-policy data is suboptimal its own, it reflects what the policy is literally capable of producing, as opposed to what humans perceive it is capable of: for example, people systematically overestimate the chance of extreme outcomes and underestimate the chance of typical ones. We then prove that SPO/GRPO-style clipping clipping-originally originally introduced to just stabilize training (Schulman et af.) (9017)iimliccite recovers a special case of this perceptual bias, as formalized in prospect theory. In other words, tfte-of-thhe-a alignment methods are, to some extent, perceptual losses already ($4).
If the success of PPO/GRPO can be ascribed to them being perceptual losses, then we do not nec- essarily need online on-policy data: we can source data from anywhere-onling offline, on-policy, off- off-policy-anc and selectively from anywhere while not sacrificing performance, then state-of- ofttc----the-rrt art post-training becomes much faster and cheaper. To this end, we propose design pattern for creating variant of most alignment objectives (including DPO, KTO, and GRPO) that explicitly incorporates these perceptual distor- tions of probability while keeping the rest of the pipeline intact. This amounts to: (1) syncing the reference model with the previous version of the policy at the end of k steps; (2) asymmetrically clipping the log-probability ratio of each token upstream of the loss. These simple changes, when applied correctly, create what we call the humanline variant of the original objective.
in manner that reflects human perception. If we source data
We consider two testbeds ($5). where an LLM is aligned to better at: (1) instruction-folloviii with unverifiable rewards; (2) mathematical reasoning, with verifiable feedback. On instruction- following, an LLM trained with the online variant of DPO/KTO/GRPO has 1.3x to 1.6x higher winrates against frontier model than one trained with the offline variant of the objective; when the same offline data is fed to the humanline variant, the gap with online alignment vanishes (Figure []. Even on mathematical reasoning, where human utility is seemingly irrelevant, humanline GRPO allows training data to be sampled up to 64x less frequently without performance degradation.
We do not claim that can match the performance of online alignment by simply applying the humanline variant to any offline data; data quality always matters. Rather, what we find is that if the data has sufficiently high average token likelihood under the reference model at the start of training, then using a humanline variant can fully bridge the gap with online alignment-thin this an empirical regularity that follows from our theory. Humanline variants offer the flexibility source good- quality data from anywhere, which has the potential to not only make post-training many times faster and cheaper, but also make models much more adaptable to new tasks and populations.
BACKGROUND
We provide high-level overview of alignment methods and leave more detailed survey to Ap- pendix A For the sake brevity. we will at times refer to the online (o-folicy off- off-policy) variant of method as the online(offline) variant, in line with the literature. In most alignment al-
2

---

Published as a conference paper at ICLR 2026
gorithms, including all those discussed in this paper, two copies are made of our initial model: reference model Tee that serves as an anchor, whose weights are not backpropagated through; and a policy Th whose parameters 0 are updated minimize the loss.
Online On-policy Alignment Samples are drawn from the policy, labeled with feedback-frin learned reward model, ground-truth verifier, etc.-and fed into a loss function, which is minimized by updating 0. This is done iteratively until the desired level of progress has been made, with the reference model periodically synced with the policy. The choice of loss function depends on many factors. Proximal Policy Optimization (PPO) (Schulman et al.) 2017) has long been the default, since its clipped objective helps reduce training instability. Given that Grouped Relative Policy Optimization (GRPO) simplifies PPO while often improving performance (Shao et al. 2024). use instead. Its objective to maximize:
LGRPO (0)
E BewD (2 (x
skl
min|rooi, A) clip(ra(i, -,, 1+ ++ - &KL[To] (1)
roo3
where ys is an output sequence, Oula is the last policy (what we call the reference D. AL.
(R, mmam(m)) R))/std(R R) is the sequence-level advantage of output yi compared to other outputs (applied per token). {c, G, }} are constants, and vo(i,t) = To To (k,ct)/ Te... (Bi,e Vi,<c)) is a token-wise probability ratio. KL denotes the token-wise forward KL divergence between the policy and fixed baseline To (e.g., the initial model).
Offiine Off-policy Alignment Online alignment often unstable and slow, since new data needs to be continually sampled and labeled. For this reason, offline off-policy alignment has emerged as a popular alternative. Here, outputs are not drawn from the policy but from another source (e.g.,
human demonstrations), then fed into closed-form loss that is minimized by updating 8. The choice of loss again depends on many factors, but the most popular options are DPO (Rafalovv et al.
2023
which operates preference pairs (xr, yor 31) where yhe > yt. and KTO (Ethayarajh et al. which operates on unpaired feedback (Ir, Vwr) and (xr, 31) (see Appendix for precise definitions).
2024
Online vs. Offline Recent work concurs that online alignment has a higher performance ceiling, although this comes at the expense of more compute, training time, and instability (Xu et al. 2024bb Ivison et al. 2024). Recognizing their complementary strengths, some have proposed online ver- sions of offline methods and vice-versa. For example, online DPO trains samples generated from the latest version of the policy, closing much-but but not all-of the gap with standard PPO (Guo et al, 2024 Xu et al. 2024b). Conversely, offline PPO-where the reference model is never synced and training data is static-performs similarly to offline DPO (Ethayarajh et al., (2024). Explanations of why online alignment works better have traditionally been rooted in RL theory (Song et al. 2024
Tang et al. 2024a Swamy et al. 2025) and are thus complementary to this work.
Verifiability The literature increasingly focuses on verifiable tasks whose correctness can be checked programmatically, such mathematical reasoning (Lambert et al., 2025). When correct- ness is determined by preferences or open-ended judgments, the task is considered unverifiable.
3 ALIGNMENT AS PROSPECT THEORETIC OPTIMIZATION
Given a gamble that returns +$100 with 80% probability and $100 with 20% probability, how much would a player have to be offered to forgo playing? Classical decision theory tells us that an agent meeting certain axioms of rationality (Von Neumann & Morgenstern 1947) would have to be offered the expected value of the gamble: 0.8(+$100) + 0.2( -S100) situation accept far less than $60 however, even though in expectation they could make more money gambling. Prospect theory offers a general framework of why, when presented with an uncertain
$60. Most humans in this
Note that under our terminology Te. would be called the reference model, since it determines the ratio (i f), and wo, which is called the reference in (Sao et al. 20044) would be called the baseline. This is to ensure terminological consistency with offline methods.
3

---

Published as a conference paper at ICLR 2026
Implied Cumulative Probability
1.0
online on-policy
0.8
offline off- oaflpaliy (from worse model)
offline off pflly (from better model)
0.4
human-perceived
0.2
humanline-sampled
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Cumulative Probability under Policy
Figure 2: To estimate human utility, outputs should be sampled from the typical human-perceived distribution of what the policy can produce, whose inverted S-shape comes from prospect theory. Online on-policy sampling (dashed black) is superior to offline off-policy- both from worse (red) and better (blue) models-becaus, the latter deviate more from human perception (solid black). Rejection-samplie with perceptual bias gives us humanline sampling (green) that can mimic this, and special case of it simplifies to the humanline clipping used in design pattern.
event, humans may choose not to maximize their expected value (Kahneman & Tversky Tversky & Kahneman| [992). model of human utility is as follows:
1979.
Definition 3.1. A value function Z - R maps outcome 2, relative to reference point 20, to its subjective value as perceived by the human. When & is vallev, the typical form of is:
u(z;,, [(= 20) 1->(z--1) z)a if < 20
=
where A, a E RR+ are constants.
The salient qualities of value function are: the existence of reference point 2u used determine the relative gain or loss; concavity in relative gains (a < 1), known as risk aversion; and greater sensitivity relative losses than gains (A 1), known loss aversion. Under these settings, is easy to see how the subjective expected value-as as induced by v-could be less than $60.
Definition 3.2. The weighting function w. when applied probability. Let P4 denote the objective probability of outcome and Q capacity function that maps cumulative probabilities perceived cumulative probabilities. typical functional form for the capacity function is
outcome z;. supplants its objective
a
2 (a:n)
(1-
(a7
where E IR+ is constant.
recovers the objective probability but lies in (0. 1) for most humans. Letting 2, denote a positive outcome relative to 20 and 21,..,, 2n the ordered outcomes from least most positive, the weights are then:
s (E - 22 (E Pj)
if <
w(z))
(4)
2 (p.)
if i
If were instead negative outcome relative 20, then it would be compared to outcomes even more negative than it, with separate function Ω (a: following the same form (3).
For example, suppose that in our gamble, there were now two positive outcomes instead of one: winning +550 with 60% probability and winning +1100 with 20% probability. The probability of an outcome as good or better than $50 is 0.8 0.6 + 0.2) and one good or better than $100 0.2 (itself); these are our cumulative probabilities. Since Ω2+ captures the human tendency to overweight extreme outcomes at the expense of moderate ones, let us say the perceived cumulative probabilities are 0.8 and 0.3 respectively. By applying 44. we then get weights of 0.5 (= 0.8 0.3)
for the $50 outcome and 0.3 for the $100 outcome. That is, the extreme outcome of winning S100 has been up- weiwhittted from its objective probability of 0.2 to a subjective probability of 0.3 while the moderate outcome of winning $50 has been down-weighted from 0.6 to 0.5.

---

Published as a conference paper at ICLR 2026
Definition 3.3.
The subjective expected utility of a random variable Z is weighted combination of the subjective values of its outcomes: ul u(; _ == a,,
a,,
Although every human has unique value and capacity function, the functional forms in (2) and describe those belonging to the majority of people in human studies (Tveesky & Kahneman 1992
In the original literature, random variables were only studied in monetary context, where outcomes can be measured in dollars. |Ethayarajh et al. (2024) were the first to extend prospect theory to the alignment generative models. Taking any output (token or sequence) given context z, they treat the surprisal term Zæ.y log[To (z)) Tee [x)] as the outcome, whose units are nats of information. They propose that the goal of alignment is to modify 0 such that desirable outputs have Ea.y > Ao and undesirable outputs have Z 20. formally proving that all the commonly used alignment objectives-DPO KTO, PPO, and GRPO eRRPOeecoode prospect theoretic model of utility differing only in the shape of their value function and the distribution over which the expected surprisal is taken to construct 20-
However, they ignore the weighting function, assuming that the human perception of probability is effectively objective when it comes to generative model outputs. But what if it were not? It is intractable to infer the human-perceived probability distribution over large output spaces (e.g.. token vocabularies), which is why the original prospect theory experiments were limited to handful of possible monetary outcomes (Tversky & Kahneman 1992). Because of this, we will assume that the perceptual distortion of probability in the generative model setting has the same shape as in the monetary setting (Figure 2) allowing us to use the well-established parameterization in Definitions 3.1 through 3.3]
Proposition 3.4. For any input x and bounded value function v, let the outcome of an output be its surprisal log[ (y)z)) / [y]=]] and Q be candidate distribution over outcomes. Then to guarantee (w) w( Q)1 S 8 for some 2 0, it suffices that v /KL(w]IQ) s 8/ (V2])oll<).
The proof is deferred to Appendix B. Even if we had oracle access to value function and thus knew exactly which alignment objective to use, we could not necessarily maximize human utility. As the proposition suggests, the simplest way do would be to sample generations according to the subjective distribution that was implicitly assigned to the outputs.
This offers human-centric explanation for why online on on-policy sampling is superior to offfine off- policy sampling for alignment, one that is complementary to the RL-theoretic explanations in prior work. As illustrated in Figure 2 if the median human capacity function (solid black) is function of the probabilities from the current version of the policy, then the subjective probabilities will loosely track online on-policy sampling (dashed black). In contrast, offiine off-policy sampling can deviate sharply from both. Consider the desirable outputs for some context x::
1.
When sampling from model worse than the policy, the outputs' surprisals- computed under the current policy-will on average be lower (i.e., less positive), since they are more likely under the worse model and less likely under the current one. Recall that in (3), positive outcomes are ordered from least good to most good. This means that the implied capacity function (red) will saturate much more quickly than the human capacity function.
2. Conversely, when sampling from model better than the current policy, surprisals will be larger. Given that more-positive outputs are more plentiful than they would be under the policy, the implied capacity function (blue) will saturate more slowly.
Returning Proposition 3.4, if we cannot directly sample the perceived distribution. then a natu- ral solution is to rejection-sample our outputs to simulate the drawing of tokens according to their subjective probabilities!" Moreover, this allows us to use data sourced from anywhere instead of limiting ourselves to online on-policy data. In $4 we modify the standard rejection sampling al- gorithm to capture the perceptual bias in ④ which we call humanline sampling. By tweaking its hyperparameters, we can mimic wide range of distributions (Figure 2 green).
*Other parameterizations have been proposed, however: Prolec | Gonzalez & Wu (199), inter alia. 'Even losses without explicit surprisal term, such as PPO and GRPO, do have tokenwise likelihood ratio (mo (f))))/==<<(()))) that can he framed as the exponentiated surprisal.
'Athhough importance sampling is another option, it comes with its own problems in the context of genera- tive models, such degenerate importance weights.
5

---

Published as a conference paper at ICLR 2026
4 CLIPPING RECOVERS PERCEPTUAL BIAS
In 3 we established that human utility can be maximized when outputs are drawn according to the human-perceiied distribution. Given that do not have access to anyone's perceived distribution, we will instead modify the standard rejection sampling algorithm to simulate drawing from the typical human's, as formalized by prospect theory (4) We call this humanline sampling.
Proposition 4.1. Under typical conditions, for any context x, simulating output sequences y from is equivalent to performing token-wise rejection sampling with the rejection criterion
(Be V<<r// Tee((V (z; ge1) < M
where B ~ Betal7, 1), M2 is finite upper bound on the token-level likelihood ratio under the vocabulary (i.e., V yb. Rec(M (xg)eee)
ne (Ir (xt))))
< MMg.. and y (0, 1] is the capacity function constant.
We defer the proof to Appendix [ Still, applying rejection sampling during training comes with several practical concerns. For one, change, and the objective probabilities that are fed into a human observer's capacity function could reflect exposure either the current policy the previous one. Second, resampling only those tokens that have been rejected while leaving the others untouched will not guarantee that the final output is coherent or relevant. Third, zeroing out the rejected tokens could destabilize sequence- wise losses like KTO whose training dynamics are affected by the saturation induced by all tokens. Taking this into account, we propose humanline sampling:
online setting, both the reference and policy models
Definition 4.2. Given output sequence y, humanline sampling rejects tokens ye that meet the fol- lowing rejection criteria by detaching them from the computational graph:
To [e( < Mp or Tee M1 yet)
Tee | [et))) Ve(e)
MR Bn
(5)
To
where Mp, MR are constants such that To (o < Mp Meex) and Ter (2;; yer) ( 2; ) for all yt, Bp ~ Beta(?P, 3p) and BR ~ Beta( BR) are independent Beta random variables, and yp, "/R, 3p, 3R are Beta distributionnsppclie constants.
<
The two-sided criteria address the first concern about the origins of the objective probabilities that are fed to a human observer's capacity functions an online setting. Keeping the rejected tokens in the sequence addresses the second and third concerns, while detaching the tokens from the compu- tational graph (i.e., stopping gradient flow for those tokens) ensures that they do not contribute to the updates of that minimize the loss. Even though control an the policy (i.e., more exploitation); if 7)p YR, there is more emphasis exploration.
not resampling tokens, "/n: "/r effectively trade-off. If yp < 7/R, there more emphasis on drawing from
Theorem 4.3. The clipped component in PPO/GRPO is special case of humanline sampling that arises under limit conditions.
We defer the proof to Appendix The intuition that there exists construction such that sampling from the Beta distributions is equivalent to deterministically sampling their means. The two criteria can then be combined into a range that the likelihood ratio must fall in, analogous to the clipping range. In both cases, the gradient is zero outside this range: the clipping function due to its deriva- tive, and humanline sampling because it explicitly stops the gradients for those tokens from flowing through the graph. However, the unclipped component in PPO/GRPO does allow ratios outside this range to affect the overall gradient the loss; to more fully integrate this perceptual bias, we would need clip the ratios upstream of the objective, not just within ($5)
HUMANLINE VARIANTS
5.1
METHOD
If the success of PPO/GRPO can be ascribed to them being perceptual losses ($3) $4). then need not limit ourselves to using online on-policy data; we can source data from anywhere and selectively train on it in manner that reflects the prospect theoretic model of perceived probability. To this

---

Published as a conference paper at ICLR 2026
Offline
Online
Humanline Syncing
Fref
training
re Rg
Fref/+1 F8p
Figure 3: In offline objectives (left), the reference model does not change during training. In online objectives (middle), the reference is synced with the policy at the current step; at scale, some asyn- chrony is permitted lag of one step is depicted here). In humanline syncing (right), every steps, the reference is synced with the policy from the previous step (k = 1 is depicted here).
clipping
ER
r)
1 +
GRPO
rdii f)
+
GRPO (humanline)
Ep
symmetric clipping
asymmetric clipping
symmetric clipping
Figure 4: In humanline clipping, the token-wir wise likelihood ratios ro(i, f) are asymmetrically clipped to [cp, cal upstream of the loss. In the humanline variant of GRPO, instead of there being an unclipped 7a and a [1 [1-c, e, 1 <]-clipped r3 as in (I). we have a once-clipped and twice-clipped rg- Though humanline clipping should in theory be most impactful for losses without any clipping to begin with (e.g., DPO, KTO), still benefits GRPO (see Figure 5 left).
end, propose creating a humanline variant of any alignment objective that is a function of both policy my and reference model Tie This is done by applying a two-part design pattern:
1. Humanline Syncing: Every i steps, after the loss calculated but before the optimizer step is taken, sync the weights of Thef with Ty (Figure 3) In general, lower &: leads to better performance but also more instability (Figure 8 Appendix
2. Humanline Clipping: Clip all losen-wise likelihood ratios тe T (z,
( Vece)
to the range [EP,ER] even before they are fed into the loss, where EP; ER Et and the range can be asymmetric. Losses that already do some clipping, such as GRPO, will do clipping twice over (Figure We clip in log-space for greater numerical precision.
The motivation behind humanline syncing is that as the policy changes over the course of training, the standard against which the policy is judged also changes. Since the outcome is defined as the surprisal logí mo (y)l2)) ( this means that the reference model must change well, at a rate controlled by k. We choose to implement humanline clipping instead of the humanline sampling proposed in $ for few different reasons. For one, humanline clipping is special case of hu- manline sampling, one that arises under limit conditions (Theorem 4.3 However, is much faster (since new tensors have be allocated), requires fewer hyperparameters, and is more stable than humanline sampling while being as or more performant (Figure S right). Clipping multiple times (Team al. 2025) and asymmetric clipping Yuu al. 2025) have been explored in past work, but to our knowledge, the specific formulation in humanline clipping has not been used.
Note that the humanline variant of each method can be used with both online on-policy data and offline off-policy data, which we denote as online+humanlin In contrast, the online variant of method only used with online on-policy data; the offline variant, only with offline off-policy data. Alignment objectives without reference model, such as SimPO (Meng et al. 2024). cannot have a humanline variant because neither change is applicable.
and aofline+hmmnlinnn respectively.
As defined in 2 the reference model is the one against which the surprisal is calculated (explicitly, in the log-ratios of DPO and KTO; implicitly in the ratios of PPO and GRPO).

---

Published as a conference paper at ICLR 2026
30
offline humanline w/o clipping
humanline sampliing humanline clipping
25
w/o syncing
LC Winrate (%)
w/o either
LC Winrate (%)
20-
20-
15
15
10
KTO
GRPO
DPO
KTO
GRPO
Figure 5: The majority of the improvement comes from humanline syncing (left), However, human- line clipping is still necessary-Synciing alone is not competitive with online alignment. Although humanline clipping is special case of the more general humanline sampling ($4), it performs as well while being stabler and simpler to implement (right).
5.2 EXPERIMENTS
We create humanline variants of DPO/KTO/GRPO and compare them to both their offline and online counterparts, ensuring that the number of examples seen by the different variants the same. Details on how we created the online version of DPO/KTO and the offline version of GRPO can be found in Appendix C We test these variants in an unverifiable reward setting where the goal is to follow open- ended instructions and a verifiable reward setting where the goal is to do mathematical reasoning.
5.2.1 UNVERIFIABLE REWARDS
Using offline DPO/KTO/GRPO, we first align 1lamma-88 -B-Instucct (AI@ 2024) on an instruction-following dataset called UltraFeedback ArmoRM (Meng et al. 2024). For online DPO/KTO/GRPO, we use the same contexts but sample completions from the policy, score them with the ArmoRM reward model (Wang et al., 2024). and then construct preference pairs. This is how the offline data was constructed as well-but but sampled from different apples-to-apples comparison between the online and offline variants of the same objective. Using the humanline variants with either the online or offline data does not require further changes. The models are evaluated with AlpacaEval2 (Dubois et al, 2024 6
for an
For all objectives, the variant (p < 0.05 and is on par with the online variant, as seen in Figure | The magnitude of improvement is large, with offlinc+humanline GRPO performing 1.6x better than its offline coun- terpart. Improvements persist at the 27B scale and with different model families (Appendix D However, the online+humanline variants only slightly better than their online counterparts. This not surprising: under our theory ($3), online on- on-policy sampling is superior to offline off-policy sampling because it deviates far less from human perception; the marginal benefit of humanline objectives will naturally be smaller in this case.
variant performs significantly better than the offline
Humanline objectives do not obviate the need for good-quality data. We stress that although offline+humanline variants can match the performance of their online counterparts, this is not given for any offline data (Appendix D Table 5). Fortunately, we find that the average token log- probability of the output under Ter (before training starts) is good proxy for whether the offline data will be "good enough'. Training on the lowest quartile- -with average token log- probaabiiity in the range [-1.03, -0.36] to significantly worse results than training the rest (Figure 6 This can be ascribed to lower sample efficiency that arises from more frequent humanline clipping.
Humanline syncing is responsible for most of the improvement; humanline clipping is needed to fully close the gap. Figure 5 we plot the drop in performance as one or both changes are ablated. Humanline syncing, done here every step (A: 1) is the more crucial ingredient; without it, the performance would be as bad as with the offline variant. However, can be done as infrequently
"We use GPT 4.1 as the judge instead of the default GPT-4-Turbbe as it is cheaper and more performant.
We apply the Moom--onferee correction adjust for multiple comparisons (Holm 1979 Dunn 1961
8

---

Published as a conference paper at ICLR 2026
Offline
Offline Humanline
online
LC
before alignment
Q1
Q2
Q3
Q4
Quartile (Average Token Log-Probability under n.,))
Figure 6: Data quality matters, even when using humanline variants. As seen here, the average token log-probability of the output under net (at step 0) is good proxy for offline data quality; if it is too low, as in the first quartile of data, the DPO-aligned model's performance will be worse.
Accuracy Reward
Format Reward
0.6
0.45
0.30
Online
Offline (64x)
0.2
Oflinn++ummnlinee
0.15
Offline+Humaninn (64x, k-20)
0.9
250
500
750
1000
Training Steps
Training Steps
Figure 7: For mathematical reasoning (MATH500), sampling data 64x less frequently (orange) than in online GRPO (blue) leads to significantly worse performance, even though the total volume of data seen remains the same. In contrast, using the humanline variant of GRPO while being 64x more offline does not incur performance degradation (green). Less frequent humanline syncing (k 20, violet) leads to slower but more stable learning; at 1, the instability would cause collapse.
as =
4 without a loss in performance (Appendix D Figure 8). Note that not all kinds of syncing
equal: trust region-style syncing (Gorbatovski al. 2024). which happens after the policy
is updated-the rendering the policy and reference equal leads to worse results (Appendix D Figure TOp. Humanline clipping is still needed for the offine+humanline variants to match the performance of their online counterparts (Figure 5 left). For instruction worettoonn..loooiinnn.o clipping range
of log cp 1.5, log n 1.5 < Cp 0.22, cn 4.48 works best for the humanline variants of all methods, and performance is robust to small changes (Appendix Table T)).
Humanline variants do not require changing method-specific hyperparameters, but the learn- ing rate or maximum gradient norm need to be adjusted. The use of humanline variant in- troduces two counteracting forces. On one hand, likelihood ratios ro can get smaller compared to offline learning explicitly, due to clipping, and implicitly, due to the syncing of the reference model, since Te TT(y)z) cannot drift too far from ( causing gradients to get smaller. The learning rate maximum gradient norm needs to increase make up for this. Conversely, updat- ing the reference model introduces more training instability, which demands a lower learning rate maximum gradient norm. Therefore, depending the cireumstances, this shift could require increasing or decreasing the learning rate/gradient norm by 0.1x-4x (Appendix D Table
aifinn++hmmnling GRPO is 6x faster to train with than the online variant, while attain- ing equal performance. Compared to offline GRPO, the offline+humanline variant takes roughly twice as long run when syncing every step (Appendix D Figure TT]. However, this is a com- paratively small price to pay to match the performance of online GRPO, which takes over 12x the wall-clock time of offline GRPO.
5.2.2 VERIFIABLE REWARDS
When doing alignment for mathematical reasoning, it is standard to be fully online on-policy and the correctness of the final output the only reward (DeepSeek-Al et al. 2025). Our goal

---

Published as a conference paper at ICLR 2026
with the humanline variants will be to push the extent to which the data can be offline off off-policy. For example, sampling completions every 10 steps instead of every step would make the process much more efficient: in the fully online on-policy setup, training waits the next batch of samples from the current policy and inference requires the policy to finish training on the current batch; by sampling less frequently, training, inference, and labeling can all be asynchronously overlapped.
Humanline GRPO allows data to be sampled up to 64x less often with no performance degra- dation on mathematical reasoning. We first align Qwen2. 5B ruct Yang et al. 2025) with online GRPO on the MATH500 training set (Lightman et al. 2023), largely following the setup in Open-R1 (Hugging Face 2025) and assigning rewards based on formatting and cor- rectness. Instead of sampling every step, we then sample 64 times as much data every 64 steps to get a model that significantly worse (p < 0.05) (Figure 7). Running the same off-policy setup with the humanline variant of GRPO closes the gap in rewards within 1000 steps. After 1600 steps, the @ass@1 accuracy on the MATH500 test set 0.593 0.019 for both the online and humanline runs. The degree of humanline clipping remains the same as in instruction Ep = -15, log en = 1.5), suggesting that it works as a strong default for a wide variety of tasks. However, we find that syncing too frequently (A = 1) leads to reward collapse. Increasing leads to slower but stabler training, with any & E [12, 24] closing the gap with online alignment in 1000 steps while avoiding collapse. Although human utility seems irrelevant mathematical correctness, the fact that reasoning is still expressed in language, a human abstraction, may help explain why incorporating perceptual biases via humanline objective is still useful for this task.
LIMITATIONS & FUTURE WORK
We stress that although humanline variants trained with offline off-policy data are able to match the performance of their online counterparts, this is still an empirical regularity as opposed to a formal guarantee. In addition metrics that we can use to quantify what makes offline data 'good grood-quality?? Conversely, are there settings under which alignment data must necessarily be online and on-policy? We leave these as directions for future work.
the average token log-probability of the output under Tret are there other
The model of human utility discussed in this paper comes directly from prospect theory, which was originally developed in the context of monetary random variables. Although it has since been em- pirically validated in other contexts, there is no guarantee that it naturally extends to the generative modeling setting. Assuming that it does is another limitation of our work, one we accept because our primary goal is to improve the post-training of generative models, and because experimentally inferring biases in human perception over very large output spaces is intractable. Developing new theories of human probability perception as relates to generative models is another future direction.
Lastly, humanline variants raise practical questions: How large the systems gains from fully overlapping triniinginferrenc/llblligg? Can we reduce the cost of syncing (e.g.. by only syncing some of the model weights)? Should be personalized instead of using one setting for all?
CONCLUSION
Based a prospect theoretic framework, we proposed that the online-offline dichotomy central to post-training is incidental to actually maximizing utility: what matters is not the source of data per se, but whether it reflects the human-perceived distribution over model outcomes. This perspective interprets PPO/GRPO's 's clipping as recovering form of probability distortion, suggesting that these state-of-tie the-t--rtttrttt objectives are successful because they are perceptual losses. We then proposed a generic design pattern for explicitly incorporating perceptual biases into commonly used alignment objectives, giving us humanline variants of DPO/KTO/GRPO. When trained with offline off- pooicy data, the humanline variants were able to match the performance of their online counterparts, closing 1.3-1.6x gaps in winrate for instrection-following and enabling up to 64x less frequent sampling in mathematical reasoning without performance degradation. This opens the door to cheaper, faster, and more parallelizable alignment that is not constrained by the need for online on- oolool data.
SSinee mathematical reasoning on MATH500 requires the generation of many intermediate reasoning to- kens, we were foreed to use a smaller model than in $5.2. 1 due memory constraints.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Al@Meta. Llama 3 model card.
2024. 1lama3/blob wwaii/MMaa. CARD mc
URL https: https::/github.co wwt.ol/.ooommmeeeeeeeeeee 11mma/
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. arXiv preprint arXiv:2408. 1791, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204 05862, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint
2022b.
Lawrence Chan, Andrew Critch, and Anca Dragan. Human irrationality: both bad and good for reward inference. arXiv preprint arXiv:211 (1.0690,, 2021.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ulrafeedback: Boosting language models with scaled ai feedback. In International Conference Machine Learning, pp. 9722 9744. PMLR, 2024.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye. Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu. Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou. Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda
11

---

Published as a conference paper at ICLR 2026
Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforce- ment learning, 2025. URI.tttps: pattts://arxiv. org. hwagg//bb//// /201 12948
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang. Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprint arXiv:2402.01139) 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlln al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477 2024.
Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical Associ- ation, 56(293):52-644 64, 1961.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model align- ment as prospect theoretic optimization. In International Conference on Machine Learning, pp- 12634-12651. PMLR, 2024.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code iims in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089, 2024.
Richard Gonzalez and George Wu. On the shape of the probability weighting function. Cognitive psychology, 38(1):12-1666, 1999.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint urXiv:2404.09655 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint M.XXi:22022.47922 2024.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Nickum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: learning from human feedback without rl. arXiv preprint ar 1xX.:23310.133:3 2023.
Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint a7X1v:2403.07691, 2024.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https:
//github. cttebb..ommuuuee http//ugginng ng face/ open-r1
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin. Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances in neural information processing sys- fems, 01:36602-366333 2024.
Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung- Woon On. Binary classifier opti- mization for large language model alignment. arXiv preprint arXiv:2404.046566 2024.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econo- metrica, 47(2):263-292, 1979.
Minae Kwon, Erdem Biyik, Aditi Talati, Karan Bhasin, Dylan P Losey, and Dorsa Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction, pp. 43 43-52, 2020.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llm.. arxiv preprint arXiv:2406.1880 2024.
12

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https: wax.i..or//aao......o.o
15124
Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang. Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Bridging offline and online reinforcement learning for Ilms, 2025a.
21495
Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang. Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, et al. Bridging offline and online reinforcement learning for llms. arXiv preprint arXiv:2506.21442 /495, 2025b.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21 213228, 2022.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Yen-Ting Lin, Di Jin, Tengyu Xu, Tianhao Wu, Sainbayar Sukhbaatar, Chen Zhu, Yun He, Yun- Nung Chen, Jason Weston, Yuandong Tian, al. Step-kto: Optimizing mathematical reasoning through stepwise binary feedback. arXiv preprint mrXiv:2501.10777 2025.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Rein- forcement learning from unit test feedback. arXiv preprint arXiv:2307.04344 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward, 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint Stti::23120088866 18, 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. 2021.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville, Asynchronous rlhf: Faster and more efficient off-policy for language models. arXiv preprint ar 21v:241002.825555 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, March 2022. URL http:://a /ttp///aivv/////////v///aa///.at orq abs attaabb////2222 2203.000000 C2155 arXiv:2203.02111 [cs].
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:1166177116637,, 2024.
Drazen Prelec. The probability weighting function. Econometrica, pp- 497-527, 1998.
13

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2023. URL http//axivvv.oo aabeg/////// /20../3355.111 18290
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.0337 037// 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL htttpst: //triii/a..vv...//aa wwtpp/////////...ooo//// 06347
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL htthstts: //tr:iv.a.iii.aa/aa arg/abs/2402
03300
Yuda Song, Gokul Swamy, Aarti Singh, Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. Advances in Neural Information Processing 12270, 2024.
Systems,
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Liting Sun, Wei Zhan, Yeping Hu, and Masayoshi Tomizuka. Interpretable modelling of driving behaviors in interactive driving scenarios based on cumulative prospect theory. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3329-4333 IEEE, 2019.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint vwXii:2503.01067 2025.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.068 08848, 2024a.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernaro Ávila Pires, and Bilal Piot. Gen- eralized preference optimization: A unified approach to offline alignment. arXiv preprint 255::22002.05749, 2024b.
Yunhao Tang, Sid Wang, Lovish Madaan, and Rémi Munos. Beyond verifiable rewards: Scaling re- inforcement learning for language models unverifiable data. arXiv preprint arXiv:2503. 191118 2025.
Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong, Jannik Straube, Manveer Basra, Aaron Pazdera, Kushal Thaman, Matthew Di Ferrante, Felix Gabriel, et al. Intellect-2: A reasoning model trained through globally decentralized reinforcement learning. arXiv preprint arXiv:2505.07291, 2025.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323, 1992.
14

---

Published as a conference paper at ICLR 2026
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and and outcome-based feedback. arXiv preprint arXiv:2211.14275. 2022.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 2nd rev. 1947.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts 2024. URL https://aaxiv.
org/abs/ 2406. 12845
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-stey without human annotations. arXiv preprint arXiv:2312.089 080935, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939, 2025a.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20577 2025b.
Junkang WU, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 8-dpo: Direct preference optimization with dynamic B. Advances in Neural Information Processing Systems, 37:1 199121994412966, 2024a.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. arXiv preprint arXiv:2407. 2024b.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008-59033 2023.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit qe-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21044 2024.
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseck-proven Advancing theorem proving Ilms through large-scale synthetic data. arXiv preprint 24X.::2400...333 2024.
Wei Xiong. Hanze Dong, Chenlu Yc, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.1 /456, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401. 17, 2024a.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Iterative preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? comprehensive study. In International Conference on Machine Learning, pp. 54983-54998. PMLR, 2024b.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint NKKv:25005009888 2025.
15

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang. Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL
https: / /wt.aaaavvv..../...ae.eaeee.ae.. wtt.ii..og/aabaa....oooa/ abs 2503. 14476
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504. 138337 2025.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025.
Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. Wpo: Enhancing rlhf with weighted preference optimiza- tion. arXiv preprint arXiv:2406. 11827, 2024.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493, 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano. and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0883 08593, 2019.
Yuxin Zuo Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
16

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Alignment Methods Reinforcement Learning from Human Feedback (RLHF) involves training a reward model on human preference data and then using fine-tune policy, commonly via online reinforcement learning (Christiano et al. 2017 Schulman et al. 2017]] Nakano et al. 2021 Ouyang
al. 2022). The complexity of online RL has motivated line of research on simpler, offline meth-
ods that optimize policy on static dataset (Ziegler et al.] [2019] Rafailov et al., [202] Ethayarajh
al., 2024; Hejna al., 2023: Azar al.
2024 Hong et al. |024, Munos et a.. [023 Xu al.
2024a Jung et al., 2024 Wu et al.) 2024a
Xie et al., 2024 Pang et al. 2024 Tang et al. b024b).
Other work aims to bridge the gap between offline and online methods via iteratively collecting new
data from the policy (Stiennon et al.
2020
Xu et al.
2023 Xiong ct al. 2023] Wu et al. 2024b
Chen et al. 2024 Rosset et al. 2024 Pang et al. 2024 Lanchantin et al. 2025b), reweighting offline
loss terms (hoou al.) 2024), recasting offline methods online (Guo al. 2024). Recently, there has been an increase in interest in online reinforcement learning with verifiable rewards, in-
cluding training hyperparameters (Yu al., 2025 Liu et al., 20255 Wang et al. 2025a) and other
aspects (Wang et al., [2025b] Zuo et al. 20225 Shao et al. 2025 Yue et al. 22025 Zhao et al.) 2025))
The humanline design pattern be applied most alignment algorithms both offlnne and online
settings.
Sources of Feedback The performance of aligmment algorithms is directly linked to the type and quality of the feedback signal. This signal often comes from direct human judgment (Bai et al., 2022a Wu et al. 2023). To improve scalability, researchers have also explored the use of AI- generated feedback (RLAIF) (Bai et al. 2022b). For more objective domains such as coding, verifi-
able feedback can be derived from execution results and unit tests (Le et al., [022] Liu et al. [023] Gehring al. 2024 Dou et al. 2024). Due the difficulty of obtaining feedback in
do- mains, researchers are exploring learning without external feedback (Tang et al. 2025 Zhou et al.] 2025). Another axis of differentiation is whether feedback based on the final output the model (outcome-based) (Xin et al., 2024, Ankner et al. 2024 DeepSeek-A1 et al. (0255) or its intermedi- ate steps (process-based) (Uesato et al.) 2022 Lightman et al.) 2023 Wang al. 2023 Lai al.
2024 Lin et al., (202)). The humanline paradigm works with different forms of feedback and is
independent of whether that feedback is outcome- or process-based.
Having revolutionized behavioral economics, prospect theory (Kahneman & Tversky 1979 Tversky & Kahneman, 1992) has recently been incorporated into LLM alignment
Prospect Theory
via Kahneman- versky Optimization (KTO) (Ethayarajh et al. (2024). Previously. it has had only
limited impact in machine learning, mostly in human-robot interaction research (Kwon et al. 2020
Sun al. 2019 Chan et al. 2021). While KTO focuses human biases in the value function
the humanline design pattern does so for the weighting function
17

---

Published as a conference paper at ICLR 2026
B PROOFS
Proposition 3.4 (restated) For any input a and bounded value function v, let the outcome of an output y be its surprisal (y)()(()lz)/m)g ())e(((())))))))))))))))))) u/]=)] and Q be candidate distribution outcomes.
Then guarantee [u(Z:w) u (u(Z: Q) S for some d 2 0, suffices that VKL(w]IQ) z 5/ (V/llulloo).
Proof. Let Zz,y denote the outcome an input-output pair (x, y). where as in Ethayarajh et al. 2024), it is measured the surprisal term logí To (v)]=)/T=<r(((z)) Assume that the human value function is bounded (as the case prospect theory), that the human-perceived distribution has support subsuming that of Q (i.e., supp(Q) S supp(w)). and that Za.y is measurable with respect to the support of both distributions. Note that w(2z,)) denotes the subjective probability (weight) assigned output y based on its outcome 2 not a probability distribution over y itself. That is, w is a distortion of the cumulative distribution over outcomes (surprisals), as defined in Eq. (4). Then using Definition (3.3 of subjective utility:
[u(Zww) u(z:Q)1=
Σ (((2)))((ZZ))) Σ
Q (e
Σ (w(zz))) - Q(2z,)))((Zz))
- Σ |u( (W(2x))) - ((x,,))
(triangle inequality)
x ooolo ||W Q11
s |blloo KL(WIQ)
(Pinsker's inequality)
Then if / QQ) < 8/ (3//v2222) (v))) we get [u(Z:w) u 2)1
(restated) Under typical conditions, for any context simulating output se- quences y from w is equivalent to performing token-wise rejection sampling with the rejection
Proposition 4.1
criterion
To Uct)/ / V<+) < MDB
where Beta(y, 1). Mt finite upper bound on the LHS for all tokens in the vocabulary, and E (0, 1] is the capacity function constant.
Proof. We consider the following conditions:
1. The proposal distribution is the current iteration of the reference model: i.e., any output se- quence was produced by autoregressively sampling tokens from [- ct). Moreover, Trer(-aæ, Vi, and T [z, 35,<t) have the same support and finite likelihood ratio bound. In practice, this means that they share a vocabulary (which holds because they are identical at 0); and that each possible token has non-zero probability, which arises trivially from a softmax output distribution.
2. The capacity functions Ω ,2 have the standard functional form 3 implying that the human-biased CDF has the same structural form for all outcomes. This was found hold in Tversky & Kahneman (1922) (for probability in more general sense).
3. For context with output sequence yt, let surprisal outcome. The cumulative probability of outcomes with higher absolute surprisal than z; is negligible (i.e., the vast majority of all possible output sequences is nonsensical or irrele- vant, which holds trivially). To be more specific, for any given prompt, the possible output space of n-length sequences is very large. Only small minority of possible completions are good or bad enough that we want to explicitly align towards or away from them, and the cumulative probability mass of better or worse completions is negligible.
log mo (og()((())))//ma(()))))) denote the
18

---

Published as a conference paper at ICLR 2026
Under these conditions, given outcome 24, from 4 the weight (i.e., subjective probability) of a sequence is
E, Ps z 0 for S and E Pj 0 Ω Following
p"
w(z)
(p* + p)T)
For sufficiently long sequence, the denominator will approach meaning wi(z) p". The nu- merator (and thus the weight) can be factorized over tokens as p" = II. Ph. meaning that instead entire sequences, we can just rejection sample one token at time. Because the policy and reference models yield softmax distributions over finite shared vocabulary, for any given (x. Mex)) the likelihood ratio for a fixed (policy, reference) pair takes finitely many positive values, so there exists finite bound Mý- Our results only require that this maximum exists; they do not require Mg to be small, known, or independent of vocabulary size. Note that our results only need finite bound for each fixed not uniform covering argument over the entire parameter space. Then:
of
To (V11 Vee))
To 2; Bee))
Mo U
Mg U
Teot (1112; [0; Vee)
Tref (3)
where U ~ Uniform 1).
Let B & Us, where y E (0, 1] is the capacity function constant in (3. To get the density this new random variable, we apply the transformation rule, noting that because U is uniform on [0, 1], f))) =
d b" 1 1
(B(b) (()))))(b))) fb)
db
This is the density of Beta(7, 1). Therefore,
B Beta(7, 1), Ma Mg
Theorem 43
(restated) The clipped component in PPO/GRPO is special case of humanline
sampling that arises under limit conditions.
Proof. Let Bp denote the Beta random variable in Definition 4.2 and Mp its corresponding constant that bounds the likelihood ratio. By definition, its mean and variance are
B[Bp]
P/P
Var[Bp]
Tr
YP + 8p
(Lp + 8p) 3p 1)
kse. Let k, Ep E R+ be constants such that Ep < Mp. Setting 7p Mp
3p = k(1 - # we get
k EP Mr kP + k(1 Mr
Ep Mp
E[Bp]
sp) Mp
k: MM k(1 Mp
EP
1
Ep Mp
Mp
Var Bp]
(k SP + k(1
Mp
) (k sP
Mr
+ k(1
Mr
+1)
kc + 1
As k + 00, 48> 0, Pr[ - ep/Mpl 2 6) + 0 (i.e., we deterministically sample the mean).
Similarly, for BR, MR in Definition 4.2 let en E R+ be such that en 1/MR and set 7R
k/(eRMR), 3R = k(1 MR))
Then as k + oo, A 8 > O, Pr([BR 1/ERMRl 2 8) + 0.
Thus as ls oc, the rejection criteria in token-level humanline sampling simplify to:
Te Te | vet) Trof (yel | y-()
SP
no (e Heer) To Wee)
1
Mp
EP,
MR En
Mp
ER
which means that the tokens that are accepted satisfy: ne (v xz ycc))
Sp <
< ER.
Trof (yh (2; Vec))
19

---

Published as a conference paper at ICLR 2026
Recall that Mp, Ma upper bounds on the likelihood ratios, and given that ne. aeet distribu- tions over the same support and are generally not identical, there will exist tokens for which these ratios are both greater than 1. Thus Mp 2 1 and Mr 1. For any fixed (0, it is therefore given that (1 - c) < Mp and (1 + e) > 1/Mrs meeting the constraints imposed earlier on ep, en- Letting en 1 + and Ep tokens:
get the following inequality that is satisfied by accepted
To xi yec))
< Tref (Yo [2; yec))
+1+
This recovers the clipped term PPO and GRPO, where the ratio (g(y)/z; yee)/ (er yee)) for each token clipped the range [1 c, + e]. Note that the equivalence is only due to the likelihood ratios being bound to the same range, but also due to ratios outside the range contributing nothing the gradient, either due the shape of the clipping function (in PPO/GRPO) or due to being explicitly detached from the computation graph (in humanline sampling).
20

---

Published as a conference paper at ICLR 2026
C ALGORITHMS
C.I DEFINITIONS
C.1.1 OFFLINE DPO/KTO
DPO RRaaillv et 2023 and KTO (Ethayarajh al.) 2024) were originally proposed as offline algorithms, and we use the original definitions without any change for offline DPO/KTO. Where (r, Yw: 31) is tuple from an offline dataset D representing a preference for output yo over yr given context x, the DPO loss is:
To To
(an (x) Teet (yr x))
LDPO(TT: Tret) E W. Bn
log [Blog
Trer(yllz) 3 log
(6)
where B E IR+ is a hyperparameter and is the sigmoid function.
Instead of paired preferences, KTO frames outputs as undesirable or desirable. Where Ay E R+ denotes (A()) when y is desirable(undeeirabbe)) respectively, the default KTO loss is:
LKTO(TT, Tee)) = Br.gDDA - v(1
(7)
where
ro(2,)) y) log To (yllz)
50 KL(TO( (x) []]] (y)(((())))
(Bor y) - 20)) y Vdssirable v(z 3) A re(a, v)) if Vondesirable 2
There is no backpropagation through 20: it exists purely to control the loss saturation. In practice, for the sake of efficiency, shared KL estimate is used for all examples in the same batch by taking the average Ta over mismatched input-output pairs (æ. y'). In our experiments, for an apples-to- apples comparison across methods, break up DPO preference pairs to get unpaired data for KTO, although we use twice the batch size so that the same number of steps are taken.
C.1.2 ONLINE DPO/KTO
Our implementation of online DPO combines features of the online DPO implementation in Guo et al. (2024), iterative DPO in Xu et al. (2024b), and semi-online DPO in Lanchantin et al. (2025a). Like Guo al. (202)) Lanchantin et al. (2025a), we sample completions from the policy being actively aligned. However, like LLachantin et al. (2025a), we do not sample every step, because is slower, more computationally expensive, and leads to worse results. Asynchronous training is typical in RLHF, especially in large-scale distributed settings (Noukhovitch al. 2024).
We find that sampling once every 1024 contexts (i.e., 32 steps) leads to best performance in the cestruction-followwee setting, which we call one round. For each of the 1024 contexts in round, we sample completions [ ~ Ty (-|z), score them with a reward model, compare the highest- and lowest-scoring y for each I, and construct paired preference (xc, yie, U1) if the difference in score exceeds 7 = 0.01. We use this methodology because is nearly identical to how the offline mnstruction-folloiin data was constructed (Meng al.) (024), even using the exact same reward model (Wang et al., 2024) and contexts æ (Cui et al. 2024) to enable an apples-to-apples compari- son. The differences are: (1) our use of threshold T. which needed to construct feedback that is sufficiently discriminative; (2) using 22% more contexts than in the offline data to adjust for the fact that using leads to roughly 18% the preferences (albeit low-signal preferences) being discarded. Therefore the volume of data seen by both offline DPO and online DPO is approximately the same, although the latter sees more diversity in contexts, which may provide an additional advantage. At the end of round, the policy is checkpointed, and at the start of the next round, the new policy and reference model are loaded from this checkpoint.
Online KTO is implemented the exact same way, albeit the final loss is calculated with 7 instead of
6). Notably, we construct DPO-style paired preferences before breaking them up to create unpaired data for KTO, instead of directly creating unpaired data using positive and negative thresholds. Not only does this allow for a better comparison with DPO, but also works better in practice. See Al Algorithm for the pseudo-code of Online DPO/KTO.
21

---

Published as a conference paper at ICLR 2026
Algorithm Online DPO / KTO
Input: initial policy model Te reward model ryp reward threshold prompts D; hyperparameters
8,
Adssmmble Aundesirable
Output: policy model RT
Initiate policy model Tq <- 40uk
2: for round to N do
3: Set reference model Jie + Te; Dwain + Ø
4: Sample batch of contexts Dj from D, where [D, 1024
5: for prompt E Do do
>> Online sampling and relabeling
6: Sample G outputs {u:) [= TT( x), where G Compute rewards {r, [i=1 for each y via rp if max, rj miny Tyy 2 then
7:
8:
18% samples will be filtered out
9:
(ywr y1) & (arg max, Tj arg miny r5) Drrain t- Dirain U {(x, yor )x
10:
11:
end if end for
13:
for batch E Durain do
Train with newly generated preference data
14:
Compute token-level surprisal f. for every token in (y Vr via xy and Tee
15:
Update Ty by maximizing the DPO KTO objective (Eq. 6
16:
end for
end for
Algorithm Offline GRPO
Input: initial policy model Te... prompts and completions D; hyperparameters B,
Output: policy model my
Initiate policy model Ta To.. reference model Tee f πg
2: for I, .... M do
3: Sample a batch D, from
Train with off- off-policy preference data
4: for prompt D, do
5:
b Default group size = Compute token-level surprisal f.,, for every token t in {3 y1 via Te and Net
Set G {ywr, 3 from off-policy (æ,, Dor 91) tuples
6:
7:
Compute Å,, for every token in {y vI through group relative advantage estimation 8: Update Ty by maximizing the GRPO objective (Eq.
9:
end for
10:
end for
C.1.3 OFFLINE GRPO
Given that GRPO inherently online method (Shao al. 2024), we make few different changes to create an offline variant, which largely follow those made by Ethayarajh et al. (2024) to make an offline variant of PPO. For one, instead of sampling new completions, we take tuples (x, Vu 31) in an offline preference dataset (e.g.. UltraFeedback (Cui et al. 2024)) and treat them as group of two: {y 122 = (yu: yr The reference model is never updated: its weights remain those of the policy at initialization. See Algorithm| for the pseudo-code of offline GRPO.
C.1.4 ONLINE GRPO
Instead of sampling every step, we sample data as in Online DPO/KTO, the only difference being that we retain the raw scores from the scoring step so that they can later be fed into the loss calcula- tion (I). We use this approach to allow for an apples-to-apples comparison with Online DPO/KTO, as well as because some asynchronicity is usually permitted in practice and we find that sampling once per round (i.e., roughly every 32 steps) does not degrade performance. A consequence of this choice is that the group size is exactly 2 for all contexts, making the relative advantages either -1 or 1. Naturally, this does not unlock the full potential of GRPO, since one of its strengths its ability to leverage scalar rewards. However, we consider more important that the volume of training data to be roughly the same across different variants and methods. We also use DAPO-style normalization
22

---

Published as a conference paper at ICLR 2026
(i.e., taking the average loss over the number of tokens in the batch instead of within sequence), as we find this leads to better performance on instruction followingg (Yu et al. 2025). Lastly, we reuse the reference model the base model for calculating the KL penalty, both because it saves us the space storing a third model and because prior work has identified the KL. penalty to not be of much import, allowing its estimate to be less precise.
C.2 HUMANLINE IMPLEMENTATION
The instruction foolowwing experiments were done in fork of the HALOs repository. Below, we provide relatively straightforward implementation of the humanline design pattern for DPO, KTO and GRPO, which is triggered by setting sel f. conf ig. human ine True in our codebase.
The mathematical reasoning experiments were implemented in fork of the Open-RI repository (Hugging Face 2025), which itself is based on Huggingface's TRL library. Because of this, hu- manline syncing is implemented differently, by over-writing callback methods: before the optimizer step happens, the current state of the policy is stored locally; at the end of the step, the stored policy loaded into the reference model. This improves stability when doing distributed training with ZeRO2 (Rajbhandari et al. 2020).
C.2.1 HUMANLINE SYNCING
In our codebase, we first modify the training loop to implement humanline syncing. For the sake of brevity, we highlight only the relevant changes train () and omit code used for logging. The accelerator object is used to manage distributed training with FSDP in our own codebase:
def ()
2
self.optimizen zerooerreeerrreee sero_grad()
loss, metrics wel.fgee_bbatce metrics (batch)
sel self.accelerator. backward (1oss)
5
grad_norm self acl.facceeerator. cl io f.clffoppocoo moy.pa paramteers (),
self self.config.me. mor
if self.config... loss. syne stnesssseeeeeeeeeeeeeeeeeeeeeese reference self self.config humanline:
10
self htt.:syscc_eeeeeeee with pelicy()
11
sclf.eptimizer.steg
sclf.schcduler.stegg)
16
def
Update the reference model to have the policy weights.
***
20
if self self.batch batch b self, config. sync_ freq == 0:
state_dict self.acceleres self accelacceleratorr unwrap_moded aaw.ceeeeeertttr..rrrraemneeeeee (self. referrence_mmdel) load__otate_di diet (state_dict)
wwwwap_modell (self. lcyl state_dict
self watp:rrrttttttttttteeeeeeeeeeeee ()
wait
Humanline clipping is even easier to implement, although it has to be implemented in two different
places our codebase because of the different abstractions used for DPO/KTO and GRPO:
def get_sequence_rewatt (self, policy_logps: torch.Floate Floatrensor, lfference_logper torch.Float Float 4 length honnthhnoomalliadPPaaess. ): "**
2
3
7
reguiar alignment, return the surprisal for the sequence (log (policy (ywf///eeeeeeceeeeeeeee ()
8
This is called the "sequence reward", following DPC terminology. Apply humanling ihe if speci fied.
23

---

FloatTTnoo,,,,sorr, Published as a conference paper at ICLR 2026
Args:
policy_logpet token-level probabilities according to policy mntcrobatch_eize0 maximum sequence length)
14
reforence_ token-level probabilities according to reference model (microbatch_size, maximum sequence length)
length_ divide the sequence reward by the number of non-rejected tokens
18
19
Returns:
The sequence-Level rewards (microbatch size,
23
if self hronfig.humanlig humanline:
token_rewards (policy_legps reference_logss) clamp(
self.confa epsiccnn self. self.config. Ltr.tiig..ogg_ppp.eee_eeeee epsilon RJ
else:
tcken_rewards policy_logps reference_loges
28
= (token_rewards.abs) () 1= 0) flcat () sum(-1)
if Lormtl_dermalized else
sequence_rewarde token_rewards. sum(-1) normalization factor
return requence_rewarrd
class DPOTrainer (Paireed
def lcss (self,
batch: Dict,
policy_chosen_loop:: flrch..loattessor,
policy_rejected_ logps: Float
reference__hooem__ torch, FloatTesor, reffereneerejeccte__ooo *args,
):
"""ooppute the DFO loss for a batch of policy and reference model token-leve! log probabilities. amn
apply humanl ine clipping via ret_seguence_ rewa rds teven-levei log probabilities before they fed into loss computation chosen rewards
www.eesseeeeneeeeeeeeeee reference reference_chosen ctt_seheoogsss logps)
(policy_chosen, chosen logsee
rejected rewards = self. weelfleee_seeeeeeeeee
seeuttsseeeeeee__eee rewards gpolicy_refected referenne_rrjeectedoooo
chosen_rewards + self.conie bots.beta
re jejected_rewardde = self. loss. beta
lcsses logsiggnno logsigmoid gmoid (chesen_rewarde
rejected_rewards)
60
return losses, roosen__ewwrdsddeee
rewardsd ()
class KTOTrainer redreereerreeeeeeennnettlrrrrri
def
loss (self, batch: Dict, volicy_chosed ctwp_chhssen_osss:: torch. FloatTenrr,,,,,
policy_rejjected logps: Float.
policy_Kt_logps: Floateen,,,,,,,o,
69
fcrch.FloatTonsoz.
refercnce_rejeetedd
Kefferece__K__100 KL, logps: torch. Flach.nloat *args,
73
):
74
"*"Compute the KTO loss for à batch of policy and reference model log probabilities.
24

---

Published as a conference paper at ICLR 2026
If generation y o desirable,, have the 'desirable' loss: Líx, y) : sigmold (beta (flog p_policy
79
-log p_reference (ylx)] KL (p_policy 11 P_roforence))) If generation q_undesirable, we have the 'undosirable' loss:
L(x, : sigmoid (Kipp_olic 11 p_reference) [log (_lolicy(ylx) log p_reference (ylx)
84
The desirable losses are weighed by desieaaaeee.esrrbeeeeeeeeeeeee The undesirable Losses are weighed by config. losiig.ooss... undesirable weightt This should be used to address imbalances in the ratio of desirable:undent rable examples respectively.
88
The Kh term is estimated by matching x with unrelated outputs y', then calculating the average log ratio log p_policy (x)
log p refeerence (x).
***
if policy_chosee w shape != 0:
93
chosen_rewarde sclf wolff.get_scueence_cooo
policy_chosen_logs, reforence_chocon_loge
else:
chosen chosen_rewards rewards Tensor (I)) to self.accelerato..dee
to
97
98
if
shapel]] != U:
rejected rewards self get..get___.eeeeeeeeeeeee.eeeeet sequence_rewards/
rewards
101
rejour__ejeeceed_egeeee reference_rejecteeene
102
else:
103
rejected_rewards - Tensor
Tensor ([]) to (self, polpyy..pocccddyppee to
self self.
106
For кTO, humanline also applies to the term
107
KL rewards saaf.get_sequene_reewaaee rewaard (policy_KL_ dettss detach (),
108
[iterence_Kt_logee (espssh)ctcch)))
KL (KL_ / (KL_rewards.e abs != 0) .float () sum .item() .clamp (min=0)
110
111
112
if pel hwwpchoseennoossshhpppe. !- 0:
chosen_losses self.cenfic loss doss.desi rable e_wee
(1 F.si gmoid s. (self.config. loss. loss.beta (chosen_rewards KI)))
115
else:
116
chosen losses accleecceeertoor.. device)
Tensor (11) to (self. (selffpolic...........o policcydtyeeee I
119
if p shape
120
rejected_losses = www..cco..fccccccccooccocccocccc loss, undeoirable__wiiqh
121
(1 F F.s.gmoidi (sef. loss.bbb..l.s...b.....ll beta (KL rejected_rewardss)
else:
rejected_ losses Terch.Tensen (1) (oll: (
124
self self.accelerator.ddvvve
125
126
Losses corch.cat (chosen_ Lesses),) 0)
losses, Kb.detach ()
dettaarddddedchhh... detach rejected_rewardd.com (),
129
For both DPO and KTO, we apply the same token-wise likelihood clipping with the function get_sequence.rewarke
as shown above. For GRPO, we do the same but with the function get_ratios to return the probability ratio under the policy and the reference models instead of the log probability ratio, as defined in the clipped surrogate objective. Note that we clamp in log- space for greater numerical precision.
1 def get_ratica (self, policy_logps, reference__ogge ):
25

---

Published as a conference paper at ICLR 2026
regular alignment, return the token-level probability zatio under the policy ve the reference (policy http://///ccccccccccccco Apply humanline if specificd.
7
Args:
policy_logps, token-level probabilitia according to policy (microbatch_ size, maximum sequence length)
12
reference reference_Loges logps: token-ievel probabilities according to reference model (microbatch size, maximum sequence length)
16
Returns:
The probability ratios (microbatch_size, sequence length) if requence_levell otherwise, (microbatch_ size, 1
if self (policy_logps reference_logps) . sclf. solf.confgg... 1og_ self. Log_opsilon__
logratic =
else:
legratio policy_logps
reference_logps
ratic = Logratio.ex0 26
return ratic
30
class GRPOTrainer (BasicTrainer):
def lcss (self,
batch: Dict, policy_logps: torch. FlcatTensor, rofcrence_logps: torch. Flcatrensor, advantages: Forch.FloatTenser:: group_size: Float.ee,,,, *args, ): #**
35
Compute the GRPO 1os8.
44
Args:
policy_logps: logps: log probability of the output under the policy imicrobatch_size, peeuence__eegthh reference_logps log probability of output under the reference model aiir soquence_length advantages: sequence leve! advantages
48
(isrtechhsizze group_size size: number of outputs (in entire batch) belonging to prompt associated with sequence (microbatch_size,
53
Returns:
average loss, average KL, average weighted advantage, average unweighted advantage
***
apply humanline clipping via get_ratios on token-level
log probabilities which returns probability ratios
ratic self.get_ratios
ratios (policy _oopss, reference reference_logpst
masks (batchi"target_laee labellsss 1:] 1= -100) cllone( to sclf.policy_dtype
62
advantages advantages unsqueeze (-1) grcup_size group_size unsqueeze(1]
weighted_ade adv advantages ratio
67
probability ratios get clipped again in the GRPO surrogate objective controlled by separate hyperparaneter epsilon weighted adv elipped advantages aatio.clamt
26

---

Published as a conference paper at ICLR 2026
1 self.config loss epsilln, self.config. loss epsilon)
72
humanline clipping does not apply to KL term in GRPO ger_token_Kee KL = torch.exp (reference__oggg (referance_logps per per_token_loss loss torch.miii (weighted_ade adv, weighted_adv_cliipee
policy_logps)
pelicy_logps
77
+ self.config loss. beta * per_token KL
do DAPO-styLe normalization
masked_mean (per_tcken_loss,, masks), masked_mean bee__tokk__K..detcchh (), masks),
masked_mean weeighted_ddv.abs () detach (), masks),
advantages.. abs () mean ()
27

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
Table 1: Hyperparameters that are common to all of our instruction-following experiments, across different alignment objectives and different variants for each objective. Note that we generate much more data than we ultimately use during online training (see Appendix [c.]] for details). Note the data volume for KTO/GRPO twice that of DPO because the latter operates on paired preferences containing two sequences each.
Hyperparameter
Value
Optimizer
Adam
A W 31
0.9
Adam W $2
0.999
Adam W
1e-5
Weight Decay
1e-2
Warmup
10%
Offline Training Examples
10K (DPO) 20K (KTO, GRPO)
Offline Batch Size
32 (DPO) / 64 (KTO, GRPO)
Online Training Contexts
12288
Online Batch Size
32 (DPO) 64 (KTO, GRPO) 1024
Round (Number of Contexts)
Generations per Context (sampled)
Generations per Context (after filtering)
2 0
Maximum Generation Length
2048
Top-p (Nucleus Sampling)
0.95
Sampling Temperature
0.7
Reward Threshold
0.01
Humanline log eP
-1.5
Humanline log F.R Humanline k
1.5
Table 2: Hyperparameters that are common all of our mathematical reasoning experiments with GRPO. We use the setup in Huggingface's Open-R1 (Hugging Face (025). except instead of plac- ing equal weight on the tag count, format, and accuracy rewards, we place weights of 1, 1, and 8 respectively (i.e., emphasizing accuracy over the rest). Doing humanline syncing every step (k = 1) will lead to collapse this setup because of the smaller models involved; h: E [12, 20 closes the gap with the online reward curves within 1000 steps.
Hyperparameter
Value
Optimizer
Adam W
AdamW $1
0.9
AdamW B3
0.999
AdamW
1e-8
Weight Decay
0
Warmup
10%
Learning Rate
le-6
Max Gradient Norm
Training Contexts
12K
Batch Size (incl. duplicate prompts due to groups)
256
Group Size
8
Maximum Generation Length
2048
Top-p (Nucleus Sampling)
1.0
Sampling Temperature
0.7
Humanline log cp
-1.5
Humanline log CR
1.5
Humanline
[12, 24]
28

---

Published as a conference paper at ICLR 2026
Table 3: The performance of Llama3-88 BBB 1123-B--ING -----BIIIuctt trained with all variants of all objectives on AlpacaEval2, along with the objective-specific hyperparameters. Note that while humanline alignment usually reduces the average length (Length) of completions, this is not universal char- acteristic of humanline variants, but of the data and hyperparameters used; results from training on different version of UltraFeedback, where completions are sampled from different model, lead to offlnn=+hmannine variants having roughly the same length as their offline counterparts (Table The hyperparameters that require the most adjusting are the learning rate (LR) and the maximum gradient clipping norm (Max Norm); going from offline to online requires the LR and Max Norm to be scaled by 0.5x-Ix, but adding humanline variant top can increase or decrease the Max Norm (see $5.2.1 for a discussion of why). Objective-specife hyperparameters remain fixed across variants to allow for fair comparison, with the exception of Offline GRPO which needs to be much larger when the reference model is fixed.
Objective
LR
An
Ay
Max Norm
LC-WR
1
Length
Offiine KTO
5.0c-6
0.25
1.1
1.0
17.40
14.13
1658
Offlnee+ KTO
5,0e-6
0.25
1.1
4.0
22.19
14.70
1407
Online KTO
2.5e-6
0.25
1.1
0.5
22.45
19.47
1744
Online+Humanline
2.5e-6
0.25
0.1
22.79
18.78
1663
Objective
LR
Max Norm
LC-WR
WR
Length
Offiine DPO
5.0e-6
0.10
1.0
18.07
16.07
1767
Offine-Huumnline DPO
5.0e-6
0.10
1.0
24.82
20.18
1637
Online DPO
2.5e-6
0.10
0.5
24.96
22.99
1828
Online+Humanline DPO
2.5e-6
0.10
1.0
26.84
23.64
1774
Objective
LR
Max Norm
LC-WR
WR 1
Length
Offline GRPO
5.0c-6
0.01
0.50
1.0
15.52
12.61
1648
OfflinieHumanline GRPO
5.0c-6
1.0
25.24
18.11
1488
Online GRPO
5.0c-6
0.01
0.15
1.0
25.05
18.82
1529
Online+Humanligg GRPO
5.0e-6
0.01
0.15
0.5
26.10
21.57
1647
Table 4: AlpacaEval2 results when of offline UltraFeedback ArmoRM (Meng et al., 2024), one where completions are generated by Llamaa--88 GoB-Instruct (a separate unaligned version producing offline data, not to be confused with the policy) and another where completions are generated by Gemma - ruct. The contexts the same in both cases. Significant differences (p < 0.05) are highlighted in red. Although the performance is not significantly different in most cases, when trained on the L. amaa completions, the offline-humanline-lll offline counterparts; when trained on the Gemma2 completions, this is not necessarily the case. Using humanline variants does not permit one ignore the data, will always make difference in the quality of the aligned model.
is trained on two different versions
policies tend produce shorter completions than their
Mamaa-88-IIntruut Data
Gemma2-99 -----------nn-c-- Data
Objective
LC-WR
WR t
Length
LC-WR
WR f
Length
Offline KTO
17.40
14.13
1658
18.10 10
15.28
1698
OfllineHummnnine KTO
22.19
14.70
1407
22.18
20.30
1836
Offline DPO
18.07
16.07
1767
18.63
15.44
1690
OfflineHumannine DPO
24.82
20.18
1637
26.26
21.91
1642
Offline GRPO
15.52
12.61
1648
12.64
10.84
1696
Offlnn+Humanline GRPO
25.24
18.11
1488
24.24
18.99
1587
29

---

Published as a conference paper at ICLR 2026
Table 5: AlpacaEval2 results when ferent versions of offline UltraFeebbcck ArmoRM (Meng et al. 2024). one where completions are generated by Liama -- 10--8--nnn 133-Instuct
is aligned with DPO dif-
(L3-8B) and another where completions are generated by Gemma2-9B I (G2 - -B), The contexts are the same in both cases. Note that using the ofuline+humanline variant only able match the performance of the online variant when the
offline off-pollcy data comes from the latter of the two sources.
Objective (DPO Variants)
LC-WR
WR 1
Std. Err
LR
Baseline
45.90
35.45
1.54
Offline (L3-8B 8B Completions)
48.59
32.36
1.58
2.5e-6
0.3
Humanine (1.3-8B Completions)
56.27
44.49
1.67
2.5e-6
0.3
Offine (G2- 9B Completions)
56.58
45.17
1.68
2.5e-6
0.1
Offlnineummanine (G2-95 Completions)
67.45
61.37
1.64
2.5e-6
0.1
Online
66.49
74.22
1.48
2.5e-6
0.1
aligned with humanline DPO with different choices of humanline clipping hyperparameters Ep, Humanline syncing is done every step (A 1) and other hyperparameters are fixed. The performance of the aligned model fairly robust to the choice of clipping values in both directions, with most length- winrates (LC- WR) falling within the standard error of the highest one. However, the length of the outputs does grow monotonically the clipping range gets looser.
Table 6: AlpacaEval2 results when -aaaam----Inntteee
log Cp
log CR
LC-WR t
Std. Err
Length
-2.0
1.5
23.33
1.35
1663
-1.5
1.5
24.82
1.36
1637
-1.0
1.5
24.37
1.33
1588
-1.0
2.0
22.55
1.31
1619
1.0
3.0
23.98
1.33
1636
with humanline GRPO, the performance on instruction-follooing---- -meawin here the length-controlled winrate against GPT-4-Turbo baseline-is is robust to the frequency of humanline syncing up to frequently leads to a log-linear decline in performance. In other setups not shown here, such as mathematical reasoning experiments with Owen2. 5-1, 1eg-Instruct.. syncing less frequently not only beneficial but necessary, since anything less than it 12 introduces too much instability and leads to reward collapse.
Figure When aligning
Past that point, syncing less
30
GRPO (offline+humanline)
95% CI
25
LC Winrate (%)
20
15
10
2
21
2²
23
2
25
26
2
Syncing Frequency (log-scale)
30

---

Published as a conference paper at ICLR 2026
Figure 9: The performance benefits of the humanline variants of KTO/DPO/GRPO persist larger scale with different model families, with - 2 seeing a 1.15-1.30x improvement in performance. This is slightly smaller than the relative improvement seen by Ll -aaaam--B-Instzune and can be ascribed to the former being a better base model.
Offline
Offline Humanline
65
Length- Controleed Winrate (%)
+1.16x
+ 23x
+1.30x
(GPT A.l-as-a-judeee
50
45
DPO
KTO
GRPO
Figure 10: Trust region-style syncing (Gorbatovski et al. 2024) performs much worse than human- line syncing. In offlnne+tuust region, we sync the reference model with the policy after the update every 1024 steps, the best performing setup both in Gorbatovski et al. (2024) and in our hyperpa- rameter sweep. This suggests that enough to merely syne the reference model; the way in which it is done matters as well.
Offline
Offline + Trust Region
30
Offline + Humanline
LC Winrate (%)
25
20
15
10
DPO
KTO
GRPO
31

---

Published as a conference paper at ICLR 2026
Figure 11: Average wall-clock time for aligning L1ama3-88 8B Instruct UltraFeedback using FSDP across 4xH100 GPUs, reported with standard eπror and 95% confidence intervals across random seeds. Note that offline+humanlnne GRPO takes almost twice as long as offline GRPO due to the syncing of the reference model weights. However, this is still less than 1/6 of the time needed run online GRPO (without any overlapping of training/inference) while reaching the same performance (Figure
w/o humanline
w/ humanline
Wall-Clock Time (hr)
5
4
2
Offline GRPO
Online GRPO
32

---

Published as a conference paper at ICLR 2026
Table Generations from the different DPO-aligned versions of Gemma2 -tnstttuuuttIItutttttttt given a simple math question. Note that only the offline+humanlnn output explicitly states the weighting idea, explains why simple sum/divide is only valid with equal weights, and then shows the per- quarter contributions but without the verbosity of offline DPO.
Prompt
this math problem, we are asked to figure out the percentage total profits that are made in each quarter. They are 90% in the first quarter, 92% in the second quarter, 81% in the third quarter and 88% in the fourth quarter. What is the total profit percent period of time, based upon these quarters?
Baseline
Here's how calculate the average profit percentage over the year: Add up the profit percentages: 90% 92% + 81% 88% 351% 2. Divide the total percentage by the number quarters: 351% % = 87.75% Therefore, the average profit percentage over the year is 87.75%.
Offline
Here's ' how to calculate the total profit percentage over the year:
1. You can't t directly add percentages like this. Percentages represent parts of a whole, and simply adding them could result number over 100%, which doesn't make sense in this context.
Think of weighted averages: Each quarter's profit percentage reflects its contribu- tion to the overall yearly profit. To find the total profit percentage, we need to calculate a weighted average.
3. Here's the formula:
8 Weighted Average (Q1% Q1 Weight) + (Q2% Q2 Weight) + (Q3% Q3 Weight)
+
(Q4% Q4 Weight)
Where Q1% Q2%, Q3%, and Q4% are the profit percentages for cach quarter.
* The weight for each quarter is typically 1/4 (since there are four quarters in ycar).
Calculation:
Total Profit Percentage (0.90 0.25) (0.92 0.25) (0.81 0.25) (0.88 0.25)
Total Profit Percentage 0.225 + 0.23 + 0.2025 + 0.22
Total Profit Percentage 0.8775 or 87.75 %.
Therefore, the total profit percentage
the year is 87.75%
Offline-Humanlnn
Here's 's how to calculate the total profit percentage over the 1. Understand it's not simple sum: You just add 90%. + 92% + 81% 88% 88% 351% % and divide by 4. Percentages represent portions the whole, simply adding them like this would imply profits exceeding 100%, which isn't possible.
2. Weighted Average Approach:
Think of each quarter's profit weighted contribution: The weight is the proportion
of the year that quarter represents (1/4 25%).
* Calculate the weighted profit for each quarter:
QI: 90% 0.25 22.5%
* Q2: 92% 0.25 23%
* Q3: 81% * 0.25 -0.225%
* Q4: 88% 0.25 22%
Add the weighted profits: 22.5% + 23% + 20.25% + 22% 87.75%
Therefore, the total profit percentage over the year is 87.75%
Online
Here's how to calculate the average profit percentage over the year:
1. Add up the percentages: 90% + 92% 81% + 88% 351%
2. Divide the total the number of quarters: 351 % 4 87.75% Therefore, the average profit percentage
the year 87.75%.
Important Note: This is an average. It doesn't mean the company made profit of 87.75%
overall. It shows the average across the quarters, which were individually quite variable.
33


Reviews:
[{"review_id": "LY0ByvZyBz", "reviewer": "Reviewer_Thz1", "summary": "This paper talks about why online on-policy methods tend to outperform offline off-policy methods, and offers a human-centric answer grounded in prospect theory. Specifically, the authors claim that (1) online on-policy sampling better approximates the human-perceived distribution of what the model can produce, and (2) PPO/GRPO recovers a perceptual bias in how humans perceive probability. They then propose a practical humanline design pattern applicable to DPO/KTO/GRPO: (1) humanline syncing (sync the reference with the previous policy every k steps) and (2) humanline clipping (asymmetric clipping of token log-ratio). Empirically, the authors show that offline + humanline matches online performance on instruction following and allows 64-times less frequent sampling without degradation on math reasoning, which gives large speedups versus standard online training.", "questions": "See weakness. Happy to raise the score if authors could help clarify on the above issues.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\n1. The paper reframes the online/offline gap through prospect theory from behavioral economics, which is quite a unique point of view. Though it is not the first paper to extend prospect theory to alignment, it considers the probability weighting and the inverted-S curve shown in the paper intuitively explains why online sampling is closer to the human-perceived distribution. \n2. The paper theoretically links GRPO/PPO clipping to perceptual weighting via rejection sampling and limit arguments. This gives a fresh interpretation of a widely used heuristic.\n\n## Quality\n1. All claims are backed with theoretical proof. \n2. Empirically, the paper gives a practical yet minimal recipe to highlight the change. Such a plug-and-run recipe can be useful for future researchers. \n\n## Clarity\nThe presentation is clear and easy to follow. There is a good example throughout the prospect theory section making it easy to understand. \n\n## Significance\n1. If robust, the result meaningfully relaxes the need for fully online training. This would significantly improves the overall training walltime while maintains the same level of performance, greatly speed up the model iterations.", "weaknesses": "1. Lack of implications of assumptions. The authors did not cover enough discussion on assumptions 4.1-4.3, e.g., what do those assumptions imply in the language of LLM alignment? Especially for 4.1, same support and finite likelihood ratio seem too restrict in LLM as the action space is enormous; for 4.3, does LLM usually gives you light tail? Any data/study support?\n2. Offline data quality dependency. The paper also shows that not all data could match the online performance, but stops discussing more on the implication and in practice how this could be resolve or any efficient ideas to detect an offline dataset with \"good quality\". Within good/bad data, do they share some common attributes?\n3. Data–method confound. Given the offline data quality dependency, can naive offline methods without humanline achieve comparable performance by a carefully selected set of data? If so, then how can we verify the efficacy of humanline? Some carefully designed abalation tests on this would be good.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWDuQVYj6B", "reviewer": "Reviewer_fVrX", "summary": "In this paper, the authors provide a theoretical study of the reasons why online alignment is better than offline alignment. In particular, this work shows that online on-policy algorithms are good at approximating human perception. Moreover, the offline and online training have been shown they be identical in maximizing human utility. Motivated by these investigations, the authors propose a new design for the training algorithm, which is evaluated on experiments.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work studies an important domain - trying to explain why online algorithms are usually better than offline algorithms.\n2. The presentation of this paper is easy to follow. \n3. The theoretical and empirical investigations are both provided.", "weaknesses": "1. From the experimental results, the humanline indeed can help with offline alignments following the lack of human perceptions. However, it is not clear why the online alignments (the authors have shown that those online alignments have demonstrated the capacity for modeling human perceptions) can also be greatly improved with humanline.\n\n2. The modeling of the value function as well as the utility function is hypothetical, which means that if they reflect the true system is not clear; otherwise, the authors need to verify them.\n\n3. Based on the definitions under prospect theory, the authors provide a new analysis for answering the question about the comparison between the offline and online alignments. However, from another perspective, the offline off-policy is indeed a more challenging process compared to online on-policy, which is more computational cost but involves more exploration opportunities. In this case, how to validate the benefits of online alignments is mostly from the explanation under the prospect theory? \n\n4. As the performance difference between offline and online alignments is also witnessed in large model alignments. I am wondering how to verify that the phenomenon observed for small models can also be extended to large models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KM3JrI6O2s", "reviewer": "Reviewer_r7Xi", "summary": "This paper proposes a human-centric explanation of why online, on-policy alignment (e.g., GRPO/PPO) often outperforms offline, off-policy alignment (e.g., DPO/KTO). The key idea is to model human utility via prospect theory and include probability weighting (capacity) in addition to the subjective value. From this, the authors argue that sampling from the current policy better matches the human-perceived outcome distribution, giving intuition on why online > offline.\n\nOn the practical side, they introduce a two-part humanline design pattern applicable to DPO/KTO/GRPO: Humanline syncing and Humanline clipping. Empirically, humanline variants trained on offline data close the observed performance gap with online variants on instruction-following and allow 64× less frequent sampling in math-reasoning. Ablations suggest syncing provides the bulk of the gain; upstream clipping adds a smaller but positive effect.", "questions": "**Q1.** Could the authors provide quantitative results (e.g., support overlap or KL divergence) to verify when Assumption 4.1 holds and how it relates to performance differences when using different datasets?\n\n**Q2.** See other questions in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1. Originality:** Extends alignment as prospect-theoretic optimization by explicitly modeling probability weighting (not just value).\n\n**S2. Practicality:** The humanline recipe (syncing and upstream asymmetric clipping) is simple to implement and integrate into existing algorithms.\n\n**S3. Quality/robustness:** Ablations isolate the roles of syncing vs clipping; results hold across tasks and model sizes.\n\n**S4. Significance:** If adopted, the approach can lower alignment cost and increase offline data usage without sacrificing quality.", "weaknesses": "**W1. Dependence on offline data.**\nThe method depends on the choice of offline dataset (e.g., offline data generated by Gemma-9B vs Llama-8B). The paper attributes this to a possible violation of Assumption 4.1 (support overlap and bounded likelihood ratio), but never verifies it empirically. It would strengthen the work to measure these assumptions directly. For instance, by reporting a coverage or divergence metric and examining how they correlate with performance across datasets.\n\n**W2. Gains stem from syncing, not clipping.**\n\nThe ablation study shows that most performance improvements come from syncing the reference model ($\\pi_{\\text{ref}}$). In contrast, the proposed clipping mechanism, a central contribution of the paper, performs comparably to the offline baseline when used alone. This suggests that clipping contributes little to the observed gains while adding two extra hyperparameters and complexity.\n\n**W3. Unfair comparison to trust-region baselines.**\n\nThe comparison to trust-region methods is not hyperparameter-fair. The baseline’s update frequency $k_{tr}=1024$ was taken directly from the original paper, where it was tuned for a different task. In contrast, the proposed method’s equivalent parameter $k$ is shown to be highly sensitive and was tuned for this work. The trust-region baseline could likely perform better if $k_{tr}$ were tuned under the same conditions, so the reported advantage may be overstated.\n\n**W4. Weak theoretical justification.**\nProposition 3.4 only suggests that a small KL divergence is beneficial; it does not theoretically prove that online > offline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgjVIrN9cy", "reviewer": "Reviewer_xZQd", "summary": "The paper studied the problem of reasons and explanations of online alignment better than offline alignment. Through the lens of the prospect theory framework, the authors provided a human-centric explanation. By proving PPO/GRPO in online on-policy clipping recovers a perceptual bias, the authors showed that online alignment acts as a perceptual loss. With these findings, the post-training is more efficient without restriction on the data source and without the cost of performance. Finally, they also test their results in an unverifiable reward setting to follow open-ended instructions and a verifiable reward setting for the math reasoning task.", "questions": "1. How is Assumption 4.3 in practice, especially \"The cumulative probability of outcomes with higher absolute surprisal than $z_i$ is negligible\"? Is this assumption also used in previous work? How did your experimental setting satisfy this assumption?\n2. What is the intuition of Definition 4.5 to propose humanline sampling?\n3. The term perception plays a central role in your framing, yet its precise mathematical definition is unclear. Could you provide a more formal explanation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The perspective to explain online alignment is interesting by adopting some methods from behavioral economics.\n2. The writing of the paper is clean. The authors clearly state the problem, methods, and their results.\n3. Most of the related work is cited and discussed. And the authors provided additional related work in Appendix A.\n4. The paper provided both theoretical results and empirical results.", "weaknesses": "1. The assumption is strong, e.g., Assumption 4.3.\n2. Some statement is not rigorous. For example, \"If the success of PPO/GRPO can be ascribed to them being perceptual losses\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)