Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Authors: Yueqin Yin
Shentao Yang
Yujia Xie
Ziyi Yang
Yuting Sun
Hany Hassan Awadalla
Weizhu Chen
Mingyuan Zhou
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Previous RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. Our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.

Published in Transactions on Machine Learning Research (09/2025)
Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Yueqin Yin
wwegin.giin@uueens.ce
The University of Texas at Austin, Microsoft
Shentao Yang"
shentao, gengglmncccmbs..ueeee....
The University of Texas at Austin
Yujia Xiel Microsoft
yugiaaiie@mmccrooofftccom
Ziyi Yang! Microsoft
aldyangg@microooft.com
Yuting Sun
vutingsun@miccroooft.ccom
Microsoft
Hany Awadalla
hanyhh@microoftt.com
Microsoft
Weizhu Chen
wzchenn@microsoft.comm
Microsoft
Mingyuan Zhoul
mingguan.ahouu@mccoml..uuu
The University of Texas at Austin
Reviewed on OpenReview: https: openreview. net/ forum? id= YALLgDOUN
Abstract
Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Previons RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token In this paper, soek to get the best of both by training and utilizing segment-level reward model, which assigns reward to each semantically complete text. segment that spans over short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequenca-preferene datasets. For effective RI-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. Our method performs competitively three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method. Our code can be viewed at https//ww........e.ee///////n/ea
action may be oversubtle to proper reward assignment.
"Equal contribution. This work done during internship Microsoft.
i "Corresponding Author.

---

Published in Transactions on Machine Learning Research (09/2025)
1 Introduction
Reid et al.) 2024 with human values, reinforcement learning (RL, Sutton & Barto 2018 methods have been widely adopted to optimize the non-differentiable human preference, leading to the paradigm of reinforcement learning from human feedback (RLHF, (uyang et al. 2022 Bai et al. 2022b). A prevailing approach in RLHF is to optimize the LMs by proximal policy optimization (PPO, Schulman et al. 2017) against a bandit reward model Icarned from human preference data, with KL regularization lowards a pre-specilied larget distribution to avoid over-optimization on the reward model (Ziegler et al., 2019: Stiennon et al. 2020 Castricato et al., 2022). While this bandit approach is easier for reward modeling and has achieved remarkable success, language generation is intrinsically sequential, rather than simultancous. Thus, from the view of optimizing human preference, assigning bandit reward to the entire text sequence induces the sparse reward (delayed leedback) issue (Andrychowicz
To align language models (LMs, e.g.,
OpenAT 2023
et al. 2017; Marbach & Tiitsiklis 2003), that often hurts RL-based LM training by increasing gradient
variance and lowering sample efficiency (Takanobu et al. 2019 Wang et al. 2020 Guo et al. 2022 Snell
et al. 2022).
To mitigate this sparse reward issue, prior works have developed methods to "ground" the sequence-level preference label into a dense token-level reward model y (Yang et al., 2023 Zhong et al., 2024]. While dense per-token reward signal reduces the optimization complexity (Laidlaw et al.) 2023). each action, however, is then defined as a single token, i.c., a sub-word that is finer-grained than a word, especially with the BPE-style tokenizers (Gage) 1994 Sennrich et al. 2016). For instance, Llama 3.1's tokenizer (Dubey et al. 2024] has tokens as (Brit, ce, cod, neo, redd, that have less clear semantic meaning per se in any given context. The contribution of those tokens to the text sequence will inevitably depend on later tokens, making reward/crode /crodit assignment harder, espocially under the prevailing RLHF paradigm of implementing the reward model as an ofl-the-shelf decoder-only transformer (e.g., Ouyang et al., 2022, Bai et al., 2022b; Menick et al. 2022). Further, token-level reward implicitly assumes that the basic unit of a text sequence is token, which may not follow linguistics, where more meaningful decomposition of text may be phrase (including word) that can be more semantically complete and generally consists of a short sequence of tokens.
To retain the optimization benefit of dense reward for RLHF, while mitigating its reward assignment issue and linguistic counter-intuition in this paper, we seek to train and utilize a segment-level reward model, which assigns reward to each semantically meaningful text segment that constitutes a small number of (or just one) tokens. With this design, we define the action space in RLHF "text segment," interpolating between the finest "per token" and the coarsest "full sequence" and potentially getting the benefit of both worlds: easier RI-based LM training owing to denser feedback and more accurale training guidance from the semantic completeness of each action. Although prior work has explored fine-grained RLHF through sentencc-level foodback (Wu et al. 2023), such mothods often rely on manual annotation or external APIs, and may miss finer compositional structures below the sentence level. In contrast, our method automatically segments text based on predictive entropy, offering fully automated, segment-level feedback from only binary sequence preferences.
Technically, we are motivated by prior works (Malinin & Gales 2018 Li et al. 2024a to dynamically segment a text sequence by thresholding the entropy of LM's predictive distributions, under the assumption that tokens within a semantically complete text segment can be more certainly predicted by prior tokens, while the beginning of a new segment is not (Wang et al.) 2024b). To allow training the segment- level reward model by the standard sequence-preference labels via Bradley-Terry (BT, Bradley & Terry 1952) loss, we differentiably aggregate segment rewards in a text sequence into a parametrized sequence cvaluation. The learned segment-level reward model is then utilized in PPO-based policy learning, where we observe the unsutabiiliy of classical reward normalizers, ie., the mean and standard deviation (std) of full sequence rewards. We address this issue by generalizing the classical bandit normalizers of scalar mean and std into a mean and a std function that output the reward normalizers at arbitrary locations of the text sequence. In addition, we enhance PPO training by within-segment reward interpolation, which further densifies training signal and improves results.
We test our method on the performance of PPP-trrined LM policy, On three popular RLHF benchmarks for LM policy: AlpacaFval 2.0, Arena-Hard, and MT- MT-Bench, our method achieves competitive performance gain
2

---

Published in Transactions on Machine Learning Research (09/2025)
Segment-level Reward Training
PPO-based LM Policy Learning
Prompt x:
Policy LM TT
What the fundtion of rooks in plants?
Sampled Resporse:
Entropy
Roots unchor the plant in thescal and alborb water und ruttiients.
Reward Model rip
Seginent Iegir
Rodts anchor the plant absuch watee nutrients.
DeitAnn BOUl a Eruit with seed and
beet milk suil pecuttssle
Ce og)
(a())))))) Paits
Reward aotaatiittion
Patr,
roles. 0.5
Oa ogl 1.4
Tpltt.ng) 1.6
Aggregation Function fO): Average
Rools
anchor
plant in soil absoch water nutrients.
Reward Interpolation
ex(x,y")
cp(x,y')
Figi:
02
02
Loss: Liog ($))
RLHF PPO Training
Figure 1: Overview of training and usage our segment-level reward model. Numerical values shown are illustrative. Each text segment is represented by a different color, with its starting word underlined in the figure.
against both the classical bandit design and the recent token-level design. We conduct extensive ablation studies to verify our design choices and further probe into our method.
2
Main Method
2.1
Notations and Background
In this section, will define generic notations, provide background the classical bandit RLHFF, and then discuss RL formulation of LM generation underlying recent elforts on dense-reward RLHF.
Generic Notations. Both reward modeling and LM policy learning require text prompt x and the corre- sponding response y. Reward model training turns the supervised finc-tuned (SFT) model TsFr(-I)) (without the final unembedding layer) into a parametrized scalar-output. model ro(:.) with parameter & that scores its input. The LM policy सe is then optimized against T
Bandit Reward Model Training. Reward model training assumes dataset Dpret {(x, yd (')))) of prompt and the corresponding winning/chosen chosen response y and losing, rejected response v², where the label comes from human evaluation on the entire text sequence y and y¹. In the classical bandit RLHF, reward model rs is trained by the binary classification BT loss
Lbandit (Φ) - -x(2, - DDDerr [loga (ro(x,v") - -r (x( v1)]
(1)
where (u) 1/(1 + exp (-u)) denotes the sigmoid function.
PPO-based Bandit Policy Learning. In policy learning, set Dpot {x} of text prompts is given. The LM policy is trained to generate outputs on Dpot optimizing the bandit reward from To, with KL penalty towards TSFT to avoid reward over-optimization. Collectively, the objective is
(o [o(U|z) (x)
max E
a~ pol you (x)
ro(x, 4) - 8 x log
(2)
TSFT
3

---

Published in Transactions on Machine Learning Research (09/2025)
where 3 is the KL coefficient. In practice, for PPO's training stability, the value of r2(2,)) y) is de-mean and de-sid normalized based on statistics calculated on calibration dataset, e.g,, Prref-
RL Formulation of LM Generation. By its sequential nature, LM generation is formulated as a Markov Decision Process (MDP) M = (3, P.R,7) (Sutton & Barto, 2018). y. Concretely, for state space 8, the state at timestep 1, 8t, consists of the prompt x and all generated tokens so far act =: (do...., with aco =: Ø, i.e., St =: [x, acc] A is the action space, where the action ar at step is a short-sequence/segneeee of tokens from the vocabulary in segment-level design, whereas a, is a single token in the token-level design. Transition function P deterministically appends the newly sampled tokens after the previous ones, L.c., S1+1 = (st, ax] [zr, aga]. r(s,a) S x A + R scores the action choice (segment/ /token selection) a at state/context 8 and is typically substituted by the learned reward model r ? E [0, 1] is the discount factor. In what follows, we will focus on our segment--evel design where each action a, E A is a semantically complete text segment, consisting of a non-deterministic number of consecutive tokens. The response y for prompt x then contains a variable number of geaments/actions generically denoted as y/ (ao..... 0r-1] where T is the number of segments in y and varies across responses. We denole single token in y as y whose generation context is [z, yei].
Fig. D
overviews key components in our method. A detailed algorithm box is in Appendix A
2.2 Reward Model Training
Overview. In training our segment-level reward model, we follow the data assumption set forth in Sec- tion 2.1 where the dataset Dpref = { y" "y'}} contains only binary sequence-level preference labels, without any process supervision (Uesato et al. 2022) The reward model ra(Se, at) is configured to output a scalar reward for each text segment choice a at the generation context 8t. ra is trained such that its induced pa- rameterized text sequence evaluations, aggregated from all segment-level rewards in the respective sequence, align with the preference labels in pref- This is inspired by the imitation learning literature (e.g., Christiano et al. 2017 Brown et al. Collectively, the BT loss for training our segment-level reward function To is
2019 2020) and prior token-level reward modeling in RLHF (Yang et al. 2023
Long (0) = - E (aay" D log (ea(x, - -esla:"")-oatt.) e
(3)
Vy E ",""""" es(z,y) (((ra(s)
where eo is the parameterized sequence evaluation induced by ré. constructed by aggregating all segment- level rewards (ra(se, (t)))aee in the text sequence y by a selected aggregation function ʃ(-).
Entropy-based Segmentation. As discussed in Section [ we intend to split the given text sequence y E {u""./ } into semantically complete segments, so that the reward assignment to each action (segment) can be easier, especially under the common implementatiin of the reward model as a casual LMM Recent works on LMs (e.g., Li et al., 2024a] Wang et al.) (202bb) have noticed that tokens within a semantically complete text segment can be more predictable by the corresponding generation context, since they are continuation of the designated semanties; whereas the starting token of new segment is comparably less predictable, its semantic binding with prior words is relatively weaker. For casual LMs. the predictability of each token can be conveniently measured by the entropy of the wext-token-predictii distribution from which the token is sampled (Malinin & Gales, 2018). To make text sequence segmentation a one-time data pre-processing in the reward model training stage, we choose to use the prediction distribution from the supervised fine-tuned model TSFT, from which the reward model is initialized before training. With a selected entropy cutoff Cent: token yi starts a new segment if the Shannon entropy H(-) of TSFT's 's predictive distribution of the i-th token surpasses Cent, d.e., H(TSPT(- æ. V<c)) > Cent, in which case Vi-1 ends the previous segment.
Choice of the Aggregation Function f(.). Aggregation function f(-) provides inductive bias on the relation between the quality of each segmen/fcctinn and the preferability of entire lext sequence. While several designs have been proposed in literature (Christiano et al. 2017 Kim et al., 2023 Yang et al. 2023) after looking into the datasot, in our experiments, we select Average to differentiably highlight the better average quality of the chosen responses over the rejected ones. With this choice of f(), the parametrized sequence evaluation es (x. y) in Eq. (3) is constructed as
4

---

Published in Transactions on Machine Learning Research (09/2025)
T-1 cs(2, v) = ((tra(8), T Σ (z)
(4)
An Alternative Interpretation. Comparing our segment-level reward training loss Eq. (3 with the classical bandit. loss Eq. 1 one may alternatively interpret eœ and [((ra)) in Eq. (3 as a re-parametrization of the learned sequence-level feedback that differentiably aggregates the quality/ /contribution of each text segment, and thereby connects a denser evaluation To of cach semantically complete text segment with the information in ground-truth sequence-level preference label.
2.3 PPO-based Policy Learning
Overview. In policy learning, we again follow the classical bandit setting in Section 2.1 Lo optimize the LM policy My on a given prompt set Poo = {=}. But unlike the bandit objective in Eq. 2 we adopt the full RL setting (Sutton & Barto 2018) to maximize ne expected sum of per-segment/stee rewards. This enables directly plugging our segment-level reward model re into most off- the-th RLHF PPO implementation. With this design, the policy learning objective for va is
To xr) TSST (3) 22) ]
max E
Σ
Erolst,an at) - $ log 1=
2 pal
Ta (t)
where again, each a is a segment tokens (chopped by TSFT), 8t = [z, ao.... a,-1] is the generation context at step i, and y = [ is the response to prompt æ sampled from the learning LM policy Te.
Recall from Section 2.1 that the output values from the reward model Te need to be normalized for the stability of PPO training. With our segment-level design, it is no longer suitable to normalize each per-step reward (3 az) by the mean and std of entire sequences' rewards as in the bandit setting, since the latter may not be on proper scale. Further, the on-policy nature of PPO induces an extra complexity: each step of PPO samples new text sequences, whose total length, segment lengths, and segment locations are all stochastic and can differ from the reward calibration dataset, e.g., Ppref- Appendix [H] provides an extended discussion on reward normalization in PPO-based LM training. Below, we discuss our approach to construct the reward value normalizers, followed by interpolating the segment-level reward into per-token signal to helpfully provide an even denser training guidance.
Location-aware Reward Normalizers via Regression. While the length of the sampled response y and the lengths and locations of segments { in y are all stochastic, we know that each a₂ is somewhere in y. Correspondingly, cach input (st, (4) to T is linked to a normalized location p E (0, 1] of y, and p can be simply defined as t/T, where t the index of the segment ae in y, since PPO routine has fully sampled y before calculating rewards. On each datapoint the calibration set, normalized location p E (0, 1] again, with the linked segment-level reward available. Across all data points in the calibration set, we construct new dataset Dnorm {p, Hp. op)), where p runs over all values of normalized location in the calibration set, Mp and Tp respectively denote sample mean and sid of all segment-level rewards corresponding to p in the calibration set. With Duorm, we run simple lincar regressions to estimate the relation between the log- tranfornddd normalized location log(p) and the mean/std of segment-level rewards at p. The regression formula is given by:
Mean(p) = wy log(p) + by: Std(p) uy, log(p) + bo:
(6)
where the independent variable log(p), and the regression coefficients (wyy by)) and (wr, bw) are obtained as ordinary least squares (OLS) solutions, with Hn and op as the corresponding response variables.
Note that the classical bandit normalizers of the mean and std of full sequences' rewards correspond to evaluate Mean(p) and Std(p) at p 1.0. In this regard, our mean and std functions in Eq. (0) generalize the classical scalar normalizers into location-aware functions able to output proper reward normalizers at an arbitrary (normalized) location p of the text sequence. With Mean(-) and Sud(-) and the corresponding (s, ar) is normalized by ra(Se; dt) + (ro(st, a.) - Meaa(p))/Std((p))
Within-segment Reward Interpolation. Depending on the specific tokenizer in use, we observed that semantically complete text segments may contain around twenty tokens. The corresponding action space
5

---

Published in Transactions on Machine Learning Research (09/2025)
A might still be large and the resulting segment-level design might not sufficiently address the sample inefficiency issue in the classical bandit RLHF and could again lead to inferior PPO-based RL training. To further densify the RL training signal, we evenly split the segment-level reward ra(se, az) for a segment ae to cach token yi E ae. This induces a token-level credit assignment that Vy E ag, Fo yesl, ys) = To(Kt; (e)/ where Ved is the generation context of loken y/s and |a,] is the length of segment ar- To can then directly substitute ro in Eq. (5), since T-1 (s) = Σi EE To(st::))/(0n ac)//agl).
Note that F3 is still intrinsically segment level, since all token selections Yi within segment a, receive the same feedback, i.e., the average of segment-leve reward ro(st, 00)/(ar) This is in contrast to prior works on token-level reward models (Yang et al., 2023: Zhong ct al., 2024). where cach token selection is evaluated separately and thus their token-level feedback vary for each token.
Summary. With the learned segment-level reward model rg, in PPO training of the LM policy no, we first normalize each ro(st,a.) in the sampled sequence by the corresponding normalizers Mean(p) and Std(p). Normalized segment-level rewards are then interpolated into the per-token feedback signal Tg. Finally, we plug F directly into an off-tho-shelf RLHF PPO routinc. A more theoretical viewpoint of using finor-grained reward over the classical bandit reward is provided in Appendix B
3 Related Work
Training Signals for RL-based Language Model (LM) Training. In RL-based LM fine-tuning, a classical training signal for adapting LMs to the specific downstream task is the native trajectory-level downstream test metrics (e.g., Ryang & Abekawal 2012 Ranzato et al. 2015). This approach intrinsically uses bandit formulation of LM generation that treats the entire generated sequence as single action. As discussed in Section D ignoring the sequential nature of LM ion, this bandit training signal delays the feedback to each token/phrase selection, and can thus incur optimization difficulty (Guo et al. 2022
Snell et al.j 2022). With various forms of stronger data or compute requirements, task-specific per-step training signals have been proposed to mitigate this sparse reward issue. Assuming abundant golden expert
data for supervised (pre-)training, (Shi et al. (2018) construct per-step reward via inverse RL [Russell
1998
Guo et al. (2018)
hicrarchical approach: Yang et al. (2018) learn LM discriminators; Lin et al. (2017)
and Yu et al. (2017) use the expensive and high-variance Monte Carlo rollout to estimate per-step reward from a sequence-leve adversarial reward function trained in the first place; while Le et al. (2022) use some
rule-based intermediate training signal dorived from the oracle soquencc-level evaluation, without explicitly
learning per-step reward.
Similarly, in RLHF, to move forward from the classical bandit formulation, methods have recently been proposed to ground sparse preference labels into dense per-step feedback, with applications in task-oriented dialog systems (e.g., Ramachandran et al. 2021 Feng et al. 2023) and variable-length text-sequence gener- et al. 2024 has also explored the token-level action space
ation (Yang et al.) 2023). Recently, RTO for RLHF, which we adopt as one of our baselines. Our paper seeks to reconcile dense u.s. sparse training signal in RLHF by distributing feedback to the level of semantically complete "text segment", interpolating between the densest. "token level" and the sparsest "sequence level" and ideally getting the benefit of both worlds: casicr RL training and accurate optimization signal. Fine-grained rewards were also explored in Wu et al. 2023), which demonstrated their advantages over bandit rewards in detoxification and long-form QA tasks. However, their approach relies on manual segment annotation. In contrast, shown in Section 2 our method overcomes this limitation through entropy-based automated segmentation and systematically explores the integration of segment rewards with PPO training.
In this paper, we seek to refine RL-based LM preference alignment by re-thinking the suitable action space in the RL formulation that allows both denser immodiate feedback while not joopardizing the feodback accuracy. Our segment-level design validated through numeric and example in Section 4 We discuss a broader set of related works in Appendix G
6

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Performance comparison among different action definitions on PPO-trained LM policy, with the backbone model being Phi3-mini Instruct. # {char, token} measures the average response length in the benchmark tests. Highest value of each column in bold.
Action
AlpacaEval 2.0
Arena-Hard
MT-Bench
Definition
LC(%)
WR(%) (%)
# char
WR%
# token
GPT-4o
Phi3-mini Instruct.
18.89
14.41
1473
25.1
490
7.33
Bandit (Sequence)
27.05
29.07
2164
31.3
623
7.46
Sentence
25.56
32.92
2626
32.8
671
7.51
Token
27.82
26.46
1940
27.2
533
7.58
Segment (Ours)
31.05
34.53
2257
34.0
593
7.65
Bandit as Segment
14.39
6.46
691
11.1
308
6.61
Segment as Bandit
27.15
28.20
2079
30.9
620
7.38
4 Experiments
4.1 Experimental Setups and Implementation
Datasets. For reward model training, we use the Preference-700K dataset which is a diverse collection of open-source preference datasets, such as HH-RLHF (Bai al. 2022a), Stanford Human Preferences Dataset (SHP) (Ethayarajh et al. 2022), and HelpSteer (Wang et. al. 2023). PPO-based LM policy training is conducted on Uttrafedback dataset (Cui et al. 2023), from which only use prompts to sample responses during PPO training.
Evaluation Benchmarks. The (PPO-trained) LM policy is evaluated on three popular open-ended
entructioo-following benchmarks: AlpacaEval 2.0 (Li ct al.) 2023 Arcna-Hard (Li ct al. 2024b), and MT- Bench (Zheng el al. (223),, where GPT-4o is used as the judge. For AlpacaEval 2.0, we report two metrios: Win Rate (WR), calculated as the mean of the LLM judge (GPT-4o)'s soft preference probabilities over all prompts, comparing the evaluated model to the bascline (typically GPT-4 Turbo); and Length-Controlle Win Rate (LC), which debiases for response length via logistic regression by simulating both models having equal output length, with the reported value stiil being the average of (adjusted) soft preference probabilities. Further evaluation details are deferred to Appendix D.
Implementation. We implement our method onto the open-sourced 3.8B Phi3-mini Instruct (Abdin et al. 2024), the SFT checkpoint of Phi3.1-mini Instruct, and the popular SFT checkpoint of Tlama-3-8B (Dubey et al. 2024) released by RLHFlow (Dong al. [ The backbone model is used as the starting points of both reward model training and PPO-based LM policy learning, in the latter initializing the models for value function, learning policy, and reference policy. Our implementation is built upon the open-source RLHF framework OpenRLHF (Hu et al. 2024). We maximally follow the default hyperparameters in OpenRLHF.
Due to space limit, we defer further implementation details to Appendix D
4.2 Main Experimental Comparisons
Baselines. To demonstrate our unique consideration of RLHF's action space, the main experiment, we compare our design of segment-level action space with the coarsest coarser sentence-level space, and the finest loken-level space, in terms of performance of the PPO-trained LM policy. For PPO training, a reward model is first trained under the specified action definition. The sentence-level models are implemented by splitting the text sequences using sentence splitters {".", "!", "?", "An", ";", "...", ",", ":"} and/or their foreign language equivalents. To further illustrate our segment-level reward model and denser segment-level reward assignment, we additionally compare with two hybrid approaches: (A) the reward model trained using bandit-level reward (only using the EOS token),
action space, the

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Performance comparison among different action definitions on PPO-trained LM policies. The top four rows correspond the 3.8B SFT checkpoint of Phi3.1-ming Instruct, and the bottom four rows correspond to the 8B SFT checkpoint of Llama-3 released by RLHFlow. Table format follows Table
AlpacaEval 2.0
Arena-Haue
Backbone Model
Action Definition
LC (%)
WR (%)
# char
WR (%)
# token
GPT-1o
Raw Backbone
14.93
10.19
1271
14.5
476
7.00
Bandit (Soquence)
19.39
14.78
1542
19.5
524
7.26
Phi3.1-mini-SFT
Token
22.48 48
19.25
1687
23.2
525
7.43
Segment (Ours)
26.19
23.85
1795
28.5
585
7.49
Raw Backhone
16.31
9.50
1221
10.4
469
6.82
Bandit (Sequence)
21.20
20.99
2218
18.7
513
7.11
LJama-3-8B-SFT
Token
23.84
20.87
1744
26.0
622
7.13
Segment (Ours)
25.11
28.57
2264
30.4
616
7.15
but during PPO training, rewards are computod at cach segment by fooding the corresponding logits into the reward model ("Bandit Segment "); and (B) the reward model is trained to produce segment-level rewards, but during PPO training, all segment rewards are aggregated into a single bandit reward applied at the EOS token ("Segment as Bandit"), where the bandit reward is implemented via the parametric sequence evaluator es in Eq. 4 For all baselines, we follow the standard training receipts and tune them to the extent of ensuring a fair comparison.
Results, Table|compantt our PPO-trained LM policy with alternative RLHF action spaces and two hybrid approaches using the Phiß-mini Instruct backbone. Key findings are as follows.
(1) Our segment-level approuch improves RLHF training while not suffering from length hacking. As seen in Table [ our LM policy performs better than the baselines across all three benchmarks: AlpacaEval 2.0, Arena-Hard, and MT--Bench. Notably, our model's average response length on AlpacaEval 2.0 and Arena- Hard is not significantly larger than the baseline models', in contrast to the LM policy from the sentence-level action space. Together, these results manifest the merit of our segment-level approach in truly improving the quality of the gencrated responses while not cheating the benchmark evaluations by response- length hacking (Dubois et al. 2024).
(2) Not all finer action spaces can help RLHF training over the classical bandit formulation. Apart from our denser segment-level approach, in Table 1 see that the other two finer action space specifications: design,
per-sentence and per-token, both fail to generally improve over the classical Sandit/sequencee especially on AlpacaEval 2.0 and Arena-Hard. This validates our design of segment-level reward assignment for RLHF PPO training, that offers more granular feedback than sentence-level and can be more accurate than the semantically incomplete token-level.
(9) A segment-level reward model is necessary for segment-level reward assignment, and vice versa. One may wonder if we can use the classical bandit reward model to assign segment-level reward in the PPO training. As shown by the results of "Bandit as Segment" in Table | this approach performs significantly worse than the original pure bandit, which in turn under-performs our segment-level design. These comparisons justify the necessity to train a segment-level reward model for segment-level reward assignment. Conversely, using our segment-level reward model to provide only bandit feedback in PPO training ('Segment. as Bandit") lcads to slight performance degradation over pure bandit design. Compared with our main results, we see that "Segment as Bandit" does not fully benefit from our proposal of (consistent) segment-level action space. Its weaker results again highlight the gain of denser reward assignment in PPO-based RLHF training.
(4) The benefit of segment-level design extends to SFT model and the larger 8B model. We swap the backbone model to the SFT checkpoint of 1hi3.1-mini Instruct and the larger 8B SFT checkpoint of Llama- 3, as shown in Table Table 2 It is clear the gain of our segment-level design over the prior bandit and token-level design is not scoped within the alrcady DPO'ed Phi3-mini Instruct. Rather, our advantage extends to both the SFT checkpoint of Phi3.1-mini Instruct and the larger Llama-3-8B-SFT, which veriñies the value and versatility of our method in the practical post-training pipeline.
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Performance comparison between our segment- PPO and DPO.
Model
Method
AlpacaEval 2.0 (ICC/WR)
MT--Bench
Arena-Hard
GSM8K
DPO
29.24
29.69
7.27
28.1
81.6
Phiß-mini Instruct
Segment-PPO (Ours)
31.05
34.53
7.65
34.0
86.7
DPO
24.65
22.73
7.45
26.6
82.7
1-T
Segment-PPO (Ours)
26.19
23.85
7.49
28.5
85.1
DPO Segment-PPO (Ours)
24.71 / 26.71
7.13
24.9
82.5
Laaa--3SB-SFFT
25.11
28.57
7.15
30.4
82.7
Table 4: Comparison of fixed n-gram and autropy-based segmentation on PPO-trained LM policy.
AlpacaEval 2.0
MT--Bench
Fixed n-gram
LC (%)
# char
GPT--o
2
26.00
2805
7.57
5
27.88
2224
7.51
n 10
28.55
2968
7.61
20
24.32
3369
7.58
Ours
31.05
2257
7.65
Appendix B provides generation examples from main LM policy. Table 7 in Appendix C compares the LM policies in Table D on OpenLLM Leaderboard. Both show that our method, while achieving strong RLHF training, does not suffer from the "alignment tax" (Askell et al. 2021..
(5) Segment-level PPO consistently outperforms DPO across models and benchmarks. Although the primary goal of our work to rethink the appropriate action space for PPO in RL.F, we also compare our proposed segment-level PPO with the widely adopted Direct Preference Optimization (DPO) that removes the need for explicit reward model. As shown in Table our segment-level PPO achieves stronger performance than DPO [Rafailov et al. 2023 across all metrics. In particular, it achieves up to +5.9 absolute gain Arrna-Hard (34.0 vs. 28.1) and +5.1 on GSM8K (86.7 vs. 81.6) for Phi3-mini Instruct. Similar gains are observed with the Phi3. 1ii.1-mini-SFTT and Llama-3-8B-SFT backbones, while DPO exhibits signs of alignment tax, particularly on GSMSK (Cobbe et al. 2021 benchmark.
4.3 Ablation Study
This section considers the following research questions to better understand our method. Unless otherwise specified, all ablation studies are performed on the 3.8B Phi3-mini Instruct model used in Table [
(a): What will the performance be if we segment text by the "simpler" flaed n-gram?
To compare with recent work CChai et al. 2025). swap entropy- based text segmentation for the "simpler" heuristic of fixed n-gram, where every non-overlapping tokens in the text constitute a text segment, without considering semantics. Table compares the performance of PPO-trained LM policy from our entropy-based segmentation against fixed n-gram with n E (2, 5, 10,20).
It is clear in Table that while fixed n-gram yields reasonable results, all of them under-performs our entropy-based segmentation, in terms of lower benchmark scores and higher response lengths. As will be discussed in the following part (b) and Fig. our entropy-based approach segments text sequence based semantic completeness rather than the rigid token count, which should benefit reward assignment and thus policy learning.
(b): Can our method reasonably segment lexl and assign rewords?
In Fig. 2 (Top), we compare dense reward assignments from our segment-level reward model with the token- level and fixed n-gram model on normal text. We choose n-gram with as the resulted LM policy in Table 4 does not exhibit. the response-length hacking issue, and so the reward model should have higher

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt: Explain Buddhism?
Entrop Seguent
Model
Buddhism spiritual fia founded by Siddhartha Gautama (the Buddha) the 5th century Its teachings Include Four Noble Truis, which nature of suffering and the path to its cessation, and the Eightfold Path guide to efrical living and mental development, Budchism emphasizes medkation, mincfulness. and compassion, With the ullimale goel of achieving anfighttemmen Nirvans
Fixed 5-Gram Reward Model:
Budchism a spirtuua traditio founded by Biddhartha Goutama (the Buddha) in the 5th century BCE. Its ccre teachings include the Four Noble Truths, which address the nature suffarting and the path rs dissatton, and the Eightfold Path i guice to athical living and mantal development Buddhtam ahs meditition, mindtiiness, and sompassion, with the ulimale
achieving enlighle meett Nivarsi
Token-leve Reward Mode):
Buddhiemm a spiritual traditon founded by Siddhartha Gautama (the Buddha) the Sih certury BCE. Its core teachings include the Four Noble Truths which addrees the net ura of suftoring and the path to its cessation, and the Eightfold Pam luide ethical living and mental development. Bucdhism emphasines meditation, minctuiness, and co mussssion with the uliimtee goal ol acciiiiingg enlightenment Nivanel
Prompt: rauses earthquakes?
Entrop-based Seguent
Model
release of energy in which cause the ground lo shake. Earthquakes most commonly occur along feull lines where plales meel. The movement of leclonic plates and accumulated stress are the pomary/ causes uakes The mevement of Inctanic sletes and accumulated stress a are the primary causes ae of earthquakas The movement tectoric Rittes and www.tullted 503833 are the RODUSY seuess, sorthwuaees
Earthquakes caused
movement plates.
energy generates saismic waves,
Tokon-level Reward Model:
Eartiquakes are caused by the sucden release of enargy in the Eart''s crust due the movemem of sectanic plates, This release energy gene seismic waves, which ausa the ground ahake Earting saks most commanly nocur along falll ines where plils meat The movement of heclame plates and acnuvtlated atrass are the primary Ga uses earihqusless The mevement tectous, elsues acovoulated streee are Ite sucer sawees sottaweess mewewea bodons Rites aow.oM/atee sioe Ins 8000 souses of sarthuusss.
Examples of dense reward assignment for text sequences encountered in PPO training. In the Top half, we compare our segment-level reward model with the token-level and fixed n-gram models with n = 5 on normal text. In the Bottom half, we compare our segment-level reward model with the token-level model on text with verbosity /repeettttreeetttion, where repeated sentences are rredderrieeed Darker color indicates higher reward.
Figure 2:
Table 5: Comparison of different constructions of segment-level reward normalizers, on performance of the resulted PPO-trained LM policies.
Reward Normalizer
AlpacaEval 2.0
MT-Bench
LC (%)
# char
CPT-4o
No Reward Normalization
19.64
2446
7.25
Global Statistics of All Statistics of the Last Rewards
17.34
2420
7.14
20.30
2551
7.10
Regression based (Section 2.3
31.05
2257
7.65
quality. The color blocks in Fig. (Top) demonstrate that our entropy-based approach segments text into meaningful semantic units. In cont.rast, the token-level design, a token often represents only part of a word, and thus the reward model often inconsistently highlights only parts of words (e.g., "Truths," " "meditation," " "compassion"). The fixed n-gram approach rigidly segments text without considering semantics, and thus can lead to unnatural breaks, such splitting "a guide to ethical living" into two segments: "a guide to eth" and "ical living".
In Fig. (Bottom), we compare our segment-level reward model with the token-level model on text with verbosity /repetttitte.eeiiinnnn We see that our model oassigns consistent low rewards to the repeated sentences, effectively refraining the LM from verbosity. In contrast, the token-level model still assigns high rewards to tokens in the repetitions, even in the second repeat, which undoubtedly undesirable. This comparison further shows the benefit of our design of semantically complete action space for more accurate reward assignment.
(c): How will PPO troining perform if we use different constructions of reward normalizers?
Recall that in our PPO training (Section 2.3 we use simple linear regression to fit location-aware mean and std functions that provide reward normalizers at arbitrary locations of the text sequence. To study if this design is enggeengg,,,,, we compare our main method with three simpler constructions of segment- level reward normalizers: (A) reward normalization; (B) using the scalar global mean and std over all
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: Comparison different within-segment reward interpolation strategies. Shown are the results of the resulted PPO-trained LM policies.
Interpolation
AlpacaEval 2.0
MT-Bench
Strategy
L,C (%)
# char
GPT-4o
No Interpolation
25.98
2666
7.45
Repeat Segment Reward
26.34
1795
7.42
Even Split (Section 2.3
31.05
2257
7.65
1.75
2.25
27.5
32
77
13.0
30
32
7.5
7.
28
0.
36
24
73
(a) Avg. Len (# token)
(b) AlpacaEval (LC)
(c) AlpacaEval (WR)
(d) Arena-Hard
(e) MT-Bench
Figure 3: Performance comparison among different entropy cutoffs for autropy-based text segmentation, com- paring PPO-trained LM policy's benchmark scores and average segment length ("Avg. Len") in terms of number of tokens.
segment-level rewards in the reward calibration dataset; and (C) using the scalar mean and std over the last segment-level rewards in each response of the calibration set, mimicking the normalizers in the classical bandit approach. Table compares the resulted LM policies.
In Table 5 we clearly see that normalizing (dense) reward by improper reward statistics is akin to no reward normalization, as all three baselines have significantly lower benchmark scores than our regression-baser approach and undesirable longer response lengths. As discussed in details in Appendix H the linguistic structure of the response leads to certain correlation between the mean and std of segment-level reward values and the normalized location of segment in the response, e.g., in the early or middle or later part. This necessitates design of location-aware reward normalizers that are able to capture the reward statistics at each arbitrary location of the sampled text sequence, since constant normalization statistics insufficient to properly normalize the rewards of text segments at different parts of the text sequence, as verified in Table 5 Future work may extend our linear regression-based normalizer functions in Section 2.3 with non-lincarity and/or more features.
be
(d): What will happen if we use different strategies for within-segment reward interpolation!
Recall from Section 2.3 that, to further densify the learning signal in RLHF for enhancing training, we interpolate the segment-level rewards by evenly splitting the reward of segment each of its constituting token. We now compare this even-split interpolation strategy with two other intuitive alternatives: (A) no
interpolation on the segment-level rewards, use 0 for technical padding in PPO ("No Interpolation" "); (B)
repeat the segment-level reward a segment to each token in it ("Repeat Segment Reward"). Table 0 shows the performance of the resulted PPO-trained LM policies.
In conjunction with our main result Table D in Table (relatively) effective RLHF training on Phi3. 1-mini Instruct, in reference to the results of the classical bandit approach in Table Nevertheless, we see that the generation length from "No Interpolation" is significantly longer, while "Repcat Segment Reward" is too short. Probing into those long text sequences encountered in PPO training, we found that they typically contain some very negative segment-level rewards that refrains the behavior long generation from being learned by the policy LM. Such very negative reward signals may be diluted by the technical zero-padding in "No Interpolation", Icading to overly long text generation; whereas they are overly amplified in "Repeat Segment Reward", resulting in too-strong punishment for long texts and hence too-short generations. By contrast, the even-split interpolation strategy in main method
that these two alternatives still provide
11

---

Published in Transactions on Machine Learning Research (09/2025)
provides densified learning signal of proper scale, which attribute to the implicit (segment-) length normalization inherited from the operation of dividing by segment length in even split. Future work may design a proper non-even split of segment-level reward over each token in the text segment.
(e): With a different entropy cutoff Cent for text segmentation, how will our method perform?
As discussed in Section 4. for main results, we use entropy cutoff Cent 1.75 for entropy-based text
segmentation. To investigate the impact of Cent, in Fig. 3 we vary the value of Cent E {1.5, 1.75, 2.0, 2.25), and compare the performance of the resulted PPO-trained LM policies well as the average segment length of the PPO-trained LM policy.
As seen in Fig. B similar to the discussion of token-level approach in Section | a smaller Cent = 1.5, which chops text sequence into finer pieces with smaller average segment length, may result in semantically less complete segments, leading to less accurate reward modeling and the subsequent weaker LM policy. A reasonably larger entropy cutoff, such as Cent [1.75, 2.25] that corresponds to an average segment length of 10 to 22 in Fig. 3a (or about to 7 words), leads to much better PPO-trained LMs. This coincides with the advantage of our segment-level design over the prior token-level design in Table Table 2 and verifies our goal of a more semantically complete action space.
(f): How does our segment-based PPO policy compare on the AlpacaEval win rate-KL frontier?
Fig. shows the AlpacaEval LC win rate versus KL to the reference policy (Phi3.-mini--FTT), For all methods, the win rate increases as KL grows, but our segment-based based PPO achieves consistently higher win rates than the bandit token-level baselines across the entire KT. range. This indicates that our approach makes the most effective of the optimization budget (KL), and sets new fronticr for real alignment performance.
KL Win Rate
26
Bandit Win
Token Win Rate
Segment Win Rate
24
Rate (%)
20
18
10
20
Divergence
Figure 4: The AlpacaEval LC win rate-KL vs KL to the reference SFT policy.
(g): How does segment-level reward modeling affect code generation performance?
Figure 5 compares the performance of Bandit, Token, and Segment action spaces on three code generation benchmarks (MBPP Base (Austin et al. 2021), MBPP Plus, LiveCode (Jain et al. 2024)), ), for both Phi3. I-TT.--mii and Alama-3-8B-Si
backbones. All models are trained on the Ultrafeedback dataset. Across all benchmarks and model backbones, our segment- PPO consistently achieves the highest scores, surpassing both token-level and bandit baselines. This demonstrates that segment- action space provides better and more robust performance even structured data types such code.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Uaama
70
Action space
Actio space Bandi:
80-
Segiment.
Segmmeet
40-
Score
20-
20-
10
HBPP Rase
M80P Plus
UveCouleench
MBPP Rase
M809 Plus
UveCadleench
Figure 5: Performance comparison on code generation benchmarks (MBPP Base, MBPP Plus, Live- CodeBench) for different action spaces (Bandit, Token, Segment) and model backbones (Phi, Llama).
5 Conclusion
In this paper, we propose to train and utilize segment-level reward model for improved RLHF in LMs. motivated by both a denser reward signal in RL-based LM training and semantic completeness of cach action for accurate reward assignment. Our method and insight are validated through extensive experiments, ablation studies, and backbone models of different sizes, offering a promising research direction for further exploration of fine-grained action spaces in RLHF
13

---

Published in Transactions on Machine Learning Research (09/2025)
Limitations
While our proposed segment-level reward model demonstrates promising improvements in RLHF, certain aspects warrant further investigation. As an initial exploration into refining the action space in RLHF, our experiments have so far been limited to PPO training on free-form dialog-style datasets and instruction- following benchmark evaluations. Future work will focus on scaling our approach to even larger LMs, extending its applicability to diverse tasks such as mathematical reasoning and code generation, and exploring its integration with alternative RL algorithms, such GRPO (Shao el. al. 2024), and REINFORCE++ (Hu
2025
Impact Statement
Segment-PPO advances RLHF by introducing segment-level reward modeling, improving language model alignment while addressing sparse reward issues. This refinement enhances response quality, benefiting applications like conversational AT and automated content generation. However, segment-level optimization requires careful calibration to mitigate potential biases and unintended generation patterns. Additionally, as RLHF inffuences AI makiin. responsible deployment is crucial to prevent misuse in misinformation propagation or biased outputs. By refining reward learning at a more semantically meaningful level, our work underscores the importance of balancing AI advancements with ethical considerations.
14

---

Published in Transactions on Machine Learning Research (09/2025)
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14211 2024.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference- based policy learning, In Machine Learn- ing and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, Seplember 5-9, 2011. Proceedings, Part 11, pp. 12 27. Springer, 2011.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advonces in neural information processing systems, 30, 2017.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXi::2112.0086 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arX::22108.077 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ar preprinl ar arXiv:2204. 05866, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai foedback. arXiv preprint (xXi::2212008033 2022b.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. Hugging Face, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on ma- chine learning, pp. 192. PMLR, 2019.
Daniel Brown, Wonjoon Goo, and Scott Nickum. Better-hhan-demonstaator imitation learning via automattiaall--aanee demonstrations. In Conference on robot learning, pp. 330 359. PMLR, 2020.
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Drle: Reinforcement learning with dense rewards from Ilm critic. arXiv preprint arXtm:2401.07388 2024.
Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Tan Yang, Spencer Frazier, and Mark Riedl. Robust preference learning for storytelling via contrastive reinforcement learning. arXiv preprint ar arx::22200007700 2022.
Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, and Hua Wu. Ma-rlhf: Reinforcement learning from human feedback with macro actions. ICLR, 2025.
Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human foodback. arXiv preprint arXim:4402.0078. 2024.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. Xix preprint or Xin: 1604.66174 2016.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. 2024.
arXiv preprint arXie:2401.0000
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vineet, Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Uttrfeedback: Boosting language modcls with high-quality feedback. arXiv preprint arXiv:2310. 013777 2023.
T Dao, DY Fu, S Ermon, A Rudra, and C Flashattention Ré. Fast and memory-ofiiciet exact attention with io-awareness. URL hllps:/ /urriv. ory/aby 0tti:/2255..444 14195, 2022.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhí workflow: From reward modeling to ouline rlhf. aix preprinl arXiv:2405.0788 07867, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint wxXi:2447.21783 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. contrllcon alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 244444444 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 6008. PMLR, 2022.
Yihao Feng, Shentao Yang, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. Fantastic rewards and how to tame them: A case study reward learning for task-oriented dialogue systems. In The Eleventh International Conference on Learning Representalions, 2023.
Chelsea Finn, Paul Francis Christiano, P. Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. ArXiv, abs/1611.03853 2016.
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-based reinforce- ment learning: formal framework and a policy iteration algorithm. Machine learning. 89:123-156, 2012.
Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. Beyond imitation: Leveraging fine-grained quality signals for alignment. arXiv preprint Stw:2311104477 2023.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Efficient (soft) q-learning for text generation with limited good data. Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6969 6991, 2022.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adver- sarial training with leaked information. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trIX: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8578 8578-8595, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ http//22..mmmmmmmin..... URL. httpstss::// htttps://aclantoolll/ wtwphhoology.og/222233333 emnnp-main..n.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Jocy Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based rl without reward function. arXiv preprint ar Xrx:22335. 15363, 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puraník, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint (rXie:2105.0 099388, 2021.
Jian Hu. + +: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501. 033222200.0326 2025.
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlif: An easy-to-use use, scalable and high-performance rlhf framework. arXiv preprint 2rXiv:2405.1114 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXic preprint Axxi:2443.00977, 2024.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun Craig Ferguson, Agala Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint ar Xix:2000 058488 2020.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lln-blender: Ensembling large language models with pairwise ranking and generalive fusion. aixiv preprinl urXio:2306. Si::2636.00561,, 2023.
Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. In ICML, pp. PMIR, 2016.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for RL. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///ppparrreeeeooooooortrrrp wttt//ooruuiidioo111s...
Diederík Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International
Conference Learning Representations, 2014.
Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging rl theory and practice with the effective horizon. arXiv preprint Xiv:20004 00985,,,00 2023.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mas- tering code generation through pretrained models and deep reinforcement learning. Advances in Newral Information Processing Systems, 35:21314-211 -13228, 2022.
Bolian Li, Yifan Wang, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding-tine time alignment. Xixiv preprint arXiv:2406. 16306, 2024a.
Tianle Li, Wii-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph Gonzalez, and lon Stoica. From live data to high-quality benchmarks: The arena-hard pipcline, 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-followingg models, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 205. 2023.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming- Ming-Ting Sun. Adversarial ranking for language generation. Advances in neural information processing systems, 30, 2017.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Peter Marbach and John N Tsitsiklis. Approximate gradient methods in policy-space optimization of markov reward processes. Discrele Fvent Dynamic Systems, 13:111-148, 2003.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillighhe Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2208.1114 2022.
OpenAL. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, ct al. Training language models to follow instructions with human feedback. arXiv preprinl Xix: 02155, 2022.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society: Series C (Applied
Slatisties), 24(2):193 202, 1975.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea. Finn. Direct preference optimization: Your language model is secretly a reward model. In Thärly-seventh Con- ference on Neural Information Processing Systems, 2023. URL https://openreviert wtwprrrrriieeeeteoutnuuiio.i net/forum?id-
HPuSIXJaa9
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to Q*: Your language model is secretly a Q-function. arXiv preprint 24f.::2444..266 2024.
Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong. Causal-aware safe policy improvement for task-oriented dialogue. arXiv preprint 2Xxx:2103.06370, 2021.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXio:1511.067 2015.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0550 2024.
Stuart Russell, Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using reinforcement lcarning. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Process- ing and Compulational Natural Languoge Learning, pp. 256 265, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arxiv: 707.063777 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1723, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/PP 1162. URL https://aclaathhologg org/P16-1165 1162
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepscckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprinl Xrx::22402 00230000 2024.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. arXiv preprint ar arxii:18044 11 11258, 2018
Charlie Snell, Πya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. Xi preprint 220. 2022.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 33:3008 3021, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. arXiv preprint arxiv: 10719, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finc-tuned chat models. arxiv preprint or orXiv:2307009880 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, GeolTrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211. 14275, 2022.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expert arXiv preprint 1XXi::2406.18855 2024a.
Huimin Wang, Baolin Peng, and Kam-Fai Wong. Learning efficient dialogue policy from demonstrations through shaping. In Proceedingg of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 6355-6365, 2020.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. my answer is c': First-token probabilities do not match text answers in Instruction-tunet language models. arXiv preprint ar arXiv:2402. 1 4499, 2024b.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint vrXiv:2311.09522 2023.
Zhilin Wang, Yi Dong, Olivier Delallcau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top- performing reward models. arXiv preprint arXiv:2406.088 0867773,. 2024c.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhir, Prithviraj Ammanabrolu, Noah Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. NewrIPS, 36:59008 59033, 2023.
Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, and Jaeyoung Do. Aligning large language models via fine-grained supervision. arxi preprint arXiv:2406. 02756, 2024.
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference- grounded token-level guidance for language model fine-tuning. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Birg-Kirkp.atrck. Unsupervised text style transfer using language models as discriminators. Advances in Neural Information Processing Systems, 31, 2018
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung- Woon On, Mark A Hasogawa-Johnson, Sungwoong Kim, and Chang D Yoo. Tler: Token-level continuous reward for fine-grained reinforcement learning from human feedback. arXiv preprinl arXiv:2407.1657 2024.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. Xiv preprint arXiv:3305.11 10425, 2023.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mi-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.1880 18922, 2024.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse rein- forcement learning. In Proc. AAAI, pp. 4433-1438, 2008
Daniel M Ziegler, Nisan Stiemnon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- liano, and GeofTrey Irving. Fine-tuning language models from human preferences. ar preprinl 1909 08593, 2019.
20

---

Published in Transactions on Machine Learning Research (09/2025)
A
Algorithm Box
Algorithm 1 summarizes our method in Section 2 on training the segment-level reward model and utilizing it in PPO-based RLHF' LM training. Note that all operations in Algorithm can be efficiently conducted in batch mode, parallel for multiple sample points at oncc.
Algorithm 1 Training and Utilizing Our Segment-level Reward.
Input: Binary preference dataset Pret = {(x, y", y')} for reward model training, prompt set Dpol = {x) for policy learning, supervised fine-tuned model TSFT; reward model training steps Myews LM policy training steps Mpol, entropy cutoff Cent, KL coefficient for RLHF PPO training.
Initialization: Initialize the segment-leved reward model ra and LM policy Ta from TSFT, fix the aggre- gation function f(-) as the Average in Eq. initialize other components in the off-the-shelf RLHF PPO routine as specified.
// Training the segment-lewe reward model Use TSFT and Cent to split the responses {y'", , in Dpret {1x, y²" ./)) into segments. for iter (.,...,,A.es...
do Sample minibatch B {<r, yi" y.))s D Drref-
With f()) and T, calculate e (2 v(t) and ca( by Eq. 44 for (xi, o vt) E B.
Optimize reward model by Eq. 3
end for
// Uuilizing the segment-level reward model PPO-based LMM policy learning Estimate the reward normalizer functions Mean(p) and Std(p) as described in Section 2.3 for iter (1...., Mp } do
Sample a minibatch B {2i}; ~ Ppol-
Sample a response Yi To (-)))) for each 2; E B
Use FSFT and Cent to segment each yı; record the completion portion of each segment. Use ro to assign segment-level reward to each segment a, in each yi Normalize each segment reward (8,, a.) as (o(8),0e) + (ro(s), ax) Mean(p))/St((p)) Interpolate rø(st, az) each token Yi, as a, E y. E a,, (olr, Vesl yi) re (st, a With KL coefficient, B, optimize policy LM ma against ip by the PPO routine. end for
B Justification of Using Finer-grained Reward over the Bandit Reward
Since finer-grained reward applicd to the standard PPO algorithm, the standard gradient calculation of PPO should hold our method. The benefit of fine-grained reward manifests in the estimation of value and advantage functions. For discussion discussion/eqquation /equation simplicity, below we take as an example the action-value of the first segment/ /action Qzz, oo) and ignore the KL-regularization term, where x denotes the prompt. This discussion holds for action values at other state-actions, the value V function, the advantage function, and adding back KTl-regularization
Denote y = [ao, di....,00 aT as the entire response, consisting of T segments.
Q(z, on) Under Bandit Reward. The standard bandit reward on PPO is implemented as assigning a reward R(x, y) to the last sogment/action and padding all intermediate rewards as O. With the standard selection of 7 = in LM's RLHE, we have
Bandie (x,)ao) E er..er (x [R(x,))
(7)
21

---

Published in Transactions on Machine Learning Research (09/2025)
(xx, oo) Under Segment-Level Reward. Our denser segment-level reward model assigns reward
r(s,, ax) to each segment dz- We have
Qseg(2; (r, on) = R [o (ea (xa (lle)) [r(s1, 01) + r(82, 02) + (s8,ar)]
(8)
Interchanging expectation and summation, we have
Qseg ao) (ajor((Isss) [1]]
xx)x(() 1=2))))))))))))))))))))))))))))) [r 02)] +...
(9)
Comparison between QBandit (I, ao) and Qsep (2, ao). We see that Q ao) is defined solely on the product space of all segments:
a1 (a.....arr] x(- $1) x T(- 82) x ... x x(. ST).
(10)
By contrast, our Qseg(z; ao) decomposed into T terms. The first term is the expectation over a single segment, the second term over the product space of the first two segments, and for estimating each term in Qseg og) can be order of magnitude smaller compared to Qanddi (I, ao); @seg( un) can be estimated more accurately with finile samples, which is our motivation for designing a denser reward. A formal theoretical treatment comparing dense and sparse rewards, reaching a similar conclusion, is presented in [Laidlaw et al. 2023
Since the sample space
Our variance reduction technique in spirit close the slep-wise Importance Sampling (IS) estimator off- off-policy evaluation (v.s. the trajectory-wise IS estimator). For related discussion, see Section 3.2.2 of (Jiang
& Li 2016
C
Additional Results
Table presents the evaluation results of different. LM policies from Table on the HuggingFace OpenLLM Leaderboard (Beeching et al. 2023
Table 7: Evaluation results of downstream tasks on the HuggingFace OpenLLM Leaderboard (Beeching comparing LM policies in Table [
et al. 2023
Action Definition
ARC
TruthfulQA
Winograd
HellaSwag
MMLU
GSM8K
Average
Phi-Instruct
64.76
54.44
74.51
79.03
70.41
81.6
70.79
Bandit (Sequence)
64.76
55.11
74.35
79.32
70.42
77.8
70.29
Sentence
63.40
53.99
72.93
79.34
70.42
84.1
70.70
Token
62.71
53.94
71.43
79.46
70.55
87.3
70.90
Segment (Ours)
62.71
54.74
72.06
79.23
70.42
86.7
70.98
Bandit as Segment
64.16
54.62
74.66
78.95
70.55
81.0
70.66
Segment as Bandit
64.33
54.81
74.74
79.23
70.39
78.6
70.35
D More Implementation Details
Implementation Details. We tabulate detailed parameter settings in Table X and Table g Most of them are the same as the default setting in OpenRLHF]] Both the reward model and PPO training employ the Adam optimizer (Kingma & Bal 2014), with 81 = 0.9 and 82 = 0.95. To save GPU memory, we use gradient checkpointing (Chen et al. 2016)) and flash attention (Dao et al. 2022).
For reward model training, we set the maximum prompt sequence length as 1792 tokens, with the total sequence length (including both prompt and response) capped at 2048 tokens. During data preprocessing, we apply left truncation to the prompt and right truncation to the response. If the EOS token in the response is truncated, manually change the last token in the truncated response to the EOS token. The
22

---

Published in Transactions on Machine Learning Research (09/2025)
global mini batch size for reward model training is set to 128, with each GPU processing micro batch size of 8. To facilitate distributed training, we utilize DeepSpeed ZeRO-3 For our segment-level reward model, the entropy threshold is set to Cent = 1.75 for training with the Phi-series models and Cent = 2 for the Llama-3-8B model. The baseline bandit reward model is technically implemented as setting the entropy threshold Cent = 1000, restricting reward computation to the EOS token only, while the baseline token-level reward model is implemented as sctting the entropy threshold Cent = 0, ensuring that reward is computed for each token in the text sequence.
For PPO training, the replay buffer size Frolloutbbatch_ batch_atttiia is set 1024, while the batch size per GPU for generation (micro_ (micro_rollout rollout, batch_ batch_size) is configured as 16 for Phi-mini and 4 for Llama-3-8B. The maximum prompt sequence length is set as 1024 tokens, and the maximum generated sequence length is also set to 1024 tokens. In PPO's on-policy sampling, for each prompt in the mini-batch. a single response is sampled via top-p sampling with p = 1.0 and sampling temperature 1.0. We use DeepSpeed ZeRO-2 for distributed training. The actor learning rate is set to the default value of 5 x 10-7, and the critic lcarning rale is also the default value of 9 10-6 The clipping coelficient for value loss (value clip) is set to 0.25 for PPO training based on segment- and token-level reward model, and as default to 0.2 for bandit-reward-baat PPO training. The clipping cocfficient for policy loss (eps clip) is set to 0.2. The KL coefficient is kept to the default value of B = 0.01.
Below, we provide further clarification of the two hybrid baselines in Table ! which are important for interpreting our experimental results and understanding the interaction between reward model training and PPO training strategies: For the "Bandit as Segment" baseline, during reward model training, only the hidden state of the EOS token is fed into the LM head to compute the hiddrd-hidden states at other positions are not used for supervision. However, during PPO training, instead of using the reward only at the EOS token, we feed the hidden states of all positions (not just the EOS token) into the reward model's LM head to compute per-segment rewards. This baseline illustrates that if the reward model is trained with only bandit-level supervision, but during PPO training, we attempt to provide segment-lovel rewards, the resulting mismatch leads to inferior performance compared to our approach. This highlights the importance of jointly considering both reward model training and PPO training strategies.
In the "Segment as Bandit" bascline, we the same segment-level reward model as in our main PPO training. The difference is that, during PPO training, we adopt the conventional bandit action space, assigning reward only at the EOS token (i.e., only the final segment receives a non-zero reward, while all other positions receive zero). In contrast, our proposed method assigns distinct rewards to all segments. The inferior results of "Segment as Bandit" demonstrate the benefit of using segment-level rewards throughout PPO training, rather than reverting to bandit-level reward signal.
Table 8: Hyperparameters used in reward model training.
Hyperparameter
Value
Batch Size
128
Training Epochs
Max prompt length
1728
Max total length
2048
DeepSpeed ZeRO stage
3
Optimizer
Adam
Learning Rate
le-6
Gradient clipping norm
1.0
Entropy threshold Cent
1.75
E
Examples of Generated Responses
Below, we provide examples of generated responses from our LM policy in the main results (Table 1 on the benchmarks AlpacaEval 2.0 (first two examples) and Arena-Hard (last two examples).
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table Hyperparameters used in PPO-based LM training.
Hyperparameter
Phi-mini
Llama-3-8B
Batch Size
128
128
Rollout batch size
1024
1024
Micro rollout batch size
16
4
Training Epochs
2
1
Max prompl length
1024
1024
Max generation length
1024
1024
DeepSpeed ZeRO stage
2
2
Actor learning rate
50-7
5e-7
Critic learning rate
9e-6
96-6
Gradient clipping norm
1.0
1.0
Entropy threshold Cent
1.75
2.0
Value clipping
0.25
0.25
KL cocfficient/ /
0.01
0.01
Prompt: When are hops added the brewing process?
Hops, the flowers of the hop plant Humulus lupulus, crucial ingredient in brewing beer, primarily used for their bittering, flavoring, and aromatic properties. They are typically added to the brewing process at various stages, depending on the desired outcome in terms of bitterness, flavor, and aroma. The timing of hop addition a critical aspect of brewing that significantly influences the final beer's character.
Response:
Boiling Stage (Early Boil): This the most common stage for adding hops. Adding hops early in the boil (usually within the first 15 minutes) primarily contributes to the beer's bitterness. The longer the hops are boiled, the more bitterness is extracted due to the breakdown of alpha acids, which are responsible for this bitterness. This process is known as isomerization. Farly boil hops can also contribute some flavor, but the impact on aroma is minimal because the volatile compounds responsible for aroma are largely evaporated during this long boil.
Middle of the Boil: Adding hops the middle of the boil (around 30 minutes) can strike a balance between flavor and aroma. The bitterness level is moderate, and the hop flavors and some aromatic compounds are preserved better than in the early boil, thanks to the shorter exposure time.
3. Late Boil (Last 10-15 minutes): Hops added late in the boil contribute significantly to the beer's aroma and flavor, with minimal to no contribution to bitterness. This is because the shorter boiling time allows the volatile aromatic compounds to remain intact, while the alpha acids responsible for bitterness are less extracted. This technique is often used in styles that emphasize hop aroma and flavor, such as India Pale Ales (IPAs) and New England IPAs (NEIPAs).
4. Fermentation Stage: Some brewers also add hops during the fermentation process, technique known as dry hopping. This is done after the primary fermentation has completed, and the beer is transferred to secondary fermenter directly to the bottle/keg. Dry hopping euhances the beer's aroma without significantly affecting its bitterness. The extended contact time allows the hop oils to infuse the beer, imparting fresh, fruity, and floral aroma.
Fost-teermntttion (Aging): some cases, especially with barrel-aged beers, hops may be added during the aging process impart additional flavor and aroma characteristies. This can be a experimental approach, the interaction between the hops and the aging environment (e.g., oak barrels, different beer styles) can lead to unique and complex profiles.
The choice of when to add hops is critical decision that brewers make based on the style of beer they are aiming to produce and the specific flavor, aroma, and bitterness profiles they desire. Experimentation
24

---

Published in Transactions on Machine Learning Research (09/2025)
and experience play significant roles in fine-tuning this aspect of the brewing process to achieve the perfect balance for each batch.
Prompt: Introduce the "financial markets and institutions" by Frederic S. Mishkin.
Response: In his comprehensive work, The Economics of Money, Banking, and Financial Markets, Frederic S. Mishkin delves into the intricate world of financial markets and institutions, providing a thorough examination of their roles, functions, and significance in the global economy. The book, published in 2000, is well-regarded resource for students, professionals, and policymakers alike, as it explores the complexities of financial systems and their impact on economic growth, stability, and development.
Financial markets and institutions are integral components of the broader economic system, serving as the backhone for the efficient allocation of resources, risk management, and the facilitation of transac- tions between various participants. Mishkin's book offers a deep dive into the various types of financial markets, such capital markets, money markets, and foreign exchange markets, as well as the in- stitutions that operate within these environments, including banks, investment firms, and regulatory bodies.
The book begins by establishing a solid foundation in the basic concepts and principles of money, banking, and financial markets, ensuring that readers have a clear understanding of the fundamental ideas that underpin the entire field. Mishkin then progresses to explore the role of financial intermodi- aries, such as banks and other institutions, in connecting savers and borrowers, and how these entities contribute to the overall efficiency and stability of the financial system.
One of the key themes in Mishkin's work is the importance of financial markets and institutions in promoting economic growth and development. He examines the ways in which these entities facilitate the flow of funds from savers to productive investments, thereby supporting innovation, entrepreneur- ship, and job creation. Furthermore, Mishkin discusses the role financial markets providing risk management tools, such as derivatives and insurance products, which help individuals and businesses protect themselves against potential losses.
Another crucial aspect of Mishkin's exploration is the examination of the regulatory framework that governs financial markets and institutions. The author emphasizes the need for effective oversight and regulation to ensure the stability of the financial system, protect consumers, and prevent financial crises. Throughout the book, Mishkin provides real-world examples and case studics to illustrate the conso- quences of both well-functioning and poorly regulated financial markets, highlighting the importance maintaining balanced and robust regulatory environment.
In addition to discussing the thcoretical aspects of financial markets and institutions, Mishkin also addresses practical considerations, such the impact of technology on the financial industry, the role of central banks in monetary policy, and the challenges posed by globalization and financial integration. By offering comprehensive and well-rounded perspective on the subject, Mishkin's is work serves as an invaluable resource for anyone secking to understand the complexities and nuances financial markets and institutions in the modern economic landscape.
25

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt: Provide me with breakfast recipe that is quick to make and is high in protein (at least 30 grams) and has variety of ingredients.
Certainly! Here's a quick and delicious high-protein breakfast recipe called "Spinach and Feta Egg Muffins.' This recipe not only packed with protein but also incorporates a variety of in- gredients, including spinach, feta cheese, and eggs. It's perfect for healthy, on-the-go breakfast satisfying meal prep option.
Response:
Ingredients:
large eggs
cup fresh spinach, chopped
1/2 cup feta cheese, crumbled
1/4 cup red onion, finely chopped
1/4 cup cherry tomatoes, halved
1/4 cup green bell pepper, finely chopped
1/4 cup mushrooms, finely chopped
1/4 cup fresh parsley, chopped
Salt and pepper, to taste
1/4 teaspoon garlic powder (optional)
/4 leaspoon paprika (optional)
1/4 teaspoon dried oregano (optional)
Instructions:
1. Preheat your oven to 375'F (1909C) and lightly grease muffin tin with cooking spray or a small amount of oil.
2. In large mixing bowl, whisk the eggs until well combined.
3. Add the chopped spinach, feta cheese, red onion, cherry tomatoes, green bell pepper, mush- rooms, and parsley to the egg mixture. Stir until all the ingredients are evenly distributed.
4. Season the mixture with salt, pepper, garlic powder, paprika, and dried oregano (if using) to taste.
5. Pour the egg mixture into the prepared muffin tin, filling each cup about 3/4 full.
6. Place the muffin tin in the preheated oven and bake for 20-25 minutes, or until the egg muffins are set and the tops are lightly golden.
7. Remove the muffin tin from the oven and let the egg muffins cool for few minutes before transferring them to a wire rack to cool completely.
8. Once cooled, you can store the egg muffins in an airtight container in the refrigerator for up to
3-4 days or freeze them for up to 3 months.
26

---

Published in Transactions on Machine Learning Research (09/2025)
9. To reheat, simply place the egg muffins in microwavo-safe dish and heat for 15-20 seconds, until warmed through.
Enjoy your high-protein, delicious, and nutritious Spinach and Feta Egg Muffins!
27

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt:
Write a python function that solves a quadratic equation.
Response:
To solve quadratic equation of the form ax² + c - 0, we can use the quadratic
formula:
b V
4ac
I
2a
This formula gives us the roots of the quadratic equation, where a, and are coefficients of the equation, and represents the variable. The term under the square root, 8 4ac, is known the discriminant. The discriminant determines the nature of the roots:
If the discriminant is positive, there are two distinct real roots. If the discriminant is zero, there is exactly one real root (also known as a repeated or double root). the discriminant is negative, there are no real roots, but two complex roots.
Here's Python function that calculates the roots of quadratic equation, handling all three cases: import cmath
def solve_ quadraticaa b, c):
...
Solves a quadratic equation of the form ar 2 + bx + = 0.
Parameters:
Coefficient of x²2
b: Coesficient of
- Constant term
Returns:
A tuple containing the roots of the equation. If the roots are compler. they are returned compler numbers.
# Calculate the discriminant discriminant = emath. sqrt (b++2- - 4*a*c)
# Calculate the two roots using the quadratic formula
root (-b + discriminant) / (2*a)
root2 (-b discriminant) / (2+a)
return (root1, root2)
# Ezample usage
a, b, c = 1, 5, 6 # Coefficients for the equation x²2 + 5x + 6 = 0
roots = solve quadratic (a, b, c)
print (f" The roots, the equation
equuttttooo are tr [0]} and froots [1]} ')
# For compler discriminant
a, b, = 1, 2, 5 # Coefficients for the equalion x 2x + 5 0 roots = solve_ quadratic b, c) print
are ineeurroote [0]} uand anndufroots rroots [1]}')
This function uses the 'emath module to handle complex square roots, ensuring that it can return complex roots when necessary. The function calculates and returns both roots, which can be real or complex numbers, depending the discriminant.
28

---

First, 60,000 data points are randomly sampled from the Preference-700k dataset, which includes pairs of prompls, chosen responses, and rejected responses. Each response is processed by a segment reward model, where the segments within the response are indexed by their respective normalized location. Specifically, the normalized location p E (0, 1] is computed for each segment a₂ as p T: within the response and T represents the total number of segments in the response. The model then provides Published in Transactions on Machine Learning Research (09/2025)
Chat Template. Below is the chat template used for PPO training, reward model training, and evaluation sampling with Phi-3. The original Phi-3-Mini chat template includes an additional FOS token, which we removed in experiments due to incompatibility, leaving only the < 
 token.


(the user query) </end]
<lassistant|>
(Assistant Response) <
</end|>
For Llama-3, the chat. template is follows:
<|begin_ of text (start, header su/sr <lend header id/>
(the user query) quert]<(eot_idde </start header id asssstantt < </end header id
(Assistant Response|</eot id/>
F
Computation of Location-Aware Reward Normalizers via Regression
t. where the index of the segment
the reward for each segment, producing set of data points that consist of the segment's 's normalized location and its corresponding reward.
To estimate the relationship between the normalized location and the reward statistics, we employ linear regression approach using the HuberRegressor from the akle library, which is robust to outliers. We perform the regression on the log-transformed normalized locations, log(p), to model the dependence of the mean reward Pp and the standard deviation op of rewards at each normalizod location. The regression formulas are given by:
Mp) = ,,, log(p) + by. Std(p) ww log(p) + by:
(11)
Here, di and by are the regression coefficients for the mean reward, and wa and b, are those for the standard deviation.
Once the regression coefficients obtained, we them to compute the normalized rewards for each segment-level reward during the PPO training. The normalized reward ra(s,, az) is computed according to the location-aware normalizers:
ac) ro(st, a.) - Mcan(p) Std(p)
(12)
G More Related Work
Reward Models in RLHF. In the classical RLHF paradigm, policy LM is optimized against bandit reward model trained firstly by binary classification loss on the preference datasct, with KL penalty to a specified prior distribution to avoid reward over-optimization (Ziegler et al., 2019 Stiennon et al.) 20200 Jaques et al. 2020 Bai et al. 2022a Ouyang et al. 2022). Under the same bandit formulation, recent works have enhanced the bandit reward model by dircctly modcling the probability of one response being preferred over the other (e.g., Jiang al. 2023 Zhao et al. 2023 Dong et al. 2024 or factorizing human preference
into multiple facets via multi-objective modeling Touvron et al. 2023 Wang et al. 2024c a). Despite its
29

---

Published in Transactions on Machine Learning Research (09/2025)
popularity, from the angle of RL-based optimization of human preference captured by the reward model, such a bandit reward may lead to inferior training, due to the sparse reward issue intrinsic to the bandit formulation of LM generation and credit assignment (e.g., Takanobu et al. 2019; Guo et al. 2022).
Viewing the weakness of bandit RLHF, efforts have been making to densify the reward signal for RLHF LM training. Yang et al.] (2023), ,, Xu et al. (2024) and Chan et al. (2024) train token-level reward models by the binary preference classification loss. Zhong et al. (2024) and Rafailov et al. (2024) use an LM trained by
DPO (Rafailov et al. 2023) firstly for token-level reward assignment, which is later used in PPO training
rearch-based algorithms. Guo et al. (2023), Cao et al. (2024). , and Yoon et al. (0224) assign continuous
fixed fine-grained rewards (e.g.. +1) by accessing an external powerful large LM or the oracle environmental
reward; while Chen et al. 2024) require the extra task and datasets of erroneous solution rewriting. Apart
from potential extra requirements, as discussed in Section [ the semantic incompleteness of token in text may challenge the efficacy of per-token credit assignment, especially with the prevailing implementation of reward model as a decoder-only transformer that cannot look ahead into later tokens.
Close to our segment-level reward, process reward models (PRMs, e.g., Uesato et al. 2022 Lightman et al, 2023) in reasoning-alike tasks also assign reward to each step, defined as a short sequence of tokens. However, PRMs typically require per-step human annotations - impractical for general text generation tasks like summarization or dialogue where only full text. sequences can be properly evaluated. In contrast, our method (Section 2 is developed for the most basic yet general RLHF setting, where (human) preference is only provided in a dataset of binary sequence-level preference with diverse prompt-respouse forms.
classically takes two-stage approach where re- ward model is first trained on dataset of binary multiple ranking via maximizing the choice model likelihood (Bradley & Terry 1952 Plackett 1975). before optimizing the RL/control policy against the learned reward model by RL algorithms (Akrour et al. 2011 Fürnkranz et al. 2012). Earlier application in policy for robotics conttol tasks (e.g.. Chris- tiano et al. 2017 Hejna & Sadigh 2023). Implanting its success in robotics, in natural language generation, this two-stage learning-Trom- prrfeeeeeeeeeeeecceeeeeeeee paradigm has been scaled up and popularized the post-training stage to align LMs with specific human values, with applications ranging from text summarization (Ziegler ct al.
Learning- fraiinng-fooom--eeeeeeee
Learning-from-ppreeer
deep learning mainly focuses on relatively simple neural-networ
2019 Stiennon et al. 20200; ; prompt generation (Yang et al. 2023). to (task-oricnted) conversational
agent
c.g.: Ouyang et al. 2022 Feng et al. 2023 OpenAl 2023
To alleviate the complexity in fitting an explicit reward model, motivated by the theory of maximum-entrop control and RL (Zicbart et al. 2008 Finn et al. 2016). direct preference optimization methods (DPO, e.g., Rafailov et al. 2023 ZZhao el al. 2023 were recently proposed to directly train LMs on a preference dataset
by using their log-density-ratio as the classification logit.
H More on the Reward Normalizers in PPO Training
To center the assigned rewards from the reward model and reduce their variance, in most open-source (bandit) RLHF PPO implementations (e.g., Havrilla et al. 2023 Hu et al., 2024), the bandit reward of the newly sampled response y is first "7-score" normalized, before being fed into the PPO routine. Concretely, for the prompt x and sampled response y, the bandit reward ro(2, 3) is normalized as ro(x,, y) + (ro(æ, y) p)/a, where μ and a are respectively the mean and standard deviation of (bandit) rewards in the reward calibration dataset. The PPO routine starts by using this normalized r (zt, y), e.g., first subtract it by the KT. regularizer, and then calculate the advantage estimates and value function training target, etc.
For the segment-level action space, we will then need to normalize the reward ra(s;, a,) for each segment ae. As shown in Table 5 ("Global Statistics of All"), the most intuitive idea of simply using the global mean and standard deviation over all segment-level rewards in the reward calibration datasct does not train a good LM. Looking inlo the responses sampled in PPO training and in the reward calibration datasel, we find that, for example, the beginning segments of the responses are typically greeting alike phrases that are less informational and/or essential respond to the given prompt, which tend to receive relatively lower rewards. If we normalize the segment-level rewards of those carly segments by the global mean and standard deviation, those normalized rewards will be significantly negative, rather than centered around 0. This will
30

---

Published in Transactions on Machine Learning Research (09/2025)
undesirably refrain the generation of necessary greeting alike phrases, resulting in "impolite LM" and thus inferior benchmark results. More generally, the linguistic structure of the response leads to certain correlation between the mean and standard deviation of segment-level reward values and the normalized location of segment in the response, e.g., in the early or middle or later part. This observation motivates us design location-aware reward normalizers that can approximately capture the reward statistics al an arbitrary location of the response, so that the normalized segment-level rewards can be more centered and less varying. It important to have proper reward normalizers an arbilrary location of the response, because the response sampled in PPO training will have stochastic total length, nondeterministic number of segments, and less-controllable length of each segment. These considerations motivate our design of the regression-based reward normalizer functions in Section 2.3
More visualization results of segment
We provide additional visualization results to further demonstrate the effectiveness and interpretability of our entropy-based segmentation method. Specifically, we randomly sampled three examples from the preference- 700K training set and applied our method with an entropy threshold of 1.5. As shown in Fig. 6 Fig. 7 and Fig. S we present direct comparisons between human-annotated segments and our automatic entropy- based segmentation, demonstrating that our approach produces semantically meaningful and consistent segments. Furthermore, Fig. g. Fig. TO and Fig. 11 showcase representative examples of entropy-based segmentation across diverse prompts, highlighting the robustness and adaptability of our method to various question lypes and response styles. Notably, the coding samples in Fig, TO and Fig. 11 are randomly selected from the APPS training set (Hendrycks et al., 2021). In these cases, we observe that the entropy-based segmentation naturally tends to split the code at meaningful boundarics, such as line breaks (Qn) or logical code blocks. This behavior indicales that to some exlent, our method can aulomatically capture the structure and semantics of programming outputs without the need for explicit rules supervision.
31

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt: What initiatives partnerships has Jeep established to promote sustainability in the automotive industry?
Thank for reaching out! I'm happy to
help you with your question.
To be honest,
must inform you that
Jeep, as car brand,
does not have
specific initiatives or partnerships
dedicated solely to promoting sustainability the automotive industry.
However, Jeep's parent
Fiat Chrysler Automobiles (FCA),
has made some efforts to reduce
Its environmental impact and FCA announced its commitment to achieve carbon neutrality its set goals to increase the use of renewable energy in its facilities and reduce waste and water usage.
promote susta sustaina iabiiity in its operations.
In 2019,
reduce its greenhouse gas emissions
by 50% and
operations by 2025
The company has also
Furthermore, Jeep has introduced some eco-friendiy features some of its recent vehicle models,
such as the Jeep Renegade, Hellcat, which offers an eco driving mode that
helps to reduce fuel consumption and emissions.
Additionally, some Jeep models come with that helps to improve fuel efficiency and reduce em emissions.
mild- hild-hybrid system
While Jeep has not specifically focused on sustainability in the
automotive industry, it
clear that the company is
committed to reducing its environmental impact
and promoting sustainability in
operations.
hope this information helps answer your question.
you have any further queries,
please feel free to ask!
(a) Human-Annotated Segments
Thank for reaching out I''m happy to help you with your question To be honest,
must inform you that Jeep, brand, does not have any specific initiatives partnerships dedicated solely to promoting sustainability in the automotive industry.
However, Jeep's parent company Flat Chrysler Automobiles (FCA), has made some efforts to reduce
its environmental impact and promote sustainability its operations. 2019,
announced commitment reduce greenhouse gas emissions by 50% and achieve carbon neutrality in its use of renewable energy in its facilities and reduce waste and water usage. Furthermore,
operations by 2025
The company has also set goals to increase the
Jeep introduced some eco
friendly features in some its recent vehicle models, Hellcat, which offers an eco-driving mode that
such the Jeep Renegade
helps to reduce fuel consumption and emissions. Additionally, some Jeep models come with
mild hybrid system that helps improve fuel efficiency and reduce emissions.
While Jeep has not specifically focused on sustainability the automotive industry, is
clear that the company committed to reducing its environmental
impact and promoting sustainability in its operations. hope this information helps answer your
question. you any further queries, please free ask!
(b) Entropy-based Segments
Figure 6: Visualization results comparing tion (b).
segmentation (a) with entropy-based segmenta-
32

---

Published in Transactions on Machine Learning Research (09/202))
Prompt: How has the music scene in Nashville, United State changed over the past century?
Nathellle, Tennetsnn
Poon ling boeen knowum
CMa
hacerobed
morly
jnd matio.
1800
weukesen andlesernois Mare
Bessi
Hanayy forred theuly
country
beggn cmarge
popuaa geveer
Opry
countly
loaddedie Oucyhetedd
Nestvelle ruticnal
In 100Es, rock and roll begant
become pepprine Reabela--eeedd
arttib suth Tresley
13
popalee
and iidhrp Can't halped
pspelance the garte.
3250
Naahnv became canter
for lilk mask.
Many faforsoolumno vint, suchis
Joan Boucord bob
perforedd cly
curing time.
1811s
Nastwille continuddtobe
centerto countrymadd
SATCe Nastrvilee contrued
muae Many popilar country
Dekly Parton
jnnilfier Ropers,
Marha
dsplllyrette heney Wegrr srrenges
Nesladie dhengghhs lave
Nabeella chertia
the www.innneeee moveman. Arcitasuch Eremyja Hards helped populatte gasse.
and Lovett
to antt
the siice Nostreile thrahtertto noseeee
popp//ariz gare, 2000, country
ouch Naverdaa Topeloheened popelatta gerr
3000
popurar ouary
popda ouunty
emegediier
eneeged
Today, Martiile u.a
Chaine www.cccccccccoooo lith
addde oegeeee rnpresrtted
bocay Machillise deesser Cit with
wede gooeeeeeeeeneeeeee
Dann manic aillll
wwr.ssipppplee MATER
Machella, oilo sannes path authcc (M06
Cruntry must the Howood popudn grere
badhill, www.dei,ccaaeeegge grom arhasmei, pPp
ard top abc populit chy Ei
home man sccrtingtaatoos
and hwo aho
popular, dia horetomerry wesrderreeerree dtudkn
massic venues, moordlases Redtediees (lteapop destiaatnnior
Nach
popular desteazon
anintersted
Beurstt whe inoeestedi mutko
Thermais whahe
shangdd le www.ttecccooiin Just mo
The masicaare lestvvile taacharged ooe
buttherty malits
aubort muSk NNztlille saddy
Uat endeng. andits scenels
brant music. Hattell conatard wwching.
32 morptoos eecked see (MS mestorrtaraftrasc Wand
escestion. eocteed what
mast nea in anvieeel Line)
(a) Human-Annotated Segments
(b) Entropy-based Segments
Figure 7: Visualization results comparing human-annotated segmentation (a) with entropy-based segmenta- tion (b).
33

---

Published in Transactions on Machine Learning Research (09/2025)
Prompt: Detailed Instructions: In this task you will break down question into the basic steps required to answer it.'n A question decomposition is numbered list of operations that must be performed to answer the original question. Imagine explaining your question to a friendly droid by listing each action it should take in order for the question to be answered. Each step in our decomposition should refer either an entity (known or unknown), a propery of an entity or query operation (count, group, union, etc.)'n Here are the list of step templates and their deseription''n Seleet: A select step is used to return set of objects. There are no references to previous steps in a select step. template: Return [attributes]lun Inn Filter: A filter step is used to return results from a previous step to which a certain condition applies. template: Return [#step] [condition]is inn Project: A project step should return certain attributes of the results of previous step. template: Return [attributes] of [#step]]nn Aggregate: An aggreeatte step returns an aggregator function applied step's result. template: Return the [aggregator] of [#step]] An Group: A group step is an aggregator applied on attributes, template: Return the [aggregator] of [#step] for each [attribute]]n |n Superlative: superlative step is used to return the result with a highest/lowest attribute among other results. template: Return [#step]] 1] [where] [#step2] [is] [highest lowest]]nn in Comparative: A comparative step is used when we need to compare attribute with number to filter results. template: Return [#step]] [where] [#step2] [comparator] [number] 'nn Union: A union step is used to return results of two steps together. template: Return [#step]] [or 4] [#step2]]n Intersection: An intersection step returns the result that two steps have in common. template: Return [attribute] of both [#stepl] and [#step2]]nn Discard: A discard step returns result of a step and excludes result of another step from it. template: Return [#step1] besides [#step2]]nn In Sort: A sort returns result of another step in specific order. template: Return [#stepl] [ordered/ sorted by] [#step2]|n Is true: An is true step checks condition another result and returns a true or false. template: Return [is if] [condition]] Jin Arithmetic: An arithmatic step operates an arithmatic operation on one or more steps. template: Return the [arithmetic op.] of [#step1] [and] [#step2].inO] :nQ question: How many years after the Treaty of Verdun did Philip IV of France become King?
Greati Ib hoppy to help you odit
your quettion.
To anower your queet on, se need
Creat! I'd be happy to help you with your quesiios aroowr cuestian, need
decomposs kirto smaller stass
Here's tow
Wee ca break I:dawn)
decemmpoe ntoo smolleee ieen ittrrd' how (ow can brrek it cinan :5 lect Swturn ot letuibuuee of Distop In this cass, we need to select the sttributes step that
Return lnttliboss) of (nstal li coes) we need to the
bosts the stop that
santain decisoornzion we're for,
thi
noed th attrbott
tabiin the info mation we're looking for the care, we need to select the "Year"
Fllmr Salurr Imada from
which cartain sandiicnaupp.e
atribbut: Filter Return results from apretious stop worch cortain concition appllis
Inwithh aertien cord spplies
We Wennedd the
reudis Sndithe
your
King
when Philpp Hol France Hicaneeinng
We conditine "lear
concition "Year 1314" totteeer the rosults mayaaaaatt Treaty Verdus
Uneywar
Projects tietrr cartann atte bu be moallss of previcus We rwer to project
stop.
the "Yaar" attribute from the fitered results has egate: Returr agaregono function
Aaprepate Return aegregoor
on sreph
to
applied on atep's a noult, We need app the aggragator funstins 'unn' the projected "Year"
nggregaoo oanctionn the projected "rear"
otoribute find the number years tht the Tresty of Vendun that Phill p IV France
fumbee of year
treary verdun
Philp france becamering
become Knng Groupt faturn tne aggregator lunct on applidd on resulty NNe need group resultshyyhe the sttrbuut spply the apprerttr functton "sug' Millip france
Groupe Beturn the aggrapatr function appl led the neuutt by the "Mer" arttbute the the
pnaset of rendtss
group
find
namber Check concttion on snother result return ortetse
nitheer Insaty teerdun
aggrepato hunntionn "wag")
check
fn thaceegge au merrotyears inar Phils Souoee (sesure king aiserse Thddy Veitton
I1 Philip 21 Fiaroal became King alter the Tready Vandun (aas signad, whichis tru condition. Now
Isoue Chick conddile ona rosutt) and rstame pue or We need check
put all tngether thet answerto yourquestonn Return the (sum
it MP of frunce bocams sing alter the Treaty of danwas sigred
which sa bus car diiinn
ytars) trat Fhilp fV Fronce became King ahtrene Treaty venuu (ong otyaami
New, leth putil (al) coyethor the
arnverto qusation Botam your tram yours)
selecting the wir buse flltwing the neults to condiiton "Wer> 13:44" projscinnn
thee Pys Nor Koree becams King sherthe Tresty afaleda (Dog seann
atrrbutee aggrggtting ressutss by thompinghhnnilyy andaapn appreppter funstion "aag".
stribute
bocciiiingg the "No"" vil Booo) fitering hee Bostsltt to wiliiin "howr
pojocing the "oar" ataibooe uegnopringge rosits by groppng themlyy the "Haa atritatt and applying the aggpeeeeer farction
The Theaswwwrrc 1328
(a) Human-Annotated Segments
(b) Entropy-based Segments
Figure 8: Visualization results comparing human- annotatnd segmentation (a) with entropy-based segmenta- tion (b).
34

---

Published in Transactions on Machine Learning Research (09/2025)
Part #1: Translate text from English to Urdu
Usage:
"The
Survey further notes that India
's energy intensity of GDP
started declining at a much lower level of per capita GDP as compared to the developed world
سركت ب مزيد حوال بهع ك ct ديتا a بباتك ترانان كي يطست كم شدت لكي
ونه ططي مترر
مين سسط كططط مشيوط فصس معيشست مقارن كم مبن
ترقيياقت ممالكك Explanation
The original text is translated from English to Urdu
so it meets the criteria.
Part #2: The translation must not omit or add information to the original sentence
Usage: The translated text" بهص بي مزيب سرفء لكي بون كم شدت تواناني كي معيشست كي بهارت كم at دينا حواد
كي ططر متتصر بالكل شخص في قت طرف ترق كم دنيا مين مقارن كي معيسست مشبرط ممالك بافت
contains all the key information from the original sentence without any omissions or
additions. The translation specifically mentions the decline of India's energy intensity of GDP,
the comparison to the developed world and that it started at lower per capita
GDP. Explanation: The translated text contains all the information from the original sentence so it meets the criteria.
Part #3: The translation must convey the same meaning as the original sentence. Usage:
The translated text "اللك لكك بوم كم شست تواناني كي معيشست كم بهارت كح at ديتا حوال بني
بي مزيد سرت
شننسس في س كطر x طرد مختصر ترقي كم دنيا مين مقارن كي معيطست بوط
ممالك باات
conveys the same
meaning as "The Survey further notes that India's 's energy intensity of GDP started declining at much lower level of per capita GDP as compared to the developed world
Explanation: The translated text conveys the same idea and meaning the original sentence, so meets the criteria.
Figure 9: Example of entropy- basod segmentation.
("import ("importysshinptttt systiipppt-s-sssssiiirrrediinnn inn ans 0\n
in rageggaiinnniiiiitiiiireeieiiiin
for i in range(n - 11:1n =i continueln else:)n ans max(len
(bin(diff)) 2, ans)\n ali 1
"for t in rangel//ntiippoo/x3xxennninninnn [int() for in input().split(jInn m a v=
for in v= m m if 0:An print(0)) else:)n p 1\n c= 0\ while p
p == c== c= 1|n p
for in ittpag/iiiiiitttttttoooiiiiinnni n int(input()))n 11 hitp://pppiiiniiiuuuttpspttttt/ answer 0\n
max_so max_so_far= for in range(1, n):nn if [1 2 max_so_far.) _o__r_ max_so_farr so__sofff__ol]la else:)n
answer= maxanswe,,,, (max_so_fat far lengghhnt \n \n print(answer)")
Figure 10: Example of ropy- segmentation on coding response.
35

---

Published in Transactions on Machine Learning Research (09/2025)
I"n ("n=int(inputt) \na, = map(int, input().plitt))ndd= =I[ for range(n 1)] forj range(n 1)/lnn
for ronge(nn:nn dp(o][i] a[]] range(1, n):(n for range(n -i 1):(n
dpfill))=pp(-1I1/ 1][i] ^ dpli 1]lj 1 for in range(1, n):(n forj range(n -i):(n
dpflll)=maxx(piilii) dp 1]1ii. dp(i 1]lj + 1]])nfor in ranee((n(n((imut))))
1 rr=map(int, input()split())) print(dp(r- I[[I 1]) "# immprr os, sys, atexit|n
# from io import Bytes/O,
StringlO/nfrrom sys itdin, stdoutin in# rypo os. (ifot(().st_size)).rreeeee
#_OUTPUT_BUFFER= Stringi00\n8 stsp::/nssssssssss__ssssuo =yssdoout==OUUTPUBBBFFFEEE
resssrtte def
write(::n@ wttp:odwwwwee....oo.......EEEEE/ Indef colullacccaat len(array)
finalarray = []|n findaarcy.copeendoorrraa/lll (nae while (n!=1):|n for x in range(n-1):\n
return finolarryyn Indefsolve())nn n int(inputt)) prray=(0]]) [0]|n
owwpppeteeeetttiioooooieieeeeeer subArayy []|n for ronge(n+1):\n
subArrays..ppeendf(0 (n+1))\n finllaray==calccllteeeaaaayy # print print
for in range(1,n+1)) for range(x ,n+1)) print (y-x+1
print hore://uAAuurrrooooooooooooonnnn in range(2,n+11)1n
witpr//sss||iii////////r/ssssssa print (subArrays)
for oaryiinanggeeda--)/)/oooonnnnnnn
print (subArrays)\n
q = int(input(i)) for in range(q))n
I, http:///ww.....aeaoooooooooooota
Intry:)n solve()In
except Excepttin print (e)In)n solve()ltit /titte [t lejtitit \tititin "#t=int(input()nnt
=1|nfor_ Inn n=int(input())nn |n |+iist/mpp(imt/nnputtssssptttt/t pi
range1,,))) in
for ortpa//eee....ooiooooooooooo/ooa
for in range(g)))n
wywpapathhhpuuupiiiiii--------ih "importsss\n inputssy..tdinn.eallinnn sireiparrrhhooaiittteemmiiiiiiip ndicc [i=[]\nfor i in range(n):)n xx=[]\n
(aapranngggnllnnnxxxpeeeeeeeee liapppend(xx.oopy)))1 for in range(n):)n for in range(n-i)AAn if
(i=0)) [ii]]]]a[]] else:)n [UJ[)=II[[-1jj]M[[--l[jj)))))))
for rang range(n-i))n |f
range((nt(inputt))Ann
(r-map(intiipputt)splttt))))) print(([[r-[[(-1]) "#I/usr/bin/env python3|n #\n# XOR-p yramdyn#\nimport sys, os)n)ndef read [ad__in(): return int(input())nneefre read
Lints(): return list(map(int, input(..plitt()))n= -#In
Inn read_int()</span>>>> na read [[0] ' for range(n)]\nf =[[0] for range(n)]\n
in range(n 1,-,, -1)Antt f[i][i] a([])n]foo]]])) in range(i n):nntttt f[illj] f[illj fli 1]lj]ln
for range(n 1,, -1)Anttdp([[[] f[][i]]nttfor range(i n)/Anttttdpp(l]]]] max(f(ili), dp[i][j-
1]. dpli 1][]]\nq (read_int()(nfor.. iim.anndchhttt/eeeesneennnnnnnno http::/neooo.iooooooonooonooonnn
Figure 11: Example of entropy-based segmentation on coding response.
36


Reviews:
[{"review_id": "LqgkK0ulAY", "reviewer": "Reviewer_XNou", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. Experiments across several different models show the superiority of the approach relative to multiple different baselines.\n3. Additional justification, results, and details in the appendix do a nice job of supporting the main paper and giving more context. The comparable observations motivating location-aware normalization, for instance, was interesting.\n\nWeaknesses:\n1. Certain design decisions seem rather ad hoc, such as the choice to distribute token rewards within a segment equally and the specific relationship between segment level rewards and the sequence-level reward. It is useful to know that they seem to work alright in practice though.\n2. It is not clear why this proposed approach should be better than an approach like RTO that also uses token level rewards, especially if they are well-motivated to approximate actual token level rewards.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests a modification of the classic bandit formulation of RLHF by proposing a specific instantiation of reward shaping to provide denser rewards in the training of the policy model. Namely, the authors break a sequence into segments based on entropy (with the start of a segment being a token with high conditional entropy) and train a segment level reward model with a specific functional relation to the sequence level pairwise feedback information in the data (i.e., average of the segment-level rewards is the sequence-level reward). Then during policy optimization, the reward for a given segment is evenly distributed across tokens in the segment, after being normalized with a learned mean and standard deviation which take into account the relative location of the segment. Experiments show that this setting outperforms baselines, including those with use coarse grain and fine grained rewards, and ablations show that the particular design choices empirically outperform other simple counterparts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could the authors provide more justification for why this proposed approach should outperform RTO? The paper seeks to differentiate itself from other works which utilize token level rewards, but ultimately the proposed approach is another version of assigning token level rewards in a way such that all tokens within a cycle and get the same reward. Intuitively, I would expect that this constraint is actually harmful, because for instance it's not clear why a predictable continuation of a word or phrase should be rewarded the same as the token which begins the word or phrase.\n2. Can the authors clarify how they implemented the baselines they compared to in table 1 as well as relative accuracies and losses? Specifically for the sentence baseline, how the sentence level reward model was trained, the relative accuracies of the reward models, and ideally the differences in the number of tokens in a typical sentence versus a segment? For the token baseline, the relative accuracies of the implicit reward model vs. the segment level reward model, the reward normalization, and train loss / reward scores / KL achieved in the evaluated checkpoints?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QDwPTXnB9L", "reviewer": "Reviewer_xEau", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe paper proposes a natural and simple approach to training and using segment-level reward models in RLHF. I particularly appreciated the extensive set of ablation studies that verify that each component of the proposed approach adds value.\n\n## Weaknesses\nThe work is motivated by the idea that assigning rewards to individual tokens is problematic, because 1) tokens are not necessarily intuitively meaningful units, and 2) \"The contribution of those tokens to the text sequence will inevitably depend on later tokens,\nmaking reward/credit assignment harder\". In my opinion this is not very convincing, because \"meaningfulness\" is quite a vague concept, and the second point also applies to larger chunks such as segments. I am therefore not entirely convinced by the motivation, but the results do show improvements which is ultimately most important.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a segment-level reward modelling method, to improve RLHF for LLMs. Existing RLHF methods either use a bandit formulation, where a reward is assigned to the complete response, or provide token-level rewards. Bandit methods suffer from sparse feedback which is known to make RL difficult, and token-level methods suffer from semantic ambiguity (how much reward should we assign to a sub-word like \"ing\"?). The proposed takes the middle ground, rewarding segments determined by entropy-based segmentation.\n\nTo train the reward model, the authors propose to aggregate segment-level rewards to match existing sequence-level preference labels with a Bradley-Terry loss. During RL (PPO), they use location-aware reward normalization and within-segment reward interpolation, to evenly distribute segment rewards among tokens. \n\nResults show that the model significantly outperforms bandit and token-level approaches on multiple benchmarks (AlpacaEval 2, Arena-hard, MT-Bench) and also outperforms DPO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J1yC6NtKZ1", "reviewer": "Reviewer_VeAx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :-\n\n- The core idea of segmenting text into semantically meaningful chunks offers a compelling middle ground between the coarse sequence-level and the often oversubtle token-level reward assignments. \n- The proposed method demonstrates superior performance across multiple benchmarks (AlpacaEval 2.0, Arena-Hard, and MT-Bench) when compared to established baselines like bandit, sentence-level, and token-level RLHF. \n- The use of predictive entropy for text segmentation is an elegant and fully automated solution, which circumvents the need for manual annotation or external tools that other fine-grained RLHF methods might require.\n- The paper shows that their segment-level PPO approach consistently outperforms DPO, a widely adopted alternative to RLHF that does away with an explicit reward model. For example, on Phi3-mini Instruct, Segment-PPO achieves an 86.7 on GSM8K, compared to DPO's 81.6 (Table 3). This provides strong evidence for the continued relevance of PPO-based methods when coupled with a well-designed reward structure.\n\nWeaknesses :-\n\n- The paper's method for segmentation hinges on the assumption that lower entropy in a language model's predictive distribution corresponds to a \"semantically complete text segment.\" While this is a reasonable heuristic, the paper does not provide a rigorous justification for this link. It is possible that low entropy could also be a result of other factors, such as repetitive or formulaic language, and may not always align with true semantic boundaries. The paper states, \"...tokens within a semantically complete text segment can be more predictable by the corresponding generation context...\" (Section 2.2), but this remains an assumption.\n\n- The authors claim their method improves quality without \"cheating the benchmark evaluations by response-length hacking\" (Section 4.2). However, the mechanism for how segment-level rewards might influence the length of generated text is not fully explored. Since the final reward is an average of segment rewards, this could inadvertently incentivize the model to produce a larger number of shorter, high-reward segments, which might not always lead to a better overall response. A more detailed analysis of the relationship between segment length, reward, and output length would strengthen this claim.\n\n-  The paper proposes using a simple linear regression on the log-transformed normalized location to create location-aware reward normalizers. The paper acknowledges this limitation, stating, \"Future work may extend our linear regression-based normalizer functions in Section 2.3 with non-linearity and/or more features\" (Section 4.3, part c). While the linear model shows good results, it may not capture more complex relationships between a segment's position and its expected reward, especially in long or structurally complex responses.\n\n- The decision to evenly split the segment-level reward among its constituent tokens is another heuristic choice. The paper notes, \"Future work may design a proper non-even split of segment-level reward over each token in the text segment\" (Section 4.3, part d). This even split ignores the possibility that some tokens within a segment might be more critical to its overall quality than others.\n\n- While the benchmarks used are standard for instruction-following and dialogue tasks, the paper's claims would be more robust if the method were tested on a wider range of applications. The limitations section acknowledges this, stating, \"Future work will focus on scaling our approach to even larger LMs, extending its applicability to diverse tasks such as mathematical reasoning and code generation...\" (Section 5). The current evaluation does not provide evidence that the entropy-based segmentation and reward model would be effective in these more structured domains.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel approach to improve RLHF by addressing the sparse reward problem inherent in traditional bandit-based formulations. Instead of assigning a single reward to an entire generated sequence, the authors propose a segment-level reward model. This model assigns rewards to semantically coherent text segments, which are sequences of tokens that are more meaningful than individual tokens but more granular than the full text. The paper introduces a method to dynamically segment text by thresholding the entropy of the language model's predictive distribution, based on the idea that beginnings of new segments are less predictable. To train this segment-level reward model using standard sequence-preference datasets, the authors aggregate the individual segment rewards into a single sequence-level score. For the policy learning phase using PPO, the paper introduces location-aware reward normalizers, which are functions that adjust the reward normalization based on the segment's position within the sequence. This is achieved by learning a regression from the normalized location of a segment to the mean and standard deviation of rewards. Furthermore, the segment-level reward is interpolated across its constituent tokens to create an even denser training signal. The authors validate their method on several RLHF benchmarks, including AlpacaEval 2.0, Arena-Hard, and MT-Bench, and show that their approach outperforms both traditional sequence-level and more recent token-level RLHF methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The core assumption of the paper is that text can be segmented into \"semantically complete\" units by thresholding the entropy of the language model's predictive distribution. This is stated in Section 2.2: \"Recent works... have noticed that tokens within a semantically complete text segment can be more predictable... whereas the starting token of a new segment is comparably less predictable\". While this is an intuitive and elegant heuristic, it remains an assumption. To make the paper's primary contribution more robust, I suggest the authors to provide a more rigorous justification for this link. Proposed Action: I recommend conducting a small-scale qualitative analysis. The authors could randomly sample 50-100 responses from a test set, have a human annotator segment them based on semantic completeness, and then compare these manual segmentations to the ones generated by the entropy-based method. Providing a few illustrative examples in the Appendix, showing where the method succeeds and perhaps where it fails, would substantially strengthen the foundation of the work. This would provide direct evidence supporting the central hypothesis beyond the (strong) downstream results.\n\n\n- The two hybrid baselines in Table 1 are important for the paper's argument, but their descriptions are very brief. For instance, for \"Bandit as Segment,\" it is stated that \"...rewards are computed at each segment by feeding the corresponding logits into the reward model\". It is not perfectly clear what \"logits\" are being referred to here (e.g., the logits of the tokens in the segment? An aggregated representation?). Proposed Action: I suggest adding a few more precise details in Section 4.2 or the Appendix explaining the exact inputs to the reward model for these two specific baselines. This would improve the reproducibility and clarity of these important comparisons.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XzFVjZ4AfJ", "reviewer": "Reviewer_3Cy9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": ">**Strengths:**\n\n1. The paper is generally well-written and easy to follow. The research problem and motivation has been clearly articulated in the introductions.\n2. The core idea of using segment-level reward assignment is interesting. Further, each component of the proposed approach such as segmentation technique, reward normalization etc has been clearly explained. \n\n3. The empirical evaluations seems comprehensive in terms of models and datasets. The authors performed several ablations, which provide useful insights. \n\n\n>**Weaknesses:**\n\n1. The proposed method is evaluated exclusively within a PPO-based RLHF framework. While the paper shows better performance compared to a baseline DPO model, it is unclear if the segment-level reward modeling concept could be adapted to offline algorithms such as DPO. \n\n2. The text segmentation is based on the entropy of the SFT model, both for reward model training and during PPO training. The rationale for not using the current learning policy's entropy for segmentation is not fully justified. Since responses are sampled from the current policy during PPO, using its entropy for segmentation seems more intuitive. \n\n3. The paper does not have any analysis on the KL divergence between the trained policy and the SFT model. \n\n4. Based on current setup, it is difficult to comprehend how the segment-level rewards are distributed for codinginstruction prompts. The authors should also add evaluation for these kind of prompts.\n\n>**Questions:**\n\n1. \"after looking into the dataset, in our experiments, we select Average to differentiably highlight the better average quality of the chosen responses over the rejected one.\" -- Does this imply that the choice of the aggregation function is a dataset-dependent hyperparameter that may require manual tuning for different datasets?\n\n2. The win-rate in Tables 1 and 2 appear somewhat low. Is the win-rate relative to SFT policy? How is the win-rate calculated: (#win + #tie/2)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the challenge of reward sparsity in RLHF training by proposing a segment-level reward assignment. They argue that in standard RLHF approaches, which are typically based on a bandit formulation, rewards are only assigned after the end-of-sequence (EOS) token, resulting in a sparse feedback. While token-level rewards could provide more dense signals, current tokenization methods often generate tokens that are not semantically meaningful, making this approach suboptimal. To mitigate this, the authors introduce segment-level credit assignment, where segments are defined by thresholding on entropy. Empirical evaluations on AlpacaEval 2.0, Arena-Hard, and MT-Bench demonstrate the effectiveness of segment-level rewards over both sequence-level and token-level credit assignment strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  It is important to visualize the relationship between performance and KL divergence. Specifically, the authors should focus on analyzing reward vs. KL or win-Rate vs. KL trade-off for the different baseline methods. This would provide clear evidence of efficiency of the proposed segment-level training.\n\n2. The current evaluation focuses primarily on instruction-following benchmarks like AlpacaEval and MT-Bench. Evaluation on coding instruction prompts or specifically multi-step reasoning tasks would be interesting.\n \n3. I understand due to space constraints the evaluations metrics have been deferred to Appendix. For improved readability, the result tables in the main paper should be more self-contained. Currently, metrics such as \"LC\" and \"WR\" are used in Tables 1, 2, and 3 , but their definitions are deferred to Appendix D.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)