MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Authors: Yekun Chai
Haoran Sun
Huang Fang
Shuohuan Wang
Yu Sun
Hua Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Human Alignment
Large Language Models
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces MA-RLHF, a framework that incorporates macro actions into RLHF for large language models, addressing the credit assignment problem and significantly improving learning efficiency and performance across various tasks.

Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences. However, token-level RLHF suffers from the credit assignment problem over long sequences, where delayed rewards make it challenging for the model to discern which actions contributed to preferred outcomes. This hinders learning efficiency and slows convergence.In this paper, we propose MA-RLHF, a simple yet effective RLHF framework that incorporates macro actions --- sequences of tokens or higher-level language constructs --- into the learning process. By operating at higher level of abstraction, our approach reduces the temporal distance between actions and rewards, facilitating faster and more accurate credit assignment. This results in more stable policy gradient estimates and enhances learning efficiency within each episode, all without increasing computational complexity during training or inference. We validate our approach through extensive experiments across various model sizes and tasks, including text summarization, dialogue generation, question answering, and program synthesis. Our method achieves substantial performance improvements over standard RLHF, with performance gains of up to 30\% in text summarization and code generation, 18\% in dialogue, and 8\% in question answering tasks. Notably, our approach reaches parity with vanilla RLHF $1.7 \sim 2$ times faster in terms of training time and continues to outperform it with further training. We make our code and data publicly available at \url{https://github.com/ernie-research/MA-RLHF}.

Published as a conference paper at ICLR 2025
MA-RLHE REINFORCEMENT LEARNING FROM HU- MAN FEEDBACK WITH MACRO ACTIONS
Yekun Chai" Haoran Sun' Huang Fang Shuohuan Wang Yu Sun Hua Wu
Baidu Inc.
{cha i yekun, fanghuang, wangshgchannnnununnunnnee @ba sunhaoran 4 020 com
@ba idu.com
ABSTRACT
Reinforcement learning from human feedback (RLHF) has demonstrated effec- tiveness in aligning large language models (LLMs) with human preferences. How- ever, token-level RLHF suffers from the credit assignment problem over long sequences, where delayed rewards make it challenging for the model to discern which actions contributed to preferred outcomes. This hinders learning efficiency and slows convergence. In this paper, propose MA-RLHF, simple yel ef- fective RLHF framework that incorporates macro actions sequences of tokens higher-level language constructs into the learning process. By operating at higher level of abstraction, our approach reduces the temporal distance be- tween actions and rewards, facilitating faster and more accurate credit assignment. This results in more stable policy gradient estimates and enhances learning effi- ciency within each episode, all without increasing computational complexity dur- ing training or inference. We validate across various model sizes and tasks, including text summarization, dialogue gen- eration, question answering, and program synthesis. Our method achieves sub- stantial performance improvements over standard RLHF, with performance gains of up 30% in text summarization and code generation, 18% in dialogue, and 8% in question answering tasks. Notably, our approach reaches parity with vanilla RLHF 1.7 2 times faster in terms of training time and continues to outper- form it with further training. We make our code and data publicly available at mttps://gitaee com/ werneerreseacee wwweeearrhh///EEeEEeE
approach through extensive experiments
INTRODUCTION
Recent advancements large language models (LLMs) have revolutionized natural language pro- cessing tasks, demonstrating impressive capabilities across a wide range of applications such as code generation (Roziere al., 2023; Chai et al., 2023; Lozhkov al., 2024), mathematical rea- soning (Lewkowyez et al., 2022; Anil et al., 2023), and dialogue assistance (OpenAI, 2023; Team et al,, 2023; Anthropic). Despite these successes, aligning LLMs with human values and preferences remains a critical challenge. Reinforcement learning from human feedback (RLHF) has emerged as a promising approach to address this alignment issue by incorporating human evaluations into the training process (Christiano et al., 2017; Ziegler et al., 2019; Stiennon al., 2020).
Existing RLHF (Ouyang et al., 2022: Bai et al., 2022; Askell et al., 2021) methods mainly opti- mize decisions at the level of individual tokens, and require process vast number of minute adjustments. However, this fine-grained training paradigm can lead to the credit assignment prob- lem (Kaelbling et al., 1996; Pang et al., 2019; Machado al., 2023b; Pignatelli al., 2023), partic- ularly when dealing with long-distance dependencies. As LLM agents attempt to optimize decisions across extensive sequences, the difficulty in attributing the credits of actions to specific tokens com- plicates the reinforcement learning (RL) process (Pignatelli et al., 2024). Moreover, the use of subword tokenization, such Byte-Pair Encoding (Sennrich al., 2016), often splits words into
Equual contribution. Correspondence to: YC. Work done during HSSss internship at Baidu.

---

Published as a conference paper at ICLR 2025
smaller pieces. For instance. OpenAl's ChatGPT' treats each token three quarters of a word on average, resulting in sequences that are 33% longer than word counts (OpenAI, 2024) and further exacerbates the credit assignment problem.
Additionally, standard RLHF methods may overlook essential local co-occurrence patterns or in- herent structures between adjacent tokens in natural language. For example, consider the phrase Big Apple', treating Big and Apple e as isolated decisions misses the cohesive meaning of the term, which actually refers to the "New York City". The token-level granularity of natural language can hinder the agent's 's ability to capture high-level language constructs in RL optimization, as some sequences are better understood when evaluated holistically.
To address these challenges, we propose a new framework called macro-action RLHF (MA-RLHF) that incorporate macro action sequences of tokens or high-level language constructs into the RLHF framework. The concept of macro actions, has been explored in the literature of planning (Iba, 1989; Korf, 1985; Sacerdoti, 1974) and reinforcement learning (Thrun Schwartz, 1994; Precup et al., 1997; Hauskrecht et al., 2013), simplifies decision-making by operating at high levels of temporal abstraction under the framework of semi-Markov Decision Processes (SMDPs) (Sutton et al., 1999b). Macro actions leverage temporal abstraction by chunking the sequences and reducing the decision resolution, enabling the agent to learn from "long-sighted" macro-level actions instead of "short-sighted" token-level actions. This can potentially lead to improved learning efficiency and scalability. Alternatively, MA-RLHF can also be interpreted from the perspective of reversing tokenization; MA-RLHF serves as de- de-tokenization process reconstruct high-level language units from subword pieces. By merging tokens into macro actions, we reduce the number of decision points and shorten decision trajectories, alleviating the credit assignment problem caused by long temporal distances.
To conclude, our main contributions are as follows:
We propose MA-RLHF, a simple yer effective RLHF framework that integrates the macro ac- tions into RLHF to align LLMs with human preference. We demonstrate the effectiveness of our approach through extensive experiments across various datasets and tasks, including text summarization, dialogue generation, question answering, and code generation.
We show that MA-RLHF achieves 1.7x to 2 faster learning efficiency in reward scores during training compared the standard token-leve RLHF, without introducing additional computa- tional costs during training or inference. MA-RLHF also exhibits strong scalability across model sizes ranging from 2B 27B parameters.
Our analysis reveals that MA-RLHF exhibits robust generalization capabilities under varying experimental settings, such as temperature values and rejection sampling, consistently outper- forms the standard RLHF approaches.
2
PRELIMINARIES
We introduce some basic concepts and notations used in RL and RLHE
2.1 REINFORCEMENT LEARNING AND POLICY OPTIMIZATION
Problem Definition RL addresses the problem of finding a policy make optimal sequential de- cisions in environments modeled as Markov Decision Process (MDP) (Sutton & Barto, 1999). An MDP is defined by the tuple (S, A. P, r,, Po: Po: 7), where S denotes a finite set of states, A is a finite set of actions, P S x A x S + [0, 1] represents the state transition probability distribution, r : x A + is the reward function, Po : S + [0, 1] defines the initial state distribution, and y (0, 1) is the discount factor that determines the importance of future rewards.
Given a trajectory (so. ao. $1, a1 ..., a reward Pt = r(st, az) is received at each time t. The state-action value function (= (+) 1.......1.... [E/o rre++1] measures the expected return of taking action de at state St and following policy thereafter. The value function V*(ss) Ee. 7 rrtti] estimates the expected return from state s, under the policy л. The ad- vantage function (s,, a,) (n, 4,) V, (s;) reflects the relative value of taking action a, at state s, compared to the average value of the state.
'nttps://platfomm itw.ppppaaooooooooii....mmm www./o/////cccccooooooooo http:://eewwwiiiiiaa.i..oo /tpp
2

---

Published as a conference paper at ICLR 2025
The goal of RL is to find optimal policy (a(a s), parameterized by 8, that maximizes the pected cumulative discounted reward: J(Ø) = ap........ I=== 2'r,). where Su ~ Pu($o) represents the initial state distribution, dr ( 8e) denotes the action selection based on the policy, and 8141 P(8++1 Se; 0t) specifies the state transition dynamics.
common approach for optimizing policies by estimating the gradient of a performance objective with respect to the policy parameters --== A, V a log (e(a; 8e]]. where the expectation E|:] is taken over the randomness of the initial state, policy, and state-transition. The policy gradient guides us how adjust the policy parameters to improve the expected return. Among the family of policy gradient methods, Proximal Policy Optimization (Schulman et al., 2017, PPO) is perhaps the most widely-used one due to its simplicity and empirical effectiveness. PPO simplifies TRPO (Schulman et al., 2015) by using clipped surrogate objective function to penalize large deviations from the old policy, thereby ensuring more stable updates. Specifically, PPO introduces a clipped objective function:
Proximal Policy Optimization Policy gradient methods
0. The policy gradient is given by: (O)
To(ai st) Teau (as (1))
As, clip( To(a |si) To (ar sc)
(P E, min
e, 1 (,111) e)4, (.11)))))
(I)
where is hyperparameter that defines the range for clipping. The expectation E, [...] indicates the empirical average over finite batch of samples. Nowadays, PPO usually comes as the first choice for RL practitioners.
2.2 RLHF FOR HUMAN ALIGNMENT
The post-training of LLMs (Stiennon al., 2020; Ouyang al., 2022) is multi-stage training paradigm to align LLMs with human preferences. Post-training typically involves three stages:
(1) Supervised Fine-Tuning (SFT) stage: pre-trained language model (LM) fine-tuned on dataset of human demonstrations, learning to generate responses that align with human instructions and preferences.
(2) Reward Modeling (RM) stage: A reward model is trained on a labeled preference dataset D NN (I v M/ Ji=1 consisting of prompts x; and pairs of responses (y4" ..;)) where y, preferred over yi by human annotators. The reward model ra (I:, y), parameterized by @,, is trained using the ranking loss: LRM-
log or (log(r&(z, y+) rI y-))), where denotes the sigmoid function.
(3) RLHF stage: The RL fine-tuning utilizes the RM to provide feedback on the generated outputs, optimizing the policy using RL methods such as PPO. The reward signal is modified by incorpo- rating a Kullback-Leible (KL) divergence penalty to balance the exploration of new policies with adherence the SFT model. The reshaped reward is defined as:
R(x, y) = ( v) DDLL(TT x:) || Tan. x));
where my represents the policy learned through RL, Tah the policy produced from the SFT stage, and B > 0 is hyperparameter that controls the strength of the KL penalty.
In the RLHF stage, the PPO algorithm, as detailed Equation (1), employed to optimize the RL policy. In the context of RLHF, we denote the state (so, dp, 41,.... at } as the sequence of tokens generated up to time step I, while so represents the initial states, iee., the prompt, and a, represents the token selected at the t-th position.
3 MARCO-ACTION RLHF
3.1 REVISITING MACRO ACTIONS (OPTIONS)
Macro actions, also referred to as options (Sutton et al., 1999b), are high-level constructs that encapsulate sequence of primitive actions (i.e,, subword tokens); by its definition, macro actions allows an agent to operate at a coarser temporal scale.
Formally, macro action is characterized by three components: (1) a policy S A [0
which guides the action selection among actions; (2) a termination condition Ç S+ - 1]. which determines where the macro action should end; (3) a initiation set I C S, which is subset of states
3

---

Published as a conference paper at ICLR 2025
Vanilla RLHF
MA-RLHF
Reward Model
MDP
Optiize
R()
Macro actions over MDP
PPO
MA-PPO
reward
Jotel
RM
action mecro action
END
the trajectory
episode
Figure I: Illustration of the MA-RLHF optimization framework. Standard RLHF makes decisions and evaluates value scores at the token level, while MA-RLHF makes decisions over sequences of tokens at a coarser temporal scale.
that macro actions can begin with. Once initiated with state E I, the macro action follows policy until it reaches the termination condition according to Ç. Intuitively, the use of carefully designed macro actions can extend decision-making temporally, it allows the agent to avoid "short-sighted" token-level decisions and encourage "long-sighted" macro-level decisions, thereby simplifies the decision-makinn process and potentially enhances learning efficiency.
3.2
RLHF WITH MACRO ACTIONS
We describe how we integrate macro-actions into the existing RLHF framework, the resulting frame- work is named as macro-action RLHF (MA-RLHF).
3.2.1 FORMALIZATION OF MACRO ACTIONS
We denote macro actions as w1, W2,...2....... wy. In the context of LLMs, macro action W, consists of a sequence of consecutive tokens, i.e., w, {at,, 0t,+1, ......., 0t,+1-1), where t, is the starting index of the 7-th macro action. We let |w>| denotes the number of primitive actions that w- contains. Unless otherwise specified, use r index macro actions/states and use t to index primitive actions/states.
As mentioned in $3.1, macro actions are defined by the policy model, the termination condition and the initiation set. In MA-RLHF, we set the policy model the same as the standard token-level RLHF and let the initiation used in MA-RLHF is decided solely by the termination condition, which plays a crucial rule in the MA-RLHF framework. We explore three termination conditions in this work:
be any possible sequence of tokens. Therefore, the macro action
n-gram based termination:
Following Vezhnevets et al. (2016), we find that n-grams serve as simple yer effective termination condition for macro actions, i.e., |w>|| = n, where n represents the length of the n- n-gram. We consider two variants of the n-gram termination condition: (a) Fixed n- gram: We group tokens into fixed-lengtl re-grams, simplifying the action space while maintaining common linguistic patterns. We empirically find fixed r-gram macro action perform best and use it as the default setup. (b) Randomized n-gram: We randomly select the length n-gram from predefined list of lengths {2, 3, 5, 10) to introduce variability, allowing the policy to adapt to different sequence lengths.
based termination: wy is derived from syntactic semantic parsing of the input text, aligning macro actions with grammatical structures like phrases or clauses. Concretely, we tra- verse the constituent terminal nodes until current non-terminal state contains no more than a specified threshold of leaf' tokens, set at C'
of the entire sequence using depth-first search (DFS), expanding
(PPL) termination: Perplexity measures the likelihood of sequence of to- kens. Here, the perplexity of macro action proportional to the averaged entropy of the token within it, i.c., pp](w>) & 1
Perplexity-
[oo] Eaew, log Pa- A macro action terminates until it reaches
token that has negative impact the perplexity of the macro action. Mathematically, struct w, = (a....... -1141-13 such that ppl(w, U atrm)) > ppl(w,) and ppl( (allfat...... ..)) 2
ppl( ((a,,,,...,,...,.. (1+1)) for all < i lr+1
4

---

Published as a conference paper at ICLR 2025
After determining the macro action based the termination condition, apply the state value function and importance sampling at the macro level Equation (1). We provide the details of imple- mentation in Appendix D.1.
3.2.2 POLICY OPTIMIZATION WITH MACRO ACTIONS
In MA-RLHE, we adapt the PPO algorithm for optimization, referred as MA-PPO. In the context of LLMs, expanding the action space with additional macro actions/tokens results in re-architecting the LLM's vocabulary and retraining the model, which is computationally prohibitive. Thus, we maintain the original action space pretrained LLMs, which can be treated "single-step" prim- itive options as noted in (Sutton et al., 1999b). The policy To still outputs probabilities over individual tokens, but for optimization,
consider the joint probability of the macro action: To(w, 9r) I1 Te(ar act). The macro reward for executing the macro action W, the macro time step is defined R, = =[[ (w=11 p're,ts sy], where Ti is the reward received time step t, and we set the discount factor = our experiments.
Each macro action represents contiguous sequence of tokens, and is treated as option in the SMDP framework. The option-level value function with macro action is then estimated as:
V (s+: = E R, >V (st.a)) ST, Wr]
where y is the discount factor for future rewards beyond the macro action.
The advantage function An (S7,477) in MA-PPO determines how much the chosen macro action outperforms the average, which is defined as Ar (+r,w>) Qn(8r,
V (s()). Similar to definition stated $2, (* (r) is the expected return conditioned on executing w, at state 8y, which is calculated by summing the immediate macro rewards from the macro action with the discounted value of the subsequent state.
In MA-PPO, the objective function is adapted for MA-level evaluation. The policy gradient is computed based on the advantage of the MA sequences:
To (wi 87) Å,, clip Th (w+ 8T)
To(wr ST) e, c,1+e) A, }+
(MA-PPO(0)) (0)=E, min
Tee Ww sT)
where A, is the estimated advantage at macro time step т, is constant that defines the range for clipping, and Teas the policy before the update.
3.2.3 CONNECTION TO PREVIOUS METHODS
MA-RLHE builds and generalizes prior work in the RLHF literature by varying the length of macro actions. When the macro action length is set to 1, MA-RLHF reduces to the standard token- level RLHF (Stiennon al., 2020; Ouyang al., 2022), operating
MDP. Conversely, if allow | + 00, then MA-RLHF converges toward methods like RLOO (Ahmadian et al., 2024), REINFORCE (Williams, 1992; Sutton et al., 1999a), and GRPO (Shao al., 2024), approximating a contextual bandit problem where decisions are made based on the entire sequence context. By varying the length of macro actions |wy|. MA-RLHF provides a flexible framework that balances the granularity of action decisions. We provide further analysis on the impact of |w,| in $4.3.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Tasks and Datasets We evaluate MA-RLHI on three different datasets for open-ended generation tasks: TL:DR (Stiennon et al., 2020) dataset for text summarization, Anthropic Helpful and Harm- less (HH-RLHF) (Bai al.. 2022) for dialogue generation and WebGPT Comparison (Nakano et al., 2021) for question answering. Additionally, we evaluate MA-RLHE on code generation using the APPS (Hendrycks al., 2021) dataset. More details can be found in Appendix B.I.

---

Published as a conference paper at ICLR 2025
Vanilla MA PPO
RM score (28)
Percentage
7.5%
RM
Venta
Veniin
4.00
3000
2000
2002
4000
1990
Training step
Training stop
RM Score
Figure 2: Test RM scores of Gemma-2B and Gemmaa7B 7B models on the TL;DR dataset. The shaded regions represent the standard deviation on test RM scores across training runs.
Figure 3: RM score distribution
for PPO and MA-PPO (2B) fi- nal steps (4.6k) on TL:DR.
Win
Ti:
Logg
Win
Loss
GPT- Evaluation
Gemma-28
75%
22%
GPT-4 Evaluation
Germa-22 26
34%
38%
GPT- Evaluation
Gemma.78 7B
86%
14%
CPT-4 Evaluation
Gemma 7B
72%
25 25%
Human Evaluation
Gemma-22 2B
Human Evaluation
Gemma- 28
52%
Human Evaluation
Gemma-78
69%
Human Evaluation
Gemma-77 78
56%
20%
Conerente
Cancual
Uher Pertomiit Alannas
Win Rate
% Win Rate
Figure 4: Win rates of MA-PPO against vanilla PPO TL;DR (left), HH-RLHF (middle) and WebGPT Comparisons (right), estimated by GPT-4 and Human.
Base Models and Training Details For open-ended generation tasks, 2B (Team et al., 2024) as our base model; we further adopt Gemma-78 7B and Gemma-2-27B to test the scaling trend. For the program synthesis task, we use CodeGemma-l.1-- 1.1-7B-it as our base models. The data split for SFT RM / PPO and the hyperparameters used in SFT / RM/ / PPO stages are detailed in Appendix B.2. The implementation details of MA-PPO can be found in Appendix E.
pre-trained Gemma-
and CodeGemma-
Evaluation For open-ended generation tasks, our evaluation metrics includes RM scores, GPT-4 pairwise evaluation, and human pairwise evaluation. To compute the RM score, we randomly sam- ple 2k validation instances for the TL;DR and HH-RLHF datasets and use the default validation set of the WebGPT dataset. For GPT-4 and human evaluations, we simulate the win-rate on 50 instances that are drawn from the instances used in the RM evaluation. The GPT-4 and human evaluations are based task-specific criterion: relevance, coherence, consistency, and fluency for TL;DR; help- fulness for :HH:: factual accuracy, coherence, and usefulness for WebGPT. We followed prior studies (Askell al., 2021; Zheng al., 2024) by randomizing the order of responses during evalu- ation to mitigating potential evaluation biases. The prompts used by the GPT-4 evaluation are placed in A Appendix F, and the annotation rules used for human evaluation are given in Appendix F.2. For the program synthesis task, we utilize pass 1 and @ metrics to assess the performance of the model, evaluated on the provided 5k test set.
4.2
MAIN RESULTS
In this section, we present the main results of applying MA-PPO across three key tasks: summa- rization, dialogue, and question answering. The main takeaway is that MA-PPO consistently out- performs vanilla PPO terms of both training efficiency and generation quality; MA-PPO obtains a significant improvement in testing reward model scores and human/GPT-4 evaluation win rates.
TL;DR Summarization For the TL:DR summarization task, MA-PPO shows marked improve- ment over vanilla PPO. As shown in Figure 2, MA-PPO achieves parity with vanilla PPO approx- imately 1.7 times faster during training. Specifically, Gemma-2B trained with 1.7k MA-PPO updates reaches similar testing RM scores obtained by vanilla PPO trained with 3.7k steps. We also find similar trends when scaling up the parameter sizes to 7B, demonstrating the generalized capability of MA-PPO on model sizes.
Moreover, Figure highlights the distribution of RM scores, where MA-PPO consistently shifts towards higher RM sores compared to vanilla PPO. Further evaluation using GPT-4 given in the left figure of Figure 4, shows that MA-PPO achieves 78% and 86% win rate over vanilla PPO for the 2B and 7B models, respectively. Human evaluation gives similar results, where MA-PPO obtains win rates of 74% and 69%, further demonstrating the effectiveness of macro actions. The final testing RM scores of MA-PPO and vanilla PPO are given in Table 2.

---

Published as a conference paper at ICLR 2025
Table 1:
Agreement among RM, GPT-
Table 2: Test RM scores of vanilla PPO and MA-PPO on TL:DR, HH-RLHF, and WebGPT datasets.
4, and human evaluations on TL;DR.
#Param
RM
GPT-4
Human
RM
100%
Model
TL;DR
HH-RLHF
WebGPT
GPT-4
2B
78%
100%
Vanilla PPO (2B)
0.84
-0.62
Human
76%
58%
100%
MA-PPO (2B)
1.41
1.55-189
-0.60.3s
RM
100%
GPT-4
7B
78%
100%
Vanilla PPO (7B)
1.90
1.05
-0.61
Human
74%
64%
100%
MA-PPO (7B)
2.47.008
1.24.18%
-0.56.a%
1.50
Mulno (andorbbet
1.25
RM score
0.50
Vanilla PPO
0.25
MA-PPO (Fixed)
MA-PPO (PPL)
GPT-4
MA-PPO (Random (zed]
-0.25
MA PPO (Parsing)
-0.50
reference
1000
2000
3000
4000
Training step
Relevance Coherence Consistency Fluency
Figure 5: Performance of MA-PPO with various macro action termination strategies on the TL;DR dataset using Gemma-2B. Left: Test RM scores for different termination strategies. Right: GPT- 4 evaluation across four dimensions relevance, coherence, consistency, and fluency comparing different MA lermination methods.
HH-RLHF Dialogue We use the HH-RLHF dataset to evaluate the helpfulness and harmlessness of single-turn dialogues. MA-PPO shows clear advantages over vanilla PPO, as depicted in the middle figure of Figure 4. GPT-4 evaluations show that MA-PPO yields a 72% win rate for the Gemma-7B model, compared to 58% for the Gemma-2B model. Human evaluation results align with these findings, with the win rate increasing from 52% to 56% as model size scales from 2B to 7B. The testing RM score of MA-PPO and vanilla PPO are presented in Table These results highlight the scalability and effectiveness of MA-PPO in dialogue tasks. We refer to Appendix C.1 for detailed experimental results.
We evaluate MP-PPO the WebGPT Comparison dataset for question- answering tasks. As shown in Figure 4 (Right), MA-PPO consistently outperforms vanilla PPO, with GPT-4 evaluations yielding win rate of 64% for the Gemma-7B model. This result demonstrate the robustness of MA-PPO across different tasks, including more structured tasks like question answering. More experimental details refer to Appendix C.2.
WebGPT Comparisons
Validating Model-based Judgments with Human Evaluation We evaluate the reliability of our evaluation methods by calculating the agreement between the reward model, GPT-4, and human evaluators. Since GPT-4 and human evaluations are conducted pairwise, we determine the reward model's win rate by selecting the summary with the higher RM score. The results, shown in Table 1, demonstrate that the reward model aligns more closely with both GPT-4 and human evaluations. Furthermore, the agreement between GPT-4 and human evaluators averaged 62% across models, reinforcing the consistency and validity of our evaluation framework.
4.3
ANALYZING THE USE OF MACRO ACTIONS
We study the performance of various termination strategies. Unless otherwise specified, we conduct analysis the TL;DR dataset.
4.3.1
EXPLORING DIFFERENT STRATEGIES FOR MA TERMINATION (C)
In MA-RLHF, the termination condition (() for macro actions is critical as it determines when macro action should conclude. We compare the performance of various termination strategies, par- ticularly on reward maximization and linguistic coherence. The termination strategies studied in this section including fixed randomized n-gram-based, nation, aforementioned in $3.2.1; please Figure 12 for detailed illustration.
baasdd, and perplexity-base termi-
Figure 5 illustrates the overall test-set performance on RM scores (Left) and GPT-4 evaluation scores (Right) with different MA termination strategies. All macro action termination strategies outperform the vanilla PPO approach, underscoring the importance of temporal abstraction in decision decision-making making. Figure 5 (Left) shows that n-gram based approach, both fixed and randomized, achieves the opti-
7

---

Published as a conference paper at ICLR 2025
175
3222
Vord
150 pmo
0:50
M&PPPO in
MA-PPO in 101
156 PMo
022
030
2233
X00 Trining stop
220
4030
1300
2000
5003
<0ia 5306
Training step
Figure 6: Test RM scores of different values in MA-PPO evaluated by corresponding RM on the TL;DR (left) and HH- RLHF (right) dataset.
Figure 7: GPT4 scores of
vanilla PPO and MA-PPO with different n values on TL;DR.
SFT
1.00
0.6
1.68
RM Score
1.25
3.15
1.48
TeA2
Ted.8
1:20
Two2 1-0.
Tw02 T=0.
T-04
T-10
T-24
T-14
T-24
T-10
0.0
T-066
T-12
T-2.6 T-12
T-2.6 T-12
Best-of-N Figure 8: The effect of temperature on RM scores for varying sample sizes (Best-of-N) across models. (Left): RM score of the SFT model under different temperatures and sample sizes. (Mid): RM score of vanilla PPO under the same settings. (Right): RM score of MA-PPO.
Best-of-N
Best-of-N
mal results among others. Notably, randomized n-gram-based termination performs the best across multiple dimensions, including relevance, coherence, and consistency, shown in Figure 5 (Right). As expected, the perplexity-base prioritize smooth and natural language generation. Furthermore, parsing-base termination shows promising ability to handle complex grammar, as is designed to better capture linguistic structures.
termination enhances fluency, and is most suited for tasks that
4.3.2 ABLATION STUDY: VARYING n IN MA-RLHF
The n--gram based macro action strategy in MA-RLHF uses hyper-parameter to control the length of macro actions. Notably, when n = 1, MA-PPO is equivalent to vanilla PPO, and treats the problem as a traditional Markov Decision Process (MDP), making decisions token by token. In contrast, setting n + oo corresponds to the REINFORCE algorithm (McGovem & Sutton, 1998), where the entire sequence is treated as single macro action, akin a contextual bandit problem, as discussed in 3.2.3. For intermediate values of n (i.e., E (1, oo) MA-PPO falls under the SMDP framework, which allows for temporally extended actions: see $3, This continuum between MDPs and contextual bandits highlights the flexibility of the MA-RLHF approach in handling varying levels of temporal abstraction.
RM Scores We conducted experiments with varying values of n (n {3, 5, 10, oo}) on the TL;DR and HH-RLHF datasets. Figure 6 shows that all values of lead to performance improvements over the vanilla PPO (n 1). indicating the advantage of modeling sequences of tokens as macro actions. Notably, for the TL:DR dataset, n oo yields the highest RM score, suggesting that treating the entire sequence For the HH-RLHF dataset, setting n = 10 gives the best performance, likely because this task benefits from moderate-lengee macro actions that capture essential linguistic structures while maintaining sufficient granularity.
macro action is particularly effective for the summarization task.
GPT-4 Evaluation Analysis As shown in Figure 7, setting n 5 strikes a good balance between relevance, coherence, consistency; it outperforms both smaller and larger values of n. These findings align with the semi-MDP framework: increasing allows for better credit assignment and context retention, but excessive abstraction (e.g., n = oo) sacrifices fine-grained control. Overall, moderate values of n = 5 and n 10 provide the best trade-offs. highlighting the adaptability across tasks. 4.4 GENERALIZATION PROBING IN MACRO ACTIONS
Robustness on Rejection Sampling vs. Temperature Best-of-N (a.k., rejection sampling) (Tou- vron et al., 2023) enhances response quality by selecting the highest-reward response from N samples generated by the policy model. We compare MA-PPO, SFT, and vanilla PPO using the best-of-N sampling across various temperatures T e {0.2, 0.4, 0.6, 0.8, 1.0, 1.2) and sample sizes N [4, 8, 16, 32). As shown in Figure 8, best-of-N sampling improves RM scores for all methods,
8

---

Published as a conference paper at ICLR 2025
RM Score
GPT4 Eval
Human Eval
MA-PPO
3.
6.2%
Reward Score
Win Raits
RM
1.2%
1000
Training stop
2006
5008
4003
0.0%
RM Score
28 7B 27B
28 78 279
28 7B
27B
Figure 9: Evaluation results for vanilla PPO and MA-PPO on Gemma-2-27B using the TL:DR
dataset. Left: RM scores on validation set. Mid: Distribution of RM scores for vanilla PPO and MA-PPO (27B) at final steps (4.6k). Right: Scaling trending on TL;DR dataset across 2B, 7B, and 27B model size, showing RM scores, GPT-4 evaluation, human evaluation results.
Do0 Scon
20000 10 TM
SF RM Score
SFT RM Scor
S5T 8M Score
3FT Score
Figure 10: RM score shifting pattern after RLHF training; Left: RM scores of best-of-N (N 8) sampling compared to the SFT model. Mid Left: RM scores of vanilla PPO compared to the SFT model. Mid Right: RM scores of MA-PPO (n = 5) compared the SFT model. Right: RM scores of MA-PPO (n bo) compared to the SFT model.
with performance increasing as N grows. We observe that SFT and vanilla PPO are sensitive to temperature variations, requiring specific adjustments to achieve optimal results. In contrast, MA- PPO demonstrates robustness in sampling temperature, it consistently delivers the best performance at T 1.2 and shows consistent improvement across all tested temperatures. Moreover, MA-PPO maintains stable performance across varying temperature settings. as detailed in Appendix D.4, highlighting its robustness and generalization capabilities under different sampling temperatures.
Scaling Trends up to 27B Models We evaluate the performance of MA-PPO across different model sizes, specifically Gemma-2B, 7B, and 27B. As demonstrated in Figure 9 (Left and Mid), MA- PPO consistently surpasses vanilla PPO, exhibiting higher RM scores throughout training. Figure 9 (Right) presents the scaling trend of MA-PPO across the 2B, 7B, and 27B models terms of testing RM scores, GPT-4, and human evaluations. The experimental results underscore the scalability and robust performance of MA-PPO across varying model sizes.
Analyzing the Impact on RM Score Distribution We evaluate the RM score distribution shift after applying RLHF using vanilla PPO and MA-PPO on the TL:DR dataset, with the SFT model serving as the baseline. To further contextualize the impact of RLHF, we include the Best-of-N sampling (N= = 8) on the SFT model. As illustrated in Figure 10, Best-of-N enhances overall response quality but falls short compared to RLHF. While vanilla PPO shifts the distribution towards higher RM scores, it leaves a significant number of low-quality, long-tailed instances. In contrast, MA-PPO demonstrates more pronounced positive impact, effectively reduces the number of low-quality outliers and improves overall score distribution compared with the vanilla PPO. This highlights the robustness of MA-PPO in enhancing response quality through RLHF.
4.5 ADDITIONAL ANALYSIS
Impact on L2-Norm of Advantage and Q Values We present the L2-norm of both the advantage and Q-values for MA-PPO and vanilla PPO during training in Figure 11. The advantage function, which reflects the difference between the expected return (Q-value) and the baseline, is critical in guiding policy optimization. A lower nornomm of both the advantage and Q-values suggests more stable and less noisy policy updates, likely contributing to faster learning speed observed in $4.2.
The policy gradient for a sequence of length T is given by: VoJ E ΣT Va log Ta (als) R). where R is the sequence reward provided by the RM. In the case of using n-gram based macro actions, the sequence length is reduced by factor of shortening the decision horizon: T + T/n. This reduction in the number of actions, T/n, where n > 1, implies that the temporal distance between actions and corresponding rewards is decreased, thus reducing the variance in the gradient

---

Published as a conference paper at ICLR 2025
12 Nomm Advantages
PRG
La Norm of Values
Table 3: Pass@ (k (1,53) metric evaluated on the APPS test set.
Vort 7PO
Method
CodeGemma-28
CodeGemma-78
PPO
MA-PPO
PPO
MA-PPO
Inter
2.82
,,25,
4,26
6.22,vv
200h
Step
Step
passi
Comp.
Inro.
All
15.26
0.92
4.92
5.45,
16.56.
0.94.2
20.90
6.98
1.21
26.74,09
2/00,09
9,48,
Figure 11: Ly Norm of advantages and Q-values during training for MA-PPO and vamilla PPO. Left: L2 norm of ad- vantages over training steps; Right: T-2 norm of Q-values.
Inter.
4.19
4.37,70
6.57
8,37
Iatro.
17.30
18,30..
23,30
30.30
Comp.
1.70
1.00,s
2,30
4,30,
All
6.26
6,00,9
9.06
11.74,37
estimate and improving credit assignment. We refer readers to Mann & Mannor (2014) for the theoretical foundations of variance reduction through macro actions and their benefits in RL.
Case Study We show some qualitative examples in Appendix G.I, demonstrating that MA-PPO can produce more coherent and contextually appropriate responses compared to vanilla PPO, capturing both short/long-tern dependencies effectively.
Extended Experiments: Code Generation We further assess the effectiveness of MA-PPO on the code generation task. Following Shojaee et al. (2023); Liu et al. (2023). we utilize the compiler sig- nal the final reward; see Appendix B.5 for implementation details. We compare the performance of MA-PPO and vanilla PPO using the pass @K k (k=1, 5) metric (Chen et al., 2021) on the 5k test set of the APPS dataset (Hendrycks et al., 2021). As shown in Table 3, MA-PPO significantly out- performs vanilla PPO in both pass @ and pass @ 5 metrics, with more pronounced improvements as model size scales. Notably, for the 7B model, MA-PPO achieves an improvement of +35% in pass @1 I and +30% in pass @5 over vanilla PPO, demonstrating the effectiveness of our approach in code generation tasks.
5 RELATED WORK
LLM Alignment RLHF have shown impressive success in aligning LLMs with human preferences through multi-stage training, including SFT, RM, and RL fine-tuning (Ziegler et al., 2019; Stien- non et al., 2020; Ouyang et al., 2022: Sun et al., 2025). Recent research has explored optimization methods for RL in LLMs, employing both online (Ahmadian et al., 2024; Farebrother et al., 2024; Shen et al., 2024; Chakraborty et al., 2024; Shao et al., 2024) and offline RL algorithms (Snell et al., 2023: Hu al., 2023; Yu et al., 2024) to address training instability, improve efficiency (Tang al., 2024) and diversity (Sun et al., 2025). Improvements to RM learning have been proposed, such as parameter scaling (Gao et al., 2023), fine-grained reward (Wu et al., 2023), tool use (Li al., 2024), and model merging (Ramé et al., 2024; Rame et al., 2024). Alternatively. direct policy optimiza- tion (Rafailov et al., 2024; Ethayarajh et al., 2024; Gheshlaghi Azar et al., 2023; Rosset et al., 2024) has emerged as promising approach, bypassing the instability of RL while directly aligning mod- els to human preferences. In this paper, we enhance the RLHF action space by integrating macro actions, a well-established concept in RL (Sutton et al., 1999b; Mann & Mannor, 2014).
Macro Action in RL Macro actions introduce temporal abstraction in RL by grouping sequences of primitive actions, reducing decision complexity and improving long-horizon credit assignment (Pre- cup et al., 1997; Hauskrecht et al., 2013; Sutton et al., 1999b; Pignatelli et al., 2024; Machado et al., 2023a). This method has demonstrated its utility in speeding up convergence and stabilizing policy updates in various domains (Mann & Mannor, 2014; Solway et al., 2014). Our work applies macro actions RLHF in LLM training, leveraging this structure to enhance scalability and optimize credit assignment over extended sequences.
6 CONCLUSION AND FUTURE WORK
In this paper, we introduced MA-RLHF, a novel framework that incorporates macro actions into RLHF to enhance the alignment of LLMs with human preferences. Our approach demonstrates con- sistent improvements across multiple tasks, including summarization, dialogue generation, question answering, and code generation. Notably, MA-RLHF achieves parity with vanilla RLHF 1.7x to 2x faster in reward scores without incurring additional computational overhead, showing robust scala- bility across model sizes ranging from 2B to 27B parameters. is promising to explore MA-RLHF in complex step-by-step reasoning tasks for future research.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We committed to ensuring the reproducibility of the experiments presented in Section 4. To this end, we make the source code and model checkpoints publicly available at https: / wwt//eennerrsearee ch ChAAMMMAHEE The detailed source code for training and evaluating both the conventional RLHF and our proposed MA-RLHF approach is included in the supplementary materials. We believe that these efforts will enable researchers to rigorously verify findings and build upon our work.
ACKNOWLEDGMENTS
We would like to express our gratitude to the anonymous reviewers for their insightful and construc- tive feedback.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint urXiv:2402. 14740, 2024.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Pas- sOs, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Mor- eira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Chool Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Fein- berg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. CoRR, abs/2305, 10403, 2023. doi: 10.48550/arXiv.2305. 10403. URL BRLhttps: /twi. wrg/10.../..oooo..o
111111...88550// 48550/ wwt.xi...30000 2305 10403.
Anthropic. Introducing the next generation of Claude - anthropic.com. anthropic. www.coo//eeeeeeee www.:eeww/eeeeeee wtt./////wwww///c/eeeeeee
family. [Accessed 22-07-2024]..
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant laboratory for alignment. CoRR, abs/21 12.00861, 2021. URL httpt::t////tliiioee httpss:///xx.v.or///eeeeo 21 2112. 00861.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jack- son Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862. 2022. doi: (0.48550arXXv..224..555 220400 URL. https: / /ow...///.//// www.or//// 48.50.855//aaa0 2204.22222.2 05862.
Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. ERNIE-code: Beyond English-centric cross-lingual pretraining for programming languages. In Anna Rogers, Jordan Boyd-Graber,
Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguis- tics: ACL 2023, pp. 10628-10650, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 2000655//112222 http:////1222....nnnnn-eg.lll/l findings-acl.676. URL https: /aclantthoogyy orr/ 2023. fi i
676.
11

---

Published as a conference paper at ICLR 2025
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.0892). 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- htcp//aper/211///// /
4299-4307, 2017. URL https: http://pooceedeigeeee / neur Abstract..................-.a... catttrrrtt.ttl......
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5; Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558- 3567,, 2019.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.03950, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10835- 10866. PMLR, 2023. URL https: /ttp/proccee ppwpeedincgnnseeee mlr press / v22 //////2.h.h..m//////// 3ao23h.html. html.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. general theoretical paradigm to understand learning from human preferences. arXiv e-prints, pp. arXiv-2310, 2023.
Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas L Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. arXiv preprint arXiv:1301.7381, 2013.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Bench- marks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: -at.aaset---eenccateeeeeeeee hash/ c2 4 htta://22c.cffeeee111eeeeeaeeee.
neurips ec http//appr/222211 Abst round2. roupraaoee--ttttaaarooou.eeeeee html.
Jian Hu, Li Tao, June Yang, and Chandler Zhou. Aligning language models with offline reinforce- ment learning from human feedback. CoRR, abs/2308..1202 2023. doi: 10.48550/ARXIV.2000. 12050. URL httph:tt///. /oi. oor.lorgg111 118550/aaXaa 2308. 12050.
Glenn A Iba. A heuristic approach the discovery of macro-operatoors Machine Learning, 3:
285-317, 1989.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
12

---

Published as a conference paper at ICLR 2025
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: survey. J. Artif. Int. Res., 4(1):237-28)) May 1996. ISSN 1076-9757.
Richard E Korf. Learning to solve problems by searching for macro-operators, 1985.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman- Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solv- ing quantitative reasoning problems with language models. In NeurIPS, 2022. http://papers.t www.iisspppaaaeeeeee firaaperfllleeeppperrreeee /ttpperr/222/hhaee 1 Babbeefcct2200115550000EE. htt.:://eeeS..aaaaaaaaaaaaaa---- Consraact-ooferencee
URL
html.
Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. Tool- augmented reward modeling. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https:ttp////..... http///pppeeeeeeeeeee net/ forum?idi ?immmiiiiii d9400dT1XXxx
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang. and Deheng Ye. RLTF: re- inforcement learning from unit test feedback. Trans. Mach. Learn. Res., 2023, 2023. URL.
https: /ttp::///peeeeeeeeeeee iew. net/ forum? j
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirien Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402.1 /9/73, 2024.
Marlos C. Machado, Andre Barreto, Doina Precup, and Michael Bowling. Temporal abstraction in reinforcement learning with the successor representation. Journal of Machine Learning Research, 24(80):1-69, 2023a. URL http: : / /1111//1/////211111333 html.
Marlos C Machado, Andre Barreto, Doina Precup, and Michael Bowling. Temporal abstraction in reinforcement learning with the successor representation. Journal of Machine Learning Research, 24(80):1-69, 2023b.
Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Bet- policies with fewer iterations. In Eric Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 127-135, Bejing, China, 22-26 24 Jun 2014, PMLR. URL https: / //proceedings. mlr. press /veesss//////e/ 2222//an14...m html.
Amy McGovem and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. Computer Science Department Faculty Publication Series, pp. 15, 1998.
Relichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09333 2021.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10:48500anXin.2908.87744
URL htt.tttts://ee //oii org 10. .11.1...8555555 48550/ 201555////iii.. .20.l......3..
OpenAl. What are tokens and how to count them? articles wo000--wwaaaaee-tttteaaaaooooooa cessed 30-09-2024).
httpp::/heelp.peeaa.. compe///.ccm///////////e 2024. [Ac-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. URL hwwp: /appers//pppppapp/ep/ nips cc/paper files//eeeeeeeee.. http:/pppprre/eee//epppp 22t/2h22//hh/// bl
64 7438534444777144400000000 -tte:
Abst Conferennceeeeeeee - html.
Zhen-Jia Pang, Ruo-Ze Liu, Zhou- Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4691-4698, 2019.
13

---

Published as a conference paper at ICLR 2025
Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment deep reinforcement learning. arXiv preprint arXiv:2312.01072. 2023.
Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Trans. Mach. Learn. Res., 2024, 2024. URL https: htttt:://penneeeeeeeeee /oppnreveeeeeeeeeeeeeeeeeeeeeeee net forum? iirrummiddbbbt25S122..
Doina Precup, Richard S Sutton, and Satinder P Singh. Planning with closed-loop macro actions. In Working notes of the 1997 AAAI Fall Symposium on Model-directed Autonomous Systems, pp- 70-76. Citeseer, 1997.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, lean-Baptist Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by terpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: on the benefits of weight averaged reward models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https::////c..... 47RDnNUJy6.
https:://peneeeytst /ttp:///ppeeeiiieeeeom www.:nnunffoiiiooooooooo
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03775. 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. htt:::238..2999990 /2950, 2023.
Earl D Sacerdoti. Planning hierarchy of abstraction spaces. Artificial intelliggnne, 552):115-135, 1974.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.::1777.06477 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/1//11-111 wtg/p16-2000 1162.
URL https://////... htt///acaanattoooo
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li. Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprini urXiv:2402.03300 2024.
Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, and Yang Liu. Improv- ing reinforcement learning from human feedback using contrastive rewards. arXiv preprint arXiv:2403.07708, 2024.
Parshin Shojaee, Aneesh Jain, Sindhu Tipimeni, and Chandan K. Reddy. Execution-based code generation using deep reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023. URL
hitts://cpeereet net forum? ttte://ooommmddoouaaaaaeeee OxORuaxXF.G..
14

---

Published as a conference paper at ICLR 2025
Charlie Snell, Ilya Kostrikov, Yi Su, Sherry Yang, and Sergey Levine. Offline RL for natural lan- guage generation with implicit language Q learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL hwtpt:tps///.......... /wpppppeeeeiieeeeeeeeeeee net forum? i BBaaaDddEevot
Alec Solway, Carlos Diuk, Natalia Córdova, Debbie Yee, Andrew G Barto, Yael Niv, and Matthew M Botvinick. Optimal behavioral hierarchy. PLoS computational biology,
2014.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler. Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, 0ts/2000901325, 2020. URL https: //axxiv org/ aba 015
Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Curiosity-driven reinforcement learning from human feedback. arXiv preprint arXiv:2501./146, 2025.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2): 229-235, 1999.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999a.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-
211, 1999b.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. CoRR, abs/2405.08444 088448 2024. doi: 10.48550/ARXIV.2405.000 2405. 08448.
URL https: doi. wtwpooooog/..... 48150//x5555/aai1
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint ar arXiv:2403.08295, 2024.
Sebastian Thrun and Anton Schwartz. Finding structure reinforcement learning. Advances in neural information processing systems, 7, 1994.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu. Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09288 2023. doi: https: //t:///doii wrg/oo.........oo..ooooo........ 48540/ arwiv. 230. 2307.09288.
09288.. URL
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Aga- piou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems, 29, 2016.
15

---

Published as a conference paper at ICLR 2025
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. (5,-599,20000 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL pettps://opennee
net forum? t
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed- chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint 20w:2388.01200 2023.
Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, and Hongxia Yang. ma (B) S-coerr Value- based deep reinforcement learning for program synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: httsss///peenevie..co net/ farettfformmide ELf589bx1f.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08594 2019.
A LIMITATIONS
While work demonstrates the effectiveness of MA-RLHF across multiple tasks, there are sev- eral limitations that leave room for future improvements. our implementation, we apply the identical action / vocabulary space pretrained LLMs, considering the fact that defining macro actions as one options (e.g., one macro action per n--gram) would require re-architecting the LLM's vocabulary and retraining the model, which is computationally infeasible. Meanwhile, our macro action termination methods are rule-based, including linguistics- or perplexity-driven approaches; future research could explore more complex learnable termination strategies to further enhance performance. Furthermore, regarding the generalization of MA-RLHF, our experiments are con- ducted using models with up 27B parameters; exploring more advanced models, such as LLaMA 3.1 405B (Dubey et al., 2024) or other fate--ofthh--e art architectures and tasks (e.g., mathematical and complex reasoning), may provide additional insights into the scalability of MA-RLHE. Lastly, although we observe significant improvements in training efficiency, further investigation into the trade-offs between training stability and performance under diverse real- worl-world conditions is neces- sary. Addressing these limitations will pave the way for more robust applications of MA-RLHF.
B EXPERIMENTAL DETAILS
B. DATASETS AND TASKS
TL;DR Summarization In this task, the policy asked to generate summarizations for Reddit posts. This dataset consists of 93k human-annotated preference pairs and 86k pairs for validation. The trainable pairs are derived from the Reddit TL.DR (Völske et al., 2017) dataset. Additionally, a portion of the validation pairs is sourced from the CNN Daily Mails, which serves as the test set for out-of-distribution generalization.
HH-RLHF
With the Anthropic HH-RLHF dataset, the policy is asked to generate helpful and harmless response given a single-turn dialogue or multi-turn dialogue. This dataset provides 112k leference-labeled instances for training, and 12.5k for validation.
16

---

Published as a conference paper at ICLR 2025
WebGPT Comparisons The WebGPT Comparisons dataset contains QA pairs from the ELI5 (Fan et al., 2019) and the TriviaQA (Joshi et al., 2017). The policy is responsible for information retrieval and response generation. In experimental setup, focus exclusively The policy must generate a response that balances factual accuracy and coherence. This dataset contains 19.6k instances for training. We split 5% instances for validation, as separate validation set is provided.
the generation task.
Code Generation For this task, we leverage the APPS dataset, which contains 5k training and 5k validation instances. The policy must write executable code based on a natural language described in the question, using Python as the target programming language.
We present the data statistics in Table 4.
Table 4: Statistics of datasets involved in experiments. The number of tokens are calculated with
Gemma-2B tokenizer.
Num. of Comparisons
Num. of Train Samples
Num. of Sumples
Avg. Tokens Prompt
Avg. Tokens in Chosen
Avg. Tokens
Dataset
Rejected
Anthropic HH--RHHF
127.5k
112k
12.5%
160
83
OpenAT Summarization
17%
92.9k
86.
325
35
OpenAI WebGPT
19.6k
18.5k
979
149
137
APPS
203
B.2 TRAINING DETAILS
Following the procedure used by InstructGPT (Ouyang et al., 2022), we fine-tune both the SFT model and the reward model on the same dataset to avoid distribution gap. We implement training code with the Deepspeed-Char package (Yao et al., 2023).
SFT Training We split the dataset into three parts, allocating 20% of the data in the supervised fine- tuning stage. We use the prompts and the chosen sentences as the instruction data. For the TL;DR Summarize dataset, concatenate the post and summarization following the approach of Stiennon et al. (2020). For the single-turn dialogue and the question answering dataset, we apply a human- assistant chat template format the instructions. For the program synthesis dataset, we format the instruction data in line with Hendrycks et al. (2021).
Reward Modeling In this stage, we use 40% of the data to train the reward model for each dataset, formatting the preference data the same way as in the SFT training stage. We initialize the reward model using the fine-tuned SFT model. Due to the lack of preference pairs in the program synthesis dataset, this stage is omitted for this task.
PPO Training Similar to previous stages, the remaining 40% of the data is used to optimize the policy model. The SFT model initializes the policy model, and the reward model initializes the critic model. For the program synthesis dataset, 80% of the data is used in this stage, with both the policy and critic models initialized using the SFT model. The pass @ metric serves the reward signal for program synthesis, compensating for the absence of reward model. While training 7B model on TL;DR dataset using MA-PPO, encountered unstable training with KL coefficient of 0.05. Reducing the coefficient to 0.01 for the 7B model led to more stable optimization.
Table 5 lists the hyperparameters used across all training stages for each task.
B.3 NOTATIONS
In Table 6, we present the notations used in our paper.
B.4 DETAILS OF MACRO ACTION TERMINATION
The general form of the segmentation rule is thus f,+1 = fr + |wr], where |w,| is determined by the chosen criterion, such n-grams, random, parsing, perplexity-baser segmentation.
1. Fixed n-gram length: For all macro actions, set |wr| n, where is constant value.
17

---

Published as a conference paper at ICLR 2025
Table 5:
for training Gemma series of models in MA-PPO and vanilla PPO.
Gemma
CodeGemma
Hyper-Purameler
2B 64 WebGPT
7B
27B
2B
7B
Batch
128
512 others
WebGPT for others
SET
Epachs
3
WebGPT 5c-5 for others cosine 0.1
Learning rate
20-5
5c-6
50-6
2c-6
LR scheduler Warmup ratio
cosine 0.1
cosáne 0.1
cosine
cosine
WebGPT
Batch size
64 far others
64 for I11I RLUP
128
WchGPT
RM
Epochs
2e-5 for WebGPT
Learning rate
le-6
8c-6
Ic-5 for others cosine
LR scheduler
cosine
cosine
Warmup ratio
Batch size
256 1.5e-5
256
256
Policy learning
le-6
7e-7
5e:7
Se:7
Critic learning rate
1.5e-5
le-6
1e-6
5e-5
Se-5
for WebGPT
for WebGPT
Epochs
I for others
for others
epochs
PPO
Rollout
1
Clip ratio
0.2
0.2
0.2
in GAE
0.95
0.95
0.95
0.95
0.95
y in GAF
0.1 WebGPT 0.05 for others
Kl. cvefficient
0.1
0.05
0.05
prompt length
512
512
600
Max response length
512
512
512
512
512
Warmup steps
200
200
0
20
Temperature
0.8
0.8
1.0
Top-p
1.0
1.0
1.0
1.0
1.0
Top-k
50
n-gram Based Termination
Randomized Based Termination
Actions
10
Gg
dc-1
d2
Parsing Based Terminatioon
Randimm Select
Macro Action
Termination
the
crew
repair
Perplexity Based Termination
The spaceship
PPL:
1.82
Macro Actions
from
Re-1
60
Figure 12: Illustration of four termination rules for macro actions in the MA-RLHF framework. Each termination rule outputs a list of w.r In the parsing based termination, the macro action is determined when the token number the current node is less than C 4, which is represented as number in the tree node.
2. Randomized n-gram length: We define list of [[w+]) = (2, 3,5, 10) model macro actions. This list is repeated multiple times to cover the length of the sample, in practice, we repeat this list 3 times. If the total length of macro actions can match the number of tokens, large number will be considered as an additional |w>| to mitigate this gap. which is similar to the |oo| = co. We shuffle the list and take this as random based length.
Parsing-based length: We parse the response into a constituent tree and perform a depth- first search (DFS) identify macro action length. Two rules guide the termination of wy] (1) nodes
18

---

Published as a conference paper at ICLR 2025
Table 6: List of notation used in this paper.
Sym.
Meaning
RL
S
A finite of states.
A
A finite set of actions.
P
The state transition probability distribution.
r
The reward function.
Pu
The imitial state distribution.
The discount factor related with future rewards.
To(a
Policy parameterized by 6.
((T))
The expected cumulative discount reward.
at
The actions selected by the policy.
(s)
The state-action value function.
(+t)
The state value function.
(s ())
The advantage function.
G
The expected return.
RLHF
ro(a, v)
The reward model parameterized by ø. Prompt.
y
Chosen response.
Rejected response.
KL coefficient.
The range for clipping in PPO.
t
Time step of tokens.
Macro Action
S
Termination condition.
I
Initiation set.
The index of macro action/stateerewaard
Wy
Macro action at time step т.
ty
Time step of macro actions.
ar The weight used measure the value of maero action.
with fewer than C tokens mark the end of macro action; (2) nodes with single token included in the last macro action, avoiding single-token termination conditions like punctuation. Due to differences between the training and parsing tokenizers, revert to the standard PPO method when discrepancies occur. We set the cut-off threshold C = 5, providing optimal granularity in practice.
4. Perplexity- -based length: Given response generated by policy model, we calculate the per- plexity Pt any time step / by treating ys* as the ground truth response. This process lever- ages the logits from the reference model, avoiding additional forward passes. Intuitively, se- lecting the macro actions based on perplexity P {po- Pi.... :Py/ can be defined as se- lecting tokens which consistently attribute to the decrease of the perplexity given partial sen- tence. Mathematically, it can be represented as wi {a,, (t, +1,...; ((,+/u--11 where Pt, (Pt,,
exhibits monotonic decreasing pattern.
B.5 TRAINING SETTINGS OF PROGRAM SYNTHESIS
Defining the reward score solely based on the state "Accept" or "Wrong Answer" is somewhat re- strictive, some generated code may pass certain unit tests while failing others. These actions should also receive positive signals to encourage the policy to maximize the number of passed unit tests. To address this, incorporate adaptive compiler signal into the reward feedback previ-
19

---

Published as a conference paper at ICLR 2025
24-
726
Vanill 1913 MGPPO
Percentage
RM score (28)
3.0
AN 200
30%
M4.10 M4-9P0
1083
120
1000
RM Score
Figure 14: Distribution test RM scores for vanilla PPO and MA-PPO (2B) at final steps (5.6k) on the HH-RLHF dataset.
Figure 13: Test RM scores evaluated by corresponding re- ward model of Gemma-2B and Gemma-7B model on HH- RLHF dataset.
-266
MS--PO
RM (28)
-055
RM score (78)
-0.00
Percentae
6%
-2.00
1.0%
-10
XA-710
M4-7P0
1269
RM Score
Training
Figure 16: Distribution of test RM scores for vanilla PPO and MA-PPO (2B) at final steps (3.2k) on WebGPT dataset.
Figure 15: Test RM scores evaluated by corresponding reward model of Gemma-2B and Gemma-7B model on the WebGPT Comparisons dataset.
ous work (Shojaee et al., 2023; Liu et al., 2023):
N pass
0.3 + 1.3.
if y successfully compiled.
N pass + Nfall
R(x,y)
0.6,
if y received runtime emror
1.0,
y received compile rrror.
where æ represents the prompt, and represents the code snippet generated by the policy model.
C ADDITIONAL EXPERIMENTS RESULTS
C.I RESULTS OF DIALOGUE GENERATION
In Figure 13, we demonstrate the RM scores on the validation set of vanilla PPO and MA-PPO. shows that MA-PPO surpasses vanilla PPO under RM evaluation, MA-PPO achieves parity perfor- mance at 3100 step and 2600 step for 2B and 7B models, respectively, while vanilla PPO at 5100 step and 5400 step. Generally, MA-PPO is 1.6-2x faster than vanilla PPO. Figure 14 compares the RM score distribution of both methods.
C.2 RESULTS OF QUESTION ANSWERING
We assess the performance of MA-PPO on the OpenAl WebGPT Comparison dataset, which focuses on the question answering task.
Figure 15 presents the evaluation results based on the reward model. We observe that the policy model challenging to optimize in this task. likely due to the suboptimal performance of the reward model. We applied early stopping during PPO training since the policy model exhibited reward hacking behavior which generated repetition tokens to inflate higher reward scores towards the end of training. Despite this, evaluations the saved checkpoints show that MA-PPO still outperforms vanilla PPO across both tested model sizes. The reward score distribution in Figure 16 further confirms that MA-PPO achieves superior reward scores.
20

---

Published as a conference paper at ICLR 2025
Table 7: Test RM scores of SFT model. vanilla PPO, MA-PPO, and baselines: DPO and RLOO on TL;DR and HH-RLHF datasets.
TE:D0
24%
72%
RLDO PPO
TL,0A
50%
48%
MX-PD
11.00
DPO v.5. FPG TL.0R
34%
50%
Method
RM Score
RM Score
DPO v.S. MA FPO
(TL:DR)
(HH-RLHFF
SFT
-0.64
0.13
HH-ALMF
DPO
0.03
0.64
140
Win
RLOO
0.81
PPO
0.83
1.31
Win rates of DPO and RLOO against PPO and MA-PPO on TL:DR and HH-RLHF estimated by GPT-4.
Figure 17:
MA-PPO (n=5)
1.40
1.55
When using GPT-4 as the judge, we consider three different metrics to evaluate the answers generated by the policy: factual accuracy, coherence, and usefulness overall, following previous work (Nakano et al., 2021). The win rates depicted in Figure 4 (Right) show that MA-PPO consis- tently outperforms the policy trained with vanilla PPO across all criteria. Notably, MA-PPO achieves higher win rates in coherence and usefulness compared to factual accuracy. Human evaluation was conducted to select the preferred answer between those generated by the two policy models. Re- sults in Figure 4 (Right) show that answers produced by MA-PPO were predominantly preferred by human annotators.
C.3 COMPARING WITH ADDITIONAL BASELINES
In this section, compare MA-PPO with two additional baselines: DPO (Rafailov et al., 2024) and RLOO (Ahmadian et al., 2024) on Gemma-2B model. Both of the methods are implemented with Deepspeed-Char with the same data split as we used when training PPO. RLOO model is trained on TL;DR dataset only, with the same policy and reward model initialization as PPO. For the training details of DPO, the learning rate is set to 2e-7, with / 0.1 for TL:DR and 8 0.01 for HH-RLHF. The policy and reference models are initialized using the same SFT model as in PPO. For RLOO, the learning rate for the policy model is set to 1.5e-5, and the number of online samples is K 4. All other hyperparameters are kept consistent with PPO.
Specifically, DPO models trained TL:DR and HH-RLHF datasets,
We demonstrate the results evaluated by reward model score in Table 7, and win rates estimated by GPT-4 in Figure 17. On TL;DR dataset, DPO fails to gain improvement compared PPO and MA-PPO, while RLOO achieves similar performance compared to PPO. but outperformed by MA- PPO. On HH-RLHF dataset, DPO exhibits superior performance than PPO but still underperforms the MA-PPO.
C.4 EXPERIMENTS ON LLAMA-3.2-3B
We conduct experiments on Llama-3.2-3B
model validate the generalizability of our method across different model families. The experiments are conducted on TL:DR dataset, following the same data split as Gemma-2B. We set the learning rates actor and critic to 5e-6 and le-5, and the KL coefficient is set to 0.1. Table 8 demonstrate the results evaluated by RM score, show MA-PPO still remark- ably outperforms vanilla PPO. Using GPT-4 to assess the win rate, MA-PPO obtains 61% win, 4% tie and 34% loss rate compared against PPO. These results prove the generalizability of our method.
Table 8: Test RM scores of Llama-3.2-3B models on TL;DR dataset.
Method
RM Score (TLDDR)
SFT
2.38
PPO
3.33
MA-PPO (n=5)
3,96
21

---

Published as a conference paper at ICLR 2025
Actions
Ag
22
a3
Equal Assignment
de
0.5
0.5
0.5
0.5
o
Value Function
Ohtic Model
V(,,)
0.6
0.9
(ss,)
0.9
1.5
0.3
Position Decay Assignment
Unit Assignment
1/3
2/3
2/3
(s+)
((++)
Figure 18:
Illustration of value function of macro actions in MA-RLHF framework. takes outputs from the value function of tokens input, and returns the value of macro actions with different o, assignment.
Table 9: Pass@1 1 metric evaluated when apply- ing different termination conditions on APPS dataset.
Table 10: Test RM scores and GPT-4 win rates when applying different termina- tion conditions on TL:DR and HH-RLHF datasets.
GPT-4 Win Rate (Na.. PPO)
Termination
Termination
Fixed 10-gram
Parsing
PPL
Fixed 5-gram
1.40
78%
TL:DR
Parsing
1.37
78%
Inter.
3.25
3,04
PPL
1,27
72%
pas@@
Intro.
16.56
17.05
16.36
Fixed 5-gram
1,55
58%
Comp.
0.94
1.24
0.80
HH-RLHP
Parsing
1.64
62%
All
5.45
5.56
5.26
D FURTHER ANALYSIS
D.1 VALUE FUNCTION ESTIMATION OF MACRO ACTION
When implementing the macro actions, the value function of macro actions is estimated through the value function of tokens. This process can be formulated as: V (,)
E ot. Ve-+VV V 1/"(st,4is (st, where o, ( ,t.tlwl control the contribution of each value function of tokens.
In this section, explore several assignments of a and their effectiveness MA-PPO. Figure 18 illustrates macro action value function with different a, assignments:
1. Equal assignment: We treats the contributions of each value function of tokens equally when considering the value function of macro actions, i.e., o, = { M This is the naive assign- ment in MA-PPO used in all our experiments.
Unit assignment Since a macro action a higher-level construct sequence actions, we can use the value function of the last action as the macro action's value function, where {0, ...
Position decayed assignment The contributions of each value function of tokens are determined by taking the position into consideration. We define a, based on the position of the token, i.e.,
o (1
|-l-1 where H
Σ
-
this construction ensures Eo [-)
We tested these approaches with fixed n-gram based termination on TL;DR dataset, with n 5. We report the RM score and GPT-4 score previous. Results in Figure 19 show that the equal assignment yields higher RM scores. However, the unit assignment achieves the best consistency and fluency according to GPT-4 evaluations.
22

---

Published as a conference paper at ICLR 2025
1.50
Vanilaa PPD
1.25
MA-PPD (Equai]
MA-PPD (Unii)
RM score
1.00
MA-PPD (Position)
0.75
GPT-4 Score
Vanilla PPO
0.25
MA-PPO (Equal)
MA-PPO (Unit)
0.00
MA-PPO (Position)
-0.25
reference
-0.00
1000
2000
4000
Training step
Relevance Coherence Consistency Fluency
Performance of MA-PPO with different value function estimations in MA-PPO TL;DR dataset for Gemma-2B model. Left test RM scores. Right GPT-4 scores on dimensions.
Figure 19:
D.2 TERMINATION CONDITIONS ON DIFFERENT TASKS
In this section, we analysis the effectiveness of termination conditions on TL:DR, HH-RLHF, and APPS datasets. When implementing parsing-based termination condition on APPS dataset, we use a programming-languagee
parser.* The results of TL;DR and HH-RLHF datasets are shown in Table 9 and Table 10. We can notice that parsing-based termination condition performs well on the HH-RLHF tasks, with higher RM score and win rate than fixed 5-gram based termination condition. While the TL:DR dataset, parsing-based termination condition also achieves excellent performance compared to fixed 5 5-gram termination condition. On APPS dataset, gausing-basee termination condition achieves the best results, except for the interview level task. These results demonstrate that construct macro action with linguistic information indeed brings performance gain MA-PPO.
D.3 IMPACT OF RLHF ON REWARD SCORE DISTRIBUTION
05
23
Figure 20: RM score shifting pattern after RLHF training. Left presents the RM score of best of 8 sampling on vanilla PPO compared to the vanilla PPO. Mid Left presents the RM score best of 8 sampling MA-PPO compared to the MA-PPO. Mid Right presents the RM score of MA-PPO (n = 5) compared to the vanilla PPO model. Right presents the RM scores of MA-PPO (n = oo) compared to the vanilla PPO model.
We apply Best-of-N sampling on both vanilla PPO and MA-PPO. The RM score shifting patterns for these methods are illustrated in Figure 20 (Left and Mid Left). From the results, we can conclude that ofst-of-N sampling continues to enhance the performance of RLHF models effectively.
In Figure 20 (Mid Right and Right), we compare the MA-PPO with vanilla PPO using settings of and oo, both of which demonstrate positive effects on the RM score distribution.
D.4 IMPACT OF SAMPLING TEMPERATURE
In the previous experiments, the results sampled with temperature temp 0.8 align with the sampling strategy used during training. In this section, we examine the effect of sampling
23

---

Published as a conference paper at ICLR 2025
2.50
1.75
275
Reward score
1.50
Reward SCO
232
0.75
0.30
1.52
0.22
1:2
O.R
1.0
102
0.8
1.0
Temperature (2B)
Temperature (7B)
Figure 21: Test reward scores evaluated by the corresponding reward model for summarizations generated with different sampling temperature the TL;DR dataset.
Token-Level RLHF
Macro Action RLHF
RL Optimization
Rewarda
Rawarrss
Vilaes
Actions
Mocro Action
Termination
leyl
Hoy Ma
Via
V(s,..n)))
R
GAI Function
Aisartage
detion Saie
Quaryy
Volues
Poley Hoee
Rroui Shaee
Values (FL
Qc
Query
V(Ne
Figure 22: Illustration of the macro action-RLHF (MA-RLHI) framework.
temperature on response quality. We vary the temperature temp E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0), and report the results in Figure 21. The performance of both methods remains stable when temp 0.8. However, the performance vanilla PPO begins to decline after temp = 0.8, whereas MA-PPO continues to demonstrate stable performance,
temp 1.0.
Algorithm 1: Framework of Macro Action RLHE
Input: Prompts: X {20, -1, Fn }: Policy model: ppolicy :Rfference model: Tref: Critic model: Teritiei Reward model: Tim Termination rule <(-) in Section 3.2.1; Value function estimation ot Section D.1.
Output: Policy loss CPPn, Critic loss Lvalue.
foreach prompt x; in X do
Make experience using policy model y := Toolicy (z); Get value V(*)): == Teritie (z, 8z) at every time step / E [0, lw]);
Get reward score current experience r = Trm (x. y);
Compute macro actions {w, ) based on the termination rule (w, }m = C(y);
foreach macro action W/r in ),m do Compute macro action value function
V (T w+) E.mo
(FL.
Obtain Å, and Q with GAE(V* w)),
Optimize _epo Ê min
na(welas) Te (w-1
Ar, clip( No(w-1or)
woald (w+ (w-18)) e, e)A
Optimize Evalue I8 [|V w/) Q-12
E MA-RLHE ALGORITHMM
Figure 22 illustrates the framework of MA-RLHE. In practice, to implement MA-RLHE, once the macro actions are obtained via the termination function, we compute their value (as estimated by the critic model) and rewards (based on a per-token KL penalty) using the value function estimation. With these values and rewards, apply Generalized Advantage Estimation (GAE) without modi- fication to derive advantage estimates and state-action value functions. These advantage estimates and state-action value functions are then used all tokens within the macro action during the opti-
24

---

Published as a conference paper at ICLR 2025
mization of both the policy and critic models. The macro action RLHF algorithm, utilizing PPO, is detailed in Algorithm 1.
In this implementation, the introduced additional time complexity is in the option termination. While fixed n-gram based, randomized n-gram based, and perplexity based terminations achieves same time complexity, the time complexity of parsing based termination is related the constituent tree which we applied DFS to obtain | During the inference stage, our MA-PPO will not introduce additional complexity since it only works the training stage.
We provide the Pytorch code for implementation the macro action in PPO below:
25

---

Published as a conference paper at ICLR 2025
Obtain Macro Action Positions
def get_ ctt.actooaciioooppeeiiiiiississo (sclf, start, mask, termination= ngram', n_gram: int=None, ppl: List [torch. float16] =None, repeat_times: int= None, cutoff: int=None):
sequence = [start]
if termination - 'ngram':
assert gram is not None
current current_count for i in range (maskl start: J.sze (1) - 1):
current_coumt +- mask[0, start + il .item)
if current_count n_gram:
sequence
current_coune
elif termination == = aaandomizze__nraatt
k <_iist torch. tenshreeecrrrcc (12, 3, 5, 10], dtype=int)
k_list torch. rocahhrrpeaa_iitteeeeee (k_iist, 3)
k_list k
(k_tdddpprrrm_tttstoeeee. wtt.:/isssssssi.. [-1])]
k torch. cwmsummmmmmm.... 1k_tist, dim=-1)
sequence [n for in range (start, mask[ start:).sien (e) 1)]
indexed k list (x.item() for x indexed list if x.item
< len (sequence)
sequence = [start] + [sequence for i in indexed k 1ist] elif termination
'ppl';
assert ppl not None
for i in range(1, (pp()))))
if [i][i] ppl[i 1]:
elif termination sequence.append 'parser': if len (endee. < 1:
i)
return False, ma_length +
if len (node aequence.appenddmaa_
cutoff:
node Leaves
True, ma_length + node. leaves
for nxt_node in node.childs
state, ma_length_ dfs (nxt_node, ma_length) if !state:
sequence[-1] ma length return True, ma_length eequence.appende (int (mask.sizo 1 return sequence
ma length ma _eagthh
Calculate Values Rewards Macro Action
def get action vetion_valueslleeeeeeee
values (self, values, mask, start, sequence)
split list torch. (torch.tenson tensor (sequence) .tolist splited_values torch, split (values [:, start:l, split_list, dim=-1) splited_mask = forch.splic (mask [;, start:l, split_list, dim=-1)
inplace_values
len (splii__list) dtype-values dtype
(voolvalu device)
for idx, (value_i, mask_i) in enumerate seaa splited_mask))
lasked_valuse value_i [mask_i != 0] Inplace_valuen [0, idx] torch.men march.mean values) if vasked_vaaues.nee return inplace_values
else 0.0
26

---

Published as a conference paper at ICLR 2025
Calculate Policy Model Loss
def policy_lool
seall_oos__ma__ macroo_attiienssll. logprobs, olg_rogp, advantages, mask, sequence):
leg_ratio (logprobs cod_roopo mask ratic = torch.ex; rlogiratio)
calculate loss with macro action list torch.diff tenso (sequence)) .toliit tollist O
split_ ratio = torch. torchssplittratioo, spiit_ list, dim=-1)
split_mask torch.split (mask split_list, dim=-1)
pg_loss 0.0
total mask sum 0.0
for i in range (len =
ratio_i = mask_i split_mask[i]
advantages_i advantagesli, i]
Pq_lossl - -advantages_i ratio_i
1o_1 i torch (aapchcclammppati___ 1.0 set f.
cl cliprange,
self.cliprange)
pg loss += torch. (terch.erm)) materch.maxxee lossi, pg pg_ lo__) mask total mask sum += mask sum
pg loss = loss total mask sum return pg_loss
Calculate Critic Model Loss
def critiic_los__merr__coo larss__aro__cttioo (self, values, old_values, returns, mask, sequence):
values_clipped
values clipped = torch.clanp.
values,
old values self.cliprange value,
old_values self.cliiprange_aalee,
if self. htt.fccoppte__332222 values values.. float()
values_clipper
calculate loss with macro action split_list barch.diff ttosrr (sequence) ) tolist ()
splited_values torch split (values, split_list, dim=-1)
splited_values__tt pppppd torch. split (values_cll ippd split_lists dim=-1)
splited_mask - torch. it (mask, pplit_ist, dim--1)
total loss 0.0
total mask sum 0.0
for i in range (len (sl[teelled_
vf_loss1 (splited_values[1) returnsl;, +*2
vf_loss2 [=pitted_vaauus_clipeee]]] returns [:, i]) x.2
vf_loss 0.5 torch,
torch.mar (vf_lcssi vf_loss2) splited_mask [i])
tetal_vf_loss += vf_loss
total_mask_snt
splited_mast mask [i] sum()
total_vf_loss total_vi_lose total sum.
return total_vf vf loss
27

---

Published as a conference paper at ICLR 2025
PPO
In PPO algorithm
start prompts. size [-1]
acticn_man attention_mask 1:]
sequence - jet_merr_aationneettiiisttttttr action_mask, terminattoon
agram' , a_gramen_gramm
macro macro_action
get_nacro_ectioovvvaluss (old
action, action_mask, mask, start, sequence)
pacro_action_old_reaa
get_ ( (
action_mask, start, sequence)
advantages, returns ge (11_
policy_ loisy_losss__ = policy_loss_aaeeeepppppplllslssp less_osss______ mac_maacr__aa cetteacciionnn (poliyyyyy_ probl____pbpppcooooooo__ start:l,
log_probsl;, start:l, advantages, action_maskli, start:), sequence)
critic_loss
aaaaaiiooooavleiaaiiii (value start:l, (i
start:l, returns, action (;, start:l, sequence)
F EVALUATION DETAILS
F.1 GPT-4 EVALUATION PROMPTS
In our experiments, we take GPT-4 as main judgment of the quality of policy models. The prompts used generate win rates using GPT-4 are listed below. We utilize the gpt-4o-055 13-13 for all of our experiments. The order of the responses generated by policy models is randomly chosen for all experiments.
TL;DR GPT-4 Evaluation Prompt
You will be given two summaries written for an article. Your task is to pick the better one between them, based on the four criteria. Please make sure you read and understand these instructions carefully. Relevance selection of important content from the source. The summary should include only impor- tant information from the source document. Annotators contained redundancies and excess information.
instructed penalize summaries which
Coherence the collective quality of all sentences. We align this dimension with the DUC quality ques- tion of structure and coherence whereby "the summary should be well-structured and well-organized. The summary should not just be heap of related information, but should build from sentence to coherent body of information about a topic."
Consistency the factual alignment between the summary and the summarized source. factually consistent summary contains only statements that are entailed by the source document. Annotators were also asked to penalize summaries that contained hallucinated facts.
Fluency the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
You should output single character to indicate which summary you think is better. 'A' stands for Summary A and 'B' stands for Summary B. If you think both summaries are equally good, output 'E'. Article Post: aarticle / }
Summary A: summary a)
Summary B: sunma ry b)
Your Choice (only a single character):
28

---

Published as a conference paper at ICLR 2025
HH-RLHF GPT-4 Evaluation Prompt
For the following query chathot assistant, which response First provide a one-sentence comparison of the two responses and explain which you more helpful. Second, on a new line, state only 'A' or 'B' to indicate which response is more helpful. If they are equally good or bad, state 'E". Your response should use the json format, with "comparison" and "choice" keys.
helpful?
Query: {query)
Response A: (response a
Response B: (response b} Your Judgment:
WebGPT Comparisons GPT-4 Evaluation Prompt
You will be given two response written for an question. Your task is to pick the better one between them, based on these criteria.
Factual accuracy which answer is more factually accurate?
Coherence which answer casier follow? Usefulness overall all things considered, which answer would more helpful to the person who asked this question?
You should output with json format where the key is the criteria and the value is the choice you made, using 'A' stands for Response A and 'B' stands for Response B. If you think both responses equally good. output 'E'.
Question: [question]
Answer A: (answer a)
Answer B: :answer b)
Your Judgment (you should also output the reason, that you are allowed think both responses
are equally good, then output with 'E'):
F.2 HUMAN EVALUATION
To estimate the quality from human perspective, we collect human preference data on the TL:DR, HH-RLHF, and WebGPT datasets. Human annotators select the preferred response based on task- specific criteria. For TL:DR, the evaluation criteria focus on three main perspectives:
this considers whether the generated summary includes any additional informa- tion not present in the original post article.
Hallucination:
Verbosity: this assesses if the summary includes unnecessary context that could be removed without negatively impacting its quality.
3. Overall Quality: this measures the general coherence, informativeness, and readability of the generated summary.
For evaluation on TL;DR dataset, the annotators should first compare the overall quality of two responses. If overall qualities are equally good for responses, then they should choose the winner based on hallucination and verbosity.
In the context of HH-RLHE, annotators focus on the helpfulness of the responses:
1. Instruction Following: whether the generated response follows the requirements in the instruc- tion
Usefulness: whether the advices in the response are applicable, and does the response ideally guide the user on what to do next.
Annotators are instructed to choose the response based these aspects, while excluding superfi- cial replies such as "You're welcome." For the WebGPT dataset, the primary evaluation factor is factual accuracy. Annotators are provided with retrieval information relevant to the question from the dataset to aid their judgment. They are tasked with selecting the answer that most accurately matches the retrieved information.
During the evaluation process, annotators are presented with a prompt and two responses, each generated by either vanilla PPO MA-PPO. To ensure impartiality and prevent annotators from
29

---

Published as a conference paper at ICLR 2025
guessing which model produced which response, shuffle the positions of the responses. Anno- tators are given three choices: response wins, response wins, or a tie. The results are then collected to calculate the win rates for each model.
For evaluations on the TL;DR and HH-RLHF datasets using 7B models, we conduct the human evaluation with 3 different annotators and collect their preference data report the win rates. For all other human evaluations, we conduct them with a single annotator. The inter-rater agreement achieves average of 68% total 100 samples. On the TL;DR dataset the agreement 64%, and on the HH-RLHF dataset the agreement is 72% across 50 samples per task.
G GENERATED EXAMPLES
G.1 CASE STUDY
When evaluating the responses of MA-RLHF with human annotators, we observe that the MA- RLHF exhibits coherence and contextual appropriate abilities. We illustrate this phenomenon with an example by comparing MA-RLHF with the baseline Table 11. We found that the MA-PPO method tends to generate responses with phrases more than the baseline method. Specifically, for nouns, it commonly includes adjectives for modification to make the generated summaries more accurate, such as "feeding indoor cat food".
G.2 EXAMPLE RESPONSES
In this section, we demonstrate some examples of validation sets to highlight the superiority of MA-PPO. In Table 12, we feature examples from the TL:DR dataset. Compared the responses generated by vanilla PPO, the responses from MA-PPO offer more concise and relevant details about the situation. Table showcases dialogue focusing on the differences between Spanish and Italian. MA-PPO provides more helpful response, addressing both cultural and grammatical perspectives. Table 14, we compare answers from the WebGPT dataset. The responses provided by MA-PPO more factual and accurate compared to those generated by vanilla PPO.
30

---

Published as a conference paper at ICLR 2025
Table 11: Example response from TL:DR dataset to illustrate the responses generated by MA-RLHF is more coherence and contextual appropriate compared to baseline. We highlight the context that we observed with green
Prompt
POST Subreddit: r/cats
Hello everyone! One of my cats is about 10 years old now, she is pretty much strictly indvors save for some time she spends on our screened in porch each day. (She likes to watch the birds in the yard while she suns herself by the pool, quite the princess), Anyway, when she was younger she was very active and quite small, however with age she has put on pretty hefty amount of weight. feed her indoor cat food for weight control, I've switched brands few times trying to find something that works, I've cut hack feeding her by lot (she gets very angry and demanding when she wants food but I don't give in) however, nothing really seems to work. I've tried loys, and bought harness thinking I could try walk her but she just lays down and looks at me like I'm stupid. Basically I just want to know you all have any suggestions for exercise food. care about her and don't want this get any worse. I also have another cat that the same amount and type of food her and is completely normal weight only a year younger. however he is a male, not sure that makes a difference in predisposition for weight gain. They are also both fixed.
Vanilla PPO
10 year old cat has gained noticeable amount of weight and is becoming increas- ingly inactive. Looking for suggestions for exercise or food help her get healthier. Male and female cats are both affected.
Fixed 3-gram
10 year old cat put on hefty amount of weight despite feeding indoor cat food for weight control. Any suggestions for exercise or food to help with weight control and prevent further weight gain? Specifically wondering if there are any suggestions for cats that are hoth female and fixed.
Fixed 5-gram
10 year old indoor cat has put on significant amount of weight despite being fed low calorie/weight control food. Need suggestions for exercise or food to help with weight management. Also have another cat that eats same type/amount of fond as her and completely normal weight and only a year younger, however he is male. Any insights would be appreciated.
Fixed 10-gram
10 year old cat despite feeding indour cat food for weight control Any suggestions for ex- ercise or food to help her lose weight and keep her healthy? Also, is there a difference in predisposition for weight gain between male and female cats when fed same amount/type food?
putting on significant weight
Fixed oc-gram
One 10 year old cat has put on hefty amount of weight with age. feed induor cat food for weight control, tried various brands and back feeding by lot, nothing reully seems to work in terms of exercise food, also have another cat same age and weight as her and com- pletely normal weight and no issues currently however male and not sure that makes difference predisposition for weight gain both fixed.
Randomized n-gram
10 year old cat has put on a lot of weight, feed her indoor cat food for weight control yet nothing scems to work. Any suggestions for exercise or food address this? Other cat that cats same amount and type of food as her is normal weight and year younger, however is male, not sure if that makes difference in predisposition for weight gain.
Perplexity based
amount of weight despite diet change and other attempts at exercise/activity. Any suggestions for exereise food to help her lose weight? Also, any differences hetween males and females regarding predisposition for weight gain?
10 yo cal has put
significant
Parsing based
10 year old indour cat has gained significant amount of weight despite proper diet and exercise potentially due to age reluted health issues Need suggestions for exercise or food changes to help reduce weight and maintain proper body weight. Is weight gain common older cats, and does gender play factor in predisposition
31

---

Published as a conference paper at ICLR 2025
Table 12: Example responses from TL;DR dataset. The summarization of MA-PPO exactly capture the details of the post.
Prompt
POST Subreddit: r/rumning
Last night experienced my first ever absolutely miscrable run where in- stead of feeling energized and accomplished when it was over, lay on my bathroom floor shivering and hugging the toilet.
I have had frustrating runs before and have had runs when my knee/an- kle/toe hurt and made me want to quit but last night was far worse than any of those.
Earlier in the day. ate sandwich and fell asleep on the couch for a two hour surprise nap. When woke up my stomach was slightly unsettled like I was about to have heartburn. decided to do out for my long run anyways because always have heartburn and doesn't 't usually stop me. This run was scheduled to be my longest yet at 1:30:00 (I am training for half). At about 45 minutes I started to get stomach cramps so I slowed how much water I was taking in. The entire run I had walk several times which is unusual for me but my times were better than expected so thought was just pushing too hard and tried to take it easy the way back.
My post long run ritual is usually to drink several long drinks once I get back to the car since now won't be jostling my innards around, can do what I want. When I got home, I felt like I probably shouldn't try cating right away so I had about 1/3 bottle of Gatorade.
Well..GGtoorade was the wrong answer. Within 20 minutes I was violently heaving into my kitchen garbage and lost all of the Gatorade and likely most of the water had while running.
My question is, do blame my sandwich from earlier or my run? I am worried that 1:30:00 is my limit. haven't considered quitting my half marathon training until now because everything up to this point has been fine.
Vanilla PPO 2B
1:30:00 run extremely stressful and painful. Do I blame sandwich from earlier or run itself? am worried that 1:30:00 is my limit.
RM Score: 0.75
MA-PPO 2B
1:30:00 long run resulted in severe stomach cramps and forced heavy heav- ing into kitchen garbage. Do blame my sandwich from earlier or my run? Concerned that 1:30:00 is my limit.
RM Score: 1.55
Vanilla PPO 7B
1:30:00 run caused serious stomach pains and I puked up all of the water I drank while running. Do I blame the sandwich for my stomach pain or the run itself?
RM Score: 1.71
MA-PPO 7B
1:30:00 long run, had stomach cramps and had to walk often, drank Gatorade post run, vomited violently within 20 minutes, now worried that 1:30 is my limit, do blame sandwich or run? Worried about quitting half marathon training. Help?
RM Score: 2.61
Vanilla PPO 27B
45 minutes into my run my stomach started to cramp and was vomiting profusely within 20 minutes of finishing my 1:30:00 long run. Do I blame my sandwich from earlier or my run? Does this mean my limit is 1:30:00 or is it something else?
RM Score: 3.45
MA-PPO 27B
I went out for 1:30:00 long yesterday and came home with stomach cramps and nausea. I drank 1/3 bottle of Gatorade when I got home and got violently sick. lost that and most of the water I drank while running. Do I blame my sandwich from earlier or my run? I am worried that 1:30:00 is my limit and that I won't 't be able finish my half marathon training.
RM Score: 3.95
32

---

Published as a conference paper at ICLR 2025
Table 13:
Example responses from HH-RLHF dataset, MA-PPO method presents more helpful response compared to vanilla PPO, and is prefered by GPT-4 and human.
Prompt
Human: Would you be able to explain the differences between the Spanish and Italian language? Assistant: Of course. Can you tell me more about the specific areas where you're re interested knowing more? Human: I'm thinking between the Spanish spoken in Mexico and Italian spoken in Italy. Assistant:
Vanilla PPO 2B R.M Score: 1.07
The main difference between Spanish and Italian is that Spanish is an Indo- European language with very rich historical background. while Italian is an Italian Romance language derived from Latin. Spanish has influences from many different languages, while Italian has fewer and more limited influences from other languages.
MA-PPO 2B
Here are some of the main differences between Spanish spoken in Mexico and Italian spoken in Italy: 1. Spanish spoken in Mexico tends to be more relaxed and casual, while Italian spoken in Italy tends to be more formal and courteous. 2. Spanish Mexico heavily influenced by indigenous languages and has a rich vocabulary, while Italian is mostly derived from Latin and has a more simple and elegant sound. 3. Spanish spoken throughout Latin America, while Italian is confined to the Italian peninsula. Overall, Spanish and Italian have many subtle differences that contribute to their unique cultural identities.
RM Score: 1.24
Vanilla PPO 7B R.M Score: 0.80
The main differences between the Spanish spoken in Mexico and the Italian spoken in Italy are in the pronunciation and grammar of the two languages. In Mexico, the Spanish spoken is much more similar to the Spanish spoken in Spain. In Italy, the Italian spoken is similar to the Italian spoken in other parts of Europe, but with its own unique style of pronunciation. In both languages, the grammar is very similar, though the Italian language does have some unique grammar rules.
MA-PPO 7B
Sure, here are some of the main differences between the Spanish spoken in Mexico and the Italian spoken in Italy: Spanish is a latin language, which a language that was derived from the latin language of ancient Rome. Italian is based on the ancient language of the romans of ancient Rome. Spanish is spoken in a number of different countries throughout the world, whereas Italian is the native language of Italy, and is only spoken in some parts of northern Italy. Spanish is spoken primarily in Latin America, and has taken many local variations and differences due to the influence of local indigenous languages (i.e. Mexican Spanish is quite different from Spanish spoken in Argentina). Italian primarily spoken in Italy and has more localized variations. There are many differences in grammar, vo- cabulary, and pronunciation between Spanish and Italian that are reflected in the differences between Mcxican Spanish and Italian. Some of the most important differences in grammar include the use of masculine and femi- nine gender to refer to objects, and the different use of the tenses. There are many differences in vocabulary that also exist between the two languages, and the pronunciation of each language also varies.
R.M Score: 1.22
33

---

Published as a conference paper at ICLR 2025
Table 14: Example responses from WebGPT dataset.
Prompt
What causes that sudden sinking, heavy, hot flush feeling when you become anxious and how are the effects felt so rapidly?
Vanilla PPO 2B
Anxiety has been linked to several physical symptoms, including a rapid heartbeat, trembling, dizziness, and sense of impending doom. While some of these symp- toms may be due to the anxiety-provode may be the result of an anxiety disorder itself. The symptoms that result from anx- iety can be divided into physical symptoms, emotional symptoms, and behavioral symptoms. Physical symptoms of anxiety include nausea, dizziness, and queasy or empty feeling in the stomach. Emotional symptoms include feeling helpless, hopeless, afraid, irritahle, and worried, Behavioral symptoms include avoiding so- cial interactions, remaining in bed all day, and not performing daily activitics.
RM Score: -0.68
situation causing the symptoms, some
MA-PPO 2B
The "hot flush" sensation that some people experience when they become anxious is the result the nervous system's 's fight-or-fliht response. When the body is faced with a stressful or dangerous situation, the nervous system's adrenaline re- lease causes blood to rush through the cardiovascular system to the muscles. This causes the body to become more efficient using oxygen and energy. and also cause the skin to turn red and hot. This response designed help the body deal with stress and danger, and can include range of symptoms that impact different parts of the body. For example, elevated blood pressure can cause the kid- neys to release more urine, and adrenaline speed up the heart rate. It can also cause the mouth to dry addition, anxiety can cause the chest tighten and breathing to become quicker and shallower, which can cause the heart rate to increase. It can also cause the digestive system to slow down, and the muscles of the gastrointestinal tract to tighten, which can cause nausea and abdominal cramping.
RM Score: -0.41
the nervous system reacts to the perceived threat. In
Vanilla PPO 7B RM Score: -0.41
There are several possible explanations as to why we feel hot and uncomfortable when anxious. One explanation is that anxicty causes the body to relcase hormones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. different explanation is that a reaction to stress causes the body to release hormones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. Another explanation is that the body's 's stress response causes the release of hor- mones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. fourth explanation is that anxiety triggers the release of cortisol, which increases blood sugar levels and causes the body to burn fat and muscle tissue for energy.
MA-PPO 7B
When we become anxious, is because our brain has encountered something that it sees as a threat. The brain then sends out a warning signal, which takes the form of a fight-or-fliht response. This response causes a number of physical effects, including increased heart rate and breathing rate, increased blood flow to muscles, and elevated blood sugar. This is because when are threatened our body needs to able to run or fight, and to do this we need increased oxygen and energy. Our body also needs to be able to protect itself, so we have increased blood flow to our muscles. When we are in high-stress situation, our body produces scrotonin and dopamine, which help time, our body produces norepinephrine and cortisol. Norepinephrine and cortisol work together to increase our heart rate, breath rate, and hlood pressure, which in turn increases the flow of oxygen and nutrients to our muscles. Thus, when we are anxious, our body' way of protecting us is by increasing our awareness and increasing the flow of nutrients and oxygen to our muscles, so can run or fight if needed. All of these processes occur so quickly because our body" systems are already set up to respond instantly to threats.
RM Score: -0.39
feel good and focus on the task hand. At the same
34


Reviews:
[{"review_id": "mXClcEhg8m", "reviewer": "Reviewer_oNFY", "summary": "The current manuscript proposes to tackle the credit assignment problem when aligning large language models (LLMs) to human preferences via reinforcement learning from human feedback (RLHF). The core idea is to consider a group of tokens as a macro action, instead of individual actions, and use rewards accumulated within a macro action with proximal policy optimization (PPO) to align the language model. This approach is labeled MA-RLHF. Experiments on three tasks (text summarization, dialog generation, question answering, and program synthesis) show improvements of MA-RLHF over PPO for a chosen LLM (Gemma) both in terms of performance and training speed.", "questions": "Typos:\nL68: leads -> lead", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(S1) The direction of the current work to use macro actions (or a group of low-level actions together) is novel and interesting, for the setting of aligning a language model with human preferences (RLHF scope). Please see W1 for more discussion.\n\n(S2) The manuscript is well-written, with sufficient details on all the technical details making it easy to understand and comprehend. I thoroughly enjoyed reading the paper.\n\n(S3) The ablation and generalization studies presented in the experiment section (Sec.4.3 and 4.4) were interesting to better understand the behavior of the proposed approach in the context of RLHF.", "weaknesses": "(W1) The idea of gathering multiple actions together as a “macro-action” is not novel in general. Referred to as “action chunking” [A], similar ideas have been explored in other domains. As noted in (S1), this is still useful in the context of RLHF, but with a diminished novelty factor.\n\n(W2) The current work mostly experiments with a single LLM (Gemma and its variants). Given (W1) and lack of generalizability across different models, the usefulness of the current approach over PPO is not clearly understood to benefit the community. Demonstrating similar benefits in an LLM-agnostic manner will tremendously be useful.\n\n(W3) Comparisons to the more recent approaches like DPO [B] are missing in the experimental tables. Though there is benefit over PPO for the chosen model, the experiments fall short to gauge the improvements compared to one of the current arts. Request the authors to include this comparison to better understand the proposed approach and its efficacy. \n\nReferences\n* [A] Action chunking as conditional policy compression. https://osf.io/preprints/psyarxiv/z8yrv\n* [B] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.\nhttps://osf.io/preprints/psyarxiv/z8yrv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xAiWVrFo4f", "reviewer": "Reviewer_YfQu", "summary": "They conduct empirical analysis of the use of macro actions in RLHF instead of the standard per-token actions. They study a variety of approaches to breaking responses into macro actions, finding that N-gram based macro actions tend to perform best. In general, they find macro actions have the potential to outperform standard PPO with per token actions. They attribute this to better training stability and reduced variance in the learning problem.", "questions": "See weaknesses for questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* They conduct thorough analysis on many different tasks, using several different sized base models.\n* They conduct many ablations of their method.\n* The paper is clearly written and easy to follow.\n* This empirical analysis of macro actions present a useful contribution to our understanding of RLHF training.", "weaknesses": "* Some of the differences in performance could be due to tuning one method more than another. It would be great if the paper could at least document the steps they went through to tune hyper-parameters for their method and baselines.\n* My understanding is that the primary functional difference between per-token PPO and macro action PPO is the granularity of the value function, importance sampling, and discount factor. It is a fairly small modification, but this is not necessarily clear when reading the paper and could be better emphasized.\n* The GRPO method proposed in https://arxiv.org/pdf/2402.03300, can be viewed as a special case of their macro action PPO. They don't discuss this in the related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9EXi4v1qXe", "reviewer": "Reviewer_85YK", "summary": "This paper introduces MA-RLHF, a novel framework that incorporates macro actions (sequences of tokens or higher-level language constructs) into the RLHF training process for LLMs. The key innovation is operating at a higher level of abstraction by grouping tokens into meaningful units, which helps address the credit assignment problem in long sequences. The authors demonstrate that this approach leads to faster convergence (1.7x to 2x) and better performance across various tasks including text summarization, dialogue generation, question answering, and program synthesis, without increasing computational complexity during training or inference.", "questions": "* How sensitive is the method to the choice of macro action size? Is the macro action sizes sensitive to different types of tasks - i’m particularly interested in difference between different categories of tasks, for example, difference between natural language (TLDR, HH-RLHF done in the paper) and coding (APPS). In figure 6, it seems the difference between different n choice is not too large on HH-RLHF, wondering if that would change for coding tasks.\n\n* Will be nice to see some qualitative examples of the macro-actions (e.g., how tokens are grouped together in a concrete setence).\n\n* What is the difference & commonalities between this paper and ArCHer [1]?\n[1] ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a critical limitation in token-level RLHF - the credit assignment problem over long sequences - and proposes a simple yet effctive solution by incorporating macro actions. The approach is well-motivated by both theoretical considerations (credit assignment, temporal abstraction) and practical issues (subword tokenization challenges).\n\n- The paper is well-written and easy to follow.\n\n- The experimental evaluation is thorough, covering multiple tasks, model sizes (2B to 27B parameters), and evaluation metrics. The authors also conducted detailed ablation studies on different strategy of termination and find n-gram performs the best, analyzed the impact of macro action size, and did best-of-N to further validate the effectiveness of the proposed approach.\n\n- The proposed method is simple (maintaining the same computational complexity) yet effectively improve performance (up to 30% in text summarization and code generation). The faster convergence and better performance make this a valuable contribution to practical applications.", "weaknesses": "- While the paper explores different termination conditions for macro actions (n-gram based, perplexity based, parsing based), there could be more analysis of how different types of macro actions affect different types of tasks. For example, which macro action strategies work best for which types of generation tasks? For example, i would expect parsing-based termination might also work well on code generation tasks if a programming-language-based parser was used.\n\n- Lack details of Human Evaluation: The evaluation involves human evaluation of model responses. It would be nice to show the detailed protocol of how human evaluation is performed, and what is the inter-rater agreement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zjecMXwAYv", "reviewer": "Reviewer_QYzk", "summary": "The main idea is to use a larger chunk of text as the action in RLHF, instead of just one token (e.g., one word or one sub-word unit). What’s a larger chunk? The authors have tried multiple strategies: n-gram-based, parsing-based, perplexity-based “termination” of macro actions (i.e., segmenting tokens into a macro-action) – those are simple heuristics-based strategies introduced on page 4. Algorithms are built on top of naive PPO. \n\nExperiments are done using Gemma-2B, Gemma-7B, and Gemma-2-27B. Results on multiple tasks (summarization, dialogue or instruction following including HH-RLHF and WebGPT, code generation) show improvements.", "questions": "A natural way is based on BPE-like automatic construction of a dictionary with most frequent n-grams. Have the authors considered this strategy? \n\nDo the authors think that leave-one-out REINFORCE or other recent PPO variants would also benefit from the authors’ algorithm (macro-actions)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The ideas are intuitive. Recent progress has shown that RL is promising in general so it’s great that the authors are investigating this direction. \n\nMultiple macro action termination strategies are explored. Experiments on code generation / program synthesis is helpful.\n\nInteresting analysis on rejection sampling vs. temperature.\n\nHelpful figures & illustrations in the appendix.", "weaknesses": "Results are great and surprising (not weaknesses); e.g., in Figure 4, the newly trained models are strongly preferred over the baseline (vanilla PPO). But:\n- What’s the main reason here? Is there a comprehensive analysis that makes sure that the win rate is not based on simple features like length? \n- Are baselines trained with a sufficient number of steps? \n\n\nThe equation for R is incorrect in Section 2.2.\n- KL is between two distributions, but the authors wrote that the KL is between two real numbers.\n- The last term should be -KL( \\pi_\\theta ( \\cdot | x) || \\pi_\\text{ref} (\\cdot | x)); in particular, KL(p || q) does not equal to -KL(q || p). Currently if you expand the KL, the interpretation is that if \\pi_sft (y|x) is large, then we should make \\pi_\\theta (y|x) really small, in order to maximize R. \n\nOther issues in Section 2\n- Can gamma be equal to 1? (Section 2.1)\n- In Eq. (1) it’s unclear if A_t is computed using \\theta or \\theta_old – it’s never defined; similarly, how about Section 3 line 233 – what model would the authors use to compute the advantage?\n- In Eq. (1) what’s the expectation over? \n- At the end of page 2, the indices start from 0 (e.g., s_0, a_0); but on page 3 line 145, the indices start from 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xajwGMn38i", "reviewer": "Reviewer_ziZM", "summary": "The paper proposes the incorporation of options (in the RL sense) into the RL component of LLM fine tuning with RLHF, with the motivation being better temporal abstractions over generation of long token sequences. The method is evaluated on several benchmarks and model sizes and is shown to overall outperform RLHF by vanilla PPO.", "questions": "Somewhat more conceptually, the motivation about \"essential linguistic structures and dependencies between adjacent tokens\" (Line 54) is questionable. While the intuition is reasonable, in fact the very same argumentation can be made against the pre-training stage, where LLMs are trained to make single token predictions. But pre-trained LLMs already generate fluent/coherent output (surprisingly or not). It is a strong claim that making (one of) the fine-tuning step(s) more \"global\" can correct for the extensive pre-training. The power in RL methods might be the ability to design global reward/objectives (see also Ranzato et al. 2015, Choshen et al. 2019) which are better suited for aligning to designer preferences, but this is still true even if the model has to make only \"micro\" decisions. I think the paper can benefit from at least acknowledging these questions.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is (as mentioned by the authors) simple and could be of potential interest to practitioners.\nThere is also an extensive evaluation of the method both in terms of tasks/datasets and in terms of method (GPT-based and human annotators)", "weaknesses": "There are some significant conceptual weaknesses, as well as many technical issues.\n\nMost importantly, there is a considerable gap between the stated/declared motivation and goals and what is actually being implemented. \nThe MA-RLHF algorithm only learns *one* option that is being applied everywhere -- there isn't a set of different options being learned, and there is no \"macro\" policy that chooses between them. As a result, generation really is driven by the (single) low-level policy alone, contrary to the authors statement that \n- \"Macro actions leverage meaningful linguistic structures\" (Line 66; which is clearly false for several of the \"termination\" strategies later used such as n-grams!)\n- \"MA-RLHF operates at the macro action level, making decisions over sequences of tokens at a coarser temporal scale\" (line 176-177).\n- And so on.\n\nWhat remains, effectively, is a particular way to chunk the rewards (and Advantage / Value estimates) that feed into the RL algorithm, in a way that might be different than the standard one. It could potentially be the case that this pushes the value estimation component more towards Monte-Carlo (rather than TD-learning / bootstrap) because Q (and A) are now fitted to a Bellman backup update that first sums over $\\vert \\omega_{\\tau}\\rvert$ observed rewards before bootstrapping (See $R_\\tau$ definition; Line 218. This is also reflected in the authors analysis of Section 4.3.2, line 374, see more on that later). If this is the source of the performance gain, it can be implemented and evaluated much more directly, and the story about temporal abstractions is merely a distraction.\n\nIf the authors have a a good reason / explanation for this particular choice, it should be stated. And in any case, this limitation (or choice) must be clearly stated and acknowledged, preferably early on. Right now it's not even mentioned in the Limitations section (which is unjustifiably hidden in the appendix, Line 810).\n\nOther than this main issue, there are plenty of technical issues, to a varying degree of severity. I expect some/most of them can be addressed in revisions. I have divided those into two sets, but I suspect these are not-exhaustive: the paper can definitely benefit from a closer proof-reading.\n\n**\"Moderate\" Issues**\n- The (Bengio et al. 2013) cite for the credit assignment problem (Abstract line 14, Intro Line 46) is out of place. That paper doesn't really deal with the RL credit assignment problem at all (although the term is mentioned, once, in the text).\n  \n- The comment \"setting n → ∞ corresponds to the REINFORCE algorithm where the entire sequence is treated as a single macro action\" (Line 399 and also Line 243) is completely unclear to me. REINFORCE most certainly **does not** treat the entire sequence as one action -- the policy is trained to generate standard \"micro\" actions at each time-step individually. The authors *might* mean to state something inline with the TD-based vs Monte-Carlo based method for estimating the Value/return, with PPO being an Actor-Critic variant and vanilla REINFORCE being a fully Monte-Carlo method. But this difference concerns the \"critic\" component rather than the actor. Moreover, the  (McGovern and Sutton 1998) cite for REINFORCE is wrong or at best confusing. (for REINFORCE the authors should cite Williams 92, or Sutton et al. 1999 if the claim is more generally on PG methods).\n\n- What do we learn from Figure 3 that's not already shown in Figure 2?  There's nothing particularly interesting about the shape of the shown distributions besides the mean and variance, which are already presented in Fig. 2. If anything, other than showing the marginal histograms alone, there could be potential value in showing the correlation over individual examples and see if there's any structure in that (is the performance gain come from more easier/harder examples, or more uniform, etc). Also, if that figure is kept, it should be specified for which of the models and at which step at training the histograms are shown (judging by the scale I assume this is the 2B model at the end of training?).\n  \n- The authors claim that \"MA-PPO achieves parity with vanilla PPO approximately 1.7 – 2 times faster during trainin\" (Line 303). However the main difference in Figure 2 seems to be the final performance level which is higher for MA-PPO compared to PPO, and not the learning / convergence speed which is really comparable. This should either be made more quantitative / carefully evaluated, or less boldly stated.\n\n- Figure 10: how come the marginal distribution of SFT RM scores changed between the first two panels and the last two panels? \n\n**Minor issues**\nThere are, once again, many issues with phrasings, notation, etc.\n\nLine 92: \"An policy\" -> \"A policy\"\nLine 93: \"The Markov Decision Processes\" -> \"A Markov decision process\"\nLine 98: The definition of Q(s,a) should include a conditioning on the fact that s_0 =s, a_0=a. It would also be better to write Q(s,a) rather than Q(s_t, a_t).\nLine 98: The authors specify that reward is a function r(s,a) but then in defining Q use the formulation of r(s).\n\nLine 108: The PPO discussion is rather dense. I feel that for a reader unfamiliar with the method, it will not be too helpful, and for familiar reader it will be redundant. Can be replaced by a reference to the appropriate paper, and shortened to a brief discussion of what Policy Gradient does (roughly, the inline Eq. in line 110), to be merged with the general RL formulation.\n\nLine 111: Randomness of *the* initial state and so on.\nLine 128: The acronyms for SFT and RM should be spelled out explicitly\n\nLine 130: There's no use in writing r(x,y) if the reader don't know what x,y stand for. Better write \"A reward model is trained to predict ...\", then explain what the inputs are and what the goal is.\n\nLine 140: Shouldn't the the equation read r(x,y) - beta KL? The way now written, the KL is not a penalty at all but a \"bonus\" to the reward (so higher KL will be encouraged)\n\nFigure 1: I don't understand what we should read from the figure other than the fact that temporal abstractions are \"a thing\". And it is overall somewhat misleading, since it seems to imply that the optimized policy chooses macro actions (the MA-RLHF column, arrow from the MA-PPO network to the omega_tau action).\n\nLine 145: Doesn't the state include the prompt as well? (other than the tokens already generated)\n\nLine 236: The use of $r$ to denote the likelihood ratio or pi_theta/pi_theta_old is a really bad choice, because $r$ is reserved to denote the reward function. Is there actually a need to introduce more notation for the ratio? (note that in Equation 1, line 119, this notation is not being used).\n\nFigure 2: Move the \"(2B)\" and \"(7B)\" to the ylabel (\"RM Score (2B)\"), or somewhere else -- otherwise it is confusing whether this 2B indicate scale of the time axis.\n\nFigure 4: The fonts are tiny (in all figures really, but particularly here, only readable at 200% zoom or so).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)