MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

## Abstract

Abstract Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences.
However, token-level RLHF suffers from the credit assignment problem over long sequences (Bengio et al., 2013 ) , where delayed rewards make it challenging for the model to discern which actions contributed to successful outcomes. This hinders learning efficiency and slows convergence (Mann & Mannor, 2014 ; Machado et al., 2023 ) .
In this paper, we propose MA-RLHF, a simple yet effective RLHF framework that incorporates macro actions — sequences of tokens or higher-level language constructs—into the learning process.
By operating at this higher level of abstraction, our approach reduces the temporal distance between actions and rewards, facilitating faster and more accurate credit assignment. This results in more stable policy gradient estimates and enhances learning efficiency within each episode, all without increasing computational complexity during training or inference.
We validate our approach through extensive experiments across various model sizes and tasks, including text summarization, dialogue generation, question answering, and program synthesis. Our method achieves substantial performance improvements over standard RLHF, with performance gains of up to 30% in text summarization and code generation, 18% in dialogue, and 8% in question answering tasks.
Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in terms of training time and continues to outperform it with further training.
We will make our code and data publicly available at https://github.com/ernie-research/MA-RLHF .

## 1 Introduction

Recent advancements in large language models (LLMs) have revolutionized natural language processing tasks, demonstrating impressive capabilities across a wide range of applications such as code generation (Roziere et al., ; Chai et al., ; Lozhkov et al., ), mathematical reasoning (Lewkowycz et al., ; Anil et al., ), and dialogue assistance (OpenAI, ; Team et al., ; ). Despite these successes, aligning LLMs with human values and preferences remains a critical challenge. Reinforcement learning from human feedback (RLHF) has emerged as a promising approach to address this alignment issue by incorporating human evaluations into the training process (Christiano et al., ; Ziegler et al., ; Stiennon et al., ).

Existing RLHF (Ouyang et al., ; Bai et al., ; Askell et al., ) methods mainly optimize decisions at the level of individual tokens, and require to process a vast number of minute adjustments. However, this fine-grained training paradigm can lead to the credit assignment problem (Bengio et al., ; Kaelbling et al., ), particularly when dealing with long-distance dependencies. As LLM agents attempt to optimize decisions across extensive sequences, the difficulty in attributing the credits of actions to specific tokens complicates the reinforcement learning (RL) process (Pignatelli et al., ). Moreover, the use of subword tokenization, such as Byte-Pair Encoding (Sennrich et al., ), often splits words into smaller pieces. For instance, OpenAI’s ChatGPT(^1^11[https://platform.openai.com/tokenizer](https://platform.openai.com/tokenizer)) treats each token as three quarters of a word on average, resulting in sequences that are 33% longer than word counts (OpenAI, ) and further exacerbates the credit assignment problem.

Additionally, standard RLHF methods may overlook essential linguistic structures and dependencies between adjacent tokens in natural language. For example, consider the phrase Big Apple(^2^22[https://en.wikipedia.org/wiki/Big_Apple](https://en.wikipedia.org/wiki/Big_Apple)), treating Big and Apple as isolated decisions misses the cohesive meaning of the term, which actually refers to the “New York City”. The token-level granularity of natural language can hinder the agent’s ability to capture high-level language constructs in RL optimization, as some sequences are better understood when evaluated holistically.

To address these challenges, we propose a new framework called macro-action RLHF (MA-RLHF) that incorporate macro action — sequences of tokens or high-level language constructs — into the RLHF framework. The concept of macro actions, has been explored in the literature of planning  (Iba, ; Korf, ; Sacerdoti, ) and reinforcement learning (Thrun & Schwartz, ; Precup et al., ; Hauskrecht et al., ), simplifies decision-making by operating at high levels of temporal abstraction under the framework of semi-Markov Decision Processes (SMDPs) (Sutton et al., ). Macro actions leverage meaningful linguistic structures and allow the agent to learn from “long-sighted” macro-level actions instead of “short-sighted” token-level actions, which can potentially leads to improved learning efficiency and scalability.
Alternatively, MA-RLHF can also be interpreted from the perspective of reversing tokenization; MA-RLHF serves as a de-tokenization process to reconstruct high-level language units from subword pieces. By merging tokens into macro actions, we reduce the number of decision points and shorten decision trajectories, alleviating the credit assignment problem caused by long temporal distances.

To conclude, our main contributions are as follows:

- •
We propose MA-RLHF, a simple yet effective RLHF framework that integrate the macro actions into RLHF to align LLMs with human preference. We demonstrate the effectiveness of our approach through extensive experiments across various datasets and tasks, including text summarization, dialogue generation, question answering, and program synthesis.
- •
We show that MA-RLHF achieves 1.7x to 2x faster convergence in reward scores during training compared to the standard token-level RLHF, without introducing additional computational costs during training or inference. MA-RLHF also exhibits strong scalability across model sizes ranging from 2B to 27B parameters.
- •
Our analysis reveals that MA-RLHF exhibits robust generalization capabilities under varying experimental settings, such as temperature values and rejection sampling, consistently outperforms the standard RLHF approaches.

## 2 Preliminaries

We introduce some basic concepts and notations used in RL and RLHF.

### 2.1 Reinforcement Learning and Policy Optimization

Problem Definition   RL addresses the problem of finding an policy to make optimal sequential decisions in environments modeled as the Markov Decision Processes (MDPs) (Sutton & Barto, ). An MDP is defined by the tuple $(\mathcal{S},\mathcal{A},P,r,\rho_{0},\gamma)$, where $\mathcal{S}$ denotes a finite set of states, $\mathcal{A}$ is a finite set of actions, $P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ represents the state transition probability distribution, $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function, $\rho_{0}:\mathcal{S}\rightarrow[0,1]$ defines the initial state distribution, and $\gamma\in(0,1)$ is the discount factor that determines the importance of future rewards.

The state-action value function $Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{s_{t+1},a_{t+1},\dots}\left[\sum_{l=0}^{
\infty}\gamma^{l}r(s_{t+l})\right]$ measures the expected return of taking action $a_{t}$ at state $s_{t}$ and following policy $\pi$ thereafter. The value function $V_{\pi}(s_{t})=\mathbb{E}_{a_{t},s_{t+1},\dots}\left[\sum_{l=0}^{\infty}\gamma
^{l}r(s_{t+l})\right]$ estimates the expected return from state $s_{t}$ under the policy $\pi$. The advantage function $A_{\pi}(s_{t},a_{t})=Q_{\pi}(s_{t},a_{t})-V_{\pi}(s_{t})$ reflects the relative value of taking action $a_{t}$ at state $s_{t}$ compared to the average value of the state.

The goal of RL is to find an optimal policy $\pi_{\theta}(a\mid s)$, parameterized by $\theta$, that maximizes the expected cumulative discounted reward: $J(\theta)=\mathbb{E}_{s_{0},a_{0},\dots}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s
_{t})\right]$,
where $s_{0}\sim\rho_{0}(s_{0})$ represents the initial state distribution, $a_{t}\sim\pi_{\theta}(a_{t}\mid s_{t})$ denotes the action selection based on the policy, and $s_{t+1}\sim P(s_{t+1}\mid s_{t},a_{t})$ specifies the state transition dynamics.

Proximal Policy Optimization   Policy gradient methods are a common approach for optimizing policies by estimating the gradient of a performance objective with respect to the policy parameters $\theta$. The policy gradient is given by: $\nabla_{\theta}J(\theta)=\mathbb{E}\left[\sum_{t=0}^{\infty}A_{t}\nabla_{
\theta}\log\pi_{\theta}(a_{t}\mid s_{t})\right]$, where the expectation $\mathbb{E}[\cdot]$ is taken over the randomness of initial state, policy, and state-transition. The policy gradient guides us how to adjust the policy parameters to improve the expected return.
Among the family of policy gradient methods,
Proximal Policy Optimization (Schulman et al., , PPO) is perhaps the most widely-used one due to its simplicity and empirical effectiveness. PPO simplifies TRPO (Schulman et al., ) by using a clipped surrogate objective function to penalize large deviations from the old policy, thereby ensuring more stable updates. Specifically, PPO introduces a clipped objective function:

$$ $J^{\text{ppo-clip}}(\theta)=\mathbb{E}\left[\min\left(\frac{\pi_{\theta}(a_{t} \mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})}{A}_{t},\text{clip}( \frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t} )},1-\epsilon,1+\epsilon){A}_{t}\right)\right],$ (1) $$

where $\epsilon$ is a hyperparameter that defines the range for clipping. Nowadays, PPO usually comes as the first choice for RL practitioners.

### 2.2 RLHF for Human Alignment

The post-training of LLMs (Stiennon et al., ; Ouyang et al., ) is a multi-stage training paradigm to align LLMs with human preferences. Post-training typically involves three stages:

(1) SFT stage: A pre-trained language model (LM) is fine-tuned on a dataset of human demonstrations, learning to generate responses that align with human instructions and preferences.

(2) RM stage: A reward model $r_{\phi}(x,y)$, parameterized by $\phi$, is trained to predict human preference scores for model outputs. The training data $\mathcal{D}={(x_{i},y_{i}^{+},y_{i}^{-})}_{i=1}^{N}$ consists of prompts $x_{i}$ and pairs of responses $(y_{i}^{+},y_{i}^{-})$, where $y_{i}^{+}$ is preferred over $y_{i}^{-}$ by human annotators. The reward model is trained using the ranking loss: $\mathcal{L}_{\textrm{RM}}=-\log\sigma(\log(r_{\phi}(x,y_{+})-r_{\phi}(x,y_{-})))$,
where $\sigma$ denotes the sigmoid function, and $r_{\phi}$ represents the reward function parameterized by $\phi$.

(3) RLHF stage: The RL fine-tuning utilizes the RM to provide feedback on the generated outputs, optimizing the policy using RL methods such as PPO. The reward signal is modified by incorporating a Kullback-Leibler (KL) divergence penalty to balance the exploration of new policies with adherence to the SFT model. The reshaped reward is defined as:

$$ $R(x,y)=r_{\phi}(x,y)+\beta D_{\mathrm{KL}}(\pi_{\mathrm{sft}}(y~{}|~{}x)~{}||~ {}\pi_{\theta}(y~{}|~{}x)),$ $$

where $\pi_{\theta}$ represents the policy learned through RL, $\pi_{\mathrm{sft}}$ is the policy produced from the SFT stage, and $\beta>0$ is a hyperparameter that controls the strength of the KL penalty.

In the RLHF stage, the PPO algorithm, as detailed in eq. 1, is employed to optimize the RL policy. In the context of RLHF, we denote the state $s_{t}=\{a_{1},a_{2},\dots,a_{t-1}\}$ as the sequence of tokens generated up to time step $t$, while $a_{t}$ represents the token selected at the $t$-th position.

## 3 Marco-Action RLHF

### 3.1 Revisiting Macro Actions (Options)

Macro actions, also referred to as options (Sutton et al., ), are high-level constructs that encapsulate a sequence of primitive actions (*i.e.*, subword tokens); by its definition, macro actions allows an agent to operate at a coarser temporal scale.

Formally, a macro action is characterized by three components: (1) a policy $\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ which guides the action selection among macro actions; (2) a termination condition $\zeta:\mathcal{S}^{+}\rightarrow[0,1]$, which determines where the macro action should end; (3) a initiation set $\mathcal{I}\subseteq\mathcal{S}$, which is a subset of states that macro actions can begin with.
Once initiated with a state $s_{0}\in\mathcal{I}$, the macro action follows policy $\pi$ until it reaches the termination condition according to $\zeta$. Intuitively, the use of carefully designed macro actions can extend decision-making temporally, it allows the agent to avoid “short-sighted” token-level decisions and encourage “long-sighted” macro-level decisions, thereby simplifies the decision-making process and potentially enhances learning efficiency.

### 3.2 RLHF with Macro Actions

Figure: Figure 1: Illustration of the MA-RLHF framework. Standard RLHF makes decisions and evaluates value scores at the token level, while MA-RLHF operates at the macro action level, making decisions over sequences of tokens at a coarser temporal scale.
Refer to caption: x1.png

We describe how we integrate macro-actions into the existing RLHF framework, the resulting framework is named as macro-action RLHF (MA-RLHF).

#### 3.2.1 Formalization of Macro Actions

We denote macro actions as $\omega_{1},\omega_{2},\ldots,\omega_{\tau}$. In the context of LLMs, a macro action $\omega_{\tau}$ consists of a sequence of consecutive tokens, $i.e.$, $\omega_{\tau}=\{a_{t_{\tau}},a_{t_{\tau}+1},\ldots,a_{t_{\tau+1}-1}\}$, where $t_{\tau}$ is the starting index of the $\tau$-th macro action. We let $|\omega_{\tau}|$ denotes the number of primitive actions that $\omega_{\tau}$ contains. Unless otherwise specified, we use $\tau$ to index macro actions/states and use $t$ to index primitive actions/states.

As mentioned in §3.1, macro actions are defined by the policy model, the termination condition and the initiation set. In MA-RLHF, we set the policy model the same as the standard token-level RLHF and let the initiation set to be any possible sequence of tokens. Therefore, the macro action used in MA-RLHF is decided solely by the termination condition, which plays a crucial rule in the MA-RLHF framework. We explore three termination conditions in this work:

- •
$n$-gram based termination: Following Vezhnevets et al. (), we find that $n$-grams serve as a simple yet effective termination condition for macro actions, *i.e.*, $|\omega_{\tau}|=n$, where $n$ represents the length of the $n$-gram. We consider two variants of the $n$-gram termination condition: (a) Fixed $n$-gram: We group tokens into fixed-length $n$-grams, simplifying the action space while maintaining common linguistic patterns. We empirically find fixed $n$-gram macro action perform best and use it as the default setup. (b) Randomized $n$-gram: We randomly select the length of a $n$-gram from a predefined list of lengths $n\in\{2,3,5,10\}$ to introduce variability, allowing the policy to adapt to different sequence lengths.
- •
Parsing-based termination: $\omega_{\tau}$ is derived from syntactic or semantic parsing of the input text, aligning macro actions with grammatical structures like phrases or clauses. Concretely, we traverse the constituent tree of the entire sequence using depth-first search (DFS), expanding non-terminal nodes until current non-terminal state contains no more than a specified threshold of leaf tokens, set at $C=5$.
- •
Perplexity-based (PPL) termination: Perplexity measures the likelihood of a sequence of tokens. Here, the perplexity of a macro action is proportional to the averaged entropy of the token within it, $i.e.$, $\mathrm{ppl}(\omega_{\tau})\propto-\frac{1}{|\omega_{\tau}|}\sum_{a\in\omega_{
\tau}}\log p_{a}$. A macro action terminates until it reaches a token that has negative impact on the perplexity of the macro action. Mathematically, we construct $\omega_{\tau}=\{a_{t_{\tau}},\ldots,a_{t_{\tau+1}-1}\}$ such that $\mathrm{ppl}(\omega_{\tau}\cup a_{\tau+1})>\mathrm{ppl}(\omega_{\tau})$ and $\mathrm{ppl}(\{a_{\tau},\ldots,a_{i}\})\geq\mathrm{ppl}(\{a_{\tau},\ldots,a_{i
+1}\})$ for all $t_{\tau}\leq i\leq t_{\tau+1}$.

#### 3.2.2 Policy Optimization with Macro Actions

In MA-RLHF, we adapt the PPO algorithm to operate at the macro action level, referred to as MA-PPO. The policy $\pi_{\theta}$ still outputs probabilities over individual tokens, but for optimization, we consider the joint probability of the macro action: $\pi_{\theta}(\omega_{\tau}~{}|~{}s_{\tau})=\prod_{t=t_{\tau}}^{t_{\tau+1}}\pi_
{\theta}(a_{t}~{}|~{}a_{<t})$.
The macro reward for executing the macro action $\omega_{\tau}$ at the macro time step $\tau$ is defined as:
$R_{\tau}=\mathbb{E}\big{[}\sum_{i=0}^{|\omega_{\tau}|-1}\rho^{i}r_{t_{\tau}+i}
~{}\big{|}~{}s_{\tau}\big{]},$
where $r_{t}$ is the reward received at time step $t$, and we set the discount factor $\rho=1$ in our experiments.

Each macro action represents a contiguous sequence of tokens, and is treated as an option in the SMDP framework.
The option-level value function with macro action is then estimated as:

$$ $V^{\pi}(s_{\tau},\omega_{\tau})=\mathbb{E}\left[R_{\tau}+\gamma V^{\pi}(s_{t_{ \tau+1}})\ \middle|\ s_{\tau},\omega_{\tau}\right],$ $$

where $\gamma$ is the discount factor for future rewards beyond the macro action.

The advantage function $A_{\pi}(s_{\tau},\omega_{\tau})$ in MA-PPO determines how much the chosen macro action outperforms the average, which is defined as $A_{\pi}(s_{\tau},\omega_{\tau})=Q_{\pi}(s_{\tau},\omega_{\tau})-V^{\pi}(s_{
\tau})$. Similar to the definition stated in §2, $Q_{\pi}(s_{\tau},\omega_{\tau})$ is the expected return conditioned on executing $\omega_{\tau}$ at state $s_{\tau}$, which is calculated by summing the immediate macro rewards from the macro action with the discounted value of the subsequent state.

In MA-PPO, the objective function is adapted for MA-level evaluation. The policy gradient is computed based on the advantage of the MA sequences:

$$ $\mathcal{L}^{\text{MA-PPO}}(\theta)={\mathbb{E}}_{\tau}\left[\min\left(r_{\tau }(\theta)\hat{A}_{\tau},\ \text{clip}\left(r_{\tau}(\theta),1-\epsilon,1+ \epsilon\right)\hat{A}_{\tau}\right)\right],$ $$

where $r_{\tau}(\theta)=\pi_{\theta}\big{(}\omega_{\tau}\mid s_{\tau}\big{)}/\pi_{
\theta_{\text{old}}}\big{(}\omega_{\tau}\mid s_{\tau}\big{)}$, and $\hat{A}_{\tau}$ is the estimated advantage at macro time step $\tau$, $\epsilon$ is a constant that defines the range for clipping, and $\pi_{\theta_{\text{old}}}$ is the policy before the update.

#### 3.2.3 Connection to Previous Methods

MA-RLHF builds on and generalizes prior work in the RLHF literature by varying the length of macro actions. When the macro action length is set to $1$, MA-RLHF reduces to the standard token-level RLHF (Stiennon et al., ; Ouyang et al., ), operating as an MDP. Conversely, if we allow $|\omega_{\tau}|\rightarrow\infty$, then MA-RLHF converges toward methods like RLOO (Ahmadian et al., ) and REINFORCE (McGovern & Sutton, ), approximating a contextual bandit problem where decisions are made based on the entire sequence context. By varying the length of macro actions $|\omega_{\tau}|$, MA-RLHF provides a flexible framework that balances the granularity of action decisions. We provide further analysis on the impact of $|\omega_{\tau}|$ in §4.3.

## 4 Experiments

### 4.1 Experimental Settings

Tasks and Datasets  We evaluate MA-RLHF on three different datasets for open-ended generation tasks: TL;DR (Stiennon et al., ) dataset for text summarization, Anthropic Helpful and Harmless (HH-RLHF) (Bai et al., ) for dialogue generation(^3^33[https://huggingface.co/datasets/Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf)), and WebGPT Comparison (Nakano et al., ) for question answering. Additionally, we evaluate MA-RLHF on code generation using the APPS (Hendrycks et al., ) dataset. More details of the datasets can be found in Section B.1.

Base models and training details   For open-ended generation tasks, we use pre-trained Gemma-2B (Team et al., ) as our base model; we further adopt Gemma-7B and Gemma-2-27B to test the scaling trend. For the program synthesis task, we use CodeGemma-1.1-2B and CodeGemma-1.1-7B-it as our base models. The data split for SFT/RM/PPO and the hyperparameters used in SFT/RM/PPO stages are detailed in Section B.2. The implementation details of MA-PPO along with the code snippet can be found in Appendix E.

Evaluation  For open-ended generation tasks, our evaluation metrics includes RM scores, GPT-4 pairwise evaluation, and human pairwise evaluation.
To compute the RM score, we randomly sample 2k validation instances for the TL;DR and HH-RLHF datasets and use the default validation set of the WebGPT dataset.
For GPT-4 and human evaluations, we simulate the win-rate on 50 instances that are drawn from the instances used in the RM evaluation. The GPT-4 and human evaluations are based on task-specific criterion: relevance, coherence, consistency, and fluency for TL;DR; helpfulness for HH-RLHF; factual accuracy, coherence, and usefulness for WebGPT. We followed prior studies (Askell et al., ; Zheng et al., ) by randomizing the order of responses during evaluation to mitigating potential evaluation biases. The prompts used by the GPT-4 evaluation are placed in Section F.1, and the annotation rules used for human evaluation are given in Section F.2. For the program synthesis task, we utilize pass@1 and pass@5 metrics to assess the performance of the model, evaluated on the provided 5k test set.

### 4.2 Main Results

In this section, we present the main results of applying MA-PPO across three key tasks: summarization, dialogue, and question answering. The main takeaway is that MA-PPO consistently outperforms vanilla PPO in terms of both training efficiency and generation quality; MA-PPO obtains a significant improvement in testing reward model scores and human/GPT-4 evaluation win rates.

Figure: Figure 2: Test RM scores of Gemma-2B and Gemma-7B models on the TL;DR dataset. The shaded regions represent the standard deviation on test RM scores across training runs.
Refer to caption: x2.png

Figure: Figure 4: The win rates of MA-PPO against vanilla PPO, estimated by GPT-4 and Human. The left/middle/right figures are results for TL;DR, HH-RLHF and WebGPT Comparisons respectively.
Refer to caption: x5.png

TL;DR Summarization   For the TL;DR summarization task, MA-PPO shows a marked improvement over vanilla PPO. As shown in Figure 3, MA-PPO achieves parity with vanilla PPO approximately 1.7 – 2 times faster during training. Specifically, Gemma-2B trained with 1.7k MA-PPO updates reaches similar testing RM scores obtained by vanilla PPO trained with 3.7k steps. We also find similar trends when scaling up the parameter sizes to 7B, demonstrating the generalized capability of MA-PPO on model sizes.

Moreover, Figure 3 highlights the distribution of RM scores, where MA-PPO consistently shifts towards higher RM sores compared to vanilla PPO. Further evaluation using GPT-4, given in the left figure of Figure 4, shows that MA-PPO achieves 78% and 86% win rate over vanilla PPO for the 2B and 7B models, respectively. Human evaluation gives similar result, where MA-PPO obtains win rates of 74% and 69%, and further confirming the effectiveness of macro actions. The final testing RM scores of MA-PPO and vanilla PPO are given in Table 2.

HH-RLHF Dialogue   We use the HH-RLHF dataset to evaluate the helpfulness and harmlessness of single-turn dialogues. MA-PPO shows clear advantages over vanilla PPO, as depicted in the middle figure of Figure 4. GPT-4 evaluations show that MA-PPO yields a 72% win rate for the Gemma-7B model, compared to 58% for the Gemma-2B model. Human evaluation results align with these findings, with the win rate increasing from 52% to 56% as model size scales from 2B to 7B. The testing RM score of MA-PPO and vanilla PPO are presented in Table 2. These results highlight the scalability and effectiveness of MA-PPO in dialogue tasks. We refer to Section C.1 for detailed experimental results.

WebGPT Comparisons   We evaluate MA-PPO on the WebGPT Comparison dataset for question-answering tasks. As shown in Figure 4 (Right), MA-PPO consistently outperforms vanilla PPO, with GPT-4 evaluations yielding a win rate of 64% for the Gemma-7B model. This result demonstrate the robustness of MA-PPO across different tasks, including more structured tasks like question answering. More experimental details refer to Section C.2.

**Table 1: Agreement among RM, GPT-4, and human evaluations on TL;DR.**
|  | #Param | RM | GPT-4 | Human |
| --- | --- | --- | --- | --- |
| RM | 2B | 100% | - | - |
| GPT-4 | 78% | 100% | - |  |
| Human | 76% | 58% | 100% |  |
| RM | 7B | 100% | - | - |
| GPT-4 | 78% | 100% | - |  |
| Human | 74% | 64% | 100% |  |

Validating Model-based Judgments with Human Evaluation   We evaluate the reliability of our evaluation methods by calculating the agreement between the reward model, GPT-4, and human evaluators. Since GPT-4 and human evaluations are conducted pairwise, we determine the reward model’s win rate by selecting the summary with the higher RM score. The results, shown in Table 2, demonstrate that the reward model aligns more closely with both GPT-4 and human evaluations. Furthermore, the agreement between GPT-4 and human evaluators averaged 62% across models, reinforcing the consistency and validity of our evaluation framework.

Figure: Figure 5: Performance of MA-PPO with various macro action termination strategies on the TL;DR dataset using Gemma-2B. Left: Test RM scores for different termination strategies. Right: GPT-4 evaluation across four dimensions–relevance, coherence, consistency, and fluency–comparing different MA termination methods.
Refer to caption: x8.png

### 4.3 Analyzing the Use of Macro Actions

We study the performance of various termination strategies. Unless otherwise specified, we conduct our analysis on the TL;DR dataset.

#### 4.3.1 Exploring Different Strategies for MA Termination ( ζ 𝜁 \zeta italic_ζ )

In MA-RLHF, the termination condition ($\zeta$) for macro actions is critical as it determines when a macro action should conclude. We compare the performance of various termination strategies, particularly on reward maximization and linguistic coherence.
The termination strategies studied in this section including fixed / randomized $n$-gram-based, parsing-based, and perplexity-based termination, as aforementioned in §3.2.1; please see Figure 13 for detailed illustration.

Figure 5 illustrates the overall test-set performance on RM scores (Left) and GPT-4 evaluation scores (Right) with different MA termination strategies. All macro action termination strategies outperform the vanilla PPO approach, underscoring the importance of temporal abstraction in decision-making. Figure 5 (Left) shows that $n$-gram based approach, both fixed and randomized, achieves the optimal results among others. Notably, randomized $n$-gram-based termination performs the best across multiple dimensions, including relevance, coherence, and consistency, as shown in Figure 5 (Right). As expected, the perplexity-based termination enhances fluency, and is most suited for tasks that prioritize smooth and natural language generation. Furthermore, parsing-based termination shows promising ability to handle complex grammar, as it is designed to better capture linguistic structures.

#### 4.3.2 Ablation Study: Varying n 𝑛 n italic_n in MA-RLHF

The $n$-gram based macro action strategy in MA-RLHF uses a hyper-parameter $n$ to control the length of macro actions. Notably, when $n=1$, MA-PPO is equivalent to vanilla PPO, and treats the problem as a traditional Markov Decision Process (MDP), making decisions token by token. In contrast, setting $n\rightarrow\infty$ corresponds to the REINFORCE algorithm (McGovern & Sutton, ), where the entire sequence is treated as a single macro action, akin to a contextual bandit problem, as discussed in § 3.2.3. For intermediate values of $n$ (i.e., $n\in(1,\infty)$), MA-PPO falls under the SMDP framework, which allows for temporally extended actions; see §3. This continuum between MDPs and contextual bandits highlights the flexibility of the MA-RLHF approach in handling varying levels of temporal abstraction.

RM Scores   We conducted experiments with varying values of $n$ ($n\in\{3,5,10,\infty\}$) on the TL;DR and HH-RLHF datasets. Figure 7 shows that all values of $n$ lead to performance improvements over the vanilla PPO ($n=1$), indicating the advantage of modeling sequences of tokens as macro actions. Notably, for the TL;DR dataset, $n=\infty$ yields the highest RM score, suggesting that treating the entire sequence as a macro action is particularly effective for the summarization task. For the HH-RLHF dataset, setting $n=10$ gives the best performance, likely because this task benefits from moderate-length macro actions that can capture essential linguistic structures while maintaining sufficient granularity.

GPT-4 Evaluation Analysis   As shown in Figure 7, setting $n=5$ strikes a good balance between relevance, coherence, consistency; it outperforms both smaller and larger values of $n$. These findings align with the Semi-MDP framework: increasing $n$ allows for better credit assignment and context retention, but excessive abstraction (e.g., $n=\infty$) sacrifices fine-grained control. Overall, moderate values of $n=5$ and $n=10$ provide the best trade-offs, highlighting the adaptability of MA-RLHF across tasks.

Figure: Figure 6: Test RM scores of different $n$ values in MA-PPO evaluated by corresponding RM on the TL;DR (left) and HH-RLHF (right) dataset.
Refer to caption: x10.png

### 4.4 Generalization Probing in Macro Actions

Figure: Figure 8: The effect of temperature on RM scores for varying sample sizes (Best-of-$N$) across models. (Left): RM score of the SFT model under different temperatures and sample sizes. (Mid): RM score of vanilla PPO under the same settings. (Right): RM score of MA-PPO.
Refer to caption: x13.png

Robustness on Rejection Sampling vs. Temperature   Best-of-$N$ (*a.k.a*, rejection sampling) (Touvron et al., ) enhances response quality by selecting the highest-reward response from $N$ samples generated by the policy model. We compare MA-PPO, SFT, and vanilla PPO using the best-of-$N$ sampling across various temperatures $T\in\{0.2,0.4,0.6,0.8,1.0,1.2\}$ and sample sizes $N\in\{4,8,16,32\}$. As shown in Figure 8, best-of-$N$ sampling improves RM scores for all methods, with performance increasing as $N$ grows. We observe that SFT and vanilla PPO are sensitive to temperature variations, requiring specific adjustments to achieve optimal results. In contrast, MA-PPO demonstrates robustness in sampling temperature, it consistently delivers the best performance at $T=1.2$ and shows consistent improvement across all tested temperatures. Moreover, MA-PPO maintains stable performance across varying temperature settings, as further detailed in Section D.3, highlighting its robustness and generalization capabilities under different sampling temperatures.

Figure: Figure 9: Evaluation results for vanilla PPO and MA-PPO on Gemma-2-27B using the TL;DR dataset. Left: RM scores on validation set. Mid: Distribution of RM scores for vanilla PPO and MA-PPO. Right: Scaling trending on TL;DR dataset across 2B, 7B, and 27B model size, showing RM scores, GPT-4 evaluation, human evaluation results.
Refer to caption: x16.png

Scaling Trends up to 27B Models   We evaluate the performance of MA-PPO across different model sizes, specifically Gemma-2B, 7B, and 27B. As demonstrated in Figure 9 (Left and Mid), MA-PPO consistently surpasses vanilla PPO, exhibiting higher RM scores throughout training. Figure 9 (Right) presents the scaling trend of MA-PPO across the 2B, 7B, and 27B models in terms of testing RM scores, GPT-4, and human evaluations. The experimental results underscore the scalability and robust performance of MA-PPO across varying model sizes.

Figure: Figure 10: RM score shifting pattern after RLHF training; Left: RM scores of best-of-$N$ ($N=8$) sampling compared to the SFT model. Mid Left: RM scores of vanilla PPO compared to the SFT model. Mid Right: RM scores of MA-PPO ($n=5$) compared to the SFT model. Right: RM scores of MA-PPO ($n=\infty$) compared to the SFT model.
Refer to caption: x19.png

Analyzing the Impact on RM Score Distribution   We evaluate the RM score distribution shift after applying RLHF using vanilla PPO and MA-PPO on the TL;DR dataset, with the SFT model serving as the baseline. To further contextualize the impact of RLHF, we include the Best-of-$N$ sampling ($N=8$) on the SFT model. As illustrated in Figure 10, Best-of-$N$ enhances overall response quality but falls short compared to RLHF. While vanilla PPO shifts the distribution towards higher RM scores, it leaves a significant number of low-quality, long-tailed instances. In contrast, MA-PPO demonstrates a more pronounced positive impact, effectively reduces the number of low-quality outliers and improves overall score distribution compared with the vanilla PPO. This highlights the robustness of MA-PPO in enhancing response quality through RLHF.

### 4.5 Additional Analysis

Impact on $L_{2}$-norm of Advantage and Q Values   We present the $L_{2}$-norm of both the advantage and Q-values for MA-PPO and vanilla PPO during training in Figure 12. The advantage function, which reflects the difference between the expected return (Q-value) and the baseline, plays a critical role in guiding policy optimization. A lower $L_{2}$-norm of both the advantage and Q-values suggests more stable and less noisy policy updates, likely contributing to faster convergence observed in §4.2.

The policy gradient for a sequence of length $T$ is given by: $\nabla_{\theta}J=\mathbb{E}\big{[}\sum_{t=1}^{T}\nabla_{\theta}\log\pi_{\theta
}(a|s)\cdot R\big{]}$, where $R$ is the sequence reward provided by the RM.
In the case of using $n$-gram based macro actions, the sequence length is reduced by a factor of $n$, shortening the decision horizon: $T\rightarrow T/n$. This reduction in the number of actions, $T/n$, where $n>1$, implies that the temporal distance between actions and corresponding rewards is decreased, thus reducing the variance in the gradient estimate and improving credit assignment.
We refer readers to Mann & Mannor () for the theoretical foundations of variance reduction through macro actions and their benefits in RL.

Figure: Figure 11: $L_{2}$ Norm of advantages and Q-values during training for MA-PPO and vanilla PPO. Left: $L_{2}$ norm of advantages over training steps; Right: $L_{2}$ norm of Q-values.
Refer to caption: x23.png

Case Study   We show some qualitative examples in Section G.1, demonstrating that MA-PPO can produce more coherent and contextually appropriate responses compared to vanilla PPO, capturing both short/long-term dependencies effectively.

Extended Experiments: Program Synthesis   We further assess the effectiveness of MA-PPO on the program synthesis task. Following (Shojaee et al., ; Liu et al., ), we utilize the compiler signal as the final reward; see Section B.5 for implementation details.
We compare the performance of MA-PPO and vanilla PPO using the pass@k (k=1, 5) metric Chen et al. () on the 5k test set from APPS dataset (Hendrycks et al., ). As shown in Figure 12, MA-PPO significantly outperforms vanilla PPO across both pass@1 and pass@5 metrics, with more pronounced improvements as model size scales. Notably, for the 7B model, MA-PPO achieves an improvement of +35% in pass@1 and +30% in pass@5 over vanilla PPO, demonstrating the effectiveness of our approach in code generation tasks.

## 5 Related Work

LLM Alignment   RLHF have shown impressive success in aligning LLMs with human preferences through multi-stage training, including SFT, RM, and RL fine-tuning (Ziegler et al., ; Stiennon et al., ; Ouyang et al., ). Recent research has explored optimization methods for RL in LLMs, employing both online (Ahmadian et al., ; Farebrother et al., ; Shen et al., ; Chakraborty et al., ) and offline RL algorithms (Snell et al., ; Hu et al., ; Yu et al., ) to address training instability and improve efficiency (Tang et al., ). Improvements to RM learning have been proposed, such as parameter scaling (Gao et al., ), fine-grained reward (Wu et al., ), tool use (Li et al., ), and model merging (Ramé et al., ; Rame et al., ). Alternatively, direct policy optimization (Rafailov et al., ; Ethayarajh et al., ; Gheshlaghi Azar et al., ; Rosset et al., ) has emerged as a promising approach, bypassing the instability of RL while directly aligning models to human preferences. In this paper, we enhance the RLHF action space by integrating macro actions, a well-established concept in RL(Sutton et al., ; Mann & Mannor, ).

Macro Action in RL   Macro actions introduce temporal abstraction in RL by grouping sequences of primitive actions, reducing decision complexity and improving long-horizon credit assignment (Precup et al., ; Hauskrecht et al., ; Sutton et al., ; Pignatelli et al., ; Machado et al., ). This method has demonstrated its utility in speeding up convergence and stabilizing policy updates in various domains (Mann & Mannor, ; Solway et al., ). Our work applies macro actions to RLHF in LLM training, leveraging this structure to enhance scalability and optimize credit assignment over extended sequences.

## 6 Conclusion and Future Work

In this paper, we introduced MA-RLHF, a novel framework that incorporates macro actions into RLHF to enhance the alignment of LLMs with human preferences. Our approach demonstrates consistent improvements across multiple tasks, including summarization, dialogue generation, question answering, and program synthesis. Notably, MA-RLHF achieves 1.7x to 2x faster convergence in reward scores without incurring additional computational overhead, showing robust scalability across model sizes ranging from 2B to 27B parameters. It is promising to explore more fine-grained or learnable macro action strategies for future research.

## Appendix A Limitations

While our work demonstrates the effectiveness of MA-RLHF across multiple tasks, there are several limitations that leave room for future improvements. First, our macro action termination methods are rule-based, including linguistics- or perplexity-driven approaches; future research could explore more complex or learnable termination strategies to further enhance performance. Second, our experiments are conducted using models with up to 27B parameters; exploring more advanced models, such as LLaMA 3.1 405B (Dubey et al., ) or other state-of-the-art architectures, may provide additional insights into the scalability of MA-RLHF. Lastly, although we observe significant improvements in training efficiency and convergence speed, further investigation into the trade-offs between training stability and performance under diverse real-world conditions is necessary. Addressing these limitations will pave the way for more robust applications of MA-RLHF.

## Appendix B Experimental Details

### B.1 Datasets and Tasks

TL;DR Summarization   In this task, the policy is asked to generate summarizations for Reddit posts. This dataset consists of 93k human-annotated preference pairs and 86k pairs for validation. The trainable pairs are derived from the Reddit TL;DR (Völske et al., ) dataset. Additionally, a portion of the validation pairs is sourced from the CNN Daily Mails, which serves as the test set for out-of-distribution generalization.

HH-RLHF   With the Anthropic HH-RLHF dataset, the policy is asked to generate a helpful and harmless response given a single-turn dialogue or multi-turn dialogue. This dataset provides 112k preference-labeled instances for training, and 12.5k for validation.

WebGPT Comparisons   The WebGPT Comparisons dataset contains QA pairs from the ELI5 (Fan et al., ) and the TriviaQA (Joshi et al., ). The policy is responsible for information retrieval and response generation. In our experimental setup, we focus exclusively on the generation task. The policy must generate a response that balances factual accuracy and coherence. This dataset contains 19.6k instances for training. We split 5% instances for validation, as no separate validation set is provided.

Code Generation   For this task, we leverage the APPS dataset, which contains 5k training and 5k validation instances. The policy must write executable code based on a natural language described in the question, using Python as the target programming language.

We present the data statistics in Table 3.

**Table 3: Statistics of datasets involved in experiments. The number of tokens are calculated with Gemma-2B tokenizer.**
| Dataset | Num. of<br>Comparisons | Num. of<br>Train Samples | Num. of<br>Test Samples | Avg. Tokens<br>in Prompt | Avg. Tokens<br>in Chosen | Avg. Tokens<br>in Rejected |
| --- | --- | --- | --- | --- | --- | --- |
| Anthropic HH-RLHF | 127.5k | 112k | 12.5k | 160 | 83 | 75 |
| OpenAI Summarization | 179k | 92.9k | 86.1k | 325 | 35 | 33 |
| OpenAI WebGPT | 19.6k | 18.5k | 979 | 49 | 149 | 137 |
| APPS | 10k | 5k | 5k | 453 | 203 | - |

### B.2 Training Details

**Table 4: Hyper-parameters for training Gemma series of models in MA-PPO and vanilla PPO.**
|  | Hyper-Parameter | Gemma | CodeGemma |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | 2B | 7B | 27B | 2B | 7B |  |
| SFT | Batch size | 64 for WebGPT<br>512 for others | 128 | 128 | 16 | 32 |
| Epochs | 3 | 5 for WebGPT<br>1 for others | 3 | 1 | 1 |  |
| Learning rate | 1e-4 for WebGPT<br>5e-5 for others | 2e-5 | 5e-6 | 5e-6 | 2e-6 |  |
| LR scheduler | cosine | cosine | cosine | cosine | cosine |  |
| Warmup ratio | 0.1 | 0.1 | 0.1 | 0 | 0 |  |
| RM | Batch size | 32 for WebGPT<br>64 for others | 128 for TL;DR<br>64 for HH-RLHF<br>32 for WebGPT | 128 | - | - |
| Epochs | 1 | 1 | 1 | - | - |  |
| Learning rate | 2e-5 for WebGPT<br>1e-5 for others | 1e-6 | 8e-6 | - | - |  |
| LR scheduler | cosine | cosine | cosine | - | - |  |
| Warmup ratio | 0.1 | 0.1 | 0.1 | - | - |  |
| PPO | Batch size | 256 | 256 | 256 | 16 | 16 |
| Policy learning rate | 1.5e-5 | 1e-6 | 7e-7 | 5e-7 | 5e-7 |  |
| Critic learning rate | 1.5e-5 | 1e-6 | 1e-6 | 5e-5 | 5e-5 |  |
| Epochs | 4 for WebGPT<br>1 for others | 4 for WebGPT<br>1 for others | 1 | 1 | 1 |  |
| PPO epochs | 1 | 1 | 1 | 1 | 1 |  |
| Rollout | 1 | 1 | 1 | 1 | 1 |  |
| Clip ratio | 0.2 | 0.2 | 0.2 | 0.2 | 0.2 |  |
| $\lambda$ in GAE | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |  |
| $\gamma$ in GAE | 1 | 1 | 1 | 1 | 1 |  |
| KL coefficient | 0.05 | 0.1 for WebGPT<br>0.05 for others | 0.1 | 0.05 | 0.05 |  |
| Max prompt length | 512 | 512 | 512 | 600 | 600 |  |
| Max response length | 512 | 512 | 512 | 512 | 512 |  |
| Warmup steps | 200 | 200 | 0 | 20 | 20 |  |
|  | Temperature | 0.8 | 0.8 | 0.8 | 1.0 | 1.0 |
|  | Top-$p$ | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
|  | Top-$k$ | 50 | 50 | 50 | 5 | 5 |

Following the procedure used by InstructGPT (Ouyang et al., ), we fine-tune both the SFT model and the reward model on the same dataset to avoid a distribution gap. We implement our training code with the Deepspeed-Chat package (Yao et al., ).

SFT Training   We split the dataset into three parts, allocating 20% of the data in the supervised fine-tuning stage. We use the prompts and the chosen sentences as the instruction data. For the TL;DR Summarize dataset, we concatenate the post and summarization following the approach of Stiennon et al. (). For the single-turn dialogue and the question answering dataset, we apply a human-assistant chat template to format the instructions. For the program synthesis dataset, we format the instruction data in line with Hendrycks et al. ().

Reward Modeling   In this stage, we use 40% of the data to train the reward model for each dataset, formatting the preference data the same way as in the SFT training stage. We initialize the reward model using the fine-tuned SFT model. Due to the lack of preference pairs in the program synthesis dataset, this stage is omitted for this task.

PPO Training   Similar to previous stages, the remaining 40% of the data is used to optimize the policy model. The SFT model initializes the policy model, and the reward model initializes the critic model. For the program synthesis dataset, 80% of the data is used in this stage, with both the policy and critic models initialized using the SFT model. The pass@1 metric serves as the reward signal for program synthesis, compensating for the absence of a reward model. While training 7B model on TL;DR dataset using MA-PPO, we encountered unstable training with a KL coefficient of 0.05. Reducing the coefficient to 0.01 for the 7B model led to more stable optimization.

Table 4 lists the hyperparameters used across all training stages for each task.

### B.3 Notations

**Table 5: List of notation used in this paper.**
| Sym. | Meaning |
| --- | --- |
| RL |  |
| $\mathcal{S}$ | A finite set of states. |
| $\mathcal{A}$ | A finite set of actions. |
| $P$ | The state transition probability distribution. |
| $r$ | The reward function. |
| $\rho_{0}$ | The initial state distribution. |
| $\gamma$ | The discount factor related with future rewards. |
| $\pi_{\theta}(a\mid s)$ | Policy parameterized by $\theta$. |
| $\eta(\pi)$ | The expected cumulative discount reward. |
| $a_{t}$ | The actions selected by the policy. |
| $Q_{\pi}(s_{t},a_{t})$ | The state-action value function. |
| $V_{\pi}(s_{t})$ | The state value function. |
| $A_{\pi}(s_{t},a_{t})$ | The advantage function. |
| $G_{t}$ | The expected return. |
| RLHF |  |
| $r_{\phi}(x,y)$ | The reward model parameterized by $\phi$. |
| $x$ | Prompt. |
| $y^{+}$ | Chosen response. |
| $y^{-}$ | Rejected response. |
| $\beta$ | KL coefficient. |
| $\eta$ | The range for clipping in PPO. |
| $t$ | Time step of tokens. |
| Macro Action |  |
| $\zeta$ | Termination condition. |
| $\mathcal{I}$ | Initiation set. |
| $\tau$ | The index of macro action/state/reward. |
| $\omega_{\tau}$ | Macro action at time step $\tau$. |
| $t_{\tau}$ | Time step of macro actions. |
| $\sigma_{\tau}$ | The weight used to measure the value of macro action. |

In Table 5, we present the notations used in our paper.

Figure: Figure 13: Illustration of four termination rules for macro actions in the MA-RLHF framework. Each termination rule outputs a list of $|\omega_{\tau}|$. In the parsing based termination, the macro action is determined when the token number of the current node is less than $C=4$, which is represented as a number in the tree node.
Refer to caption: x25.png

### B.4 Details of Macro Action Termination

The general form of the segmentation rule is thus $t_{\tau+1}=t_{\tau}+|\omega_{\tau}|$, where $|\omega_{\tau}|$ is determined by the chosen criterion, such as $n$-grams, random, parsing, or perplexity-based segmentation.

- 1.
Fixed $n$-gram length: For all macro actions, we set $|\omega_{\tau}|=n$, where $n$ is a constant value.
- 2.
Randomized $n$-gram length: We define a list of $\{|\omega_{\tau}|\}=\{2,3,5,10\}$ to model macro actions. This list is repeated multiple times to cover the length of the sample, in practice, we repeat this list 3 times. If the total length of macro actions can not match the number of tokens, a large number will be considered as an additional $|\omega_{\tau}|$ to mitigate this gap, which is similar to the $|\omega_{\tau}|=\infty$. We shuffle the list and take this as a random-based length.
- 3.
Parsing-based length: We parse the response into a constituent tree and perform a depth-first search (DFS) to identify macro action length. Two rules guide the termination of $|\omega_{\tau}|$: (1) nodes with fewer than $C$ tokens mark the end of a macro action; (2) nodes with single token are included in the last macro action, avoiding single-token termination conditions like punctuation. Due to differences between the training and parsing tokenizers, we revert to the standard PPO method when discrepancies occur. We set the cut-off threshold $C=5$, providing optimal granularity in practice.
- 4.
Perplexity-based length: Given a response $y$ generated by policy model, we calculate the perplexity $p_{t}$ at any time step $t$ by treating $y_{\leq t}$ as the ground truth response. This process leverages the logits from the reference model, avoiding additional forward passes.
Intuitively, selecting the macro actions based on perplexity $\mathcal{P}=\{p_{0},p_{1},\dots,p_{|y|}\}$ can be defined as selecting tokens which consistently attribute to the decrease of the perplexity given partial sentence. Mathematically, it can be represented as $\omega_{\tau}=\{a_{t_{\tau}},a_{t_{\tau}+1},\dots,a_{t_{\tau}+|\omega_{\tau}|-
1}\}$ where $\mathcal{P}_{t_{\tau}}=\{p_{t_{\tau}},p_{t_{\tau}+1},\dots,p_{t_{\tau}+|\omega
_{\tau}|-1}\}$ exhibits a monotonic decreasing pattern.

### B.5 Training Settings of Program Synthesis

Defining the reward score solely based on the state “Accept” or “Wrong Answer” is somewhat restrictive, as some generated code may pass certain unit tests while failing others. These actions should also receive positive signals to encourage the policy to maximize the number of passed unit tests. To address this, we incorporate an adaptive compiler signal into the reward feedback as previous work (Shojaee et al., ; Liu et al., ):

$$ $R(x)=\left\{\begin{aligned} &-0.3+1.3\cdot\frac{N_{\text{pass}}}{N_{\text{pass }}+N_{\text{fail}}},&\text{if }x\text{ successfully compiled.}\\ &-0.6,&\text{if }x\text{ received runtime error.}\\ &-1.0,&\text{if }x\text{ received compile rrror.}\end{aligned}\right.$ $$

where $x$ represents the code snippet generated by the policy model.

## Appendix C Additional Experiments Results

### C.1 Results of Dialogue Generation

Figure: Figure 14: Test RM scores evaluated by corresponding reward model of Gemma-2B and Gemma-7B model on HH-RLHF dataset.
Refer to caption: x26.png

In Figure 15, we demonstrate the RM scores on the validation set of vanilla PPO and MA-PPO. It shows that MA-PPO surpasses vanilla PPO under RM evaluation, MA-PPO achieves parity performance at 3100 step and 2600 step for 2B and 7B models, respectively, while vanilla PPO at 5100 step and 5400 step. Generally, MA-PPO is 1.6-2x faster than vanilla PPO. Figure 15 compares the RM score distribution of both methods.

### C.2 Results of Question Answering

Figure: Figure 16: Test RM scores evaluated by corresponding reward model of Gemma-2B and Gemma-7B model on the WebGPT Comparisons dataset.
Refer to caption: x29.png

We assess the performance of MA-PPO on the OpenAI WebGPT Comparison dataset, which focuses on the question answering task.

Figure 17 presents the evaluation results based on the reward model. We observe that the policy model is challenging to optimize in this task, likely due to the suboptimal performance of the reward model. We applied early stopping during PPO training since the policy model exhibited reward hacking behavior which generated repetition tokens to inflate higher reward scores towards the end of training. Despite this, evaluations on the saved checkpoints show that MA-PPO still outperforms vanilla PPO across both tested model sizes. The reward score distribution in Figure 17 further confirms that MA-PPO achieves superior reward scores.

When using GPT-4 as the judge, we consider three different metrics to evaluate the answers generated by the policy: factual accuracy, coherence, and usefulness overall, following previous work (Nakano et al., ). The win rates depicted in Figure 4 (Right) show that MA-PPO consistently outperforms the policy trained with vanilla PPO across all criteria. Notably, MA-PPO achieves higher win rates in coherence and usefulness compared to factual accuracy. Human evaluation was conducted to select the preferred answer between those generated by the two policy models. Results in Figure 4 (Right) show that answers produced by MA-PPO were predominantly preferred by human annotators.

## Appendix D Further Analysis

### D.1 Value Function Estimation of Macro Action

When implementing the macro actions, the value function of macro actions is estimated through the value function of tokens. This process can be formulated as: $V^{\pi}(s_{\tau},\omega_{\tau})=\sum_{i=0}^{|\omega_{\tau}|}\sigma_{t_{\tau}+i
}V^{\pi}(s_{t_{\tau}+i},a_{t_{\tau}+i})$, where $\sigma_{\tau}=\{\sigma_{t_{\tau}},\cdots,\sigma_{t_{\tau}+|\omega_{\tau}|}\}$ control the contribution of each value function of tokens.

In this section, we explore several assignments of $\sigma_{\tau}$ and their effectiveness on MA-PPO. Figure 18 illustrates macro action value function with different $\sigma_{\tau}$ assignments:

- 1.
Equal assignment: We treats the contributions of each value function of tokens equally when considering the value function of macro actions, $i.e.$, $\sigma_{\tau}=\{\frac{1}{|\omega_{\tau}|}\}_{i=1}^{\tau}$. This is the naive assignment in MA-PPO used in all our experiments.
- 2.
Unit assignment Since a macro action is a higher-level construct of a sequence of actions, we can use the value function of the last action as the macro action’s value function, where $\sigma_{\tau}=\{0,0,\cdots,0,1\}$.
- 3.
Position decayed assignment The contributions of each value function of tokens are determined by taking the position into consideration. We define $\sigma_{\tau}$ based on the position of the token, *i.e.*, $\sigma_{\tau}=\{\frac{1}{(|\omega_{\tau}|-i)\cdot\mathcal{H}}\}_{i=0}^{|\omega
_{\tau}|-1}$, where $\mathcal{H}=\sum_{i=0}^{|\omega_{\tau}|-1}\frac{1}{(|\omega_{\tau}|-i)}$, this construction ensures $\sum_{\sigma\in\sigma_{\tau}}\sigma=1$.

We tested these approaches with fixed $n$-gram based termination on TL;DR dataset, with $n=5$. We report the RM score and GPT-4 score as previous. Results in Figure 19 show that the equal assignment yields higher RM scores. However, the unit assignment achieves the best consistency and fluency according to GPT-4 evaluations.

Figure: Figure 18: Illustration of value function of macro actions in MA-RLHF framework. It takes the outputs from the value function of tokens as input, and returns the value of macro actions with different $\sigma_{\tau}$ assignment.
Refer to caption: x32.png

Figure: Figure 19: Performance of MA-PPO with different value function estimations in MA-PPO on TL;DR dataset for Gemma-2B model. Left test RM scores. Right GPT-4 scores on 4 dimensions.
Refer to caption: x33.png

### D.2 Impact of RLHF on Reward Score Distribution

Figure: Figure 20: RM score shifting pattern after RLHF training. Left presents the RM score of best of 8 sampling on vanilla PPO compared to the vanilla PPO. Right presents the RM score of best of 8 sampling on MA-PPO compared to the MA-PPO.
Refer to caption: x35.png

Figure: Figure 21: Test reward scores evaluated by the corresponding reward model for summarizations generated with different sampling temperature on the TL;DR dataset.
Refer to caption: x37.png

We also apply Best-of-$N$ sampling on both vanilla PPO and MA-PPO. The RM score shifting patterns for these methods are illustrated in Figure 20. From the results, we can conclude that Best-of-$N$ sampling continues to enhance the performance of RLHF models effectively.

Figure: Figure 22: Illustration of the macro action-RLHF (MA-RLHF) framework.
Refer to caption: x39.png

### D.3 Impact of Sampling Temperature

In the previous experiments, the results were sampled with a temperature $temp=0.8$ to align with the sampling strategy used during training. In this section, we examine the effect of sampling temperature on response quality. We vary the temperature $temp\in\{0.0,0.2,0.4,0.6,0.8,1.0\}$, and report the results in Figure 21. The performance of both methods remains stable when $temp<0.8$. However, the performance of vanilla PPO begins to decline after $temp=0.8$, whereas MA-PPO continues to demonstrate stable performance, even at $temp=1.0$.

Figure: Algorithm 1 Framework of Macro Action RLHF.

## Appendix E MA-RLHF Algorithms

Figure 22 illustrate the framework of MA-RLHF. In practice, to implement MA-RLHF, once the macro actions are obtained via the termination function, we compute their value (as estimated by the critic model) and rewards (based on a per-token KL penalty) using the value function estimation. With these values and rewards, we apply Generalized Advantage Estimation (GAE) without modification to derive advantage estimates and state-action value functions. These advantage estimates and state-action value functions are then used to all tokens within the macro action during the optimization of both the policy and critic models. The macro action RLHF algorithm, utilizing PPO, is detailed in Algorithm 1.

In this implementation, the introduced additional time complexity is in the option termination. While fixed $n$-gram based, randomized $n$-gram based, and perplexity based terminations achieves same time complexity, the time complexity of parsing based termination is related to the constituent tree which we applied DFS to obtain $|\omega_{\tau}|$. During the inference stage, our MA-PPO will not introduce additional complexity since it only works at the training stage.

We provide the Pytorch code for implementation of the macro action in PPO below:

| Obtain Macro Action Positions |
| --- |
| [⬇](data:text/plain;base64,ZGVmIGdldF9tYWNyb19hY3Rpb25fcG9zaXRpb25zKHNlbGYsIHN0YXJ0LCBtYXNrLCB0ZXJtaW5hdGlvbj0nbmdyYW0nLCBuX2dyYW06IGludD1Ob25lLCBwcGw6IExpc3RbdG9yY2guZmxvYXQxNl09Tm9uZSwgcmVwZWF0X3RpbWVzOiBpbnQ9Tm9uZSwgY3V0b2ZmOiBpbnQ9Tm9uZSk6CiAgICBzZXF1ZW5jZSA9IFtzdGFydF0KICAgIGlmIHRlcm1pbmF0aW9uID09ICduZ3JhbSc6CiAgICAgICAgYXNzZXJ0IG5fZ3JhbSBpcyBub3QgTm9uZQogICAgICAgIGN1cnJlbnRfY291bnQgPSAwCiAgICAgICAgZm9yIGkgaW4gcmFuZ2UobWFza1s6LCBzdGFydDpdLnNpemUoMSkgLSAxKToKICAgICAgICAgICAgY3VycmVudF9jb3VudCArPSBtYXNrWzAsIHN0YXJ0ICsgaV0uaXRlbSgpCiAgICAgICAgICAgIGlmIGN1cnJlbnRfY291bnQgPT0gbl9ncmFtOgogICAgICAgICAgICAgICAgc2VxdWVuY2UuYXBwZW5kKHN0YXJ0ICsgaSArIDEpCiAgICAgICAgICAgICAgICBjdXJyZW50X2NvdW50ID0gMAogICAgZWxpZiB0ZXJtaW5hdGlvbiA9PSAncmFuZG9taXplZF9uZ3JhbSc6CiAgICAgICAga19saXN0ID0gdG9yY2gudGVuc29yKFsyLCAzLCA1LCAxMF0sIGR0eXBlPWludCkKICAgICAgICBrX2xpc3QgPSB0b3JjaC5yZXBlYXRfaW50ZXJsZWF2ZShrX2xpc3QsIDMpCiAgICAgICAga19saXN0ID0ga19saXN0W3RvcmNoLnJhbmRwZXJtKGtfbGlzdC5zaXplKClbLTFdKV0KICAgICAgICBpbmRleGVkX2tfbGlzdCA9IHRvcmNoLmN1bXN1bShrX2xpc3QsIGRpbT0tMSkKCiAgICAgICAgc2VxdWVuY2UgPSBbbiBmb3IgbiBpbiByYW5nZShzdGFydCwgbWFza1s6LCBzdGFydDpdLnNpemUoMSkgLSAxKV0KICAgICAgICBpbmRleGVkX2tfbGlzdCA9IFt4Lml0ZW0oKSBmb3IgeCBpbiBpbmRleGVkX2tfbGlzdCBpZiB4Lml0ZW0oKSA8IGxlbihzZXF1ZW5jZSldCiAgICAgICAgc2VxdWVuY2UgPSBbc3RhcnRdICsgW3NlcXVlbmNlW2ldIGZvciBpIGluIGluZGV4ZWRfa19saXN0XQogICAgZWxpZiB0ZXJtaW5hdGlvbiA9PSAncHBsJzoKICAgICAgICBhc3NlcnQgcHBsIGlzIG5vdCBOb25lCiAgICAgICAgZm9yIGkgaW4gcmFuZ2UoMSwgbGVuKHBwbCkpOgogICAgICAgICAgICBpZiBwcGxbaV0gPiBwcGxbaSAtIDFdOgogICAgICAgICAgICAgICAgc2VxdWVuY2UuYXBwZW5kKHN0YXJ0ICsgaSkKICAgIGVsaWYgdGVybWluYXRpb24gPT0gJ3BhcnNlcic6CiAgICAgICAgaWYgbGVuKG5vZGUubGVhdmVzKCkpIDwgMToKICAgICAgICAgICAgcmV0dXJuIEZhbHNlLCBtYV9sZW5ndGggKyAxCiAgICAgICAgaWYgbGVuKG5vZGUubGVhdmVzKCkpIDwgY3V0b2ZmOgogICAgICAgICAgICBzZXF1ZW5jZS5hcHBlbmQobWFfbGVuZ3RoICsgbm9kZS5sZWF2ZXMoKSkKICAgICAgICAgICAgcmV0dXJuIFRydWUsIG1hX2xlbmd0aCArIG5vZGUubGVhdmVzKCkKICAgICAgICBmb3Igbnh0X25vZGUgaW4gbm9kZS5jaGlsZHMoKToKICAgICAgICAgICAgc3RhdGUsIG1hX2xlbmd0aF8gPSBkZnMobnh0X25vZGUsIG1hX2xlbmd0aCkKICAgICAgICAgICAgaWYgIXN0YXRlOgogICAgICAgICAgICAgICAgc2VxdWVuY2VbLTFdID0gbWFfbGVuZ3RoXwogICAgICAgICAgICBtYV9sZW5ndGggPSBtYV9sZW5ndGhfCiAgICAgICAgcmV0dXJuIFRydWUsIG1hX2xlbmd0aAogICAgc2VxdWVuY2UuYXBwZW5kKGludChtYXNrLnNpemUoMSkgLSAxKSkKICAgIHJldHVybiBzZXF1ZW5jZQ==)<br>def get_macro_action_positions(self, start, mask, termination=’ngram’, n_gram: int=None, ppl: List[torch.float16]=None, repeat_times: int=None, cutoff: int=None):<br>sequence = [start]<br>if termination == ’ngram’:<br>assert n_gram is not None<br>current_count = 0<br>for i in range(mask[:, start:].size(1) - 1):<br>current_count += mask[0, start + i].item()<br>if current_count == n_gram:<br>sequence.append(start + i + 1)<br>current_count = 0<br>elif termination == ’randomized_ngram’:<br>k_list = torch.tensor([2, 3, 5, 10], dtype=int)<br>k_list = torch.repeat_interleave(k_list, 3)<br>k_list = k_list[torch.randperm(k_list.size()[-1])]<br>indexed_k_list = torch.cumsum(k_list, dim=-1)<br>sequence = [n for n in range(start, mask[:, start:].size(1) - 1)]<br>indexed_k_list = [x.item() for x in indexed_k_list if x.item() < len(sequence)]<br>sequence = [start] + [sequence[i] for i in indexed_k_list]<br>elif termination == ’ppl’:<br>assert ppl is not None<br>for i in range(1, len(ppl)):<br>if ppl[i] > ppl[i - 1]:<br>sequence.append(start + i)<br>elif termination == ’parser’:<br>if len(node.leaves()) < 1:<br>return False, ma_length + 1<br>if len(node.leaves()) < cutoff:<br>sequence.append(ma_length + node.leaves())<br>return True, ma_length + node.leaves()<br>for nxt_node in node.childs():<br>state, ma_length_ = dfs(nxt_node, ma_length)<br>if !state:<br>sequence[-1] = ma_length_<br>ma_length = ma_length_<br>return True, ma_length<br>sequence.append(int(mask.size(1) - 1))<br>return sequence |
| Calculate Values / Rewards of Macro Action |
| [⬇](data:text/plain;base64,ZGVmIGdldF9tYWNyb19hY3Rpb25fdmFsdWVzKHNlbGYsIHZhbHVlcywgbWFzaywgc3RhcnQsIHNlcXVlbmNlKToKICAgIHNwbGl0X2xpc3QgPSB0b3JjaC5kaWZmKHRvcmNoLnRlbnNvcihzZXF1ZW5jZSkpLnRvbGlzdCgpCiAgICBzcGxpdGVkX3ZhbHVlcyA9IHRvcmNoLnNwbGl0KHZhbHVlc1s6LCBzdGFydDpdLCBzcGxpdF9saXN0LCBkaW09LTEpCiAgICBzcGxpdGVkX21hc2sgPSB0b3JjaC5zcGxpdChtYXNrWzosIHN0YXJ0Ol0sIHNwbGl0X2xpc3QsIGRpbT0tMSkKICAgIGlucGxhY2VfdmFsdWVzID0gdG9yY2guemVyb3MoMSwgbGVuKHNwbGl0X2xpc3QpLCBkdHlwZT12YWx1ZXMuZHR5cGUpLnRvKHZhbHVlcy5kZXZpY2UpCiAgICBmb3IgaWR4LCAodmFsdWVfaSwgbWFza19pKSBpbiBlbnVtZXJhdGUoemlwKHNwbGl0ZWRfdmFsdWVzLCBzcGxpdGVkX21hc2spKToKICAgICAgICBtYXNrZWRfdmFsdWVzID0gdmFsdWVfaVttYXNrX2kgIT0gMF0KICAgICAgICBpbnBsYWNlX3ZhbHVlc1swLCBpZHhdID0gdG9yY2gubWVhbihtYXNrZWRfdmFsdWVzKSBpZiBtYXNrZWRfdmFsdWVzLm51bWVsKCkgPiAwIGVsc2UgMC4wCiAgICByZXR1cm4gaW5wbGFjZV92YWx1ZXM=)<br>def get_macro_action_values(self, values, mask, start, sequence):<br>split_list = torch.diff(torch.tensor(sequence)).tolist()<br>splited_values = torch.split(values[:, start:], split_list, dim=-1)<br>splited_mask = torch.split(mask[:, start:], split_list, dim=-1)<br>inplace_values = torch.zeros(1, len(split_list), dtype=values.dtype).to(values.device)<br>for idx, (value_i, mask_i) in enumerate(zip(splited_values, splited_mask)):<br>masked_values = value_i[mask_i != 0]<br>inplace_values[0, idx] = torch.mean(masked_values) if masked_values.numel() > 0 else 0.0<br>return inplace_values |

| Calculate Policy Model Loss |
| --- |
| [⬇](data:text/plain;base64,ZGVmIHBvbGljeV9sb3NzX21hY3JvX2FjdGlvbihzZWxmLCBsb2dwcm9icywgb2xkX2xvZ3Byb2JzLCBhZHZhbnRhZ2VzLCBtYXNrLCBzZXF1ZW5jZSk6CiAgICBsb2dfcmF0aW8gPSAobG9ncHJvYnMgLSBvbGRfbG9ncHJvYnMpICogbWFzawogICAgcmF0aW8gPSB0b3JjaC5leHAobG9nX3JhdGlvKQoKICAgICMgY2FsY3VsYXRlIGxvc3Mgd2l0aCBtYWNybyBhY3Rpb24KICAgIHNwbGl0X2xpc3QgPSB0b3JjaC5kaWZmKHRvcmNoLnRlbnNvcihzZXF1ZW5jZSkpLnRvbGlzdCgpCiAgICBzcGxpdF9yYXRpbyA9IHRvcmNoLnNwbGl0KHJhdGlvLCBzcGxpdF9saXN0LCBkaW09LTEpCiAgICBzcGxpdF9tYXNrID0gdG9yY2guc3BsaXQobWFzaywgc3BsaXRfbGlzdCwgZGltPS0xKQoKICAgIHBnX2xvc3MgPSAwLjAKICAgIHRvdGFsX21hc2tfc3VtID0gMC4wCiAgICBmb3IgaSBpbiByYW5nZShsZW4oc3BsaXRfbGlzdCkpOgogICAgICAgIHJhdGlvX2kgPSBzcGxpdF9yYXRpb1tpXQogICAgICAgIG1hc2tfaSA9IHNwbGl0X21hc2tbaV0KICAgICAgICBhZHZhbnRhZ2VzX2kgPSBhZHZhbnRhZ2VzWzosIGldCgogICAgICAgIHBnX2xvc3MxID0gLWFkdmFudGFnZXNfaSAqIHJhdGlvX2kKICAgICAgICBwZ19sb3NzMiA9IC1hZHZhbnRhZ2VzX2kgKiB0b3JjaC5jbGFtcChyYXRpb19pLCAxLjAgLSBzZWxmLmNsaXByYW5nZSwgMS4wICsgc2VsZi5jbGlwcmFuZ2UpCiAgICAgICAgcGdfbG9zcyArPSB0b3JjaC5zdW0odG9yY2gubWF4KHBnX2xvc3MxLCBwZ19sb3NzMikgKiBtYXNrX2kpCiAgICAgICAgdG90YWxfbWFza19zdW0gKz0gbWFza19pLnN1bSgpCiAgICBwZ19sb3NzID0gcGdfbG9zcyAvIHRvdGFsX21hc2tfc3VtCiAgICByZXR1cm4gcGdfbG9zcw==)<br>def policy_loss_macro_action(self, logprobs, old_logprobs, advantages, mask, sequence):<br>log_ratio = (logprobs - old_logprobs) * mask<br>ratio = torch.exp(log_ratio)<br># calculate loss with macro action<br>split_list = torch.diff(torch.tensor(sequence)).tolist()<br>split_ratio = torch.split(ratio, split_list, dim=-1)<br>split_mask = torch.split(mask, split_list, dim=-1)<br>pg_loss = 0.0<br>total_mask_sum = 0.0<br>for i in range(len(split_list)):<br>ratio_i = split_ratio[i]<br>mask_i = split_mask[i]<br>advantages_i = advantages[:, i]<br>pg_loss1 = -advantages_i * ratio_i<br>pg_loss2 = -advantages_i * torch.clamp(ratio_i, 1.0 - self.cliprange, 1.0 + self.cliprange)<br>pg_loss += torch.sum(torch.max(pg_loss1, pg_loss2) * mask_i)<br>total_mask_sum += mask_i.sum()<br>pg_loss = pg_loss / total_mask_sum<br>return pg_loss |
| Calculate Critic Model Loss |
| [⬇](data:text/plain;base64,ZGVmIGNyaXRpY19sb3NzX21hY3JvX2FjdGlvbihzZWxmLCB2YWx1ZXMsIG9sZF92YWx1ZXMsIHJldHVybnMsIG1hc2ssIHNlcXVlbmNlKToKICAgIHZhbHVlc19jbGlwcGVkID0gdG9yY2guY2xhbXAoCiAgICAgICAgdmFsdWVzLAogICAgICAgIG9sZF92YWx1ZXMgLSBzZWxmLmNsaXByYW5nZV92YWx1ZSwKICAgICAgICBvbGRfdmFsdWVzICsgc2VsZi5jbGlwcmFuZ2VfdmFsdWUsCiAgICApCiAgICBpZiBzZWxmLmNvbXB1dGVfZnAzMl9sb3NzOgogICAgICAgIHZhbHVlcyA9IHZhbHVlcy5mbG9hdCgpCiAgICAgICAgdmFsdWVzX2NsaXBwZWQgPSB2YWx1ZXNfY2xpcHBlZC5mbG9hdCgpCgogICAgIyBjYWxjdWxhdGUgbG9zcyB3aXRoIG1hY3JvIGFjdGlvbgogICAgc3BsaXRfbGlzdCA9IHRvcmNoLmRpZmYodG9yY2gudGVuc29yKHNlcXVlbmNlKSkudG9saXN0KCkKICAgIHNwbGl0ZWRfdmFsdWVzID0gdG9yY2guc3BsaXQodmFsdWVzLCBzcGxpdF9saXN0LCBkaW09LTEpCiAgICBzcGxpdGVkX3ZhbHVlc19jbGlwcGVkID0gdG9yY2guc3BsaXQodmFsdWVzX2NsaXBwZWQsIHNwbGl0X2xpc3QsIGRpbT0tMSkKICAgIHNwbGl0ZWRfbWFzayA9IHRvcmNoLnNwbGl0KG1hc2ssIHNwbGl0X2xpc3QsIGRpbT0tMSkKCiAgICB0b3RhbF92Zl9sb3NzID0gMC4wCiAgICB0b3RhbF9tYXNrX3N1bSA9IDAuMAoKICAgIGZvciBpIGluIHJhbmdlKGxlbihzcGxpdGVkX3ZhbHVlcykpOgogICAgICAgIHZmX2xvc3MxID0gKHNwbGl0ZWRfdmFsdWVzW2ldIC0gcmV0dXJuc1s6LCBpXSkqKjIKICAgICAgICB2Zl9sb3NzMiA9IChzcGxpdGVkX3ZhbHVlc19jbGlwcGVkW2ldIC0gcmV0dXJuc1s6LCBpXSkqKjIKICAgICAgICB2Zl9sb3NzID0gMC41ICogdG9yY2guc3VtKAogICAgICAgICAgICB0b3JjaC5tYXgodmZfbG9zczEsIHZmX2xvc3MyKSAqIHNwbGl0ZWRfbWFza1tpXSkKICAgICAgICB0b3RhbF92Zl9sb3NzICs9IHZmX2xvc3MKICAgICAgICB0b3RhbF9tYXNrX3N1bSArPSBzcGxpdGVkX21hc2tbaV0uc3VtKCkKICAgIHRvdGFsX3ZmX2xvc3MgPSB0b3RhbF92Zl9sb3NzIC8gdG90YWxfbWFza19zdW0KICAgIHJldHVybiB0b3RhbF92Zl9sb3Nz)<br>def critic_loss_macro_action(self, values, old_values, returns, mask, sequence):<br>values_clipped = torch.clamp(<br>values,<br>old_values - self.cliprange_value,<br>old_values + self.cliprange_value,<br>)<br>if self.compute_fp32_loss:<br>values = values.float()<br>values_clipped = values_clipped.float()<br># calculate loss with macro action<br>split_list = torch.diff(torch.tensor(sequence)).tolist()<br>splited_values = torch.split(values, split_list, dim=-1)<br>splited_values_clipped = torch.split(values_clipped, split_list, dim=-1)<br>splited_mask = torch.split(mask, split_list, dim=-1)<br>total_vf_loss = 0.0<br>total_mask_sum = 0.0<br>for i in range(len(splited_values)):<br>vf_loss1 = (splited_values[i] - returns[:, i])**2<br>vf_loss2 = (splited_values_clipped[i] - returns[:, i])**2<br>vf_loss = 0.5 * torch.sum(<br>torch.max(vf_loss1, vf_loss2) * splited_mask[i])<br>total_vf_loss += vf_loss<br>total_mask_sum += splited_mask[i].sum()<br>total_vf_loss = total_vf_loss / total_mask_sum<br>return total_vf_loss |

| PPO |
| --- |
| [⬇](data:text/plain;base64,IyBJbiBQUE8gYWxnb3JpdGhtCnN0YXJ0ID0gcHJvbXB0cy5zaXplKClbLTFdIC0gMQphY3Rpb25fbWFzayA9IGF0dGVudGlvbl9tYXNrWzosIDE6XQouLi4Kc2VxdWVuY2UgPSBnZXRfbWFjcm9fYWN0aW9uX3Bvc2l0aW9ucyhzdGFydCwgYWN0aW9uX21hc2ssIHRlcm1pbmF0aW9uPSduZ3JhbScsIG5fZ3JhbT1uX2dyYW0pCm1hY3JvX2FjdGlvbl9vbGRfdmFsdWVzID0gZ2V0X21hY3JvX2FjdGlvbl92YWx1ZXMob2xkX3ZhbHVlcywgYWN0aW9uX21hc2ssIHN0YXJ0LCBzZXF1ZW5jZSkKbWFjcm9fYWN0aW9uX29sZF9yZXdhcmRzID0gZ2V0X21hY3JvX2FjdGlvbl92YWx1ZXMob2xkX3Jld2FyZHMsIGFjdGlvbl9tYXNrLCBzdGFydCwgc2VxdWVuY2UpCmFkdmFudGFnZXMsIHJldHVybnMgPSBnZXRfYWR2YW50YWdlc19hbmRfcmV0dXJucyhzdW1lZF9vbGRfdmFsdWVzLCBzdW1lZF9vbGRfcmV3YXJkcykKCnBvbGljeV9sb3NzID0gcG9saWN5X2xvc3NfbWFjcm9fYWN0aW9uKHBvbGljeV9sb2dfcHJvYls6LCBzdGFydDpdLCBsb2dfcHJvYnNbOiwgc3RhcnQ6XSwgYWR2YW50YWdlcywgYWN0aW9uX21hc2tbOiwgc3RhcnQ6XSwgc2VxdWVuY2UpCmNyaXRpY19sb3NzID0gY3JpdGljX2xvc3NfbWFjcm9fYWN0aW9uKHZhbHVlWzosIHN0YXJ0Ol0sIG9sZF92YWx1ZXNbOiwgc3RhcnQ6XSwgcmV0dXJucywgYWN0aW9uX21hc2tbOiwgc3RhcnQ6XSwgc2VxdWVuY2Up)<br># In PPO algorithm<br>start = prompts.size()[-1] - 1<br>action_mask = attention_mask[:, 1:]<br>...<br>sequence = get_macro_action_positions(start, action_mask, termination=’ngram’, n_gram=n_gram)<br>macro_action_old_values = get_macro_action_values(old_values, action_mask, start, sequence)<br>macro_action_old_rewards = get_macro_action_values(old_rewards, action_mask, start, sequence)<br>advantages, returns = get_advantages_and_returns(sumed_old_values, sumed_old_rewards)<br>policy_loss = policy_loss_macro_action(policy_log_prob[:, start:], log_probs[:, start:], advantages, action_mask[:, start:], sequence)<br>critic_loss = critic_loss_macro_action(value[:, start:], old_values[:, start:], returns, action_mask[:, start:], sequence) |

## Appendix F Evaluation Details

### F.1 GPT-4 Evaluation Prompts

In our experiments, we take GPT-4 as a main judgment of the quality of policy models. The prompts used to generate win rates using GPT-4 are listed below. We utilize the gpt-4o-05-13 for all of our experiments. The order of the responses generated by policy models is randomly chosen for all experiments.

### F.2 Human Evaluation

To estimate the quality from a human perspective, we collect human preference data on the TL;DR, HH-RLHF, and WebGPT datasets. Human annotators select the preferred response based on task-specific criteria. For TL;DR, the evaluation criteria focus on three main perspectives:

- 1.
Hallucination: this considers whether the generated summary includes any additional information not present in the original post or article.
- 2.
Verbosity: this assesses if the summary includes unnecessary context that could be removed without negatively impacting its quality.
- 3.
Overall Quality: this measures the general coherence, informativeness, and readability of the generated summary.

In the context of HH-RLHF, annotators focus on the helpfulness of the responses. They are instructed to choose the response that provides more useful information for a given dialogue, while excluding superficial replies such as ”You’re welcome.” For the WebGPT dataset, the primary evaluation factor is factual accuracy. Annotators are provided with retrieval information relevant to the question from the dataset to aid in their judgment. They are tasked with selecting the answer that most accurately matches the retrieved information.

During the evaluation process, annotators are presented with a prompt and two responses, each generated by either vanilla PPO or MA-PPO. To ensure impartiality and prevent annotators from guessing which model produced which response, we shuffle the positions of the responses. Annotators are given three choices: response A wins, response B wins, or a tie. The results are then collected to calculate the win rates for each model.

For evaluations on the TL;DR and HH-RLHF datasets using 7B models, we conduct the human evaluation with 3 different annotators and collect their preference data to report the win rates. For all other human evaluations, we conduct them with a single annotator.

## Appendix G Generated Examples

### G.1 Case Study

When evaluating the responses of MA-RLHF with human annotators, we observe that the MA-RLHF exhibits coherence and contextual appropriate abilities. We illustrate this phenomenon with an example by comparing MA-RLHF with the baseline in Table 6. We found that the MA-PPO method tends to generate responses with phrases more than the baseline method. Specifically, for nouns, it commonly includes adjectives for modification to make the generated summaries more accurate, such as “feeding indoor cat food”.

**Table 6: Example response from TL;DR dataset to illustrate the responses generated by MA-RLHF is more coherence and contextual appropriate compared to baseline. We highlight the context that we observed with green.**
| Prompt | POST Subreddit: r/cats<br>Hello everyone! One of my cats is about 10 years old now, she is pretty much strictly indoors save for some time she spends on our screened in porch each day. (She likes to watch the birds in the yard while she suns herself by the pool, quite the princess). Anyway, when she was younger she was very active and quite small, however with age she has put on a pretty hefty amount of weight. I feed her indoor cat food for weight control, I’ve switched brands a few times trying to find something that works, I’ve cut back on feeding her by a lot (she gets very angry and demanding when she wants food but I don’t give in) however, nothing really seems to work. I’ve tried cat toys, and bought a harness thinking I could try to walk her but she just lays down and looks at me like I’m stupid. Basically I just want to know if you all have any suggestions for exercise or food. I care about her and don’t want this to get any worse. I also have another cat that eats the same amount and type of food as her and is a completely normal weight and only a year younger, however he is a male, not sure if that makes a difference in predisposition for weight gain. They are also both fixed. |
| --- | --- |
| Vanilla PPO | 10 year old cat has gained a noticeable amount of weight and is becoming increasingly inactive. Looking for suggestions for exercise or food to help her get healthier. Male and female cats are both affected. |
| Fixed 3-gram | 10 year old cat put on hefty amount of weight despite feeding indoor cat food for weight control. Any suggestions for exercise or food to help with weight control and prevent further weight gain? Specifically wondering if there are any suggestions for cats that are both female and fixed. |
| Fixed 5-gram | 10 year old indoor cat has put on significant amount of weight despite being fed low calorie/weight control food. Need suggestions for exercise or food to help with weight management. Also have another cat that eats same type/amount of food as her and is a completely normal weight and only a year younger, however he is a male. Any insights would be appreciated. |
| Fixed 10-gram | 10 year old cat is putting on significant weight despite feeding indoor cat food for weight control. Any suggestions for exercise or food to help her lose weight and keep her healthy? Also, is there a difference in predisposition for weight gain between male and female cats when fed same amount/type of food? |
| Fixed $\infty$-gram | One 10 year old cat has put on hefty amount of weight with age, feed indoor cat food for weight control, tried various brands and cut back feeding by lot, nothing really seems to work in terms of exercise or food, also have another cat same age and weight as her and completely normal weight and no issues currently however male and not sure if that makes difference predisposition for weight gain, both fixed. |
| Randomized n-gram | 10 year old cat has put on a lot of weight, feed her indoor cat food for weight control yet nothing seems to work. Any suggestions for exercise or food to address this? Other cat that eats same amount and type of food as her is a normal weight and a year younger, however is male, not sure if that makes a difference in predisposition for weight gain. |
| Perplexity based | 10 yo cat has put on significant amount of weight despite diet change and other attempts at exercise/activity. Any suggestions for exercise or food to help her lose weight? Also, any differences between males and females regarding predisposition for weight gain? |
| Parsing based | 10 year old indoor cat has gained significant amount of weight despite proper diet and exercise, potentially due to age related health issues. Need suggestions for exercise or food changes to help reduce weight and maintain proper body weight. Is weight gain common in older cats, and does gender play a factor in predisposition? |

### G.2 Example Responses

In this section, we demonstrate some examples of validation sets to highlight the superiority of MA-PPO. In Table 7, we feature examples from the TL;DR dataset. Compared to the responses generated by vanilla PPO, the responses from MA-PPO offer more concise and relevant details about the situation. Table 8 showcases a dialogue focusing on the differences between Spanish and Italian. MA-PPO provides a more helpful response, addressing both cultural and grammatical perspectives. In Table 9, we compare answers from the WebGPT dataset. The responses provided by MA-PPO are more factual and accurate compared to those generated by vanilla PPO.

**Table 7: Example responses from TL;DR dataset. The summarization of MA-PPO exactly capture the details of the post.**
| Prompt | POST Subreddit: r/running<br>Last night I experienced my first ever absolutely miserable run where instead of feeling energized and accomplished when it was over, I lay on my bathroom floor shivering and hugging the toilet.<br>I have had frustrating runs before and I have had runs when my knee/ankle/toe hurt and made me want to quit but last night was far worse than any of those.<br>Earlier in the day, I ate a sandwich and fell asleep on the couch for a two hour surprise nap. When I woke up my stomach was slightly unsettled like I was about to have heartburn. I decided to do out for my long run anyways because I always have heartburn and it doesn’t usually stop me. This run was scheduled to be my longest yet at 1:30:00 (I am training for a half).<br>At about 45 minutes I started to get stomach cramps so I slowed how much water I was taking in. The entire run I had to walk several times which is unusual for me but my times were better than expected so I thought I was just pushing too hard and tried to take it easy on the way back.<br>My post long run ritual is usually to drink several long drinks once I get back to the car since now I won’t be jostling my innards around, I can do what I want. When I got home, I felt like I probably shouldn’t try eating right away so I had about 1/3 bottle of Gatorade.<br>Well…Gatorade was the wrong answer. Within 20 minutes I was violently heaving into my kitchen garbage and I lost all of the Gatorade and likely most of the water I had while running.<br>My question is, do I blame my sandwich from earlier or my run? I am worried that 1:30:00 is my limit. I haven’t considered quitting my half marathon training until now because everything up to this point has been fine. |
| --- | --- |
| Vanilla PPO 2B<br>RM Score: 0.75 | 1:30:00 run was extremely stressful and painful. Do I blame sandwich from earlier or run itself? I am worried that 1:30:00 is my limit. |
| MA-PPO 2B<br>RM Score: 1.55 | 1:30:00 long run resulted in severe stomach cramps and forced heavy heaving into kitchen garbage. Do I blame my sandwich from earlier or my run? Concerned that 1:30:00 is my limit. |
| Vanilla PPO 7B<br>RM Score: 1.71 | 1:30:00 run caused serious stomach pains and I puked up all of the water I drank while running. Do I blame the sandwich for my stomach pain or the run itself? |
| MA-PPO 7B<br>RM Score: 2.61 | 1:30:00 long run, had stomach cramps and had to walk often, drank Gatorade post run, vomited violently within 20 minutes, now worried that 1:30 is my limit, do I blame sandwich or run? Worried about quitting half marathon training. Help? |
| Vanilla PPO 27B<br>RM Score: 3.45 | 45 minutes into my run my stomach started to cramp and I was vomiting profusely within 20 minutes of finishing my 1:30:00 long run. Do I blame my sandwich from earlier or my run? Does this mean my limit is 1:30:00 or is it something else? |
| MA-PPO 27B<br>RM Score: 3.95 | I went out for a 1:30:00 long run yesterday and came home with stomach cramps and nausea. I drank 1/3 bottle of Gatorade when I got home and got violently sick. I lost that and most of the water I drank while running. Do I blame my sandwich from earlier or my run? I am worried that 1:30:00 is my limit and that I won’t be able to finish my half marathon training. |

**Table 8: Example responses from HH-RLHF dataset, MA-PPO method presents more helpful response compared to vanilla PPO, and is prefered by GPT-4 and human.**
| Prompt | Human: Would you be able to explain the differences between the Spanish and Italian language? Assistant: Of course. Can you tell me more about the specific areas where you’re interested in knowing more? Human: I’m thinking between the Spanish spoken in Mexico and Italian spoken in Italy. Assistant: |
| --- | --- |
| Vanilla PPO 2B<br>RM Score: 1.07 | The main difference between Spanish and Italian is that Spanish is an Indo-European language with a very rich historical background, while Italian is an Italian Romance language derived from Latin. Spanish has influences from many different languages, while Italian has fewer and more limited influences from other languages. |
| MA-PPO 2B<br>RM Score: 1.24 | Here are some of the main differences between Spanish spoken in Mexico and Italian spoken in Italy: 1. Spanish spoken in Mexico tends to be more relaxed and casual, while Italian spoken in Italy tends to be more formal and courteous. 2. Spanish in Mexico is heavily influenced by indigenous languages and has a rich vocabulary, while Italian is mostly derived from Latin and has a more simple and elegant sound. 3. Spanish is spoken throughout Latin America, while Italian is confined to the Italian peninsula. Overall, Spanish and Italian have many subtle differences that contribute to their unique cultural identities. |
| Vanilla PPO 7B<br>RM Score: 0.80 | The main differences between the Spanish spoken in Mexico and the Italian spoken in Italy are in the pronunciation and grammar of the two languages. In Mexico, the Spanish spoken is much more similar to the Spanish spoken in Spain. In Italy, the Italian spoken is similar to the Italian spoken in other parts of Europe, but with its own unique style of pronunciation. In both languages, the grammar is very similar, though the Italian language does have some unique grammar rules. |
| MA-PPO 7B<br>RM Score: 1.22 | Sure, here are some of the main differences between the Spanish spoken in Mexico and the Italian spoken in Italy: - Spanish is a latin language, which is a language that was derived from the latin language of ancient Rome. Italian is based on the ancient language of the romans of ancient Rome. - Spanish is spoken in a number of different countries throughout the world, whereas Italian is the native language of Italy, and is only spoken in some parts of northern Italy. - Spanish is spoken primarily in Latin America, and has taken on many local variations and differences due to the influence of local indigenous languages (i.e. Mexican Spanish is quite different from Spanish spoken in Argentina). Italian is primarily spoken in Italy and has more localized variations. - There are many differences in grammar, vocabulary, and pronunciation between Spanish and Italian that are reflected in the differences between Mexican Spanish and Italian. Some of the most important differences in grammar include the use of masculine and feminine gender to refer to objects, and the different use of the tenses. There are many differences in vocabulary that also exist between the two languages, and the pronunciation of each language also varies. |

**Table 9: Example responses from WebGPT dataset.**
| Prompt | What causes that sudden sinking, heavy, hot flush feeling when you become anxious and how are the effects felt so rapidly? |
| --- | --- |
| Vanilla PPO 2B<br>RM Score: -0.68 | Anxiety has been linked to several physical symptoms, including a rapid heartbeat, trembling, dizziness, and a sense of impending doom. While some of these symptoms may be due to the anxiety-provoking situation causing the symptoms, some may be the result of an anxiety disorder itself. The symptoms that result from anxiety can be divided into physical symptoms, emotional symptoms, and behavioral symptoms. Physical symptoms of anxiety include nausea, dizziness, and a queasy or empty feeling in the stomach. Emotional symptoms include feeling helpless, hopeless, afraid, irritable, and worried. Behavioral symptoms include avoiding social interactions, remaining in bed all day, and not performing daily activities. |
| MA-PPO 2B<br>RM Score: -0.41 | The “hot flush” sensation that some people experience when they become anxious is the result of the nervous system’s fight-or-flight response. When the body is faced with a stressful or dangerous situation, the nervous system’s adrenaline release causes blood to rush through the cardiovascular system to the muscles. This causes the body to become more efficient at using oxygen and energy, and it can also cause the skin to turn red and hot. This response is designed to help the body deal with stress and danger, and it can include a range of symptoms that impact different parts of the body. For example, elevated blood pressure can cause the kidneys to release more urine, and adrenaline can speed up the heart rate. It can also cause the mouth to dry out as the nervous system reacts to the perceived threat. In addition, anxiety can cause the chest to tighten and breathing to become quicker and shallower, which can cause the heart rate to increase. It can also cause the digestive system to slow down, and the muscles of the gastrointestinal tract to tighten, which can cause nausea and abdominal cramping. |
| Vanilla PPO 7B<br>RM Score: -0.41 | There are several possible explanations as to why we feel hot and uncomfortable when anxious. One explanation is that anxiety causes the body to release hormones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. A different explanation is that a reaction to stress causes the body to release hormones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. Another explanation is that the body’s stress response causes the release of hormones and chemicals into the bloodstream, which then circulate throughout the body and affect organs, tissues, and nerve cells. A fourth explanation is that anxiety triggers the release of cortisol, which increases blood sugar levels and causes the body to burn fat and muscle tissue for energy. |
| MA-PPO 7B<br>RM Score: -0.39 | When we become anxious, it is because our brain has encountered something that it sees as a threat. The brain then sends out a warning signal, which takes the form of a fight-or-flight response. This response causes a number of physical effects, including increased heart rate and breathing rate, increased blood flow to muscles, and elevated blood sugar. This is because when we are threatened our body needs to be able to run or fight, and to do this we need increased oxygen and energy. Our body also needs to be able to protect itself, so we have increased blood flow to our muscles. When we are in a high-stress situation, our body produces serotonin and dopamine, which help us to feel good and focus on the task at hand. At the same time, our body produces norepinephrine and cortisol. Norepinephrine and cortisol work together to increase our heart rate, breath rate, and blood pressure, which in turn increases the flow of oxygen and nutrients to our muscles. Thus, when we are anxious, our body’ way of protecting us is by increasing our awareness and increasing the flow of nutrients and oxygen to our muscles, so we can run or fight if needed. All of these processes occur so quickly because our body’ systems are already set up to respond instantly to threats. |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Typos:
L68: leads -> lead
soundness: 2
strengths: (S1) The direction of the current work to use macro actions (or a group of low-level actions together) is novel and interesting, for the setting of aligning a language model with human preferences (RLHF scope). Please see W1 for more discussion.

(S2) The manuscript is well-written, with sufficient details on all the technical details making it easy to understand and comprehend. I thoroughly enjoyed reading the paper.

(S3) The ablation and generalization studies presented in the experiment section (Sec.4.3 and 4.4) were interesting to better understand the behavior of the proposed approach in the context of RLHF.
summary: The current manuscript proposes to tackle the credit assignment problem when aligning large language models (LLMs) to human preferences via reinforcement learning from human feedback (RLHF). The core idea is to consider a group of tokens as a macro action, instead of individual actions, and use rewards accumulated within a macro action with proximal policy optimization (PPO) to align the language model. This approach is labeled MA-RLHF. Experiments on three tasks (text summarization, dialog generation, question answering, and program synthesis) show improvements of MA-RLHF over PPO for a chosen LLM (Gemma) both in terms of performance and training speed.
weaknesses: (W1) The idea of gathering multiple actions together as a “macro-action” is not novel in general. Referred to as “action chunking” [A], similar ideas have been explored in other domains. As noted in (S1), this is still useful in the context of RLHF, but with a diminished novelty factor.

(W2) The current work mostly experiments with a single LLM (Gemma and its variants). Given (W1) and lack of generalizability across different models, the usefulness of the current approach over PPO is not clearly understood to benefit the community. Demonstrating similar benefits in an LLM-agnostic manner will tremendously be useful.

(W3) Comparisons to the more recent approaches like DPO [B] are missing in the experimental tables. Though there is benefit over PPO for the chosen model, the experiments fall short to gauge the improvements compared to one of the current arts. Request the authors to include this comparison to better understand the proposed approach and its efficacy. 

References
* [A] Action chunking as conditional policy compression. https://osf.io/preprints/psyarxiv/z8yrv
* [B] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.
https://osf.io/preprints/psyarxiv/z8yrv

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses for questions.
soundness: 3
strengths: * They conduct thorough analysis on many different tasks, using several different sized base models.
* They conduct many ablations of their method.
* The paper is clearly written and easy to follow.
* This empirical analysis of macro actions present a useful contribution to our understanding of RLHF training.
summary: They conduct empirical analysis of the use of macro actions in RLHF instead of the standard per-token actions. They study a variety of approaches to breaking responses into macro actions, finding that N-gram based macro actions tend to perform best. In general, they find macro actions have the potential to outperform standard PPO with per token actions. They attribute this to better training stability and reduced variance in the learning problem.
weaknesses: * Some of the differences in performance could be due to tuning one method more than another. It would be great if the paper could at least document the steps they went through to tune hyper-parameters for their method and baselines.
* My understanding is that the primary functional difference between per-token PPO and macro action PPO is the granularity of the value function, importance sampling, and discount factor. It is a fairly small modification, but this is not necessarily clear when reading the paper and could be better emphasized.
* The GRPO method proposed in https://arxiv.org/pdf/2402.03300, can be viewed as a special case of their macro action PPO. They don't discuss this in the related work.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * How sensitive is the method to the choice of macro action size? Is the macro action sizes sensitive to different types of tasks - i’m particularly interested in difference between different categories of tasks, for example, difference between natural language (TLDR, HH-RLHF done in the paper) and coding (APPS). In figure 6, it seems the difference between different n choice is not too large on HH-RLHF, wondering if that would change for coding tasks.

* Will be nice to see some qualitative examples of the macro-actions (e.g., how tokens are grouped together in a concrete setence).

* What is the difference & commonalities between this paper and ArCHer [1]?
[1] ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
soundness: 3
strengths: - The paper identifies a critical limitation in token-level RLHF - the credit assignment problem over long sequences - and proposes a simple yet effctive solution by incorporating macro actions. The approach is well-motivated by both theoretical considerations (credit assignment, temporal abstraction) and practical issues (subword tokenization challenges).

- The paper is well-written and easy to follow.

- The experimental evaluation is thorough, covering multiple tasks, model sizes (2B to 27B parameters), and evaluation metrics. The authors also conducted detailed ablation studies on different strategy of termination and find n-gram performs the best, analyzed the impact of macro action size, and did best-of-N to further validate the effectiveness of the proposed approach.

- The proposed method is simple (maintaining the same computational complexity) yet effectively improve performance (up to 30% in text summarization and code generation). The faster convergence and better performance make this a valuable contribution to practical applications.
summary: This paper introduces MA-RLHF, a novel framework that incorporates macro actions (sequences of tokens or higher-level language constructs) into the RLHF training process for LLMs. The key innovation is operating at a higher level of abstraction by grouping tokens into meaningful units, which helps address the credit assignment problem in long sequences. The authors demonstrate that this approach leads to faster convergence (1.7x to 2x) and better performance across various tasks including text summarization, dialogue generation, question answering, and program synthesis, without increasing computational complexity during training or inference.
weaknesses: - While the paper explores different termination conditions for macro actions (n-gram based, perplexity based, parsing based), there could be more analysis of how different types of macro actions affect different types of tasks. For example, which macro action strategies work best for which types of generation tasks? For example, i would expect parsing-based termination might also work well on code generation tasks if a programming-language-based parser was used.

- Lack details of Human Evaluation: The evaluation involves human evaluation of model responses. It would be nice to show the detailed protocol of how human evaluation is performed, and what is the inter-rater agreement.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: A natural way is based on BPE-like automatic construction of a dictionary with most frequent n-grams. Have the authors considered this strategy? 

Do the authors think that leave-one-out REINFORCE or other recent PPO variants would also benefit from the authors’ algorithm (macro-actions)?
soundness: 3
strengths: The ideas are intuitive. Recent progress has shown that RL is promising in general so it’s great that the authors are investigating this direction. 

Multiple macro action termination strategies are explored. Experiments on code generation / program synthesis is helpful.

Interesting analysis on rejection sampling vs. temperature.

Helpful figures & illustrations in the appendix.
summary: The main idea is to use a larger chunk of text as the action in RLHF, instead of just one token (e.g., one word or one sub-word unit). What’s a larger chunk? The authors have tried multiple strategies: n-gram-based, parsing-based, perplexity-based “termination” of macro actions (i.e., segmenting tokens into a macro-action) – those are simple heuristics-based strategies introduced on page 4. Algorithms are built on top of naive PPO. 

Experiments are done using Gemma-2B, Gemma-7B, and Gemma-2-27B. Results on multiple tasks (summarization, dialogue or instruction following including HH-RLHF and WebGPT, code generation) show improvements.
weaknesses: Results are great and surprising (not weaknesses); e.g., in Figure 4, the newly trained models are strongly preferred over the baseline (vanilla PPO). But:
- What’s the main reason here? Is there a comprehensive analysis that makes sure that the win rate is not based on simple features like length? 
- Are baselines trained with a sufficient number of steps? 


The equation for R is incorrect in Section 2.2.
- KL is between two distributions, but the authors wrote that the KL is between two real numbers.
- The last term should be -KL( \pi_\theta ( \cdot | x) || \pi_\text{ref} (\cdot | x)); in particular, KL(p || q) does not equal to -KL(q || p). Currently if you expand the KL, the interpretation is that if \pi_sft (y|x) is large, then we should make \pi_\theta (y|x) really small, in order to maximize R. 

Other issues in Section 2
- Can gamma be equal to 1? (Section 2.1)
- In Eq. (1) it’s unclear if A_t is computed using \theta or \theta_old – it’s never defined; similarly, how about Section 3 line 233 – what model would the authors use to compute the advantage?
- In Eq. (1) what’s the expectation over? 
- At the end of page 2, the indices start from 0 (e.g., s_0, a_0); but on page 3 line 145, the indices start from 1

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Somewhat more conceptually, the motivation about "essential linguistic structures and dependencies between adjacent tokens" (Line 54) is questionable. While the intuition is reasonable, in fact the very same argumentation can be made against the pre-training stage, where LLMs are trained to make single token predictions. But pre-trained LLMs already generate fluent/coherent output (surprisingly or not). It is a strong claim that making (one of) the fine-tuning step(s) more "global" can correct for the extensive pre-training. The power in RL methods might be the ability to design global reward/objectives (see also Ranzato et al. 2015, Choshen et al. 2019) which are better suited for aligning to designer preferences, but this is still true even if the model has to make only "micro" decisions. I think the paper can benefit from at least acknowledging these questions.
soundness: 2
strengths: The proposed method is (as mentioned by the authors) simple and could be of potential interest to practitioners.
There is also an extensive evaluation of the method both in terms of tasks/datasets and in terms of method (GPT-based and human annotators)
summary: The paper proposes the incorporation of options (in the RL sense) into the RL component of LLM fine tuning with RLHF, with the motivation being better temporal abstractions over generation of long token sequences. The method is evaluated on several benchmarks and model sizes and is shown to overall outperform RLHF by vanilla PPO.
weaknesses: There are some significant conceptual weaknesses, as well as many technical issues.

Most importantly, there is a considerable gap between the stated/declared motivation and goals and what is actually being implemented. 
The MA-RLHF algorithm only learns *one* option that is being applied everywhere -- there isn't a set of different options being learned, and there is no "macro" policy that chooses between them. As a result, generation really is driven by the (single) low-level policy alone, contrary to the authors statement that 
- "Macro actions leverage meaningful linguistic structures" (Line 66; which is clearly false for several of the "termination" strategies later used such as n-grams!)
- "MA-RLHF operates at the macro action level, making decisions over sequences of tokens at a coarser temporal scale" (line 176-177).
- And so on.

What remains, effectively, is a particular way to chunk the rewards (and Advantage / Value estimates) that feed into the RL algorithm, in a way that might be different than the standard one. It could potentially be the case that this pushes the value estimation component more towards Monte-Carlo (rather than TD-learning / bootstrap) because Q (and A) are now fitted to a Bellman backup update that first sums over $\vert \omega_{\tau}\rvert$ observed rewards before bootstrapping (See $R_\tau$ definition; Line 218. This is also reflected in the authors analysis of Section 4.3.2, line 374, see more on that later). If this is the source of the performance gain, it can be implemented and evaluated much more directly, and the story about temporal abstractions is merely a distraction.

If the authors have a a good reason / explanation for this particular choice, it should be stated. And in any case, this limitation (or choice) must be clearly stated and acknowledged, preferably early on. Right now it's not even mentioned in the Limitations section (which is unjustifiably hidden in the appendix, Line 810).

Other than this main issue, there are plenty of technical issues, to a varying degree of severity. I expect some/most of them can be addressed in revisions. I have divided those into two sets, but I suspect these are not-exhaustive: the paper can definitely benefit from a closer proof-reading.

**"Moderate" Issues**
- The (Bengio et al. 2013) cite for the credit assignment problem (Abstract line 14, Intro Line 46) is out of place. That paper doesn't really deal with the RL credit assignment problem at all (although the term is mentioned, once, in the text).
  
- The comment "setting n → ∞ corresponds to the REINFORCE algorithm where the entire sequence is treated as a single macro action" (Line 399 and also Line 243) is completely unclear to me. REINFORCE most certainly **does not** treat the entire sequence as one action -- the policy is trained to generate standard "micro" actions at each time-step individually. The authors *might* mean to state something inline with the TD-based vs Monte-Carlo based method for estimating the Value/return, with PPO being an Actor-Critic variant and vanilla REINFORCE being a fully Monte-Carlo method. But this difference concerns the "critic" component rather than the actor. Moreover, the  (McGovern and Sutton 1998) cite for REINFORCE is wrong or at best confusing. (for REINFORCE the authors should cite Williams 92, or Sutton et al. 1999 if the claim is more generally on PG methods).

- What do we learn from Figure 3 that's not already shown in Figure 2?  There's nothing particularly interesting about the shape of the shown distributions besides the mean and variance, which are already presented in Fig. 2. If anything, other than showing the marginal histograms alone, there could be potential value in showing the correlation over individual examples and see if there's any structure in that (is the performance gain come from more easier/harder examples, or more uniform, etc). Also, if that figure is kept, it should be specified for which of the models and at which step at training the histograms are shown (judging by the scale I assume this is the 2B model at the end of training?).
  
- The authors claim that "MA-PPO achieves parity with vanilla PPO approximately 1.7 – 2 times faster during trainin" (Line 303). However the main difference in Figure 2 seems to be the final performance level which is higher for MA-PPO compared to PPO, and not the learning / convergence speed which is really comparable. This should either be made more quantitative / carefully evaluated, or less boldly stated.

- Figure 10: how come the marginal distribution of SFT RM scores changed between the first two panels and the last two panels? 

**Minor issues**
There are, once again, many issues with phrasings, notation, etc.

Line 92: "An policy" -> "A policy"
Line 93: "The Markov Decision Processes" -> "A Markov decision process"
Line 98: The definition of Q(s,a) should include a conditioning on the fact that s_0 =s, a_0=a. It would also be better to write Q(s,a) rather than Q(s_t, a_t).
Line 98: The authors specify that reward is a function r(s,a) but then in defining Q use the formulation of r(s).

Line 108: The PPO discussion is rather dense. I feel that for a reader unfamiliar with the method, it will not be too helpful, and for familiar reader it will be redundant. Can be replaced by a reference to the appropriate paper, and shortened to a brief discussion of what Policy Gradient does (roughly, the inline Eq. in line 110), to be merged with the general RL formulation.

Line 111: Randomness of *the* initial state and so on.
Line 128: The acronyms for SFT and RM should be spelled out explicitly

Line 130: There's no use in writing r(x,y) if the reader don't know what x,y stand for. Better write "A reward model is trained to predict ...", then explain what the inputs are and what the goal is.

Line 140: Shouldn't the the equation read r(x,y) - beta KL? The way now written, the KL is not a penalty at all but a "bonus" to the reward (so higher KL will be encouraged)

Figure 1: I don't understand what we should read from the figure other than the fact that temporal abstractions are "a thing". And it is overall somewhat misleading, since it seems to imply that the optimized policy chooses macro actions (the MA-RLHF column, arrow from the MA-PPO network to the omega_tau action).

Line 145: Doesn't the state include the prompt as well? (other than the tokens already generated)

Line 236: The use of $r$ to denote the likelihood ratio or pi_theta/pi_theta_old is a really bad choice, because $r$ is reserved to denote the reward function. Is there actually a need to introduce more notation for the ratio? (note that in Equation 1, line 119, this notation is not being used).

Figure 2: Move the "(2B)" and "(7B)" to the ylabel ("RM Score (2B)"), or somewhere else -- otherwise it is confusing whether this 2B indicate scale of the time axis.

Figure 4: The fonts are tiny (in all figures really, but particularly here, only readable at 200% zoom or so).

Meta-review: The paper introduces a framework that incorporates macro actions—sequences of tokens or higher-level language constructs—into the RLHF process for large language models (LLMs). This approach aims to address the credit assignment problem in token-level RLHF, where delayed rewards over long sequences hinder learning efficiency.  Empirical results demonstrate significant performance improvements across various tasks, including text summarization, dialogue generation, question answering, and program synthesis. There are some concerns (which the authors addressed well during the rebuttal) including application to more LLMs, comparison with DPO/PPO, training parameters in experiments, macro action definitions, etc. Please address these comments in the final paper.

Decision: Accept (Poster)