MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Authors: name: Yekun Chai; name: Haoran Sun; name: Huang Fang; name: Shuohuan Wang; name: Yu Sun; name: Hua Wu
Published: 2024-10-03 17:55:13

Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated
effectiveness in aligning large language models (LLMs) with human preferences.
However, token-level RLHF suffers from the credit assignment problem over long
sequences, where delayed rewards make it challenging for the model to discern
which actions contributed to successful outcomes. This hinders learning
efficiency and slows convergence. In this paper, we propose MA-RLHF, a simple
yet effective RLHF framework that incorporates macro actions -- sequences of
tokens or higher-level language constructs -- into the learning process. By
operating at this higher level of abstraction, our approach reduces the
temporal distance between actions and rewards, facilitating faster and more
accurate credit assignment. This results in more stable policy gradient
estimates and enhances learning efficiency within each episode, all without
increasing computational complexity during training or inference. We validate
our approach through extensive experiments across various model sizes and
tasks, including text summarization, dialogue generation, question answering,
and program synthesis. Our method achieves substantial performance improvements
over standard RLHF, with performance gains of up to 30% in text summarization
and code generation, 18% in dialogue, and 8% in question answering tasks.
Notably, our approach reaches parity with vanilla RLHF 1.7x to 2x faster in
terms of training time and continues to outperform it with further training. We
will make our code and data publicly available at
https://github.com/ernie-research/MA-RLHF .