Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
Authors: name: Wei Shen; name: Chuheng Zhang
Published: 2024-09-11 02:40:38

Abstract: Reinforcement learning from human feedback (RLHF) is one of the key
techniques that helps large language models (LLMs) to follow instructions and
provide helpful and harmless responses. While direct policy optimization
methods exist, state-of-the-art LLMs adopt RL-based methods (usually PPO) in
RLHF to train the policy to generate good responses guided by a reward model
learned from preference data. The main challenge of these methods is the
inaccuracy of the intermediate reward model, especially in code generation
tasks that require long and complex reasoning to score a response. We find that
the reliability of the reward model varies across responses assigned with
different rewards. This motivates us to filter the samples whose rewards may be
unreliable to improve signal-to-noise ratio during policy learning, resulting
in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a
proper policy filtration strategy for a given reward model, the coefficient of
determination (R^2) between rewards and actual scores on filtered samples
serves as a good metrics and helps us find several promising strategies. We
provide extensive experiments to validate the effectiveness of PF-PPO in code
generation tasks, and find that some variants of PF-PPO are highly effective
and achieve new state-of-the-art performance across 7-billion-parameter models
on HumanEval, MBPP, and a new and more challenging LeetCode Contest benchmark.