ReFT: Reasoning with Reinforced Fine-Tuning
Authors: name: Trung Quoc Luong; name: Xinbo Zhang; name: Zhanming Jie; name: Peng Sun; name: Xiaoran Jin; name: Hang Li
Published: 2024-01-17 04:43:21

Abstract: One way to enhance the reasoning capability of Large Language Models (LLMs)
is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
annotations. This approach does not show sufficiently strong generalization
ability, however, because the training only relies on the given CoT data. In
math problem-solving, for example, there is usually only one annotated
reasoning path for each question in the training data. Intuitively, it would be
better for the algorithm to learn from multiple annotated reasoning paths given
a question. To address this issue, we propose a simple yet effective approach
called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
learning LLMs for reasoning, with math problem-solving as an example. ReFT
first warmups the model with SFT, and then employs on-line reinforcement
learning, specifically the PPO algorithm in this paper, to further fine-tune
the model, where an abundance of reasoning paths are automatically sampled
given the question and the rewards are naturally derived from the ground-truth
answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
ReFT significantly outperforms SFT, and the performance can be potentially
further boosted by combining inference-time strategies such as majority voting
and re-ranking. Note that ReFT obtains the improvement by learning from the
same training questions as SFT, without relying on extra or augmented training
questions. This indicates a superior generalization ability for ReFT.