Title: ReST-RL: Reinforcing LLM Reasoning through Self-Training and Value-Guided Decoding
Primary area: reinforcement learning
Keywords: Large Language Models, Reinforcement Learning, Code Reasoning, Self Training

Abstract: With respect to improving the reasoning accuracy of LLMs, the representative reinforcement learning (RL) method — Group Relative Policy Optimization (GRPO) — has achieved critical success, yet it still suffers from the issue of insignificant reward variance. This paper introduces ReST-RL, a unified LLM RL paradigm that combines an improved GRPO algorithm with a meticulously designed test-time decoding method to improve LLM’s code reasoning ability. As the first stage of policy reinforcement, ReST-GRPO adopts an optimized ReST algorithm to increase the reward variance of GRPO sampling, thereby improving training effectiveness. Building on this foundation, we further introduce a test-time decoding optimization method, VM-MCTS, which employs an adapted Monte-Carlo Tree Search (MCTS) guided by a trained Value Model (VM) to provide precise process signals and verification scores, further enhancing LLM reasoning accuracy. We validate our RL paradigm on multiple coding benchmarks (e.g., APPS, BigCodeBench, and HumanEval), where it significantly outperforms other reinforcement training baselines (e.g., naive GRPO and ReST-DPO), as well as decoding and verification baselines (e.g., PRM-BoN and ORM-MCTS), indicating its power to strengthen LLM’s reasoning capability. Notably, our approach achieves strong performance with limited data, showcasing its effectiveness, efficiency, and generalizability.

TL;DR: We propose a novel LLM RL framework that significantly improves LLM's code reasoning capabilities through optimized self-training and decoding.