Iterative Value Function Optimization for Guided Decoding
Authors: name: Zhenhua Liu; name: Lijun Li; name: Ruizhe Chen; name: Yuxian Jiang; name: Tong Zhu; name: Wenliang Chen; name: Jing Shao
Published: 2025-03-04 07:49:10

Abstract: While Reinforcement Learning from Human Feedback (RLHF) has become the
predominant method for controlling language model outputs, it suffers from high
computational costs and training instability. Guided decoding, especially
value-guided methods, offers a cost-effective alternative by controlling
outputs without re-training models. However, the accuracy of the value function
is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
decision-making and degraded performance. Existing methods struggle with
accurately estimating the optimal value function, leading to less effective
control. We propose Iterative Value Function Optimization, a novel framework
that addresses these limitations through two key components: Monte Carlo Value
Estimation, which reduces estimation variance by exploring diverse
trajectories, and Iterative On-Policy Optimization, which progressively
improves value estimation through collecting trajectories from value-guided
policies. Extensive experiments on text summarization, multi-turn dialogue, and
instruction following demonstrate the effectiveness of value-guided decoding
approaches in aligning language models. These approaches not only achieve
alignment but also significantly reduce computational costs by leveraging
principled value function optimization for efficient and effective control.