Iterative Value Function Optimization for Guided Decoding
Authors: name: Zhenhua Liu, name: Lijun Li, name: Ruizhe Chen, name: Yuxian Jiang, name: Tong Zhu, name: Wenliang Chen, name: Jing Shao

While Reinforcement Learning from Human Feedback (RLHF) has become the
predominant method for controlling language model outputs, it suffers from high
computational costs and training instability. Guided decoding, especially
value-guided methods, offers a cost-effective alternative by controlling
outputs without re-training models. However, the accuracy of the value function
is crucial for value-guided decoding, as inaccuracies can lead to suboptimal
decision-making and degraded performance. Existing methods struggle with
accurately estimating the optimal value function, leading to less effective
control. We propose Iterative Value Function Optimization, a novel framework
that addresses these limitations through two key components: Monte Carlo Value
Estimation, which reduces estimation variance by exploring diverse
trajectories, and Iterative On-Policy Optimization, which progressively
improves value estimation through collecting trajectories from value-guided
policies. Extensive experiments on text summarization, multi-turn dialogue, and
instruction following demonstrate the effectiveness of value-guided decoding
approaches in aligning language models. These approaches not only achieve
alignment but also significantly reduce computational costs by leveraging
principled value function optimization for efficient and effective control.