GenARM: Reward Guided Generation with Autoregressive Reward Model for
  Test-time Alignment
Authors: name: Yuancheng Xu; name: Udari Madhushani Sehwag; name: Alec Koppel; name: Sicheng Zhu; name: Bang An; name: Furong Huang; name: Sumitra Ganesh
Published: 2024-10-10 17:58:24

Abstract: Large Language Models (LLMs) exhibit impressive capabilities but require
careful alignment with human preferences. Traditional training-time methods
finetune LLMs using human preference datasets but incur significant training
costs and require repeated training to handle diverse user preferences.
Test-time alignment methods address this by using reward models (RMs) to guide
frozen LLMs without retraining. However, existing test-time approaches rely on
trajectory-level RMs which are designed to evaluate complete responses, making
them unsuitable for autoregressive text generation that requires computing
next-token rewards from partial responses. To address this, we introduce
GenARM, a test-time alignment approach that leverages the Autoregressive Reward
Model--a novel reward parametrization designed to predict next-token rewards
for efficient and effective autoregressive generation. Theoretically, we
demonstrate that this parametrization can provably guide frozen LLMs toward any
distribution achievable by traditional RMs within the KL-regularized
reinforcement learning framework. Experimental results show that GenARM
significantly outperforms prior test-time alignment baselines and matches the
performance of training-time methods. Additionally, GenARM enables efficient
weak-to-strong guidance, aligning larger LLMs with smaller RMs without the high
costs of training larger models. Furthermore, GenARM supports multi-objective
alignment, allowing real-time trade-offs between preference dimensions and
catering to diverse user preferences without retraining.