Reward Steering with Evolutionary Heuristics for Decoding-time Alignment
Authors: name: Chia-Yu Hung; name: Navonil Majumder; name: Ambuj Mehrish; name: Soujanya Poria
Published: 2024-06-21 14:35:16

Abstract: The widespread applicability and increasing omnipresence of LLMs have
instigated a need to align LLM responses to user and stakeholder preferences.
Many preference optimization approaches have been proposed that fine-tune LLM
parameters to achieve good alignment. However, such parameter tuning is known
to interfere with model performance on many tasks. Moreover, keeping up with
shifting user preferences is tricky in such a situation. Decoding-time
alignment with reward model guidance solves these issues at the cost of
increased inference time. However, most of such methods fail to strike the
right balance between exploration and exploitation of reward -- often due to
the conflated formulation of these two aspects - to give well-aligned
responses. To remedy this we decouple these two aspects and implement them in
an evolutionary fashion: exploration is enforced by decoding from mutated
instructions and exploitation is represented as the periodic replacement of
poorly-rewarded generations with well-rewarded ones. Empirical evidences
indicate that this strategy outperforms many preference optimization and
decode-time alignment approaches on two widely accepted alignment benchmarks
AlpacaEval 2 and MT-Bench. Our implementation will be available at:
https://darwin-alignment.github.io.