DeAL: Decoding-time Alignment for Large Language Models
Authors: name: James Y. Huang; name: Sailik Sengupta; name: Daniele Bonadiman; name: Yi-an Lai; name: Arshit Gupta; name: Nikolaos Pappas; name: Saab Mansour; name: Katrin Kirchoff; name: Dan Roth
Published: 2024-02-05 06:12:29

Abstract: Large Language Models (LLMs) are nowadays expected to generate content
aligned with human preferences. Current work focuses on alignment at model
training time, through techniques such as Reinforcement Learning with Human
Feedback (RLHF). However, it is unclear if such methods are an effective choice
to teach alignment objectives to the model. First, the inability to incorporate
multiple, custom rewards and reliance on a model developer's view of universal
and static principles are key limitations. Second, the residual gaps in model
training and the reliability of such approaches are also questionable (e.g.
susceptibility to jail-breaking even after safety training). To address these,
we propose DeAL, a framework that allows the user to customize reward functions
and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view
decoding as a heuristic-guided search process and facilitate the use of a wide
variety of alignment objectives. Our experiments with programmatic constraints
such as keyword and length constraints (studied widely in the pre-LLM era) and
abstract objectives such as harmlessness and helpfulness (proposed in the
post-LLM era) show that we can DeAL with fine-grained trade-offs, improve
adherence to alignment objectives, and address residual gaps in LLMs. Lastly,
while DeAL can be effectively paired with RLHF and prompting techniques, its
generality makes decoding slower, an optimization we leave for future work.