Controllable Safety Alignment: Inference-Time Adaptation to Diverse
  Safety Requirements
Authors: name: Jingyu Zhang; name: Ahmed Elgohary; name: Ahmed Magooda; name: Daniel Khashabi; name: Benjamin Van Durme
Published: 2024-10-11 16:38:01

Abstract: The current paradigm for safety alignment of large language models (LLMs)
follows a one-size-fits-all approach: the model refuses to interact with any
content deemed unsafe by the model provider. This approach lacks flexibility in
the face of varying social norms across cultures and regions. In addition,
users may have diverse safety needs, making a model with static safety
standards too restrictive to be useful, as well as too costly to be re-aligned.
  We propose Controllable Safety Alignment (CoSA), a framework designed to
adapt models to diverse safety requirements without re-training. Instead of
aligning a fixed model, we align models to follow safety configs -- free-form
natural language descriptions of the desired safety behaviors -- that are
provided as part of the system prompt. To adjust model safety behavior,
authorized users only need to modify such safety configs at inference time. To
enable that, we propose CoSAlign, a data-centric method for aligning LLMs to
easily adapt to diverse safety configs. Furthermore, we devise a novel
controllability evaluation protocol that considers both helpfulness and
configured safety, summarizing them into CoSA-Score, and construct CoSApien, a
human-authored benchmark that consists of real-world LLM use cases with diverse
safety requirements and corresponding evaluation prompts.
  We show that CoSAlign leads to substantial gains of controllability over
strong baselines including in-context alignment. Our framework encourages
better representation and adaptation to pluralistic human values in LLMs, and
thereby increasing their practicality.