Conditioned Language Policy: A General Framework for Steerable
  Multi-Objective Finetuning
Authors: name: Kaiwen Wang; name: Rahul Kidambi; name: Ryan Sullivan; name: Alekh Agarwal; name: Christoph Dann; name: Andrea Michi; name: Marco Gelmi; name: Yunxuan Li; name: Raghav Gupta; name: Avinava Dubey; name: Alexandre Ramé; name: Johan Ferret; name: Geoffrey Cideron; name: Le Hou; name: Hongkun Yu; name: Amr Ahmed; name: Aranyak Mehta; name: Léonard Hussenot; name: Olivier Bachem; name: Edouard Leurent
Published: 2024-07-22 16:13:38

Abstract: Reward-based finetuning is crucial for aligning language policies with
intended behaviors (e.g., creativity and safety). A key challenge here is to
develop steerable language models that trade-off multiple (conflicting)
objectives in a flexible and efficient manner. This paper presents Conditioned
Language Policy (CLP), a general framework for finetuning language models on
multiple objectives. Building on techniques from multi-task training and
parameter-efficient finetuning, CLP can learn steerable models that effectively
trade-off conflicting objectives at inference time. Notably, this does not
require training or maintaining multiple models to achieve different trade-offs
between the objectives. Through an extensive set of experiments and ablations,
we show that the CLP framework learns steerable models that outperform and
Pareto-dominate the current state-of-the-art approaches for multi-objective
finetuning.