Conditioned Language Policy: A General Framework for Steerable
  Multi-Objective Finetuning
Authors: name: Kaiwen Wang, name: Rahul Kidambi, name: Ryan Sullivan, name: Alekh Agarwal, name: Christoph Dann, name: Andrea Michi, name: Marco Gelmi, name: Yunxuan Li, name: Raghav Gupta, name: Avinava Dubey, name: Alexandre Ramé, name: Johan Ferret, name: Geoffrey Cideron, name: Le Hou, name: Hongkun Yu, name: Amr Ahmed, name: Aranyak Mehta, name: Léonard Hussenot, name: Olivier Bachem, name: Edouard Leurent

Reward-based finetuning is crucial for aligning language policies with
intended behaviors (e.g., creativity and safety). A key challenge here is to
develop steerable language models that trade-off multiple (conflicting)
objectives in a flexible and efficient manner. This paper presents Conditioned
Language Policy (CLP), a general framework for finetuning language models on
multiple objectives. Building on techniques from multi-task training and
parameter-efficient finetuning, CLP can learn steerable models that effectively
trade-off conflicting objectives at inference time. Notably, this does not
require training or maintaining multiple models to achieve different trade-offs
between the objectives. Through an extensive set of experiments and ablations,
we show that the CLP framework learns steerable models that outperform and
Pareto-dominate the current state-of-the-art approaches for multi-objective
finetuning.