Response Tuning: Aligning Large Language Models without Instruction
Authors: name: Seokhyun An; name: Hyounghun Kim
Published: 2024-10-03 13:15:19

Abstract: Instruction tuning-supervised fine-tuning using instruction-response pairs-is
a foundational step in transitioning pre-trained Large Language Models (LLMs)
into helpful and safe chat assistants. Our hypothesis is that establishing an
adequate output space can enable such a transition given the capabilities
inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),
which eliminates the instruction-conditioning step in instruction tuning and
solely focuses on response space supervision. Our experiments demonstrate that
RT models, trained only using responses, can effectively respond to a wide
range of instructions and exhibit helpfulness comparable to that of their
instruction-tuned counterparts. Furthermore, we observe that controlling the
training response distribution can significantly improve their user preference
or elicit target behaviors such as refusing assistance for unsafe queries. Our
findings illuminate the role of establishing an adequate output space in
alignment, highlighting the potential of the extensive inherent capabilities of
pre-trained LLMs.