Tuning Language Models by Proxy
Authors: name: Alisa Liu; name: Xiaochuang Han; name: Yizhong Wang; name: Yulia Tsvetkov; name: Yejin Choi; name: Noah A. Smith
Published: 2024-01-16 18:49:55

Abstract: Despite the general capabilities of large pretrained language models, they
consistently benefit from further adaptation to better achieve desired
behaviors. However, tuning these models has become increasingly
resource-intensive, or impossible when model weights are private. We introduce
proxy-tuning, a lightweight decoding-time algorithm that operates on top of
black-box LMs to achieve the result of directly tuning the model, but by
accessing only its prediction over the output vocabulary. Our method instead
tunes a smaller LM, then applies the difference between the predictions of the
small tuned and untuned LMs to shift the original predictions of the base model
in the direction of tuning, while retaining the benefits of larger scale
pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using
proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its
truly-tuned chat version, when evaluated across knowledge, reasoning, and
safety benchmarks. Interestingly, when tested on TruthfulQA, proxy-tuned models
are actually more truthful than directly tuned models, possibly because
decoding-time guidance better retains the model's factual knowledge. We then
demonstrate the generality of proxy-tuning by applying it for domain adaptation
on code, and task-specific finetuning on question-answering and math problems.
Our work demonstrates the promise of using small tuned LMs to efficiently
customize large, potentially proprietary LMs through decoding-time guidance.