Well Begun is Half Done: Low-resource Preference Alignment by
  Weak-to-Strong Decoding
Authors: name: Feifan Song, name: Shaohang Wei, name: Wen Luo, name: Yuxuan Fan, name: Tianyu Liu, name: Guoyin Wang, name: Houfeng Wang

Large Language Models (LLMs) require alignment with human preferences to
avoid generating offensive, false, or meaningless content. Recently,
low-resource methods for LLM alignment have been popular, while still facing
challenges in obtaining both high-quality and aligned content. Motivated by the
observation that the difficulty of generating aligned responses is concentrated
at the beginning of decoding, we propose a novel framework, Weak-to-Strong
Decoding (WSD), to enhance the alignment ability of base models by the guidance
of a small aligned model. The small model first drafts well-aligned beginnings,
followed by the large base model to continue the rest, controlled by a
well-designed auto-switch mechanism. We also collect a new dataset, GenerAlign,
to fine-tune a small-sized Pilot-3B as the draft model, which effectively
enhances different base models under the WSD framework to outperform all
baseline methods, while avoiding degradation on downstream tasks, termed as the
alignment tax. Extensive experiments are further conducted to examine the
impact of different settings and time efficiency, as well as analyses on the
intrinsic mechanisms of WSD in depth.