Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Authors: name: Haoyu Wang; name: Guozheng Ma; name: Ziqiao Meng; name: Zeyu Qin; name: Li Shen; name: Zhong Zhang; name: Bingzhe Wu; name: Liu Liu; name: Yatao Bian; name: Tingyang Xu; name: Xueqian Wang; name: Peilin Zhao
Published: 2024-02-12 12:30:42

Abstract: Self-alignment is an effective way to reduce the cost of human annotation
while ensuring promising model capability. However, most current methods
complete the data collection and training steps in a single round, which may
overlook the continuously improving ability of self-aligned models. This gives
rise to a key query: What if we do multi-time bootstrapping self-alignment?
Does this strategy enhance model performance or lead to rapid degradation? In
this paper, our pioneering exploration delves into the impact of bootstrapping
self-alignment on large language models. Our findings reveal that bootstrapping
self-alignment markedly surpasses the single-round approach, by guaranteeing
data diversity from in-context learning. To further exploit the capabilities of
bootstrapping, we investigate and adjust the training order of data, which
yields improved performance of the model. Drawing on these findings, we propose
Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
training recipe, we propose SOFT+ which further boost self-alignment's
performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
various classification and generation tasks, highlighting the potential of
bootstrapping self-alignment on continually enhancing model alignment
performance.