I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative
  Self-Enhancement Paradigm
Authors: name: Yiming Liang; name: Ge Zhang; name: Xingwei Qu; name: Tianyu Zheng; name: Jiawei Guo; name: Xinrun Du; name: Zhenzhu Yang; name: Jiaheng Liu; name: Chenghua Lin; name: Lei Ma; name: Wenhao Huang; name: Jiajun Zhang
Published: 2024-08-15 10:44:38

Abstract: Large Language Models (LLMs) have achieved significant advancements, however,
the common learning paradigm treats LLMs as passive information repositories,
neglecting their potential for active learning and alignment. Some approaches
train LLMs using their own generated synthetic data, exploring the possibility
of active alignment. However, there is still a huge gap between these one-time
alignment methods and the continuous automatic alignment of humans. In this
paper, we introduce I-SHEEP, an Iterative
Self-EnHancEmEnt Paradigm.This
human-like paradigm enables LLMs to continuously self-align from
scratch with nothing. Compared to the one-time alignment method Dromedary
sun2023principledriven, which refers to the first iteration in this
paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama
models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca
Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval
accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally,
I-SHEEP surpasses the base model in various standard benchmark generation
tasks, achieving an average improvement of 24.77\% in code generation tasks,
12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based
on the experiment results. Our codes, datasets, and models are available at
https://anonymous.4open.science/r/I-SHEEP.