SPaR: Self-Play with Tree-Search Refinement to Improve
  Instruction-Following in Large Language Models
Authors: name: Jiale Cheng, name: Xiao Liu, name: Cunxiang Wang, name: Xiaotao Gu, name: Yida Lu, name: Dan Zhang, name: Yuxiao Dong, name: Jie Tang, name: Hongning Wang, name: Minlie Huang

Instruction-following is a fundamental capability of language models,
requiring the model to recognize even the most subtle requirements in the
instructions and accurately reflect them in its output. Such an ability is
well-suited for and often optimized by preference learning. However, existing
methods often directly sample multiple independent responses from the model
when creating preference pairs. Such practice can introduce content variations
irrelevant to whether the instruction is precisely followed (e.g., different
expressions about the same semantic), interfering with the goal of teaching
models to recognize the key differences that lead to improved instruction
following. In light of this, we introduce SPaR, a self-play framework
integrating tree-search self-refinement to yield valid and comparable
preference pairs free from distractions. By playing against itself, an LLM
employs a tree-search strategy to refine its previous responses with respect to
the instruction while minimizing unnecessary variations. Our experiments show
that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses
GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
Furthermore, SPaR demonstrates promising scalability and transferability,
greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how
inference scaling in tree search would impact model performance. Our code and
data are publicly available at https://github.com/thu-coai/SPaR.