SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models

Authors: Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, instruction-following, self-improvement

Abstract:
Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output.
Such an ability is well-suited for and often optimized by preference learning.
However, existing methods often directly sample multiple independent responses from the model when creating preference pairs.
Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following.
In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions.
By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations.
Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. 
Furthermore, SPaR demonstrates promising scalability, greatly enhancing models like GLM-4-9B and LLaMA3-70B.
We also identify how inference scaling in tree search would impact model performance.
Our code and data are publicly available at https://github.com/thu-coai/SPaR.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729148478190

thread_type:
official_review

<thread_object>
creator: reviewer_ashf

<content>
summary:
The authors introduce SPAR, an automated and scalable approach designed for self-improvement in instruction-following tasks through self-play. The core idea is to create paired responses with minimal irrelevant variations, allowing for precise training of the model's instruction-following capabilities. In the SPAR framework, the authors fully leverage test-time scaling: using tree search to obtain higher-quality data for training the model's instruction-following abilities, and using self-consistency to acquire higher-quality data for training the model's discriminative and refinement abilities. Experimental results show that the SPAR framework significantly outperforms various self-critique baselines.

strengths:
- Constructing tailored and distinct instruction-following response pairs for the model by eliminating irrelevant content is a strong motivation for enhancing the model's instruction-following abilities.  
- The SPAR framework's proposal to use test-time scaling during the training phase to obtain high-quality data for training the model's 
- The experimental setup is reasonable, and the results appear promising.  
-  The writing in the paper is clear and easy to understand.

weaknesses:
Using test-time scaling (more accurately, inference-time scaling) during the training phase to obtain high-quality data for self-critique is well-motivated, but it undoubtedly introduces significant training overhead. Therefore, providing a detailed comparison of the training costs of different methods, or comparing the gains when the costs are aligned, would make the paper's conclusions more convincing.

typo: line 239 needs a blank after 'refiner'.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730446769191

thread_type:
official_review

<thread_object>
creator: reviewer_7agy

<content>
summary:
This paper proposes a novel self-play framework (called SPAR) integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions, so as to teach models to recognize the key differences that lead to improved instruction following. To gain a good start for Actor and Refiner, the authors construct a high-quality dataset with 43K complex instruction-following prompts and an SFT dataset for improving the instruction-following capabilities of LLMs. Through extensive experiments on several LLMs, the authors demonstrate the effectiveness of proposed SPAR.

strengths:
* The proposed approach is intuitive and has strong motivation.

* This paper is well-written and presents clear ideas.

* The authors conduct extensive experiments to validate the effectiveness of proposed SPAR.

weaknesses:
* **SPAR introduces additianal training overheads.** SPAR initially requires constructing additional data to train the actor and trainer. Building on this, it needs to incorporate iterative training with tree search and self-consistency, which greatly increases the training cost compared to self-rewarding.

* **Some crucial information is missing in the experiment section.**  For example, what is the average number of search nodes in the tree search, and does it decrease with the iterations? How does LLaMA3-70B perform at different iterations (SPAR-70B-SFT, SPAR-70B-DPO-iter1, SPAR-70B-DPO-iter2)?

questions:
See weaknesses. 

In addition:

(1) line 527, GPT-4-Turbo or GPT-4o-mini?

(2) Can you compare the training cost of SPAR, Self-Rewarding and Meta-Rewarding?

(3) Does more iteration brings higher performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721553374

thread_type:
official_review

<thread_object>
creator: reviewer_2gma

<content>
summary:
The paper introduces SPAR (Self-Play with Tree-Search Refinement), a self-improvement framework that enhances the instruction-following capabilities of LLMs by minimizing extraneous factors and highlighting key differences in preference pairs. This method involves an iterative training process where a model (actor) performs tasks, and a paired model (refiner) evaluates and refines the imperfect responses using a tree-search algorithm through structured feedback loops. The authors evaluate SPAR with two LLMs on the IFEval and FollowBench benchmarks. Additionally, they contribute a dataset with 43k complex instruction-following prompts and an SFT dataset that can improve the instruction-following capabilities of LLMs.

strengths:
1. Effective in reducing noise. By minimizing content variations in preference pairs, SPAR helps the model focus on essential elements, which improves its instruction-following accuracy.

2. Comprehensive ablation experiments. The authors conducted extensive ablation studies to verify the impact of interfering factors on preference learning and to assess the rationality of each component in the framework.

3. Generalization without degradation. The approach does not degrade general language model capabilities, suggesting a balanced enhancement in alignment without compromising overall functionality.

4. Contribution of datasets. The authors provide valuable datasets that benefit the development of this research area.

weaknesses:
1. Limited validation across models. The effectiveness of the method was validated on only three models. Further exploration is needed to assess the framework's applicability to other models.

2. Reliance on complex setup and compute resources. The framework's iterative training, including tree-search refinement and multiple model roles, may require significant computational resources. Therefore, the performance-cost trade-off needs further clarification. 

3. Lack of comparative details. The paper lacks sufficient details in its comparisons with other methods, such as how each baseline initializes the model.

questions:
1. The paper only lists results from the first three iterations, and the data indicate that the model's performance still has room for improvement. Could you provide a simple analysis of when the model might reach optimal performance?

2. Does the framework heavily depend on the model's initial performance? Can it be directly applied to the raw models provided officially?

3. It is suggested to directly specify the strong LLM used in Section 2.2.2
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728112343

thread_type:
official_review

<thread_object>
creator: reviewer_pyv7

<content>
summary:
This study presents SPAR, a self-play framework that enhances LLM‘s instruction-following capabilities by training with refined preference pairs. Unlike traditional methods that rely on independent response sampling, SPAR refines pairs to reduce irrelevant factors, thereby emphasizing critical distinctions, leading to notable improvements in instruction adherence. SPAR’s iterative process enhances instruction-following, judgment, and refinement, offering a pathway for continuous model improvement.

strengths:
1. The motivation makes sense. The fine-grained refinement is essential for further improving the model's instruction-following abilities.
2. The design of the proposed method is sound, allowing it to effectively achieve its intended motivation.
3. The experiments are comprehensive, demonstrating relatively strong performance.

weaknesses:
1. The applicability of the method may be limited. It might be suitable primarily for further improvement of models that already possess strong instruction-following capabilities, as the experiments were conducted on models that had already undergone instruction fine-tuning. Additionally, a strong LLM is required for warm-up training before iteration (This also raises concerns about the fairness of comparisons.), and one of the goals of dataset construction is to introduce more complex instructions.
2. Missing comparison with a key baseline：Self-Alignment with Instruction Backtranslation.

questions:
1. Why are judgment and refinement performed by the same model? What would happen if they were separated, or combined with the actor model, using a single model for all tasks?
2. I haven't closely checked the details of the baselines. Do they use a strong LLM, or do they rely solely on the model being evolved?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732023468824

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their thoughtful comments and constructive suggestions, which significantly helped us strengthen our paper. 
We are encouraged to see that the reviewers recognize the novelty and significance of our proposed SPAR framework (Reviewer pYV7, 2gma, 7aGY, AshF), its sound design and comprehensive experimental validation (Reviewer pYV7, 2gma, 7aGY, AshF), and the clarity of our presentation (Reviewer pYV7, 7aGY, AshF).

In response to the reviewers' feedback, we have submitted an updated version of our paper, which now includes more experimental details, the Humpback baseline comparison, additional experiments with more models such as GLM-4-9B, and a more detailed analysis of the cost associated with our method.

Several reviewers suggested including more information regarding the cost of our method. We would like to emphasize that leveraging inference-time scaling for training to improve model performance is well-motivated (as recognized by Reviewer AshF) and is actually more efficient compared to traditional pretraining solutions or producing high-quality data with human annotators.
Furthermore, our overall cost—averaging around 3.7 expanded tree nodes—is within a reasonable range, ensuring the feasibility and scalability of the approach.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary: 
The paper introduces SPaR, a self-play framework that enhances instruction-following capabilities in language models by refining preference pairs through tree-search self-refinement. Unlike traditional methods that rely on independent response sampling, SPaR minimizes irrelevant factors and emphasizes critical distinctions, leading to improved instruction adherence. Through an iterative training process, SPaR guides LLMs to recognize key differences crucial for enhanced instruction. Experimentation demonstrates the effectiveness of SPaR, with an LLaMA3-8B model trained using SPaR surpassing GPT-4-Turbo on the IFEval benchmark without compromising general capabilities. Additionally, SPaR shows promising scalability and greatly improves the performance of LLaMA3-70B. The framework sheds novel insights into continuous model improvement. 

## Strengths:
1. The paper's motivation for fine-grained refinement of responses for instruction following is well-founded and crucial for continuous model improvement.
1. The tree search-based negative refinement is effective by minimizing content variations, allowing the model to focus on essential elements for enhancing instruction-following accuracy.
1. Comprehensive ablation experiments validate the impact of interfering factors on preference learning and the rationality of each component in the framework.
1. The approach maintains generalization without compromising overall language model capabilities, indicating a balanced enhancement.
1. The paper presents comprehensive experiments with clear writing.

## Weaknesses:
1. Lack of some experiments: the initial draft does apply the proposed method to finetuning weaker base models or other different models; Comparison to more baselines are needed; Using the same vs. separate models for judgment and refinement tasks; Training for more iterations. The rebuttal addressed these concerns by reporting the corresponding experimental results. 
1. Concerns about the extra overhead introduced by the data exploration process (tree search and decoding). An analysis and measurement of the cost is needed. A comparison with baselines under the same computation budget is necessary. It would be better to compare the trade-off between data exploration cost and the instruction-following improvement. 
1. BFS and DFS as two basic tree search algorithms have been studied. It would be more interesting to try other search algorithms such as greedy search, A*, and MCTS, and compare their performance. 
1. There are several concurrent works adopting the LLM + tree search idea to generate better data for LLM finetuning. It would be helpful to include discussions with them in the related work section.   

## Decision:
The reviewers raised several concerns mainly regarding experimental comparisons and computational time overhead. The authors provided further clarifications and additional experimental results in the rebuttal, which addressed most concerns, as confirmed by three out of the four reviewers. The efforts of the authors convinced the reviewers, resulting in all positive ratings (8666). The meta-reviewer is familiar with the field and carefully reads the paper, all the comments, the rebuttals, and the discussions. The paper addresses an important open challenge for the self-improvement of LLMs: how to generate more informative and relevant preference pairs. Using tree search to refine the negative responses is an effective and intuitive idea, as demonstrated by the comprehensive experiments in the paper. By strengthening the paper with all the results presented in the discussion, the meta-reviewer believes this paper is valuable and inspirational to the community, hence acceptance is recommended.