SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models
Authors: Jiale Cheng
Xiao Liu
Cunxiang Wang
Xiaotao Gu
Yida Lu
Dan Zhang
Yuxiao Dong
Jie Tang
Hongning Wang
Minlie Huang
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
instruction-following
self-improvement
Primary area: foundation or frontier models, including LLMs
TL;DR: A new self-improvement framework for LLMs in instruction-follow tasks

Abstract: Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output.
Such an ability is well-suited for and often optimized by preference learning.
However, existing methods often directly sample multiple independent responses from the model when creating preference pairs.
Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following.
In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions.
By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations.
Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. 
Furthermore, SPaR demonstrates promising scalability, greatly enhancing models like GLM-4-9B and LLaMA3-70B.
We also identify how inference scaling in tree search would impact model performance.
Our code and data are publicly available at https://github.com/thu-coai/SPaR.

Published as a conference paper at ICLR 2025
SPAR: SELF-PLAY WITH TREE-SEARCH REFINEMENT TO IMPROVE INSTRUCTION-FOLLOWWE IN LARGE LANGUAGE MODELS
Jiale Chengt Xiao Liu Cunxiang Wang Xiaotao Gu², Yida Lu Dan Zhang" Yuxiao Dong Jie Tang", Hongning Wang', Minlie Huang
'The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University
2 ZZhipu AI
The Knowledge Engineering Group (KEG), Tsinghua University
2mmmg (ma12mmmmiii ew..ls..aaa.e....mmmmmaam..lla.e wts.ssssiii. wtw.iss..ssssiiii. tsinghuu. edu. en, sihnang@tsinghue,.eee
ABSTRACT
iestruction-folloinng is fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them its output. Such ability well-suited for and of- ten optimized by preference learning. However, existing methods often directly sample multiple independent responses from the model when creating preference pairs. Such practice can introduce content variations irrelevant to whether the in- struction is precisely followed (e.g., different expressions about the same seman- tic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following. In light of this, we introduce SPAR, self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. By playing against itself, an LLM employs tree-search strategy to refine its previous responses with respect the instruction while minimizing unnecessary variations. Our experiments show that LLAMA3-8B model, trained over three iterations guided by SPAR, passes GPT-4-Turbo on the IFEval benchmark without losing general capabilities. Furthermore, SPAR demonstrates promising scalability, greatly enhancing mod- els like GLM-4-9B and LLaMA3-70B. We also identify how inference scaling in tree search would impact model performance. Our code and data are publicly available at https: /ttpt/gittttt htt.huu..ooottuucteeeeee www./ttutcaa///ee/eoeo
Write story end it with 'The devil is in detaiss."
82.0
81.5
GPT-c-Tubo
Ressonse 1:
Response
Refined Response:
Avergge Spores IFENva
81.0
siney
story:
stwry!
80,5
80.0
<Hansel and Gretels
<Little Red Riding Hoods
<Little Red Riding Heads
79.5
..... they defasted
... in the end, they lived happily together.
... and, they lived hepplly by together. The devil distall.
79,0
the withh The devil
78.5
78.0
Recinement Resporse Puirs
Diseetly Samplal Pains
77.5
DPU-
DPU-
Learn about the story (nterfering) ending (oxpacted)
about
Figure 1: An example of the interfering factors (story content) independently sampled multiple responses (Left). Refined response pairs exclude these factors, highlight the key difference (ending sentence), and lead to improved performance on iteratively trained LLaMA3 8B-Instruc (Right).
Equal contributions.
Work done when JC and YL interned at Zhipu AL.
1 *Corrsspnndigg author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
To date, Large Language Models (LLMs) have achieved great success in wide range of tasks (Brown et al. 2020 Zeng et al. 2022 Chowdhery et al. 2023 Touvron et al., 2023 GLM et al. 2024]. As LLMs are applied to various scenarios, their instructionffollowig capability becomes crucial (Ouyang et al. 2022, Bai et al.
2022). especially to follow instructions with multiple con- straints (Zeng al. 2023 hoou al. 2023 liang et al.) 2023b). The failure to accurately follow
instructions even lead to safety issues (Ruan et al. 2023).
Subtle nuances can determine the success of instruction-follovinng tasks (Zhou et al. 2023), making preference learning (Rafailov et al. 2024| Hou et al. 2024) well-suited solution. However, existing methods usually sample multiple independent responses from the target model (Yuan et al. 2024]
Wu et al.) 2024; Dong et al.] (024), inadvertently introducing irrelevant variations to whether the instruction was successfully tollowed. As illustrated in Figure D given the instruction: "Write a story and end it with The devil is in the details", sampling multiple independent responses from L.M can result responses as different as the story Little Red Riding Hood vs. Hansel and Gretel. This variation in the narrative content can interfere with the model's ability to learn how realize the critical requirement- specified ending sentence-and and ultimately mislead the comparison within the preference pair. Therefore, effective learning from preference pairs necessitates excluding these extraneous factors and focusing on the key differences that drive the success of instruction foltooing..
In this paper, we propose SPAR, self-play method integrated with tree-search refinement to en- hance cntructtionfolloviii capabilities of L.MMs. The key lies in iteratively teaching L.MSs to learn instruction-follin following from nuances by playing against itself with structured tree search. each turn of self-play, LLM takes two different roles: the actor and the refiner, which are both initialized from the same model. The actor executes complex instructions while the refiner critiques and refines the actor's ' responses. During the iteration, we first collect the actor's responses which fail to follow the instructions accurately, as judged by the refiner. Starting from those failed responses, we apply a tree-search algorithm for refinement, which ensures consistent improvements against previous turns and naturally creates valid comparison counterparts for model training.
We conduct experiments on several LLMs, LLaMA3 series (MetaAl 2024). GLMM-499B (GLM jet al., [202]]. and Mistral-7B-Instrucc (Jiang et al.] (023a), over multiple iterations. Through ex- tensive experiments, we demonstrate significant improvements in the models' instruction-follooinn capability, outperforming other self- self-improvement methods such as self-rewarding (Yuan et al. 2024 and meta-rewarding (Wu et al. 2024). Notably, after three iterations. SPAR improves LLaMA3-8B. Instruct over GPT-4- Turbo on the IFEval benchmark (Zhou et al. 2023). Moreover, scaling test- time compute by integrating tree-search refinement during inference can further improve the quality of instruction following. Additionally, we find that with several iterations, the refiner's judgment and refinement capabilities can match or even exceed those of the distilled I.M,, indicating great potential for continuous self-improvement without being limited by the initial bootstrapping data. Ablation studies demonstrate the importance each component within our framework. Importantly, method does not degrade performance general benchmarks. In summary, contributions are:
We reveal that preference pairs derived from independently sampled responses often contain interfering factors, hampering preference learning to improve instruction following. As a re- sult, a performing solution has to minimize such interference and highlight the key diffcrences contributing to the success of instruction following.
We introduce SPAR, a novel self-play framework that enables continuous self-improvemment in instruction-followwe tasks. Through three iterations, our method boosts LLaMA3-8B-Instme achieve GPT4-level performance and scales effectively enhance LLaMA3-70B-Inse
We construct a high-quality dataset with 43K complex instruction-followiii prompts and an SFT dataset that can improve the essructtionfoolowinn capabilities of LLMs.
2
METHOD
We introduce SPAR, an automated and scalable approach designed for smproimmrree of instruction-folllwiing tasks through self-play. The core idea is create paired responses with
2

---

Published as a conference paper at ICLR 2025
Negative Data Collection
Tree Search
Astor
Reftner
Judge
Refinement
jo
M,
R
Prompr
Response
Audgement Megenve Cate
Lx 2h Ji
( Pi Je
Model Training
Achor
Rediner
(Ar A
M,,
Rist
DPO Training
RFT Training
Openations:
Actor Data
Bettner Data
3 41
(
Reiner
Refine >5 3% Judge 3g Ju
Conrect Responso
Figure 2: SPAR iterative training framework. At iteration t, the refiner RG first judges the generated responses from the actor Mj to collect negative data. Next, a tree-search algorithm is employed to refine these imperfect responses. Finally, using the data from the above steps, we optimize the actor and refiner for the next iteration, aiming for continuous self-improvement.
minimal irrelevant variations, thereby highlighting the key differences that manifest the success of cdtructioo-folloiing.
2.1 OVERALI FRAMEWORK
The overall framework of SPAR is illustrated in Figure 2 model and refiner model, which are both initialized from the same base model. The actor generates responses to given instructions while the refiner judges and refines these responses. This iterative self-play process, involving response generation, judgment, juddmmt, and refinement, fosters continuous self- improvement.
Briefly, our framework involves an actor
Formally, in each iteration, given instruction from the prompt set, the actor generates response The refiner identifies the responses that do not follow the instructions accurately, termed as negative responses. Our objective to refine the negative response (represented as yo Figure 2) into a correct response (represented as in the figure). These generated refinement pairs, c.g., (x, Us > Yo), are collected and used optimize the actor via Direct Preference Optimization (DPO) Fine- Tine-Tuning (RFT) (Yuan al.) 2023) to improve the refiner. This process prepares both models for the next iteration of self-improvement.
[Rafallov et al.) 2024). Simultaneously, we apply Rejection-sampe
In this iterative process, we face two major challenges: the scarcity complex instruction-followin data and the difficulty of achieving successful refinements. To address the lack of high-quality multi- constraint cnstruction-folloinn datasets, we generate complex instructions using a taxonomy-base approach and create corresponding SFT datasets to initialize the actor and refiner models ($2.2). To ensure a high success rate in refining negative responses, we employ a tree search strategy that systematically explores refinement paths and facilitates subsequent training ($2.3).
2.2
DATA CONSTRUCTION
2.2. 1
PROMPT CREATION
Given the scarcity of high-quality data for costruction-followiine tasks, especially those with multiple constraints, we start by creating a high-quality dataset of eesttrutiionfollowwii prompts.
Seed Prompts. To ensure the quality and diversity of our dataset, and to prevent issues like in- sufficient diversity even model collapse (Liu et al. 2024, Shumailov et al. 2024), we use a seed set of prompts derived from the Infinity-Instrun dataset (Zhao et al. (024)) which contains ten mil- lion high-quality conversations. After applying rule-based filtering based length, keywords, and self-BLEU, we obtain approximately 50k seed prompts.
Taxonomy- Prompt Construction. Complex prompts constructed without human interven- tion tend to be poorly diversified, as the types of constraints added are often distributed unevenly (Sun et al. 2024). Therefore, we adopt taxonomy- based mechanism to make constraint types com- prehensive and balanced. Our taxonomy for forructiion-olloiinn constraints is derived from Cheng eet al.) (2024) and further refined to be more comprehensive. After building the constraint taxonomy, employ construct complex instruction-folloviii tasks based seed prompts. We sample
3

---

For each prompt I, we first sample K responses {y1. ..,..., .8K} from the actor model. This step ensures that there are enough negative responses to support subsequent learning. Then, for each prompt and response pair, utilize the refiner to generate a judgment, which contains two parts: label suggesting whether the response follows the instruction and an explanation about the assessment. To make this judgment more accurate, incorporate the self-consistency mechanism (Wang et al.) 2022), which is also applied in the subsequent refinement process. Specifically, we obtain multiple judgments from the refiner and determine the final label through majority voting, as detailed in Appendix [E.4] After majority voting, we randomly select one judgment that matches the voted label to serve as the final judgment. This process allows us identify challenging prompts Published as a conference paper at ICLR 2025
main constraint type and employ a strong LLM to add several other constraints to make the original prompt more complex. Moreover, we leverage the strong LLM to assess the validity of the generated prompt, ensuring that the constraints do not conflict with each other create unreasonable scenarios with the original task. The detailed taxonomy and prompt can be found in Appendix A
2.2.2 ACTOR AND REFINER INITIALIZATION
The taxonomy-based prompt construction results in about 43k prompts. We utilize 8k prompts for actor initialization, another 5k for the refiner, and save 30k for further self-play training.
Actor Data Creation. To bootstrap the actor model with strong cnstructioo-folloviii capabilities, first collect strong LLM'' responses these complex prompts, thereby producing supervised fine-tuning (SFT) data (xr, y) E Daur for the actor model, where x is the complex instruction and the strong LLM's response. Then, we fine-tune the base model to get an initial actor Mo.
Refiner Data Creation. To bootstrap the refiner model with strong judgment and refinement ca- pability, we sample responses from the initial actor My. Then, we collect the judgments from strong LLM to form a dataset, (I, V- j) DISFT. We collect responses that are judged not to ac- curately follow instructions and term them as negative responses. For these negative responses, we use the strong LLM to correct them with minimal revisions to avoid irrelevant variations. In this way, we get a refinement dataset, (z, Unegariee Jn Prefined) E DRSET. The refiner is then trained with DRetner = DIsFT U DRSFT create the initial refiner Ru-
Training Strategy. For both actor and refiner models, we use standard supervised fine-tuning with the loss function:
C
log (rgl(raarrea).
where y denotes the input, signifies the target response, and N represents the length of r. For actor training, we have input q = x and target r = When it comes to the refiner, we use input q (x, y) and target r = j for DISFT, and input q = (z, Ynegative: f) and target r Frefined for DRSFT-
2.3 TREE-SEARCH INTEGRATED SELF-PLAY TRAINING
After initializing the actor and refiner models, we embark on iterative process for continuous self-improvement. In each iteration, we first collect the negative data, where the responses fail to accurately follow the instructions (22.3.I). Then, utilize tree-search algorithm to refine the negative responses ($2.3.) and form the training data for the next iteration of the actor ($2.3.3) and refiner 2.3.4). This iterative self-play pipeline allows us to continuously improve both models.
2 NEGATIVE DATA COLLECTION
that elicit responses that do not accurately follow the instructions, yielding tuples in the form of (ar, Unegative: j), where Unegative is the incorrect response and j its corresponding judgment.
2.3.2 TREE-SEARCH REFINEMENT
After collecting these negative instances, the core step is to refine the responses form preference pairs. These self-refined pairs are crucial for highlighting the subtle differences that can determine the success of instruction-follooiin tasks, thereby facilitating effective learning. Given that direct
4

---

Published as a conference paper at ICLR 2025
refinement often results in low success rate, employ a tree-search approach. We implement both breadth- first search (BFS) and depth-firss first search (DFS) strategies for this refinement. Detailed algorithms for these methods provided in Appendix B
To illustrate our process, we take BFS as an example and illustrate the procedure in Figure Starting with incorrect instruction-resoonss pair and its judgment the root node, expand the search tree level-by-level until correct response is found. At each intermediate node, we generate potential refinements for the current response and evaluate its correctness using the refiner. The number of generated refinements corresponds to the number of branches. Specifically, at a level of the tree, the refiner: 1). generates potential refinements for each node in the current level; 2). judges the correctness of these refinements. This creates a set of child nodes with new responses and their corresponding, judgments. The search process continues until we obtain tuple (ar, Unegative: Vrefined), where Vrefined is the newly refined, correct response. Importantly, SPAR combines the strengths of both tree-search and self-refinemmnnt, exploring multiple refinement paths while minimizing the interfering factors, producing effective preference learning data.
2.3.3 ACTOR TRAINING
To optimize the actor model, we leverage the refinement pairs for preference learning using DPO. At iteration i, we train the actor model M, with refinement pairs (Unegative Frefined), ), treating Unegative as the rejected response (yr) and lrreiind as the chosen response (yw.) The training dataset is denoted as D and the DPO loss is described follows:
Th ((wlla)) 3 log The (Ur(2z)
Tee yn (x: )
( Tra) = - (e.g.. log a Blog
(2)
Tea (err(Wulz))
where Th represents the actor model Mr, and the reference model Tre initialized with Mr remains fixed during the training process. This results in new actor model, M,+1, for the next iteration.
2.3.4 REFINER TRAINING
Given that the input for the refiner is templated, we RFT to obtain the new refiner Re+1- The RFT training data consists of two components: the refinement data and the judgment data for improving the refiner's corresponding capabilities.
Refinement Training Data. The refinement training data consists of tuples that capture the pro- cess of refining incorrect responses. For each incorrect response from the tree-search based refine- ment step, we collect tuples in the form of (a, yp- jp: Bretine),, where (æ. Up) represents the parent node of the final correct response in the refinement tree, and !/refined is the correctly refined response.
Judgment Training Data. The judgment training data is derived both from the negative data collection and nodes of the tree-search process. This dataset consists of tuples (æ,, Vs, where is the prompt, yi is response to x, and ji is the judgment consistent with majority voting.
Then, we perform supervised fine-tuning using the constructed training data. For the refinement data DE use the tuples (I, yp Jp. Mrefined) with input q = (z, Up: and target r 3refined For the judgment data Duddge we use the tuples (I., 3<. with input q = (x,, y:) and target r = Js. The supervised fine-tuning loss given by Eq (I). By employing this self-play training process with the tree-search based self-refinement strategy, SPAR iteratively enhances both the actor and refiner models, aiming for continuous self-improvement instruction-folloiing tasks.
3 EXPERIMENTS
3.1 EXPERIMENT SETUP
Backbone Models. We have conducted experiments on several popular LLMs:
LLaMA3 Series (MetaAl 2024) are the best best- dest--errorming models of their size, showcasing top- tier instruction-follwwee capabilities among open-source LLMss.
5

---

Published as a conference paper at ICLR 2025
GLM-4-9B- (GLM al. 2024) excels in instruction-following tasks, offering competitive performance under 10B parameters.
Mistral-7B-Instrucl @Jiag et al. 2023a) is one of the most popular LLMs and has shown good performance across wide range of tasks.
Settings. In this work, we focus enhancing the instruction crrruuctioonfolooiing following abilities of LLMs in self- play fashion. We utilize SFT to bootstrap models under 10B parameters as actor and refiner models. For the more advanced -LLMMA3-70--nstruuct we perform a three-iteration self-play training using 10k prompts per iteration from our generated dataset. In each iteration, we apply DPO for the actor and RFT for the refiner. We refer to the trained -LaMA3--B-Instruus as SPAR-8B, LLaMA--77B-Inssti as SPAR-9B and Mistral-7B-Instruee as SPAR-7B. More implementation details can be found in Appendix C Description of baseline methods is provided in Appendix D
directly employ it in both roles. Following this,
as SPAR-70B, GLM-4-9B-Chaa
3.2 EVALUATION BENCHIMARKS
As both the actor and refiner continually evolve within our framework, it's crucial to comprehen- sively evaluate both of their capabilities.
Capability. To assess the actor's ability to follow instructions, we rely on two widely-used benchmarks: IFEval (Zhou et al. 2023 and FollowBence Bench (Jiang et al.) 2023b). IFEval offers 541 verifiable instructions specifically designed for code-based evaluation. These instructions cover 25 verifiable types, including tasks like Keyword Frequency and Number of Words. FollowBench, the other hand, encompasses five categories of more subjective constraints: Content, Situation, Style, Format, and Example. This dataset features 820 meticulously curated instructions across five difficulty levels and utilizes hybrid assessment approach combining rule- based and LLM-as-judge evaluations.
Actor's Instruction-follove
Refiner's Judgment and Refinement Capability. For assessing the refiner's judgment capability, we turn to LLMBar (Zeng et al. 2023), dataset designed to measure the assessment ability of LLMs in the context of foolloition-oolloinng tasks. LLMBar includes 419 instruction-respone pairs, categorized into two subsets: Natural and Adversarial. Originally, the task involves pair-wise comparisons to identify successful and failed responses. We adapted to a point-wise judgment task, asking the model to determine whether each instruction-followee following task is successful.
To evaluate the refiner's 's capability in refinement, we split 200 the refiner after three rounds of training.
200 samples from the DESFT to create a test
set, and we employ both GPT-40 and as judges to evaluate whether the refined responses are accurately following the instructions.
3.3 ACTOR EVALUATION RESULTS
SPAR significantly improves instruction-follviii ability. As illustrated in Table I the iter- atively trained LLMs demonstrate substantial improvements in both the IFEval and FollowBench benchmarks. Remarkably, after three training iterations, SPAR-8B-DPO 4-Turb (81.3% average accuracy) IFEval. Moreover, incorporating the tree-search refine- ment technique during the inference stage significantly boosts performance. Additionally, the SPAR showcases excellent scalability with respect to model size, which substantially enhances the cestruction-ollowwi/ abilities of the LLaMA3-70B-Instree model.
even surpasses GPT-
SPAR does not damage general abilities. As shown in Appendix E.2 we assessed each itera- tion's 's performance on general benchmarks, including GSM8k (Cobbe et al. 2021). TriviaQA (Joshi et al. 2017), MMLU (Hendrycks et al. 2020), and HumanEval (Chen et al. (221)) The results in- dicate that SPAR maintains or even improves general performance, particularly GSM8k and Hu- manEval benchmarks, demonstrating that enhanced instraction-following capabilities support over- all LLM alignment.
SPAR outperforms other baselines significantly. Figure 3 demonstrates the improvements on Eval with each training iteration. In every iteration, SPAR outperforms other methods. Notably,

---

Published as a conference paper at ICLR 2025
Table 1: Main results of iteratively trained LLMSs instruction-followinng benchmarks (Cf. Table for full results). P stands for prompt level, and represents instruction level. L and S denote loose and strict evaluations, respectively. Avg. indicates average results and Lv means level. Results using inference-time tree search are highlighted in green The highest results for each backhone model
bolded. Scores marked with are sourced directly from the original paper.
IFEval
FollowBench (SSR)
Model
P (
ILL)
P())
(S)
Avg.
Lv-1
Lv-2
Lv-3
Ly-4
Lv-5
Avg.
LLaMA3-8B Models
LLaMA3-8B-
77.6
84.5
70.6
78.9
77.9
69.4
62.2
63.1
61.9
60.9
63.5
AutoIF-SBt
43.1
56.0
28.8
42.2
42.5
54.6
52.1
50.0
49.0
43.7
49.9
SELF
78.2
84.5
76.0
82.9
80.4
68.3
65.7
65.2
62.2
62.4
64.8
Humpback
72.5
80.2
70.1
78.1
60.2
62.6
64.6
Relf-Rewrd
77.3
84.2
74.1
81.7
79.3
72.8
66.6
66.8
64,9
64.1
67.0
Meta-Rewarding
77.8
84.1
75.4
82.3
79.9
73.9
71.9
66.0
62.3
62.6
67.3
SPAR-8B-SFT
75.4
82.5
73.4
80.6
78.0
73.9
67.4
68.1
63.1
61.3
66.8
78.0
84.7
75.8
82.6
64.7
62.3
67.5
SPAR-8B-DPO-iE
78.9
85.0
77.1
83.3
81.1
73.9
71.9
69.1
64.0
62.2
68.2
MAR-8BB-DPO-ter3
79.9
85.4
78.0
83.7
81.8
73.0
72.3
70.0
64.1
64.7
68.8
w/ trée search
82.4
87.5
79.5
85.3
83.7
73.9
71.7
70.3
66.8
64.1
69.4
GLM-4-9B Models
GLM-4-9B-Chal
71.5
79.9
68.0
77.2
74.2
80.8
75.1
67.4
64.3
65.4
70.6
SPAR-9B-SFT
71.5
80.5
68.8
78.1
74.7
79.4
70.9
68.2
65.1
63.7
69.5
SPAR-9B-DPO-ites
77.3
84.1
73.6
81.4
82.7
76.7
67.9
68.3
64.2
72.0
LLaMA3-70B Models
LLaMA3-70B -
83.7
88.9
77.1
83.8
83.4
77.
72.5
69.4
68.7
66.3
70.8
AutolF-70B 85.6
90.4
80.2
86.7
67.2
63.5
66.5
SPAR-70B 70B-DPO-itedt
85.6
90.2
81.3
87.3
86.1
80.3
75.7
71.4
73.7
70.5
74.3
even after three iterations, other methods fail to surpass the performance SPAR's first it- eration. Generally, method and SELF out- perform rewarding and meta-rewarding ap- proaches, underscoring the importance of learn- ing from refinement and excluding the interfer- ing factors in instruction fowlttiionoooooiingggg.o tasks. Fur- thermore, SPAR's superior performance com- pared SELF indicates that contrastive refine- ment response pairs can highlight key differ- ences, which are difficult to learn using only correct responses. Additionally, only SPAR- 8B-SFT outperforms the original LLaMA3-8B- Instruct, which suggests that incorporating the judgment SFT or refinement SFT data would reduce performance, likely due to the huge task gap and reduced diversity in the data.
82
Selfnowzeding
Mota-twarddngg
Ours
80
Avrrage Soome
78
SFT
Iterttio-1
Bentioo-22
Iteaaionn
Figure 3: Comparison with baseline methods across iterations (Cr. Figure 9 for SPAR-7B).
SPAR-8B consistently surpasses all baselines.
3.4 REFINER EVALUATION RESULTS
SPAR iteratively enhances judgment capability. Our analysis in T that SPAR itera- tions notably improve the model's ability to evaluate instruction-follovimn tasks. By iteration three, the refiner SPAR-8 PPAR-8B-REE 8B-RFT-iter3 surpasses GPT-4o-Mini, the model used to construct the judgment SFT dataset. This finding highlights the potential for continuous self- self-imprrvveentt, as the supervised fine-tuning data is not bottleneck. Interestingly, our refiner greatly outperforms GPT-4o-Min on adversarial test sets, suggesting that the similar positive and negative examples generated during tree search make model more robust against adversarial samples.

---

Published as a conference paper at ICLR 2025
Table 2: for Bestrall-B-Insttuee results.) Acc. denotes accuracy. The highest scores for each base model are highlighted in bold.
Evaluation of judgment capability for iteratively trained LLMs LLMBar. (Cf. Table
Adversarial
Natural
Average
Model
GPTInst
GPTOut
Manual
Neighbor
Average
Acc.
F1
Acc.
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
GPT-4o-Mini
74.5
70.5
69.2
61.6
60.9
51.4
59.8
51.9
72.8
66.4
65.7
57.8
67.4
60.4
L.aMAAT-BB Models
LLaMA3-SPP -B--s------
60.0
51.8
55.4
46.1
47.9
39.5
51.1
36.6
54.5
45.0
52.2
41.8
53.8
43.8
SELF
69.5
61.6
62.0
50.7
64.9
54.8
57.6
41.8
64.6
51.3
62.2
49.6
63.7
52.0
Self- Ritteeeanding
71.0
66.3
70.1
66.7
63.8
59.5
62.0
55.7
67.5
61.7
65.9
60.9
66.9
61.9
Metu-Rewurding
70.5
66.3
68.5
64.6
64,9
60.2
64.1
58.3
69.0
63.1
66.6
61.6
67,4
62.5
SPAR-8B-SFT
68.5
62.4
59.6
68.3
SPAR-SB-RFT-terl
68.5
63.2
60.6
63.8
62.0
53.3
66.8
59.0
64.9
57.1
58.3
SPAR-8B-RFT-teC.
70.5
64.2
66.8
61.6
66.0
60.0
65.2
57.9
69.0
62.4
66.8
60.5
67.5
61.2
SPAR-883-RITT-terr
70.5
65.9
70.7
66.7
63.8
57.5
68.5
63.3
68.3
62.2
67.8
62.4
68.3
63.1
GLM-4-9B Models
GLM-4-9B-Chat
74.5
76.5
74.5
75.9
57.4
62.3
53.3
56.6
69.8
72.0
63.7
66.7
65.9
68.6
SPAR-9B-SFT
70.5
65.5
72.8
70.2
S9.6
55.8
64.1
53.5
71.3
67.2
66.9
61.7
67.7
62.5
SPAR-9B-RFT-Bn
71.0
68.8
75.5
74.6
58.5
55.2
68.5
64.2
68.7
65.9
67.8
68.4
63.7
LLMMA3-70B Models
LJaMA3-70B-Insee
75.0
71.9
73.4
69.6
69.1
66.7
66.3
60.8
69.0
63.4
69.5
65.1
70.6
66.5
SPAR-70B 708-RFT--errr
78.0
74.7
78.8
76.9
64.9
61.2
67.4
59.5
72.4
68.1
70.9
66.4
72.3
68.1
SPAR progressively improves refinement capability. Table 3 demonstrates continuous improvement in refine- ment accuracy (success rate) of with each training iteration, eventually matching the level of GPT-4o-Mini, the strong LLM for SFT data construc- tion. This further showcases promising way for self- evolution cntrructionfoolowwiiii tasks. However, it also points potential issue of self-evaluation bias: when the refiner self-evaluates refinement accuracy, it performs significantly better than when evaluated by GPT-4o.
Table Refinement evaluation results. Acc-GPT uses GPT-4o as judge; -PARR uses SPAR-SB-RFT-ters.
Model
Acc-GPT
Acc-SPAR
GPT-4o-Mini
79.0
71.0
SPAR-80 SPAR-8B-SFT
73.5
71.0
SPAR-8B MAR-SB-RFF-iterl
77.5
77.0
SPAR-SB-RET-ite
74.5
76.0
-PAR-8B-RFT-iee
90.5
3.5 ABLATIONS AND ANALYSIS
Refinement preference pairs enhance instruction-folloving capability more effectively. To verify that the interfering factors indeed affect preference learning and motivate the need to highlight the key differences, we have conducted synthetic data experiment featuring two tasks:
Character Sequence Generation: The model needs generate specified number of given letters, with no restrictions on letter case, such as generating 12 letters a. For each prompt, we first construct negative response lowercase. In order to introduce disturbing factors, we have the correct response in uppercase for interfering pairs while maintaining refined pairs lowercase correctness.
Start/End Story Generation: The model is asked to generate story that starts with sentence and ends with sentence 2. The negative response lacks either sentence 2. Interfering pairs have a different story concatenated with these sentences; refined pairs keep the same story intact.
Figure shows that refinement pairs significantly outperform interfering pairs in both tasks, with larger and more effective improvements. Particularly in story generation, diverging stories results in worse accuracy than the original model. Moreover, in the character generation task, we can clearly observe that the interfering factor (upperease ratio) is learned quickly. However, the task not performed as well as the refinement setting, highlighting the necessity of focusing on key differences and excluding possible interfering factors.
8

---

Published as a conference paper at ICLR 2025
Instruction--esee Accancy Uppprruse Ratio
100-
Refinement Pains (Aec)
55.0
Refinement Pairs
Interferigg Pairs (Acc)
(Upperccse)
Insinattommieeel.cc Accuracy
52.5
Interfaring
Interfeting (Uppercase)
60
47.5
45.0
42.5
20
37.5
140
20
40
60
80
100
120
140
160
Training Steps
Training Staps
Figure 4: Synthetic data experiment results: Churacter Sequence Generation (left) and Sturt/End Story Generation (right). For Character Sequence Generation, interfering pairs show rapid learning of the upperease ratio (interfering factor) but perform worse than refinement pairs. In the Start/End Story Generation task, refinement pairs outperform interfering pairs, which even underperform the original model step O.
Table 4: Ablation study the actor.
Table Ablation study on the refiner.
IHKval
FollowBenchh (SSR)
Natural
Adversarial
Model
Prompt(S)
Instruction(S)
Avg.
Acc.
F1
Acc.
Fi
SPAR-8B- 1R--8--DDO-itrr3
78.0
83.7
68.8
SPAR-8P 8B-RFT-tter3
70.5
65.9
67.8
62.4
w/ Tree Search
-2.0
-0.8
-1.7
w/ Iterative Training
-0.9
-0.2
-2.0
Tree Search
-0.5
-1.2
w/ Refinement
-2.6
-3.1
w/o Iterative Training
-0.5
-2.5
-1.7
-3.5
Furthermore, the ablation study on actor's performance in Table further reveals significant drop when refinement data is omitted. SPAR's superiority over self-rewarding and meta-rewarding meth- ods in Table IT also underscores the importance of using refinement pairs to eliminate interfering factors. Additionally, the string-level similarity of refinement response pairs 0.90, much higher than 0.85 of the independently sampled response pairs.
Each element is crucial in SPAR. The primary elements of SPAR include the tree-search finement process and iterative training. We thus conduct ablation studies to assess the significance of these elements. For the tree-search process, shown in Table 4 excluding tree search signifi- cantly reduces the actor's performance. This might be due to a lack of difficult samples that require more iterations refine and a reduced number of preference pairs. Table I0 illustrates that tree search greatly outperforms greedy decoding in response refinement and surpasses other methods, such as best-of-N refinement simple iterative refinement. Furthermore, tree search is essential for improving judgment capability, especially against adversarial inputs, as indicated in Table 5 Simi- lar responses with opposite labels generated during the tree-search process can enhance robustness against challenging scenarios. Moreover, the results presented in Tables and underscore the im- portance itcrative training for both the actor and the refiner. This iterative training process ensures mutual improvement, which is crucial for the overall effectiveness of our framework.
Scaling test-time compute significantly boosts model performance. Inspired by the recent developments in test-time compute scaling (Snell et al.] 2024). we investigate various decoding strategies during inference on -PAR-8B-DPO-iten Figure [5] shows that in- creasing inference times remarkably enhances model performance, outperforming the results of greedy decoding. Notably, while tree search refinement's 's performance growth is slower. it ultimately achieves superior results compared best-of-N generation. This indicates that
Avenge Spore on IFTval
83.50
83.25
83.00
Best-of-N
82.50
Greedy Decoding
82.25
82.00
Inference Times
Figure 5: Comparison of decoding strategies.

---

Published as a conference paper at ICLR 2025
refinement is more powerful than generation and could be better suited for scaling test-tims compute in the instuctioonffollwwie task.
RELATED WORK
4.1 INSTRUCTION FOLLOWING
Instruction-followiin is fundamental capability of LLMs and is central to LLM alignment (Ouyang ct al. 2022; Cheng et al., 2023 Lou et al. (2024). Many studies have evaluated instruction-foll following
capabilities from various perspectives (Li et al.) 2023b; Zheng et al. 2023 Zeng et al. 2023 Liu
et al. 2023a Xia et al. 2024). With the expanding application of L.Mss, s the tasks they are expected perform become more intricate (Liu et al. (002b)), often involving composite instructions with numerous constraints. Consequently, several benchmarks have been developed to test LLMs' abil-
ity to follow these complex instructions (Zhou al., 2023 Jiang et al. 2023b Qin al. 2024
Wen et al. 2024). Additionally, multiple studies have Tocused on enhancing LLMS' instruction-
following capabilities (Lou et al. 2023] Zhou et al., 2024, Sun et al. 2024). One crucial aspect of the instruction foolowiwnn task is that subtle differences in responses can significantly impact their
correctness (Zhou al., (223). Considering this, we introduce SPAR framework construct pref-
erence pairs that reduce extraneous elements to highlight these subtle variations for effective im- provements.
4.2 AUTONOMOUS LLM ALIGNMENT
Given the high cost of manually collecting alignment data, many studies focus exploring tonomous I.LM alignment methods (Cao et al. 2024). One common strategy involves using data distilled from advanced models to improve less powerful ones (Peng al., 2023 Xu et al. 2023
Cheng et al. 2024). Alternatively, as the LLMs become stronger, several studies (Wang et al. 2023
Yuan al. 2024 Zhang al. 2024) investigate how self-evolving LLMs capabilities. Self-
Instruct (Wang et 20033 generates instructions by employing the model's 's in-context learning ability. Reinforced Self Self-Trainin (Gulcehre al., 2023) samples data from LLM policy and uti- lizes the dataset to enhance the policy through offline RL algorithms. Moreover, recent research has incorporated feedback from diverse sources. SELF (Lu al., (023) trains LLMS to acquire meta- skills of self-feedback and self-refinement, enabling the models to self-evolve iteratively. AutolF (Dong et al. 2024) introduces the code execution feedback. Self-rewarding (Yuan et al. 2024) and Meta-rewarding (Wu et al. 2024) leverage the LLM-as-judge ability to evaluate its responses, thereby constructing preference pairs. However, these methods usually direct sample multiple in- dependent responses from the actor model, which is likely to introduce the interfering factors and thus affect the model's 's capture of the key differences. Thus, we propose a new framework that con- structs preference pairs by self-refining the model's responses, minimizing extraneous elements, and promoting more effective autonomous improvement.
5 CONCLUSION
In this study, we introduce a new self-play framework. SPAR, designed to improve the instruction- following capabilities of LLMs through training with refinement pairs. We reveal that, unlike tradi- tional approaches that rely on sampling multiple independent responses from the model to construct preference pairs, refining preference pairs to minimize extraneous factors and highlight key differ- ences lead to significant improvements in instruction-follooinn tasks. Remarkably, the LLaMA3- 8B-Instruct model, trained iteratively using our framework, outperforms GPT-4-Turbo on IFEval. With inference time compute scaling, its performance can be further improved. Moreover, the it- erative enhancement of instruction forluuttioooflooiiingg judgment, and refinement abilities brought about by SPAR underscores a promising path to continuous self inffiiipooveee.....
6 ACKNOWLEDGEMENT
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604). This work was also supported by Tsinghua University Initiative Scientific
10

---

Published as a conference paper at ICLR 2025
Research Program. We would also like to thank Zhipu Al for sponsoring GPU computing and API cost consumed this study.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 22d1:2204000802. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He. Xianpei Han, et al. Towards scalable automated alignment of llms: A survey. arXiv preprint arXiv:2406.012 212220006.122 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 100v:21077.03374 2021.
Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongming Wang, Yuxiao Dong, Jie Tang, and Minlic Huang. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint (xXiv:2311.0//35 2023.
Jiale Cheng, Yida Lu, Xiaotao Gu, Pei Ke, Xiao Liu, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. Autodetect: Towards unified framework for automated weakness detection in large language models. arXiv preprint arXiv:2406. 16714, 2024.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-folloiing capabilities of large language models. arXiv preprint arXiv:2406. 13542, 2024.
Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongming Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang. Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 130b to glm-4 tools, 2024.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.00 0.89988808,, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.000 033000 2020.
11

---

Published as a conference paper at ICLR 2025
Zhenyu Hou, Yiin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, et al. Chatglm-rlhff Practices of aligning large language models with human feedback. arXiv preprint arXiv:2404.0092 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23/0.06883 2023a.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang. Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint w:Xiv:23/0.2044 10, 2023b.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqu: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1601-1611, 2017.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint 23w.:::30000005599 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooinng models. atttp::/ggithut www.tut..ooottttttsuuouou watp:alaaaaaaaaaaaaaaaaaaaaaaaaa
2023b.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, al. Best practices and lessons learned synthetic data for language models. arXiv preprint arXiv:2404.07503. 2024.
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang. Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18744 2023a.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating ilms agents. arXiv preprint arXiv:2308.03666 2023b.
Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. Muffin: Curating multi-faceted instructions for improving instruction-ollooiing arXiv preprint arXiv:2312.022436 2023.
Renze Lou, Kai Zhang, and Wenpeng Yin. Large language model instruction following: A survey of progresses and challenges. Computational Linguistics, pp. 1-10, 2024.
Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. Self: Language-driven self-evolution for large language model. arXiv preprint 2rXiv:231000055: 2023.
MetaAI. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL https://ai.meta wtw.aaccooccccccccoeee ccom/blog/mett///eco 1lama
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Baolin Peng, Chunyuan Li. Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint wrXiv:2304.03277, 2023.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401. 240,0000, 2024.
12

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. Identifying the risks of Im agents with an Im-emulated sandbox. arXiv preprint arXiv:2309. 15817, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022)755-759,
2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 24Xx:22008.33// 2024.
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained fnslluction-oolloiing ability of large language models. arXiv preprint arXhv:2404.02823 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprini awXiv:2203..1111 /1171, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 6lst Annual Meeting the Association for Computational Linguistics (Volume 2023.
1: Long Papers), pp.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. Benchmarking complex instucttion-olllviin with multiple constraints composition. arXiv preprint arXiv:2407.03978, 2024.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with ann-a--a-meta-judgee arXiv preprint ar arXiv:2407.1 195944 2024.
Congying Xia, Chen Xing, Jiangshu Du Xinyi Yang. Yihao Feng, Ran Xu, Wenpeng Yi,, and Caiming Xiong. Fofo: A benchmark to evaluate lims' format-following capability. arXiv preprint arXiv:2402.1 186667 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.00 018000 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng. Tanya Goyal, and Dangi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self- training via process reward guided tree search. arXiv preprint arXiv:2406.0388 038/6 2024.
13

---

Published as a conference paper at ICLR 2025
Hanyu Zhao, Zhao, Li Du, Yiming Ju, Chengwei Wu, and Tengfei Pan. Beyond iid: Optimizing instruction learning from the perspective instruction interaction and dependency. 2024. URL https:
//arxiv. org 220.//249000099
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 466633 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followig evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
A DATASET INFORMATION
Constraint Taxonomy. We take the taxonomy from Cheng et al. (2024), and further refine it to be more comprehensive ensure the diversity of our prompts. The refined taxonomy shown in Figure
Length
Text Format
Format
Character Format
Punstuation
Numeric Format
Keyword
Symbol
Insllocti
Content
Sentence
Language
Situationi
General
Language Style
Sentiment
Figure 6: The detailed taxonomy constraints for prompt evolution.
Prompt Template. Here, we give the prompt for constructing complex prompts in Figure 7 For the refiner, the prompt template for judgment is provided in Figure As for the refinement task, form as a multi-tumn task after judgment, with the prompt template provided in Figure g
B TREE-SEARCH ALGORITHM
We show the detailed process of BFS and DFS refinement in Algorithm I and Algorithm 2
C IMPLEMENTATION DETAILS
The SFT dataset for the actor comprises 8k examples, while the refiner dataset includes approxi- mately 9k examples for judgment training and 3k for refinement training, formatted as multi-turn
14

---

Published as a conference paper at ICLR 2025
Prompt Construction Template
In this task, you need to refine the procpt to make it more specific and complex. Transfer the prompt to an nntructionnfollowiii
enhance specificity, refer constraints from the following txxonomy: [Taxonomy]
prompt: [[surt]] [[end]]
Main comstraint:
[Main Constraint]
You must include the main constraint (unless the main constraint contradicts the original prompt), and need to add 1-3 other construints:
2. You need
that these constriints reasonable and do conflict with each other. 3. The added constraints need to make sense with the original prompt and not conflict with it. ensure constraints
diverse possible, you can types not mentioned taxonomy above. 5. The priority is given constraint rypes other than longth constraints.
Do not added constraints in points.
7. Don't
response your refined prompt.
8. the Prompt a mathermttic or coding problem, just output "Nome" as the refined prompt.
conflicting constraints unreasonable constraints original task, just output refined prompt. 10. Never involve visual elements.
output the following format: [[start]] [your refined prompt]][end
Figure 7: The prompt template applied for prompt evolution.
Judgement Template
expert in ovaluating the capabilitios cettruction-fllowwiie In cirnectionffollwwiin Output nceds ceeesstlyypeciielyllle follow the given Prompt. Your task to earefully judge whether the Output bonestlyjpreisslyyllosdd follows the given Prompt. there are any constraints in the Prompt that not satisfied the Output, please list all the constraints that not satisfied.
Prompt: "Instruction)"
Output: "[Response)"
Please carcfully judge if each coustraint is perfectly satisfied and give final judgement weather the Output accuratoly follows the Prompt the following format:
Step by-step verification: xxx
Final Judgement the Output accurately follows Prompt):: (Yees or No)
Refinement Template
Based your judgement, refine Outpot sure honestlyipeccidyy/deeeey follows the given Prompt.
Please carefully refine the Outpat to meet all the constraints the Prompt.
Plense format like this:
Reflection on refine the Output: xxx Final Refined Output: [[start] [[snd]]
Figure The prompt template applied for the refiner's judgment and refinement.
15

---

Published as a conference paper at ICLR 2025
Algorithm MFFF-Refinemen
Algorithm 2 SDFS-RReinement
Require: Instruction æ, Response y, Judgment J. Require: Current state 8, depth I, Refiner Rv. depth limit d, threshold utas branch limit b if > then record output & = (z,
Refiner Rv, depth limit d, branch limit b.
So {z.3.j}
for / = -,.... do
end if
E St-1.// Ry(s,b)) b get judgment S, + {1z, Ji St,j' Vi((s)
S: - [[x.
for s' E Rs, b) do if Ry's') < Veh then DFS(s', +
> refinement judgment
V, < (S))
end for
end if end for
return arg max,es, (s)s)
task following the first turn's judgment. These two datasets are both constructed with GPT-4o-Miin Both the actor and refiner are trained with a learning rate of 2e-6 and a warmup ratio of 0.1, using the A optimizer with 31 0.9 and /3 0.999. The actor is trained over 5 epochs with batch size of 64, and the refiner trained for 3 epochs with the same batch size. In the data construction process, we a tree search budget of to strike balance between performance and efficiency. The average number of expanded tree nodes is around 3.7 in our experiments, which is an acceptable level. Specifically, for 3.4 across different iterations, demonstrating decreasing trend as the model becomes stronger. For the actor iterative training, each iteration uses around 5k examples for DPO. To enhance training stability as suggested by (Hou al. 2024), an additional SFT loss is added to the chosen response with a weight of 0.1. Here, the learning rate is set to 2e-7. 3 to 0.1, with a warmup ratio of 0.1, and training is conducted for epoch with batch size of 32. For the refiner, each iteration utilizes about 10k examples, including 4k refinement samples. We ensure the judgment training dataset maintains balance of positive and negative samples. The training configuration remains the same for SFT, except the learning rate is set to 1e-6. All experiments are performed on an 8x80G Nvidia A100 setup.
the average expanded node numbers 4.3, 3.7, and
For our baseline methods, we have maintained uniform settings to ensure fairness. For SELF, we initialize with our constructed datasets, D Actor and DRefiner- In the case of self-rewarding and meta-rewarding, we start with Dcttrr and DJSFT- For Humpback, we create the seed dataset by combining about 3k data from the Oass dataset and 5k data from Actorr We also control the number of training samples to be nearly identical for fair comparisons.
D BASELINES
We compare our method with four popular self-improvement approaches, including:
AutolF (Dong et al.
2024) incorporates code feedback and online DPO training to improve instraction-following ability in both distillation and self-evolution settings.
SELF (Lu et al.
2023) proposes leveraging language feedback to guide response generation in
order to achieve iterative seff-improvement.
Self-rewarding (Yuan et al. 2024) proposes to combine the reward model and policy model to enhance alignment capabilities simultaneously.
Meta-rewarding [Wu et al. 2024) further introduces a meta-judge to address judgment capa- the self-rewarding framework.
bility limitations, building
Humpback (Li et al. (023a) proposes training an instruction generation model synthesize high-quality data using web resources.
E EXPERIMENT RESULTS
E.
INSTRUCTION-FOLLOWWNS EVALUATION RESULTS.
The evaluation results instruction-follovinn benchmarks shown in Table Our method outperforms all baselines on these benchmarks and show substantial improvements in each iteration (Figure 9.
16

---

Published as a conference paper at ICLR 2025
E.2 GENERAL PERFORMANCE EVALUATION
Our analysis in Table reveals that SPAR training not only doesn't harm general performance, but it can also even bring enhancements.
E.3 JUDGMENT EVALUATION RESULTS.
As shown in Table 8 the judgment capability improves in each iteration and the accuracy outper- forms all baselines.
E,4 ABLATION STUDY ON JUDGMENT CAPABILITY.
In experiments, employ majority voting for iterative improvements for judgment capability. We show the results of the refiner SPAR-SB-SFT's sampling times and performance on LLMBar in Table 9
To balance the performance and computation time, we choose majority voting@5..
E.5 ABLATION STUDY ON REFINEMENT CAPABILITY.
Table I0 shows the results of different decoding strategies for the refinement task on SPAR-8B. For methods except greedy decoding, we use the same inference budget. We can see that the tree search algorithms largely outperform other methods, verifying the importance of incorporating tree search refinement.
E.6 INFERENCE-TIME SCALING COMPARISON
Figure I0 presents a comparison between SPAR and self-rewarding, focusing on their scalabil- ity with regard to inference times, measured by the number of response generations in our study. Our analysis includes both the LLaMA3-8B-Intrrue and Mistral-7B-Insttee models. The results demonstrate that SPAR outperforms the self-rewarding method when additional computational re- sources are allocated for inference time, leading to enhanced performance.
76
Self-rewarding
Meta-rewarding
74
Average Scores on IFEval
SBLF
Ours
72-
70
68
SFT
Ieraaiion11
lteration-2
Iteration-3
Figure 9: Comparison with baseline methods across iterations. SPAR-7B consistently surpasses all baselines.
17

---

Published as a conference paper at ICLR 2025
71
SPaR
SPalt
SelErewading
Sellmmandingg
IFEval Performance
80.0
IFEval Performance
69
79.6
68
79.4
79.2
79.0
66
10
Inference Times
Inference Times
Figure 10: Inference-time scaling comparison IFEval. The left panel showcases results for
LLaMA3-8B-Instruc) while the right panel presents findings for Mistral-7B-Instruet.
Table 6: Full results of SPAR-7B, SPAR-9B, and SPAR-70B instruction-follooing benchmarks. P stands for prompt level, and represents instruction level. L and S denote loose and strict evalu- ations, respectively. Avg. indicates average results and Lv means level. Scores marked with sourced directly from the original paper.
IFEval
FollowBench (SSR)
Model
P (L)
(L)
(S)
(S)
Avg.
Lv-1
Lv-4
Lv-5
Avg.
Mistral-71 7B Models
Mistral-7B-Instruet
55.1
64.9
49.9
60.22
57.5
65.1
61.6
61.6
56.8
57.2
60).4
SELF
71.3
79.7
68.0
76.9
74.0
71.5
64.2
60.8
58.0
57.0
62.3
Humpback
60.4
71.0
56.6
67.6
63.9
70.7
63.9
63.8
59.8
57.9
63.2
Self-Rewarding
64.3
73.5
61.0
70.7
67.4
70.8
64.8
62.3
61.9
58.3
63.6
Meta-Rewarding
65.1
74.7
61.0
71.1
68.0
73.2
64.6
64.5
60.6
57.6
64.1
SPAR-7B-SITT
62.7
72.3
59.3
68.7
65.8
74.4
64.3
62.5
58.2
55.0
62.9
SPAR-7B-DPO-ier
68.2
76.6
64.7
73.6
70.8
73.2
63.1
60.3
56.6
63.6
BE-R-7B-DPO-ies
70.0
78.1
74.2
72.0
57.5
63.8
SPAR-7B-DPO-Heer
74.1
80.9
69.7
77.1
74.6
61.0
64.7
GLM-4-9B Models
GLM-4-9B-OC
71.5
79.9
68.0
77.2
74.2
75.1
64.3
70.6
SPAR-9B-SFT
71.5
80.5
68.8
78.1
74.7
79.4
70.9
68.2
65.1
63.7
69.5
SPAR-9B-DPO-ie
73.8
81.2
70.6
78.5
76.0
82.6
76.0
67.9
64.9
63.6
71.0
D.AR-9B-DPO-ie
76.7
83.3
73.2
80.9
78.5
80.4
76.6
67.4
68.7
64.1
71.4
77.3
84.1
73.6
81.4
79.1
82.7
76.7
67.9
68.3
64.2
72.0
LLaMA3-70B Models
LLaMA3-70B Instruct
83.7
88.9
77.1
83.8
83.4
77.1
72.5
69.4
68.7
66.3
70.8
A
85.6
90.4
80.2
86.7
85.7
66.2
64.6
63.5
66.5
SPAR-70B-DE
70B DD-O-iterr
84.5
89.2
80.2
85.7
84.9
77.6
74.0
70.2
70.6
66.9
71.9
SPAR R-RR8---DPP---e
85.0
89.4
81.5
87.2
85.8
80.4
76.4
69.9
73.7
70.2
74.1
SPAR-70B-DPO-iter
85.6
90.2
81.3
87.3
86.1
80.3
75.7
71.4
73.7
70.5
74.3
18

---

Published as a conference paper at ICLR 2025
Table Performance on general benchmarks. SPAR maintains the model's general capabilities.
Model
GSM8K
TriviaQA
MMLU
HvmanEval
Average
Mistral-7B Models
Mistral-7B-Inan InIsrructt
42.9
72.5
57.9
32.9
51.6
SPAR-7B-SFT
56.4
72.8
56.7
44.5
57.6 (16.0)
FEAR-7B-DPO-ice.
55.6
72.2
55.3
46.3
57.4 (45.)
SPAR-7B-DPO-ittr2
54.4
72.1
55.8
45.1
56.9 (+5.3)
SPAR-7B-DPP -
58.2
71.6
55.1
46.3
57.8 (16.2)
LLaMA3-8B Models
LLaMA3-8F (1-A3388-Innee Instructctt
-anttat-B-Inttructt
75.4
75.9
63.6
55.5
67.6
SPAR-SB-SFT
75.6
76.0
64.0
61.6
69.3 (+1.7)
SPAR-8B UPO-iterl
78.8
75.2
63.8
60.4
69.6 (v2.0)
MARR-BB-DP--itet
77.0
74.9
63.1
60.4
68.9 (+1.3)
3PAR-SB-DPO-iter3
77.7
75.1
63.1
60.9
69.2 (+1.6)
GLM-4-9B Models
GLM-4-9B-CHI
80,6
69.7
71.9
74.3
74.1
SPAR-9B-SS 9B-SFT
82.9
69.4
71.8
73.8
74.5 (+0.4)
SPAR-9B-DPO-iter
82.6
68.8
71.6
75.0
74.5 (10.4)
SPAR-9B-DPO-iE
82.8
68.9
71.8
73.8
74.3 (40.2)
83.0
69.0
72.1
73.2
74.3 (+0.2)
LLaMA3-70B Models
LLaMAA3-0BB-mstruut
92.2
87.2
80.8
79.3
84.9
SPAR-70B-DPO-tterl
92.5
90.4
81.0
79.3
85.8 (10.9)
SPAR-70B-DPO 2PO-ie2--
92.9
89.5
80.4
78.7
85.4 (+0.5)
PPAR-70B-DPO-iter
93.4
86.7
80.6
79.9
85.2 (+0.3)
Table Judgment evalution results on LLMBar for SPAR-7B, Acc. stands for accuracy.
Adversarial
Natural
Average
Model
GPTInst
GPTOut
Manual
Neighbor
Average Acc.
Acc.
Acc.
Fl
Acc.
F]
Acc.
Fl
Acc.
Fl
F11
Acc.
Mistral-77--Instrunl
58.0
69.1
57.1
68.8
50.0
64.1
45.6
61.5
47.8
62.6
50.1
64.3
51.7
65,2
SELF
68.0
65.2
71.2
68.7
56.4
56.8
62.0
52.6
67.5
62.3
64.3
60.1
65.0
61.1
Self-Rewarding
68.0
69.0
59.6
53.7
63.0
57.5
69,4
64.3
59.8
60.6
Meta- Rewarding
67.5
62.4
71.7
56.4
51.8
63.0
56.4
66.8
62.1
59.7
65.1
60.3
SPAR-7B-SFT
69.5
63.9
71.7
67.5
55.3
48.8
55.4
45.3
69.4
62.3
63.0
56.1
64.3
57.6
SPAR-7B-RE ---7--rFT-etrrrl
67.0
62.1
66.3
62.7
56.4
52.9
60.9
52.6
64.2
60.7
61.9
57.2
63.0
58.2
R-R--RFF-etr
68.0
64.4
68.5
64.6
60.6
57.5
62.0
52.1
64.2
60.0
63.8
58.5
64.7
59.7
SPAR-7B 3
72.3
57.4
62.6
Table 9: Comparison of decoding strategies on LLMBar.
Natural
Adversarial
Method
Acc.
Fl
Acc.
Fl
Greedy Decoding 68.0 60.7 63.9 55.1
Majority Voting@3 69.0
60.8 63.7
54.5
Majority Voting @5 68.5
60.9
64.7
56.5
Majority Voting @7 66.5 58.8
65.7
56.7
Majority Voting@9 69.0
61.2
65.8
57.1
19

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of different decoding strategies for refinement task. Ace-GPT stands for the accuracy of using GPT-4o as judge, and Ace-SPAR for the accuracy of using SPAR-8B-RFTiice judge.
Method
Acc-GPT
Acc-SPAR
Greedy Decoding
69.5
65.0
Best of N
74.0
80.0
Iterative Refinement
71.0
82.0
BFS
79.0
90.5
DFS
79.0
90.0
20


Reviews:
[{"review_id": "uDYTFuHKf1", "reviewer": "Reviewer_pYV7", "summary": "This study presents SPAR, a self-play framework that enhances LLM‘s instruction-following capabilities by training with refined preference pairs. Unlike traditional methods that rely on independent response sampling, SPAR refines pairs to reduce irrelevant factors, thereby emphasizing critical distinctions, leading to notable improvements in instruction adherence. SPAR’s iterative process enhances instruction-following, judgment, and refinement, offering a pathway for continuous model improvement.", "questions": "1. Why are judgment and refinement performed by the same model? What would happen if they were separated, or combined with the actor model, using a single model for all tasks?\n2. I haven't closely checked the details of the baselines. Do they use a strong LLM, or do they rely solely on the model being evolved?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The motivation makes sense. The fine-grained refinement is essential for further improving the model's instruction-following abilities.\n2. The design of the proposed method is sound, allowing it to effectively achieve its intended motivation.\n3. The experiments are comprehensive, demonstrating relatively strong performance.", "weaknesses": "1. The applicability of the method may be limited. It might be suitable primarily for further improvement of models that already possess strong instruction-following capabilities, as the experiments were conducted on models that had already undergone instruction fine-tuning. Additionally, a strong LLM is required for warm-up training before iteration (This also raises concerns about the fairness of comparisons.), and one of the goals of dataset construction is to introduce more complex instructions.\n2. Missing comparison with a key baseline：Self-Alignment with Instruction Backtranslation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ByEtSqwQ3", "reviewer": "Reviewer_2gma", "summary": "The paper introduces SPAR (Self-Play with Tree-Search Refinement), a self-improvement framework that enhances the instruction-following capabilities of LLMs by minimizing extraneous factors and highlighting key differences in preference pairs. This method involves an iterative training process where a model (actor) performs tasks, and a paired model (refiner) evaluates and refines the imperfect responses using a tree-search algorithm through structured feedback loops. The authors evaluate SPAR with two LLMs on the IFEval and FollowBench benchmarks. Additionally, they contribute a dataset with 43k complex instruction-following prompts and an SFT dataset that can improve the instruction-following capabilities of LLMs.", "questions": "1. The paper only lists results from the first three iterations, and the data indicate that the model's performance still has room for improvement. Could you provide a simple analysis of when the model might reach optimal performance?\n\n2. Does the framework heavily depend on the model's initial performance? Can it be directly applied to the raw models provided officially?\n\n3. It is suggested to directly specify the strong LLM used in Section 2.2.2", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Effective in reducing noise. By minimizing content variations in preference pairs, SPAR helps the model focus on essential elements, which improves its instruction-following accuracy.\n\n2. Comprehensive ablation experiments. The authors conducted extensive ablation studies to verify the impact of interfering factors on preference learning and to assess the rationality of each component in the framework.\n\n3. Generalization without degradation. The approach does not degrade general language model capabilities, suggesting a balanced enhancement in alignment without compromising overall functionality.\n\n4. Contribution of datasets. The authors provide valuable datasets that benefit the development of this research area.", "weaknesses": "1. Limited validation across models. The effectiveness of the method was validated on only three models. Further exploration is needed to assess the framework's applicability to other models.\n\n2. Reliance on complex setup and compute resources. The framework's iterative training, including tree-search refinement and multiple model roles, may require significant computational resources. Therefore, the performance-cost trade-off needs further clarification. \n\n3. Lack of comparative details. The paper lacks sufficient details in its comparisons with other methods, such as how each baseline initializes the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEfye1NU6b", "reviewer": "Reviewer_7aGY", "summary": "This paper proposes a novel self-play framework (called SPAR) integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions, so as to teach models to recognize the key differences that lead to improved instruction following. To gain a good start for Actor and Refiner, the authors construct a high-quality dataset with 43K complex instruction-following prompts and an SFT dataset for improving the instruction-following capabilities of LLMs. Through extensive experiments on several LLMs, the authors demonstrate the effectiveness of proposed SPAR.", "questions": "See weaknesses. \n\nIn addition:\n\n(1) line 527, GPT-4-Turbo or GPT-4o-mini?\n\n(2) Can you compare the training cost of SPAR, Self-Rewarding and Meta-Rewarding?\n\n(3) Does more iteration brings higher performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed approach is intuitive and has strong motivation.\n\n* This paper is well-written and presents clear ideas.\n\n* The authors conduct extensive experiments to validate the effectiveness of proposed SPAR.", "weaknesses": "* **SPAR introduces additianal training overheads.** SPAR initially requires constructing additional data to train the actor and trainer. Building on this, it needs to incorporate iterative training with tree search and self-consistency, which greatly increases the training cost compared to self-rewarding.\n\n* **Some crucial information is missing in the experiment section.**  For example, what is the average number of search nodes in the tree search, and does it decrease with the iterations? How does LLaMA3-70B perform at different iterations (SPAR-70B-SFT, SPAR-70B-DPO-iter1, SPAR-70B-DPO-iter2)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPjfM9EWje", "reviewer": "Reviewer_AshF", "summary": "The authors introduce SPAR, an automated and scalable approach designed for self-improvement in instruction-following tasks through self-play. The core idea is to create paired responses with minimal irrelevant variations, allowing for precise training of the model's instruction-following capabilities. In the SPAR framework, the authors fully leverage test-time scaling: using tree search to obtain higher-quality data for training the model's instruction-following abilities, and using self-consistency to acquire higher-quality data for training the model's discriminative and refinement abilities. Experimental results show that the SPAR framework significantly outperforms various self-critique baselines.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Constructing tailored and distinct instruction-following response pairs for the model by eliminating irrelevant content is a strong motivation for enhancing the model's instruction-following abilities.  \n- The SPAR framework's proposal to use test-time scaling during the training phase to obtain high-quality data for training the model's \n- The experimental setup is reasonable, and the results appear promising.  \n-  The writing in the paper is clear and easy to understand.", "weaknesses": "Using test-time scaling (more accurately, inference-time scaling) during the training phase to obtain high-quality data for self-critique is well-motivated, but it undoubtedly introduces significant training overhead. Therefore, providing a detailed comparison of the training costs of different methods, or comparing the gains when the costs are aligned, would make the paper's conclusions more convincing.\n\ntypo: line 239 needs a blank after 'refiner'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)