Self-Boosting Large Language Models with  Synthetic Preference Data
Authors: Qingxiu Dong
Li Dong
Xingxing Zhang
Zhifang Sui
Furu Wei
Venue: ICLR 2025 Poster (2025)
Keywords: preference optimization
synthetic data
LLM alignment
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces a self-boosting mechanism for LLMs, enabling them to self-synthesize preference data for iterative improvement.

Abstract: Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and  eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

Published as a conference paper at ICLR 2025
SELF-BOOSTING LARGE LANGUAGE MODELS WITH SYNTHETIC PREFERENCE DATA
Zhitang Sui State Key Laboratory of Multimedia Information Processing, School Computer Science, Peking University Miirosooft Research
Qingxiu Dong Li Dong Xingxing Zhang?
Furu Wei
ABSTRACT
Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is resource-intensive and creativtty-dmaadding process, especially for the continual improvement of L.MMs. We introduce SynPO, self-boosting paradigm that leverages synthetic prefer- ence data for model alignment. SynPO employs an iterative mechanism wherein self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs autonomously learn the generative rewards for their own outputs and eliminates the need for large- scale annotation of prompts and human preferences. After four SynPO itera- tions, Llama3-8B and Mistral-7B show significant enhancements in instruction- following abilities, achieving over 22.1% 1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.
INTRODUCTION
Large Language Models (LLMs) have made remarkable progress in following user instructions and generating honest, harmless, and helpful responses (Achiam et al., 2023: Dubey et al., 2024). This advancement is primarily achieved in the model alignment stage, which involves training reward models or LLMS directly on datasets curated from human preferences (Ouyang et al., 2022b; Bai et al., 2022a) typically employing Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022b) or Direct Preference Optimization (DPO) (Rafailov et al., 2024).
Recent research has made significant strides in model alignment by collecting high-quality prefer- ence data (Hu et al., 2024), sampling and ranking on-policy responses (Meng et al., 2024; Wu al., 2024b), or introducing LLM-as audge Cui et al., 2023). However, most work still relies on static, pre-collected preference datasets from human stronger L.MM annotation. As LLMS improve rapidly, collecting large, high-quality prefer- ence data for effective learning becomes increasingly challenging and costly, whether from humans or stronger models (Shi al., 2023). According to Yin al. (2024), directly sampling preference pairs, which closely resembles an on-policy setting, can result in performance declines due to in- herent volatility and inefficiency. Therefore, constructing effective preference data continuously improve LLMs remains critical research problem.
substitutes for human preferences (Yuan et al., 2024;
In this work, we present self-boosting paradigm for LLM alignment, SynPO. This paradigm lever- ages a small set of supervised fine-tuning (SFT) data to steer the generation of synthetic preference data, thereby enabling L.MSs to iteratively extend their capabilities through optimizing on synthetic data. To support iterative preference learning across diverse scenarios, SynPO first trains a self- prompt generator to create large-scale synthetic prompts. Unlike previous approaches that require more powerful LLMs and instruction examples (Wang et al., 2022), our generator utilizes only the LLM itself and three random keywords as input. To generate preference pairs for the synthetic prompts, SynPO utilizes the initial model generated responses rejected candidates and employs response improver to refine these responses into chosen ones. The response improver comes from two straightforward intuitions: (1) LLMS excel at identifying distribution gaps between texts (Zhong

---

Published as a conference paper at ICLR 2025
Model
Size LC (%)
WR (%)
gpt4 1ee__116__pP______ _i0e__eeeieeww
50.0
50.0
28.6
GPT-4 (03/14)
35,3
22.1
Length-controo
Win Rate [
25
1asaama---77--nnttttue
70B
34.4
33.2
134
Mistral-Base-SynnO Nerr
7B
34.0
36.4
20
Mistral Large (24/02)
123B
32.7
21.4
Mistral-Base-SynPO Ierr
31.7
13.3
GPT-4 (06/13)
30.2
15.8
10
10.6
Claude
28.2
17.2
Claude 2.1
27.3
17.0
5.4
Mistral-Base-Syn'O Iter
7B
25.7
28.1
SFT
Iter
Iter 2
Iter 3
Iter
gemini-pro
24.4
18.2
Liama3 8B (Durs) m...:Laama3-700 1a033--0--nnntteee Instruct
GGTT 0613
Mixtral-8x 1a-1--8x7--Issret---0
8x7B
23.7
18.3
Mistral-7B (Ours) -iistral---agg--222222 Gemini-Pro
Botrrl-7B-Instruuee
7B
17.1
14.7
GPT-4-0314
Distral-Base-Synle Ieerl
7B
13,3
15.3
Mistral-Baae Bas
7B
6.6
3.6
Figure 1: Length-controlled win on Al- pacaEval 2.0 improves with SynPO iterations, approaching GPT-4 level for the base versions of Llama3-8B and Mistral-7B.
Table 1: Results AlpacaEval 2.0 leaderboard. LC and WR represent length-controlled and raw win rate, respectively.
al., 2022; Singh al., 2022), and (2) refining a response is generally easier than generating high- quality response from scratch (Madaan et al., 2023; Lu et al., 2023; Nguyen et al., 2024). In each iteration, train the same initial model be response improver, focusing identifying distri- bution gaps between current model outputs and gold standard responses in seed data. We then use the response improver refine the initial model outputs, thereby providing generative rewards to the responses. Through iterative training the initial model on synthetic preference data, SynPO al- lows the LLM make subtle improvements and gradually push its boundaries. By leveraging small high-quality data and the current model state to guide the generation of synthetic data, we introduce stronger supervision in an iterative manner.
Experimental results demonstrate that SynPO not only benefits LLM alignment with human pref- erences, but also improves generalist capabilities across various tasks. Trained solely on synthetic data, SynPO significantly improves the coatuuction-olllwiinn abilities of Llama3-8B and Mistral-7B (as shown Figure/ and Table 1), achieving over a 26% length-controlled win rate improvement AlpacaEval 2.0 (Dubois et al., 2024) and 22% 30% improvement on Arena-hard (Li et al., 2024c) (as shown in Table 2). Furthermore, self-boosted models achieve 3.2% to 5.0% higher aver- age performance than SFT models the Open L.M leaderboard (Beeching et al., 2023), indicating SynPO also enhances general LLM performance.
To summarize, contribution includes:
We introduce SynPO, a self-boosting mechanism that iteratively induces LLMs synthesize high- quality data for training. Without requiring human-annotated preference data, SynPO significantly enhances the diversity and quality of synthetic prompts and responses.
SynPO dynamically guides LLMs to improve their own outputs, using pre- and post-refinement generations as synthetic preference pairs for training. This approach effectively integrates gener- ative rewards for preference learning. enabling LLMs to gradually push their boundaries.
SynPO significantly enhances both the instruction followiinnn capabilities and the general perfor- mance of LLMs, showing substantial improvements over three to four iterations.
SELF-BOOSTING LLM WITH SYNTHETIC PREFERENCE DATA
SynPO self-boosting scheme designed to iteratively generate high-quality preference data, An overview of SynPO is presented in Figure 2. It begins with small set of SFT data as seed data, denoted as {(x;, >>>>>>>>>> and the initial policy model T By incorporating both the self-prompt generator and the response improver, SynPO provides sufficient prompts for iterative training and leverages the generative rewards in the synthetic preference data. This approach allows the policy model make subtle improvements and gradually expand its boundaries.
2

---

Published as a conference paper at ICLR 2025
R Response Improver Training
Preference O Optimization
Input Given promptx" and current model response yi-1, improve the response.
Seed Data
y
(x",y"
Next-iteration model
Output: y" (seed data response)
Seed prompt
x²
M
Vi-1
Rt
Train response improver for M,-1
M₁
yynthetic Preference Data Creation
Training
Synthetic
Refine the original response with R₂
Select pairs
Preference Dala
prompt
x
M
yt
yt-1
(x > yol
Figure 2: Overview of SynPO in the tth iteration. Starting with the previous iteration model Mc-1. SynPO first learns a response improver Rg to identify discrepancies between model responses (w--1) and gold standard responses (v") seed data, and learns to refine model responses. Subse- quently, on the self generated prompts x (elaborated in Section 2.1). SynPO employs R, to refine the M,-1 responses (Yt-1) into improved responses (y1-)) The valid synthetic prompts æ, refined responses (y1-1) 1), and initial model Mn responses (yn) to form synthetic preference data. These data incorporated into the synthetic preference dataset for preference optimization, resulting in updated M, for the next iteration. The iterative process continually enhances L.LM capabilities in instructionfoolloiing and task performance.
Question Generation Prompt
Background
Design ONE question/lustructan (and its solution) with given keywords.
### Requirements
The question question/Instruction /insrructtiinn should to contain the keywords in '## Given Keywords., The solu- MUST BE CORRECT, including detailed reasoning steps and dense knowledge.
## Given Keywords
(keywords]
Output Format
= GENERATED QUESTION & CORRECT SOLUTION ==
question>> question content </question>>
<. </solution> correct solution </solution>>
# Output
GENERATED QUESTION & CORRECT SOLUTION
Figure 3: Prompt used in SynPO for LLMs self-prompt generators.
2.1 SYNTHETIC PROMPT CREATION
Diverse and ample prompts are crucial for effective preference learning (Shi et al., 2023; Yuan al., 2024; Song et al., 2024). Diversity facilitates generalization and a sufficient number of prompts allows data selection from large candidate pool. In SynPO, we propose novel strategy for syn- thetic prompt generation. We design keywords-to-text task to guide the training of a self-prompt generator and create pscudo-label data from the seed SFT data.
Self-Prompt Generator Training We train the LLM itself to serve as high-quality prompt gen- erator. For each prompt x² in seed data, we randomly extract two keywords from x and one noise keyword from x5. where j E (1, n) \ {i). The inclusion of the noise keyword enhances the robustness the prompt generator. learns to filter out irrelevant keywords during training and ensure that the generated prompts are fluent. This process yields keyword list, k; for x². Next, insert k; into a prompt template (see Figure 3) to create a prompt and use (x'y?) as the correspond- ing completion. This process constructs training data for the prompt generator. We then optimize 6o through SFT to transform the model into a prompt generator g. 9 possesses the capability to generate unlimited, diverse, and high-quality user instructions, controlled by the given keywords.
3

---

Published as a conference paper at ICLR 2025
Mupe and youR
costure
12.
UltraReadback
Self-instruct
Sports and firees
Education
Density
UltraChat
Farily parstting.
Health and wethees
Banulation
others
Cheste
media
carner
weite
History nostalgia
0.05
6.15
0.25
Pershout development
Speek
noneenal Eamiing and technology
Cusitt Money and finance
Average Similarity
Enerept Extreprenerraan
Cruativlly
Literttire and weiting
Science and Innuration
Whlloruphy and envics
Falitics and eeents
Figure 5: Inter-prompp similarity distribu- tions for 1,000 randomly sampled prompts from SynPO, UltraFeedback (Cui et al., 2023), Self-Instruct (Wang et al., 2022), and Ultra- Chat (Ding al., 2023). We used Sentence- Transformer (Reimers & Gurevych, 2019) to compute sentence embeddings and calculated the cosine similarity between each prompt and others, then averaged these values for each prompt. The results suggest that our method, SynPO, generates more diverse prompts with lower inter-prompt similarity.
style
Travel
Entettaliinen
Olher teshhul
Figure 4: The top 25 most common topics (outer circle) and the top most common intentions (inner circle) in SynPO generated prompts. We aggregate the other topics and in- tentions to the 'Others' group.
Keywords Sampling and Prompt Generating To enhance the overall diversity of prompts, we sample keywords from pretrain corpus paragraphs. We select three keywords from a same para- graph to maintain the inherent distribution between keywords, and sample from different paragraphs to ensure overall diversity. Specifically, we randomly sample keyword lists from the RefinedWeb paragraphs (Penedo et al., 2023) and generate synthetic prompts, denoted (x, 1m X; ~ G (((1k)).
Benefiting from the self-prompt generator, our approach requires only a small SFT data to enable the model to generate diverse prompts independently, no in-context learning examples predefined topic lists are required. Due to the high combinatorial possibilitics of keyword sampling. the self- prompt generator can produce huge variety of synthetic prompts for preference learning.
For further analysis, we randomly sampled 1k self self-generated prompts from the Llama3-88 ---3---BB-u BB-Base prompt generator and used GPT-4 Turbo classify the intentions and topics behind these prompts! The results, shown in Figure 4, demonstrate significant diversity across various topics and user intentions. Even compared prompts from GPT3.5-Turbe (Ding al., 2023) or collection of prompts from different sources (Cui et al., 2023), as shown in Figure 5, SynPO generated prompts exhibit lower inter-pret piterpprompt similarity and greater diversity.
2.2 SYNTHETIC PREFERENCE GENERATION
A primary challenge in leveraging synthetic prompts is the lack of high-quality responses to pro- vide sufficient supervision (Li al., 2024b). To address this, we introduce response improver to enhance the quality of model responses to synthetic prompts. Pre- and post-improvement responses naturally become the rejected and chosen candidates, respectively, with the chosen ones providing clear guidance what approximates gold standard response.
Response Improver Training each iteration, we train the LLM as a response improver to further reduce the gap between policy model outputs and gold standard responses. Formally, let To.-1 denote the policy model at the beginning of the f-th iteration. We generate outputs from Te.- for the seed data prompts: Y[r-1),; with the seed data responses, form the training set for the response improver, following the template provided in Appendix B. Each training example consists of the prompt and the policy model output (i-1))))) as the input, and the gold standard response y as the output. We fine-tune na, on the
(+)x), i E {1, .I, m). These outputs, along
Experimenttl details in Appendix
4

---

Published as a conference paper at ICLR 2025
training set to obtain the response improver R2. This response improver refines the policy model outputs, aligning them more closely with the gold standard responses.
Response Improving Subsequently, we use R₁ refine model responses to synthetic prompts, obtaining pre- and post-improvement responses as synthetic preference pairs. For each synthetic prompt XG, we first obtain the current model output (t-1)),5 ~ To, (-Ix,), for E {I...., .......,... m.. The response improver then refines this completion produce ((t-1),i ~ R2 the chosen response. As we fine-tune the initial model in each iteration, the initial policy model output Y()), serves as the on-policy rejected response for x,. Here, Y(o), To, (.x,). This method generates numerous synthetic preference candidates, including both chosen and rejected responses.
considered
Data Filtering Unlike data from humans or strong teacher LLMs, which come with clear standard responses, self-generated data require proper filtering to ensure quality (Gulcehre et al., 2023). As our policy model improves, many responses no longer need refining; we only need to retain data with a preference gap between the chosen and rejected responses. Instead of using GPT4-Turbo- as-a-Judge for data filtering (Rosset et al., 2024), SynPO employs only a small model (e.g., 0.4B PairRM (Jiang al., 2023)) or the model itself for scoring. Similar to SimPO (Meng et al., 2024) and SPPO (Wu et al., 2024b), this ensures the process does not rely a more powerful teacher model. We retain Y(t-1), and ((),, with significant preference differences (i.e., large score gap). (t- is regarded as the chosen response, y,,, while Y(o). is regarded as the rejected response, y!- Along with the corresponding prompt x,. they form valid instance (x,, y)) All valid data is then integrated into the synthetic preference data for subsequent iterations.
2.3 SYNTHETIC PREFERENCE OPTIMIZATION
The large-scale synthetic preference data naturally facilitate the multi-iteration process of self- boosting. In each iteration, we follow SimPO (Meng et al., 2024) for training; actually, our method also compatible with other preference optimization training methods, such as DPO (Rafailov al., 2024) and KTO (Ethayarajh et al., 2024). Denoting DD as the synthetic preference data, we have:
log no, (yl x.) -7)
0. + arg min E (x. log 0
log To (y." x;)
and are hyperparameters. Different from the vanilla SimPO, SynPO is iterative process and all the preference data are synthetic ones. The response improver continuously refines the generation distribution align with the ideal data distribution across multiple iterations.
Overall, the response improver automatically learns to generate implicit generative rewards for the outputs of the LLM. Unlike using a discriminative reward model straightforwarddyy this approach helps the model learning to improve its outputs. We present the Synthetic Preference Optimization algorithm in Appendix A. The entire optimization process is performed on synthetic preference data, requiring only small amount of high-quality data for validation. This strategy maintains two key advantages: (1) Compared the limited and hard-to-collect preference data, SynPO generates an unlimited amount of new self-synthetic data to meet the needs of iterative model improvement. (2) Using small, high-quality validation data prevents the model from deviating during training and consistently guides the generation of more relevant synthetic data.
EXPERIMENTS
We carry out comprehensive experiments demonstrate the effectiveness of SynPO in enhancing model alignment and improving general model performance.
3.1 EXPERIMENTAL SETUP
Models and Training We perform synthetic preference optimization on both Mistral-Bass 7B and Llama3-8B Base. Following Meng et al. (2024), we employ supervised fine-tuned models as the initial models. Specifically, the Mistral-Bast 7B model (mistralai/Mistral- 7B-v0.1) and the Llama3- 8B Base model (meta-llama//Metall -areeaaa-aaa------------ were fine-tuned on the UltraChat-200 dataset
5

---

Published as a conference paper at ICLR 2025
Mistral-Base (7B)
Llama3-Base (8B)
Data Construction
AlpacaEval 2.0
Arena-Hard
AlpacaEval 2.0
Arena-Hlard
T.C (%)
WR (%)
WR (%)
L.C (%)
WR (%)
WR (%)
SFT
6.6
3.6
2.0
5.4
3.1
2.7
Manual Collection
21.5
20.8
16.8
22.0
19.8
23,2
Cumpling-Rankinng
Iterl
6.5
4.4
4.1
7.2
4.3
4.4
Sampling-Ranie Ranking
Iter2
9.3
6.4
7.7
Sampling-Rankinn Ranning
Iter3
10.6
7.5
13.8
Sampling-Rani Ranking
Iter4
11.6
8.0
9.6
14.2
8.4
10.4
Self- Rewarding
Iterl
19.5
19.8
11.9
20.1
20.3
20.8
Self Rewarding
Iter2
22.4
23.5
19.2
21.7
22.4
20.5
Self- Rewarieng
Iter3
24.6
26.3
20.8
22.4
24.1
23.8
Saf-RRearddin
Iter4
26.1
28.0
21.1
24.8
25.6
25.0
SYNPO Ieerl
13.3
15.3
10.6
10.7
SYNPO Iter
25.7
28.1
20.8
23.4
24.1
24.6
SYNPO
Iter3
31.7
33.8
24.1
28.6
31.5
32.5
SYNPO Her4
34.0
36.4
22.8
32.1
33.6
31.4
Table Results on AlpacaEval 2.0 and Arena-Hard. LC and WR denote length-controlled and raw win rates, respectively. After four SYNPO iterations, Mistral-Base and Llama3-Base increase LC by 27.4% and 26. 7%, respectively, on AlpacaEval 2.0. Arena- SYNPO achieves the highest WR by the third iteration, improving both models by over 22.1%.
as part of the Zephyr (Tunstall et al., 2023) training pipeline. Subsequently, we utilize 18k seed data to SFT the self-prompt generator and then generate 50k synthetic prompts per iteration. For data filtering, we employ the 0.4B PairRM (Jiang al., 2023) as small pairwise scoring model for the Mistral-Base 7B. For 1lam33-Base 8B, we use Llama3 itself ArmoRM--Llama--- -B-v.0. 10)) as scoring model for data filtering, given its superior alignment with human scoring (Meng et al., 2024). In each iteration / = 1,.... 7, we use model Te, from the previous iteration to generate synthetic preference data and then preference optimize the initial models again. More details on training parameters, filtering thresholds, and implementation environments provided in Appendix C.
Seed Data Construction We randomly sample UltraFeedback (Cui et al., 2023) prompts and their GPT-4 Turbo completions as our seed data. The seed data is multipurposely transformed for the training of self-prompp generator, response improver, and the validation of synthetic preference op- timization. The complete UltraFeedback dataset contains 61k instructions from sources including TruthfulQA (Lin et al., 2021), FalseQA (Hu et al., 2023), Evol-Instruct (Xu et al., 2023a), Ultra- Chat (Ding al., 2023), and ShareGPT (Chiang al., 2023). To construct seed SFT data with high-quality responses, randomly sampled 18k prompts and obtained the corresponding comple- tions generated by GPT-4 Turbo.
Baselines As baselines, we use the initial supervised fine-tuned models and those optimized with data from various preference construction methods, including manual collection and iterative ap- proaches. We recognize the UltraFeedback preference data (Cui al., 2023) as product of manual collection. is gathered from six high-quality datasets and various models, with prefer- ences annotated by GPT-4 (Achiam al., 2023). For iterative construction approaches, we com- pare against Sampling-Rankinn (2024) and
and Self-Rewarding For Sampling-Ranking, similar to Meng et al. al. (2024b), we use LLMs sampling five responses per prompt in each iteration. The same scoring models, i.c., PairRM and ArmoRM-Llame --a--amm------o 10.1, then used to select the highest and lowest scoring responses as the chosen and rejected responses, respectively. For the Self- Rewarding method (Yuan al., 2024), generate preference data based model's via LLLM-as-a-Judge prompting. We employ our 18k seed data as the initial instruction-folloviii data. Given that Self-Rewarding requires additional LLLM-as-a-JJdge training data, generate 16k
rewards
httpps https/:://hgggggggggceee co/a1 gwwentt http:///eoooooaaeeeeeoooooooo//e http://gg.....oooccccttooooooooo c http////w......oo///o/o.....ii.. 1tt.://mmmMiiiiaa------000
aft- full and https:

---

Published as a conference paper at ICLR 2025
Mistral-Base
Llama3-Base
Data Construction
Reaserinng 5.2
Turn 1 Turn
Turn 1 TTrn 2
Math 5.2
bay
LFm
SFT
6.04
5.65
6.55
5.36
8.9
Uaama3-iter
Uama3-lter2
Manual Collection
6.73
6.82
7.29
7.00
Laama3-iter
Shapling-Ranking Iters
6.83
6.18
7.06
6.99
Llama3-itera
Self- Rewarding Itery*
6.71
6.63
7.30
7.28
Coding 6.0
Writing
SYNPO Herl
6.53
6.41
6.99
6.65
SYNPO Iter
6.66
6.65
7.34
7.30
SYNPO Ite3
6.86
7.34
7.34
SYNPO Iier
6.73
6.69
7.43
7.04
9.8
1..
Extraction
Humanities
Table Multi-turn evaluation on MT-Bench. An asterisk (*) denotes the best score across multi- ple iterations. For Sampling-Rande Ranking, Llama's best is from iteration 4 and Mistral's from itera- tion For Self-Rewarding. both are from itera- tion 3. SYNPO progressively enhances the multi- turn instruction mnstruction-folloinng capabilities of LLMs.
STEM
Figure Radar chart for Llama3-8B-Ban SYNPO on MT-Bench. SYNPO achieves notable improvements across various prompt categories, particularly in RolePlay, STEM, Reasoning, and Coding tasks.
seed data with GPT-4 Turbo. For all settings, adopt SimPO (Meng al., 2024) for preference optimization. More detailed are elaborated in Appendix C.5.
3.2 PREFERENCE ALIGNMENT
We evaluate the model aligmment performance on three benchmarks: AlpacaEval 2.0 (Dubois et al., 2024), Arena-Hard (Li et al., 2024c), and MT-Bench (Zheng et al., 2024). AlpacaEval 2.0 includes 805 user prompts and utilizes pair-wise comparison with LLMM-as-aJJudee Specifically, the win rate against the baseline GPT-4 Turbo model determined based on GPT-4 Turbo evaluation. Arena- Hard includes 500 more challenging
queries, employing GPT-4-Turbo judge the model sponses against GPT-4. MT-Bench features 80 multi-turn questions spanning various domains, with GPT-4 scoring the model responses out of 10.*
Single- Turn Dialogues We compare the Eestructionfoollwiigg and human preference alignment capabilities on AlpacaEval 2.0 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024c) in Table 2. Compared to the initial model SFS SynPO shows sustained improvement over four iterations in win rate against GPT-4 Turbo or GPT-4. On AlpacaEval 2.0, Mistral-Base achieves a 27.4% increase in length-controlled win and 32.8% increase in raw win rate after four iterations. Similarly. Llama3 exhibits 26.7% rise in length-controlles win rate and a 30.5% improvement raw win rate after the same number of iterations. In the more challenging Arena-Hard setting, SynPO reaches the highest win rate after the third iteration. Compared to the baseline methods, SynPO's iterative preference learning on synthetic data yielded more significant improvements.
Multi-Turn Dialogues For the multi-turn benchmark MT-Bench, we report both the first-turn and second-turn scores (in Table 3) as well as radar chart depicting performance across different ques- tion types (refer to Figure 6). The results indicate that SynPO enhances not only first-turn perfor- mance, with an increase of over 0.7 points, but also subsequent turns, with increase of over 1.2 points. Compared to the initial model, SynPO shows improved performance across various question types, particularly in humanities, writing, STEM, and roleplaying.
3.3 DOWNSTREAM TASK PERFORMANCE
In terms of the general model performance on various tasks, we report the average scores on the well-recognized Open LLM Leaderboard (Beeching et al., 2023) and 6 additional benchmarks from Language Model Evaluation Harness library (LLM Harness) (Gao al., 2024). Open LLM Leaderboard (Beeching et al., 2023) is recognized as a standard assessment for the general per-
'More details on API usage for LLM judgement are listed in Appendix E and G.
SSimilar results for Mistral-7B shown Figure 10 in Appendix G.

---

Published as a conference paper at ICLR 2025
Model
Are
HellaSwag
TQA
MMLU
Winogrande
GSM8k
Average
Mistral-Base-SFT
58.02
80.89
40.37
58.78
76.40
34.72
58.20
Manual Collection
62.71
83.39
50.69
58.47
77.35
32.83
60.91
Campling-Rankingg
Iters
60.32
81.80
44.43
59.09
76.95
36.85
59.91
Self-Rewarding Iters*
60.15
81.84
43.25
58.98
76.48
34.72
59.24
Iterl
60.49
82.25
50.36
59.00
76.48
36.39
60.83
Mistral-Base- S
Ite2
63.65
83.24
58.04
58.74
76.48
27.35
61.25
Mistral-Basee -aee--aase----PP-
SYNPO
Iter3
63.54
83.14
58.11
58.37
75.77
25.26
60.70
Mistra--Baxe-SNNOO
hte4
63.57
83.04
56.
58.75
75.77
31.08
61.39
LLama3-Base-SFT
60.92
81,28
45.37
63.80
76.72
51.93
63.34
Manual Collection
66.72
82.89
59.47
63.10
77.82
45.72
65.95
Sampling-Ranking Ranking Iters
66.38
82.71
59.84
63.37
77.27
54.40
67.33
Sell-Rewarding Iters*
64.76
82.48
55.54
63.42
77.03
54.59
66.30
LLama3- Base--ase-----ee SYNPO
Iterl
63.99
82.66
54.20
64.02
77.51
56.10
66.41
-Lamaa--ase-S/ SYNPO Iter2
65.70
83.22
61.73
64.03
76.56
56.25
67.92
-Lama3-Base -aae--SNPPOOO
Iter3 66.55
83.57
63.53
63.91
76.80
55.27
68.27
Lamaa-Base-SYNPO
Iter4 66.47
83.44
63.69
63.79
76.90
55.72
68.34
Table 4: Open LLM Leaderboard results. TQA stands for TruthfulQA. The asterisk (*) represents the best performance across multiple iterations. Compared to the SFT versions, SYNPO achieves an overall improvement of 3. 19% for Mistral and 5.00% for Llama3 on the average score.
formance LLMs. includes six different datasets, evaluating LLMs on commonsense rea- soning (Are (Clark al., 2018), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021)), wide knowledge (MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2021)), and math (GSM8K (Cobbe al., 2021)). The six addtional LLM Harness tasks include Openbook Question Answering (OBQA) (Mihaylov et al., 2018) and Haerae (Son et al., 2023) for model knowledge, MathQA (Amini al., 2019), XNLI (Conneau al., 2018), and PROST (Aroca-Ouellett al., 2021) for reasoning, as well as Toxigen (Hartvigsen et al., 2022) for toxicity evaluation."
Open LLM Leaderboard of 3.1%% of Mistral-Base-SFFT and 5.00% of Llama3-Base-SFT on the average score. Specifically, SynPO achieves over 6% improvement on the ARC challenge and over 16% on TruthfulQA com- pared the SFT model, after four rounds of self-boosting, Notably, the performance of Mistral-Base on GSM8K has experienced decline, whereas Llama3-Base has demonstrated an improvement of nearly 4 points the same benchmark. This disparity likely stems from the superior data filtering capability ArmoRM- MaMMm---aa3-88 -B-vo0. compared to the 0.4B PairRM. ArmoRM MMMMM---ama--8 --. effectively mitigates erroneous responses and enhances mathematical problem-solving performance.
the Open LLM leaderboard, we observe an overall improvement
LLM Harness Tasks The advantages above are also reflected in the results for more diverse tasks in LLM Harness, as evidenced by Table 5. Previous works (Wu al., 2024b; Meng al., 2024) demonstrate that preference optimization can induce the "alignment tax" aligning models with human preferences can improve performance for only 1~2 iterations or even degrade over- all performance on downstream tasks (Askell et al., 2021). Our method exhibits similar behavior on MathQA; however, overall, SynPO shows improvements across more iterations on other tasks. This is because synthesizing better chosen candidates introduces additional supervision, partially mitigating the alignment tax issue and enabling LLMs to continuously enhance their capabilities on downstream tasks at the same time of alignment.
4 ABLATION STUDIES
4.1 SYNTHETIC PROMPTS AND RESPONSES
We have demonstrated the diversity of prompts generated by SynPO in Section 2.1. To fur- ther validate the self-prompt generator, compare the generated prompts with manual collected
We follow the standard few-shot setting Open L.M Leaderboard, as elaborated in Appendix G. For the six additional tasks, employ fixed 5-shot setting for evaluation.
8

---

Published as a conference paper at ICLR 2025
Model
OBQA
Haerae
MathQA
XNLI
Toxigen
PROST
Average
Mistral Base-SFT SFT
46.40
39.96
36.25
43.76
60.11
52.04
46.42
Manual Collection
50.20
40.35
36.72
44.65
62.83
54.66
48.24
Campling-Rankinng
Iters*
47.60
40.15
36.21
44.71
62.45
53.74
47.48
Rell-Rewarding Iters*
48.60
40.15
36.25
44.17
61.28
53.29
47.29
Histral-Base-Syntee
Ierll
48,00
39.78
36.05
44.23
60.74
$2.39
46.87
Mistral-Baan
Iter
50.40
40.05
36.85
43.97
60.96
53.04
47.55
Bastral-BaseSyyme
Iter
51.20
40.51
35.88
44.37
60.74
53.45
47.69
Mistral- -aael-aasS--PP SynPO
Iter
51,40
40.88
36.48
44.47
63.40
55.05
48.61
Llama--Bac-SSH SFT
46.20
61.78
42.04
45.47
68.83
52,40
52.79
Manual Collection
51.60
61.59
42.51
44.77
74.38
55,96
55.14
Sampling-Rane Ranking Iters*
50.80
62.05
42.81
46.67
74.68
55.41
55,40
Self- Rewarding Iters*
48.60
62.42
42.78
46.31
74.15
54.23
54.75
Llama3- Slama--Base-ae Base- SynPO
Ierl
48.20
61.96
42.71
46.15
71.17
53.77
53.99
Iter2
50.80
62.60
42.75
46.34
74.04
54.85
55.23
Llama3-Bass Baera--BeeSSSie -SyPP
Iter3
51.00
62.51
42.78
46.37
75.11
55.29
55.51
Llama3- Haama--Base-oe Base-SynPP SynPO
Iter
52.00
62.51
42.65
46.27
75.18
56.14
55.79
Table 5: Downstream performance in each SynPO iteration on six tasks in LM Evaluation Harness. An asterisk (*) represents the best performance across multiple iterations.
prompts (Cui et al., 2023) and Self-Instruct prompts (Wang et al., 2022). For each prompt construc- tion approach, randomly sample 20k prompts and construct response pairs through both SynPO and Sampling-Rante Rannng.. We compare the single iteration results of Llama3-8B. As shown in Table 6, whether through self-refinement or Bampling-Ranking, synthetic prompts generated by SynPO lead to better-aligned models, validating the quality of these prompts. It is worth mentioning that SynPO prompts are even more effective than the superset of its seed training data, UltraFeedback prompts. This increased effectiveness may be attributed to the greater diversity of SynPO prompts, achieved through the keyword sampling process in prompt synthesis. Results of mixing SynPO and manual collected prompts further indicate the potential of SynPO in augmenting existing prompts.
4.2 IMPACT OF SEED DATA
SynPO involves training LLMs solely on synthetic preference data while using seed SFT data for validation. To investigate the maximum impact of the seed SFT data, we compare SynPO with the following settings: 1) Seed SFT: Directly fine-tuning the LLM using seed data. 2) Seed PO: For each prompt the seed SFT data, using the gold standard response in the seed data as the chosen response and the initial policy model response as the rejected response for preference optimization. 3) Seed SFT+ PO: To avoid distribution shifts in directly using seed SFT data, we first obtain a model fine- tuned on seed data as in 1), then construct preference data using the model output and gold standard responses. 4) Seed SFT + "O"": Training on data from 3) for multiple epochs. 7 The results on AlpacaEval 2.0 are presented in Table 7. Among the evaluated methods except for SynPO, setting 3) is most analogous to SynPO, and proves to be the most effective. However, due to the limited quantity of seed data, the improvement is less than that achieved by iterative SynPO on synthetic data. Training under such conditions for multiple epochs does not yield further improvements and even degrades performance. These findings validate that SynPO a promising approach construct preference data and maximize the utilization of minimal high-quality data.
5 RELATED WORK
Preference Data Construction Preference data are triplets consisting of user prompts, user- preferred responses, and non-preferred responses. Acquiring preference data from humans can be resource intensive, often constrained by the data collection platform (Ouyang et al., 2022a) or the cost of human annotation (Bai al., 2022a; Ethayarajh al., 2022; Nakano et al., 2021). To allevi- ate this problem, researchers have started using teacher LLMs, such as GPT-4 (Achiam et al., 2023),
We compare the results of epochs and 3 epochs and select the hetfer-performing 2 epochs. More experi- mental details Appendix

---

Published as a conference paper at ICLR 2025
Prompts
SynPO
Method
LC (%)
WR (%)
LLC (%)
WR (%)
LC (%)
WR (%)
Seed SFT
20.1
19.7
Manual Collection
23.8
25.6
7.8
6.2
Seed PO
11.6
10.7
Self-Instruct
21.7
21.4
9.1
4.6
Seed SFT + PO
24.6
20.9
SynPO
24.3
24.5
9.2
5.3
Seed SFT PO
22.4
15.0
SynPO Mix.
23.4
29.4
14.2
6,9
SynPO Iter4
32.1
33.6
Table 6: Comparison of various prompt generation methods on AlpacaEval 2.0. SynPO Mix. combines SynPO prompts with manually collected prompts.
Table 7: Impact analysis of seed data on AlpacaEval 2.0. PO"H refers to preference optimization over multiple epochs.
to simulate human preferences (Cui et al., 2023; Ding et al.. 2023; Huang et al., 2024). Given user prompts and candidate responses, this line of work employs stronger model annotate prefer- ences, thereby overcoming the scarcity and constraints of existing preference data (Cui et al., 2023). However, the prompts still need to be collected (Cui al., 2023), which limits the domain, diversity, and quantity of the data. Both human annotation and the utilization of large teacher model APIs incur substantial costs (Shi et al., 2023). Moreover, relying solely on reward scores or win-lose annotation fails fully capture the subtleties and complexities of human preferences.
LLM Self-Boosting Previous work has advanced the self-boosting LLMs by searching for high-reward behaviors (Tian et al., 2024; Zhang et al., 2024), using LLMs as judges to select sponses (Yuan et al., 2024; Wang et al., 2024a; Wu et al., 2024a; Kim et al., 2024), and leveraging self-play strategies (Chen et al., 2024; Cheng et al., 2024; Luo et al., 2024; Wu et al., 2024b). These works typically use a fixed set of cxisting prompts, limiting the LLM ability to learn across wide scenarios. Furthermore, the deterministic reward signals in these methods do not help the model recognize subtle discrepancies between its responses and ideal responses. Prior to our work, Con- stitutional AI (Bai et al., 2022b) and SELF (Lu et al., 2023) used AI generate non-deterministic feedback for training. Constitutional AI used refinement data for reward models, while SELF em- ployed GPT-4 for data generation and taught models relf-refinemet However, these methods did not utilize comparative information between pre- and post-revision texts for training.
Synthetic Data for LLMs Acquiring human-generated data is costly and time-consuming. leading to the use of synthetic data for LLM training (Wang et al., 2022; Xu et al., 2023a; Li et al., 2024b). Unnatural Instructions (Honovich et al., 2022) and Self-Instruct (Wang structions to generate new prompts, while WizardLM (Xu et al., 2023a) and WizardMath (Luo et al., 2023) rewrite these instructions into more complex forms using ChatGPT. Seed topics also produce textbook-like data (Li et al., 2023; 2024b) or self-chat dialogues (Xu et al., 2023b; Ding et al., 2023) for instruction tuning. These methods often require strong LLMs or examples, benefiting from model distillation (Xu et al., 2023a; Li et al., 2024b;a). Our approach uses the model itself to gener- ate prompts and responses without needing carefully designed topics, extending beyond the model's inherent sampling space. This brings generative rewards to the LLM self-boosting process, partic- ularly benefiting initially weaker models. It resembles direct preference knowledge distillation (Li al., 2024d) but does rely large-scale teacher model responses for supervision. Additionally, our work shares a similar spirit with CommonGen (Lin et al., 2020) in generating sentences from keywords and CHEF (Seo et al., 2023) in creating synthetic data for contrastive learning.
al.. 2022) use seed in-
6 CONCLUSION
We introduce self-boosting LLM with synthetic preference data, SynPO, a method for LLM align- ment through iterative training on synthetic data. In SynPO, we innovatively base the entire training process on synthetic data and only employ limited SFT data for validation. SynPO diversifies the prompts and dynamically guides LLMs to improve their own output, using pre- and post-refinement generations as synthetic preference pairs for training in the next iteration. Experimental results show that SynPO leads to significant improvements performance. This strategy sheds light on high-quality synthetic data generation and self-alignment with minimal supervision, both of which critical for the continuous development of L.Mss.
both cntruuction-oolloiiit capabilities and task
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This paper is supported by NSFC project 62476009.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 2rXiv:2303.08774, 2023.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms, 2019.
Stéphane Proca-Ouellette Cory Paik, Alessandro Roncone, and Katharina Kann. PROST: Physical reasoning about objects through space and time. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4597-4608, Online, August 2021. Association for Compu- tational Linguisties. URL https: / /aclanthologyy org/ 2021 f findings-aee 404.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova Dassarma, Nelson Elhage, Zac Hatfiel-Doddss Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A general lan- guage assistant as a laboratory for alignment. ArXiv, abs/21 1 00061, 2021. URL https: //api. semant cacholar atchollrr.oo. org Swtg/CCrruusttteeeo 244799669
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli. Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kemionn, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodss, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeff Ladish, Landau, Kamal Ndousse, Kamilé Lukoiūté, Liane Lovitt Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noem'i Mercado, Nova Das- sarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravee, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. ArXiv, abs/2212.0800 2022b. WtLhttts::/and wwwantiinniiommm casholaa...... Ttg/Corpus 254222448338
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm
leaderboard. attps://huggingg cw.cee.ccceee httpeeccccppppppppeeeeeeeeee leta-llm-eesdd Lem-Leadebborrd-oddddd
open laa__lmmmmmaaa__ www_eeaderbbard. 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du. Self- playing adversarial language game enhances Ilm reasoning. ArXiv, abs/2404. 10642, 2024. URL. https:/ //api.
semamtnntt nt escholar www./orgoooooo ctgg/corpusipp 269157364.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An
11

---

Published as a conference paper at ICLR 2025
open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: ///mmys.....
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 000507, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2018.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrrfeedback:: Boosting language models with high-quality feedback, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learming. volume 162 of Proceedings of Machine Learming Research, pp. 5988-6008. PMLR, 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. ArXiv, abs/2402.01306, 2024. URL https:/ api. wewante wwc.nticccclar...... org/ Corpus 26740606::666688888
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL htt.t::/://// http::::///eeoo...om org/ records/ 12608602.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Ab- hishek Sharma, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, A. Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling. ArXiv, Abs/2308.08999 2023. URL aittps://apll COrpusID:2610340
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap. Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generatte dataset for implicit and adversarial hate speech detec- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 03300, 2020.
12

---

Published as a conference paper at ICLR 2025
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv, abs/2212.09689, 2022. URL https: / //api wwwanticscholeeeeeeeeeeee Itt.:aap....r.ooo....D.... 254455555
Shengding Hu, Yi-Xiao Luo, Huadong Wang. Xingyi Cheng. Zhiyuan Liu, and Maosong Sun. Won't get fooled again: Answering questions with false premises. In Annual Meeting of the Associa- tion for Computational Linguistics, 2023. URL https: /api sppi... semant cschol .r.gg// org/
CoZpusID:25934177
Yulan Hu, Qingyang Li, Ouyang Sheng, Ge Chen, Kaihui Chen, Lijun Mei. Xucheng Ye, Fuzheng Zhang, and Yong Liu. Towards comprehensive preference data collection for reward mod- eling. ArXiv, (as/22006114880 2024. URL https: /aaps:///ap...//// www.nmiccmattchooooeeoo org// CorpusID:270700
Shengyi Costa Huang, Agustín Piqueres, Kashif Rasul, Philipp Schmid, Daniel Vila, and Lewis Tunstall. Open hermes preferences. htttps://uugginggaal 2024.
/t/ddaaaatt///rgiiiaaaeaa aarillaa/
Dongfu Jiang. Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anmual Meeting of the Association for Com- putational Linguistics, 2023. URL https::////. //aps:///p....// http://ap...mmanssssseeeees www.nticscholeeeee......eeeeeeee wtt.:aarr.o//ooouIDDD
259075564.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-ICCIIPP), pp. 2567-2577, 2019.
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Aligning large language mod- els with self-generated preference data, ArXiv, abs/2406.04am 04412, 2024. URL https://api.
semant remanticschoaaz wttpe/ollarrrrrrerroeeeeeeoeee wtt/ Catp//orpussaaa I 270357971.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Process- ing. pp. 785-794, Copenhagen, Denmark, September 2017. Association for Computational Lin- guistics. doi: 10. 10.18653/v//D17--U0
URL https:
aanthology.ore -
Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang. Weidong Guo, Yukang Gan, Jeffrey Xu Yu, Dan Tong Wang. and Ying Shan. Hrlaif: Improvements in helpfulness and harmlessness in open-domain rein- forcement learning from ai feedback. ArXiv, abs/2403.08309, 2024a. URL https://api. semant cscchoaar Stgg/orppusiii ID 268379322
Haoran Li. Qingxiu Dong, Zhengyang Tang. Chaojun Wang. Xingxing Zhang, Haoyang Huang, Shaohan Huang. Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang. Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. ArXiv, abs/2402.1300 2024b. URL https: //api. semanticschola: Io:puSID:2677530
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonza- lez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and bench- builder pipeline. ArXiv, 1tt//2406.11999, 2024c. URL https: https///..p...aaaaaaaa.ee semanticscholar: :t:
13

---

Published as a conference paper at ICLR 2025
Yixing Li, Yuxian Gu, Li Dong, Dequan Wang. Yu Cheng, and Furu Wei. Direct preference knowl- edge distillation for large language models. ArXiv, 1bb/2200..1974,, 2024d. URL https:
sep...msatncccoollaa... wtt. 270845777777775
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprinl arXiv:2309.0544 2023.
Percy Liang Rishi Bommasani, Tony Lec, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Erie Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel J. Orr, Llcia Zheng, Mert Yük- sekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Hen- derson. Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Trans. Mach. Learn. Res., 2023, 2023. URL https: http////peeeeeeeeeeeeeeeeeeeeeee net/ fest/foornte 2
Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Dipanjan Das, Hannanch Hajishirzi, Andrew McCallum, and Sameer Singh (eds.), Conference on Automated Knowledge Base Construction, AKBC 2020, Virtual, June 22-24, 2020, 2020. URL https: / www. akbe w 2020/ papers/ yytp
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0990 2021.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020.
Jianqiao Lu, Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi, Baojun Wang. Weichao Wang, Lifeng Shang, and Qun Liu. Self: Self-evolution with language feedback. 2023. URL https: // /appi. semant www.chccccaacccccoooo org/
wwweaaarrrrrrrrrorreeeeeeeee Corppsrrpuuuaaa : 223334133555
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.099 095583. 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. Arena learning: Build data flywheel for llms post- training via simulated chatbot arena. ArXiv, abs/2407.10627, 2024. URL https://aag semantics www.nticscchaar.. Terg/CorpusIDD 0e::2272233099
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Itera- tive refinement with self-feedback. ArXiv, abs/2303.116551 2023. URL https://api. semanticschol. ar. wwweerrg////eee waag//Coruustin 20:2:57900 257900877.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 24Xiv:4405.14734, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories, 2016.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question with human feedback. In arXiv, 2021.
14

---

Published as a conference paper at ICLR 2025
Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, and Xian Li. Better alignment with instruction back-and- translation. 2024. URL https:
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022a.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022b. URL https: /ttp//app..... sem.t/ cscholar. CorpusID: : 246426909.
Samuel J. Paech. Eq-bench: An emotional intelligence benchmark for large language models, 2023.
Anselmo Peñas, Eduard H. Hovy, Pamela Forner, Álvaro Rodrigo, Richard F. E. Sutcliffe, and Roser Morante. Qa4mre 2011-2013: Overview of question answering for machine reading evaluation.
In CLEF,2015
Guilherme Penedo. Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 11, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxxt
arg/abs// /1t///9900 1908. 10084.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. ArXiv, abs/2404.03715, 2024, URL https: //pppapppppp/// semanticscoooeeeee... org/ CorpusID: 268987748..88888.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, 2019.
Jachyung Seo, Hyeonseok Moon, Jaewook Lee, Sugyeong Eo, Chanjun Park, and Heuiseok Lim. CHEF in the language kitchen: A generative data augmentation leveraging korean morpheme ingredients. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10, 2023, pp. 6014 Association for Computational Linguistics, 2023. doi: 10.18653/V1/0033.1 B...23..MMPP-PP. 1MMNL--MANN366. URL https::///// stttps://doi.. 1ttp://0..863//122222222 ma emm.p-main..3
Taiwei Shi, Kai Chen, and Jieyu Zhao. Safer- Aligning language models with automated preference data. ArXiv, abs/2311.08688 2023. URL https:://// app.ppi.... wewpo.ssmmmtnnccchhheooooo
org/CCrrpsID: 265212888.
Chandan Singh, John Xavier Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao. Explaining patterns in data with language models via interpretable autoprompting. 2022.
15

---

Published as a conference paper at ICLR 2025
Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jaecheol Lee, Je Won Yeom, Jihyu Jung, Jung Woo Kim, and Songseong Kim. Hae-rae bench: Evaluation of korean knowledge in language models, 2023.
Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li. Scaling data diversity for fine-tuning language models in human alignment. In Inter- national Conference on Language Resources and Evaluation, 2024. URL https:/ //pp:///a /appi.
semanticscholass wwwphholrrrrrrraaaoeeeeeeeeeeeee Serg/CorpussD) : 268512826.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self- improvement of Ilms via imagination, searching, and criticizing. ArXiv, abs/2404.12255 2024.
URL httph:tp:///// pttttpps//ap...mmaaa.t.. wwwantssseaiiiieaaeaaeaa.c... iescholaa. org/ Co
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Tianlu Wang. Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-YY Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi,, Jason Weston, and Xian Li. Self-taught eval- uators. 2024a. URL https: apii wwwantiesccaaaeeeeeee
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https:// api. wweemntiisccol... org/ wtw.:aaaaa...oooooosiIIiaa. :oo.pssD::254877011
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024b.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-metaa-uddee 2024a. URL mnttps://agp semant i cecttclccclarr..o.co org CorpusID: 271533411.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. 2024b.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint 2r0vv:2304. /2244, 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint wxXiv:230400/196 2023b.
Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, and Mingyuan Zhou. Self-augmented prefer- ence optimization: Off-policy paradigms for language model alignment. ArXiv, abs/2405.20830, 2024. URL https:////. / api. www.ste
arggCoorpuIID.... 270199610.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, ArXiv, abs/2401.100200 2024. URL https:///// //api.
semanticscholar org/ wow.pssI:::7703002122
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), 2018
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv: /00::000500000 2019.
16

---

Published as a conference paper at ICLR 2025
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Lim self- training via process reward guided tree search. ArXiv, abs/2406.03816, 2024. URL https:
saap...etattiicccoolaa ortp/oorpssI::
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pp- 27099-27116. PMLR, 2022.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
17

---

Published as a conference paper at ICLR 2025
LIMITATIONS
Our approach begins with small SFT dataset and does not necessitate specifically labeled data for training response improver. We assume that model-generated outputs closely resemble the gold standard responses in the SFT, enabling them to serve as training data for the response improver. This necessitates filtering out pairs where the gold standard is inferior the model-generated output. Such data can cause the response improver to rewrite text through paraphrasing or substantial alter- ation, as the training data comprises pseudo pairs rather than minimally edited original responses. While prompting a more powerful LLM to generate cewriting-specitng data, as suggested by Lu et al. (2023), can alleviate this, it sacrifices the benefit of learning the distribution gap.
SynPO leverages small, high-quality dataset repeatedly to guide synthetic data generation, making the seed data quality vital. This approach requires only small amount of high-quality data for validation, significantly reducing annotation costs. Additionally, recent work on direct on-policy sampling methods (Wu et al., 2024b), which do not need additional SFT data, shows considerable promise. After our final round of improvements, the model-generated responses already of high quality. Future enhancements can incorporate on on-policy preference optimization techniques to further refine the model.
APPENDIX
A ALGORITHM
We provide the overall pipeline of SynPO in Algorithm 1.
Algorithm Synthetic Preference Optimi zat in (SynPO)
1: Input: Initial policy no, validation set [(x],,)))? keyword list set K,, prompt generator G, data filter F. synthetic preference data D
2: for 1,2... do
3: Generate m synthetic prompts {x:] with x, ~ G(-lk)), where k, represents list
keywords randomly sampled from K.
Train the response improver R2 from 0g. R2 arg ming Li A
where (t-1)). ~ -e._,(-(x;; for í (I.....n..
Generate No... completions
self-refined completions on synthetic prompts: Yj,-1), ~ mB. ( (-Ix)). Y(t-1), ~ Mttnns)
(y) for E (I....,m.
Filter invalid refinements and integrate valid data into the synthetic preference dataset:
D e DU (xx..
F(x; F1-1), valid, e (......m).. Yoo,, (-1x))
7: Optimize me, using the SimPO (Meng et al., 2024) objective, where and are hyperparam-
eters:
8
B
0: + arg min E(x, No
log
log ( xx) Yoolll
log To(y x.) -2)
8: end for
B PROMPT FOR SESPONSE-REFINES
The prompt template used for training and inference in the response improver is shown in Figure 7.
Self- Self-Improver Prompt
You smart Al assistant, For given question-answer pair, improve the answer by correcting errors, bolstering informativeness, aligning with the question, and providing comprehensive detail.
Given Question:
(self-generated question] Original Answer: (original_ moal___mdel____
completion) _add_mmm______lpttinnn
Rewritten Answer:
Figure 7: Prompt in SynPO for the LLM to act as a response-refinee
18

---

Published as a conference paper at ICLR 2025
C
EXPERIMENTAL DETAILS
Here list additional experimental details for our implementation and experiments.
C.I SELF-PROMPT GENERATOR TRAINING
The hyperparameters for self--rree celf-prompt generator training are detailed below. During SFT for the self- prompt generator, we employ learning rate of 1.0 x 10 10-6 for Mistral-Baa -Bse and Llama3-Basc, with batch size of 32, a warm-up ratio of 0.1, and an AdamW optimizer. We set the maximum sequence length to 8,000 and train the model for 3 epochs.
To generate diverse synthetic prompts, we randomly sampled million paragraphs from Refined- Web (Penedo et al., 2023) and randomly selected 3 keywords from each paragraph. This process yields large keyword list pool containing million keyword lists. These keyword lists serve as the input for the self-promp generator in each iteration. For each iteration, we generate between 36,000 and 72,000 keyword lists (depending on the filtering ratio at each itcration) and exclude lists con- taining personal names stopwords. We use vllm for inference and set the sampling temperature to 0.7.
C.2 RESPONSE IMPROVER TRAINING
As the model iterates and self-improves, may produce responses superior to those of the seed data. Our objective for the response improver learn from its deficiencies. Therefore, we iden- tify instances where the model output is inferior to the original response using the same scoring model as the filtering stage. This ensures that the response improver only learns positive optimiza- tions or semantic paraphrasing, rather than negative optimizations. Specifically, for a given $. if the score difference between the gold standard completion and the model completion exceeds the threshold, we include this data for response improver training. In the Mistral-Base setting, we set the PairRM scoring threshold to 0.20. In the Llama3-Base setting, ArmoRM--Llama3-BB-o scoring threshold is set to 0.02.
Since the response improver data are automatically derived from SFT data conversion, the model also learns paraphrasing. Using more powerful model, such as GPT-4, to create data that introduce only minor improvements for rewriter training is a promising research direction. However, to explore the potential for self- self-boosting, we did not introduce additional data stronger models for data construction, resulting in inevitable paraphrasing by the response improver.
During SFT for the response improver, most training parameters prompt generator training. Some miner differences lie in: we set the max sequence length to 6,000.
the parameters in self-
C.3 RESPONSE IMPROVING AND FILTERING SETTING
To produce synthetic preference (chosen and rejected) completions for the t-th iteration. we utilize the current policy model generate a completion and employ the response improver refine it. We use vllm for inference, with the decoding temperature set at T 0.7.
During synthetic data filtering, we set threshold 0.20 for PairRM scores and threshold 0.02 for MaaeeMM--lama3BB...
scores. In addition, we filter out all the data that contain over 50% repetition patterns to avoid model collapse on synthetic data. In our experiments, we randomly incorporated 10,000 preference pairs from each iteration to the whole synthetic preference data.
C.4 OPTIMIZATION
As the parameter 3 is crucial for achieving optimal performance in SimPO (Meng al., 2024), individually search the in the range of [2, 4, 6, 8, 10, 12] for each optimization process. We use a fixed 7 1.6 for the Mistral-Base model and Llama3-Base.
19

---

Published as a conference paper at ICLR 2025
C.5 BASELINES
In experiments involving iterative baselines, we control various conditions to ensure faireess. We maintain the same training data size for both iterative baselines and SynPO. We adopt the SimPO loss (Meng et al., 2024) for preference optimization, as is more effective than DPO (Rafailov al., 2024). We all use self-generated prompts, which have been shown to superior prompts generated by other methods, as validated in Section 2. and Section 4.1. All preference construction processes are iterated until performance no longer improves.
Regarding the baseline models trained on UltraFeedback 61k, we straightforwardly adopt well-trained versions available from SimPO repository at ttttps://githie princeton-nie nlp/ SimmO.
C.6 DECODING HYPERPARAMETERS
For the AlpacaEival 2 (Dubois et al., 2024) evaluation, we use sampling-based decoding approach generate responses. Specifically, we employ vllm for inference, setting the temperature to 0.7 and the maximum tokens to 2048 for both the Mistral-Ban Base and Llama3- configurations. All other parameters adhere to the default settings vilm. As for MT-Bench (Zheng et al., 2024), we adhere to the official decoding setup, which specifies varying sampling temperatures tailored to distinct categories.
D ADDITIONAL DETAILS ON SEED DATA ABLATION
In setting 2), 3), and 4), to prevent cases where rejected responses are better than the chosen ones, filter the preference data using the same method as SynPO, specifically employing ArmoRM- Llama3-8B-vO.. to select valid preference data. We fix the threshold at 0.02, as our search among (0, 011, 0.2) reveal that 0.02 consistently performs the best.
E API USAGE For
GPT-4 Turbo, we all use the latest turbo-2024-04-09 API on Azure OpenAI Ser- vice https://leat
www..aaw.....oo........e..e.e... wwweeennssauueeaaeeaeeeeeeeeeeee conceptssmoddelssgptt-oo
F PROMPT ANALYSIS
Here we provide the prompt used for prompt topic and intention analysis in Figure 8, along with more detailed distribution bar plot for different intentions and topics in Figure 9, The topic word list derived from UltraChat (Ding et al., 2023), while the intention word list was designed by us.
G EVALUATION DETAILS
For instructionnffllowviii ability evaluation, Table presents the detailed information for three align- ment benchmarks use, including AlpacaEval 2.0, Arena-Hard and MTT-Bnch. Additionally, display the radar chart for MT-Bench scores on different prompt types (see Figure 10).
As for general LLM capability evaluation, we provide the few-shot example numbers Open LLM Leaderboard in Table and a comprehensive comparison of SynPO.
# Instances
Baseline Model Judge Model
Scoring Type
AlpacaEval 2.0
805
GPT-4 Turbo
GPT-4 Turbo
Pairwise comparison
Arena-Hard
500
GPT-4-0314
GPT-4 Turbo
Pairwise comparison
MT-Bench
80
GPT-4 Turbo
Single-answer grading
Table Details for three alignment benchmarks.
20

---

Published as a conference paper at ICLR 2025
Topic and Intention Classification Prompt
You are smart AI assistant. Below the *## Given text' section, user's instruction/que provided. Please determine which topic this question belongs to and output ONE most suitable topic from the "topies" list. Also, output ONE most suitable intention tey your best to use the words or phrases in the given lists, but if none of them fits, you can output new one.)
be
user from the 'intentions' list. [Note:
### Given Text (Riven_texxt)
### # Toppc List topics ("Techuology", "Health and welluess", "Travel and adventure", "Food and drink", "Art and culture", "Science and innovation", "Fashion and style", "Relationshipe and dating", "Sports and fitnes", "Nature and the euvironment". "Music and entertainment" events", "Education and learning". "Money and finance". "Work and career", "Philosophy and ethics". "History and nostalgia", "Social media and communication", "Creativity and Inspiration" "Personal growth and dovelopment", "Spirituality and faith", "Pop culture and trends", "Beauty and self-care", "Family and parenting", "Eatrepreneurshhip and business", "Literature and writing". "Gaming and technology", "Mindfulness and meditation", "Diversity and inclusion", "Travel and culture exchange")
"Politics and current
### Intention List intentions = ['Seek advice". "Design help". "Plan something". "Discuss top- ies", "Analyze something", "Evaluate something". "Search help", "Learn something", "Writing/ "Writin/pollshing pollishin help". "Quality chat", "Create something", "Fix something", "Compare something". " "Transfer something", "Calculate something", "Navigate", "Explore something new". "Play a game", "Install
Iastall/uninstall uninstall help", /cancel help". "Buy/sell suggestions", "Register/enrdd /enroli help", "Translation help". "Proofread- ing/ lng/editing help". "Mental health advice", "Recommendations" "Troubleshoot help", "Project feedback", "Creative brainstorming", "Time management help", "Organization help", "Public speaking help". "Job application help". "Networking help", "Language learning help". "Technology setup help", "Event coordination help", "Social media management help", "Couflict resolution help", "Sustainable living advice"]
### # Output Format Format chosen topic and intention only as a python dictionary with no extrancous output e.g. "topic": f..", "inteution"; ".... Each value only contain ONE topic intention.
Figure 8: Prompt for using GPT-4 Turbo as intention and topic classifier.
Topics
Intentions
100
100
Synthetic
Ultra
Synthetic
Ultra
Technology Sc sccecee
Nature
Polities
Gaming
Music
History
Creativity tityiiiy
Sports
Pashion
Family Diversit
Beauty
Leam Analyza Evalu Evaluate uate
Seek
Desion Compare re
Traubleshon
Plan
Sigrainable Book/c cancel
Project Buy Public
Fix
Register Installi eereenee
Time
Figure 9: Topic and intention distribution bar plot for the synthetic prompts in SynPO and Ultra- Feedback.
H ANALYSIS ON KEYWORDS FOR PROMPT GENERATION
In this section, we systematically investigate the impact of various factors on the quality and diversity of prompts through preliminary experiments on Llama3. We explore the effects of excluding noise keywords (Table 10), the types of keywords used (Table 11), the order of keywords (Table 12), and the number of keywords (Table 13). Our experiments involve training and evaluating the prompt generator with different configurations, measuring average similarity using Sentence-Trande Transformer and assessing prompt quality with GPT-4 Turbo as an
on a scale from 1 to 10.
Results indicate that the inclusion of noise keywords improves prompt quality as LLMs learn to ig- nore unrelated words, while using noun phrases enhances diversity and quality. Keyword order has a minimal impact, with randomization slightly enhancing diversity. Additionally, using 3-5 keywords provides the best balance between diversity and quality, with too many keywords degrading prompt quality. These findings collectively guide the implementation of the self- self-prompt generator.
I DATA LEAKAGE ANALYSIS
To ensure the robustness of our evaluation, we conduct a thorough analysis to detect any potential data leakage between our training datasets and the test sets. Specifically, we compare the n-grams
21

---

Published as a conference paper at ICLR 2025
Task
Arc
HellaSwag
TruthfulQA
MMLU Winogrande
GSM8k
Few-shot Examples
25
10
Metrics
acc_norm
acc_norm
mc2
acc
Table 9: Number of few-shot examples in Open LLM Leaderboard evaluation.
Wrrtnn
Wrrann
model
model
Humontiis
loma3--55000 SFT
Humaniics
Mistral
Lame3--B-ite
Mistral-e 7B iter2
Liam33-BB BB-hera
Mistaa-Te 7B IterS
Llana3-BB-tard
Misoal
STEM
Reasoning
STEM
Reasoning
Exraztion
Math
Extaction
Math
Coding
Coding
Figure 10: MT-Bench scores different prompt types. The left radar chart represents results from SynPO on Llama3 and the right comes from Mistral.
of our training datasets (seed SFT data, synthetic preference data, and UltraFeedback for reference) with the n-grams of the test set data to identify any overlaps. If any n-gram from test data entry appears our dataset n-grams, that entry is marked as leaked, and we calculate the proportion of leaked data for each test dataset. For datasets with candidate answers, we concatenate the question with candidate answers for analysis; for those without candidate answers, we use only the question. Following Liang et al. (2023), set the n-gram size to 13.
As shown in Table and Table 15, the overlap between training datasets (UltraFeedback, seed data, and synthetic preference data) and the test sets is very low, indicating that there is data Ieakage problem. Interestingly, synthetic preference data generally shows even less overlap with the test sets than other data. These findings enhance the robustness of our research and confirm the integrity of our evaluation process.
J EXTENDED EVALUATION ON DIVERSE TASKS
For comprehensive evaluation across broader range of tasks, include Evaluation Harness (see Table 16). including biomedical domain and complex reasoning. We also report the domain-specific results on the AGIEval benchmark (Zhong al., 2023) (in Table 17), which is a comprehensive benchmark specifically designed to assess foundation models in the con- text of
tasks from LM
standardized exams across diverse domains.
The results indicate that while SynPO is not specifically designed for complex reasoning, it performs competitively across variety of tasks, particularly in specialized domains. For instance, SynPO iters shows significant improvements over the SFT model in tasks such as EQ-Bench (Paech, 2023). In the AGIEval benchmark, SynPO iters also outperforms the SFT model each specific domains.
K PERFORMANCE GAINS ON TRUTHFULQA
Similar to SimPO Meng et al. (2024), SynPO shows significant enhancement on TruthfulQA Lin et al. (2021). There are two primary reasons for the observed performance gains on the TruthfulQA benchmark. First, there is strong correlation between preference alignment and TruthfulQA. Truth- fulQA evaluates the truthfulness of responses from language models, a goal that closely aligns with preference alignment. Our synthetic preference dataset, which includes instances emphasizing truth-
22

---

Published as a conference paper at ICLR 2025
Noise Condition
Avg. Similarity
Quality
No noise
0.0572
7.92
With noise
0.0574
8.99
Table 10: Effect of noise keywords.
Extraction Form
Avg. Similarity
Quality
Noun Phrases
0.0574
8.99
Verb Phrases
0.0865
8.74
Noun Verb Phrases
0.0610
8.67
All Phrases
0.0569
8.67
Noun Words
0.0604
8.98
Verb Words
0,0894
8.76
Noun Verb Words
0.0725
8.52
All Words
0.0598
8.61
Table 11: Effect of keyword types.
fulness, enhances the model's ability to accurately interpret context and generate truthful responses. This aligns with findings from Meng et al. (2024). Similarly, we hypothesize that the preference dataset contains instances that emphasize truthfulness, which helps the model better understand the context and generate more truthful responses." " Second, we utilize SimPO loss for preference opti- mization, which has significantly boosted TruthfulQA performance compared to other methods, as detailed Table the SimPO paper (Meng et al., 2024).
23

---

Published as a conference paper at ICLR 2025
Order
Avg. Similarity
Quality
Keep original order Random
0.0605
9.01
0.0574
8.99
Table 12: Effect of keyword order.
of keywords Avg. Similarity
Quality
0.0621
8.71
0,0574
8.99
0.0543
8.63
10
0.0541
7.08
Table 13: Effect of number of keywords.
Data
Are
HellaSwag
TQA
MMLU
Winogrande
GSM8k
UltraFeedback (for reference)
0.00085
0.00030
0.00122
0.00199
0.00000
0.00531
Seed SFT Data Synthetic Preference Data
0.00085
0.00010
0.00122
0.00036
0.00079
0.00076
0.00000
0.00000
0.00122
0.00064
0,00000
0.00000
Proportion of leaked data for various test datasets when compared with UltraFeedback, Table 14:
Seed SFT Data, and Synthetic Preference Data.
Data
AlpacaEval 2.0
Areana-Hard MT-Bench
UltraFeddbcck
0.00248
0.00600
0.01250
Seed SFT Data
0.00373
0.01200
0.01250
Synthetic Preference Data
0.00124
0.00800
0.00000
Table 15:
Proportion of leaked data for additional test datasets when compared with UltraFeedback, Seed SFT Data, and Synthetic Preference Data.
LogiQA (Liu et al., 2020)
MMLU-Pro (Wang et al., 2024b)
SIQA
QA4MRE (Peñlas al., 2013)
NQ (Kwiatkowski al., 2019)
Model
(Sap et al., 2019)
SFT
30.88 31.95
36.12 37.28
46.21 49.18
46.13 49.30
11.55 12.80
SynPO 4 iters
PubMedQA
RACE (Lai et al,, 2017)
SWAG
EQ-Bench (Paech, 2023)
Story Cloze (Mostafazaleh al., 2016)
Model
2019)
al., 2018)
SFT
73.20 75.44
44.59 46.70
76.91 77.19
46.79 46.79
10.55 12.80
SynPO iters
55,82
Table 16: Evaluation results on 10 additional LM Evaluation Harness tasks, using Llama3.
Model
History
Biology
Chemistry Physics MathQA
Llama3-SFT
38.29
34.29 39.05
24.64
32.00
24.22 28.49
SynPO 4 iters
45.53
31.88
35.00
Table 17: Evaluation results AGIEval for different domains, using Llama3.
24


Reviews:
[{"review_id": "QNODZoBq4m", "reviewer": "Reviewer_TAJC", "summary": "The paper presents \"SynPO\", which trains a self-prompt generator by controlling the keywords used. Noises (extracted also from responses) are inserted to produce the \"bad\" responses so that a SFT setup is possible where good and bad responses are now available. A separate step involves a response regenerator that is used to refine the model responses to get the good responses. \nThe diversity comes from the selection of keywords, which can be sampled from a large pretraining set like RefinedWeb, and the model improves by learning to discern bad and good responses iteratively.", "questions": "Do you have any insights on what kind of noise keywords are bad? What exactly does it make one keyword more noisy than another?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strength of the paper is that it offers a sound solution in terms of introducing diversity in the self-improvement process. While this is not exactly novel, but I do see a world where this works and improves model progressively. Moreover, use of keyword means that it provides a transparency in what we can control, the granularity of control, and ease of analysis.", "weaknesses": "The weaknesses are the following:\n1) I'm not sure if this is completely \"self-boosting\" as the title claimed, since external knowledge is needed, and careful selection of the keyword might be needed to guide the LLM to learn properly. \n2) I think the paper does not provide much insights into the types of noise (keywords) that are more effective, which seems rather important as an insights in creating the bad samples. \n3) The method relies heavily on the model's ability to generate out from a prompt containing keyword, it might or might not work with less capable LLMs. \n4) The writing is rather unclear, I think it could have been written in a much more understandable manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B28AYpeUEM", "reviewer": "Reviewer_EKkD", "summary": "The paper illustrates a synthetic data generation pipeline, SynPO, for self-generating preference data to align the model. The pipeline consists of (a) self-generating diverse prompts at scale and (b) self-generating paired responses for these prompts. The paired responses are constructed by (i) first sampling generations from the base model, (ii) training a response improver model to synthetically generate the preferred response, and (iii) pairing the preferred response with the base model response to create paired preference data. The method relies only on initial high-quality seed SFT data, and all the remaining data is synthetic. Applying the SynPO method for multiple iterations results in significant alignment performance gains.", "questions": "Data filtering:\n- For preference data, why is the rejected response across iterations the one generated by the base model? Why are the model(s) from the previous iteration(s) not used for generating rejected responses? \n- Given that the \"chosen response\" quality keeps improving in the preference data since the rejected response is from the base model, do the number of filtered-out preference pair data, as stated on L230, come down across iterations?\n\nResults:\n- I don't understand the performance gains on TruthfulQA. Any reason for/pattern to these gains?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Each component of the SynPO pipeline is intuitive and contributes to the final model performance\n- Strong and extensive empirical results, including data analysis (Figure 4, 5) and extensive ablation experiments. \n- Creative uses of the seed SFT data for multiple components of the pipeline.\n- Very well written and structured.", "weaknesses": "- Has the synthetic preference data been tested for benchmark data leakage? I didn't see anything suggesting that checks for data leakage were done.\n- While there's novelty to the pipeline as a whole, the individual components of the pipeline have been known. \n\nThere are a few typos as well:\n- L193: \"an infinite\" -> \"a huge\" (clearly what is being described is not \"infinite\") \n- L247: \\sigma -> \\beta. Also, state what is \\sigma? I assume it's the sigmoid function. \n- L314: \"involve\" -> \"compare against\"\n- L316: \"involves\" -> \"use\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h7eLNALk5S", "reviewer": "Reviewer_LmCF", "summary": "This paper proposes a framework called SynPO that leverages synthetic preference data for human alignment. The framework starts with the training of a self-prompt generator to create large-scale synthetic prompts. This prompt-generator is trained from the LLM itself, starting from some initial SFT data, and can generate prompts based on keywords. In an iterative process, these prompts are then given to the model to be trained, and passed through a response improver, which is retrained each round to reduce the gap between policy model outputs and gold standard responses. The authors perform experiments with Mistral-Base 7B and Llama3-8B, starting from versions that have undergone SFT supervision on a chat-dataset. They show improvements on three alignment benchmarks: MT-Bench, Arena Hard and AlpacaEval 2.0. They also show an overall improvement on standard benchmarks including Arc, HellaSwag, TQA and MMLU.", "questions": "The fact that some of these scores go up, especially on knowledge benchmarks, is a bit suspicious because knowledge is something that can evidently not be learned from just self-improvement loops. It makes me wonder to what extent the improvement is just coming from the GPT-4 data. Your experiments in 4.2 seem to refute that idea (though you don't look at the benchmarks there I believe), leaving me a bit puzzled. Do you have any other explanation for why the model would improve on something it shouldn't improve on?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper does quite extensive experimentation, investigate the diversity of the prompts and do ablations to understand the impact of various parts of their pipeline. The topic will be well-received I believe given the recent attention for alignment and the cost of generating human preference data.", "weaknesses": "In the introduction, it is not entirely clear to me what model is what. E.g. it is mentioned that a self-prompt generator is trained to create synthetic prompts, and after that it is mentioned that model-generated responses are used as rejected candidates and a response improver is used to improve the response. It is also mentioned that a small amount of SFT data is used to steer the generation of synthetic preference data. But what is what? Which model is used with the synthetic preference data? What is the \"response\"? Is that coming from the self-prompt generator? Are the self-prompt generator and the response improver the same model? Some of this is cleared up later, but the reader does not know that while trying to comprehend that initial paragraph.\n\nSmall point: I don't think it is entirely accurate to write that you use Mistral-base and Llama 3 base model, because those models did not undergo SFT. The models stem from these models (as does the Llama 3 8B instruct model), but they are not those base models.\n\nLastly, I am a bit confused about the improvement on the standard benchmarks. Many of these benchmarks are things learned in pretraining (e.g. knowledge). The fact that some of these scores go up, especially on knowledge benchmarks, is a bit suspicious because knowledge is something that can evidently not be learned from just self-improvement loops. It makes me wonder if the improvement is just due to the input coming from GPT-4. I would ask if you considered a basilen where you do not update the policy model in between, but just do multiple rounds trying to squeeze out more of the GPT generated data without doing iterations with the model itself, or just do more SFT with the initial data, but I think that those are the ablations presented in 4.2 and the difference seems quite large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hyv6NpkuHO", "reviewer": "Reviewer_Ftfv", "summary": "The paper introduces SynPO, a self-boosting framework that improves LLMs using synthetic preference data rather than human annotations. The key innovation is a two-part process: (1) A self-prompt generator creates diverse prompts using just three keywords, and (2) A response improver that refines model outputs. The refined outputs are used as \"chosen\" responses and the original outputs as \"rejected\" responses for preference learning. Testing on Llama3-8B and Mistral-7B shows significant improvements in instruction following (~22-30% win rate improvements on benchmarks) and general performance (3-5% increase on Open LLM leaderboard) after four iterations.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novel and Practical Solution:\nAddresses a critical challenge in LLM development - the scarcity of high-quality preference data\nProvides a scalable way to generate training data without relying on human annotations\nSimple but effective keyword-based prompt generation strategy", "weaknesses": "Limited Task Scope:\nPrimary evaluation focuses on instruction-following tasks with limited testing on complex reasoning tasks or specialized domains\nSome tasks show performance degradation (e.g., Mistral's performance on GSM8K)\n\n\nUnexplained Performance Gaps:\nThe dramatic improvement over SFT baseline (e.g., from 6.6% to 34.0% win rate) needs more analysis\nLimited ablation studies on why the synthetic data works so much better", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vVcxpCHKW", "reviewer": "Reviewer_d9yw", "summary": "The authors introduce SynPO (Self-Boosting Preference Optimization), a framework that enables LLMs to autonomously generate synthetic data and enhance their performance through iterative self-boosting.  SynPO leverages a self-prompt generator and response improver to collaboratively produce high-quality synthetic preference data for training. This process allows the model to learn responses aligned with preferences, leading to performance improvements on benchmarks like the Open LLM Leaderboard, AlpacaEval 2.0, and Arena-Hard.", "questions": "1. References are needed for studies on a word-to-text task [1] and similar augmentation research where word sets are extracted from sentences in existing datasets, combined to create new synthetic datasets, and enhanced in quality using contrastive learning [2].\n\n    [1] *Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020, November). CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1823-1840).*\n\n    [2] *Seo, J., Moon, H., Lee, J., Eo, S., Park, C., & Lim, H. S. (2023, December). CHEF in the Language Kitchen: A Generative Data Augmentation Leveraging Korean Morpheme Ingredients. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6014-6029).*", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors contribute by demonstrating infinite generation potential through experiments that examine performance changes relative to the number of synthetic datasets. They also clarify the limitations by distinguishing this capacity from the sustained utility of such datasets over time.\n\n2. The contribution to diversity within the keyword list is effectively demonstrated, with supporting experiments that validate this aspect.\n\n3. Previous research has rarely considered multi-turn interactions; however, the authors establish the effectiveness of their proposed method under multi-turn dialogue conditions.\n\n4. The self-boosting approach employed in the optimization process is appropriate and well-suited for enhancing model performance.", "weaknesses": "1. An ablation study on the effect of excluding noise keywords is necessary for Lines 156–157.\n\n2. Analysis of the types of keywords is insufficient. The authors justify random sampling based on diversity, yet the study lacks analysis on which extraction forms might be more beneficial, or order affects outcomes.\n\n3. It would be interesting to see experiments that vary the number of keywords or increase them significantly.\n\n4. The most concerning issue is the weak validation of out-of-distribution data. Since the seed dataset reflects OpenAI GPT’s preferences, as do Alpaca 2.0 and Arena-Hard, this likely influences the performance shown in Table 2. Additionally, while Table 4 shows considerable improvement on TQA included in the UltraFeedback dataset, effects are minimal on other datasets, suggesting that SynPO does not fully resolve in-distribution bias issues. There are also doubts about whether this approach could be more efficient or effective than methods in other studies applying CoT rather than simple SFT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)