Self-Boosting Large Language Models with Synthetic Preference Data

## Abstract

Abstract Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses.
However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs.
We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment.
SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively.
This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences.
After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard.
Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

## 1 Introduction

Large Language Models (LLMs) have made remarkable progress in following user instructions and generating honest, harmless, and helpful responses (Achiam et al., ; Dubey et al., ). This advancement is primarily achieved in the model alignment stage, which involves training reward models or LLMs directly on datasets curated from human preferences (Ouyang et al., ; Bai et al., )
, typically employing Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ) or Direct Preference Optimization (DPO) (Rafailov et al., ).

Recent research has made significant strides in model alignment by
collecting high-quality preference data (Hu et al., )
, sampling and ranking on-policy responses (Meng et al., ; Wu et al., ), or introducing LLM-as-a-Judge as substitutes for human preferences (Yuan et al., ; Cui et al., ).
However, most work still relies on static, pre-collected preference datasets from human or stronger LLM annotation.
As LLMs improve rapidly, collecting large, high-quality preference data for effective learning becomes increasingly challenging and costly, whether from humans or stronger models (Shi et al., ).
According to Yin et al. (), directly sampling preference pairs, which closely resembles an on-policy setting, can result in performance declines due to inherent volatility and inefficiency.
Therefore, constructing effective preference data to continuously improve LLMs remains a critical research problem.

In this work, we present a self-boosting paradigm for LLM alignment, SynPO.
This paradigm leverages a small set of supervised fine-tuning (SFT) data to steer the generation of synthetic preference data, thereby enabling LLMs to iteratively extend their capabilities through optimizing on synthetic data.
To support iterative preference learning across diverse scenarios, SynPO first trains a self-prompt generator to create large-scale synthetic prompts.
Unlike previous approaches that require more powerful LLMs and instruction examples (Wang et al., ), our generator utilizes only the LLM itself and three random keywords as input.
To generate preference pairs for the synthetic prompts,
SynPO utilizes model-generated responses as rejected candidates and employs a response improver to refine these responses into chosen ones.
The response improver comes from two straightforward intuitions: (1) LLMs excel at identifying distribution gaps between texts (Zhong et al., ; Singh et al., ), and (2) refining a response is generally easier than generating a high-quality response from scratch (Madaan et al., ; Lu et al., ; Nguyen et al., ).
In each iteration, we train the initial model to be a response improver, focusing on identifying distribution gaps between current model outputs and gold standard responses in seed data. We then use the response improver to refine the model outputs, thereby providing generative rewards to the responses.
This approach allows the LLM to make subtle improvements and gradually push its boundaries.
By leveraging small high-quality data and the current model state to guide the generation of synthetic data, we introduce stronger supervision in an iterative manner.

Experimental results demonstrate that SynPO not only benefits LLM alignment with human preferences, but also improves generalist capabilities across various tasks.
Trained solely on synthetic data, SynPO significantly improves the instruction-following abilities of Llama3-8B and Mistral-7B (as shown in Figure 1 and Table 1), achieving over a 26% length-controlled win rate improvement on AlpacaEval 2.0 (Dubois et al., ) and a 22% to 30% improvement on Arena-hard (Li et al., ) (as shown in Table 2). Furthermore, self-boosted models achieve 3.2% to 5.0% higher average performance than SFT models on the Open LLM leaderboard (Beeching et al., ), indicating SynPO also enhances general LLM performance.

To summarize, our contribution includes:

- •
We introduce SynPO, a self-boosting mechanism that iteratively induces LLMs to synthesize high-quality data for training.
Without requiring human-annotated preference data, SynPO significantly enhances the diversity and quality of synthetic prompts and responses.
- •
SynPO dynamically guides LLMs to improve their own outputs, using pre- and post-refinement generations as synthetic preference pairs for training. This approach effectively integrates generative rewards for preference learning, enabling LLMs to gradually push their boundaries.
- •
SynPO significantly enhances both the instruction-following capabilities and the general performance of LLMs, showing substantial improvements over three to four iterations.

Figure: Figure 1: Length-controlled win rate on AlpacaEval 2.0 improves with SynPO iterations, approaching GPT-4 level for the base versions of Llama3-8B and Mistral-7B.
Refer to caption: x1.png

## 2 Self-Boosting LLM with Synthetic Preference Data

Figure: Figure 2: Overview of SynPO in the $\mathbf{t^{th}}$ iteration. Starting with the previous iteration model $M_{t-1}$, SynPO first learns a response improver $R_{t}$ to identify discrepancies between model responses ($y^{*}_{t-1}$) and gold standard responses ($y^{*}$) on seed data, and learns to refine model responses. Subsequently, on the self-generated prompts $x$ (elaborated in Section 2.1), SynPO employs $R_{t}$ to refine the $M_{t-1}$ responses ($y_{t-1}$) into improved responses ($y_{t-1}^{w}$). The valid synthetic prompts $x$, refined responses ($y_{t-1}^{w}$), and initial model $M_{0}$ responses ($y_{0}$) to form synthetic preference data. These data are incorporated into the synthetic preference dataset for preference optimization, resulting in an updated $M_{t}$ for the next iteration. The iterative process continually enhances LLM capabilities in instruction-following and task performance.
Refer to caption: x2.png

SynPO is a self-boosting scheme designed to iteratively generate high-quality preference data. An overview of SynPO is presented in Figure 2.
It begins with a small set of SFT data as seed data, denoted as $\left\{(\mathbf{x}^{*}_{i},\mathbf{y}^{*}_{i})\right\}_{i=0}^{n}$, and the initial policy model $\pi_{\theta_{0}}$.
By incorporating both the self-prompt generator and the response improver, SynPO provides sufficient prompts for iterative training and leverages the generative rewards in the synthetic preference data.
This approach allows the policy model to make subtle improvements and gradually expand its boundaries.

### 2.1 Synthetic Prompt Creation

Diverse and ample prompts are crucial for effective preference learning (Shi et al., ; Yuan et al., ; Song et al., ). Diversity facilitates generalization and a sufficient number of prompts allows data selection from a large candidate pool.
In SynPO, we propose a novel strategy for synthetic prompt generation.
We design a keywords-to-text task to guide the training of a self-prompt generator and create pseudo-label data from the seed SFT data.

Figure: Figure 3: Prompt used in SynPO for LLMs to act as self-prompt generators.

#### Self-Prompt Generator Training

We train the LLM itself to serve as a high-quality prompt generator.
For each prompt $\mathbf{x}^{*}_{i}$ in seed data, we randomly extract two keywords from $\mathbf{x}^{*}_{i}$ and one noise keyword from $\mathbf{x}^{*}_{j}$, where $j\in\{1,2,\ldots,n\}\setminus\{i\}$.
The inclusion of the noise keyword enhances the robustness of the prompt generator. It learns to filter out irrelevant keywords during training and ensure that the generated prompts are fluent.
This process yields a keyword list, $k_{i}$ for $\mathbf{x}^{*}_{i}$.
Next, we insert $k_{i}$ into a prompt template (see Figure 3) to create a prompt and use $(\mathbf{x}^{*}_{i},\mathbf{y}^{*}_{i})$ as the corresponding completion. This process constructs training data for the prompt generator.
We then optimize $\theta_{0}$ through SFT to transform the model into a prompt generator $\mathcal{G}$.
$\mathcal{G}$ possesses the capability to generate unlimited, diverse, and high-quality user instructions, controlled by the given keywords.

#### Keywords Sampling and Prompt Generating

To enhance the overall diversity of prompts, we sample keywords from pretrain corpus paragraphs.
We select three keywords from a same paragraph to maintain the inherent distribution between keywords, and sample from different paragraphs to ensure overall diversity.
Specifically, we randomly sample keyword lists from the RefinedWeb paragraphs (Penedo et al., ) and generate $m$ synthetic prompts, denoted $\{\mathbf{x}_{i}\}_{i=1}^{m}$, $\mathbf{x}_{i}\sim\mathcal{G}(\cdot|\mathbf{k}_{i})$.

Benefiting from the self-prompt generator, our approach requires only a small SFT data to enable the model to generate diverse prompts independently, no in-context learning examples or predefined topic lists are required.
Due to the high combinatorial possibilities of keyword sampling, the self-prompt generator can produce an infinite variety of synthetic prompts for preference learning.

For further analysis, we randomly sampled 1k self-generated prompts from the Llama3-8B-Base prompt generator and used GPT-4 Turbo to classify the intentions and topics behind these prompts(^1^11Experimental details in Appendix F.).
The results, shown in Figure 5, demonstrate significant diversity across various topics and user intentions.
Even compared to prompts from GPT3.5-Turbo (Ding et al., ) or a collection of prompts from different sources (Cui et al., ), as shown in Figure 5, SynPO generated prompts exhibit lower inter-prompt similarity and greater diversity.

Figure: Figure 4: The top 25 most common topics (outer circle) and the top 12 most common intentions (inner circle) in SynPO generated prompts. We aggregate the other topics and intentions to the ‘Others’ group.
Refer to caption: x3.png

### 2.2 Synthetic Preference Generation

A primary challenge in leveraging synthetic prompts is the lack of high-quality responses to provide sufficient supervision (Li et al., ).
To address this, we introduce a response improver to enhance the quality of model responses to synthetic prompts.
Pre- and post-improvement responses naturally become the rejected and chosen candidates, respectively, with the chosen ones providing clear guidance on what approximates a gold standard response.

#### Response Improver Training

In each iteration, we train the LLM as a response improver to further reduce the gap between policy model outputs and gold standard responses.
Formally, let $\pi_{\bm{\theta}_{t-1}}$ denote the policy model at the beginning of the $t$-th iteration. We generate outputs from $\pi_{\bm{\theta}_{t-1}}$ for the seed data prompts: $\mathbf{y}^{*}_{(t-1),i}\sim\pi_{\bm{\theta}_{t-1}}(\cdot|\mathbf{x}^{*}_{i})$, $i\in\{1,\dots,m\}$. These outputs, along with the seed data responses, form the training set for the response improver, following the template provided in Appendix B. Each training example consists of the prompt and the policy model output $(\mathbf{x}^{*}_{i},\mathbf{y}^{*}_{(t-1),i})$ as the input, and the gold standard response $\mathbf{y}^{*}_{i}$ as the output.
We fine-tune $\pi_{\bm{\theta}_{0}}$ on the training set to obtain the response improver $\mathcal{R}_{t}$. This response improver refines the policy model outputs, aligning them more closely with the gold standard responses.

#### Response Improving

Subsequently, we use $\mathcal{R}_{t}$ to refine model responses to synthetic prompts, obtaining pre- and post-improvement responses as synthetic preference pairs. For each synthetic prompt $\mathbf{x}_{i}$, we first obtain the current model output $\mathbf{y}_{(t-1),i}\sim\pi_{\bm{\theta}_{t-1}}(\cdot|\mathbf{x}_{i})$, for $i\in\{1,\dots,m\}$. The response improver then refines this completion to produce $\overline{\mathbf{y}_{(t-1),i}}\sim\mathcal{R}_{t}(\cdot|\mathbf{x}_{i},
\mathbf{y}_{(t-1),i})$, considered the chosen response.
As we fine-tune the initial model in each iteration, the initial policy model output $\mathbf{y}_{(0),i}$ serves as the on-policy rejected response for $\mathbf{x}_{i}$. Here, $\mathbf{y}_{(0),i}\sim\pi_{\bm{\theta}_{0}}(\cdot|\mathbf{x}_{i})$.
This method generates numerous synthetic preference candidates, including both chosen and rejected responses.

#### Data Filtering

Unlike data from humans or strong teacher LLMs, which come with clear standard responses, self-generated data require proper filtering to ensure quality (Gulcehre et al., ).
As our policy model improves, many responses no longer need refining; we only need to retain data with a preference gap between the chosen and rejected responses.
Instead of using GPT4-Turbo-as-a-Judge for data filtering (Rosset et al., ), SynPO employs only a small model (e.g., a 0.4B PairRM (Jiang et al., )) or the model itself for scoring. Similar to SimPO (Meng et al., ) and SPPO (Wu et al., ), this ensures the process does not rely on a more powerful teacher model.
We retain $\overline{\mathbf{y}_{(t-1),i}}$ and $\mathbf{y}_{(0),i}$ with significant preference differences (i.e., a large score gap). $\overline{\mathbf{y}_{(t-1),i}}$ is regarded as the chosen response, $\mathbf{y}_{i}^{w}$, while $\mathbf{y}_{(0),i}$ is regarded as the rejected response, $\mathbf{y}_{i}^{l}$. Along with the corresponding prompt $\mathbf{x}_{i}$, they form a valid instance $(\mathbf{x}_{i},\mathbf{y}_{i}^{w},\mathbf{y}_{i}^{l})$.
All valid data is then integrated into the synthetic preference data for subsequent iterations.

### 2.3 Synthetic Preference Optimization

The large-scale synthetic preference data naturally facilitate the multi-iteration process of self-boosting.
In each iteration, we follow SimPO (Meng et al., ) for training; actually, our method is also compatible with other preference optimization training methods, such as DPO (Rafailov et al., ) and KTO (Ethayarajh et al., ). Denoting $\mathcal{D}$ as the synthetic preference data, we have:

$$ $\displaystyle\bm{\theta}_{t}\leftarrow\operatorname*{arg\,min}_{\bm{\theta}} \mathbb{E}_{(\mathbf{x}_{i},\mathbf{y}_{i}^{w},\mathbf{y}_{i}^{l})\sim\mathcal {D}}\left[\log\sigma\left(\frac{\beta}{|\mathbf{y}_{i}^{w}|}\log\pi_{\theta_{t -1}}(\mathbf{y}_{i}^{w}\mid\mathbf{x}_{i})-\frac{\beta}{|\mathbf{y}_{i}^{l}|} \log\pi_{\theta_{t-1}}(\mathbf{y}_{i}^{l}\mid\mathbf{x}_{i})-\gamma\right)\right]$ $$

$\sigma$ and $\gamma$ are hyperparameters.
Different from the vanilla SimPO, SynPO is a iterative process and all the preference data are synthetic ones.
The response improver continuously refines the generation distribution to align with the ideal data distribution across multiple iterations.

Overall, the response improver automatically learns to generate implicit generative rewards for the outputs of the LLM.
Unlike using a discriminative reward model straightforwardly, this approach helps the model learning to improve its outputs.
We present the Synthetic Preference Optimization algorithm in Appendix A.
The entire optimization process is performed on synthetic preference data, requiring only a small amount of high-quality data for validation. This strategy maintains two key advantages: (1) Compared to the limited and hard-to-collect preference data, SynPO generates an unlimited amount of new self-synthetic data to meet the needs of iterative model improvement. (2) Using small, high-quality validation data prevents the model from deviating during training and consistently guides the generation of more relevant synthetic data.

**Table 2: Results on AlpacaEval 2.0 and Arena-Hard. LC and WR denote length-controlled and raw win rates, respectively. After four SynPO iterations, Mistral-Base and Llama3-Base increase LC by 27.4% and 26.7%, respectively, on AlpacaEval 2.0. In Arena-Hard, SynPO achieves the highest WR by the third iteration, improving both models by over 22.1%.**
| Data Construction | Mistral-Base (7B) | Llama3-Base (8B) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| AlpacaEval 2.0 | Arena-Hard | AlpacaEval 2.0 | Arena-Hard |  |  |  |
| LC (%) | WR (%) | WR (%) | LC (%) | WR (%) | WR (%) |  |
| SFT | 6.6 | 3.6 | 2.0 | 5.4 | 3.1 | 2.7 |
| Manual Collection | 21.5 | 20.8 | 16.8 | 22.0 | 19.8 | 23.2 |
| Sampling-Ranking   Iter1 | 6.5 | 4.4 | 4.1 | 7.2 | 4.3 | 4.4 |
| Sampling-Ranking   Iter2 | 9.3 | 6.4 | 4.3 | 7.7 | 4.7 | 6.2 |
| Sampling-Ranking   Iter3 | 10.6 | 7.5 | 7.9 | 13.8 | 8.2 | 8.4 |
| Sampling-Ranking   Iter4 | 11.6 | 8.0 | 9.6 | 14.2 | 8.4 | 10.4 |
| Self-Rewarding   Iter1 | 19.5 | 19.8 | 11.9 | 20.1 | 20.3 | 20.8 |
| Self-Rewarding   Iter2 | 22.4 | 23.5 | 19.2 | 21.7 | 22.4 | 20.5 |
| Self-Rewarding   Iter3 | 24.6 | 26.3 | 20.8 | 22.4 | 24.1 | 23.8 |
| Self-Rewarding   Iter4 | 26.1 | 28.0 | 21.1 | 24.8 | 25.6 | 25.0 |
| SynPO   Iter1 | 13.3 | 15.3 | 9.8 | 10.6 | 10.7 | 11.6 |
| SynPO   Iter2 | 25.7 | 28.1 | 20.8 | 23.4 | 24.1 | 24.6 |
| SynPO   Iter3 | 31.7 | 33.8 | 24.1 | 28.6 | 31.5 | 32.5 |
| SynPO   Iter4 | 34.0 | 36.4 | 22.8 | 32.1 | 33.6 | 31.4 |

## 3 Experiments

We carry out comprehensive experiments to demonstrate the effectiveness of SynPO in enhancing model alignment and improving general model performance.

### 3.1 Experimental Setup

#### Models and Training

We perform synthetic preference optimization on both Mistral-Base 7B and Llama3-8B Base.
Following Meng et al. (), we employ supervised fine-tuned models as the initial models. Specifically, the Mistral-Base 7B model (mistralai/Mistral-7B-v0.1) and the Llama3-8B Base model (meta-llama/Meta-Llama-3-8B-Base) were fine-tuned on the UltraChat-200k dataset as part of the Zephyr (Tunstall et al., ) training pipeline.(^2^22[https://huggingface.co/alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full) and [https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT))
Subsequently, we utilize 18k seed data to SFT the self-prompt generator and then generate 50k synthetic prompts per iteration. For data filtering, we employ the 0.4B PairRM (Jiang et al., ) as a small pairwise scoring model for the Mistral-Base 7B. For Llama3-Base 8B, we use Llama3 itself (ArmoRM-Llama3-8B-v0.1(^3^33[https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1))) as a scoring model for data filtering, given its superior alignment with human scoring (Meng et al., ).
In each iteration $t=1,\ldots,T$, we use model $\pi_{\theta_{t-1}}$ from the previous iteration to generate synthetic preference data and then preference optimize the initial models again.
More details on training parameters, filtering thresholds, and implementation environments are provided in Appendix C.

#### Seed Data Construction

We randomly sample UltraFeedback (Cui et al., ) prompts and their GPT-4 Turbo completions as our seed data.
The seed data is multipurposely transformed for the training of self-prompt generator, response improver, and the validation of synthetic preference optimization.
The complete UltraFeedback dataset contains 61k instructions from sources including TruthfulQA (Lin et al., ), FalseQA (Hu et al., ), Evol-Instruct (Xu et al., ), UltraChat (Ding et al., ), and ShareGPT (Chiang et al., ).
To construct seed SFT data with high-quality responses, we randomly sampled 18k prompts and obtained the corresponding completions generated by GPT-4 Turbo.

#### Baselines

As baselines, we use the initial supervised fine-tuned models and those optimized with data from various preference construction methods, including manual collection and iterative approaches.
We recognize the UltraFeedback preference data (Cui et al., ) as a product of manual collection. It is gathered from six high-quality datasets and various models, with preferences annotated by GPT-4 (Achiam et al., ).
For iterative construction approaches, we involve Sampling-Ranking and Self-Rewarding. For Sampling-Ranking, similar to Meng et al. () and Wu et al. (), we involves LLMs in sampling five responses per prompt in each iteration. The same scoring models, i.e., PairRM and ArmoRM-Llama3-8B-v0.1, are then used to select the highest and lowest scoring responses as the chosen and rejected responses, respectively.
For the Self-Rewarding method (Yuan et al., ), we generate preference data based on model’s own rewards via LLM-as-a-Judge prompting. We employ our 18k seed data as the initial instruction-following data. Given that Self-Rewarding requires additional LLM-as-a-Judge training data, we generate 16k seed data with GPT-4 Turbo.
For all settings, we adopt SimPO (Meng et al., ) for preference optimization.
More detailed are elaborated in Appendix C.5.

**Table 3: Multi-turn evaluation on MT-Bench. An asterisk (*) denotes the best score across multiple iterations. For Sampling-Ranking, Llama’s best is from iteration 4 and Mistral’s from iteration 3. For Self-Rewarding, both are from iteration 3. SynPO progressively enhances the multi-turn instruction-following capabilities of LLMs.**
| Data Construction | Mistral-Base | Llama3-Base |  |  |
| --- | --- | --- | --- | --- |
| Turn 1 | Turn 2 | Turn 1 | Turn 2 |  |
| SFT | 6.04 | 5.65 | 6.55 | 5.36 |
| Manual Collection | 6.73 | 6.82 | 7.29 | 7.00 |
| Sampling-Ranking   Iters* | 6.83 | 6.18 | 7.06 | 6.99 |
| Self-Rewarding   Iters* | 6.71 | 6.63 | 7.30 | 7.28 |
| SynPO  Iter1 | 6.53 | 6.41 | 6.99 | 6.65 |
| SynPO  Iter2 | 6.66 | 6.65 | 7.34 | 7.30 |
| SynPO  Iter3 | 6.86 | 6.82 | 7.34 | 7.34 |
| SynPO  Iter4 | 6.73 | 6.69 | 7.43 | 7.04 |

### 3.2 Preference Alignment

We evaluate the model alignment performance on three benchmarks: AlpacaEval 2.0 (Dubois et al., ), Arena-Hard (Li et al., ), and MT-Bench (Zheng et al., ).
AlpacaEval 2.0 includes 805 user prompts and utilizes pair-wise comparison with LLM-as-a-Judge. Specifically, the win rate against the baseline GPT-4 Turbo model is determined based on GPT-4 Turbo evaluation.
Arena-Hard includes 500 more challenging user queries, employing GPT-4-Turbo to judge the model responses against GPT-4.
MT-Bench features 80 multi-turn questions spanning various domains, with GPT-4 scoring the model responses out of 10.(^4^44More details on API usage for LLM judgement are listed in Appendix E and G.)

#### Single-Turn Dialogues

We compare the instruction-following and human preference alignment capabilities on AlpacaEval 2.0 (Dubois et al., ) and Arena-Hard (Li et al., ) in Table 2. Compared to the initial model post-SFT, SynPO shows sustained improvement over four iterations in win rate against GPT-4 Turbo or GPT-4. On AlpacaEval 2.0, Mistral-Base achieves a 27.4% increase in length-controlled win rate and a 32.8% increase in raw win rate after four iterations. Similarly, Llama3 exhibits a 26.7% rise in length-controlled win rate and a 30.5% improvement in raw win rate after the same number of iterations. In the more challenging Arena-Hard setting, SynPO reaches the highest win rate after the third iteration.
Compared to the baseline methods, SynPO’s iterative preference learning on synthetic data yielded more significant improvements.

#### Multi-Turn Dialogues

For the multi-turn benchmark MT-Bench, we report both the first-turn and second-turn scores (in Table 3) as well as a radar chart depicting performance across different question types (refer to Figure 3).(^5^55Similar results for Mistral-7B are shown in Figure 10 in Appendix G.) The results indicate that SynPO enhances not only first-turn performance, with an increase of over 0.7 points, but also subsequent turns, with an increase of over 1.2 points. Compared to the initial model, SynPO shows improved performance across various question types, particularly in humanities, writing, STEM, and roleplaying.

**Table 4: Open LLM Leaderboard results. TQA stands for TruthfulQA. The asterisk (*) represents the best performance across multiple iterations. Compared to the SFT versions, SynPO achieves an overall improvement of 3.19% for Mistral and 5.00% for Llama3 on the average score.**
| Model | Arc | HellaSwag | TQA | MMLU | Winogrande | GSM8k | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral-Base-SFT | 58.02 | 80.89 | 40.37 | 58.78 | 76.40 | 34.72 | 58.20 |
| Manual Collection | 62.71 | 83.39 | 50.69 | 58.47 | 77.35 | 32.83 | 60.91 |
| Sampling-Ranking   Iters* | 60.32 | 81.80 | 44.43 | 59.09 | 76.95 | 36.85 | 59.91 |
| Self-Rewarding   Iters* | 60.15 | 81.84 | 43.25 | 58.98 | 76.48 | 34.72 | 59.24 |
| Mistral-Base-SynPO   Iter1 | 60.49 | 82.25 | 50.36 | 59.00 | 76.48 | 36.39 | 60.83 |
| Mistral-Base-SynPO   Iter2 | 63.65 | 83.24 | 58.04 | 58.74 | 76.48 | 27.35 | 61.25 |
| Mistral-Base-SynPO   Iter3 | 63.54 | 83.14 | 58.11 | 58.37 | 75.77 | 25.26 | 60.70 |
| Mistral-Base-SynPO   Iter4 | 63.57 | 83.04 | 56.12 | 58.75 | 75.77 | 31.08 | 61.39 |
| LLama3-Base-SFT | 60.92 | 81.28 | 45.37 | 63.80 | 76.72 | 51.93 | 63.34 |
| Manual Collection | 66.72 | 82.89 | 59.47 | 63.10 | 77.82 | 45.72 | 65.95 |
| Sampling-Ranking   Iters* | 66.38 | 82.71 | 59.84 | 63.37 | 77.27 | 54.40 | 67.33 |
| Self-Rewarding   Iters* | 64.76 | 82.48 | 55.54 | 63.42 | 77.03 | 54.59 | 66.30 |
| LLama3-Base-SynPO   Iter1 | 63.99 | 82.66 | 54.20 | 64.02 | 77.51 | 56.10 | 66.41 |
| LLama3-Base-SynPO   Iter2 | 65.70 | 83.22 | 61.73 | 64.03 | 76.56 | 56.25 | 67.92 |
| LLama3-Base-SynPO   Iter3 | 66.55 | 83.57 | 63.53 | 63.91 | 76.80 | 55.27 | 68.27 |
| LLama3-Base-SynPO   Iter4 | 66.47 | 83.44 | 63.69 | 63.79 | 76.90 | 55.72 | 68.34 |

### 3.3 Downstream Task Performance

In terms of the general model performance on various tasks, we report the average scores on the well-recognized Open LLM Leaderboard (Beeching et al., ) and 6 additional benchmarks from Language Model Evaluation Harness library (LLM Harness) (Gao et al., ).
Open LLM Leaderboard (Beeching et al., ) is recognized as a standard assessment for the general performance of LLMs. It includes six different datasets, evaluating LLMs on commonsense reasoning (Arc (Clark et al., ), HellaSwag (Zellers et al., ), Winogrande (Sakaguchi et al., )), wide knowledge (MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., )), and math (GSM8k (Cobbe et al., )).
The six addtional LLM Harness tasks include Openbook Question Answering (OBQA) (Mihaylov et al., ) and Haerae (Son et al., ) for model knowledge, MathQA (Amini et al., ), XNLI (Conneau et al., ), and PROST (Aroca-Ouellette et al., ) for reasoning, as well as Toxigen (Hartvigsen et al., ) for toxicity evaluation.(^6^66We follow the standard few-shot setting on Open LLM Leaderboard, as elaborated in Appendix G. For the six additional tasks, we employ a fixed 5-shot setting for evaluation.)

#### Open LLM Leaderboard

On the Open LLM leaderboard, we observe an overall improvement of 3.19% of Mistral-Base-SFT and 5.00% of Llama3-Base-SFT on the average score.
Specifically, SynPO achieves over 6% improvement on the ARC challenge and over 16% on TruthfulQA compared to the SFT model, after four rounds of self-boosting.
Notably, the performance of Mistral-Base on GSM8K has experienced a decline, whereas Llama3-Base has demonstrated an improvement of nearly 4 points on the same benchmark.
This disparity likely stems from the superior data filtering capability of ArmoRM-Llama3-8B-v0.1 compared to the 0.4B PairRM. ArmoRM-Llama3-8B-v0.1 effectively mitigates erroneous responses and enhances mathematical problem-solving performance.

#### LLM Harness Tasks

**Table 5: Downstream performance in each SynPO iteration on six tasks in LM Evaluation Harness. An asterisk (*) represents the best performance across multiple iterations.**
| Model | OBQA | Haerae | MathQA | XNLI | Toxigen | PROST | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral-Base-SFT | 46.40 | 39.96 | 36.25 | 43.76 | 60.11 | 52.04 | 46.42 |
| Manual Collection | 50.20 | 40.35 | 36.72 | 44.65 | 62.83 | 54.66 | 48.24 |
| Sampling-Ranking   Iters* | 47.60 | 40.15 | 36.21 | 44.71 | 62.45 | 53.74 | 47.48 |
| Self-Rewarding   Iters* | 48.60 | 40.15 | 36.25 | 44.17 | 61.28 | 53.29 | 47.29 |
| Mistral-Base-SynPO   Iter1 | 48.00 | 39.78 | 36.05 | 44.23 | 60.74 | 52.39 | 46.87 |
| Mistral-Base-SynPO   Iter2 | 50.40 | 40.05 | 36.85 | 43.97 | 60.96 | 53.04 | 47.55 |
| Mistral-Base-SynPO   Iter3 | 51.20 | 40.51 | 35.88 | 44.37 | 60.74 | 53.45 | 47.69 |
| Mistral-Base-SynPO   Iter4 | 51.40 | 40.88 | 36.48 | 44.47 | 63.40 | 55.05 | 48.61 |
| Llama3-Base-SFT | 46.20 | 61.78 | 42.04 | 45.47 | 68.83 | 52.40 | 52.79 |
| Manual Collection | 51.60 | 61.59 | 42.51 | 44.77 | 74.38 | 55.96 | 55.14 |
| Sampling-Ranking   Iters* | 50.80 | 62.05 | 42.81 | 46.67 | 74.68 | 55.41 | 55.40 |
| Self-Rewarding   Iters* | 48.60 | 62.42 | 42.78 | 46.31 | 74.15 | 54.23 | 54.75 |
| Llama3-Base-SynPO   Iter1 | 48.20 | 61.96 | 42.71 | 46.15 | 71.17 | 53.77 | 53.99 |
| Llama3-Base-SynPO   Iter2 | 50.80 | 62.60 | 42.75 | 46.34 | 74.04 | 54.85 | 55.23 |
| Llama3-Base-SynPO   Iter3 | 51.00 | 62.51 | 42.78 | 46.37 | 75.11 | 55.29 | 55.51 |
| Llama3-Base-SynPO   Iter4 | 52.00 | 62.51 | 42.65 | 46.27 | 75.18 | 56.14 | 55.79 |

The advantages above are also reflected in the results for more diverse tasks in LLM Harness, as evidenced by Table 5.
Previous works (Wu et al., ; Meng et al., ) demonstrate that preference optimization can induce the “alignment tax” - aligning models with human preferences can improve performance for only 1$\sim$2 iterations or even degrade overall performance on downstream tasks (Askell et al., ). Our method exhibits similar behavior on MathQA; however, overall, SynPO shows improvements across more iterations on other tasks. This is because synthesizing better chosen candidates introduces additional supervision, partially mitigating the alignment tax issue and enabling LLMs to continuously enhance their capabilities on downstream tasks at the same time of alignment.

## 4 Ablation Studies

### 4.1 Synthetic Prompts and Responses

We have demonstrated the diversity of prompts generated by SynPO in Section 2.1.
To further validate the self-prompt generator, we compare the generated prompts with manual collected prompts (Cui et al., ) and Self-Instruct prompts (Wang et al., ).
For each prompt construction approach, we randomly sample 20k prompts and construct response pairs through both SynPO and Sampling-Ranking.
We compare the single iteration results of Llama3-8B. As shown in Table 7,
whether through self-refinement or Sampling-Ranking, synthetic prompts generated by SynPO lead to better-aligned models, validating the quality of these prompts.
It is worth mentioning that SynPO prompts are even more effective than the superset of its seed training data, UltraFeedback prompts.
This increased effectiveness may be attributed to the greater diversity of SynPO prompts, achieved through the keyword sampling process in prompt synthesis.
Results of mixing SynPO and manual collected prompts further indicate the potential of SynPO in augmenting existing prompts.

### 4.2 Impact of Seed Data

SynPO involves training LLMs solely on synthetic preference data while using seed SFT data for validation.
To investigate the maximum impact of the seed SFT data, we compare SynPO with the following settings: 1) Seed SFT: Directly fine-tuning the LLM using seed data. 2) Seed PO: For each prompt in the seed SFT data, using the gold standard response in the seed data as the chosen response and the initial policy model response as the rejected response for preference optimization. 3) Seed SFT + PO: To avoid distribution shifts in directly using seed SFT data, we first obtain a model fine-tuned on seed data as in 1), then construct preference data using the model output and gold standard responses. 4) Seed SFT + PO${}^{\text{me}}$: Training on data from 3) for multiple epochs.(^7^77We compare the results of 2 epochs and 3 epochs and select the better-performing 2 epochs. More experimental details in Appendix D)
The results on AlpacaEval 2.0 are presented in Table 7.
Among the evaluated methods except for SynPO, setting 3) is most analogous to SynPO, and proves to be the most effective.
However, due to the limited quantity of seed data, the improvement is less than that achieved by iterative SynPO on synthetic data. Training under such conditions for multiple epochs does not yield further improvements and even degrades performance. These findings validate that SynPO is a promising approach to construct preference data and maximize the utilization of minimal high-quality data.

**Table 6: Comparison of various prompt generation methods on AlpacaEval 2.0. SynPO Mix. combines SynPO prompts with manually collected prompts.**
| Prompts | SynPO | Sampling-Ranking |  |  |
| --- | --- | --- | --- | --- |
| LC (%) | WR (%) | LC (%) | WR (%) |  |
| Manual Collection | 23.8 | 25.6 | 7.8 | 6.2 |
| Self-Instruct | 21.7 | 21.4 | 9.1 | 4.6 |
| SynPO | 24.3 | 24.5 | 9.2 | 5.3 |
| SynPO Mix. | 23.4 | 29.4 | 14.2 | 6.9 |

## 5 Related Work

#### Preference Data Construction

Preference data are triplets consisting of user prompts, user-preferred responses, and non-preferred responses.
Acquiring preference data from humans can be resource intensive, often constrained by the data collection platform (Ouyang et al., ) or the cost of human annotation (Bai et al., ; Ethayarajh et al., ; Nakano et al., ).
To alleviate this problem, researchers have started using teacher LLMs, such as GPT-4 (Achiam et al., ), to simulate human preferences (Cui et al., ; Ding et al., ; Huang et al., ). Given user prompts and candidate responses, this line of work employs a stronger model to annotate preferences, thereby overcoming the scarcity and constraints of existing preference data (Cui et al., ).
However, the prompts still need to be collected (Cui et al., ), which limits the domain, diversity, and quantity of the data. Both human annotation and the utilization of large teacher model APIs incur substantial costs (Shi et al., ).
Moreover, relying solely on reward scores or win-lose annotation fails to fully capture the subtleties and complexities of human preferences.

#### LLM Self-Boosting

Previous work has advanced the self-boosting of LLMs by searching for high-reward behaviors (Tian et al., ; Zhang et al., ), using LLMs as judges to select responses (Yuan et al., ; Wang et al., ; Wu et al., ; Kim et al., ), and leveraging self-play strategies (Chen et al., ; Cheng et al., ; Luo et al., ; Wu et al., ).
These works typically use a fixed set of existing prompts, limiting the LLM ability to learn across wide scenarios.
Furthermore, the deterministic reward signals in these methods do not help the model recognize subtle discrepancies between its responses and ideal responses.
Prior to our work, Constitutional AI (Bai et al., ) and SELF (Lu et al., ) used AI to generate non-deterministic feedback for training. Constitutional AI used refinement data for reward models, while SELF employed GPT-4 for data generation and taught models self-refinement. However, these methods did not utilize comparative information between pre- and post-revision texts for training.

#### Synthetic Data for LLMs

Acquiring human-generated data is costly and time-consuming, leading to the use of synthetic data for LLM training (Wang et al., ; Xu et al., ; Li et al., ). Unnatural Instructions (Honovich et al., ) and Self-Instruct (Wang et al., ) use seed instructions to generate new prompts, while WizardLM (Xu et al., ) and WizardMath (Luo et al., ) rewrite these instructions into more complex forms using ChatGPT. Seed topics also produce textbook-like data (Li et al., ; ) or self-chat dialogues (Xu et al., ; Ding et al., ) for instruction tuning. These methods often require strong LLMs or examples, benefiting from model distillation (Xu et al., ; Li et al., ; ).
Our approach uses the model itself to generate prompts and responses without needing carefully designed topics, extending beyond the model’s inherent sampling space. This brings generative rewards to the LLM self-boosting process, particularly benefiting initially weaker models. It resembles direct preference knowledge distillation (Li et al., ) but does not rely on large-scale teacher model responses for supervision.

## 6 Conclusion

We introduce self-boosting LLM with synthetic preference data, SynPO, a method for LLM alignment through iterative training on synthetic data.
In SynPO, we innovatively base the entire training process on synthetic data and only employ limited SFT data for validation.
SynPO diversifies the prompts and dynamically guides LLMs to improve their own output, using pre- and post-refinement generations as synthetic preference pairs for training in the next iteration.
Experimental results show that SynPO leads to significant improvements on both instruction-following capabilities and task performance. This strategy sheds light on high-quality synthetic data generation and self-alignment with minimal supervision, both of which are critical for the continuous development of LLMs.

## Limitations

Our approach begins with a small SFT dataset and does not necessitate specifically labeled data for training a response improver. We assume that model-generated outputs closely resemble the gold standard responses in the SFT, enabling them to serve as training data for the response improver. This necessitates filtering out pairs where the gold standard is inferior to the model-generated output. Such data can cause the response improver to rewrite text through paraphrasing or substantial alteration, as the training data comprises pseudo pairs rather than minimally edited original responses. While prompting a more powerful LLM to generate rewriting-specific data, as suggested by Lu et al. (), can alleviate this, it sacrifices the benefit of learning the distribution gap.

SynPO leverages a small, high-quality dataset repeatedly to guide synthetic data generation, making the seed data quality vital. This approach requires only a small amount of high-quality data for validation, significantly reducing annotation costs. Additionally, recent work on direct on-policy sampling methods (Wu et al., ), which do not need additional SFT data, shows considerable promise. After our final round of improvements, the model-generated responses are already of high quality. Future enhancements can incorporate on-policy preference optimization techniques to further refine the model.

## Appendix

## Appendix A Algorithm

We provide the overall pipeline of SynPO in Algorithm 1.

Figure: Algorithm 1 Synthetic Preference Optimization(SynPO)

## Appendix B Prompt for Response-refiner

The prompt template used for training and inference in the response improver is shown in Figure 7.

Figure: Figure 7: Prompt in SynPO for the LLM to act as a response-refiner.

## Appendix C Experimental Details

Here we list additional experimental details for our implementation and experiments.

### C.1 Self-Prompt Generator Training

The hyperparameters for self-prompt generator training are detailed below. During SFT for the self-prompt generator, we employ a learning rate of $1.0\times 10^{-6}$ for Mistral-Base and Llama3-Base, with a batch size of 32, a warm-up ratio of 0.1, and an AdamW optimizer. We set the maximum sequence length to 8,000 and train the model for 3 epochs.

To generate diverse synthetic prompts, we randomly sampled 1 million paragraphs from RefinedWeb (Penedo et al., ) and randomly selected 3 keywords from each paragraph. This process yields a large keyword list pool containing 1 million keyword lists. These keyword lists serve as the input for the self-prompt generator in each iteration. For each iteration, we generate between 36,000 and 72,000 keyword lists (depending on the filtering ratio at each iteration) and exclude lists containing personal names or stopwords. We use vllm for inference and set the sampling temperature to 0.7.

### C.2 Response Improver Training

As the model iterates and self-improves, it may produce responses superior to those of the seed data. Our objective is for the response improver to learn from its deficiencies. Therefore, we identify instances where the model output is inferior to the original response using the same scoring model as the filtering stage. This ensures that the response improver only learns positive optimizations or semantic paraphrasing, rather than negative optimizations.
Specifically, for a given $\mathcal{x}_{i}$, if the score difference between the gold standard completion and the model completion exceeds the threshold, we include this data for response improver training.
In the Mistral-Base setting, we set the PairRM scoring threshold to 0.20. In the Llama3-Base setting, the ArmoRM-Llama3-8B-v0.1 scoring threshold is set to 0.02.

Since the response improver data are automatically derived from SFT data conversion, the model also learns paraphrasing. Using a more powerful model, such as GPT-4, to create data that introduce only minor improvements for rewriter training is a promising research direction. However, to explore the potential for self-boosting, we did not introduce additional data or stronger models for data construction, resulting in inevitable paraphrasing by the response improver.

During SFT for the response improver, most training parameters are same to the parameters in self-prompt generator training. Some miner differences lie in: we set the max sequence length to 6,000.

### C.3 Response Improving and Filtering Setting

To produce synthetic preference (chosen and rejected) completions for the $t$-th iteration, we utilize the current policy model to generate a completion and employ the response improver to refine it. We use vllm for inference, with the decoding temperature set at $T=0.7$.

During synthetic data filtering, we set a threshold 0.20 for PairRM scores and a threshold 0.02 for ArmoRM-Llama3-8B-v0.1 scores.
In addition, we filter out all the data that contain over 50% repetition patterns to avoid model collapse on synthetic data.
In our experiments, we randomly incorporated 10,000 preference pairs from each iteration to the whole synthetic preference data.

### C.4 Optimization

As the parameter $\beta$ is crucial for achieving optimal performance in SimPO (Meng et al., ), we individually search the $\beta$ in the range of [2, 4, 6, 8, 10, 12] for each optimization process. We use a fixed $\gamma=1.6$ for the Mistral-Base model and Llama3-Base.

### C.5 Baselines

In experiments involving iterative baselines, we control various conditions to ensure fairness. We maintain the same training data size for both iterative baselines and SynPO. We adopt the SimPO loss (Meng et al., ) for preference optimization, as it is more effective than DPO (Rafailov et al., ).
We all use self-generated prompts, which have been shown to be superior to prompts generated by other methods, as validated in Section 2.1 and Section 4.1.
All preference construction processes are iterated until performance no longer improves.

Regarding the baseline models trained on UltraFeedback 61k, we straightforwardly adopt the well-trained versions available from the SimPO repository at [https://github.com/princeton-nlp/SimPO](https://github.com/princeton-nlp/SimPO).

### C.6 Decoding Hyperparameters

For the AlpacaEval 2 (Dubois et al., ) evaluation, we use a sampling-based decoding approach to generate responses. Specifically, we employ vllm for inference, setting the temperature to 0.7 and the maximum tokens to 2048 for both the Mistral-Base and Llama3-Base configurations. All other parameters adhere to the default settings in vllm.
As for MT-Bench (Zheng et al., ), we adhere to the official decoding setup, which specifies varying sampling temperatures tailored to distinct categories.

## Appendix D Additional Details on Seed Data Ablation

In setting 2), 3), and 4), to prevent cases where rejected responses are better than the chosen ones, we filter the preference data using the same method as SynPO, specifically employing ArmoRM-Llama3-8B-v0.1 to select valid preference data. We fix the threshold at 0.02, as our search among {0, 0.1, 0.2} reveal that 0.02 consistently performs the best.

## Appendix E API Usage

For GPT-4 Turbo, we all use the latest turbo-2024-04-09 API on Azure OpenAI Service [https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-turbo](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-4-turbo).

## Appendix F Prompt Analysis

Here we provide the prompt used for prompt topic and intention analysis in Figure 8, along with a more detailed distribution bar plot for different intentions and topics in Figure 9. The topic word list is derived from UltraChat (Ding et al., ), while the intention word list was designed by us.

Figure: Figure 8: Prompt for using GPT-4 Turbo as a intention and topic classifier.

Figure: Figure 9: Topic and intention distribution bar plot for the synthetic prompts in SynPO and UltraFeedback.
Refer to caption: x6.png

## Appendix G Evaluation Details

For instruction-following ability evaluation,
Table 8 presents the detailed information for three alignment benchmarks we use, including AlpacaEval 2.0, Arena-Hard and MT-Bench. Additionally, we display the radar chart for MT-Bench scores on different prompt types (see Figure 10).

As for general LLM capability evaluation, we provide the few-shot example numbers on Open LLM Leaderboard in
Table 9
and a comprehensive comparison of SynPO.

**Table 8: Details for three alignment benchmarks.**
|  | # Instances | Baseline Model | Judge Model | Scoring Type |
| --- | --- | --- | --- | --- |
| AlpacaEval 2.0 | 805 | GPT-4 Turbo | GPT-4 Turbo | Pairwise comparison |
| Arena-Hard | 500 | GPT-4-0314 | GPT-4 Turbo | Pairwise comparison |
| MT-Bench | 80 | - | GPT-4 Turbo | Single-answer grading |

**Table 9: Number of few-shot examples in Open LLM Leaderboard evaluation.**
| Task | Arc | HellaSwag | TruthfulQA | MMLU | Winogrande | GSM8k |
| --- | --- | --- | --- | --- | --- | --- |
| # Few-shot Examples | 25 | 10 | 0 | 5 | 5 | 5 |
| Metrics | acc_norm | acc_norm | mc2 | acc | acc | acc |

Figure: Figure 10: MT-Bench scores on different prompt types. The left radar chart represents results from SynPO on Llama3 and the right comes from Mistral.
Refer to caption: x7.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Do you have any insights on what kind of noise keywords are bad? What exactly does it make one keyword more noisy than another?
soundness: 2
strengths: The strength of the paper is that it offers a sound solution in terms of introducing diversity in the self-improvement process. While this is not exactly novel, but I do see a world where this works and improves model progressively. Moreover, use of keyword means that it provides a transparency in what we can control, the granularity of control, and ease of analysis.
summary: The paper presents "SynPO", which trains a self-prompt generator by controlling the keywords used. Noises (extracted also from responses) are inserted to produce the "bad" responses so that a SFT setup is possible where good and bad responses are now available. A separate step involves a response regenerator that is used to refine the model responses to get the good responses. 
The diversity comes from the selection of keywords, which can be sampled from a large pretraining set like RefinedWeb, and the model improves by learning to discern bad and good responses iteratively.
weaknesses: The weaknesses are the following:
1) I'm not sure if this is completely "self-boosting" as the title claimed, since external knowledge is needed, and careful selection of the keyword might be needed to guide the LLM to learn properly. 
2) I think the paper does not provide much insights into the types of noise (keywords) that are more effective, which seems rather important as an insights in creating the bad samples. 
3) The method relies heavily on the model's ability to generate out from a prompt containing keyword, it might or might not work with less capable LLMs. 
4) The writing is rather unclear, I think it could have been written in a much more understandable manner.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Data filtering:
- For preference data, why is the rejected response across iterations the one generated by the base model? Why are the model(s) from the previous iteration(s) not used for generating rejected responses? 
- Given that the "chosen response" quality keeps improving in the preference data since the rejected response is from the base model, do the number of filtered-out preference pair data, as stated on L230, come down across iterations?

Results:
- I don't understand the performance gains on TruthfulQA. Any reason for/pattern to these gains?
soundness: 3
strengths: - Each component of the SynPO pipeline is intuitive and contributes to the final model performance
- Strong and extensive empirical results, including data analysis (Figure 4, 5) and extensive ablation experiments. 
- Creative uses of the seed SFT data for multiple components of the pipeline.
- Very well written and structured.
summary: The paper illustrates a synthetic data generation pipeline, SynPO, for self-generating preference data to align the model. The pipeline consists of (a) self-generating diverse prompts at scale and (b) self-generating paired responses for these prompts. The paired responses are constructed by (i) first sampling generations from the base model, (ii) training a response improver model to synthetically generate the preferred response, and (iii) pairing the preferred response with the base model response to create paired preference data. The method relies only on initial high-quality seed SFT data, and all the remaining data is synthetic. Applying the SynPO method for multiple iterations results in significant alignment performance gains.
weaknesses: - Has the synthetic preference data been tested for benchmark data leakage? I didn't see anything suggesting that checks for data leakage were done.
- While there's novelty to the pipeline as a whole, the individual components of the pipeline have been known. 

There are a few typos as well:
- L193: "an infinite" -> "a huge" (clearly what is being described is not "infinite") 
- L247: \sigma -> \beta. Also, state what is \sigma? I assume it's the sigmoid function. 
- L314: "involve" -> "compare against"
- L316: "involves" -> "use"

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: The fact that some of these scores go up, especially on knowledge benchmarks, is a bit suspicious because knowledge is something that can evidently not be learned from just self-improvement loops. It makes me wonder to what extent the improvement is just coming from the GPT-4 data. Your experiments in 4.2 seem to refute that idea (though you don't look at the benchmarks there I believe), leaving me a bit puzzled. Do you have any other explanation for why the model would improve on something it shouldn't improve on?
soundness: 3
strengths: The paper does quite extensive experimentation, investigate the diversity of the prompts and do ablations to understand the impact of various parts of their pipeline. The topic will be well-received I believe given the recent attention for alignment and the cost of generating human preference data.
summary: This paper proposes a framework called SynPO that leverages synthetic preference data for human alignment. The framework starts with the training of a self-prompt generator to create large-scale synthetic prompts. This prompt-generator is trained from the LLM itself, starting from some initial SFT data, and can generate prompts based on keywords. In an iterative process, these prompts are then given to the model to be trained, and passed through a response improver, which is retrained each round to reduce the gap between policy model outputs and gold standard responses. The authors perform experiments with Mistral-Base 7B and Llama3-8B, starting from versions that have undergone SFT supervision on a chat-dataset. They show improvements on three alignment benchmarks: MT-Bench, Arena Hard and AlpacaEval 2.0. They also show an overall improvement on standard benchmarks including Arc, HellaSwag, TQA and MMLU.
weaknesses: In the introduction, it is not entirely clear to me what model is what. E.g. it is mentioned that a self-prompt generator is trained to create synthetic prompts, and after that it is mentioned that model-generated responses are used as rejected candidates and a response improver is used to improve the response. It is also mentioned that a small amount of SFT data is used to steer the generation of synthetic preference data. But what is what? Which model is used with the synthetic preference data? What is the "response"? Is that coming from the self-prompt generator? Are the self-prompt generator and the response improver the same model? Some of this is cleared up later, but the reader does not know that while trying to comprehend that initial paragraph.

Small point: I don't think it is entirely accurate to write that you use Mistral-base and Llama 3 base model, because those models did not undergo SFT. The models stem from these models (as does the Llama 3 8B instruct model), but they are not those base models.

Lastly, I am a bit confused about the improvement on the standard benchmarks. Many of these benchmarks are things learned in pretraining (e.g. knowledge). The fact that some of these scores go up, especially on knowledge benchmarks, is a bit suspicious because knowledge is something that can evidently not be learned from just self-improvement loops. It makes me wonder if the improvement is just due to the input coming from GPT-4. I would ask if you considered a basilen where you do not update the policy model in between, but just do multiple rounds trying to squeeze out more of the GPT generated data without doing iterations with the model itself, or just do more SFT with the initial data, but I think that those are the ablations presented in 4.2 and the difference seems quite large.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See weaknesses
soundness: 3
strengths: Novel and Practical Solution:
Addresses a critical challenge in LLM development - the scarcity of high-quality preference data
Provides a scalable way to generate training data without relying on human annotations
Simple but effective keyword-based prompt generation strategy
summary: The paper introduces SynPO, a self-boosting framework that improves LLMs using synthetic preference data rather than human annotations. The key innovation is a two-part process: (1) A self-prompt generator creates diverse prompts using just three keywords, and (2) A response improver that refines model outputs. The refined outputs are used as "chosen" responses and the original outputs as "rejected" responses for preference learning. Testing on Llama3-8B and Mistral-7B shows significant improvements in instruction following (~22-30% win rate improvements on benchmarks) and general performance (3-5% increase on Open LLM leaderboard) after four iterations.
weaknesses: Limited Task Scope:
Primary evaluation focuses on instruction-following tasks with limited testing on complex reasoning tasks or specialized domains
Some tasks show performance degradation (e.g., Mistral's performance on GSM8K)


Unexplained Performance Gaps:
The dramatic improvement over SFT baseline (e.g., from 6.6% to 34.0% win rate) needs more analysis
Limited ablation studies on why the synthetic data works so much better

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. References are needed for studies on a word-to-text task [1] and similar augmentation research where word sets are extracted from sentences in existing datasets, combined to create new synthetic datasets, and enhanced in quality using contrastive learning [2].

    [1] *Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020, November). CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1823-1840).*

    [2] *Seo, J., Moon, H., Lee, J., Eo, S., Park, C., & Lim, H. S. (2023, December). CHEF in the Language Kitchen: A Generative Data Augmentation Leveraging Korean Morpheme Ingredients. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6014-6029).*
soundness: 3
strengths: 1. The authors contribute by demonstrating infinite generation potential through experiments that examine performance changes relative to the number of synthetic datasets. They also clarify the limitations by distinguishing this capacity from the sustained utility of such datasets over time.

2. The contribution to diversity within the keyword list is effectively demonstrated, with supporting experiments that validate this aspect.

3. Previous research has rarely considered multi-turn interactions; however, the authors establish the effectiveness of their proposed method under multi-turn dialogue conditions.

4. The self-boosting approach employed in the optimization process is appropriate and well-suited for enhancing model performance.
summary: The authors introduce SynPO (Self-Boosting Preference Optimization), a framework that enables LLMs to autonomously generate synthetic data and enhance their performance through iterative self-boosting.  SynPO leverages a self-prompt generator and response improver to collaboratively produce high-quality synthetic preference data for training. This process allows the model to learn responses aligned with preferences, leading to performance improvements on benchmarks like the Open LLM Leaderboard, AlpacaEval 2.0, and Arena-Hard.
weaknesses: 1. An ablation study on the effect of excluding noise keywords is necessary for Lines 156–157.

2. Analysis of the types of keywords is insufficient. The authors justify random sampling based on diversity, yet the study lacks analysis on which extraction forms might be more beneficial, or order affects outcomes.

3. It would be interesting to see experiments that vary the number of keywords or increase them significantly.

4. The most concerning issue is the weak validation of out-of-distribution data. Since the seed dataset reflects OpenAI GPT’s preferences, as do Alpaca 2.0 and Arena-Hard, this likely influences the performance shown in Table 2. Additionally, while Table 4 shows considerable improvement on TQA included in the UltraFeedback dataset, effects are minimal on other datasets, suggesting that SynPO does not fully resolve in-distribution bias issues. There are also doubts about whether this approach could be more efficient or effective than methods in other studies applying CoT rather than simple SFT.

Meta-review: This paper introduces SynPO, a self-boosting framework that leverages synthetic preference data to iteratively improve the performance of large language models (LLMs). The approach trains a self-prompt generator and response improver to produce synthetic prompts and refine responses, eliminating the need for large-scale human annotation. The method is examined  on Llama3-8B and Mistral-7B, showing significant improvements in alignment tasks (e.g., AlpacaEval 2.0) and general benchmarks (e.g., MMLU, TruthfulQA).

Strengths: The paper addresses a critical challenge of lack of high-quality preference data by introducing a scalable and iterative synthetic data generation approach. The experiments are extensive, with strong empirical results supported by ablation studies and analyses of key components (e.g., noise keywords, extraction forms). The method demonstrates strong applicability to alignment tasks with clear presentations.

Weaknesses: The paper lacks detailed discussion on out-of-distribution robustness, and while the authors argue for the effectiveness of synthetic data, additional analyses on unexplained performance gains (e.g., on knowledge benchmarks) could improve the work. The introduction and framing of key components (e.g., model roles) were initially unclear, though this was improved during the rebuttal period. 

I recommend accepting this paper for its scalable method, solid empirical results and clear presentation.

Decision: Accept (Poster)