R.I.P.: Better Models by Survival of the Fittest Prompts
Authors: Ping Yu
Weizhe Yuan
Olga Golovneva
Tianhao Wu
Sainbayar Sukhbaatar
Jason E Weston
Jing Xu
Venue: ICML 2025 poster (2025)
Keywords: Data Filtering
Preference Optimization
Synthetic Data
Primary area: deep_learning->large_language_models
TL;DR: New data selection & synthetic data creation method Rejecting Instruction Preferences (RIP) dramatically improves model performance by filtering out 77% training examples.

Abstract: Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap between the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared to unfiltered data. Using Llama 3.1-8B-Instruct, RIP improves AlpacaEval2 LC Win Rate by 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 3.3-70B-Instruct, RIP improves Arena-Hard from 67.5 to 82.9, from 18th place to 6th overall in the leaderboard.

R.I.P.: Better Models by Survival of the Fittest Prompts
Ping Yu' Weizhe Yuan 12 Olga Golovneva'
Tianhao Wu Jing Xu'
Sainbayar Sukhbaatar'
Jason Weston 12
Abstract
Winrate Training Data Size
60.2
Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This is achieved by measuring the rejected response quality and the reward gap be- tween the chosen and rejected preference pair. Our method, Rejecting Instruction Preferences (RIP) can be used to filter prompts from existing training sets, or to make high quality synthetic datasets, yielding large performance gains across various benchmarks compared unfiltered data. Using Llama 3.1-8B Instruct RIP improves Al- pacaEval2 LC Win Rate by 9. 9.4%, Arena-Hard by 8.7%, and WildBench by 9.9%. Using Llama 13-77B-Innttrue RIP improves Arena-Hard from 67.5 to 82.9, which is from 18th place to 6th over- in the leaderboard.
60
57.83
58.49
52.60
GPT-4 Preview 1106
AlpacaEval LC Winrate
48.11
48.37
45
47.26.
45.64
44,93
40
* self-RIP (ours)
RIP (ours)
35
Raw Wildchat
* Jaccard Similarity Filtering
IFD Filtering
InsTag Difficulty Filtering
25
Response Filtering
SoM-as-rrompt-Quue Quallty Judge
LLaMA 3.1-8b-instruct
InsTag Diversity Filtering
20
512
2k
4k
16k
Training Data Size (in log)
Figure Our method Rejecting Instruction Preferences (RIP) for curating data, and Self-RIF for creating syn- thetic data. The x-axis represents the effective training set size (after filtering). At every data size training unfil- tered WildChat prompts is significantly outperformed by RIP. RIP also outperforms various other curation baselines. Synthetic data built by Self-RIP improves results further.
1. Introduction
In large language model (LLM) development, a primary driver for advancing frontier models is curating high-quality training examples. This curation crucial during both the pretraining (Rae al., 2021: Touvron et al., 2023a) and post-training (finetuning) phases (Touvron et al., 2023b). Despite the widespread adoption of the "scaling hypothesis" (Kaplan et al., 2020), merely increasing the size of training datasets does not guarantee improved performance if the data are of low quality (Chen et al., 2024; Li et al., 2024c; Zhou al., 2024). Without sufficient data quality, model training tends not to be fully robust to the associated noise, and final response quality from the model suffers.
model judgments given the training inputs. In this work, hypothesize that better judgments of data quality can be made by taking into account the model responses those data. Specifically, if the prompt is of low quality, then responses exhibit high variability and low quality as well. This insight leads us to develop a method for either selecting prompts, or for creating high quality synthetic prompts, both of which yield significant performance gains during post-training.
Currently, there are number of investigated techniques to curate data most of which are based on heuristics or
Our method, Rejecting Instruction Preferences (RIP), con- siders the case of instruction finetuning via preference opti- mization. It starts with a set of preference pairs consisting of input prompts and chosen and rejected responses. RIP considers specific characteristics of the preference pairs, in particular rejected response quality and the reward gap be-
Mett New York University 'UC Berkeley. Correspondence to: Jing Xu <jingxu23 @ meta.comm>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Rejecting Instruction Preferences (RIP)
Table I:
Rejecting Instruction Preferences (RIP) and Self-RIP compared to SOTA models on AlpacaEval2, Arena- Hard and WildBench. By training Llama 3.1-8B-Instruce and Llama 1.3-70B-Instrue on Wildchat instructions curated by RIP, or synthetic data created by Self-RIP, our method surpasses many existing SOTA models.
AlpacaEval2
Arena-Hard
WildBench
Standard models
LC Win
Win
Score
Score
GPT-4 Omni (05/13)
57.5
51.3
74.9
59.3
GPT-4 Turbo (04/09)
55.0
46.1
82.6
55.2
Llama 2.1-8B-Instrue
20.9
21.8
21.3
33.1
Llama 3.3-70B-Instruct
38,9
41.5
67.5
52.8
Llama 3.1-SB-Instruct + RIP (ours)
57.8
57.2
43.1
45.6
Llama 5.1-8B-Instruce + Self-RIP (ours)
60.2
61.1
42.1
42,5
Llama 3.3-70B Iitt--Instuct + RIP (ours)
67.7
73.2
82.9
58.8
tween the chosen and rejected preference pair. If the rejected quality is low or the reward gap high this is an indicator that the prompt is of low quality. We thus filter the prompts based on these metrics. The remaining prompts can subse- quently be used to fine-tune the model using RLHF methods like Direct Preference Optimization (DPO) (Rafailov et al., 2023), or for creating new synthetic prompts via few-shot prompting. Table illustrates that when trained on Wild- chat prompts (Zhao al., 2024b) and filtered by RIP, both Llama 3.1 UL-8B-Instrun and Llama 3.3- Instructt (Dubey et al., 2024) achieve large performance gains, surpassing many sfate-of-the models.
2. Related Work
Data Selection in Pretraining Data Given the high vari- ance in quality of pretraining data, data filtering a critical component for determining pretrained model quality (Hoff- mann al., 2022). In addition to heuristic preprocessing such deduplication of similar documents, removal of datasets with heavy test-set overlap, and text extractions from raw Internet content, GPT-3 (Brown et al., 2020) ap- plied text filtering to the CommonCraw/ dataset based on similarity to high-quality reference data, significantly re- ducing final pretraining text data from 45TB down a 570GB high-quality subset. As language models become more powerful, data curation can also be facilitated by us- ing LLMs as a quality judge. Llama2 and Llama3 employ model-based quality classifiers to filter out non- non-English and low-quality content from pretraining data (Touvron et al., 2023b; Dubey et al., 2024). Rae et al. (2021); Soldaini et al. (2024) also demonstrate that applying simple filter- ing on massive texts brings substantial improvements on downstream performance across the board.
Additionally, we conducted comprchensive experiments comparing the scaling behavior of our data under RIP fil- tering with that of unfiltered WildChat raw data, and six alternative filtering methods in Figure 1. Our results demon- strate that RIP significantly enhances model performance, while other filtering methods yield only marginal improve- ments. In addition to improvements observed with filtering human-written data such as Wildchat prompts or HelpSteer2 using different reward signals such as human, classifier or LLM-as-a-Juut aaddee, we also show RIP improves model perfor- mance as a method to create synthetic data.
Data Selection in Supervised Fine- Sine-Tuning Similarly, post-training also relies on high-quality data to enhance models' instruction-followii instruction-tuninn was regarded as largely dependent on the size of available instruction- tunnnn examples (Mishra et al., 2021; Wei al., 2021; Wang et al., 2022). More recent work has revealed that training on smaller yet higher-quality curated set of prompts tends to be more effective in improv- ing models' instruction-following capabilities (Zhou et al., 2024; Chen et al., 2024). To facilitate data selection, some employ traditional optimization saaaiittttio--bedddddd data data-pruning meth- ods by measuring their impact on model's generalization capabilities (Toneva et al., 2018; Yang al., 2022: Xia et al., 2024). Another stream of work studies employing powerful language models to measure the complexity, diversity and quality of instructions (Lu et al., 2023; Chen et al., 2024; Touvron et al., 2023b; Dubey et al., 2024; Li et al., 2024c). Alternative filtering approaches proposed automatic metrics such as IFD score (Li et al., 2023a), or INSTRUCTMINING
capabilities. Previously,
Analysis of our method using t-SNE shows that RIP can eliminate certain undesirable clusters. Additionally, analysis with GPT-4 reveals that RIP effectively removes noisy or low quality prompts, ambiguous prompts, unsafe prompts, and examples where preference choices are incorrect. We release our filtered datasets on HuggingFace'.
'For the Llama-3, 3. Ins-truttt filtered dataset, visit: https:/hugginnfaacccoodaaaa wow.caaattttt//ccccoooooo///a/a Www.eiihhhhtt----ieeeeee by- by-8b-Llama. For the Llama-3.3-70B, Instrrcct filtered dataset, visit:
htt.:ggginggg..eeeoddaaaeteeeeee Wiieee/oooooiiddcha----RRR -iP--iieee by-7Ob-Llama.
2

---

Rejecting Instruction Preferences (RIP)
which fits a linearly weighted score over a bag of natural language indicators (Cao et al., 2023) to select examples.
We also consider alternate pairing methods in Section A.4, We then use the preference data {a, yw, yt fees for training the model M. Note that our focus is on filtering prompts entirely, not responses to those prompts.
Data Selection in RLHF and Preference Optimiza- tion The success of preference-opiimizatio methods (Stienmon et al., 2020; Rafailov et al., 2024) has attracted more attention to collecting large scale and high quality preference data. While extensive work shows scaling up preference data through bootstrapping (Xu et al., 2023b; Yuan et al., 2024b), synthesis approaches (Lambert et al., 2024; Wang et al., 2024b), or crowdsourcing (Touvron et al., 2023b; Dubey et al., 2024), can boost model performance, the characterization and selection of high-quality pairwise examples is surprisingly underexplored. Most work involv- ing preference optimization employs existing methods de- rived from pretraining and instruction-tuning (Touvron et al., 2023b; Dubey et al., 2024), such deduplication, quality classifiers or filtering heuristics. However, such methods overlook the importance of the preference pairs (the chosen and rejected responses). Recent work Wu et al. (2024a); Khaki et al. (2024) shows that preference optimization can be highly sensitive to the choice of response pairs of differ- ent reward gaps, focusing more on pair construction than data selection.
3.2. Hypothesis on Data Selection
Although preferences are extensively used to train state- of-the-art LLMs, there is limited research on identifying unhelpful training examples in this setting. We posit that analyzing the paired model responses to given input prompts can provide valuable insights into the quality of the prompts. Specifically, we test the following two hypotheses.
Hypothesis 1: Low-quality prompts are likely to produce low-quality responses. Low-quality prompts for example those that are unclear, ambiguous, or containing conflict- ing information - are likely to lead to noisy or inaccurate model responses. While those inaccurate responses can still be used as training targets pairwise preference optimiza- tion, studies indicate that training on pairs with low-quality rejected responses might be sub-optimal. Yasunaga et al. (2024) for example shows that pairing the best with random responses works well comparing to pairing the best with the worst one with lowest reward. This suggests a potential correlation of the quality of the rejected example with the alignment outcome. Additionally, several studies (Wu et al., 2024b; Zhao et al., 2024a; Yuan et al., 2024a) have found a strong correlation between the length of responses, includ- ing rejected ones, and final performance. Therefore, consider the reward r(v(|z) and length len(u)) of rejected responses as indicators of quality of the training prompts x, i.e. large values of either of these metrics relative to other examples indicate higher quality.
3. Rejecting Instruction Preferences (RIP)
We start by defining the prompt selection problem in the pairwise preference optimization setting. In this context, we present our proposed prompt-response-piee paii--aaiiraaeee based filtering method, which develops key descriptive metrics and their use in filtering training prompts. Lastly, we describe how our method can be applied to self-instruction setups where synthetic prompts are generated from the model itself.
Hypothesis 2. Low-quality prompts are likely produce responses with larger variance Low quality prompts introduce uncertainty and ambiguity, leading to a broader range of interpretations. As the model or human generat- ing the response might guess or fill in gaps in the prompt, this results in higher variance in responses. While some responses might align well with the intent, others may devi- ate significantly. preliminary study in Wu al. (2024a) finds low-gap pairs, where chosen and rejected responses are similar, are high-quality informative pairs, leading to better performing DPO models. We therefore consider the reward gap r rV [z) r (æ) as another indicator of quality of a training prompt, i.e. small reward gaps suggest that the prompt has higher quality.
3.1. Data Curation Problem
The goal of data curation is to remove low-quality prompts that can negatively affect the general instruction following capability of the model. Given a set of prompts X = {a). we aim to find subset S S X to be used for fine-tuning a seed LLM M. We consider the preference optimization set- ting. with winning (chosen) and losing (rejected) response pairs {yw, y1 with rewards r3).: > r(y (z) for each prompt x. The response pairs and their rewards can come from human preference data, or can be generated from the model itself M and then scored using an external reward model. For the latter we use the "best-vs-worst" preference pairing method (Pace et al., 2024), where N responses are sampled, and the ones with highest and lowest rewards are the chosen and rejected, respectively:
3.3. RIP filtering
3.3.1. RIP FOR EXISTING TRAINING PROMPTS
Given the above hypotheses, we thus consider the following three metrics my
yw argmax,,, ((
(u ~ -Mxx) then
(x, mz( Bu 31) that are based on the responses:
y argmin,, (()))r))
3

---

Rejecting Instruction Preferences (RIP)
Rejected response reward: my (()lx)
focusing on two categories: human-written instructions and synthetically generated instructions. Finally, we extend our evaluation of RIP with the Llama 3.3-70B- model.
Rejected response length: m2 len(y))
Reward gap: My r(w |z) - r(ul
For each metric, we define threshold values that can be used for filtering. For the first two metrics, higher values are desired so we choose a lower-bound threshold
4.1. Human- -ariten Prompts
For human-written instructions, specifically investi- gate two setups: human-written input prompts 1) paired with model-generated responses and annotated by reward model; 2) with existing responses that have been annotated with numan-assignn rewards. We use the WildChat and Helpsteer2 datasets, see statistics in Appendix Table 13.
S {2 Th S m2 Mes 31)}-
The last reward gap metric requires an upper threshold as we want small gaps. Therefore we reduce the prompt selection problem to a threshold choice problem. To resolve this, we start with coordinate-wist experiments, analyzing model performance under various thresholds Th: for individual met- rics my (details in Section A.2). Ultimately, we perform hyperparameter selection using all 3 parameters.
4.1.1. WILDCHAT DATASET
Prompt Set We start with large pool of over 250k human-written prompts from the WildChat (Zhao et al., 2024b) dataset. We exclude any non-English prompts based on WildChat annotations, and remove around 70k Midjourney instructions", yielding 190k unique first- turn prompts. These prompts are collected from real user in- teractions without human annotations, making them highly diverse. While there are many high-quality prompts, there are also significant number of low-quality ones, such as nonsensical text or those lacking clear question.
3.3.2. SELF-RIP FOR SYNTHETIC PROMPTS
Prompt curation by RIP can also naturally be used gener- ate synthetic data. First, RIP is used to create a seed pool of high-quality prompts. Few-shot examples from this seed pool guide the model to generate training prompts, which can be further filtered by RIP. We thus propose Self-RIP, a new approach to creating high-quality synthetic prompts:
Response Generation Following Yuan et al. (2024b); Meng et al. (2024); Wu et al. (2024b) we generate our cho- sen and rejected response pairs on the WildChat prompts using our seed model M make our setup closer to the on-policy setting. We use best-vs- vs-worst as described in Sec- tion 3.1, generating N responses for each prompt using M with sampling parameters of T = 0.8, top p = 0.95.
Step 1. Few-shot prompting with RIP curated instruc- tions We start with the set of prompts S curated by our proposed method RIP as described in Section 3. To gen- erate new prompts S' we sample from our seed model M following Self-Inntruce (Wang et al., 2023; Honovich et al., 2023). For each new example we randomly select 8 prompts from S and feed them as few-shot examples to the model M generate a prompt with similar characteristics. We apply the exact processing steps in Wang et al. (2023) to new prompts 5", such as removing similar prompts (ROUGE-L similarity with any existing instructions < 0.7), and exclud- ing those that contain certain keywords (e.g.. image, picture, graph) that usually can not be processed by text-only LLMs.
Reward Annotation We then evaluate candidate responses using two different jifges:
Reward Classifier: We used the ArmoRM reward model (Wang et al., 2024a) score each response.
LLM-as-a-Judge (Zheng et al., 2023): We prompt LLama 3. B.1-4oSB-Instruct using the prompt template outlined in Yasunaga et al. (2024) to assign a score ranging from 0 to 10 for each response. For cach re- sponse, we conduct 10 independent evaluations and use the average score as the final reward.
Step 2. Filtering with RIP We further apply RIP on top of the synthetically generated prompts S" from the previous step, filtering out the self-instructions using the same thresh- old values as used before. Then the remaining subset S" is used for training the seed model M.
The training example (æ, Va 31) is selected by appointing the highest-reward one as the and the lowest-reward one as Vt- For our primary experiments, we use the default value of N = 64. However, results for N = 8, 16, 32 are provided as part of our ablation studies in Table 22, and we use N 32 for the Llama We perform early stopping using a validation set of 470 examples: 253 valid set examples from Li et al. (2024c) and
Note we use RIP filtering twice here, once in each step. This is to ensure the quality of synthetic prompts. We also explore Self-RIP using a smaller subset of as seed instruc- tions in Section A.4 as part of our ablation studies.
experiments.
4. Experimental Setup
We perform preference optimization using DPO, beginning with the Llama 3.1-8B-Instruc model as our seed model M. We evaluate both the selection and creation of prompts,
"hhey start with "As a prompt generator for a generative Al called "Midjourney", you will create image prompts..." ...".
4

---

Rejecting Instruction Preferences (RIP)
218 examples from the evol-test set of Xu et al. (2023a), with prompts that overlap with A AlpacaEval2 removed.
when it includes a greater variety of unique tags, as an- notated by the specified tagger. Using this approach, we greedily filter out data samples whose associated tags are already present in the selected dataset.
4.1.2. HELPSTEER2 DATASET
HelpSteer2 (Wang et al., 2024c) consists of around 10k human-written prompts each with a response pair sam- pled from 10 different LLMs. Each response has human- annotated rewards of helpfulness, correctness, coherence, complexity and verbosity on Likert-5 scale. We use the aggregated reward with the recommended weighting [0.65, 0.8, 0.45, 0.55, 0.4]. The main distinction from WildChat is that the rewards come from human annotations instead of an external model. We perform early stopping on the HelpStecr2 validation split, selecting checkpoints with the highest average response rewards determined by ArmoRM.
Prooppt-t-o Employing LLMS as prompt qual- ity judges has proven its efficacy in curating high-quality data (Chen et al., 2024; Dubey al., 2024; Liu et al., 2023). We employ Llama 3.1-405B-Instrus to measure the quality of prompts on both binary (useful/not useful) and point- wise scale (0-5). By sampling five Llama 3. 1-405B--Instreee predictions per prompt and taking the average of LLM-as- Prompt-Judge predictions, we filter out less useful prompts by varying the cutoff thresholds.
4.3.2. PROMPT-AND-CHOSE SEEION--EEPOSS---AASASS FILTERING
4.2. Synthetic Prompts
Perplexity We compute perplexity (ppl) of the chosen response yo with the Llama 3.B Instruct in a zero-shot manner as a filtering metric to curate training prompts. In particular, we retain examples with large ppl( (x: values, which may indicate the difficulty of the prompt.
In this setup, we generate prompts from the seed model M itself for training instead of using numan-written prompts. By varying the set of seed pool prompts used few-shot examples, we collect two sets of training prompts:
Self-Instruct: randomly select 8-shot examples from the unfiltered WildChat.
Instruction Followiing Difficulty (IFD) Li et al. (2023a) introduced the IFD to measure the model-specific difficulty of a data sample. A lower IFD score indicates that this par- ticular instruction-response pair is considered relatively easy for the language model to understand and follow without further training. We filter out examples with low IFD metric of a given pair of prompt x and chosen response yo
Self- R/P--R/P: randomly select 8-shot examples from high quality WildChat prompts filtered by RIP.
In each case, we create 20k training prompts sampled with decoding parameters T 0.8, top-p 0.95. The rest of the setup including response generations and DPO training is exactly the same as the WildChat setup where we use ArmoRM to construct response pairs (Mw, M11,, and do early stopping on the same validation set of 470 examples.
4.3.3.
REEENN---EEEEE--REEEEE FILTERING
BASED
Jaccard Similarity In addition to the reward gap between chosen and rejected responses, we explore Jaccard similar- ity, defined as the number of overlapping words divided by the overall word counts, as an alternative similarity mea- surement. We thus filter out examples with low Jaccard similarity scores (i.e. fewer overlapping words) between chosen and rejected response pairs.
4.3. Baselines
We compare our method with the existing methods below. For instruction-tuiing data selection methods which handle a single (non-pairwise) response per prompt, we apply them to the chosen responses within the response pairs. Addi- tional details on the implementation of each baseline are provided in Appendix Section A.5.
4.4. Training And Evaluation Setting
Following the Instruct setup in Meng et al. (2024), we utilize the DPO training approach with the off-the-shelff L.Lma 3.1- 8B Instructt and LLama 3.3-70B-Instruct models, leveraging the fairseq2 library (Balioglu, 2023). We use a batch size of 64 and sweep over learning rates of 7,-7,1e- 1e-6 for the I. LLLa 3. Instrruttt model, and a learning rate of 1e-6 with a batch size of 256 for the LLama 3.3-7oB- model. Both models are trained with a dropout rate of 0.0 and B value of 0.1 throughout the experiments. We conduct RIP with various cutoff thresholds, e.g. at the 25%, 50% and 75% percentile of each metric.
1. 1. FROMPT-BASED FILTERING
InsTag Complexity Lu et al. (2023) leveraged ChatGPT to create semantic and intent-based tags, subsequently fine- tuning an LLM as a data tagger using these tags. They then used the tag counts as measure of complexity. This used to filter out prompts with fewer tags enhance complexity.
InsTag Diversity The InsTag Diversity filtering method (Lu et al., 2023) characterizes a dataset as more diverse
5

---

Rejecting Instruction Preferences (RIP)
We primarily assess models' general inslowction-oolloiingg capabilities on three evaluation benchmarks: AlpacaEval2 (Li et al., 2023b), Arena-Hard (Li et al., 2024b) and Wild- Bench (Lin et al., 2024). These benchmarks cover a wide range of natural yet challenging real-world user queries, and have been widely adopted by the research community.
RIP scales different and larger models We also tried RIP a different base L.M from the Llama 3.3 family rather than 3.1, and of a larger scale, 70B rather than 8B, As shown in Table 3, RIP also works on this larger model. Filtering dramatically boosts Llama 5.3-70B-Instrua DPO trained models, with AlpacaEval2 LC win rate improved from 54.3% to 67.7%, 7%, Arena Hard from 70.5 to 82.9 and WildBench from 55.3 to 58.8, surpassing SOTA models as shown in Table 1. The prompt filtering threshold we applied to the 70B model was the same as in Llama 3.1-8B- 1-88--nnttt + RIP (see Appendix Table 14).
5. Experiment Results
Due to the large amount of unfiltered WildChat prompts, we first assess whether standard DPO training saturates as the size of the training prompts grows, As shown in Appendix Figure 2, the Armo Score on the valid set dramatically im- proves as we increase the size of training prompts, and begins to plateau afterwards. This shows growing the size of the training prompts arbitrarily does not bring additional gains, and hence quality control of the preference dataset could be important. We thus focus on 20k unique Wild- Chat prompts, denoted as WildChat-20k for Llama3.1-8BB Instruct experiments, and 40k for Llama b.sttee-.st
Weak-to-strong generalizability of RIP To explore po- tential weak-to teea-ng generalizability (Li et al., 2024a) of our method, we employ a smaller and weaker model, Llama 3. to filter data for a larger and more powerful LLM, Llama 3.3-7OB-Instruct. while the filtering capability of Llama 3.1-8B-I --8--Intttuct is not as powerful as that of Llama 3.3-70B-Instren significant improvements over baseline with no filtering. This showcases the weak-to-strong generation capabilities of our RIP, demonstrating that leveraging smaller model to assist a larger one in data filtering is a computationally efficient strategy.
Instruct. As illustrated in Table 4,
it still offers
We report Alpaca-Eval2 Sength-Controllee (LC) win rate, Arena-Hard score and WildBench WB-Score along with the number of training examples (after filtering if any) using WildChat-2Ok in Table 2, on HelpSteer2 in Table 5, and Self-Instruction data in Table 6. Existing filtering methods are provided in Table 2 as baseline comparisons. Further details, such as hyperparameters, are in Appendix Table 9 and Table 14, Our findings lead to several key observations.
Existing filtering methods derived from supervised- finetuning do not work as well on preference datasets As demonstrated in Table 2, compared to the baseline WildChat- 20k DPO (no filtering) trained on WildChat 20k prompts without any filtering, existing prompt- filtering meth- ods such as Ins -iifficuul -iiverssyyy or LLM-as- as Prompt-Judge filtering methods all lead lower win rates while outper-
When filtering human-written instructions, RIP achieves the best performance on both human-scored and model-scored preference datasets. On the WildChat dataset where pairs are annotated by the ArmоRM model, we conduct RIP with various cutoff thresholds, at the 25%, 50% and 75% percentile of each metric. Our best model is trained on examples with rejected length larger than the 50% percentile of all rejected lengths, and rejected rewards larger than the 50% percentile of all rejected rewards, and reward gap smaller than the 50% percentile. Table 2 shows that RIP significantly improves L.C win rate from the I a.ma... 1 1-BB Instruct DPO baseline without filtering from 48.4% to 57.8% by filtering out 77% training examples, surpassing GPT-4 Omni (05/13) on AlpacaEval2. Similarly, RIP scores the highest on Arena-Hard (43.1) compared to L.LM-as- Prommpp Judge filtering (42.0), Jaccard Similarity (42.6), and the no filtering baseline (37.9). RIP also achieves the highest WB-score on WildBench (45.6) compared to other filtering and no filtering baselines (41.5). As shown in Appendix Table 9 using LLMae annotated rewards, RIP also performs well. Finally, Table 5 demonstrates RIP is equally effective on HelpSteer2 where preference pairs are deter- mined by human annotators, achieving the highest scores across all 3 evaluation benchmarks compared to the base- lines (no filtering and LLM-as- ALM-as-Prompt-Junm Prompt-Jugee filtering).
Alpaca-Eval2. LLLM-as-Promp-JJuus forming certain filtering methods such as InsTag, achieves marginal gains compared to no filtering even though they are facilitated by querying a poweful LLM, Llama 3.1- N05B-Instruct. Out of all the alternative methods tried, Jaccard Similarity based filtering that takes into account response pairs for filtering achieves relatively the highest scores across the 3 benchmarks, indicating that filtering that only takes into account prompts or chosen responses does not generalize well to the pairwise preference case.
The Self-RIP method to generate synthetic data outper- forms Self-Instruct data. As shown in Table 6, Self-RIP yields better alignment results across all 3 evaluation bench- marks as compared to those trained on Self- Self-rnntt data. In particular, win rate improves from 49.1% 1% to 60.2% on AlpacaEval2,, and from 38.5% to 42.1% 12. on Arena-Hard. This result implies that our method generates better quality instructions than generating via few-shot examples from unfiltered prompts as in Ielf-Inst
RIP synthetic data outperforms human-writter in- structions In Table 6, models trained on synthetic prompts outperform those trained on 20k human-written WildChat
6

---

Rejecting Instruction Preferences (RIP)
Table 2: RIP compared to existing filtering methods on WildChat with Llama 3.1 3.1-8B-Instruct. RIP, which selects only 4538 WildChat prompts for DPO training, outperforms existing filtering methods on AlpacaEval2, Arena Hard WildBench. DPO response pairs are constructed using ArmoRM to score responses.
AlpacaEval2
Arena-Hare
WildBench
# Train
examples
LC Win
Win
Score
Score
Baseline
Llama Innstuc (seed model)
20.9
21.8
21.3
33.1
WildChat-200 20k DPO filtering)
20000
48.4
45.9
37.9
41.5
WildChat 20k DPO (best-vs-bottom- bottom-25%
20000
48.2
45.9
40.7
44.5
Prompt-Based Filtering
LLM-as- -aeaas-Prmp--uuueee Binary
4299
45.5
41.0
42.0
43.3
Pointwise
15963
47.4
47.4
40.7
45,2
Diifllllillyutty InsTag-Diersstty
10000
46.3
39.0
39.0
42.4
9952
40.1
41.1
40.4
43.4
Prompt-and- Baaettaand---ooen--eee Seiossen-Repoosee Based Filtering IFD on Prompt + Chosen Response
9902
47.6
37.6
32.2
42.2
ChhoChnnnn Response)
14851
45.6
45.5
40.8
43.4
Chosen-Reject-Resppee Response Based Filtering Jaccard Similarity(Chosen, Rejected)
9904
49.0
46.6
42.6
43.7
RIP
4538
57.8
57.2
43.1
45.6
Table 3: RIP WildChat with Llama 3.3-70B-Instruc' RIP outperforms no filtering AlpacaEval2, Arena-Hard & WildBench. DPO response pairs are constructed using ArmoRM to score responses.
AlpacaBval2
Arena-Has Hard
WildBench
# Train
examples
LC Win
Win
Score
Score
Llama 1.3-70B-Instruc (seed model) WildChat-40k DPO (no filtering)
38.9
67.5
52.8
40000
54.3
51.6
70.5
55.3
RIP
17725
67.7
73.2
82.9
58.8
Table 4: Weak to strong generation ability with RIP. RIP on Llama 3.3-70B -.strot-Instruct by employing a smaller model
Llama 3. Inssructnstruct for filtering outperforms no filtering baseline, while underperforms using its own generations.
AlpacaEval2
Arena-Hard
WildBench
Filter
Train examples
Seed Model
Model
LC Win
Win
Score
Score
Llama 3.3-70B-Instren
(no filtering)
40000
54.3
51.6
70.5
55.3
Llama 33-70B-Instruu
Llama Inssructttt Llama 3.3-70B-Instrun
18184
64.5
69.2
76.7
Llama 1.3-70B-Instrru
17725
67.7
73.2
82.9
58.8
prompts. Applying Self-RIP few-shot generation without post-filtering gives an equal amount of 20k prompts, but still increases the AlpacaEval2 LC win rate from 48.4% to 53.6%, Arena-Hard win rate from 37.9% 43.7% and WB- Score WildBench from 41.5 to 44.8. This further illus- trates the importance of training high-quality instructions. When applying the full RIP method with post-filterin results are further improved, for example achieving the best Alpa2aEval2 LC win rate of 60.2%.
ing RIP to select high quality few-shot examples but not for curating the resulting generations (post-filtering;; (ii) apply- ing standard (Self-Instruct) few-shot generation, but then applying RIP post-filtering: or (iii) applying RIP both few-shot generation and post-filtering (our default method), We find that both components of our full method are impor- tant yielding the best results, with method (i) outperforming Self-uutt and method (ii) performing better than (i), but worse than our full method (iii).
RIP seed data selection and RIP post-filtering are both important for generating Self-RIP synthetic data In Table 6, we perform ablations on Self-RIP. We try: (i) us-

---

Rejecting Instruction Preferences (RIP)
Table 5: RIP on HelpSteer2 with Llama 3.1-8B-Instrun Applying RIP to DPO models trained on HelpSteer2 outperforms the baseline of no filtering as well as using the Llama 3.1
model as a pointwise prompt quality judge.
AlpacaEval2
Arena-Hard
WildBench
# Train
HelpSteer2
examples
LC Win
Win
Score
Score
Llama 3.1-8B- --B-Instruct (seed model)
20,9
21.8
21.3
33.1
HelpSteer2 DPO (no filtering)
10161
25.2
23.1
26.8
37.1
LLM-as- Prompt-Judge filtering
5376
27.8
25.7
29.5
37.2
RIP
5081
34.6
32.8
35.0
39,5
Table 6: Self-RIP for generating high-quality synthetic instructions. Self-RIP creates prompts using few-shot samples uses few-shots from unfiltered WildChat prompts. Applying
from high-quality prompts curated by RIP, whereas RIP filtering after generation also important, and achieves the best results, significantly outperforming Self- data.
AlpacaEvvl2
Arena-Hard
WildBench
Post-
# Train examples
Train Prompts
Filtering
L.C. Win
Win
Score
Score
WildChat-20k
None
20000
48.4
45.9
37.9
41.5
WildChat-20k
RIP
4538
57.8
57.2
43.1
45.6
Self-Instruct
None
20000
49.1
46.9
38.5
40.0
Self-RIP (without post-filtering Self-Instruct with RIP post-filtering Self-RIP
None
20000
53.6
56.1
43.7
44.8
RIP
16261
58.3
53.2
40.9
44.1
RIP
18812
60.2
61.1
42.1
42.5
6. Understanding why RIP works
quality. Table 7 illustrates the prevalence of low-quality examples (with score 4 or 5 by both GPT-4 and LLama3. 1- 405t-Instruct) after applying various filtering methods. We observe that filtering based on the reward and length of the rejected response the most effective way to ensure prompt quality, compared to other methods tried. By combining those rejected response quality metrics with the reward gap, RIP reduced percentage of noisy prompts from 22.9% to 8.9%. This supports our hypothesis that very low-quality prompts, such as those in WildChat that consist of incom- plete snippets from movies, stories, or code (see sample rejected instructions in Appendix Table 32 and Table 33), often result in poor rejected responses when sampled sev- eral times. By leveraging the quality of rejected responses filtering criterion, can efficiently eliminate these extremely noisy prompts.
6.1. Filtering prompts with low quality responses
To understand what instructions are filtered out, we first visualize instructions with low quality rejected responses (as measured by low reward and short lengths) by compar- ing the t-SNE plots of unfiltered and filtered instructions (shown in Appendix Figure 5). We investigated a few clus- ters present in that tSNN plot of unfiltered prompts that are missing from the t-SNE plot of filtered ones on the right- hand-side. We find that instructions from those clusters being filtered out from the training set are either obscure, non-sensical, or they fail to elicit meaningful responses from the model, leading to lower-quality rejected responses. Such instructions can be caught by measuring the rewards and lengths of the rejected responses, with supporting evidence given in Appendix Table 32.
Furthermore, we employ GPT-4 and
Next, we employ GPT-4 and LLama3. 1-40BB Instructt to evaluate first 10,000 prompts from WildChat. Focusing solely the instructions (excluding responses) provided in WildChat, the model is tasked with scoring each prompt on a scale from I to 5. A score of 1 represents the most help- ful prompt, while a score of indicates the lowest quality. The evaluation prompt is provided in Appendix Figure 3. Manual review revealed that prompts assigned scores of and 5 were of very low quality, while those scored 3 were moderately acceptable, albeit with some quality issues still present. Notably, GPT-44 and LLama3.1 1.ama3.1-4055 1-405--Instrue oc- casionally assigned scores of 2 or 3 to a prompt of low
Instruct to respond to each WildChat prompt three times. any response declines to answer due to safety concerns, we categorize those prompts as unsafe. It's important to note that with this method, the model sometimes assigns high quality scores to prompts that are borderline unsafe. By examining the reward and the length of rejected responses, we observe RIP is also an effective approach to filter out these unsafe prompts. This approach is grounded in the ob- servation that rejected responses when dealing with unsafe instructions are typically short and have low reward scores.
8

---

Rejecting Instruction Preferences (RIP)
6.2. Filtering prompts with larger response variance
For such models, safety will also be crucial, and future work should additionally address this aspect. In experiments, the reward is not explicitly constrained by safety-related criteria. Therefore, a clear further avenue of study is to conduct safety evaluations and to explore safety filtering using our methods, with reward models built exclusively for safety existing systems (Touvron et al., 2023b).
Similarly, we visualize instructions that are filtered out by measuring the reward gap between chosen and rejected re- sponses in Appendix Figure 6, and further expand some representative groups of filtered instructions in Appendix Table 33. In particular, instructions that cover specialized domains such as coding, software, and other technical ques- tions often require precise details, well-defined objectives or targeted solutions. In those cases, lack of specificity in the instructions might lead to more variable responses. As shown in Table 33, instructions with larger reward gap are not necessarily low-quality. however we hypothesize that the combination of lack of specificity in the instruction and larger difference in the response pair make them less helpful in improving the model during preference optimization.
Given that we have shown that RIP can filter potentially unsafe prompts, this could mean in the best case that the safety of the model could potentially improve after filtering as well, with RIP being able to catch and mitigate more chal- lenging safety situations that earlier iterations cannot. From a broader perspective, this work could pave the way for methods that produce higher-quality training instructions, that are also potentially safer than organic user instructions in the wild.
Table 7: Effectiveness of Filters on Prompt Quality and Safety: we compare the number of noisy and potentially unsafe (as judged by GPT4) WildChat instructions (out of 20k) filtered by various filtering methods.
References
Balioglu, C. fairseq2, 2023. URL http://github.
com/ baaabbokkeesarch/aaaaasss
Filtering Methods
% of low-quality prompts 4.
% of unsafe prompts +
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Unfiltered Data
22.9%
12.27%
Reject Reward
10.4%
0.04%
Reject Length
13.9%
0.02%
Reward Gap
17.7%
8.07%
RIP
8.9%
0.00%
Cao, Y., Kang, Y., Wang, C., and Sun, L. Instruction min- ing: Instruction data selection for tuning large language models. arXiv preprint arXiv:2307.062:
2023.
7. Conclusion
Chen, L., Li, S., Yan, J., Wang. H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. Al- paGasus: Training better alpaca with fewer data. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: openreview .net/ Forum?id-FaVXee
This work introduces Rejecting Instruction Preferences (RIP), method for improving preference data quality by measuring the rejected response quality and the reward gap between the chosen and rejected response pair. Filtering instructions using RIP remarkably improves model align- and synthetic instruc-
ment results on both tions, and for different reward signals. In addition, we show that Self-RIP. synthetic instructions generated by few-shot prompts curated by RTP, outperforms organic user instruc- tions and the standard Self-Instruct method, achieving the highest AlpacaEval2 win rate in our experiments.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
Impact Statement
This work demonstrates the possibility of dramatically im- proving LLMs by identifying and producing high-quality training data. Studying how filtering criteria affect outputs will continue to be important for LLM training. While we have primarily focused on preference optimization, the RIP approach is general and can potentially work for any training scheme, e.g. other RL training techniques which future work should explore.
Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnat- ural instructions: Tuning language models with (almost) no human labor. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- ume Long Papers), pp. 14409-14422 Toronto, Canada,

---

Rejecting Instruction Preferences (RIP)
July 2023. Association for Computational Linguistics. doi: 10.16653w1/202333ccooon.gg. URL. https: //aclanthology.orgg///ooo
Lu, K., Yuan, H., Yuan, Z., Lin, R., Lin, J., Tan, C., Zhou, C., and Zhou, J. instag: Instruction tagging for ana- lyzing supervised fine-tuning of large language models. In The Tweifth International Conference on Learning Representations, 2023.
acl long. 606. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference- reward. arXiv preprint arXiv:2405. 14734, 2024.
arXiv preprint arXiv:2001.08360 2020.
Khaki, S., Li, J., Ma, L., Yang, L., and Ramachandra, Rs-dpo: hybrid rejection sampling and direct prefer- optimization method for alignment of large language models. arXiv preprint arXiv:2402. 10038, 2024.
Mishra, S., Khashabi, D., Baral, C., and Hajishirzi. H. Cross- task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.0877 2021.
Lambert, N., Morrison, J., Pyatkin, V., Huang. S., Ivison, H., Brahman, F., Miranda, J. V., Liu, A., Dziri, N., Lyu, S., et al. T ulu 3: Pushing frontiers in open language model post-training. arXiv preprint 1rXiv22411.15124 2024.
Pace, A., Mallinson, J., Malmi, E., Krause, S., and Severyn, A. West-of-n: Synthetic preference gener- ation for improved reward modeling, arXiv preprint arXiv:2401.12086 2024.
Rae, W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 1144, 2021.
Li, M., Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity quality: Boosting IIm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. 12032, 2023a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly areward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL httpt: forum? Frrumidd=HPuSTXaae / /ttp:////eeeeeeeeeeeeeeeemeeeeee net./
Li, M., Zhang, Y., He, S., Li, Z., Zhao, H., Wang, J., Cheng, N., and Zhou, T. Superfilterigg Neak-to-strong data filtering for fast instruction-tuni tunnonnttnnnnn--onnnn arXiv preprint arXiv:2402.0000 005300 2024a.
Li, T., Chiang, W.-L., Frick. E., Dunlap, L.. Zhu, B., Gonzalez, J. E., and Stoica, I. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL https: /ttp///mmssss//////// htt./mmsss.orrgg org.blog//// 10w--04-----rr---aaa----
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I,, Guestrin, C., Liang. P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instructionfollooinng models. https: //ss:://iii/ httbt.....u.... com/tatsu-lee lab/ aopaca_eval,, 202b.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv: 240200.900.0 2024.
Li, X., Yu, P., Zhou, C., Schick, T., Zettlemoyer, L., Levy, O., Weston, J., and Lewis, M. Self-alignment with instruc- tion backtranslation. In The Twelfth International Confer- ence on Learning Representations, 2024c. URL https: //openrevien net/ /eet/forre hwrum/orummmmmooo 1id1dd1110 1000:111100 jodo--ooijHBRs... jJJRRT.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Ad- vances Neural Information Processing Systems, 33: 3008-3021, 2020.
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichan- der, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wildhench: Benchmarking lims with challenging tasks from real users in the wild, 2024. URL https:
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A.. Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 18/2. 051.99, 2018.
// //aaxiv. / abs/ 240/ 04770.
Liu, W. Zeng, W., He, K., Jiang, Y.. and He, J. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312 15685, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302 /3971, 2023a.
10

---

Rejecting Instruction Preferences (RIP)
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Bahaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng. J., Tao, C., and Jiang, D. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.0 1tt.:23304.1224,,, 2023a.
Xu, Lee, A., Sukhbaatar, S., and Weston, J. Some things
Wang. H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12845, 2024a.
cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023b.
Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint wrXiv:2205009399 2022.
Wang, T,, Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., Weston, J. and Li, X. Self-taught evaluators. arXiv preprint 2tl.:2408.02666 2024b.
Yasunaga. M. Shamis, L., Zhou, C., Cohen, A., We- ston, J., Zeulemoyer, L., and Ghazvininejad, M. Alma: Alignment with minimal annotation. arXiv preprint arXiv:24/2.04N3 2024.
Wang, Y., Mishra, S., Alipoormolabashi, P. Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. huperr-aturalinstructioss Generalization via declarative instructions on 1600+ nip tasks. arXiv preprint arXivv2204.0000 2022.
Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., We- ston, J., and Xu, J. Following length constraints in in- structions. arXiv preprint arXiv:2406.117744 2024a.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language models with self- self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro- ceedings of the 61 Anmual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 2023.acl-long.7 754. URL http::/:///claahhoolgg....////// org/2023 a acl long. 754.
Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024b.
Zhao, H., Andriushchenko, M., Croce, F., and Flammarion, N. Long is more for alignment: A simple but tough-to- beat baseline for instruction fine-tuning. arXiv preprint arXhv:2402.0443 2024a.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatgpt interaction logs in the wild, arXiv preprint 2.MY:220550017777 2024b.
Wang, Z., Dong, Y., Delalleau, O., Zeng. J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open-source dataset for training top-pent sop-erforrmin reward models. arXiv preprint 2466:2406008673, 2024c.
Zheng, I.,, Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, Judging LLM-as-a juddee with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2023. URL https: / //openreview.co net forum? www.mm/ddduccccPI.........
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for alignment, Advances in Neural Information Processing Systems, 36, 2024.
Wu, J., Xie, Y., Yang, 7.,, Wu, J., Gao, J., Ding, B., Wang, X., and He, X. 8-dpo: Direct preference optimization with dynamic 3. arXiv preprint arXiv:2407.08639, 2024a.
Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self deff-mmpovving alignment with Ilm-an as-a-meta- judge. arXiv preprint arXiv:2407. 19594, 2024b.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.0433. 2024.

---

Rejecting Instruction Preferences (RIP)
A. Appendix
A.1. More Details on Experiment Setup
Our experiment setups are summarized Table 13. Specifically, we apply RIP to multiple popular instruction-followiin datasets as well as our own synthetic data, with reward annotated from various sources (human/reward Classifier/LLLMM M.arrs--LM---- a- Judge), indicating the generalizability of RIP method.
Table 8: Preference Dataset Statistics used for training in our experiments.
Human Written
Reward
#Prompts
#Responses
Annotator
Valid Set ( (# Examples)
WildChat- turn 20k
20,000
Yes
8,16,32,64
ArmoRM
Humpback + Evol-Instruce (470)
WildChat-turn 20k
20,000
Yes
64
a-Maas---Judg
Humpback + Evol-Instruct (470)
HelpSteer2
10,161
Yes
2
Human
HelpSteer2 valid (519)
Self-Instruct
20,000
No
64
ArmoRM
Humpback Evol-Instruct (470)
Self-RIP
20,000
No
64
ArmoRM
Humpback + Evol-Instruct (470)
We report the model performance on valid set when varying the number training WildChat prompts in Figure 2. Model training improves significantly as training data size grows to 20k and then begin saturates afterwards, therefore our main experiments are based those 20k WildChat prompts.
Figure 2: Results on DPO Training with Varying WildChat Data Sizes. Using different sizes of WildChat data for DPO training on LLaMA 2.1-8B-Instrun the performance, measured by Armo rewards on the validation set, gradually saturates as the data size increases.
Rewards Data Size
0.184
0.183
0.1831
0.1831
0.1829
0.1823
Rewards
0.182
0.1812
0.181
0.180
1794
0.179
0.178
0.
1774
Rewards Valid Set
0.177
10k
20k
30k
40k
50k
Data Size
We primarily assess our models' general instraction-followinng capabilities using three popular evaluation benchmarks: AlpacaEval-2 (Li et al., 2023b), Arena-Hard (Li et al., 2024b) and WildBench (Lin et al., 2024). AplacaEval-2 consists of 805 prompts sampled from datasets. Arena-Hard contains 500 challenging user queries sourced from Chatbot Arena and has the highest correlation and separability models commpared to Chatbot Arena among popular open-ended L.LM benchmarks (Li al., 2024b). WildBench is built from of 1024 significantly harder, challenging queries carefully curated from the WildChat project (Zhao al., 2024b) ensure diversity and complexity. The automatic evaluation of WildBench involves task-specific checklists that guide LLM judges in generating reliable and consistent judgments which demonstrate significantly high correlation with human judgments. We report the WB-Score for individual scoring.
A.2. Additional Results
LLM-as-a-Jude Juddee As Reward Annotators We explore LLM-as-a-Judge alternative reward annotator apart from the reward model ArmoRM and human reward annotations, and use a LLama3. 1-405B--nsttruc zero-shot to judge the quality of each individual response and uses its prediction to construct response pairs. For each response, we conduct 10 independent evaluations and calculate the average score to determine the final reward score. We report AlpacaEval2, Arena-Hann Hard and WildBench results on WildChat DPO models in Table 9. Similar to the observation from Table 2, RIP by filtering based on L.MMaas--Judge predictions outperforms no filtering.
12

---

Rejecting Instruction Preferences (RIP)
AlpacaEval2
Arena-Han
WildBench
# Train
WildChat with LLM-as-a-Judge as reward annotator
examples
LC Win
Win
Score
Score
Llama 3.1 3.1-SB-Inttruct (seed model)
20.9
21.8
21.3
33.1
Standard DPO (no filtering)
16837
40.1
44.9
41.1
42.5
RIP
5999
44.3
48.8
42.5
43.9
Table 9: RIP compared to baselines on WhldChat using LLM-as-a-Judge as the reward annotator. We report results on AlpacaEval2,, Arena-Hard and WildBench of various models trained using DPO on the WildChat Dataset. RIP outperforms the baseline of LLM-as -udddgejdde as the reward annotator.
Data Scaling with RIP We further scale up RIP by growing the training data size after filtering to 20k, and achieves AlpacaEval2 LC win rate of 58.49% as shown in Figure 1. While the effective training size scales from 4538 to 20k, the actual performance gain only increase slightly, suggesting that training with Llama 3.1-8B -----Instruct on existing WildChat prompts saturates, even under RIP.
RIP filtering thresholds
We report the filtering thresholds of the best checkpoints in our experiments in Table 14.
Full Evaluation Results We include full WildChat evaluation results on AlpacaEval2 and Arena-Hard in Table 15 and on WildBench in Table 16, with average response lengths, confidence intervals as well as finegrained results on subtasks. Full evaluation results on models trained on HelpSteer2 are presented in Table 17 and Table 18. In addition, full evaluation results on Self- RIP are included in Table 19 and Table 20.
Coordinate-wise Filtering results. We conduct extensive experiments by applying filtering to each individual metrie: reward on chosen or rejected response, lengths of chosen or rejected response, reward gap, average reward of all responses, etc. Results on valid set performances by applying various filtering metrics to WildChat task are included in Table 24, Table 27 and HelpSteer2 in Table 25. Both highlight strong performance hoost by filtering based on rejected reward, rejected length and reward gap.
RIP on Gemma2 models To show the effectiveness of our RIP filtering beyond Llama models, we finetune Gemma2-9B.n model with SimPO using the dataset (grincto-nil/llmmm--llltteeobbb armmrr) which are Gemma2 generations on ultrafeedback annotated by ArmoRM.
Table 10: RIP on Gemma2-9B SimPO Applying RIP on Gemma2-9B finetuning further improves Gemma2 performance on AlpacaEval from 69.48 to 73.81 by filtering out 50% train data.
Filtering
#Prompts
Alpaca I.C. Winrate
Alpaca Winrate
Gemma2-9B SimPO (no filtering) Gemma2-9B SimPO (RIP filtering)
59569
69.48
63.07
29963
73.81
62.01
RIP with Smaller-sized Reward Model different model family, we select a lightweight non-Llama-basee reward model RRay2333/GRMMgemma-233 ft", which is a Gemma2-2B based reward model, to annotate and then DPO finetune a Llama3, 1-BB ----8--Inssucc model. Below are the results on RIP filtering using reward scores by this hemma2-based RM. (ranked 36th on Reward Bench) is ranked below ArmoRM on RewardBench, and the performance gap in the two reward model quality also affects the performances of finetuning Llama3. 10mma3.1-BB-.. I-St-unttt with reward model annotations(i.e. better- quality reward model leads to better winrate of finetuned models). However, in both cases RIP filtering demonstrates its effectiveness.
To show the effectiveness of RIP if reward models are of smaller size, and from a
Comparison with DPO 3-DPO(Wu et al., 2024a) filtering is online, meaning they filter out data in every batch, whereas our approach filters data offline. This offline filtering enables more flexible and efficient generation pipelines, particularly for weak-to-stine to-stoong generation scenarios. For instance, finetuning Llama3.3-70B-Instte by a smaller Llama3.1 IB-Irutt model outperformed (Alpaca L.--wie improved from 54.3 to 64.5, Arrda-Hard from
on prompts RIP filtered
13

---

Rejecting Instruction Preferences (RIP)
Table 11: RIP with Gemma2 based Reward Model. By curating less than 5k out of 20k prompts, we can improve Llama3. DPO models from 41.1 to 49.9 on AlpacaEval LC-winrate, showing similar improvement using ArmoRM filtering (LC winrate improved from 48 38.4% to 57.8%).
Filtering
#Prompts
Alpaca LC Winrate
Alpaca Winrate
WildChat20k baseline
20000
41.1
47.3
WildChat RIP
4401
49.9
53.5
Table 12: RIP Gemma2-9B SimPO Applying RIP on Gemma2-9B finetuning further improves Gemma2 performance on AlpacaEval from 69.48 to 73.81 by filtering out 50% train data.
Filtering
#Prompts
Alpaca LC Winrate
A Alpaca Winrate
Gemma2-9B SimPO (no filtering)
59569
69.48
63.07
Gemma2-9B SimPO (RIP filtering)
29963
73.81
62.01
70.5 to 76.7). There are some noticeable distinctions between our methods and B-DPO. 3-DPO removes both small and large gaps, our method removes bigger gaps only. In addition, 3-DPO'ss filtering probabilistic, resulting in incomplete data removal, whereas our approach uses deterministic filtering ensure thorough removal of unwanted data. Given these differences, and previously mentioned, our method prioritizes Rejected Reward and Rejected Length criteria over gap-based filtering, which have demonstrated superior effectiveness in our experiments.
Table 13: Preference Dataset Statistics used for training in our experiments.
Filtering
3-DPO mode weight
#Prompts
Valid Score
Alpaca LC Winrate
Alpaca Winrate
No filtering
19803
0.1830
48.37
45.87
RIP filtering
4538
0,1898
57.83
57.16
3-DPO Filter
0.2
15842
0.1842
49.15
49.00
3-DPO Filter
0.5
9901
0.1840
46.68
42.41
3-DPO Filter
0.75
4950
0.1827
45.97
40.58
A.3. t-SNE Analysis of Filtered Instructions
We conduct t-SNE analysis on WildChat prompts filtered by rejected response length and reward in Figure 5 and those further filtered by reward gap in Figure 6. To better understand which prompts are being filtered out, we summarize prompts being filtered out due to rejected responses being of shorter length or lower reward in Table 32 and Figure 7, and those filtered out due to large reward gaps Table 33. In addition to visualizing the examples, we also conduct GPT4 analysis prompts in Section 6.2 by each criterion, to justify our hypothesis. We include one such
into quality of the filtered sample analysis below. A prompt "Write story" is filter out (rejected ARMO, gap), and is considered not useful this prompt is overly broad and lacks specific details, posing challenges in generating focused response.
A.4. Further Ablations
We report results of further ablation studies: comparing filtering and various pairing instead of filtering methods in Table 21, and robustness of RIP to choice of responses in rejection sampling in Table 22.
RIP outperforms alternative preference pairing methods We compare RIP to methods without filtering that use different response pairing methods for building pairwise preferences. Recall that in our main experiments for RIP we used the best-vs-worst pairing method as described in Section 3.1. Here explore two alternative methods: (i) rest-vs-random which is shown by existing work (Yasunaga et al., 2024; Khaki et al., 2024) to outperform best-vs-worst, and (ii) best-vs- bottom-K" K% percentile where the rejected response has the bottom K = 25, 50, 75 percentile score (K = 0 being the lowest score). Both pairing methods can effectively lower reward gap and increase quality of rejected response without removing training prompts. We report model performance on the valid set in Table 21. Out of all pairing methods, best-vs-bottom- works the best, but still under-performs compared with our RIP method (pairing with best-vs- vo-woss)). When evaluated on
14

---

Rejecting Instruction Preferences (RIP)
Wildbench
Table 14: Full Results details on number of training examples, choice of reward models, seed models, filtering metrics and thresholds chosen as well as final outcomes across 3 evaluation benchmarks.
33.1
52.8
41.5
45.5
45.6
41.0
44.1
44.8
42.5
42.5
43.9
37.1
39.5
55.3
58.8
Arena- Hard
21.3
37.9
42.3
43.1
38.5
43.7
42.1
41.1
42.5
26.8
35.0
70.5
82.9
Win
21.8
41.5
45.87
52.9
57.2
46.9
53.2
56.1
61.1
44.9
48.8
23.1
32.8
51.6
73.2
AlpacaEval2
LC Win
20.9
38.9
48.37
57.1
57.8
49.1
58.3
53.6
60.2
40.1
44.3
25.2
34.6
54.3
67.7
Rejected LLL--as-a-Judee --ddge- Reward > 8, Rejected Length > 1399, Reward Gap
Rejected Length > 1878, Rejected
Rejected Length 2 1878. Rejected Armo > 0.126, Reward Gap < 0.042 No
Rejected Length Z 1878. Rejected Armo 2 0.126 No
Rejected Length 2 1878, Rejected Armo 2 0.126
Rejected Length > 1878, Rejected Armo > 0.126. GAP & 0.042
1303
Filtering Metries
Armo 2 0.126
Rejected Length
No
No
No
No
Seed Model
Llama 1-88
Liama 3.3-
10B-Instruct
Llama 3.1-8B-
Llama 3.1-8B-
Llama 3.1-BB-
Llama 3.1-8B-
Llama 3.1-8B- Instruct
Llama 3.1-8B- Instruct
Llama 3.1-8B- Instruct
Llama 3.1-8B- Instruct
Llama 3.1-8B-
3.1-8B-
Llama 3.1-8B-
3.3-
70B Instruct
Llama 3.3-
70B 70B- Inst-ntt
Instruct
Instruct
Instruct
Instruct
Instruct
Instruct
Instruct
Instruct
Llama
Llama
Reward An-
ArmoRM
ArmoRM
ArmoRM
ArmoRM
ArmoRM
ArmoRM
ArmoRM
LLM-as-a-
ArmoRM
ArmoRM
notator
L.
Human
Human
Judge
Judge
# Re-
sponses
64
64
64
64
64
64
64
64
32
Human Writlen
Yes
Yes
Yes
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
# Train
Exam- ples
20k
6762
4538
20k
20k
18812
16.8k
5999
5081
40k
17.7k
Synthetic (few shot:
Synthetic (Tew shot:
Synthetic (few shot:
Synthetic (few shot:
Wildchat 20k
Wildchat 20k
Wildchat 201)
Wildehat 20K)
Wildchat filtered
4538 examples)
Wildchat filtered 4538 examples) Wildchat 20k
Wildchat 20k
Wildchat 40k
HelpSteer
HelpSteer
Data
Wildchar
Wilcchat
15

---

Rejecting Instruction Preferences (RIP)
Table 15: Full AlpacaEval2 & Arena-Hard Results on WildChat: we compare performances of SOTA models on AlpacaEval2 win rates and Arena-Hard scores as well as DPO models trained on the WildChat-200 dataset using various filtering methods.
AlpacaEval2
Arena-Hard
Train
Standard models
examples
LC Win
Win
Len
Score
95% CI
Len
GPT-4 Omni (05/13)
57.5
51.3
1873
74.9
(-2.5, 1.9)
668
GPT-4 Turbo (04/09)
55.0
46.1
1802
82.6
(-1.6, 1.8)
662
Gpt-4-0613
55.0
46.1
1802
37.9
(-2.8, 2.4)
354
Llama 3.1 2-1-405B-Instree
39.3
39.1
1988
67.1
(-2.2, 2.8)
658
Llama 3.1-70B-Inse Instrucct
38.1
39.1
2044
69.3
(-2.5,
658
Baseline
Llama 3.1 1-88B-Instruct
20.9
21.8
2184
21.3
(.1,9,
861
WildChat-200 20k DPO (no filtering)
20000
48.4
45.9
2134
37.9
(-2.0, 2.2)
622
WildChat-20k DPO -oss----botto--25%
20000
48.2
45.9
1971
40.7
(-2.1, 1.9)
741
Prompt-Based Filtering
Jaceard Similarity
9904
49.0
46.6
1978
42.6
(-2.4, 2.3)
LLM-as-Prae aas--roppt-Juddee Binary
4299
45.5
41.0
1859
42.0
(-1.4, 1.7)
597
SiM-as-Prompp-Judee Pointwise
15963
47.4
47.4
2056
40,7
(-2.0, 2.2)
701
Ins Tag--aafficully
10000
46.3
39.0
1752
39.0
(-2.2, 2.3)
602
Ins TiseeDDivrss
9952
40.1
41.1
1903
40.4
(-2.4, 2.8)
579
Filtering
IFD on Prompt + Chosen Response
9902
47.6
37.6
1655
32.2
(-1.7, 2.5)
533
PhhsChn Response)
14851
45.6
45.5
1930
40.8
(-2.3, 1.7)
582
Reabenn-ReeecteRReepppnn Based Filtering
M.L.-as- aas-Prompt-Judee Pointwise
15963
47.4
47.4
2056
40.7
(-2.0, 2.2)
RIP
4538
57.8
57.2
2048
43.1
(-1.5, 1.8)
638
AlpacaEval2. Arena-Hard and WildBench, the model WildChat-200 DPO improvement gain comparing to baseline WildChat-20k DPO woossss,, while still underperforming RIP as shown in Table 2. This result indicates that reward gap being small the rejected reward being high better works as an indication of low-quality prompt rather than bad response pairing.
sostt-vv-bottom-2556 only achieves a slight
Combining alternative pairing with RIP performs on par with best-vs-bottom pairing with RIP. We further apply RIP filtering to examples paired by best-vs-bottom-255 pairing. Combing best-vs-bottomm2555 with filtering out examples of low quality rejected responses yields ArmoRM Score of 0818875, slightly lower than best-vs- worss + filtering by Rejected Reward (0.1 8795). Filtering out best-vs-bottom-2556 examples of bigger reward gaps yields to Armo Score of 0.1860 on valid set compared 0818542 from best-vs- worst pairing filtering by Reward Gaps. Given the marginal performance gain between best-vs-worst and best-vs-bottom-2554 pairing with and without RIP, we thus focus on the more widely adopted best-vs- best-vs-worst pairing to experiment various filtering methods including our RIP method.
RIP robust the choice of the number of responses N. While we showed RIP provides strong performance on HelpSteer2 where only N 22 responses availabe for each prompt, and WildChat with N = 64 responses sampled per prompt, we also compare the performance of RIP by varying the choice of N the number of candidate responses generated for preference annotations in the WildChat setup. As shown in Table 22, for a wide range of values N = 64, 32, 16, 8, RIP consistently outperforms the no filtering baseline, with larger N achieving increasingly better performance, likely due to the increased quality and variability of chosen and rejected responses, allowing our RIP metrics to be more accurate in curating high quality data.
RIP works with much smaller set of high-quality seed instructions Instead of using all 4538 RIP curated high- quality instructions as seed instructions S during Step 1. few-shot generations, we sample a much shorter subset of 256 prompts from 4538 RIP-curated prompts as seed instructions, and only conduct few-shot generations by sampling prompts from the 256 seed prompts each time. We report Self-RIP with and without post-filterig in Table 23. Self-RIP based on 256 high-quality seed instructions (58.9) slightly underperforms than that based on 4538 seed prompts (60.2), but
16

---

Rejecting Instruction Preferences (RIP)
Table 16: Full Winch Results on WildChat: we compare performances of SOTA models on WildBench as well as DPO models trained on the WildChat-200 dataset using various filtering methods.
WB-Score
WB-Score: Task-specific
Weighted average
Planning & Reasoning
Math & Data Analysis
Coding & Debugging
Standard models
Creative
Information
GPT-4 Omni (05/13)
59.3
59.1
60.2
57.3
58.6
60.5
GPT-4 Turbo (04/09)
55.2
58.7
56.2
51.0
57.2
55.1
Gemini-1e 1.5-pro
53.0
55.1
53.7
48.6
52.2
55.2
47.8
54.3
50.1
42.1
52.3
44.7
Baseline
Llama 3.1-8B 8---Instrcct
33.1
45,0
37.0
23.9
37.4
29.3
WildChat-20 20k DPO (no filtering)
41.5
51.8
44.2
32.2
50.0
37.1
WildChat-20k DPO (best-vs-bottom-25%)
44.5
53.9
47.4
35.8
50.4
41.4
Crompt-Based-e Filtering
Jaccard Similarity
43.7
54.2
46.9
34.3
40.5
SAMMas-Prompt-Junse Binary
43.3
53.9
46.6
35.8
48.5
38.6
LLM-as-Prompte Pointwise
45.2
55.6
48.0
37.1
51.6
40.9
42.4
52.7
45,4
33.4
47.8
39.3
InsTag-Dvverss
43.4
53.4
46.1
35.0
49.1
40.1
Pronpraand-ChhseeeeepossseesBa Filtering
IFD
42.2
51.3
45.9
35.0
48.0
37.1
CChoSen Response)
43.4
52.5
47.0
37.2
49.4
37.6
Chosen-Reieeteed-Resppn Based Filtering
I. lilaa---roop--udgee Pointwise
45.2
55.6
48.0
37.1
51.6
40.9
RIP
45.6
56.7
48.8
36.6
51.6
41.4
Table 17: Results of our DPO models trained with HelpSteer2. Full AlpacaEval2 & Arena-Hard Results of our DPO models trained with HelpSteer2 Dataset.
AlpacaEval2
Arena-Hard
Prompts
LC Win
Win
Len
Score
95% CI
Len
Baseline
Llama 3.1-8B-JJ Instruct
20.9
21.8
2184
21.3
(-1.9, 2.2)
861
HelpSteer2 DPO (no filtering)
10161
25.2
23.1
1733
26.8
(-2.0, 2.4)
606
Prompt-Based, Coompt-Based-Fiteee Filtering
LLMMas-Prompt--une Pointwise
5376
27.8
25.7
1947
29.5
(-2.8, 2.3)
627
Frompt-Responsee Filtering
RIP
5081
34.6
32.8
1941
35.0
(-1.8, 2.2)
621
still outperforms Self-Instruct with RIP post-filtering (58.3) as well as Self-RIP based all 4538 seed prompts without post-filtering (53.6), indicating that our method Self-RIP can work well with much smaller set of high-quality seed prompts.
A.5. Details about Baselines
InsTag Complexity Lu al. (2023) utilized ChatGPT to generate semantic and intent-based tags, which were then used to fine-tune large language model (LLM) data tagger. The number of tags per prompt served complexity metrie. Building on their methodology, we employed a publicly available tagger (https: / / thub. www/co/.caa///o Saam//FA-SSS Sys/ InsTag. Note that Meta was not involved in the training of the Instag model we used.) to annotate each prompt, generating between I and 100 tags per prompt. We then categorized our training prompts into four groups based on the number of tags: more than 2, more than 3, more than 4, and more than 5. From each group. we randomly sampled 10,000 training data samples and trained distinct model for each group. It important note that threshold of > implies no filtering, with only
17

---

Rejecting Instruction Preferences (RIP)
Table 18: Results on our DPO models trained with HelpSteer2. Full WildBench results of our DPO models trained with
HelpSteer2 Dataset.
WB-Score
WB-Score: Task-specific
Weighted average
Planning & Reasoning
Math & Data Analysis
Coding & Debugging
Creative
Information
Baseline
Llama 1.1-8B-Innttrue
33.1
45.0
37.0
23.9
37.4
29.3
HelpSteer2 DPO (no filtering)
37.1
48.6
40.4
26.5
44.3
33.4
Prompt-Basee Based-Filterng
LLM-as as- SoM-as-Prommpy Prompt-Judee Pointwise
37.2
50.6
40.0
27.9
43.0
33.1
Prompt-Responsee Baseee----a------ Raxed-Filtering RIP
39.5
52.1
42.9
29.3
46.4
35.0
Table 19: Results of our DPO models trained with Delf-unstructed Dataset: Full AlpacaEval2 Arena-Hart Results comparing method with training standard Self- Inff-Instruct dataset.
AlpacaEval2
Arena-Hard
Train examples
Training Prompts
Filtering
L.C. Win
Win
Len
Score
95% CT
Len
Self-Instrrut
None
20000
49.1
46.9
1956
38.5
(-1.4, 1.6)
738
Self-RIP (without post-filtering)
None
20000
53.6
56.1
2252
43.7
(-2.3,
777
Self-Instruct with RIP post-filtering Sclf-RIP
RIP
16261
58.3
53.2
1823
40.9
(-1.9, 1.6)
560
RIP
18812
60.2
61.1
2121
42.1
(-2.0, 2.4)
606
Table 20: Results of our DPO models trained with Self-Instructed Dataset: Full WildBench Results comparing our method with training on standard Self-Instruce dataset.
WB-Score
WB-Score: Task-specific
Weighted average
Planning Reasoning
Math & Data Analysis
Coding Debugging
Training Prompts
Filtering
Creative
Information
Self-Instruct
None
41.0
51.6
43.3
31.4
47.7
38.0
Relf-RIP (without post- post-filtering) Self-Instruct with RIP post-filtering Self-R/P
None
44.8
55.3
46.9
33.5
49.7
44.6
RIP
44.1
54,8
47.3
36,4
48.2
40.3
RIP
42.5
54.1
46.2
32.8
48.6
38.2
Table 21: Results of pair selections: We report Armo scores on valid sets by varying different pairing methods instead of filtering prompts. Best pairing result 0. 1842 achieved with appointing response with bottom 25% score as rejected, although still underperforming compared our filtering method (8.1898).
Pair
Armo Score Valid
Chosen-Highesttcoorc Rejected==l.oeestScce
0.1830
ChosennHiihestSScore, Rgjoctedd-BottoomSccee25555
0.1842
Chosen==lighhestScone, Rejectedl-Bottonnecceee555
0.1839
Chosen= Rejected--BottooSScrrr
0,1821
Chosenn-Hiighessscooreeeeee Regireeooonn
0.1835
Chosen=llighessSoonne Rew.:/owooeneeeieeeeeeoooeeoooo RIP
0.1898
a random sample of 10,000 data points from WildChat. threshold of 2 means filtering out prompts with only tag. As shown in Table 28, a threshold of 2 2 yields the best performance using the InsTag Complexity filtering method. We reported the results for threshold of 2 in Table 2.
18

---

Rejecting Instruction Preferences (RIP)
Table 22: Results on Varying N = 8, 16, 32, 64 number of responses sampled per prompts in Response Generation:
Armo Score on Valid set of our DPO models trained with WildChat Dataset all increases after filtering based on RIP regardless of the choice of N in response generation step.
Before Filtering Armo Score
After Filtering Armo Score
Gain
N
0,1821
0,1860
0.0039
16
0,1827
0,1878
0.0051
32
0,1829
0,1882
0.0053
64
0.1831
0,1898
0.0067
Table 23: Self-RIP for generating high-quality synthetic instructions by varying number of fewshots. Self-RIP creates prompts using few-shot samples from high-quality prompts curated by RIP, whereas Self- belf-Instruct uses few-shots from unfiltered WildChat prompts. Applying RIP filtering after generation is also important, and achieves the best results, significantly outperforming Self-Instruct data.
AlpacaEval2
Sced Prompts
# Train examples
Train Prompts
LC Win
Win
Ilama 3.1 -.ttttttnttruct (seed model) WildChat-2Ok, + RIP
20.9
21.8
4538
57.8
57.2
Sclf-Instruct + RIP
20000
16261
58.3
3.2
Self-RIP (without post-fitering) Self-RIP (without post-filtenn filtering)
256
20000
50.0
51,2
4538
20000
53.6
56.1
Self-RIP Self RIP
256
15619
58.9
63.1
4538
18812
60.2
61.1
InsTag Diversity The InsTag Diversity filtering method (Lu et al., 2023) considers a dataset to be more diverse if it contains a larger number of unique tags, as annotated by the aforementioned tagger. We employed two metrics to manage InsTag Diversity:
1. Tag Frequency: We deem tag valid if it meets predefined frequency threshold. This approach addresses the issue of infrequent tags, such as "serve size" and "market failure," which appeared only once or twice in the entire Wildchat dataset, suggesting they may not represent valid categories. In contrast, more common tags like "creative writing" and "information retrieval" are more appropriate for categorizing prompt data.
Max prompt per Tag: This metric controls the coverage ratio of unique tags. a prompt contains only tags that have already been covered by the selected set, we discard the prompt to ensure diversity.
Table 29 presents the performance results when diversity is controlled using the two metrics described above. To ensure fairness, we downsampled the training data for each experiment to 10,000 samples. The results indicate that the model achieves optimal performance when the Tag Frequency is set to 6 and the Max Prompt per Tag is set to 3. This means we only consider tags that appear more than six times in the entire Wildchat dataset, and we allow maximum of three prompts per tag. The best performance results are reported in Table 2.
Perplexity To curate training prompts, we compute the perplexity (ppl) of the selected response yo using the Llama-3.1 1- In-Inuttuct model in a zero-shot setting. We use this perplexity as a filtering metric, specifically retaining examples with high (y (r) values, which may indicate more challenging prompts. We adjust the quantile range to control perplexity, calculating py (z) for 20,000 Wildchat data points and filtering them based on this range. Table 30 displays model performance across different ppl quantile ranges. As shown, the quantile range of 25-100 yields the best performance, and we report this model's performance in Table 2.
Fislloig Difficulty (IFD) Li al. (2023a) introduced the IFD to measure the model-specific difficulty of a data sample. In the instruction -urionnnttunnggggg process, the loss sample pair (Q, A) is calculated by continuously predicting the next tokens given the instruction Q and their proceeding words:
19

---

Rejecting Instruction Preferences (RIP)
Table 24: Performance of Different Filter Methods Across Quantile Ranges on WildChat with ArmoRM as reward anotator.
Method 0.18305
0-100
10-100
25-100
50-100
60-100
75-100
Chosen Reward
O. 0818225
0.18409
0. 0.18393
0.18380
0.18333
Rejected Reward
0.18305
18411
0,18405
O. 18566
0.18797
0.18795
Average Reward 0.18305
O. 08366
0.18392
0. 18494
0.18468
0. 0.18442
Chosen Length 0.18305
0. 08350
0.18366
0.18278
0.18226
0.18105
Rejected Length
0.18305
0. 08377
0.18340
0.18571
0.18593
0.18473
Method
0-100
0-25
0-50
50-100
Reward Gap
0.18305
0.18405
0. 18542
0.17993
Table 25: Performance of Different Filter Methods Across Quantile Ranges on HelpSteer2 valid set.
Method
0-25
0-50
0-75
0- 0-100
25-100
50-100
75-100
Chosen Human Reward
1469
0. 0.1451
0. 1456
0. 1458
0.1461
0. 0.1454
0. 0.1465
Rejected Human Reward
0. 1444
0.1455
0. 1454
041458
1,.4
0.1470
0.1461
Chosen Length
1484
0417
1455
0. 1458
1454
0. 0.1446
1449
Rejected Length
1.1421
0.1430
1445
1454
1,1499
0.1513
0.1478
Human Reward Gap
0.1482
0.1480
0. 3.1466
0.1458
0.1448
0. 0.1448
0.1441
Table 26: Results on Applying RIP individual metric. shown below, applying each individual RIP metric all yields better performance compared no filtering. In addition, apply all 3 metrics outperforms filtering with individual metric.
Valid Armo Score
Alpaca LC Winrate
Filtering Metric
No Filtering
0.1830
48.37
Rejected Armo
0.18979
56.91
Rejected Length
0.18593
53.31
Reward Gap
0. 1.58222
51.01
Apply all
0. 0.18983
57.83
Table 27: Results on Applying RIP metrics accumulatively These findings from our SimPO experiments consistent with our previous DPO experiments, which demonstrated that Rejected Armo is the most effective metric. The addition of rejected length also proved to be highly effective, while gap filtering provided some benefits, albeit to lesser extent than the other two metrics.
Alpaca LC Winrate
Alpaca LC Winrate
Filtering Metric
# Prompts
Llama3,1-88 1-8b SimPO (no filtering)
19803
$1.28
40.55
Llama3.1-48 1-8b SimPO (RIP filtering, Rejected Armo
8068
54.02
43.51
Llama3.1 1-bb SimPO (RIP filtering, Rejected Armo, Gap)
6629
53.04
43.02
Llama3. 1-8b SimPO (RIP filtering, Rejected Armo, Rejected Length, Gap)
4538
53.32
43.81
1 N
log P (w Q, @1 W wa ....... wi...... w
(I)
where N the number of words of the groundtruth answer A. They denote this averaged crossentropy loss as the Conditioned Answer Score (A Q) = (A Q).
Then they introduce the Direct Answer Score (A)
20

---

Rejecting Instruction Preferences (RIP)
Figure 3: GPT4 eval prompt.
I have collection of prompts that need to evaluate for their effectiveness in fine-tuning language model.
useful prompt should:
Clearly ask question
Be concise and specific
Directly relate to the topic of interest follow given instructions
Please assess each prompt and assign a score from to based its usefulness:
1: Pretty useful
2: Somewhat useful
3: Neutral (neither useful nor harmful)
44: Somewhat harmful
5: Harmful
Make sure to clearly indicate the score at the end of your evaluation using the format: Score: X Prompt: {prompt}
Figure 4: Self-Instruc few-shot prompt template.
Below are sample tasks from user.
1. <begin>>< (INSTRUCIITTTTTOIONN } </end>
2. <begin>> <bcggi>><INNTEEI
(INSTRUCTION
</end>
<begin>> INSTRUCTION 3)
</end>
<begin>+
INSTRUCTION 4
</end>
<begin.> INSTRUCTION 5
</end>
6. <begin>>
INSTRUCTION 6
</end>
<begin>
INSTRUCTION 7
<lend>
<begin>>
(INSTRUCTION
<lend>
Come with series of new tasks, wrapped with <hegin.>and <lend>
9.
N N
(A)
log (we wA.... 1
(2)
Finally, they estimate the Cnttruction-Foloovii Difficulty (IFD) scores IFD, on following instruction of given (Q. A) pairs by calculating the ratio between Sa(A) and (A Q)::
(A Q)
IFD;(Q.A)
(3)
Sp (A)
We calculated the IFD scores for 20,000 Wildchat data points and filtered them based on specific ranges, As shown in Table 31, filtering with range of 25-100 yielded the best performance. The performance of this model is reported in Table 2.
21

---

Rejecting Instruction Preferences (RIP)
Table 29: Model performance with InsTag Diversity Filtering
Table 28: Model performance with InsTag Complexity
Tag Fre- quency
Max prompt per Tag
Armo Reward
Filtering
on Valid Set
Tag threshold
Armo Reward on Valid Set
0.1809
1
2
0.1799
>= 1
0.1820
2
I
0.1798
>= 2
0.1826
2
2
0.1800
3
0.1812
3
0.1799 2
I
>=
0.1815
3
0.1797
>= 5
0.1818
3
3
0.1814
4
3
0.1821
5
3
0.1818
6
0.1831
Table 31: Model performance with IFD Filtering
Table 30: Model performance with Perplexity Filtering
Quantile Range
Armo Reward on Valid Set
Quantile Range
Armo Reward Valid Set
0-25
0,1815
25-100
0. 081833
0-50
0,1823
50-100
0,1827
0-75
0.1832
75-100
0.1797
25-100
0,1835
Figure 5: t-SNE plots on instructions before and after filtering by rewards and lengths of rejected responses. Red dots represent unfiltered instructions, while blue dots are instructions curated by filtering out those with low-reward and shorter rejected responses.
cluster
Figure 6: t-SNE plots on instructions before and after filtering by reward gaps. Blue dots represent instructions filtered only by rejected response, while yellow dots are instructions curated with smaller gap.
cluster
1:00
-10
49
22

---

Rejecting Instruction Preferences (RIP)
Table 32: Noisy instruction filtered based on rejected responses of lower scores and shorter lengths. We expand clusters of instructions highlighted in Figure 5 for better understanding of what instructions are being filtered out by measuring quality of rejected responses.
Cluster
Description
Rejecting Reason
Rejected Instruction
Cluster
646 instructions in the format of "give me response to ""<text>" to send in dis- cussion, VERY SHORT, CON- CISE CLEAR. ONLY RE- TURN THE RAW MESSAGE, DO NOT SAY "Hey here is the message you asked"", where <text> refers single-turn conversational message.
Around 90% of rejected responses are of shorter lengths and lower scores be- low 25% percentile, even though their scores are higher than average rejected scores. These short and concise conver- sational responses are shorter thus poten- tially more generic and less informative for the models further improve upon.
give me response to "'Im feel- ing great! Swimming around in the occan and hunting for prey never gets old. I'm always look- ing for new and exciting ways to keep busy." to send in a dis- cussion, VERY SHORT, CON- CISE & CLEAR. ONLY RE- TURN THE RAW MESSAGE, DO NOT SAY "Hey here the message you asked"
Cluster
804 instructions in the format of movie script: (In a <scene>> <namel>; linel>> >n.... Cnamek>:s :<lnnek>>>>>> without any instructions on what the model response should be.
Short rejected responses and low re- jected scores: Around 90% of rejected responses of shorter lengths, lower scores below 50% percentile. In addi- tion, over 75% response pairs are of larger score gap above 50% percentile. All of these are likely due to the obscu- rity of the instructions.
(In the school literature club- room..)) (.nm.........MMMooo.. M.niika: Natsuki. where everyone? haven't seen Sayori, Yuri, MC while, Naaasuui....
2
Cluster
279 instructions, majority of them purely excerpts from a fictional story, with no speci- fications on what the response should be. Users could be ask- ing models continue the story, or summarize, or edit it.
Short rejected responses and low jected scores: Over 90% of rejected responses are of shorter lengths, lower scores below 50% percentile. All of these are likely due to the obscurity of the user instructions.
David insists he is too strong- willed and intelligent hypnotized. He scoffs at the very idea. In this kinky script. his colleague Clare eas- ily proves him wrong, in front some amused co-workers.
3
Cluster
466 instructions, majority of are about writing comedic story about a fictional character.
Short rejected responses and low jected scores: Around 95% of rejected responses are of shorter lengths, lower scores below 25% percentile. All of these likely due to the Llama 3.1-8B- Instruct model being reluctant to provide detailed answers. These instructions are therefore less informative for improving Ilama 3.1-8B. ---B--Instrue with its own re- sponses.
Make story about Shrek the buff farting in bog water, then collecting all the fish the smell kills and cating them for dinner.
23

---

Rejecting Instruction Preferences (RIP)
Table 33: Noisy instruction clusters filtered based rejected responses of lower scores and shorter lengths. We expand 4 clusters of instructions sampled from Figure 6, that consists of both rejected and accepted instructions by RIP.
Cluster
Description
Rejecting Reason
Rejected Instructions
Accepted Instructions
Cluster
140 instructions, among those some purely code snippets without ad- ditional guidelines on what to respond. Others are requests asking optimize given piece of code.
Instructions with pure code snippets lead variable re- sponses (from code refactoring, editing, code completion, to code review and code explanation using natural guage). Instructions on "improve this code" can also incur variable responses given lack of more specified instructions.
improve this emer- geney shutdown code: import nimport primorr nimport crets thread- ing import Thread, nnrom рyn- put mouse Listener's fnform pyn- nut.keybordd..
will provide you disas- sembly from a computer game that runs in MS-DOS. written in C with Watcom compiler. Some library function calls are already identified. Ex- plain the functions give to you and suggest names and C-language function signa- tures for them, including which parameters map to which registers stack val- ues ;eode snippeti....
The game
se-
import
Cluster
237 instructions including: writing program, inquiry about online tool, software installa- tion, etc. Many instructions are short (in 120 charac- ters) and relatively high-level.
Rejected responses are on average much longer and complex compared chosen responses, despite the high scores of both chosen and rejected responses.
alignment in excel vb.net
am getting access denined when to put local files into remote server using ftp how can resolve this issuc
2
write script for del- egating fb group
Cluster
Cluster are 52 in- structions related to hypothetical or sur- real scenarios: Clus- ter 52 instruc- tions in the form of "Freedom planet ...". possibly for a cre- ative project in the video game.
Model responses vary a lot due to the obscurity or hypothetical nature of the instructions.
Can You Imagine Fictional Versions Of Silicon Valley During 1940 In Detail?
What if Cartoon Network Made The Amazing World of Gumball: Next Genera- tion
& 4
Freedom and Madness combat all characters: Hank 4th wall breaks and repetition
planet
freedom planet what Lord Brevon Wins (not kils Lilac, Carol and Milla)
24

---

Rejecting Instruction Preferences (RIP)
Figure t-SNE plots on instructions before and after filtering by rewards and lengths of rejected responses. Red dots represent unfiltered instructions, while blue dots are instructions curated by filtering out those with low-reward and shorter rejected responses. Both Cluster and consist of instructions curated or filtered the RIP metrics.
Cluster 5 consists of 307 prompts filtered out and 87 prompts selected; 282 prompts being filtered out due to shorter rejected response length. Short responses are either because the requests are underspecified or because they elicit potentially sensitive responses. Sample Rejected Instruction from Cluster 5 "I want you to help me with my research"; "Write one more short song, about Izzy's 's hatred for Joe Biden". Sample Accepted Instruction from Cluster 5: "How to comfort someone who studied for a test and got different questions than the ones he studied for"; "Lyrics for a happy song about challenges and growth in the style of The Weeknd".
Cluster consists of 385 prompts filtered due to shorter rejected responses and 218 prompts selected. Prompts leading to short rejected responses in this cluster generic chitchat messages, greetings, easy factual questions. Sample Rejected Instruction from Cluster 6; "What is the weather today in Seattle", "Do you speak Vietnamese". Sample Accepted Instruction from Cluster 6: "Hi, can you give me simple party game for 4 10 people"; "Benefits of studying in Singapore".
Cluster 5
lower quality prompts
higher quality prompts
100
Cluster 6
E-SNE Dimension
-50
E-SNE Dimension
25


Reviews:
[{"review_id": "DLl1ohLrAL", "reviewer": "Reviewer_D5vn", "summary": "The authors propose RIP, a data filtering method that leverages 3 criteria (rejected response length, rejected response reward, and reward gap) in assessing prompt quality in the context of preference fine-tuning. They find benchmark improvements of ~10% compared against non-filtered DPO. Furthermore, the authors propose Self-RIP, a synthetic data generation scheme in which few-shot examples are selected via RIP.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. >Lack of ablations and clarity surrounding the effectiveness of each of the 3 criteria individually.\n\nDue to the paper's length constraints, we have included an ablation study in the appendix (Table 19 and 20, line 990). In this study, we conducted data filtering experiments using each criterion individually, including filtering based on chosen reward, rejected reward, average reward (between chosen and rejected), chosen length, rejected length, and gap. We reported the results on our validation set.\n\nAs shown in the tables, when applying individual filtering criteria, we found that rejected reward is the most effective criterion, followed by rejected response length. The gap criterion also provided some improvement, although it was not as effective as the other two criteria. To further validate these findings, we tested the individual criteria on the Alpaca evaluation.\n\nFiltering | Reward on valid set | Alpaca Eval lc-winrate\n----------|--------------------|------------------------\nNo filtering | 0.18305          | 48.37\nRejected Armo | 0.18979         | 56.91\nRejected Length | 0.18593        | 53.31\nGap            | 0.18542         | 51.01\nMix them all   | **0.18983**     | **57.83**\n\n2. >Lack of novelty: beta-DPO also filters based on gap.\n\nWe acknowledge that beta-DPO also employs gap-based filtering, which may raise concerns about the novelty of our approach. However, as evident from Table 19 and Table 20, our primary focus lies in the effectiveness of Rejected Reward and Rejected Length as filtering criteria in addition to reward gap, which outperform the gap-only-based criterion. \n\n3. >beta-DPO filtering as baseline. \n\nThank you for the suggestion. We acknowledge that beta-DPO also employs gap-based filtering, which we cite in our paper. However, there are three key differences between their approach and ours:\n\n(a). Online vs. Offline Filtering: Beta-DPO's filtering is online, meaning they filter out data in every batch, whereas our approach filters data offline. This offline filtering enables more flexible and efficient generation pipelines, particularly for weak-to-strong generation scenarios. For instance, finetuning Llama3.3-70B-Instruct on prompts RIP filtered by a smaller Llama3.1-8B model outperformed (Alpaca LC-winrate improved from 54.3 to 64.5, Arena-Hard from 70.5 to 76.7).\n\n(b). Gap Size Thresholds: Unlike beta-DPO, which removes both small and large gaps, our method removes bigger gaps only.\n\n(c). Probabilistic vs. Deterministic Filtering: Beta-DPO's filtering is probabilistic, resulting in incomplete data removal, whereas our approach uses deterministic filtering to ensure thorough removal of unwanted data.\n\nGiven these differences, and as previously mentioned, our method prioritizes Rejected Reward and Rejected Length criteria over gap-based filtering, which have demonstrated superior effectiveness in our experiments. Consequently, when submitting our draft, we did not include a direct comparison with beta-DPO's filtering results. However, we appreciate your suggestion and have since conducted additional experiments to evaluate beta-DPO filtering on our experimental setting:\n\n\n| Filtering       | BetaDPO mode_weight | # training samples | Reward on valid set | Alapaca Eval LC-winrate | Alpaca Eval winrate |\n|-----------------|--------------------|--------------------|--------------------|------------------------|---------------------|\n| No filtering    | -                  | 19803              | 0.18305          | 48.37                  | 45.87               |\n| RIP filtering   | -                  | 4538               | **0.18983**      | **57.83**              | **57.16**           |\n| BetaDPO filter  | 0.2                | 15842              | 0.18417          | 49.15                  | 49.00               |\n| BetaDPO filter  | 0.5                | 9901               | 0.18399          | 46.68                  | 42.41               |\n| BetaDPO filter  | 0.75               | 4950               | 0.18265          | 45.97                  | 40.58               |\n\n\nThank you for diligently reviewing our work. We hope that we have thoroughly addressed all of your questions and concerns. Furthermore, we conducted additional experiments to strengthen our paper, including performing extra reasoning tasks, expanding our model suite beyond Llama by finetuning a Gemma-based model and two other reward models (GRM-Gemma2-2B-RewardModel-FT and INF-ORM-Llama3.1-70B), and applying RIP with SimPO.", "claims_and_evidence": "The key claims lie in the effectiveness of the 3 criteria used in filtering. For the reward gap, the authors cite Wu et al. (2024a) on line 139 which establishes a small reward gap being more informative. The use of rejected response length and rejected response reward seem reasonable.", "methods_and_evaluation_criteria": "The evaluation metrics are standard and criteria seem reasonable.", "theoretical_claims": "The work is entirely empirical.", "experimental_designs_or_analyses": "One of my primary concerns is missing ablations. It is unclear to me the extent to which each of the 3 provided criteria and responsible for the performance gains reported. How strong are the results when only one criteria is used at a time? Table 9 provides some incomplete insights in this direction. In that table, I see that for when fine-tuned on Wildchat 20k the performance boost provided by the reward gap filtering appears smaller than that of rejected response length + rejected response reward.", "relation_to_broader_scientific_literature": "The novelty of the paper seems rather limited. With regards to the reward gap, Wu et al. (2024a) also establishes the use of the reward gap for data filtering. They also experiment with dynamically adjusting $\\beta$ in the fine-tuning process but one of their baselines is solely using reward gap filtering (using 3$\\sigma$) with fixed $\\beta$ which appears similar to the authors' approach. This baseline was not compared against and is my second primary concern.", "essential_references_not_discussed": "The related works and preliminaries section do a good job of going over relevant works.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please address my primary concerns listed in other sections of my review:\n\n1. Lack of ablations and clarity surrounding the effectiveness of each of the 3 criteria individually.\n2. Missing baseline of \"DPO + Data Filtering\" from Wu et al. (2024a) for which the reward gap filtering criteria appears very similar.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Uggkq0iqc6", "reviewer": "Reviewer_weWw", "summary": "The paper introduces a novel data curation method called Rejecting Instruction Preferences (RIP) designed to improve the quality of training data for large language models. The core idea is to filter out low-quality prompts by examining paired model responses. Experimental evaluations on benchmarks such as AlpacaEval2, Arena-Hard, and WildBench demonstrate that models trained with RIP-filtered data (both human-written and synthetic) achieve significant improvements over unfiltered datasets and other baseline filtering methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting our strength on the novelty of RIP filtering and its significant improvements as compared to traditional prompt-based filtering methods.\n\n> 1.  Could the authors provide a more detailed explanation regarding why the rejected response length is chosen as a filtering metric? Is this measure specifically intended to support Hypothesis 2, which suggests that low-quality prompts produce a broader variance in responses?\n\nWe thank the reviewer for offering the opportunity for us to clarify our hypothesis. \nWe cited several studies in our paper Line 124 (e.g. [1]) showing correlation between response length, response quality and final performance. Given Hypothesis 1 “Low-quality prompts are likely to produce low-quality responses”,  we thus select the length of the lowest-scored responses (a.k.a the rejected response) as one of the filtering metrics, in addition to rejected response score to measure quality of the rejected response. While rejected response length might also be correlated with response variance, we consider a more straightforward metric, the reward gap between chosen and rejected responses, to capture variances in our hypothesis 2.\n\n\n> 2. \"comprehensive case studies or explicit examples that directly compare samples filtered out by this criterion with those that are selected.\"\n\nWe thank the reviewer for pointing out the importance of detailed illustrations to justify our hypothesis. We have included such analysis in the paper appendix due to page limits.\n\nIn Table 26, we summarized 4 clusters of prompts in Figure 5 t-SNE plot that are being filtered out due to large response variance. For each cluster, we manually go over the 50 ~200 instructions, and summarize patterns of those rejected instructions in the “Rejected Reason” column. We further include “side-by-side” comparison between sample instructions filtered out by this criterion with those being selected in “Rejected Instructions” and “Accepted Instructions”, to further illustrate the hypothesis 2.\n\nIn Table 25 we extract 4 clusters of prompts from Figure 4 t-SNE plot that contain thousands of prompts filtered out due to low quality rejected responses. For each cluster, we count the percentage of prompts filtered out due to shorter length (below the rejected response cutoff). We also summarize their patterns in the “Description” column and list sample rejected instruction in “Rejected Instruction” column. Those 4 clusters we investigated are predominantly prompts filtered out by RIP (barely any survived prompts). To further address the reviewer’s question on side-by-side comparison, we included 2 other clusters that contain both selected and filtered prompts (See plot in https://anonymous.4open.science/r/projects-37B8/tsne.jpg)\n\n| Cluster  | Description | Rejected Instructions | Accepted Instructions |\n|----------|------------|----------------------|----------------------|\n| **Cluster 5** | 307 prompts filtered out and 87 prompts selected; 282 prompts are being filtered out due to shorter rejected response length. Short responses are either because the requests are underspecified or because they elicit potentially sensitive responses. |  \"I want you to help me with my research\"; \"Write one more short song, about Izzy’s hatred for Joe Biden\" |  \"How to comfort someone who studied for a test and got different questions than the ones he studied for\";  \"Lyrics for a happy song about challenges and growth in the style of The Weeknd\" |\n| **Cluster 6** | 385 prompts filtered out due to shorter rejected responses and 218 prompts selected. Prompts leading to short rejected responses in this cluster are generic chitchat messages, greetings, or easy factual questions. |  \"What is the weather today in Seattle\"  ; \"Do you speak Vietnamese\" |  \"Hi, can you give me a simple party game for 4~10 people\"; \"Benefits of studying in Singapore\" |\n\nIn addition to visualizing the examples, we also conduct GPT4 analysis into quality of the filtered out prompts in Section 6.2 by each criterion, to justify our hypothesis. We also add human eval on 100 examples, due to length limit, we add one example here\n\n| **Prompt** | **RIP Filtering Results** | **Human Eval** |\n| --- | --- | --- |\n| Write a story | Filter out (rejected ARMO, gap) | Not Useful. Explanation: This prompt is overly broad and lacks specific details, posing challenges in generating a focused response. |\n\n\n\nWe will add more of those analysis in the appendix with side-by-side comparisons to illustrate our RIP filtering criteria. We hope these further analysis, in addition to our Table 25~26 will help address the reviewer’s clarification question. \n\n\n[1]  Zhao, H., etc, N. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "None", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Finding and results.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths: The use of rejected response quality metrics (reward and length) along with the reward gap to assess and filter prompt quality is a novel contribution. This pairwise evaluation provides a fresh perspective compared to traditional prompt-based filtering methods.\n\nWeakness: \nCould the authors provide a more detailed explanation regarding why the rejected response length is chosen as a filtering metric? Is this measure specifically intended to support Hypothesis 2, which suggests that low-quality prompts produce a broader variance in responses? Moreover, it would be highly beneficial if the paper included comprehensive case studies or explicit examples that directly compare samples filtered out by this criterion with those that are selected. Such detailed illustrations or side-by-side comparisons would help clarify how effectively the rejected response length differentiates between lower-quality and higher-quality prompts, thereby providing a clearer justification for its inclusion in the filtering process.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eaNUGYfcIV", "reviewer": "Reviewer_m81Y", "summary": "This paper introduces a method for filtering prompts used for preference-tuning (in this case, DPO), RIP. The method simply filters preferences based on reward, output length, and gap between chosen and rejected responses. Experiments training on datasets filtered by this method shows improvements in llm-as-a-judge based metrics over either doing no filtering or using alternate data filtering techniques. Qualitative analysis suggests the method is primarily effective in filtering out noisy and lower-quality prompts that do not elicit great responses.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. >One caveat is that the claims are very much scoped to alignment performance on llm-judge benchmarks, which is a very particular domain.\n\nThank you for bringing this to our attention. We acknowledge that in our draft, we demonstrated the effectiveness of RIP on alignment performances in general instruction-following tasks. To further validate its capabilities, we conducted additional testing on reasoning domains. Here are the results:\n\n\n| Model | GPQA-diamond | MMLU_PRO | AVG |\n| --- | --- | --- | --- |\n| Baseline DPO | 12349 | 33.5 | 52.7 | 43.1 |\n| DPO + RIP Filtering | 362 | 35.2 | 51.0 | 43.1 |\n\n\nWe used our method to filter *science reasoning 15k data* [1] with inf_orm reward model (https://huggingface.co/infly/INF-ORM-Llama3.1-70B). As you can see that with our RIP method, we successfully filtered out over 90% of data, however, the model performance remains the same on reasoning benchmarks. \n[1] Yuan et al. NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions\n\n2. >It would still be useful to validate results with human annotators instead of entirely relying on model-as-judge results.\n\nWe agreed that mixed human evaluation is the preferred approach. However, due to cost constraints, we limited our human evaluation to 100 examples. The results showed our RIP successfully filtered out 88% of the noisy data identified by human evaluators. We will provide additional details in the appendix. \n\n3. >Does model performance and/or behaviour when dealing with such ambiguous prompts change?\n\nThis is a thought-provoking question! We evaluated our method on three widely used benchmarks: Alpaca Eval, Arena-Hard, and Wildbench. Notably, Wildbench utilizes data sourced from real users in chatbot-like scenarios, which provides a more realistic and representative testbed for our approach. \n\n4. >The authors use 3 elements for RIP (response reward, response length, reward gap), but do not evaluate and ablate each component\n\nWe performed an ablation study on each component in Tables 19 and 20. Specifically, we trained Llama3.1-8B on data filtered by each component individually and evaluated the trained model on our validation set (reporting Armo reward on the validation set). This is our standard approach for checkpoint selection.\nThe numbers in Tables 19 and 20 represent ArmoRM scores on the valid set, which are close to each other due to the use of Armo scores distribution. Although the differences may seem small (e.g., 0.01), they correspond to significant performance differences.\nWe only tested performance on the validation set initially, as running model evaluations on three benchmarks (Alpaca Eval, Arena Hard, and Wildbench) is time-consuming, given their reliance on the GPT4 API. However, we recognize that this may make it challenging for readers to interpret the numbers in Table 19. To address this, we further tested the best checkpoints within Table 19 on Alpaca Eval  (see our comment to Reviewer D5vn).\n\n5. >The authors do not explore models beyond llama models\n\nTo show the effectiveness of our RIP filtering beyond Llama models, we:\n\n(1). Finetune Gemma2-9B-it model with SimPO using the dataset (princeton-nlp/llama3-ultrafeedback-armorm) which are Gemma2 generations on ultrafeedback annotated by ArmoRM.  Applying RIP on Gemma2-9B finetuning further improves Gemma2 performance on AlpacaEval from 69.48 to 73.81 by filtering out 50% train data. \n\n\n**Train Size Comparison**\n| Model | Train Size | AlpacaEval LC Winrate | Alpaca Winrate |\n| --- | --- | --- | --- |\n| Gemma2-9B SimPO (no filtering) | 59569 | 69.48% | 63.07% |\n| Gemma2-9B SimPO (RIP filtering) | 29963 | 73.81% | 62.01% |\n\n(2). Finetune Llama3 Model using a Gemma-2-2b based reward model (see our comment to Reviewer eddG).\n\n6. >You might imagine that filtering on length is less effective for a length-normalized method?\n\nTo further validate our approach, we tested our method on SimPO, a well-known length-normalized variant of the DPO algorithm.\n\n**Filter Metrics Comparison**\n| Model | # of Training Samples | AlpacaEval LC Winrate | Alpaca Winrate |\n| --- | --- | --- | --- |\n| Llama3.1-8b SimPO (no filtering) | 19803 | 51.28% | 40.55% |\n| Llama3.1-8b SimPO (RIP filtering, Rejected Armo) | 8068 | 54.02% | 43.51% |\n| Llama3.1-8b SimPO (RIP filtering, Rejected Armo, Gap) | 6629 | 53.04% | 43.02% |\n| Llama3.1-8b SimPO (RIP filtering, Rejected Armo, Rejected Length, Gap) | 4538 | 53.32% | 43.81% |\n\nThese findings from our SimPO experiments are consistent with our previous DPO experiments, which demonstrated that Rejected Armo is the most effective metric. The addition of rejected length also proved to be highly effective, while gap filtering provided some benefits, albeit to a lesser extent than the other two metrics.\n\nWe appreciate your feedback and thank you for the opportunity to strengthen our paper. We hope our comments have addressed your concerns and questions, and we look forward to your further consideration of our work.", "claims_and_evidence": "I think that the overall claim that RIP improves performance on llm-as-judge evaluations compared to no filtering or baselines is reasonably well supported, with lots of baselines and multiple evaluation settings considered. The self-RIP results are also good. One caveat is that the claims are very much scoped to alignment performance on llm-judge benchmarks, which is a very particular domain (in contrast to e.g., reasoning tasks, which are not examined in this work). This scoping is mentioned in the conclusion but not the introduction.", "methods_and_evaluation_criteria": "The benchmarks chosen are reasonable, it would still be useful to validate results with human annotators instead of entirely relying on model-as-judge results. I understand these benchmarks have strong correlations with human preferences, but it would still make me more confident that the gains are useful if human evaluation agrees.", "theoretical_claims": "This is a primarily empirical work, and the mathematical explanations where present seem correct.", "experimental_designs_or_analyses": "The use of these benchmarks while filtering with reward models also makes me wonder if there are some implicit assumptions around what sort of downstream queries the models will be used for. My understanding is that the benchmarks examined all have fairly clear, well-written questions, and so filtering out noisy prompts likely also reduces the data to prompts more similar to the downstream evals. But does model performance and/or behaviour when dealing with such ambiguous prompts change? Is this a potential concern if the aim is an LM chatbot-like application, which would likely receive such ambiguous queries? I don’t think there are great benchmarks for this now, but it would be interesting to analyse (or perhaps I have missed something in the appendix!)\n\nThe authors use 3 elements for RIP (response reward, response length, reward gap), but do not evaluate and ablate each component, nor show how the hyperparams chosen affect results in a systematic way - Tables 19 and 20 seem to do this, but I’m not sure on the setup. Are those results from models trained on data filtered with the given threshold using an LM judge? Reward score performance? Why are all the scores so close together, and are differences < 0.01 actually significantly here?\n\nThe authors do not explore models beyond llama models, so it is unclear how well results may generalize to other LMs. While they claim that looking at Llama 3.1 and 3.3 counts as different bases, my understanding (based on the HF metadata, which may not be correct) that Llama 3.3 models are still ultimately LMs from the Llama 3.1 family, just with new finetunes/post-training. Results from models from entirely different organisations and/or clearly known to have different pretraining mixes would be useful.\n\nFinally, I also wonder if examining DPO-likes would be useful? For example, work has found length-normalized DPO to be more effective [1]. You might imagine that filtering on length is less effective for a length-normalized method? I understand this explodes the experimentation space, though.\n\n[1] Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training. 2024.", "relation_to_broader_scientific_literature": "Filtering preference data and investigating methods to do so is still a relatively under-explored and interesting area for work!", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths:** The method is simple and I think the llm-judge experiment are carefully done and well-designed, with many reasonable baselines explored. The results themselves seem very strong, with quite large gains.\n\n**Weaknesses:** As noted above, I feel that it would be good to more thoroughly ablate the components of RIP, and explain the ablations in some more detail. Additionally, the paper clearly implicitly is aiming for ‘user alignment’ as a downstream target task, but this is not explicitly mentioned in the intro. This limits the applications of the method without further study, e.g. would RIP be useful for e.g. improving mathematical or reasoning performance (which preference tuning has been shown to be useful for)? How might filtering with RIP affect such performance?", "other_comments_or_suggestions": "- I don’t quite get the final sentence of the caption of table 8: “RIP outperforms the baseline of LLM-as-judge as the reward annotator.” What baseline in table 8 is using the llm-as-judge as a reward annotator? The only baselines are no filtering and the base model.\n- I would also caution against using red and green colours in tables 19/20 to accommodate red-green colourblind readers.", "questions_for_authors": "Please see my comments in “Experimental Designs Or Analyses” and “Other strengths and weaknesses”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eSjqxk67eQ", "reviewer": "Reviewer_eddG", "summary": "The paper proposes a recipe for the prompt selection problem in the pairwise preference optimization setting in RLHF called RIP filtering. The proposed method utilizes an external reward model to grade multiple completions of the candidate model and use several metrics of this random completion set as a guideline to filter the instruction prompts. The filtering procedure is based on two hypotheses that promote selecting instructions with completions with high reward and low reward variance. Extensive empirical studies are conducted to verify the effectiveness of RIP filtering.\n\n## update after rebuttal\nThe authors provided interesting experimental results that shows the proposed method works even when the reward model is chosen as a moderately strong model. I think the results are interesting and further justifies the applicability of the proposed method, therefore I have increased my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions.\n\n1. >effectiveness of RIP if reward models are of smaller size, and from a different model family\n\nThank you for this insightful feedback. To address the reviewer’s question, we select a lightweight non-Llama-based reward model “Ray2333/GRM-gemma2-2B-rewardmodel-ft” (https://huggingface.co/Ray2333/GRM-gemma2-2B-rewardmodel-ft), which is a gemma2-2B based reward model, to annotate and then DPO finetune a Llama3.1-8B-Instruct model. \n\nBelow are the results on RIP filtering using reward scores by this Gemma2-based RM. By curating less than 5k out of 20k prompts, we can improve Llama3.1-8B-Instruct DPO models from 41.1 to 49.9 on AlpacaEval LC-winrate, showing similar improvement using ArmoRM filtering (LC winrate improved from 48.4% to 57.8%). \n\n\n| Model | Train Data Size | Alpaca LC Winrate (%) | Alpaca Winrate (%) |\n| --- | --- | --- | --- |\n| WildChat20k baseline | 20000 | 41.1 | 47.3 |\n| WildChat RIP | 4401 | 49.9 | 53.5 |\n\nRay2333/GRM-gemma2-2B-rewardmodel-ft (ranked 36th on Reward Bench) is ranked below ArmoRM on RewardBench, and the performance gap in the two reward model quality also affects the performances of finetuning Llama3.1-8B-Instruct with reward model annotations(i.e. better-quality reward model leads to better winrate of finetuned models). However, in both cases RIP filtering demonstrates its effectiveness.\n\nMoreover, In the paper, we show the effectiveness of RIP filtering using various reward signals (ArmoRM, LLM-as-a-Judge, human), one of which is to use human annotated rewards from HelpSteer2 dataset as filtering criteria. We show in Table 4 that RIP filtering using *human rewards* also improve LLama3.1 winrates across all 3 benchmarks. In addition, Table 20 (performances on valid set when filtering using a single criteria), highlights that curating prompts of smaller human reward gaps boost performance.\n\nWe really appreciate the reviewer’s feedbacks, and hope that these experiment results can not only address the reviewer’s question, but further strengthen the effectiveness of RIP filterings under various reward model quality.\n\n2. > In table 9, I found several entries listing a lower bound of the reward gap, i.e., GAP > 0.042 in WildChat datasets. Does that contradict with the \"smaller GAPs work better\" hypothesis?\n\nWe really appreciate the reviewer for pointing out the typo. It should be GAP < 0.042 instead of >, since we are filtering out prompts with larger reward gaps. We will correct these typos in our updated version.", "claims_and_evidence": "From experimental evaluations along with ablations, the two hypothesis stated in section 3.2 are well supported by empirical evidence.", "methods_and_evaluation_criteria": "Yes. The evaluation procedure is sound.", "theoretical_claims": "Not applicable. There are no theory claims in the paper.", "experimental_designs_or_analyses": "Yes. The experimental design is valid and clear. I don't see any apparent issues therein", "relation_to_broader_scientific_literature": "- Efficient data selection is a critical problem for developing language models. The paper offered insights into the data selection procedure in RLHF which is valuable for the community.\n- The paper hints that distributional characteristics of the instruction-conditional distribution $p(\\cdot | x)$ could be efficiently exploited to derive selection procedures, which might further inspire future research.", "essential_references_not_discussed": "I am not an expert in the filed of RLHF. I do not recognize any important references that is not mentioned in the paper.", "other_strengths_and_weaknesses": "The paper is well-written.", "other_comments_or_suggestions": "See my questions section.", "questions_for_authors": "As the RIP procedure is based on an **external reward model** and one key hypothesis in the paper is that **smaller reward gaps indicates better instructions**, I am curious about **to what extent the effectiveness of RIP depends on the quality of the reward model itself**. In the paper the authors used ArmoRM and LLama 3.1-405B as reward models, which both originated from the LLama family. Therefore my questions are:\n- What will happen to the effectiveness of RIP if the reward model are chosen from a very different model family, or even from a lightweight model of smaller size?\n- In table 9, I found several entries listing a lower bound of the reward gap, i.e., ``GAP > 0.042`` in WildChat datasets. Does that contradict with the \"smaller GAPs work better\" hypothesis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: My decision is to accept the paper.

The paper introduces a prompt filtering protocol, RIP, for prompts used in instruction tuning models via prompt optimization. The filtering protocol filters prompts based on rejected response reward, rejected response length, and reward gap between preferred and rejected responses. The authors perform a wide range of empirical experiments, comparing tuned model performance across different filtering methods. Finally, the authors examine some potential explanations for the success of RIP.

There was some disagreement among reviewers about the overall score for this paper. Given that RIP has a 3-part filtering criteria, and that the reward gap criterion had been proposed before, there was some concern about ablations and novelty. The authors noted that the novel criteria presented in this paper seemed to contribute more to downstream performance, and expanded an existing ablation study to have a more interpretable scale (win rate). There were also questions about what signal the rejected reward and length criteria were picking up on, and the authors pointed to some qualitative analysis of filtered prompts based on these criteria. The authors also expanded experiments to reasoning/knowledge benchmarks, Gemma models, and different reward models and found similar results.

Overall, I agreed with the reviewers who supported accepting the paper.
Decision: Accept (poster)