Venue: NAACL 2025

# A Systematic Examination of Preference Learning

through the Lens of Instruction-Following

Joongwon Kim\({}^{\dagger\ddagger}\), Anirudh Goyal\({}^{\dagger}\), Aston Zhang\({}^{\dagger}\), Bo Xiong\({}^{\dagger}\), Rui Hou\({}^{\dagger}\), Melanie Kambadur\({}^{\dagger}\), Dhruv Mahajan\({}^{\dagger}\), Hannaneh Hajishirzi\({}^{\dagger\diamondsuit}\), Liang Tan\({}^{\dagger}\)

\({}^{\dagger}\)GenAI, Meta \({}^{\ddagger}\)University of Washington \({}^{\heartsuit}\)Allen Institute for AI

Work done during internship at Meta.

###### Abstract

In this work we systematically investigate how specific attributes of preference datasets affect the alignment and downstream performance of LLMs in instruction-following tasks. We use a novel synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with combinations of 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. With our synthetic prompts, we use rejection sampling (RS) and Monte Carlo Tree Search (MCTS) to obtain preference pairs. Then, we perform experiments investigating the effects of (1) the presence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the chosen, rejected responses and (3) the complexity of the training prompts. Our experiments reveal that shared prefixes provide marginal but consistent improvements and greater stability across challenging training configurations. While high-contrast preference pairs generally outperform low-contrast pairs, combining both often yields the best performance. Additionally, training on prompts of moderate difficulty leads to better generalization across different tasks. Our findings provide actionable insights into optimizing preference data curation for instruction-following tasks, offering a scalable and effective framework for enhancing LLM training and alignment.

## 1 Introduction

Aligning large language models (LLMs) with human preferences has remained a persistent challenge despite their recent success, particularly for tasks that involve generating nuanced, instruction-following responses. To address this bottleneck, **preference learning** has emerged as a vital technique applied in the final stages of LLM posttraining Stiennnon et al. (2020); Ouyang et al. (2022); Bai et al. (2022). Preference learning refines the ability of LLMs to align with human expectations by fine-tuning them on pairs of (chosen, rejected) responses. Recent successes of using preference learning to develop frontier language models OpenAI, 2023; Google (2023); Anthropic (2024); AI at Meta (2024) have led to automated methods for curating preference pairs Pang et al. (2024); Xie et al. (2024); Xu et al. (2023); Khaki et al. (2024). While these techniques yield synthetic preference pairs that significantly improve model capabilities in closed-ended tasks, it is yet unclear which attributes of the preference pairs contribute to the improved alignment and downstream capabilities.

Existing research in preference learning has largely focused on the mechanics of optimization methods, such as Direct Preference Optimization DPO, Rafailov et al. (2023)) and Proximal Policy Optimization (PPO, Schulman et al. (2017)). While these methods are critical for training models, they operate with limited insight into how the structure, quality, and complexity of the preference datasets themselves affect outcomes Ivison et al. (2024); Xiao et al. (2024). For example, questions remain about whether shared prefixes in paired responses improve learning, whether training on high-contrast pairs is always optimal, or how the difficulty of training prompts impacts generalization. Without a systematic investigation of these factors, designing effective preference datasets is largely heuristic and suboptimal.

In this work we seek to fill this gap by conducting a systematic investigation of how various attributes of automatically-curated preference datasets affect model performance. We approach this problem from the viewpoint of **instruction-following**Zhou et al. (2023); Wen et al. (2024); Zhang et al. (2024b), which are ideal for such analysis due to their complexity and capacity for integrating multiple constraints, giving us finer control over the data compared to other domains with binary correctness such as mathematics or tool

use (Hendrycks et al., 2021; Mialon et al., 2024). Furthermore, we focus on _verifiable_ constraints that can be assessed by code, which allows us to enable precision by deterministically evaluating the quality of any response with respect to the constraints, and scalability by requiring much less compute than open-ended constraints.

We first define 23 verifiable constraints spanning diverse requirements such as adherence to specific structural, stylistic, or formatting requirements, yet _distinct_ from those presented in IFevalZhou et al. (2023). These constraints form the foundation of our synthetic data generation pipeline, loosely inspired by INstruct-SkillMixKaur et al. (2024), which (1) proposes new general-purpose prompts, (2) assigns valid combinations of verifiable constraints along with the associated parameters and (3) generates new prompts that incorporate mixtures of the constraints in natural language over a wide variety of domains. Using this pipeline, we obtain a total of 48K unique prompts which contain mixtures of four, five or six verifiable constraints.

Using the synthetic prompts, we then apply two commonly-used methods for automatically curating preference pairs: rejection sampling (RS) and Monte Carlo Tree Search (MCTS). Rejection sampling presents a straightforward method to extract preference pairs for a given prompt by generating \(N\) independent responses with the policy model, scoring each response and using (high, low) scoring pairs of responses as the preference pairs Yuan et al. (2024); Khaki et al. (2024). On the other hand, Monte Carlo Tree Search presents a more complex method for extracting preference pairs - for a given tree obtained via MCTS, where each node represents the partial response generated for the given prompt, any pair of sibling nodes with (high, low) scoring pairs of responses are used as preference pairs Xie et al. (2024); Zhang et al. (2024). The RS approach is computationally efficient but there is no structure to the generated responses. Meanwhile, the MCTS approach is more resource intensive, but returns pairs of responses that share common prefixes and a more nuanced contrast in their remaining suffixes.

We use our synthetic prompts and the associated preference pairs to systematically investigate how different heuristics used for automatically curating preference pairs impact models' downstream performances. To this end, we focus on three critical dimensions that characterize a preference dataset:

1. **Shared prefixes in preference pairs**: Does structural consistency (e.g., common prefixes) between chosen and rejected responses improve learning?
2. **Contrast and quality of responses**: Is high-contrast or low-contrast pairing always superior, or does a mix of high- and low-contrast pairs offer better results?
3. **Difficulty of training prompts**: How does the complexity of training prompts affect the model's generalization ability?

Our findings reveal several actionable insights:

**1. Preference pairs with shared prefixes (MCTS) marginally outperforms preference pairs that do not (RS) consistently over different training configurations.** The performance of the MCTS-generated preference pairs is also more

Figure 1: Automatically curating preference pairs via rejection sampling (RS, left) and Monte Carlo Tree Search (MCTS, right). **RS**: We independently sample \(N\) different outputs from the policy, score each output with a verifier and take (high, low) scoring responses as the (chosen, rejected) pairs. **MCTS**: We perform tree search with the policy while generating multiple actions. Then, we use the rollouts from sibling nodes with (high, low) reward scores as the (chosen, rejected) pairs to obtain preference pairs with common prefixes up to the parent nodes.

 

stable across different training configurations than the RS-generated preference pairs - this stability is particularly valuable when response correctness is challenging to quantify.

**2. Having only high-contrast preference pairs is better than having only low-contrast preference pairs, but having a mixture of both often provides the best performance by balancing learning efficiency and diversity.** Our results also consistently indicate that the relative _contrast_ between the chosen and rejected responses have a greater impact than their absolute correctness.

**3. Training on moderately difficult prompts results in better generalization across evaluation tasks, including more complex ones.** Curating preference datasets with excessively challenging prompts returns a low yield rate of preference pairs due to the lower success rate of the model responses, and even given the same dataset size, the difficulty of the prompts tend to overwhelm the model and hinder the learning efficiency.

## 2 Background

**Preference Learning.** Preference learning is a technique that is used to align LLMs, involving pairs of (chosen, rejected) responses. Common methods for preference learning such as Direct Preference Optimization (DPO, Rafailov et al. (2023)) directly optimize the model to update its parameters to increase the likelihood of generating the chosen response over the rejected response. Meanwhile, other methods such as Proximal Policy Optimization (PPO, Schulman et al. (2017)) indirectly perform this optimization by first training a reward model to assign scores corresponding to the preferences in the training data, and then optimizing a policy model with the guidance of the reward model. Both approaches have been instrumental in aligning LLMs with human preferences and enhancing their capabilities.

**Data Curation for Preference Learning.** While various methods have been proposed for curating preference data Zhou et al. (2024); Yuan et al. (2024); Lai et al. (2024), in this paper we focus on two popular methods: rejection sampling (RS) and Monte Carlo Tree Search (MCTS). During RS, the policy model generates \(N\) independent responses to the given prompt and a verifier scores each response according to some evaluation metric. Responses with (high, low) scores are selected as the (chosen, rejected) responses Yuan et al. (2024); Khaki et al. (2024). Meanwhile, in the MCTS framework, the policy model performs tree search by generating a fixed number of tokens at each iteration and builds the tree with nodes that represent each subsequence of generated tokens Xie et al. (2024). During MCTS, the policy model performs rollouts by generating full responses and backpropagates the reward scores assigned to the rollouts. This results in a tree of possible responses to the prompt, with each node being assigned a Q-value. Here, sibling nodes with sufficient differences in Q-values or reward scores are selected such that the preference pairs contain common prefixes.

Unlike previous work, we conduct a systematic investigation into how different attributes of preference datasets affect the downstream performance. We choose instruction-following as our task to incorporate multiple constraints into the prompt and score the response on a fine-grained level. We use verifiable constraints to assign quality scores to our responses in a reliable and efficient manner.

## 3 Prompt Synthesis

We perform all our experiments with a new set of synthetic prompts that incorporate mixtures of verifiable constraints. These prompts allow us to evaluate the qualities of the generated responses in both a consistent and fine-grained manner, providing a suitable environment for us to control the attributes of the preference dataset.

### Instruction-Following with Verifiable Constraints

Our verifiable constraints resemble but are distinct from those provided in IFEval Zhou et al. (2023). We define 23 constraints which can be verified using code, including constraints such as having a TL;DR summary at the end, alliteration of a certain number of words, having a fixed number of parentheses, etc. The complete ontology of our constraints is provided in Table 6 in the appendix.

As shown in Table 1, our verifiable constraints follow the formatting of the constraints provided in IFEval - each constraint is accompanied by a set of keyword arguments that are needed to actualize the constraint for the given prompt. For example, the alliteration constraint is paired with a keyword argument num_alliteration_words, which indicates the number of words used for the alliteration.

 



### Prompt Generation

We build a pipeline that generates synthetic prompts for instruction-following, incorporating combinations of verifiable constraints to create a set of challenging prompts that stress-test models' capabilities of following complex instructions. Our pipeline is similar to Instruct-SkillMix(Kaur et al., 2024), except that we have a preexisting set of verifiable constraints functioning as the "skill" and incorporate additional layers for generating the keyword arguments before generating the final instructions. Our pipeline does not require human supervision and generates synthetic prompts only using llama-3.1-7@b-instruct(AI at Meta, 2024).

Figure 2 provides a visual summary of our synthetic prompt generation pipeline. Our pipeline can be decomposed into five steps. First, we take a set of seed instruction-following prompts (e.g., from IFEval) and remove the additional constraints originally associated with the dataset. We few-shot prompt llama-3.1-7@b-instruct and obtain a set of prompts from the seed dataset without constraints. Second, we take the base prompts and use llama-3.1-7@b-instruct to generate new prompts in their base forms, taking an approach similar to Self-Instruct(Wang et al., 2023) by using the existing base prompts as few-shot examples and prompting the model to generate 20 new prompts at a time. Third, we remove the newly generated prompts that are semantically duplicates either with any of the seed base prompts or the other

\begin{table}
\begin{tabular}{l p{85.4pt} p{196.2pt}} \hline constraint & keyword args & description \\ \hline alliteration & num\_alliteration\_words & the response should contain an alliteration, i.e. a sequence of X words starting with the same letter, where \(X=num\_alliteration\_words\). \\ max\_word\_length & max\_word\_length & the maximum length of all words in the response should be at most X characters, where \(X=max\_word\_length\). \\ num\_words\_per\_sentence & relation, num\_words & each sentence should contain \(R\in[at least,atmost]X\) words, where \(R=relation\) and \(X=num\_words\). \\ tldr\_summary & n/a & the response should end with a “TL;DR” on a new line summarizing the response. \\ \hline \end{tabular}
\end{table}
Table 1: Examples of verifiable constraints used for our synthetic prompts. Some constraints require one or more keyword arguments that materialize the constraint for its associated prompt, while others do not require any.

Figure 2: Overview of our pipeline for generating synthetic prompts with verifiable constraints. We first take a set of seed prompts from an existing dataset where the prompts contain constraints, and remove all constraints with an LLM (llama-3.1–7@b-instruct) to obtain base prompts corresponding to the original dataset. Next, we randomly sample a small subset of the base prompts and use them as few-shot examples to generate new prompts _without any constraints_. We remove duplicates among the newly-generated prompts and the existing base prompts using a sentence transformer. Then, we randomly sample a combination of \(k\in\{4,5,6\}\) of our verifiable constraints that are non-conflicting and use an LLM to generate the input parameters required for the set of selected constraints. Finally, we merge the kwargs and the new base prompts, resulting in final prompts written in natural language.



new prompts. We use all-mpnet-base-v2 Song et al. (2020), a lightweight sentence transformer, to compute semantic embeddings and use the dot product to compute similarity scores. Fourth, we randomly sample a combination of \(k\) constraints which do not contradict each other, and for each constraint in this mixture, we randomly sample or generate the associated keyword arguments. We perform random sampling for keyword arguments that can be randomly chosen without considering the prompt such as the relation argument in the num_words_per_sentence constraint or the num_alliteration_words argument in the alliteration constraint. Meanwhile, we use llama-3.1-7bb-instruct to generate the keyword arguments that require more understanding of the prompt, such as the sentence argument for the required_sentence constraint. Fifth, we take the constraints along with the keyword arguments for each prompt and use llama-3.1-70b-instruct to rewrite the base prompts into their final forms which integrate the constraints and their keyword arguments in natural language.

We use our pipeline to generate synthetic prompts with mixtures of verifiable constraints such that we can score the quality of any response on a fine-grained level. Refer to Figure 2 for examples of the synthetic prompts. In contrast to IFEval which uses at most three constraints, our pipeline allows us to integrate any number of constraints to synthesize prompts, resulting in more challenging prompts and more diverse quality of responses. We use \(k\in\{4,5,6\}\) in our experiments. Our choice of using higher values of \(k\) is to make our synthetic training prompts maximally distinct from IFEval Zhou et al. (2023) for more reliable evaluation. Refer to Appendix B for more examples and more details on the generated prompts.

## 4 Preference Data Curation

Using the prompts obtained in Section 3, we generate responses and extract preference data using the correctness scores. We employ two methods for preference data curation: rejection sampling (RS) and Monte Carlo Tree Search (MCTS). RS presents a straightforward and efficient way to obtain preference data but the (chosen, rejected) responses do not share any relationship. On the other hand, MCTS is more expensive and slower to run, but returns (chosen, rejected) responses that share a common prefix with more nuanced contrast.

**Rejection Sampling.** Refer to Figure 1 (left) for a visual overview. We first set a filtering criteria, where the chosen response must achieve a score of \(\mathbf{c}\) according to our verifier and the rejected response must achieve a score of \(\mathbf{r}\). Then, we generate \(N\) responses independently with the policy model and score each response with our verifier. Given a prompt \(x\), its associated set of verifiable constraints \(\mathcal{C}\), the response \(r\) and our verifier \(\mathcal{V}\) which verifies whether the response satisfies any given constraint \(c\), we compute the score as

\[R(r|x,\mathcal{C})=\frac{1}{\|\mathcal{C}\|}\sum_{c\in\mathcal{C}}\mathcal{V} (r|x,c)\]

We extract all preference pairs such that 1) the chosen score equals \(\mathbf{c}\), 2) the rejected score equals \(\mathbf{r}\), and 3) there are no overlapping responses.

* for a given prompt \(x\) and the MCTS tree \(T\), each node \(s_{i}\) in \(T\) represents a partial response generated for \(x\), and each edge \((s_{i},s_{j})\) in \(T\), also known as an _action_, represents a

\begin{table}
\begin{tabular}{l p{198.1pt}} \hline constraints & prompt \\ \hline ascending\_num\_words, freq\_long\_words, max\_word\_length, nth\_sent\_first\_word, start\_checker & Write a short story about a boy who gets lost in a shopping mall. Include at least 7 words that are at least 12 characters long, and ensure that the sentences have an increasing number of words, i.e. each sentence should contain more words than its previous one. Also, only include words that are at most 12 characters long. Make sure that the fifth sentence starts with the word “shotuing”, and begin your response with the sentence ”As the sounds of loud chatter and clinking of dishes filled the food court, little Tommy suddenly discovered that his parents were nowhere to be seen..” \\ \multirow{2}{*}{nth\_sent\_first\_word, num\_bold\_words, num\_exclamations, tldr\_summary, vowel\_capitalization} & Write a motivational speech for a high school graduation ceremony. Capitalize the vowels in your response, and include seven words that are bolded in HTML format (e.g., <b>word</b>). Also, ensure that the sixth sentence starts with the word “today”. Make sure that the response contains exactly three exclamation marks, and finish the response with the final line including ”TL;DR” followed by a one-sentence summary of your response. \\ \hline \end{tabular}
\end{table}
Table 2: Examples of synthetic prompts generated for \(k=5\), which combines five verifiable constraints.

 

sequence of tokens generated from \(s_{i}\to s_{j}\). In this setup, we use an LLM \(\Pi\) as the policy and our verifier as the outcome reward model via three major steps: selection, expansion and backpropagation.

_Selection._ Given the current node \(s_{t}\) and \(K\) distinct actions \((a_{1},...,a_{K})\) generated by the policy from \(s_{t}\), we balance exploitation and exploration to select the next node. Each action is a sequence of tokens under a pre-specified maximum length.

Our selection depends on \(Q(s_{t},a)\) and \(N(s_{t},a)\), the q-value and visit count of each subsequent node reached by taking action \(a\) from \(s_{t}\), respectively. We use the Predictor+Upper Confidence bounds applied to Trees (PUCT) and select the next node \(s_{t+1}\) according to the following formula:

\[s^{*}_{t+1}=\operatornamewithlimits{argmax}_{(s_{t+1}\equiv s_{t}-a_{i})} \left[Q(s_{t},a_{i})+c_{\text{par}}\cdot\Pi(a_{i}|s_{t})\frac{\sqrt{N(s_{t})}} {1+N(s_{t},a_{i})}\right]\]

Appendix C provides details on how to compute the policy score \(\Pi(a_{i}|s_{t})\). Using PUCT, we prioritize exploration during the initial stages of tree building and prioritize exploitation during the later stages of tree building as the visit counts increase. This results in a balanced trade-off between exploration and exploitation during our tree search.

_Expansion._ We perform expansion from the current node \(s_{t}\) and generate \(K\) new actions with the policy \(\Pi\). For each new action, we perform \(M\) separate rollouts and score each rollout using a linear combination of the score \(\mathcal{V}\) assigns and the self-evaluation score assigned by \(\Pi\). We use self-evaluation, denoted as \(\Pi_{\text{self-eval}}(s_{t})\), in addition to the verifier scores, to provide step-level feedback during the tree search. Again, refer to Appendix C for more information on how to compute \(\Pi_{\text{self-eval}}(s_{t})\).

After assigning reward scores to each rollout, we average the scores across the rollouts for each new action \(a_{i}\) for \(i\in[1,\dots,K]\) and add \(s_{t+1}=(s_{t}\to a_{i})\) with the averaged reward score to the tree.

_Backpropagation._ After computing the reward scores for the rollouts and adding new nodes, we backpropagate the reward scores through the parent nodes. We increment the visit counts and update the Q-values of a given (state, action) pair based on the Q-values and the visit counts of the children nodes of \(s_{t+1}=(s_{t},a)\), as the following:

\[N(s_{t})=N(s_{t})+1\]

\[Q(s_{t},a)=\frac{\sum_{i=1}^{K}Q(s_{t+1},a_{i})\cdot N(s_{t+1},a_{i})+R(s_{t+1 })}{\sum_{i=1}^{K}N(s_{t+1},a_{i})+1}\]

We repeat the threefold process over multiple iterations for each root node, and we traverse down the tree until we reach a terminal node.

Finally, we curate preference data from pairs of sibling nodes that satisfy the correctness criteria and sample their rollouts to obtain complete responses. We only use the scores from \(\mathcal{V}\) to ensure that the correctness of the responses stay consistent across both data curation methods. Appendix D provides more details of our preference pairs.

## 5 Experiments and Results

We examine three dimensions of the preference dataset: the 1) existence of common prefixes between the (chosen, rejected) responses, 2) quality of the chosen and rejected responses, and 3) difficulty of the prompts used for training. We report results for additional experiments in Appendix E.

**Data curation setup.** We implement RS by generating \(N=64\) outputs for each prompt in our training set with a temperature of 1.0, and score each output using \(\mathcal{V}\). Meanwhile, we implement MCTS with maximum depth of 5, number of actions 4, number of rollouts 4, \(c_{puct}=1.0\) and \(\lambda=0.2\).

**Training setup.** We run our experiments with llama-3.1-8b-instruct (AI at Meta, 2024) and finetune the model on the preference dataset using DPO Rafailov et al. (2023). We finetune the model for one epoch with a maximum sequence length of 2048, learning rate of 5e-7 with a linear scheduler and 4 gradient accumulation steps.

**Evaluation setup.** We evaluate our models on IFEval Zhou et al. (2023), as well as three different synthetic evaluation sets designed to be more challenging than IFEval. Our synthetic evaluation sets are created using the same pipeline in Section 3.2, but with the IFEval constraints. We use \(k\in\{4,5,6\}\) for our evaluation sets and synthesize about 500 evaluation prompts for each value of \(k\).

Using the finetuned models, we generate 16 responses to each prompt at temperature 0.7 and measure the average score. We use two metrics: 1) a hard score measuring whether the response satisfies _all_ the constraints, and 2) a soft score measuring the _ratio_ of the constraints satisfied by the response.

### Common Prefix

We investigate the effect of having a common prefix between the (chosen, rejected) responses in the 

preference dataset. Recent techniques utilize tree search to curate fine-grained preference pairs where the (chosen, rejected) responses differ after a common prefix - we examine the effect of utilizing such preference pairs as exploited by recent methods. To this end, we use rejection sampling (RS), which returns responses without common prefixes, and Monte Carlo Tree Search (MCTS), which returns responses with common prefixes.

Table 3 shows the results of our experiments. For each comparison, we fix the size of the training dataset and the number of unique prompts in each training set. Meanwhile, we vary the the correctness of the (chosen, rejected) responses and the training prompt difficulties as measured by \(k\).

**MCTS outperforms RS by a small margin consistently over different training configurations.** However, the difference is not enough to warrant the complexity of MCTS in ordinary settings.

**MCTS offers more consistent performance over different training configurations than RS.** This implies that for cases where the correctness of the response is difficult to quantify, it may be effective to use the preference dataset curated via MCTS.

### Response Quality

We investigate the effects of controlling the correctness of the (chosen, rejected) responses in the preference dataset. Here, we maintain the same training dataset size and the number of unique prompts given a fixed curation method and training prompt difficulty. Our results are shown in Table 4.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**training k=4**} \\ Method & IFEval & Ours\({}_{t=4}\) & Ours\({}_{b=5}\) & Ours\({}_{h=6}\) & Method & IFEval & Ours\({}_{h=4}\) & Ours\({}_{h=5}\) & Ours\({}_{h=6}\) \\ \hline RS, (ca+3, r=0) & 79.04 & 39.45 & 20.86 & 13.82 & RS, (cn+d, r=1) & 76.56 & 36.08 & 19.44 & 11.88 \\ RS, (en+3, r=1) & 78.66 & 38.47 & 20.56 & 14.02 & RS, (cn+d, r=2) & 74.57 & 34.09 & 17.95 & 10.93 \\ RS, (cc+3, r=2) & 74.60 & 33.59 & 18.46 & 11.23 & RS, (cn+d, r=3) & 72.80 & 31.50 & 16.68 & 10.01 \\ \hline RS, (cd, r=1) & 79.24 & 37.49 & 20.54 & 13.78 & RS, (es+r, 2) & 76.32 & 35.70 & 19.46 & 12.61 \\ RS, (cs+d, r=2) & 77.44 & 37.69 & 20.33 & 13.77 & RS, (es+r, 3) & 74.13 & 33.30 & 17.31 & 11.21 \\ RS, (cd, r=3) & 74.09 & 33.38 & 17.65 & 10.73 & RS, (es+r, 4) & 72.40 & 30.72 & 16.26 & 9.51 \\ \hline MCTS, (c=3, r=0) & 78.30 & 37.86 & 20.52 & 12.81 & MCTS, (c=d, r=1) & 76.71 & 37.24 & 19.49 & 12.37 \\ MCTS, (c=3, r=1) & 77.58 & 38.09 & 19.76 & 13.71 & MCTS, (ct=d, r=2) & 75.23 & 35.65 & 18.74 & 12.17 \\ MCTS, (rs+d, r=2) & 75.05 & 34.96 & 18.69 & 11.53 & MCTS, (cn+d, r=3) & 74.23 & 32.39 & 17.22 & 9.73 \\ \hline MCTS, (end,tr=1) & 79.48 & 39.16 & 20.39 & 14.11 & MCTS, (cs+d, r=2) & 76.59 & 37.81 & 20.06 & 12.93 \\ MCTS, (end,tr=2) & 77.76 & 38.25 & 20.00 & 14.26 & MCTS, (cs+d, r=3) & 76.14 & 36.04 & 19.39 & 12.37 \\ MCTS, (end,tr=3) & 75.65 & 35.03 & 18.80 & 12.36 & MCTS, (cs+d, r=4) & 74.06 & 32.14 & 17.81 & 10.52 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Evaluation results studying the effects of (chosen, rejected) response quality. We provide results for training prompt difficulties with \(k\in\{4,5\}\), and for both RS- and MCTS-based data caruations. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The soft score metrics for all experiments are provided in Tables 14 and 15 in the appendix.

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**training k=4**} & \multicolumn{5}{c}{**training k=5**} \\ Method & IFEval & Ours\({}_{t=4}\) & Ours\({}_{b=5}\) & Ours\({}_{h=6}\) & Method & IFeval & Ours\({}_{h=4}\) & Ours\({}_{h=5}\) & Ours\({}_{h=6}\) \\ \hline RS, (ca+, r=1) & 79.24 & 37.49 & 20.54 & 13.78 & RS, (cs+, r=2) & 76.32 & 35.70 & 19.46 & 12.61 \\ MCTS, (cat+, r=1) & 79.48 & 39.16 & 20.39 & 14.11 & MCTS, (cs+, r=2) & 76.59 & 37.81 & 20.06 & 12.93 \\ \hline RS, (ca+, r=2) & 78.86 & 38.56 & 21.66 & 14.71 & RS, (cs+, r=3) & 76.25 & 35.28 & 18.91 & 12.24 \\ MCTS, (rc+d, r=2) & 79.68 & 39.22 & 22.43 & 15.75 & MCTS, (cs+, r=3) & 76.47 & 36.73 & 19.94 & 13.62 \\ \hline RS, (ca+, r=3) & 76.74 & 35.02 & 19.40 & 12.85 & RS, (cs+, r=4) & 74.18 & 33.38 & 17.35 & 10.38 \\ MCTS, (c=d, r=3) & 79.59 & 39.05 & 21.61 & 14.96 & MCTS, (cs+, r=4) & 75.15 & 34.88 & 18.75 & 11.74 \\ \hline RS, (ca3, r=0) & 80.06 & 39.25 & 21.54 & 14.37 & RS, (cnt+, r=1) & 78.95 & 39.40 & 21.74 & 14.58 \\ MCTS, (co3, r=0) & 79.39 & 39.15 & 21.56 & 13.61 & MCTS, (cat+, r=1) & 78.63 & 39.21 & 21.19 & 14.63 \\ \hline RS, (ca3, r=1) & 80.06 & 39.15 & 21.90 & 15.20 & RS, (ent+, r=2) & 78.07 & 37.02 & 20.17 & 13.05 \\ MCTS, (ci3, r=1) & 79.94 & 39.23 & 22.15 & 14.93 & MCTS, (cat+, r=2) & 78.48 & 38.53 & 22.12 & 15.22 \\ \hline RS, (ca3, r=2) & 77.52 & 36.20 & 19.52 & 12.48 & RS, (cnt+, r=3) & 75.64 & 32.98 & 17.83 & 11.19 \\ MCTS, (ce3, r=2) & 77.89 & 38.89 & 21.51 & 13.97 & MCTS, (cat+, r=3) & 77.40 & 37.63 & 20.84 & 14.06 \\ \hline RS, (ca4, r=1/2/3) & 78.70 & 36.68 & 20.20 & 13.57 & RS, (cnt+, r=1/2/3) & 79.08 & 37.98 & 20.50 & 13.42 \\ MCTS, (ce4, r=1/2/3) & 79.97 & 39.31 & 22.19 & 15.89 & MCTS, (cat+, r=1/2/3) & 78.47 & 39.03 & 22.63 & 15.25 \\ \hline RS, (ca3+, r=0/1/2/3) & 79.10 & 37.62 & 20.98 & 13.39 & RS, (cn+d/5, r=0/1/2/3) & 78.18 & 36.81 & 19.72 & 13.29 \\ MCTS, (ca3+, r=0/1/2/3) & 79.42 & 39.37 & 22.24 & 15.20 & MCTS, (cat+, r=0/1/2/3) & 77.89 & 38.17 & 22.00 & 14.38 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Evaluation results comparing preference data without common prefixes (RS) and with common prefixes (MCTS). We show results for different training 

We also study the effects of having a mixture of high- and low-contrast pairs. To compare its performance to those of using only high- or low-contrast pairs, we fix the training dataset size and the number of unique prompts, and switch some rejected responses with low correctness scores to ones with higher correctness scores. For this experiment, we use MCTS for data curation along with \(k=4\) or \(5\) for the complexity of the training prompts.

The results of our experiments are visualized in Figure 3. We show the composition of the (chosen, rejected) responses in terms of their correctness on the x-axis, and the evaluation metric on the y-axis for each subplot. For example, for a training set with \(k=4\) constraints, we begin with (c, r) = (4, 1) and then mix in r=2 to obtain (c, r) = (4, 1/2), and then mix in r=3 to obtain (c, r) = (4, 1/2/3).

**When used alone, high-contrast preference pairs is more helpful than low-contrast preference pairs.** Our results in Table 4 show that for a fixed correctness of the chosen response, increasing the correctness of the rejected response decreases the performance consistently. This trend holds across both RS- and MCTS-based data curations, as well as across the difficulty of the training prompts.

**The margin between the (chosen, rejected) responses have a bigger impact than the absolute scores themselves.** Table 4 demonstrates that preference datasets with the same value of \(c-r\) result in similar performances across our evaluation sets. The margin between the preference pairs seems to have more influence on the downstream performance than the absolute correctness as long as the chosen responses are reasonably correct.

**Having a mixture of both high-contrast and low-contrast pairs sometimes demonstrates better performance than only using high-contrast pairs.** Our results in Figure 3 indicate that given the same training dataset size, mixing high-contrast and low-contrast preference pairs sometimes return better results than only using high-contrast pairs. Our results in Tables 16 and 17 in the appendix also hint that having a mixture of contrasts often helps, but the results are somewhat mixed.

### Prompt Difficulty

We examine how the difficulty of training prompts in a preference dataset affects performance. For each comparison, we fix the data curation method and the margin between the (chosen, rejected) responses while comparing the difficulties of our training prompts with \(k\in\{4,5,6\}\) with a fixed training data size. Then, we repeat our experiments across the other two dimensions. Table 5 summarizes the results of our experiments.

**Training on moderately difficult prompts is overall more helpful than training on extremely difficult prompts.** We find that models trained on moderately difficult prompts perform better than models trained on extreme difficulty prompts in all

Figure 3: Evaluation results showing the effects of mixing preference pairs with different margins between the (chosen, rejected) responses. The rows correspond to the number of verifiable constraints in each prompt (values of \(k\)), and each columns corresponds to an evaluation set. The x-axis indicates the correctness of the (chosen, rejected) responses with lower-margin pairs mixed in while keeping the same training size. The y-axis represents the accuracies. Results for more experiments are provided in Tables 16 and 17 in the appendix.

 

our evaluation sets. This holds across both RS- and MCTS-based preference data curation methods.

**Training on moderately difficult prompts is more helpful even for performing well on extremely difficult prompts at test time.** Surprisingly, models trained on moderately difficult prompts (\(k=4\)) outperform models trained on extremely difficult prompts (\(k=6\)) even for evaluations with \(k=6\). Our results indicate that training prompts with moderate difficulties help the model to achieve generalization to more difficult prompts, even more so than overly difficult prompts.

## 6 Conclusion

We systematically investigate the effects of various attributes of preference datasets on model capabilities via instruction-following. To this end, we first build a data generation pipeline that combines general prompts with mixtures of verifiable constraints to synthesize challenging instructions-following prompts. We then automatically curate preference pairs using rejection sampling (RS) and Monte Carlo Tree Search (MCTS). Using the preference pairs, we examine the effects of (1) the existence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the responses, and (3) the complexity of the training prompts. Our results indicate that having a common prefix in the preference pairs offers marginal yet consistent improvements, high-contrast preference pairs outperform low-contrast pairs but a mixture is sometimes better than both, and training on moderately difficult prompts is more helpful than training on extremely difficult prompts. Our work provides a systematic framework for curating different types of preference datasets and sets the groundwork for future studies that extend the scope beyond verifiable instruction-following constraints to more general constraints.

## 7 Limitations

Our work contains several limitations. First, we restrict our analysis to instruction-following with verifiable constraints. As discussed in the paper, we focus on 1) instruction-following in order to integrate multiple constraints in each prompt which allows us to assign fine-grained scores to any response, and 2) verifiable constraints in order to evaluate the quality of each response with high reliability and efficiency. Future work should explore training high-quality reward models that assign reliable scalar scores for more abstract constraints (e.g., does the response follow the topic indicated in the prompt) such that the ontology of constraints can be expanded beyond computationally verifiable constraints. Second, we perform our experiments only with models at the 8B scale. The number of different training configurations in our experiments, in addition to the number of outputs generated per prompt during evaluation, makes it difficult to scale our experiments to 70B and beyond. We expect that experiments at the 70B scale would provide further useful signals for curating preference data with larger models. Third, we restrict our preference data curation methods to rejection sampling and Monte Carlo Tree Search. Our purpose is not to exhaustively verify all methods for automatic preference data curation, but to compare two popular yet contrastive data curation methods: one which is quick and efficient but does not have any control over the structure of the preference pairs, and another which consumes more compute and is tricky to implement but generates fine-grained preference pairs with shared prefixes that can be controlled. Future work should investigate more diverse types of data curation methods to specifically examine which method offers the best performance improvements given the same constraints.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline  & \multicolumn{5}{c}{**Rejection Sampling (RS)**} \\ Method & IFFUAL & \(\text{Oars}_{s+4}\) & \(\text{Ours}_{s+5}\) & \(\text{Ours}_{b+6}\) & Method & IFEval & Ours\({}_{b+4}\) & \(\text{Ours}_{b+5}\) & \(\text{Ours}_{b+6}\) \\ \hline \(k=4\), (e=3, r=0) & 79.70 & 38.41 & 21.15 & 14.27 & \(k=4\), (e=3, r=0) & 78.94 & 39.00 & 21.60 & 14.40 \\ \(k=5\), (e=4r=1) & 79.51 & 38.66 & 21.48 & 14.70 & \(k=5\), (e=4r=1) & 78.72 & 39.42 & 21.43 & 14.25 \\ \(k=6\), (e=5, r=2) & 77.83 & 36.42 & 20.15 & 12.28 & \(k=6\), (e=5, r=2) & 78.13 & 38.55 & 21.64 & 14.45 \\ \hline \(k=4\), (e=3, r=1) & 79.36 & 39.22 & 22.00 & 14.78 & \(k=4\), (e=3, r=1) & 79.12 & 38.81 & 21.72 & 15.66 \\ \(k=5\), (e=4, r=2) & 78.58 & 37.70 & 21.42 & 13.61 & \(k=5\), (e=4, r=2) & 77.53 & 38.93 & 21.69 & 15.20 \\ \(k=6\), (e=5, r=3) & 76.73 & 36.16 & 19.96 & 12.87 & \(k=6\), (e=5, r=3) & 77.18 & 37.06 & 20.16 & 12.84 \\ \hline \end{tabular}
\end{table}
Table 5: Evaluation results investigating the effects of training prompt difficulty (\(k\in\{4,5,6\}\)) on downstream performance for evaluation sets of varying difficulties. We provide results for both RS- and MCTS-based preference data curation methods, as well as for different margins between the (chosen, rejected) responses. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The soft score metrics for all experiments are provided in Tables 18 and 19 in the appendix.

 



## References

* [1]AI at Meta. 2024. The llama 3 herd of models. _CoRR_, abs/2407.21783.
* [2] Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. [https://www-cdn.anthropic.com/debsba9b1c9ab7cbabf5c3ib8db7bbc618857627/Model_Card_Claude_3.pdf](https://www-cdn.anthropic.com/debsba9b1c9ab7cbabf5c3ib8db7bbc618857627/Model_Card_Claude_3.pdf).
* Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. _CoRR_, abs/2204.05862.
* Google [2023]Google. 2023. Gemini: A family of highly capable multimodal models. _CoRR_, abs/2312.11805.
* Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlIPS Datasets and Benchmarks 2021, December 2021, virtual_.
* Ivison et al. [2024] Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqi Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirizi. 2024. Unpacking DPO and PPO: disentangling best practices for learning from preference feedback. _CoRR_, abs/2406.09279.
* Kaur et al. [2024] Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. 2024. Invtuct-skillmix: A powerful pipeline for LLM instruction tuning. _CoRR_, abs/2408.14774.
* Khaki et al. [2024] Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. RS-DOP: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. In _Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024_, pages 1665-1680. Association for Computational Linguistics.
* Lai et al. [2024] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. _CoRR_, abs/2406.18629.
* Mialon et al. [2024] Gregoire Mialon, Clementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scilam. 2024. GAIA: a benchmark for general AI assistants. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* OpenAI [2023]OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* December 9, 2022_.
* Pang et al. [2024] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. _CoRR_, abs/2404.19733.
* 16, 2023_.
* Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _CoRR_, abs/1707.06347.
* Song et al. [2020] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlIPS 2020, December 6-12, 2020, virtual_.
* Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlIPS 2020, December 6-12, 2020, Virtual_.
* Wang et al. [2022] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _ArXiv_, abs/2203.11171.
* Wang et al. [2023] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishrizi. 2023. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 13484-13508. Association for Computational Linguistics.



Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Homing Wang, and Minlie Huang. 2024. Benchmarking complex instruction-following with multiple constraints composition. _CoRR_, abs/2407.03978.
* Xiao et al. (2024) Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, and Fei Wu. 2024. A comprehensive survey of datasets, theories, variants, and applications in direct preference optimization. _CoRR_, abs/2410.15595.
* Xie et al. (2024) Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024. Monte carlo tree search boosts reasoning via iterative preference learning. _CoRR_, abs/2405.00451.
* Xu et al. (2023) Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more CRINGE than others: Preference optimization with the pairwise crime loss. _CoRR_, abs/2312.16682.
* Yuan et al. (2024a) Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Bojil Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. 2024a. Advancing LLM reasoning generalists with preference trees. _CoRR_, abs/2404.02078.
* Yuan et al. (2024b) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024b. Self-rewarding language models. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Zhang et al. (2024a) Dan Zhang, Sining Zhoubian, Yusong Yue, Yunxiao Dong, and Jie Tang. 2024a. Rest-mcts*: LLM self-training via process reward guided tree search. _CoRR_, abs/2406.03816.
* Zhang et al. (2024b) Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, and Zenan Zhou. 2024b. Cfbench: A comprehensive constraints-following benchmark for llms. _CoRR_, abs/2408.01122.
* Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. _CoRR_, abs/2311.07911.
* Zhou et al. (2024) Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. _CoRR_, abs/2402.11411 

Verifiable Constraints

We present the complete ontology of our 23 verifiable constraints in Table 6.

## Appendix B Synthetic Prompt Details

We generate instruction-following prompts for combinations of \(k\in\{4,5,6\}\) constraints using vLLM 1. Table 7 shows the statistics that describe the prompts that we generate. For each value of \(k\), we generate about 16K synthetic prompts, resulting in a total of about 48K prompts being used in our experiments. The average length of the prompts increases as the number of constraints increases, as the complexity of the prompts increases with larger numbers of constraints.

Footnote 1: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)

Table 8 displays examples of synthetic prompts generated using our pipeline for a given combination of verifiable constraints. Each prompt contains a general-purpose base instruction such as writing a short story or a speech, and is accompanied by a combination of verifiable constraints defined in our ontology with keyword arguments that satisfy the context of the instruction - for example, the first example uses the word "shouting" for the nth_sent_first_word constraint, which is appropriate for the context of the base instruction involving a boy lost in a shopping mall. For any arbitrary response to a given prompt, we can use the associated constraints and keyword arguments to automatically assign a score indicating whether the response follows each constraint and aggregate the scores to assign an overall correctness score to the response.

## Appendix C MCTS Details

**Computing the policy score.** We compute the policy score \(\Pi(a_{i}|s_{t})\) by computing the average of the log probabilities of tokens generated for action \(a_{i}\) from state \(s_{t}\), with the denominator moderated by a hyperparameter \(\gamma\), which we set to 1.0 for our experiments. Refer to the formula below for the exact definition:

\[\Pi(a_{i}|s_{t})=\text{exp}\left[\frac{1}{\lVert a_{i}\rVert^{\gamma}}\sum_{ j=1}^{\lVert a_{i}\rVert}\Pi(t_{j}|s_{t},t_{1...j-1})\right]\]

Note that \(t_{j}\) denotes the \(j\)th token in action \(a_{i}\).

**Computing the self-evaluation score.** We compute the self-evaluation score \(\Pi_{\text{self-eval}}\) by prompting the policy with a self-evaluation prompt \(P_{\text{self-eval}}\), coupled with the response generated by the model so far, and obtaining the log probabilities of the final token of the response, denoted as \(t_{\text{final}}\in\{\text{yes},\text{no}\}\). We use self-consistency Wang et al. (2022) to obtain multiple self-evaluations of the policy of its own output over \(L\) generations and average the scores in order to improve the reliability of our self-evaluation scores via increased compute.

\[\Pi_{\text{self-eval}} =\frac{1}{L}\sum_{i=1}^{L}\Bigg{[}\frac{1+\text{exp}(\Pi(t_{ \text{final}}=\text{yes}|P_{\text{self-eval}},s_{t}))}{2}\] \[-\frac{\text{exp}(\Pi(t_{\text{final}}=\text{no}|P_{\text{self- eval}},s_{t}))}{2}\Bigg{]}\]

Our formula normalizes the score between 0 to 1, with 0.5 indicating a neutral state where the confidence scores for \(\Pi(t_{\text{final}}=\text{yes}|P_{\text{self-eval}},s_{t})=\Pi(t_{\text{final }}=\text{no}|P_{\text{self-eval}},s_{t})\). We provide the self-evaluation prompt \(P_{\text{self-eval}}\) in Figure 5.

## Appendix D Preference Pair Details

We curate preference pairs using both RS- and MCTS-based methods for our synthetic prompts with \(k\in\{4,5,6\}\), again using vLLM. Figure 4 shows the number of preference pairs obtained via both methods for \(k=5\) when the correctness criteria for (chosen, rejected) pairs is set as (5 correct, 0 correct), (5 correct, 1 correct), (5 correct, 2 correct), (5 correct, 3 correct) and (5 correct, 4 correct) pairs. The left subfigure depicts the number of unique prompts corresponding to the preference pairs extracted for each filtering criteria. Using rejection sampling (RS) returns a higher yield of preference pairs with high contrast between the (chosen, rejected) responses, while using Monte Carlo Tree Search (MCTS) returns a higher yield of preference pairs with low contrast between the responses. The right subfigure depicts the total number of preference pairs extracted for each criteria - the same observation can be made about the relative yield, with MCTS yielding a large number of preference pairs due to its tree structure. We use preference pairs collected for \(k\in\{4,5,6\}\) using both methods over different filtering criteria to perform our experiments.

 



\begin{table}
\begin{tabular}{l l l} \hline constraint & keyword args & description \\ \hline alliteration & num\_alliteration\_words & the response should contain an alliteration, i.e. a sequence of X words starting with the same letter, where X = num\_alliteration\_words. \\ ascending\_num\_words & n/a & the response should contain sentences such that the number of words in each sentence is in ascending order. \\ edit\_response & n/a & the response should be separated by dashes ‘——’ separating two responses with the second response improving upon the first response. \\ end\_quotation & n/a & the last sentence should be wrapped in quotation marks. \\ first\_letter\_capital & n/a & the first letter of each word should be capitalized. \\ frequency\_long\_words & relation, num\_words, word\_length & the response should contain sentences such that the number of words in each sentence is in ascending order. \\ keywords\_ordered & keywords \\ max\_word\_length & max\_word\_length & the maximum length of all words in the response should be at most X characters, where X = max\_word\_length. \\ no\_period & n/a & the response should contain no periods. \\ nth\_sentence\_capital & nth\_sentence & the nth\_sentence in the response should be all capitalized (and only the nth sentence). \\ nth\_sentence\_first\_word & first\_word, \\ num\_sentences & num\_words per\_sentence & relation, num\_words \\ number\_body\_words & num\_words & the response should contain R \(\in\) \\ number\_exclamations & relation, num\_exclamations & the response should contain x words in textile \\ number\_italic\_words & num\_words & the response should contain exactly X parentheses (), where X = num\_parentheses. \\ number\_parentheses & part\_splitter, & the response should contain X parts such as “Part 1”, “PART 1”, ”PART 2", ”PART 2”, ...  where part\_splitter @ {part\_parallel} & and X = num\_parts. \\ numbered\_headers & num\_headers & the response should contain X enumerated headings \\ required\_sentence & sentence & the response should italize exactly X words in textile \\ start\_checker & first\_sentence & the response should begin with first\_sentence. \\ tldr\_summary & n/a & the response should end with a “TL;DR” on a new line summarizing the response. \\ variable\_placeholder\_format & relation, & the response should contain R \(\in\) {at least, at most} X \\ num\_placeholders & variable placeholders in curly brackets, where R = & relation and X = num\_placeholders. \\ vowel\_capitalization & n/a & the response should capitalize all vowels in the response. \\ \hline \end{tabular}
\end{table}
Table 6: Complete list of verifiable constraints used for our synthetic prompts.

\begin{table}
\begin{tabular}{c c|c c c c|c c c} \hline \multicolumn{3}{c}{\(k=4\)} & \multicolumn{3}{c}{\(k=5\)} & \multicolumn{3}{c}{\(k=6\)} \\ \hline num\_prompts & mean\_words & std\_words & num\_prompts & mean\_words & std\_words & num\_prompts & mean\_words & std\_words \\ \hline
15,900 & 70.62 & 14.37 & 15,739 & 84.17 & 15.61 & 15,559 & 97.95 & 16.91 \\ \hline \end{tabular}
\end{table}
Table 7: Statistics of synthetic prompts generated by our pipeline. We generate about 16K prompts for each value of \(k\), resulting in 48K prompts total across all our experiments. Note that num\_prompts refers to the number of prompts, mean\_words refers to the average number of words in each prompt, and std\_words refers to the standard deviation of the number of words in each prompt. The number of words in each prompt increases with the number of constraints.



* 
user
 Evaluatewhethertheassistant's(partial)responsetothesgiveninstructionfollowsthemeditions specifiedintheinstructutionsofaranddoesnotviolateanyofthemoditions.Completetheevaluationbyusingtherwords"yesor"no",followerblyanexplanationforwhytheassistant'sresponsefollowsordoesnotfollowthegiveninstructionsofar. DoNOTevaluatetheconditionsthatcanbecheckedautomaticallywithPythoncode,includingtheneslistedbelow. -DON'TEVALUATE:numberofparagraphsh/sentences/words/sections DONNTEVALUATE:existenceofcertainphrases/words/characterst-0DN'TEVALUATE:capitalorlowerase Makewereonlyevaluatehemoditionsthatcanbecheckedsorfar.Forexample,yocannotcheckiftheresponsecontainsatleast2generstem 



\begin{table}
\begin{tabular}{p{142.3pt} p{280.5pt}} \hline constraints & prompt \\ \hline ascending_num_words, freq_long_words, max_word_length, nth_sent_first_word, start_checker & Write a short story about a boy who gets lost in a shopping mall. Include at least 7 words that are at least 12 characters long, and ensure that the sentences have an increasing number of words, i.e. each sentence should contain more words than its previous one. Also, only include words that are at most 12 characters long. Make sure that the fifth sentence starts with the word “shouting”, and begin your response with the sentence ”As the sounds of loud chatter and clinking of dishes filled the food court, little Tommy suddenly discovered that his parents were nowhere to be seen..". \\ \multirow{6}{*}{nth_sent_first_word, num_bold_words, num_exclamations, tldr_summary, vowel_capitalization} & Write a motivational speech for a high school graduation ceremony. Capitalize the vowels in your response, and include seven words that are bolded in HTML format (e.g., ¡b>word</b>>). Also, ensure that the sixth sentence starts with the word “today”. Make sure that the response contains exactly three exclamation marks, and finish the response with the final line including “TL;DR” followed by a one-sentence summary of your response. \\ \multirow{6}{*}{alliteration, keywords_ordered, nth_sent_first_word, number_bold_words, vowel_capitalization} & Write a score where a character walks into a room and is surprised by what they see. Capitalize the vowels in your response and include exactly six words that are bolded in HTML format (e.g., ¡b>word</b>>). Include an alliteration of five consecutive words beginning with the same letter of the alphabet, and include the words ’door’, ‘space’, ‘chaos’ in your response in the exact order provided. Moreover, make sure that the fourth sentence starts with the word “suddenly”. \\ \multirow{6}{*}{alliteration, number_italic_words, number_parts, required_sentence, start_checker} & Write a short essay on the importance of vitamin D in human health. Start your response with the sentence ”Vitamin D is an essential nutrient that has been gaining increasing attention in recent years, primarily due to its significant role in preventing various health issues, ranging from bone diseases to certain cancers.” Include the sentence ”Research has shown that vitamin D deficiency is associated with an increased risk of cardiovascular disease, diabetes, and osteoporosis.” in your response, and include one word that is italicized in textile format, wrapped between underscore characters (e.g., __word__). Divide your essay into one part marked with ’Part 1’, and include an alliteration of five consecutive words beginning with the same letter of the alphabet. \\ \multirow{6}{*}{edit_response, num_exclamations, start_checker, tddr_summary, vowel_capitalization} & Write a short script of a news anchor introducing a breaking news story. Capitalize the vowels in your response, provide two responses separated by six plus signs +++++, and include at least nine exclamation marks. Also, include a ”TL;DR” summarizing the breaking news story at the end of your response. Begin the response with the sentence ”We interrupt your regular programming to bring you this breaking news story coming in from the White House.”. \\ \multirow{6}{*}{alliteration, ascending_num_words, nth_sentence_capital, number_italic_words, number_parentheses} & Create a data model for a medical records database, including at least one relationship and one constraint. Write the first sentence of the response in all capital letters, include eight words that are italicized in textile format, and include an alliteration of five consecutive words beginning with the same letter of the alphabet. Ensure that the sentences in the response have an increasing number of words, i.e. each sentence should contain more words than its previous one. Include exactly six parentheses in the response. \\ \multirow{6}{*}{max_word_length, number_parentheses, number_parts, tldr_summary, vowel_capitalization} & Write a list of 10 ways to improve your public speaking skills. Capitalize the vowels in the response. Have one part marked with PART 1. Finish your response with the final line including ”TL;DR” followed by a one-sentence summary of your response. Only include words that are at most 14 characters long, and include exactly eight parentheses in the response. \\ \multirow{6}{*}{alliteration, nth_sentence_first_word, num_words_per_sentence, number_bold_words, vowel_capitalization} & Write a letter to a historical figure, asking for their advice on a modern issue. Ensure that the vowels in the response are capitalized and there are eight words that are bolded in HTML format (e.g., ¡b>word</b¿). Include an alliteration of three consecutive words beginning with the same letter of the alphabet. Make sure that the sixth sentence starts with the word “nonetheless”, and that each sentence in the response is less than 15 words long. \\ \end{tabular}
\end{table}
Table 8: Examples of synthetic prompts generated for \(k=5\), which combines five verifiable constraints.



Additional Experiments

We perform two additional experiments to confirm that preference learning helps our models gain instruction-following skills, and investigate the limitations of our simpler RS-based data curation method. To this end, we 1) compare the performances of SFT and DPO to ensure that our preference datasets teach meaningful skills to our models, and 2) examine how RS-based preference data curation methods scale by varying the value of \(N\), the number of samples generated for each prompt during rejection sampling.

### Sft vs. Dpo

We demonstrate that our preference learning setup teaches meaningful skills to the models and hence provide a solid ground upon which to perform the comparison experiments shown in Sections 5.1, 5.2 and 5.3. To this end, we compare the performances of the models trained via DPO with the base policy model (llama-3.1-8b-instruct) and the policy model trained via supervised fine-tuning (SFT) on the chosen responses only. For SFT, we train each model in bf16 for three epochs with a maximum sequence length of 2048, learning rate of 2e-6 with a linear scheduler and 4 gradient accumulation steps with a total batch size of 32. We fix the training dataset size for all experiments.

Tables 9 and 10 shows the results of our experiments. First, we check that DPO significantly improves over the base policy model even though the verifiable constraints in the training and evaluation sets are distinct - this indicates that training on a set of verifiable constraints can be helpful for generalizing to another set of verifiable constraints as long as they involve transferable skills. Moreover, we observe that DPO outperforms SFT across all training configurations, demonstrating the importance of our preference learning setup with the rejected responses in addition to the chosen responses.

### Scaling Rejection Sampling

We investigate how different values of \(N\) during rejection sampling affect its overall quality and ensure that our RS-based method is designed to be competitive against MCTS. To this end, we rerun the rejection sampling curation pipeline for \(N\in\{4,8,16,32,64\}\) for a fixed difficulty \(k=5\) and correctness scores of (c, r) = (4, 1) and (4, 2) and measure the performance across all of our evaluation sets.

Figure 6 shows the results of our experiments. We observe that the performance of the models trained on preference pairs curated with RS-based methods increases with \(N\), but also saturate around \(N=32\) and do not observe significant performance improvements afterward. This indicates that curating preference data via rejection sampling, while efficient and straightforward to scale, has limitations that cannot be solved by simply scaling the number of generated outputs, and further improvements may require more sophisticated search strategies such as the MCTS-based curation method implemented in this work.

## Appendix F Full Experiment Results

### Common Prefix Results

We provide the full results of our experiments investigating the effects of common prefixes in preference pairs in Tables 11, 12 and 13.

### Response Quality Results

**(chosen, rejected) response quality (unmixed).** We provide the full results of our experiments investigating the effects of the response correctness (or quality) in preference pairs in Tables 14 and 15. **(chosen, rejected) response quality (mixed).** We provide the full results of our experiments examining the effects of mixing preference pairs with different margins between the (chosen, rejected) responses in Tables 16 and 17.

### Prompt Difficulty Results

We provide the full results of our experiments investigating the effects of varying the difficulty of the training prompts, as measured by the number of verifiable constraints, in Tables 18 and 19 



\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**Monte Carlo Tree Search (MCTS)**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline policy (11am-3.1 -fb-instruct) & 71.71 / 80.12 & 27.77 / 71.53 & 14.15 / 67.76 & 7.34 / 64.43 \\ \hline SFT, \(k=4\), (real, r=1/2/3) & 73.30 / 81.83 & 29.70 / 73.48 & 15.02 / 69.16 & 7.50 / 65.56 \\ DPO, \(k=4\), (cam+r=1/2/3) & 79.97 / 86.51 & 39.31 / 78.22 & 22.19 / 72.14 & 15.89 / 70.36 \\ \hline SFT, \(k=5\), (cad, r=1/2/3) & 73.87 / 81.89 & 28.54 / 72.37 & 13.49 / 67.54 & 6.68 / 64.13 \\ DPO, \(k=5\), (ca+t, r=1/2/3) & 78.47 / 85.25 & 39.03 / 78.06 & 22.63 / 72.49 & 15.25 / 70.05 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Evaluation results comparing the performance of training on our preference datasets via DPO compared to the base policy model or running SFT on the chosen responses only. We provide results for MCTS-based preference data curation, as well as for different training prompt difficulties (\(k=4\) or \(k=5\)). Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).

Figure 6: Evaluation results for increasing the number of outputs generated for rejection sampling (RS) from \(N=4\) to \(N=64\), given a set of training prompts with \(k=5\) and (c, r) = (4, 1) or (4, 2). We observe a steady improvement in performance as more outputs are generated per prompt until \(N=32\), where it begins to saturate or even deteriorate.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**training k=5**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline RS, (c=4, r=1) & 79.24 / 85.78 & 37.49 / 77.50 & 20.54 / 72.06 & 13.78 / 70.23 \\ MCTTS, (c=4, r=1) & 79.48 / 85.92 & 39.16 / 78.11 & 20.39 / 72.04 & 14.11 / 70.10 \\ \hline RS, (c=4, r=2) & 78.86 / 85.74 & 38.56 / 76.02 & 21.66 / 72.34 & 14.71 / 70.00 \\ MCTS, (c=4, r=2) & 79.68 / 86.36 & 39.22 / 78.20 & 22.43 / 72.64 & 15.75 / 70.61 \\ \hline RS, (c=4, r=3) & 76.74 / 83.92 & 35.02 / 76.25 & 19.40 / 71.00 & 12.85 / 68.63 \\ MCTS, (c=4, r=3) & 79.59 / 86.30 & 39.05 / 77.86 & 21.61 / 72.28 & 14.96 / 70.22 \\ \hline RS, (c=3, r=0) & 80.06 / 86.69 & 39.25 / 78.55 & 21.54 / 72.44 & 14.37 / 70.32 \\ MCTS, (c=3, r=0) & 79.39 / 85.86 & 39.15 / 78.02 & 21.56 / 72.26 & 13.61 / 69.42 \\ \hline RS, (c=3, r=1) & 80.06 / 86.42 & 39.15 / 78.35 & 21.90 / 72.01 & 15.20 / 70.09 \\ MCTS, (c=3, r=1) & 79.94 / 86.37 & 39.23 / 78.37 & 22.15 / 72.34 & 14.93 / 70.00 \\ \hline RS, (c=3, r=2) & 77.52 / 84.27 & 36.20 / 76.86 & 19.52 / 70.96 & 12.48 / 68.68 \\ MCTS, (c=3, r=2) & 77.89 / 84.85 & 38.89 / 78.02 & 21.51 / 71.93 & 13.97 / 69.58 \\ \hline RS, (c=4, r=1/2) & 78.70 / 85.48 & 36.68 / 76.78 & 20.20 / 70.71 & 13.57 / 68.45 \\ MCTS, (c=4, r=1/2/3) & 79.97 / 86.51 & 39.31 / 78.22 & 22.19 / 72.14 & 15.89 / 70.36 \\ \hline RS, (c=3/4, r=0’t/12/3) & 79.10 / 85.72 & 37.62 / 77.59 & 20.98 / 71.28 & 13.99 / 68.65 \\ MCTS, (c=3/4, r=0’t/12/3) & 79.42 / 85.84 & 39.37 / 78.38 & 22.24 / 72.48 & 15.20 / 70.26 \\ \hline \end{tabular}
\end{table}
Table 11: Evaluation results comparing preference data without common prefixes (RS) and with common prefixes (MCTS). We show results for different training data configurations for \(k=4\). Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**training k=4**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline RS, (c=4, r=1) & 79.24 / 85.78 & 37.49 / 77.50 & 20.54 / 72.06 & 13.78 / 70.23 \\ MCTS, (c=4, r=1) & 79.48 / 85.92 & 39.16 / 78.11 & 20.39 / 72.04 & 14.11 / 70.10 \\ \hline RS, (c=4, r=2) & 78.86 / 85.74 & 38.56 / 76.02 & 21.66 / 72.34 & 14.71 / 70.00 \\ MCTS, (c=4, r=2) & 79.68 / 86.36 & 39.22 / 78.20 & 22.43 / 72.64 & 15.75 / 70.61 \\ \hline RS, (c=4, r=3) & 76.74 / 83.92 & 35.02 / 76.25 & 19.40 / 71.00 & 12.85 / 68.63 \\ MCTS, (c=4, r=3) & 79.59 / 86.30 & 39.05 / 77.86 & 21.61 / 72.28 & 14.96 / 70.22 \\ \hline RS, (c=3, r=0) & 80.06 / 86.69 & 39.25 / 78.55 & 21.54 / 72.44 & 14.37 / 70.32 \\ MCTS, (c=3, r=0) & 79.39 / 85.86 & 39.15 / 78.02 & 21.56 / 72.26 & 13.61 / 69.42 \\ \hline RS, (c=3, r=1) & 80.06 / 86.42 & 39.15 / 78.35 & 21.90 / 72.01 & 15.20 / 70.09 \\ MCTS, (c=3, r=1) & 79.94 / 86.37 & 39.23 / 78.37 & 22.15 / 72.34 & 14.93 / 70.00 \\ \hline RS, (c=3, r=2) & 77.52 / 84.27 & 36.20 / 76.86 & 19.52 / 70.96 & 12.48 / 68.68 \\ MCTS, (c=3, r=2) & 77.89 / 84.85 & 38.89 / 78.02 & 21.51 / 71.93 & 13.97 / 69.58 \\ \hline RS, (c=4, r=1/2) & 78.70 / 85.48 & 36.68 / 76.78 & 20.20 / 70.71 & 13.57 / 68.45 \\ MCTS, (c=4, r=1/2/3) & 79.97 / 86.51 & 39.31 / 78.22 & 22.19 / 72.14 & 15.89 / 70.36 \\ \hline RS, (c=3/4, r=0’t/12/3) & 79.10 / 85.72 & 37.62 / 77.59 & 20.98 / 71.28 & 13.99 / 68.65 \\ MCTS, (c=3/4, r=0’t/12/3) & 79.42 / 85.84 & 39.37 / 78.38 & 22.24 / 72.48 & 15.20 / 70.26 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Evaluation results comparing preference data without common prefixes (MS) and with common prefixes (MCTS). We show results for different training data configurations for \(k=5\). Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).





\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{4}{c}{**training k=5**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline RS, (c=3, r=0) & 79.04 / 85.87 & 39.45 / 78.30 & 20.86 / 72.23 & 13.82 / 70.45 \\ RS, (c=3, r=1) & 78.66 / 85.51 & 38.47 / 78.15 & 20.56 / 72.37 & 14.02 / 70.13 \\ RS, (c=3, r=2) & 74.60 / 82.39 & 33.59 / 75.58 & 18.46 / 70.62 & 11.23 / 68.15 \\ \hline RS, (c=4, r=1) & 79.24 / 85.78 & 37.49 / 77.50 & 20.54 / 72.06 & 13.78 / 70.23 \\ RS, (c=4, r=2) & 77.44 / 84.43 & 37.69 / 77.68 & 20.33 / 72.06 & 13.78 / 70.04 \\ RS, (c=4, r=3) & 74.09 / 82.16 & 33.38 / 75.30 & 17.65 / 70.21 & 10.73 / 67.79 \\ \hline MCTS, (c=3, r=0) & 78.30 / 85.19 & 37.86 / 77.42 & 20.52 / 71.92 & 12.81 / 69.30 \\ MCTS, (c=3, r=1) & 77.58 / 84.69 & 38.03 / 77.74 & 19.76 / 71.84 & 13.71 / 70.08 \\ MCTS, (c=3, r=2) & 75.05 / 82.63 & 34.96 / 76.33 & 18.69 / 71.25 & 11.53 / 68.67 \\ \hline MCTS, (c=4, r=1) & 79.48 / 85.92 & 39.16 / 78.11 & 20.39 / 72.04 & 14.11 / 70.10 \\ MCTS, (c=4, r=2) & 77.76 / 84.65 & 38.25 / 77.63 & 20.00 / 71.92 & 14.26 / 69.91 \\ MCTS, (c=4, r=3) & 75.65 / 82.99 & 35.03 / 75.68 & 18.80 / 71.88 & 12.36 / 68.81 \\ \hline \end{tabular}
\end{table}
Table 14: Evaluation results studying the effects of (chosen, rejected) response quality. We provide results for \(k=4\), as well as for both RS- and MCTS-based data curation methods. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{4}{c}{**training k=4**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline RS, (c=6, r=2) & 75.17 / 82.74 & 33.45 / 75.19 & 17.00 / 69.84 & 10.19 / 67.39 \\ MCTS, (c=6, r=2) & 75.70 / 83.11 & 34.06 / 75.71 & 17.07 / 70.27 & 10.46 / 67.64 \\ \hline RS, (c=6, r=3) & 77.47 / 84.44 & 34.96 / 77.22 & 20.05 / 71.16 & 13.27 / 68.39 \\ MCTS, (c=6, r=3) & 78.06 / 85.04 & 38.68 / 77.96 & 21.63 / 72.21 & 14.49 / 70.02 \\ \hline RS, (c=6, r=4) & 76.09 / 83.51 & 35.68 / 75.63 & 19.07 / 71.01 & 11.99 / 68.47 \\ MCTS, (c=6, r=4) & 76.26 / 83.58 & 35.81 / 75.63 & 18.41 / 71.05 & 11.56 / 68.64 \\ \hline RS, (c=5, r=1) & 75.13 / 82.77 & 34.17 / 75.56 & 18.48 / 70.93 & 11.93 / 68.49 \\ MCTS, (c=5, r=1) & 76.09 / 83.47 & 36.34 / 76.52 & 19.04 / 71.33 & 12.70 / 68.62 \\ \hline RS, (c=5, r=2) & 75.01 / 82.68 & 34.48 / 75.88 & 18.10 / 70.64 & 11.21 / 68.60 \\ MCTS, (c=5, r=2) & 76.57 / 83.83 & 36.03 / 76.72 & 18.75 / 70.93 & 11.62 / 68.68 \\ \hline RS, (c=5, r=3) & 76.73 / 83.89 & 36.16 / 76.62 & 19.96 / 70.97 & 12.87 / 68.15 \\ MCTS, (c=5, r=3) & 77.18 / 84.53 & 37.06 / 77.10 & 20.16 / 71.25 & 12.84 / 68.57 \\ \hline RS, (c=5, r=1/2/3) & 78.10 / 84.97 & 37.66 / 77.61 & 21.45 / 71.49 & 13.69 / 68.84 \\ MCTS, (c=5, r=1/2/3) & 78.33 / 85.17 & 40.12 / 78.34 & 23.12 / 72.32 & 15.28 / 69.73 \\ \hline RS, (c=5/6, r=1/2/3/4) & 78.48 / 85.28 & 37.82 / 77.86 & 22.73 / 72.02 & 14.39 / 69.48 \\ MCTS, (c=5/6, r=1/2/3/4) & 79.01 / 85.72 & 40.58 / 78.93 & 23.04 / 73.18 & 16.00 / 70.57 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Evaluation results comparing preference data without common prefixes (RS) and with common prefixes (MCTS). We show results for different training data configurations for \(k=6\). Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).





\begin{table}
\begin{tabular}{l c c c c} \hline  & \multicolumn{4}{c}{**training k=5**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline MCTS, (c=d, r=1) & 78.49 / 85.36 & 39.03 / 77.76 & 22.19 / 72.10 & 14.89 / 69.76 \\ MCTS, (c=d, r=(1,2)) & 79.77 / 86.31 & 38.97 / 78.03 & 21.99 / 71.91 & 15.24 / 70.26 \\ MCTS, (c=d, r=(1,2,3)) & 79.97 / 86.51 & 39.31 / 78.22 & 22.19 / 72.14 & 15.89 / 70.36 \\ \hline MCTS, (c=(3,4), r=(0,1)) & 78.60 / 85.31 & 39.01 / 77.76 & 22.90 / 71.89 & 14.01 / 69.06 \\ MCTS, (c=(3,4), r=(0,1,2)) & 79.60 / 86.05 & 39.89 / 78.71 & 22.69 / 72.70 & 16.10 / 70.60 \\ MCTS, (c=(3,4), r=(0,1,2,3)) & 79.42 / 85.84 & 39.37 / 78.38 & 22.24 / 72.48 & 15.20 / 70.26 \\ \hline \end{tabular}
\end{table}
Table 16: Evaluation results studying the effects of mixing preference pairs with different margins between the (chosen, rejected) responses. We provide results for \(k=4\) with MCTS-based data curation methods. Each \((c=n_{0},r=(n_{1},n_{2},n_{3}))\) indicates that the chosen response correctly addresses \(n_{0}\) constraints and the rejected response correctly addresses either \(n_{1}\), \(n_{2}\) or \(n_{3}\) constraints. The left score indicates the hard score (i.e., the proportion of all constraints that are satisfied by the responses).

\begin{table}
\begin{tabular}{l l l l l} \hline  & \multicolumn{4}{c}{**training k=5**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline MCTS, (c=d, r=1) & 76.56 / 83.74 & 36.08 / 76.81 & 19.44 / 71.42 & 11.88 / 68.84 \\ RS, (c=d, r=2) & 74.57 / 82.17 & 34.09 / 75.82 & 17.95 / 70.71 & 10.93 / 68.35 \\ RS, (c=d, r=3) & 72.80 / 89.07 & 31.50 / 74.37 & 16.68 / 69.94 & 10.01 / 67.19 \\ \hline RS, (c=5, r=2) & 76.32 / 83.55 & 35.70 / 76.64 & 19.46 / 71.51 & 12.61 / 69.39 \\ RS, (c=5, r=3) & 74.13 / 81.92 & 33.30 / 75.35 & 17.31 / 70.15 & 11.21 / 68.08 \\ RS, (c=5, r=4) & 72.40 / 80.42 & 30.72 / 73.75 & 16.26 / 69.31 & 9.51 / 66.76 \\ \hline MCTS, (c=(4, r=1)) & 76.71 / 83.90 & 37.47 / 72.73 & 19.49 / 71.52 & 12.37 / 69.19 \\ MCTS, (c=(4, r=2)) & 75.23 / 82.86 & 35.65 / 76.47 & 18.74 / 71.32 & 12.17 / 69.06 \\ MCTS, (c=(4, r=3)) & 74.23 / 82.16 & 32.39 / 74.63 & 17.22 / 69.78 & 9.73 / 67.25 \\ \hline MCTS, (c=(5, r=2)) & 76.59 / 83.81 & 37.81 / 77.58 & 20.06 / 72.01 & 12.93 / 69.85 \\ MCTS, (c=5, r=3) & 76.14 / 83.50 & 36.04 / 76.61 & 19.39 / 71.49 & 12.37 / 69.32 \\ MCTS, (c=5, r=4) & 74.06 / 81.84 & 32.14 / 74.62 & 17.81 / 70.37 & 10.52 / 67.89 \\ \hline \end{tabular}
\end{table}
Table 15: Evaluation results studying the effects of (chosen, rejected) response quality. We provide results for \(k=5\), as well as for both RS- and MCTS-based data curation methods. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).





\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{3}{c}{**Rejection Sampling (RS)**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline \(k=4\), (c=3, r=0) & 79.70 / 86.13 & 38.41 / 77.93 & 21.15 / 72.36 & 14.27 / 69.90 \\ \(k=5\), (c=4, r=1) & 79.51 / 86.02 & 38.66 / 78.17 & 21.48 / 72.23 & 14.70 / 70.04 \\ \(k=6\), (c=5, r=2) & 77.83 / 84.86 & 36.42 / 77.23 & 20.15 / 71.18 & 12.28 / 68.34 \\ \hline \(k=4\), (c=3, r=1) & 79.36 / 85.95 & 39.22 / 78.51 & 22.00 / 72.54 & 14.78 / 70.07 \\ \(k=5\), (c=4, r=2) & 78.58 / 85.46 & 37.70 / 77.60 & 21.42 / 71.84 & 13.61 / 69.14 \\ \(k=6\), (c=5, r=3) & 76.73 / 83.89 & 36.16 / 76.62 & 19.96 / 70.97 & 12.87 / 68.15 \\ \hline \hline \end{tabular}
\end{table}
Table 18: Evaluation results investigating the effects of training prompt difficulty (\(k\in\{4,5,6\}\)) on downstream performance for evaluation sets of varying difficulties. We provide results for RS-based preference data curation, as well as for different margins between the (chosen, rejected) responses. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & \multicolumn{3}{c}{**Monte Carlo Tree Search (MCTS)**} \\ Method & IFEval & Ours\({}_{k=4}\) & Ours\({}_{k=5}\) & Ours\({}_{k=6}\) \\ \hline \(k=4\), (c=3, r=0) & 78.94 / 85.62 & 39.00 / 78.10 & 21.60 / 72.54 & 14.40 / 70.04 \\ \(k=5\), (c=4, r=1) & 78.72 / 85.40 & 39.42 / 78.36 & 21.43 / 72.27 & 14.25 / 69.95 \\ \(k=6\), (c=5, r=2) & 78.13 / 85.05 & 38.55 / 77.88 & 21.64 / 72.23 & 14.45 / 69.81 \\ \hline \(k=4\), (c=3, r=1) & 79.12 / 85.83 & 38.81 / 78.24 & 21.72 / 72.48 & 15.66 / 70.45 \\ \(k=5\), (c=4, r=2) & 77.53 / 84.65 & 38.93 / 78.16 & 21.69 / 72.41 & 15.20 / 70.62 \\ \(k=6\), (c=5, r=3) & 77.18 / 84.53 & 37.06 / 77.10 & 20.16 / 71.25 & 12.84 / 68.57 \\ \hline \hline \end{tabular}
\end{table}
Table 19: Evaluation results investigating the effects of training prompt difficulty (\(k\in\{4,5,6\}\)) on downstream performance for evaluation sets of varying difficulties. We provide results for MCTS-based preference data curation, as well as for different margins between the (chosen, rejected) responses. Each \((c=n_{1},r=n_{2})\) indicates that the chosen response correctly addresses \(n_{1}\) constraints and the rejected response correctly addresses \(n_{2}\) constraints. The left score indicates the hard score (i.e., the proportion of responses that get _all_ constraints correct), and the right score indicates the soft score (i.e., the proportion of all constraints that are satisfied by the responses).