MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training
Authors: Hui Huang, Jiaheng Liu, Yancheng He, Shilong Li, Bing Xu, Conghui Zhu, Muyun Yang, Tiejun Zhao
Accepted at: acl25

MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training

Hui Huang1*, Jiaheng Liu2*, Yancheng He1*, Shilong Li3, Bing Xu1, Conghui Zhu1, Muyun Yang1, Tiejun Zhao1
1Faculty of Computing, Harbin Institute of Technology, 2Nanjing University 3School of Artificial Intelligence, Beijing University of Posts and Telecommunications
huanghui@stu.hit.edu.cn, yangmuyun@hit.edu.cn

Abstract
Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods1.

1 Introduction
Large Language Models (LLMs) have made remarkable advancements and are being wildly applied across various domains (Zhao et al., 2024; Wu et al., 2024a; He et al., 2024c; Liu et al., 2025; He et al., 2024b; Zhang et al., 2025). The instruction-following ability is fundamental and important, as it enables LLMs to generate appropriate responses to given instructions and solve corresponding tasks (OpenAI et al., 2024). While recent LLMs perform comparatively well on simple instructions, their response quality to complex instructions with elaborate constraints often falls under expectation, with some of the constraints omitted (He et al., 2024a; Jiang et al., 2024b), which hinders their application in more real-world complex scenarios.

Figure 1: An illustrative comparison between our method and Self-Reward. Note that Self-Reward cannot create effective contrast for complex instruction-following, resulting in suboptimal optimization.

To enhance the complex instruction following, the core challenge is the scarcity of high-quality complex instruction data (Lou et al., 2024). Most existing instruction datasets are constructed based on existing NLP datasets or question-answering websites with simple constraints (Wang et al., 2023; Taori et al., 2023; Lian et al., 2023; Longpre et al., 2023). To cope with the scarcity of complex instruction data, previous work such as Evol-Instruct (Xu et al., 2024), Conifer (Sun et al., 2024a), and Self-Correct (Palmeira Ferraz et al., 2024) have been proposed to construct complex instructions and responses. However, these methods typically rely on a high-performance proprietary model (e.g., GPT-4) to distill the complex instruction-following ability, which is expensive and can not be scaled up in real-world applications.

Recently, the research community has paid attention to self-alignment, to break the data bottleneck without relying on a stronger model (Wang et al., 2024; Zhang et al., 2024a). Self-Reward
(Yuan et al., 2024) proposes to utilize the model itself to both generate responses and evaluate the results, which can be incorporated into DPO training. ISHEEP (Liang et al., 2024) proposes an automatic loop to self-assess and self-filter instruction data. *Despite their effectiveness, these methods are targeted at general instruction-following ability. The self-alignment of complex instruction-following ability remains unexplored.*

In this paper, to address the above limitations, we propose a novel Multi-granularity Self-Contrastive Training framework (MuSC) in Figure 1, which mainly comprises the following components:

1) **Coarse-grained Contrast: Constraint-aware Preference Data Construction.** To improve the model’s comprehension of constraint-level distinctions, we construct preference pairs that reflect the disparities in constraint fulfillment. We achieve this by breaking down each complex instruction into atomic constraints and selectively omitting a subset to form negative instructions. The chosen response, derived from the original instruction, is paired with the rejected response, generated from the negative instruction, as a contrastive pair. Notably, no external models are utilized in this construction process.

2) **Fine-grained Contrast: Token-aware Preference Optimization.** For complex instructions, the responses often involve multiple tokens that contribute differently to fulfilling the instruction’s constraints. Therefore, we introduce a token-aware optimization framework that integrates dynamic token-level weights based on the model’s confidence. By focusing on tokens that deviate from the constraints, this approach effectively identifies and corrects tokens where the model fails to satisfy the instruction’s requirements, leading to more contextually appropriate responses.

Moreover, we need to mention that our MuSC can be applied on both pre-existing complex instruction datasets, or newly generated instruction datasets created by data synthesis methods (e.g., Self-Instruct (Wang et al., 2022)).

Our contribution can be summarized as follows:

• We propose a novel Multi-granularity Self-Contrastive Training (MuSC) framework, which creates effective contrast on both coarse and fine granularity, to enhance the complex instruction following abilities.
• For coarse-grained contrast, we construct constraint-aware preference data with instruction decomposition-recombination. For fine-grained contrast, we adopt dynamic token-level weight with confidence guidance for better preference optimization.
• We evaluate our framework on open-source LLMs, and achieve significant improvements on both complex and general instruction following benchmarks, without the help of a larger model or human supervision.

2 Related Work

Complex Instruction-Following. As one of the cores of LLM intelligence, how to improve the model’s instruction-following capability is important. The earliest works, such as Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and Camel (Wang et al., 2023), used instruction data generated by proprietary models to supervise fine-tuning of open-source models, significantly enhancing their instruction-following capabilities. However, these methods mainly focus on general instruction following, while complex instruction following still remains challenging. To cope with this challenge, a lot of methods (Yin et al., 2023; Lou et al., 2023; He et al., 2024a; Sun et al., 2024b; Chen et al., 2024b; Dong et al., 2024) have been proposed to construct complex instruction data. The earliest work is Evol-Instruct (Xu et al., 2024), which proposed to utilize GPT-4 to expand the instructions from both depth and width, thereby generating complex instructions and corresponding constraints. Conifer (Sun et al., 2024a) proposed a progressive learning strategy designed to help smaller models incrementally enhance their abilities.

Self-Alignment. Self-alignment refers to aligning the model to human preference without relying on a more advanced model or external supervision. As an early study, Self-Rewarding (Yuan et al., 2024) proposed the model itself to both generate responses and evaluate the results. Following this work, many works (Liu et al., 2024; Chen et al., 2024b,a; Pang et al., 2024; Meng et al., 2024) are conducted to obtain supervision data by the model itself. Meta-Rewarding (Wu et al., 2024b) advanced the concept by improving the model’s instruction and evaluation capabilities simultaneously. Liu et al. (2024) employed diverse prompts to guide the model to generate various responses. Despite the progress these methods have made, they all target general instruction following. For
Figure 2: The pipeline of our proposed MuSC. The process starts with constraint-aware preference data construction, which includes instruction decomposition, constraint dropout, instruction recombination and response generation. Next, the token-aware DPO is performed based on calibrated confidence to achieve token-level alignment.

complex instructions with multiple constraints, the response will be lengthy and multi-facet, resulting in challenges for the self-evaluation process.

3 Approach

The pipeline of MuSC is shown in Figure 2.

3.1 Constraint-aware Preference Data

Reinforcement-learning methods, such as PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2024b), have achieved notable success in LLM optimization. Research has shown that learning from negative samples is significantly more efficient than learning solely from positive samples (Yang et al., 2024b). However, these methods are limited by the need for high-quality preference data, which is particularly scarce for complex instructions.

To construct effective preference data for complex instruction following, we propose a novel data construction method, with the following steps1:

1. Instruction Decomposition: A complex instruction is typically a combination of multiple atomic constraints. We decompose the complex instruction into individual atomic constraints, denoted as Cons.

2. Constraint Dropout: From the decomposed constraints Cons, we randomly eliminate \( \alpha\% \) of the constraints to form Cons_{drop}.

3. Instruction Recombination: We recombine both the original and the dropped constraints Cons and Cons_{drop}, to create chosen and rejected instructions: Ins and Ins_{drop}.

4. Response Generation: Based on Ins and Ins_{drop}, we generate the chosen response Resp and the rejected response Resp_{drop}.

Previous research has suggested that the construction of effective preference pairs for optimization is non-trivial (Ivison et al., 2024). Our data construction pipeline is guided by three principles:

• Negativity: The rejected response should deviate from the instruction by omitting some constraints. Our method generates the rejected instruction based on corrupted constraints, ensuring that the rejected response deviates from the original complex instruction.

• Consistency: The rejected response should reside within the model’s decoding space (Guo

1Please refer to Appendix A.1 for implementation details.
et al., 2024). In our method, the rejected instruction is simply a recombination of the original instructions, ensuring the response falls within the decoding space, which is crucial for the optimization process.

• Contrastiveness: Chosen and rejected responses should be with a rational edit distance, to form an effective contrast (Jiang et al., 2024a). By reconstructing both chosen and rejected instructions using the same method, we ensure that the derived samples do not deviate too far from each other.

With constructed data satisfying both negativity, consistency and contrastiveness, we form a solid foundation for effective alignment. Moreover, our method does not require a stronger model or human supervision, ensuring its scalability.

Our self-construction method can be applied in different scenarios. On one hand, it can be directly applied on pre-existing complex instruction dataset. On the other hand, if there is no existing complex queries, we can adapt the Self-Instruct (Wang et al., 2022) method by first generating constraints and then generating instructions. In that case, the decomposition step can be omitted.

3.2 Token-aware Preference Optimization

A well-known issue with DPO is its uniform treatment of all tokens in both chosen and rejected examples (Wu et al., 2023; Cao et al., 2024; Li et al., 2024). However, different tokens within responses carry varying significance. Especially in scenarios involving complex instructions, the responses tend to be lengthy and multi-facet. On one hand, not all tokens in the rejected response are erroneous and should be disapproved. On the other hand, chosen response may also contain tokens that fail to meet specific constraints, therefore should not be unanimously approved.

Despite previous researchers have explored fine-grained supervision signals, the signals either come from a stronger model (Cao et al., 2024; Li et al., 2024) or human annotation (Wu et al., 2023; Lightman et al., 2023). However, in our case, it is difficult for the model to provide accurate supervision for its own response, especially when dealing with multifaceted instructions and the evaluation is at token-level. Therefore, we propose Confidence-Guided Token-aware DPO, which obtains token-level supervision based on model confidence.

3.2.1 Preliminary: Token-level DPO
Direct Preference Optimization (DPO) (Rafailov et al., 2024b) proposes a direct optimization objective that satisfies the optimal preference policy without using a reward model:

\[
\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) =
- E_{(x, y^w, y^l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w | x)}{\pi_{ref}(y_w | x)} \right) - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{ref}(y_l | x)} \right],
\]

where \( \pi_\theta \) and \( \pi_{ref} \) represent the policy model and the reference model, respectively.

Subsequently, based on the theories of Levine (2018), Rafailov et al. (2024a) derived the form of DPO in token-level Markov Decision Process\(^2\), where dynamic weight can be easily integrated for different tokens\(^3\), with the loss function as follows:

\[
\mathcal{L}_{TDPO}(\pi_\theta, D) =
- \mathbb{E}_{(\tau_w, \tau_l) \sim D} \log \sigma (\beta \sum_{t=0}^{N-1} r_t^w \log \frac{\pi_\theta(a_t^w | s_t^w)}{\pi_{ref}(a_t^w | s_t^w)} \\
- \beta \sum_{t=0}^{M-1} r_t^l \log \frac{\pi_\theta(a_t^l | s_t^l)}{\pi_{ref}(a_t^l | s_t^l)} ),
\]

where \( \tau^w \) and \( \tau^l \) represent the winning and losing trajectories, with \( N \) and \( M \) as the token numbers, and \( r_t \) represents the weight for the \( t \)-th token.

3.2.2 Calibrated Confidence as Token Weight
While Section 3.2.1 provide theoretical support for token-level DPO, it is non trivial to derive token-level supervision. In this work, we propose to use the calibrated confidence as supervision.

Given an instruction \( x \), we obtain the entropy of probability distribution over target vocabulary of size \( V \) at each decoding step as the weights:

\[
\mathrm{Ent}(y_t|x^w, \theta) = - \sum_{v=1}^V p(y_t^v|logp(y_t^v),
\]

where \( p(y_t) \) represents the conditional distribution \( p(y_t|x, y_{<t}, \theta) \), and \( \theta \) represents model parameters. If the majority of the probability mass is concentrated on a limited number of vocabulary words, it indicates that the model is confident and the token is more likely to be aligned with the instruction (Fomicheva et al., 2020). Conversely, if the probabilities resemble a uniform distribution, the resulting token is expected to be misaligned.

\footnotetext{2Please refer to Appendix C.1 for more details.}
\footnotetext{3Please refer to Appendix C.2 for the mathematical proof of the dynamic token weight in preference optimization}
While there are other attributes that could also impact the confidence score (such as fluency), in this work, we want to focus our optimization on instruction alignment. Therefore, we apply calibration to the entropy to derive the token-level supervision for chosen and rejected samples respectively:

\[
r_t = \left\{ \begin{array}{ll}
\mathrm{Ent}(y_t|x^w, \theta)/\mathrm{Ent}(y_t|x^l, \theta), & \text{for } y_t \text{ in } y^w, \\
\mathrm{Ent}(y_t|x^l, \theta)/\mathrm{Ent}(y_t|x^w, \theta), & \text{for } y_t \text{ in } y^l,
\end{array} \right.
\]

\[
r_t = \min(\Gamma, r_t),
\] (4)

where the chosen sample \((x^w, y^w)\) refers to (Ins, Resp), and the rejected sample \((x^l, y^l)\) refers to (Ins_{drop}, Resp_{drop}), and \( \Gamma \) is an upper-bound to avoid extreme cases to disrupt training, and we set \( \Gamma \) as 2 in this work.

The rationale is straightforward: for a given token in the response, if it exhibits high confidence under \( x^w \) and low confidence under \( x^l \), this suggests that the token aligns well with \( x^w \) but does not fit \( x^l \), potentially reflecting the dropped constraint. Therefore, the token requires a larger weight if it is in the rejected response (or a smaller weight if it is in the chosen response).

Calibrated confidence guided token-aware DPO allows for more targeted optimization, focusing on tokens that highlight the constraint mismatch on complex instructions, instead of unanimously optimizing all tokens, thereby improving the efficiency of complex instruction alignment.

4 Experiments

4.1 Set-up

Models. We conduct experiments on two models: LLaMA-3-8B-Instruct (Dubey et al., 2024) and Qwen2-7B-Instruct (Yang et al., 2024a). Both models have undergone alignment to possess fundamental instruction-following ability.

Setting. The experiments are carried out in two distinct settings:

1. Pre-Existing Instructions (PreInst): We leverage pre-existing complex instructions as a starting point for the model. We randomly select 2,000 instructions from the dataset of WizardLM (Xu et al., 2024).
2. Self-Generated Instructions (SelfInst): In this setting, we generate instructions using the Self-Instruct method (Wang et al., 2022), based on 10 high-quality samples from Qin et al. (2024) as in-context examples. Compared with PreInst, this setting is more challenging as we need to construct the complex instructions from scratch.

Evaluation. We mainly perform evaluations on three complex instruction-following benchmarks: CF Bench (Zhang et al., 2024b), Follow-Bench (Jiang et al., 2024b) and ComplexBench (Wen et al., 2024). We also conduct evaluations on one general instruction benchmark: AlpacaEval2 (Dubois et al., 2024). Note that all benchmarks require GPT-4 for judgment, and we use GPT-4o-0513\(^4\) as the evaluator for all of them.

Baselines. We mainly compared our method against the following self-alignment methods:

• Self-Reward (Yuan et al., 2024): This method leverages the model to first generate multiple responses and then construct rewards.
• Self-Reward + BSM: Based on Self-Reward, this method performs fine-grained evaluation based on BSM (Saha et al., 2024).
• Self-Correct (Palmeira Ferraz et al., 2024): This method generates initial output and then corrects it to construct preference data.
• ISHEEP (Liang et al., 2024): This method self-creates additional instruction-output pair, which are filtered for supervised fine-tuning.

4.2 Main Results

As demonstrated in Table 1, our proposed MuSC achieves significant improvement across both complex and general instruction-following benchmarks. The improvement is consistent among different settings, verifying its scalability. By creating preference data with both constraint-aware and token-aware contrast, the model effectively learns to address all constraints lying in the instructions.

The results of Self-Reward underperform our method, even with the help of branched evaluation (Saha et al., 2024). This is because of the limited evaluation capability of the model, especially when evaluating its own response to complex instructions. Moreover, as different responses generated from the same model to the same instruction typically do not vary significantly, it is difficult to create effective contrast samples with real negativity.

\footnotetext{4platform.openai.com/docs/models/gpt-4o}
<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th rowspan="2">Method</th>
    <th colspan="3">CF-Bench</th>
    <th colspan="2">FollowBench</th>
    <th colspan="2">ComplexBench</th>
    <th colspan="2">AlpacaEval2</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>HSR</th>
    <th>SSR</th>
    <th>Overall</th>
    <th>LC (%)</th>
    <th>Avg. Len</th>
  </tr>
  <tr>
    <td colspan="11"><i>Results on LLaMA-3-8B-Instruct</i></td>
  </tr>
  <tr>
    <td rowspan="6">SelfInst</td>
    <td>LLaMA-3-8B-Instruct</td>
    <td>0.64</td>
    <td>0.24</td>
    <td>0.34</td>
    <td>62.39</td>
    <td>73.07</td>
    <td>61.49</td>
    <td>21.07</td>
    <td>1702</td>
  </tr>
  <tr>
    <td>Self-Reward</td>
    <td>0.65</td>
    <td>0.26</td>
    <td>0.35</td>
    <td>61.20</td>
    <td>72.22</td>
    <td>62.45</td>
    <td>19.21</td>
    <td>1824</td>
  </tr>
  <tr>
    <td>Self-Reward w/ BSM</td>
    <td>0.68</td>
    <td>0.28</td>
    <td>0.39</td>
    <td>64.30</td>
    <td>73.84</td>
    <td>64.13</td>
    <td>19.03</td>
    <td>1787</td>
  </tr>
  <tr>
    <td>Self-Reward w/ GPT-4</td>
    <td>0.66</td>
    <td>0.25</td>
    <td>0.37</td>
    <td>62.18</td>
    <td>73.34</td>
    <td>64.05</td>
    <td>19.55</td>
    <td>1767</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.52</td>
    <td>0.20</td>
    <td>0.27</td>
    <td>54.38</td>
    <td>67.19</td>
    <td>55.91</td>
    <td>7.97</td>
    <td>1919</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.60</td>
    <td>0.29</td>
    <td>0.40</td>
    <td>62.77</td>
    <td>72.86</td>
    <td>62.67</td>
    <td>22.00</td>
    <td>1707</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.70</b></td>
    <td><b>0.32</b></td>
    <td><b>0.44</b></td>
    <td><b>66.71</b></td>
    <td><b>74.84</b></td>
    <td><b>65.98</b></td>
    <td><b>23.87</b></td>
    <td><b>1708</b></td>
  </tr>
  <tr>
    <td rowspan="6">PreInst</td>
    <td>Self-Reward</td>
    <td>0.66</td>
    <td>0.27</td>
    <td>0.37</td>
    <td>60.88</td>
    <td>72.17</td>
    <td>62.03</td>
    <td>19.93</td>
    <td>1789</td>
  </tr>
  <tr>
    <td>Self-Reward w/ BSM</td>
    <td>0.68</td>
    <td>0.29</td>
    <td>0.40</td>
    <td>63.96</td>
    <td>73.78</td>
    <td>64.3</td>
    <td>20.98</td>
    <td>1829</td>
  </tr>
  <tr>
    <td>Self-Reward w/ GPT-4</td>
    <td>0.66</td>
    <td>0.26</td>
    <td>0.37</td>
    <td>64.02</td>
    <td>73.26</td>
    <td>63.52</td>
    <td>18.02</td>
    <td>1804</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.60</td>
    <td>0.23</td>
    <td>0.32</td>
    <td>60.11</td>
    <td>70.94</td>
    <td>60.79</td>
    <td>6.20</td>
    <td>1593</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.67</td>
    <td>0.29</td>
    <td>0.40</td>
    <td>63.54</td>
    <td>73.21</td>
    <td>62.92</td>
    <td>20.23</td>
    <td>1703</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.56</td>
    <td>0.20</td>
    <td>0.26</td>
    <td>50.06</td>
    <td>66.48</td>
    <td>53.93</td>
    <td>10.00</td>
    <td>1079</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.69</b></td>
    <td><b>0.30</b></td>
    <td><b>0.42</b></td>
    <td><b>66.90</b></td>
    <td><b>75.11</b></td>
    <td><b>64.73</b></td>
    <td><b>23.74</b></td>
    <td><b>1631</b></td>
  </tr>
  <tr>
    <td colspan="11"><i>Results on Qwen2-7B-Instruct</i></td>
  </tr>
  <tr>
    <td rowspan="6">SelfInst</td>
    <td>Qwen2-7B-Instruct</td>
    <td>0.74</td>
    <td>0.36</td>
    <td>0.49</td>
    <td>59.81</td>
    <td>71.69</td>
    <td>67.24</td>
    <td>15.53</td>
    <td>1688</td>
  </tr>
  <tr>
    <td>Self-Reward</td>
    <td>0.75</td>
    <td>0.38</td>
    <td>0.50</td>
    <td>55.36</td>
    <td>69.71</td>
    <td>66.98</td>
    <td>16.81</td>
    <td>1756</td>
  </tr>
  <tr>
    <td>Self-Reward w/ BSM</td>
    <td>0.75</td>
    <td>0.38</td>
    <td>0.50</td>
    <td>57.83</td>
    <td>70.53</td>
    <td>67.02</td>
    <td>16.94</td>
    <td>1710</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.67</td>
    <td>0.28</td>
    <td>0.38</td>
    <td>51.98</td>
    <td>67.89</td>
    <td>64.41</td>
    <td>14.01</td>
    <td>1497</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.76</td>
    <td>0.40</td>
    <td>0.52</td>
    <td>57.01</td>
    <td>69.88</td>
    <td>67.32</td>
    <td>16.99</td>
    <td>1619</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.78</b></td>
    <td><b>0.42</b></td>
    <td><b>0.54</b></td>
    <td><b>62.60</b></td>
    <td><b>72.57</b></td>
    <td><b>69.39</b></td>
    <td><b>20.08</b></td>
    <td><b>1595</b></td>
  </tr>
  <tr>
    <td rowspan="6">PreInst</td>
    <td>Self-Reward</td>
    <td>0.75</td>
    <td>0.37</td>
    <td>0.49</td>
    <td>56.45</td>
    <td>70.00</td>
    <td>66.45</td>
    <td>15.98</td>
    <td>1796</td>
  </tr>
  <tr>
    <td>Self-Reward w/ BSM</td>
    <td>0.75</td>
    <td>0.37</td>
    <td>0.49</td>
    <td>58.02</td>
    <td>70.62</td>
    <td>67.43</td>
    <td>17.17</td>
    <td>1764</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.66</td>
    <td>0.28</td>
    <td>0.37</td>
    <td>49.47</td>
    <td>66.35</td>
    <td>64.32</td>
    <td>14.46</td>
    <td>1737</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.77</td>
    <td>0.41</td>
    <td>0.52</td>
    <td>55.52</td>
    <td>69.62</td>
    <td>67.13</td>
    <td>16.52</td>
    <td>1627</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.72</td>
    <td>0.35</td>
    <td>0.46</td>
    <td>47.36</td>
    <td>64.67</td>
    <td>65.89</td>
    <td>9.52</td>
    <td>979</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.79</b></td>
    <td><b>0.44</b></td>
    <td><b>0.55</b></td>
    <td><b>62.73</b></td>
    <td><b>73.09</b></td>
    <td><b>70.00</b></td>
    <td><b>20.29</b></td>
    <td><b>1613</b></td>
  </tr>
</table>

Table 1: Experiment results of different groups of methods on instruction following benchmarks. For more detailed results on each benchmark, please refer to Appendix D.

The improvement of I-SHEEP also underperforms, likely due to its reliance on supervised fine-tuning for optimization. Previous research also suggests that learning from negative samples is more effective than learning solely from positive ones (Yang et al., 2024b). The results of Self-Correct degrades a large margin, which might be due to the inability of the model for self-correction on complex instructions (Palmeira Ferraz et al., 2024).

On general instruction benchmarks, our method also achieves significant improvement. This aligns with the previous research, which suggests that the improvement on complex instruction-following is beneficial for the overall instruction-following ability (Xu et al., 2024; Elmadany et al., 2023).

5 Analysis

5.1 MuSC on SFT model

In Section 4, our main experiments are conducted on the Instruction-version models. To exclude the influence of an initial preference optimization process, we apply our method on SFT models.

Specifically, we selected two SFT-versions of LLaMA models, LLaMA-3-8B-UltraChat-200K and LLaMA-3-8B-Tulu-330K5. Both models have gone through and only through SFT process on open-sourced datasets. As shown in Table 2, our proposed MuSC can improve both the complex and general instruction-following ability of SFT models by a large margin. Notice we only apply 2K samples when performing preference optimization, which is roughly 1% of the amount of SFT data.

5https://huggingface.co/Magpie-Align
<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th rowspan="2">Method</th>
    <th colspan="3">CF-Bench</th>
    <th colspan="2">FollowBench</th>
    <th colspan="2">ComplexBench</th>
    <th colspan="2">AlpacaEval2</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>HSR</th>
    <th>SSR</th>
    <th>Overall</th>
    <th>LC (%)</th>
    <th>Avg. Len</th>
  </tr>
  <tr>
    <td rowspan="5">LLaMA-3-8B-Ultrachat-200K</td>
    <td>Self-Reward w/BSM</td>
    <td>0.54</td>
    <td>0.18</td>
    <td>0.25</td>
    <td>33.64</td>
    <td>51.37</td>
    <td>44.89</td>
    <td>5.94</td>
    <td>861</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.58</td>
    <td>0.2</td>
    <td>0.28</td>
    <td>39.24</td>
    <td>56.82</td>
    <td>53.69</td>
    <td>9.64</td>
    <td>1099</td>
  </tr>
  <tr>
    <td>PreInst</td>
    <td>0.32</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>18.59</td>
    <td>36.88</td>
    <td>36.38</td>
    <td>3.47</td>
    <td>575</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.57</td>
    <td>0.19</td>
    <td>0.26</td>
    <td>42.22</td>
    <td>58.87</td>
    <td>52.95</td>
    <td>10.35</td>
    <td>1283</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.66</b></td>
    <td><b>0.25</b></td>
    <td><b>0.34</b></td>
    <td><b>48.93</b></td>
    <td><b>61.54</b></td>
    <td><b>56.24</b></td>
    <td><b>11.13</b></td>
    <td><b>1112</b></td>
  </tr>
  <tr>
    <td rowspan="5">LLaMA-3-8B-Tulu-330K</td>
    <td>Self-Reward w/BSM</td>
    <td>0.56</td>
    <td>0.20</td>
    <td>0.27</td>
    <td>36.26</td>
    <td>54.66</td>
    <td>54.52</td>
    <td>6.00</td>
    <td>992</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>0.60</td>
    <td>0.22</td>
    <td>0.30</td>
    <td>41.71</td>
    <td>58.68</td>
    <td>55.84</td>
    <td>9.71</td>
    <td>1307</td>
  </tr>
  <tr>
    <td>PreInst</td>
    <td>0.31</td>
    <td>0.08</td>
    <td>0.10</td>
    <td>24.67</td>
    <td>39.94</td>
    <td>40.44</td>
    <td>3.21</td>
    <td>623</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>0.61</td>
    <td>0.22</td>
    <td>0.30</td>
    <td>43.16</td>
    <td>59.21</td>
    <td>58.20</td>
    <td>10.77</td>
    <td>1402</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td><b>0.70</b></td>
    <td><b>0.28</b></td>
    <td><b>0.40</b></td>
    <td><b>51.26</b></td>
    <td><b>63.94</b></td>
    <td><b>62.98</b></td>
    <td><b>13.20</b></td>
    <td><b>1341</b></td>
  </tr>
</table>

Table 2: Experiment results of different methods on supervised fine-tuned models.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">CF-Bench</th>
    <th colspan="2">AlpacaEval2</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC</th>
    <th>Len</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>0.64</td>
    <td>0.24</td>
    <td>0.34</td>
    <td>21.07</td>
    <td>1702</td>
  </tr>
  <tr>
    <td>Perplexity</td>
    <td>0.70</td>
    <td>0.32</td>
    <td>0.43</td>
    <td>22.99</td>
    <td>1744</td>
  </tr>
  <tr>
    <td>PMI</td>
    <td>0.69</td>
    <td>0.29</td>
    <td>0.41</td>
    <td>21.92</td>
    <td>1713</td>
  </tr>
  <tr>
    <td>KLDiv</td>
    <td>0.69</td>
    <td>0.31</td>
    <td>0.42</td>
    <td>21.86</td>
    <td>1686</td>
  </tr>
  <tr>
    <td><b>Entropy</b></td>
    <td><b>0.71</b></td>
    <td><b>0.34</b></td>
    <td><b>0.44</b></td>
    <td><b>23.74</b></td>
    <td><b>1631</b></td>
  </tr>
  <tr>
    <td>w/o calib</td>
    <td>0.68</td>
    <td>0.28</td>
    <td>0.39</td>
    <td>21.49</td>
    <td>1735</td>
  </tr>
</table>

Table 3: Results of different confidence metrics as the fine-grained weight on LLaMA-3-8B-Instruct.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">CF-Bench</th>
    <th colspan="2">AlpacaEval2</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC</th>
    <th>Len</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>0.64</td>
    <td>0.24</td>
    <td>0.34</td>
    <td>21.07</td>
    <td>1702</td>
  </tr>
  <tr>
    <th colspan="6">Results on SelfInst</th>
  </tr>
  <tr>
    <td>DPO<sub>MuSC</sub></td>
    <td><b>0.70</b></td>
    <td><b>0.32</b></td>
    <td><b>0.44</b></td>
    <td><b>23.87</b></td>
    <td><b>1708</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.68</td>
    <td>0.28</td>
    <td>0.39</td>
    <td>21.49</td>
    <td>1735</td>
  </tr>
  <tr>
    <td>SimPO<sub>MuSC</sub></td>
    <td><b>0.70</b></td>
    <td><b>0.31</b></td>
    <td><b>0.42</b></td>
    <td><b>22.92</b></td>
    <td><b>1716</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.67</td>
    <td>0.30</td>
    <td>0.40</td>
    <td>19.59</td>
    <td>1637</td>
  </tr>
  <tr>
    <td>IPO<sub>MuSC</sub></td>
    <td><b>0.73</b></td>
    <td><b>0.37</b></td>
    <td><b>0.48</b></td>
    <td><b>23.18</b></td>
    <td><b>1650</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.70</td>
    <td>0.33</td>
    <td>0.44</td>
    <td>20.42</td>
    <td>1686</td>
  </tr>
  <tr>
    <th colspan="6">Results on PreInst</th>
  </tr>
  <tr>
    <td>DPO<sub>MuSC</sub></td>
    <td><b>0.69</b></td>
    <td><b>0.30</b></td>
    <td><b>0.42</b></td>
    <td><b>23.74</b></td>
    <td><b>1631</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.69</td>
    <td>0.30</td>
    <td>0.41</td>
    <td>20.96</td>
    <td>1671</td>
  </tr>
  <tr>
    <td>SimPO<sub>MuSC</sub></td>
    <td><b>0.69</b></td>
    <td><b>0.31</b></td>
    <td><b>0.42</b></td>
    <td><b>23.28</b></td>
    <td><b>1625</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.67</td>
    <td>0.31</td>
    <td>0.41</td>
    <td>22.02</td>
    <td>1570</td>
  </tr>
  <tr>
    <td>IPO<sub>MuSC</sub></td>
    <td><b>0.68</b></td>
    <td><b>0.32</b></td>
    <td><b>0.44</b></td>
    <td><b>24.56</b></td>
    <td><b>1601</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>0.68</td>
    <td>0.30</td>
    <td>0.42</td>
    <td>22.04</td>
    <td>1531</td>
  </tr>
</table>

Table 4: Results of our method on different preference optimization methods on Llama-3-8B-Instruct.

This again verifies that learning from negative samples is comparatively more efficient than learning solely from positive samples.

5.2 The Influence of Confidence Metrics

Various confidence metrics have been established in the domain of LLM (Geng et al., 2024). This section aims to provide a comparison across different metrics as the token weight, under the framework of MuSC. We include the following metrics:

• Perplexity: The exponential of the negative log-likelihood of the token.

• PMI: Pointwise Mutual Information as defined in Takayama and Arase (2019).

• KIDiv: Kullback–Leibler divergence between the token probability distribution under chosen and rejected instructions.

As shown in Table 3, entropy-based token weight achieves the best result among all metrics, verifying its effectiveness. Both the perplexity and PMI-based score underperforms, as they only consider the probabilities of the selected tokens instead of the whole distribution, leading to biased evaluation\footnote{As the model would always select (one of) the tokens with the highest probability at each step, relying only on the the selected tokens for evaluation will result in overconfidence.}. KLDiv-based score also underperforms, this is because KLDiv is essentially a distance measurement instead of a confidence measurement, which is not adapted to our scenario.

We also experiment with removing the calibration proposed in Section 3.2.2. As can be seen, calibration is important for the effectiveness of confidence-based fine-grained weight, as it can exclude other factors such as fluency, thereby focus-
<table>
  <tr>
    <th>Method</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>HumanEval</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="5">Results on Meta-LLaMA-3-8B-Instruct</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>68.29</td>
    <td>79.08</td>
    <td>59.15</td>
    <td>51.63</td>
  </tr>
  <tr>
    <td>MuSC<sub>SelfInst</sub></td>
    <td>68.24</td>
    <td><b>79.23</b></td>
    <td><b>62.20</b></td>
    <td><b>52.42</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>67.97</td>
    <td>78.62</td>
    <td>56.71</td>
    <td>50.83</td>
  </tr>
  <tr>
    <td>MuSC<sub>PreInst</sub></td>
    <td>68.17</td>
    <td>77.41</td>
    <td><b>62.80</b></td>
    <td><b>52.10</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>67.80</td>
    <td>77.71</td>
    <td><b>60.98</b></td>
    <td>51.62</td>
  </tr>
  <tr>
    <th colspan="5">Results on Qwen2-7B-Instruct</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>70.76</td>
    <td>83.09</td>
    <td>75.61</td>
    <td>57.37</td>
  </tr>
  <tr>
    <td>MuSC<sub>SelfInst</sub></td>
    <td>70.63</td>
    <td><b>84.09</b></td>
    <td>75.61</td>
    <td><b>57.58</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>70.81</td>
    <td>82.94</td>
    <td>71.34</td>
    <td>56.27</td>
  </tr>
  <tr>
    <td>MuSC<sub>PreInst</sub></td>
    <td>70.46</td>
    <td><b>84.38</b></td>
    <td>75.78</td>
    <td><b>57.66</b></td>
  </tr>
  <tr>
    <td>w/o fgct</td>
    <td>70.63</td>
    <td><b>84.53</b></td>
    <td>73.78</td>
    <td>57.24</td>
  </tr>
</table>

Table 5: Experiment results of our methods on fundamental capability benchmarks. Bolded results denote improvements, while grayed results denote degradation.

ing the contrast on instruction alignment.

5.3 Can MuSC Scale to Other xPO Method?
While our experiments primarily focus on the DPO method, the overall framework is not limited to a single preference optimization technique. Therefore, we extended our framework to two additional xPO methods: SimPO (Meng et al., 2024) and IPO (Azar et al., 2024). We utilized the same constructed preference data and applied the entropy-based score as the token-level supervision\footnote{Please refer to Appendix A.2 for detailed implementation.}. Note that for “w/o fgct”, we just remove the fine-grained contrast in the MuSC method. In Table 4, our approach has shown consistent improvements across both SimPO and IPO, validating its scalability.

While IPO showed the best performance among different xPO methods, these performance differences were not statistically significant. Our method is not limited to DPO; instead, it’s a general framework applicable to most xPO-style methods. Given the popularity of DPO, we primarily report our experiments based on DPO in this work.

5.4 Is Fundamental Capability Harmed?
Previous research have proposed that during the alignment process, the fundamental ability of model may suffer degradation due to alignment tax (Ouyang et al., 2022). Therefore, we evaluated our proposed method on three fundamental capability benchmarks: MMLU (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021) and HumanEval (Chen et al., 2021).

![Experiment Results on Llama3-8B-Instruct.](page_1012_120_377_246.png)
(a) Experiment Results on Llama3-8B-Instruct.

![Experiment Results on Qwen2-7B-Instruct.](page_1012_370_377_246.png)
(b) Experiment Results on Qwen2-7B-Instruct.

Figure 3: Different statistical indicators during the training steps, upon different methods.

As shown in Table 5, while the results of naive MuSC may suffer slight degradation, the introduction of fine-grained contrast mitigates the degradation, which verifies the significance of token-level supervision. Under the scenario of complex instruction, the response is lengthy and should not be uniformly approved or disapproved. With fine-grained supervision, we focus the optimization on complex instruction alignment, thereby avoiding the disruption of other capabilities.

5.5 The Variation of Statistical Indicators
As different methods start from the same group of instructions, training statistics can be comparable as a quality indicator. Therefore, we display the variation of both loss and reward margins between chosen and rejected samples on different methods.

As shown in Figure 3, Self-Reward presents both higher loss and lower reward margin during training. This is because there is too much noise between the chosen and rejected pairs, making the model unable to capture the contrast related to constraint alignment. Notice that both indicators start to change drastically at the 2nd epoch, which means the learned knowledge cannot transfer between different samples at the 1st epoch. On the other hand, the optimization based on MuSC converges faster and more smoothly, verifying the effectiveness of the contrast samples.

Comparing the indicators of MuSC with and
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">CFBench</th>
    <th colspan="2">AlpacaEval2</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC</th>
    <th>Len</th>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-Inst</td>
    <td>0.59</td>
    <td>0.21</td>
    <td>0.28</td>
    <td>10.10</td>
    <td>1507</td>
  </tr>
  <tr>
    <td>+MuSC</td>
    <td><b>0.69</b></td>
    <td><b>0.32</b></td>
    <td><b>0.41</b></td>
    <td><b>11.44</b></td>
    <td>1346</td>
  </tr>
  <tr>
    <td>Qwen2.5-3B-Inst</td>
    <td>0.72</td>
    <td>0.34</td>
    <td>0.45</td>
    <td>27.44</td>
    <td>2188</td>
  </tr>
  <tr>
    <td>+MuSC</td>
    <td><b>0.76</b></td>
    <td><b>0.41</b></td>
    <td><b>0.52</b></td>
    <td><b>30.64</b></td>
    <td>1873</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst</td>
    <td>0.82</td>
    <td>0.49</td>
    <td>0.60</td>
    <td>46.05</td>
    <td>1792</td>
  </tr>
  <tr>
    <td>+MuSC</td>
    <td><b>0.85</b></td>
    <td><b>0.56</b></td>
    <td><b>0.67</b></td>
    <td><b>47.61</b></td>
    <td>1770</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst</td>
    <td>0.87</td>
    <td>0.60</td>
    <td>0.70</td>
    <td>48.63</td>
    <td>1785</td>
  </tr>
  <tr>
    <td>+MuSC</td>
    <td><b>0.89</b></td>
    <td><b>0.65</b></td>
    <td><b>0.75</b></td>
    <td><b>49.34</b></td>
    <td>1772</td>
  </tr>
</table>

Table 6: Experiment results of MuSC on Qwen2.5-Instruct of different model sizes.

without fine-grained supervision, it can also be noticed that with the introduction of fine-grained supervision, both indicators converge faster. The introduction of token-level supervision is a cheap yet effective method to improve xPO methods.

For the analysis of different instruction noising schemes and the visualization of token-level weight, please refer to Appendix B and D.

5.6 Can MuSC Scale among Model Sizes

To verify the scalability of our proposed method on models with larger sizes, we conducted extensive experiments on Qwen-2.5-Instruct models ranging from 1.5B to 32B parameters. As shown in Table 6, the results demonstrate consistent improvements across both smaller and larger models, confirming that our method is effective regardless of model size. This robust scalability is attributed to our approach being inherently self-improving and not requiring external models, which holds significant promise for advancing LLM capabilities.

6 Conclusion

In this work, we propose a Multi-granularity Contrastive Training framework, to perform complex instruction alignment without the introduction of external supervision. Experiment results show our method achieves significant improvement on instruction alignment benchmarks, surpassing previous self-improvement methods by a large margin. In the future, we will apply our MuSC on the improvement of other capabilities, such as long-form generation, multi-modal generation, etc.

Limitations

Our work still has some limitations: 1) Due to time and resource limitation, we did not validate our method on larger models, such as LLaMA-70B. Validation on larger models could help to improve the credibility of our method. 2) We mainly relied on GPT-4 based LLM-as-a-Judge to evaluate the results. Despite it has been verified that GPT-4 based evaluation achieves high correlation with human evaluators (Zheng et al., 2023), incorporating human evaluation would further improve the credibility of our methods. 3) We did not scale our method to PPO-based optimization methods, which are also wildly used in recent alignment practice. The application of our method on traditional RL methods could further improve its utility.

Ethical Considerations

Since the aligned data is generated based on the Instruct version model, the chances of generating toxic content are low. To further validate this, we randomly selected 200 samples in total—100 samples for Llama-3-8B-Instruct and 100 samples for Qwen-2-7B-Instruct—and employed three graduate students to annotate them. The results showed no instances of toxic instructions or responses.

However, there is potential risk of knowledge forgetting related to safety alignment. To address this, we recommend incorporating additional safe alignment samples during MuSC training to reinforce the model’s safety-related knowledge. We will include these recommendations in the ethical considerations section of the revised manuscript.

Acknowledgements

This work is supported by National Natural Science Foundation of China (62276077, 62376075, 62376076), Jiangsu Science and Technology Major Project (BG2024031) and Nanjing University AI & AI for Science Funding (2024300540).

References

Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR.

Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. 2024. Beyond sparse rewards: Enhancing reinforcement learning with language model critique in text generation. Preprint, arXiv:2401.07382.

Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham,
and Min Lin. 2024a. Bootstrapping language models with dpo implicit rewards. arXiv preprint arXiv:2406.09760.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzi, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.

Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. 2024b. Dog-instruct: Towards premium instruction-tuning data via text-grounded instruction wrapping. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4125–4135.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yann Dubois, Balázs Galambosi, Percy Liang, and Tat-sunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.

AbdelRahim Elmadyan, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for Arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.

Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555.

Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6577–6595, Mexico City, Mexico. Association for Computational Linguistics.

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalmans, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. 2024. Direct language model alignment from online ai feedback. Preprint, arXiv:2402.04792.

Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024a. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. Preprint, arXiv:2404.15846.

Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunze Zhou, Jiaqing Liang, and Yanghua Xiao. 2024b. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196.

Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. 2024c. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Preprint, arXiv:2406.09279.
Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, and Wei Wang. 2024a. Bridging and modeling correlations in pairwise data for direct preference optimization. Preprint, arXiv:2408.07471.

Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024b. Follow-Bench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4667–4688, Bangkok, Thailand. Association for Computational Linguistics.

Siyu Lai, Hui Huang, Dong Jing, Yufeng Chen, Jinan Xu, and Jian Liu. 2021. Saliency-based multi-view mixed language training for zero-shot cross-lingual classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 599–610, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Sergey Levine. 2018. Reinforcement learning and control as probabilistic inference: Tutorial and review. ArXiv, abs/1805.00909.

Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, and Bo Zheng. 2024. 2d-dpo: Scaling direct preference optimization with 2-dimensional supervision. Preprint, arXiv:2410.19720.

Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and “Teknium”. 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/openOrca.

Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenha Huang, and Jiajun Zhang. 2024. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm. Preprint, arXiv:2408.08072.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.

Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Xioming Wang, Jiulong Shan, Meng Cao, and Lijie Wen. 2024. Direct large language model alignment through self-rewarding contrastive prompt distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9688–9712, Bangkok, Thailand. Association for Computational Linguistics.

Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. 2025. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.17407.

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. Preprint, arXiv:2301.13688.

Renze Lou, Kai Zhang, Jian Xie, Yuxuan Sun, Janice Ahn, Hanzi Xu, Yu Su, and Wenpeng Yin. 2023. Muffin: Curating multi-faceted instructions for improving instruction following. In The Twelfth International Conference on Learning Representations.

Renze Lou, Kai Zhang, and Wenpeng Yin. 2024. Large language model instruction following: A survey of progresses and challenges. Computational Linguistics, pages 1–10.

Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS).

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayman Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madeleine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jiang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee woo Jun, Tomer Kaftan, Lukas Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal
Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Sattarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaragya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yu, Kevin Yu, Qingling Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Junting Zhuang, William Zhuk, and Barrett Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.

Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Tagyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. LLM self-correction with De-CRIM: Decompose, critique, and refine for enhanced following of instructions with multiple constraints. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7773–7812, Miami, Florida, USA. Association for Computational Linguistics.

Xianghe Pang, Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. 2024. Self-alignment of large language models via monopolylogue-based social scene simulation. arXiv preprint arXiv:2402.05699.

Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models.

Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024a. From r to q*: Your language model is secretly a q-function. Preprint, arXiv:2404.12358.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024b. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.

Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. 2024. Branch-solve-merge improves large language model evaluation and generation. Preprint, arXiv:2310.15123.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347.

Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao-hua Dong, Ran Lin, and Ruohui Huang. 2024a. Conifer: Improving complex constrained instruction-following ability of large language models. arxiv preprint arXiv:2404.02823.

Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Wayne Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Gai. 2024b. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750.

Junya Takayama and Yuki Arase. 2019. Relevant and informative response generation using pointwise mutual information. In Proceedings of the First Workshop on NLP for Conversational AI, pages 133–138, Florence, Italy. Association for Computational Linguistics.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources. In Advances in Neural Information Processing Systems, volume 36, pages 74764–74786. Curran Associates, Inc.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.

Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu Zhu, Xiang-Bo Mao, Sitaram Asur, Na, and Cheng. 2024. A comprehensive survey of llm alignment techniques: Rhf, rlaif, ppo, dpo and more. Preprint, arXiv:2407.16216.

Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. 2024. Benchmarking complex instruction-following with multiple constraints composition. Preprint, arXiv:2407.03978.

Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. 2024a. A comparative study on reasoning patterns of openai’s o1 model. arXiv preprint arXiv:2410.13639.

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. 2024b. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge. Preprint, arXiv:2407.19594.

Zequi Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. In Thirty-seventh Conference on Neural Information Processing Systems.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024a. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Zhen Yang, Ming Ding, Tinglin Huang, Yukuo Cen, Junshuai Song, Bin Xu, Yuxiao Dong, and Jie Tang. 2024b. Does negative sampling matter? a review with insights into its theory and applications. Preprint, arXiv:2402.17238.

Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. 2023. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation. arXiv preprint arXiv:2305.14327.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. Preprint, arXiv:2401.10020.

Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, et al. 2025. Codecriticbench: A holistic code critique benchmark for large language models. arXiv preprint arXiv:2502.16614.

Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Shiyu Huang, Deheng Ye, Wenbo Ding, Yaodong Yang, and Yu Wang. 2024a. A survey on self-play methods in reinforcement learning. Preprint, arXiv:2408.01072.

Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yuqing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, and Zenan Zhou. 2024b. Cfbench: A comprehensive constraints-following benchmark for llms. Preprint, arXiv:2408.01122.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2024. A survey of large language models. Preprint, arXiv:2303.18223.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics.

Brian D. Ziebart. 2010. Modeling purposeful adaptive behavior with the principle of maximum causal entropy.
A  Implementation Details

A.1  Token-aware Preference Data Construction

For all models that used for preference data construction, we adopt the following prompts presented in Figure 5, 7, 6, 8, 9 and 8. We set the temperate as 0.5 for all steps to ensure diversity. To ensure the data quality, we filter instructions with less than three constraints and more than ten constraints. We also filter preference pairs with the same chosen and rejected responses.

For constraint dropout, we set the dropout ratio \( \alpha \) to 0.3 to ensure that negative examples are sufficiently negative, meanwhile not deviate too much from the positive sample. We avoid dropout on the first constraint, as it often establishes the foundation for the task, and dropping the first one would make the recombined instruction overly biased.

A.2  Token-aware Preference Optimization

Our experiments are based on Llama-Factory (Zheng et al., 2024), and we trained all models on 8 A100-80GB SXM GPUs. The per_device_train_batch_size was set to 1, gradient_accumulation_steps to 8, leading to an overall batch size as 64, and we used bfloat16 precision. The learning rate is set as 1e-06 with cosine decay, and each model is trained with 2 epochs. We set \( \beta \) to 0.2 for all DPO-based experiments, \( \beta \) as 3.0 and \( \gamma \) as 1.0 for all SimPO-based experiments, \( \beta \) as 1.0 for all IPO-based methods referring to the settings of Meng et al. (2024). All of the final loss includes 0.1x of the SFT loss.

B  The Influence of Noising Scheme

Previous work has proposed various noising strategies in contrastive training (Lai et al., 2021). While we leverage Constraint-Dropout for negative sample generation, to make a fair comparison with other strategies, we implement the following strategies: 1) Constraint-Negate: Leverage the model to generate an opposite constraint. 2) Constraint-Substitute: Substitute the constraint with an unrelated constraint.

![Line graph showing the variation of results on CFBench and AlpacaEval2 with different dropout ratios.](page_320_1012_808_246.png)

Figure 4: The variation of results on CFBench and AlpacaEval2 with different dropout ratios.

As shown in Table 7, both the negation and substitution applied on the constraints would lead to performance degradation. After a thoroughly inspect of the derived data, we realize that instructions derived from both dropout and negation would lead to instructions too far from the positive instruction, therefore the derived negative response would also deviate too much from the original instruction. An effective negative sample should fulfill both negativity, consistency and contrastiveness, and constraint-dropout is a simple yet effective method to achieve this goal.

We also provide the variation of the results on CF-Bench and AlpacaEval2 with different constraint dropout ratios. As shown in Figure 4, with the dropout ratio increased from 0.1 to 0.5, the results on CF-Bench firstly increases and then slightly decreases. On the other hand, the results on AlpacaEval2 declines a lot with a higher dropout ratio. This denotes that a suboptimal droout ratio is essential for the balance between complex instruction and general instruction following abilities, with lower ratio may decrease the effectiveness of general instruction alignment, while higher ratio may be harmful for complex instruction alignment. Finally, we set the constraint dropout ratio as 0.3 in all experiments.
<table>
  <tr>
    <th rowspan="2">Scenario</th>
    <th rowspan="2">Method</th>
    <th colspan="5">Meta-LLaMA-3-8B-Instruct</th>
    <th colspan="5">Qwen-2-7B-Instruct</th>
  </tr>
  <tr>
    <th>CF-Bench CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC%</th>
    <th>Avg.Len</th>
    <th>CF-Bench CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC%</th>
    <th>Avg.Len</th>
  </tr>
  <tr>
    <td rowspan="4">PreInst</td>
    <td>baseline</td>
    <td>0.64</td>
    <td>0.24</td>
    <td>0.34</td>
    <td>21.07</td>
    <td>1702</td>
    <td>0.74</td>
    <td>0.36</td>
    <td>0.49</td>
    <td>15.53</td>
    <td>1688</td>
  </tr>
  <tr>
    <td>Constraint-Drop</td>
    <td><b>0.71</b></td>
    <td><b>0.34</b></td>
    <td><b>0.45</b></td>
    <td><b>23.43</b></td>
    <td>1682</td>
    <td><b>0.79</b></td>
    <td><b>0.43</b></td>
    <td><b>0.54</b></td>
    <td><b>19.31</b></td>
    <td>1675</td>
  </tr>
  <tr>
    <td>Constraint-Negate</td>
    <td>0.68</td>
    <td>0.28</td>
    <td>0.39</td>
    <td>18.94</td>
    <td>1688</td>
    <td>0.75</td>
    <td>0.37</td>
    <td>0.50</td>
    <td>17.82</td>
    <td>1663</td>
  </tr>
  <tr>
    <td>Constraint-Substitute</td>
    <td>0.68</td>
    <td>0.28</td>
    <td>0.40</td>
    <td>20.48</td>
    <td>1706</td>
    <td>0.76</td>
    <td>0.39</td>
    <td>0.51</td>
    <td>19.05</td>
    <td>1709</td>
  </tr>
</table>

Table 7: Experiment results of different noising strategies on instruction following benchmarks.

C Mathematical Derivations

C.1 Preliminary: DPO in the Token Level Markov Decision Process

As demonstrated in Rafailov et al. (2024a), the Bradley-Terry preference model in token-level Markov Decision Process (MDP) is:

\[
p^* \left( \tau^w \succeq \tau^l \right) = \frac{\exp \left( \sum_{i=1}^N r \left( s_i^w, a_i^w \right) \right)}{\exp \left( \sum_{i=1}^N r \left( s_i^w, a_i^w \right) \right) + \exp \left( \sum_{i=1}^M r \left( s_i^l, a_i^l \right) \right)}
\] (5)

The formula using the \( Q \)-function to measure the relationship between the current timestep and future returns:

\[
Q^*(s_t, a_t) = \begin{cases}
r(s_t, a_t) + \beta \log \pi_{ref}(a_t|s_t) + V^*(s_{t+1}), & \text{if } s_{t+1} \text{ is not terminal} \\
r(s_t, a_t) + \beta \log \pi_{ref}(a_t|s_t), & \text{if } s_{t+1} \text{ is terminal}
\end{cases}
\] (6)

Derive the total reward obtained along the entire trajectory based on the above definitions:

\[
\sum_{t=0}^{T-1} r(s_t, a_t) = \sum_{t=0}^{T-1} (Q^*(s_t, a_t) - \beta \log \pi_{ref}(a_t|s_t) - V^*(s_{t+1}))
\] (7)

Combining this with the fixed point solution of the optimal policy (Ziebart, 2010; Levine, 2018), we can further derive:

\[
\sum_{t=0}^{T-1} r(s_t, a_t) = Q^*(s_0, a_0) - \beta \log \pi_{ref}(a_0|s_0) + \sum_{t=1}^{T-1} (Q^*(s_t, a_t) - V^*(s_t) - \beta \log \pi_{ref}(a_t|s_t))
\]
\[
= Q^*(s_0, a_0) - \beta \log \pi_{ref}(a_0|s_0) + \sum_{t=1}^{T-1} \beta \log \frac{\pi^*(a_t|s_t)}{\pi_{ref}(a_t|s_t)}
\]
\[
= V^*(s_0) + \sum_{t=0}^{T-1} \beta \log \frac{\pi^*(a_t|s_t)}{\pi_{ref}(a_t|s_t)}
\] (10)

By substituting the above result into Eq. 5, we can eliminate \( V^*(S_0) \) in the same way as removing the partition function in DPO, obtaining the Token-level BT model that conforms to the MDP:

\[
p_{\pi^*} \left( \tau^w \succeq \tau^l \right) = \sigma \left( \sum_{t=0}^{N-1} \beta \log \frac{\pi^* \left( a_t^w | s_t^w \right)}{\pi_{ref} \left( a_t^w | s_t^w \right)} - \sum_{t=0}^{M-1} \beta \log \frac{\pi^* \left( a_t^l | s_t^l \right)}{\pi_{ref} \left( a_t^l | s_t^l \right)} \right)
\] (11)

Thus, the Loss formulation of DPO at the Token level is:

\[
\mathcal{L}(\pi_\theta, \mathcal{D}) = -\mathbb{E}_{(\tau_w, \tau_l) \sim \mathcal{D}} \left[ \log \sigma \left( \left( \sum_{t=0}^{N-1} \beta \log \frac{\pi^* \left( a_t^w | s_t^w \right)}{\pi_{ref} \left( a_t^w | s_t^w \right)} \right) - \left( \sum_{t=0}^{M-1} \beta \log \frac{\pi^* \left( a_t^l | s_t^l \right)}{\pi_{ref} \left( a_t^l | s_t^l \right)} \right) \right) \right]
\] (12)
C.2 Proof of Dynamic Token Weight in Token-level DPO

In classic RLHF methods, the optimization objective is typically formulated with an entropy bonus, expressed through a Kullback-Leibler (KL) divergence constraint as follows:

\[
\max_{\pi_\theta} \mathbb{E}_{a_t \sim \pi_\theta(\cdot|s_t)} \sum_{t=0}^T [r(s_t, a_t) - \beta \mathcal{D}_{KL}[\pi_\theta(a_t|s_t)||\pi_{ref}(a_t|s_t)]]
\] (13)

\[
= \max_{\pi_\theta} \mathbb{E}_{a_t \sim \pi_\theta(\cdot|s_t)} \sum_{t=0}^T [r(s_t, a_t) - \beta \log \frac{\pi_\theta(a_t|s_t)}{\pi_{ref}(a_t|s_t)}]
\] (14)

This can be further rewritten by separating the terms involving the reference policy and the entropy of the current policy:

\[
\max_{\pi_\theta} \mathbb{E}_{a_t \sim \pi_\theta(\cdot|s_t)} \sum_{t=0}^T (r(s_t, a_t) + \beta \log \pi_{ref}(a_t|s_t)) + \beta \mathcal{H}(\pi_\theta)|s_0 \sim \rho(s_0)]
\]

When the coefficient \( \beta \) is treated as a variable that depends on the timestep \( t \) (Li et al., 2024), the objective transforms to:

\[
\max_{\pi_\theta} \mathbb{E}_{a_t \sim \pi_\theta(\cdot|s_t)} \sum_{t=0}^T [(r(s_t, a_t) + \beta_t \log \pi_{ref}(a_t|s_t)) - \beta_t \log \pi_\theta(a_t|s_t)]
\] (15)

where \( \beta_t \) depends solely on \( a_t \) and \( s_t \). Following the formulation by Levine (2018), the above expression can be recast to incorporate the KL divergence explicitly:

\[
\max_{\pi_\theta} \mathbb{E}_{a_t \sim \pi_\theta(\cdot|s_t)} \sum_{t=0}^T [(r(s_t, a_t) + \beta_t \log \pi_{ref}(a_t|s_t)) - \beta_t \log \pi_\theta(a_t|s_t)]
\] (16)

where the value function \( V(s_t) \) is defined as:

\[
V(s_t) = \beta_t \log \int_A [\exp \frac{r(s_t, a_t)}{\beta_t} \pi_{ref}(a_t|s_t)] da_t
\] (17)

When the KL divergence term is minimized—implying that the two distributions are identical—the expectation in Eq. (14) reaches its maximum value. Therefore, the optimal policy satisfies:

\[
\pi_\theta(a_t|s_t) = \frac{1}{\exp(V(s_t))} \exp \left( \frac{r(s_t, a_t) + \beta_t \log \pi_{ref}(a_t|s_t)}{\beta_t} \right)
\] (18)

Based on this relationship, we define the optimal Q-function as:

\[
Q^*(s_t, a_t) = \begin{cases}
r(s_t, a_t) + \beta_t \log \pi_{ref}(a_t|s_t) + V^*(s_{t+1}), & \text{if } s_{t+1} \text{ is not terminal} \\
r(s_t, a_t) + \beta_t \log \pi_{ref}(a_t|s_t), & \text{if } s_{t+1} \text{ is terminal}
\end{cases}
\] (19)

Consequently, the optimal policy can be expressed as:

\[
\pi_\theta(a_t|s_t) = e^{(Q(s_t, a_t) - V(s_t))/\beta_t}
\] (20)

By taking the natural logarithm of both sides, we obtain a log-linear relationship for the optimal policy at the token level, which is expressed with the optimal Q-function:

\[
\beta_t \log \pi_\theta(a_t | s_t) = Q_\theta(s_t, a_t) - V_\theta(s_t)
\] (21)
This equation establishes a direct relationship between the scaled log-ratio of the optimal policy to the reference policy and the reward function \( r(s_t, a_t) \):

\[
\beta_t \log \frac{\pi^*(a_t \mid s_t)}{\pi_{\text{ref}}(a_t \mid s_t)} = r(s_t, a_t) + V^*(s_{t+1}) - V^*(s_t)
\] (22)

Furthermore, following the definition by Rafailov et al. (2024a)'s definition, two reward functions \( r(s_t, a_t) \) and \( r'(s_t, a_t) \) are considered equivalent if there exists a potential function \( \Phi(s) \), such that:

\[
r'(s_t, a_t) = r(s_t, a_t) + \Phi(s_{t+1}) - \Phi(s_t)
\] (23)

This equivalence implies that the optimal advantage function remains invariant under such transformations of the reward function. Consequently, we derive why the coefficient *beta* in direct preference optimization can be variable, depending on the state and action, thereby allowing for more flexible and adaptive policy optimization in RLHF frameworks.

D  Detailed Experiment Results

In this section, we presented detailed experiment results which are omitted in the main body of this paper due to space limitation. The detailed experiment results of different methods on ComplexBench, FollowBench and AlpacaEval2 are presented in Table 8, 10 and 9. The detailed results for the ablative studies of confidence metrics is presented in Table 11. The detailed results for the ablative studies of confidence metrics is presented in Table 7. We also present a case study in Table 12, which visualize the token-level weight derived from calibrated confidence score.

<table>
  <tr>
    <th rowspan="2">Scenario</th>
    <th rowspan="2">Method</th>
    <th colspan="4">ComplexBench</th>
    <th colspan="4">Qwen2-7B-Instruct</th>
  </tr>
  <tr>
    <th colspan="2">Meta-Llama3-8B-Instruct</th>
    <th colspan="2">Qwen2-7B-Instruct</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Overall</th>
    <th>And</th>
    <th>Chain Selection</th>
    <th>Overall</th>
    <th>And</th>
    <th>Chain</th>
    <th>Selection</th>
  </tr>
  <tr>
    <td rowspan="6">SelfInst</td>
    <td>baseline</td>
    <td>61.49</td>
    <td>57.22</td>
    <td>57.22</td>
    <td>53.55</td>
    <td>67.24</td>
    <td>62.58</td>
    <td>62.58</td>
    <td>58.97</td>
  </tr>
  <tr>
    <td>Self-Reward w/ BSM</td>
    <td>62.45</td>
    <td>58.23</td>
    <td>58.23</td>
    <td>54.07</td>
    <td>66.98</td>
    <td>63.02</td>
    <td>63.02</td>
    <td>57.75</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>64.13</td>
    <td>58.01</td>
    <td>58.01</td>
    <td>56.62</td>
    <td>67.02</td>
    <td>62.37</td>
    <td>62.37</td>
    <td>57.85</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>64.05</td>
    <td>59.44</td>
    <td>59.44</td>
    <td>54.78</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>55.91</td>
    <td>49.85</td>
    <td>49.85</td>
    <td>46.91</td>
    <td>64.41</td>
    <td>59.59</td>
    <td>59.59</td>
    <td>55.04</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td>62.67</td>
    <td>57.79</td>
    <td>57.79</td>
    <td>54.63</td>
    <td>67.32</td>
    <td>61.95</td>
    <td>61.95</td>
    <td>59.64</td>
  </tr>
  <tr>
    <td rowspan="6">PreInst</td>
    <td>Self-Reward w/ BSM</td>
    <td>65.98</td>
    <td>63.45</td>
    <td>63.45</td>
    <td>55.96</td>
    <td>69.39</td>
    <td>65.45</td>
    <td>65.45</td>
    <td>59.79</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>62.03</td>
    <td>56.94</td>
    <td>56.94</td>
    <td>53.09</td>
    <td>66.45</td>
    <td>61.37</td>
    <td>61.37</td>
    <td>57.64</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>64.30</td>
    <td>57.58</td>
    <td>57.58</td>
    <td>56.47</td>
    <td>67.43</td>
    <td>62.95</td>
    <td>62.95</td>
    <td>58.41</td>
  </tr>
  <tr>
    <td>ISHEEP</td>
    <td>63.52</td>
    <td>59.08</td>
    <td>59.08</td>
    <td>53.91</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>60.79</td>
    <td>55.65</td>
    <td>55.65</td>
    <td>52.02</td>
    <td>64.32</td>
    <td>60.16</td>
    <td>60.16</td>
    <td>54.63</td>
  </tr>
  <tr>
    <td>MuSC</td>
    <td>62.92</td>
    <td>56.37</td>
    <td>56.37</td>
    <td>54.83</td>
    <td>67.13</td>
    <td>64.45</td>
    <td>64.45</td>
    <td>57.54</td>
  </tr>
  <tr>
    <td></td>
    <td>SFT</td>
    <td>53.93</td>
    <td>45.77</td>
    <td>45.77</td>
    <td>44.09</td>
    <td>65.89</td>
    <td>60.16</td>
    <td>60.16</td>
    <td>57.39</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSC</td>
    <td><b>64.73</b></td>
    <td><b>59.23</b></td>
    <td><b>59.23</b></td>
    <td><b>55.91</b></td>
    <td><b>70.00</b></td>
    <td><b>66.88</b></td>
    <td><b>66.88</b></td>
    <td><b>61.38</b></td>
  </tr>
</table>

Table 8: Detailed experiment results of different methods on ComplexBench.
<table>
  <tr>
    <th rowspan="2">Scenario</th>
    <th rowspan="2">Method</th>
    <th colspan="3">FollowBench</th>
    <th colspan="3">Qwen2-7B-Instruct</th>
  </tr>
  <tr>
    <th>Meta-Llama3-8B-Instruct<br>HSR</th>
    <th>SSR</th>
    <th>CSL</th>
    <th>HSR</th>
    <th>SSR</th>
    <th>CSL</th>
  </tr>
  <tr>
    <td rowspan="1">baseline</td>
    <td></td>
    <td>62.39</td>
    <td>73.07</td>
    <td>2.76</td>
    <td>59.81</td>
    <td>71.69</td>
    <td>2.46</td>
  </tr>
  <tr>
    <td rowspan="4">SelfInst</td>
    <td>Self-Reward</td>
    <td>61.20</td>
    <td>72.22</td>
    <td>2.56</td>
    <td>55.36</td>
    <td>69.71</td>
    <td>2.34</td>
  </tr>
  <tr>
    <td>w/ BSM</td>
    <td>64.30</td>
    <td>73.84</td>
    <td>2.80</td>
    <td>57.83</td>
    <td>70.53</td>
    <td>2.41</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>62.18</td>
    <td>73.34</td>
    <td>2.66</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>54.38</td>
    <td>67.19</td>
    <td>2.02</td>
    <td>51.98</td>
    <td>67.89</td>
    <td>2.16</td>
  </tr>
  <tr>
    <td></td>
    <td>ISHEEP</td>
    <td>62.77</td>
    <td>72.86</td>
    <td>2.52</td>
    <td>57.01</td>
    <td>69.88</td>
    <td>2.36</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSC</td>
    <td><b>66.71</b></td>
    <td><b>74.84</b></td>
    <td><b>2.92</b></td>
    <td><b>62.60</b></td>
    <td><b>72.57</b></td>
    <td><b>2.82</b></td>
  </tr>
  <tr>
    <td rowspan="4">PreInst</td>
    <td>Self-Reward</td>
    <td>60.88</td>
    <td>72.17</td>
    <td>2.64</td>
    <td>56.45</td>
    <td>70.00</td>
    <td>2.44</td>
  </tr>
  <tr>
    <td>w/ BSM</td>
    <td>63.96</td>
    <td>73.78</td>
    <td>2.66</td>
    <td>58.02</td>
    <td>70.62</td>
    <td>2.42</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>64.02</td>
    <td>73.26</td>
    <td>2.64</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>60.11</td>
    <td>70.94</td>
    <td>2.70</td>
    <td>49.47</td>
    <td>66.35</td>
    <td>1.98</td>
  </tr>
  <tr>
    <td></td>
    <td>ISHEEP</td>
    <td>63.54</td>
    <td>73.21</td>
    <td>2.64</td>
    <td>55.52</td>
    <td>69.62</td>
    <td>2.28</td>
  </tr>
  <tr>
    <td></td>
    <td>SFT</td>
    <td>50.06</td>
    <td>66.48</td>
    <td>2.04</td>
    <td>47.36</td>
    <td>64.67</td>
    <td>1.96</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSC</td>
    <td><b>66.90</b></td>
    <td><b>75.11</b></td>
    <td><b>2.99</b></td>
    <td><b>62.73</b></td>
    <td><b>73.09</b></td>
    <td><b>2.86</b></td>
  </tr>
</table>

Table 9: Detailed experiment results of different methods on FollowBench.

Please break down the instruction into multiple constraints and an input, following the provided examples. Do not generate any other openings, closings or other unnecessary explanations.
Examples:
**Instruction:**
{instruction_body}
**Constraints:**
(constraint_body1)
(constraint_body2)
(constraint_body3)

**Instruction:**
{instruction_body}
**Constraints:**
(constraint_body1)
(constraint_body2)
(constraint_body3)

...
Below is the instruction that needs to be broken down:
**Instruction:**
{instruction_body}
**Constraints:**

Figure 5: The prompt template used for instruction decomposition.

<table>
  <tr>
    <th rowspan="2">Scenario</th>
    <th rowspan="2">Method</th>
    <th colspan="3">AlpacaEval2</th>
  </tr>
  <tr>
    <th>Meta-Llama3-8B-Instruct<br>LC (%)</th>
    <th>WR (%)</th>
    <th>Avg. Len</th>
    <th>Qwen2-7B-Instruct<br>LC (%)</th>
    <th>WR (%)</th>
    <th>Avg. Len</th>
  </tr>
  <tr>
    <td rowspan="1">baseline</td>
    <td></td>
    <td>21.07</td>
    <td>18.73</td>
    <td>1702</td>
    <td>15.53</td>
    <td>13.70</td>
    <td>1688</td>
  </tr>
  <tr>
    <td rowspan="4">SelfInst</td>
    <td>Self-Reward</td>
    <td>19.21</td>
    <td>19.18</td>
    <td>1824</td>
    <td>16.81</td>
    <td>15.66</td>
    <td>1756</td>
  </tr>
  <tr>
    <td>w/ BSM</td>
    <td>19.03</td>
    <td>18.34</td>
    <td>1787</td>
    <td>16.94</td>
    <td>15.09</td>
    <td>1710</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>19.55</td>
    <td>18.53</td>
    <td>1767</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>7.97</td>
    <td>9.34</td>
    <td>1919</td>
    <td>14.01</td>
    <td>10.92</td>
    <td>1497</td>
  </tr>
  <tr>
    <td></td>
    <td>ISHEEP</td>
    <td>22.00</td>
    <td>19.50</td>
    <td>1707</td>
    <td>16.99</td>
    <td>14.04</td>
    <td>1619</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSC</td>
    <td><b>23.87</b></td>
    <td><b>20.91</b></td>
    <td><b>1708</b></td>
    <td><b>20.08</b></td>
    <td><b>15.67</b></td>
    <td><b>1595</b></td>
  </tr>
  <tr>
    <td rowspan="4">PreInst</td>
    <td>Self-Reward</td>
    <td>19.93</td>
    <td>19.04</td>
    <td>1789</td>
    <td>15.98</td>
    <td>15.62</td>
    <td>1796</td>
  </tr>
  <tr>
    <td>w/ BSM</td>
    <td>20.98</td>
    <td>20.75</td>
    <td>1829</td>
    <td>17.17</td>
    <td>16.21</td>
    <td>1764</td>
  </tr>
  <tr>
    <td>w/ GPT-4</td>
    <td>18.02</td>
    <td>17.74</td>
    <td>1804</td>
    <td>—</td>
    <td>—</td>
    <td>—</td>
  </tr>
  <tr>
    <td>Self-Correct</td>
    <td>6.20</td>
    <td>5.81</td>
    <td>1593</td>
    <td>14.46</td>
    <td>14.02</td>
    <td>1737</td>
  </tr>
  <tr>
    <td></td>
    <td>ISHEEP</td>
    <td>20.23</td>
    <td>17.86</td>
    <td>1703</td>
    <td>16.52</td>
    <td>13.36</td>
    <td>1627</td>
  </tr>
  <tr>
    <td></td>
    <td>SFT</td>
    <td>10.00</td>
    <td>6.22</td>
    <td>1079</td>
    <td>9.52</td>
    <td>5.25</td>
    <td>979</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSC</td>
    <td><b>23.74</b></td>
    <td><b>19.53</b></td>
    <td><b>1631</b></td>
    <td><b>20.29</b></td>
    <td><b>15.91</b></td>
    <td><b>1613</b></td>
  </tr>
</table>

Table 10: Detailed experiment results of different methods on AlpacaEval2.
<table>
  <tr>
    <th rowspan="2">Scenario</th>
    <th rowspan="2">Method</th>
    <th colspan="5">Meta-Llama-3-8B-Instruct</th>
    <th colspan="5">Qwen-2-7B-Instruct</th>
  </tr>
  <tr>
    <th>CF-Bench CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC (%)</th>
    <th>Avg. Len</th>
    <th>CF-Bench CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>LC (%)</th>
    <th>Avg. Len</th>
  </tr>
  <tr>
    <td rowspan="4">PreInst</td>
    <td>Baseline</td>
    <td>0.64</td>
    <td>0.24</td>
    <td>0.34</td>
    <td>21.07</td>
    <td>1702</td>
    <td>0.74</td>
    <td>0.36</td>
    <td>0.49</td>
    <td>15.53</td>
    <td>1688</td>
  </tr>
  <tr>
    <td>w/ perplexity</td>
    <td>0.70</td>
    <td>0.32</td>
    <td>0.43</td>
    <td>22.99</td>
    <td>1744</td>
    <td>0.79</td>
    <td>0.43</td>
    <td>0.54</td>
    <td>19.31</td>
    <td>1675</td>
  </tr>
  <tr>
    <td>w/ PMI</td>
    <td>0.69</td>
    <td>0.29</td>
    <td>0.41</td>
    <td>21.92</td>
    <td>1713</td>
    <td>0.78</td>
    <td>0.43</td>
    <td>0.55</td>
    <td>17.42</td>
    <td>1651</td>
  </tr>
  <tr>
    <td>w/ KLDiv</td>
    <td>0.69</td>
    <td>0.31</td>
    <td>0.42</td>
    <td>21.86</td>
    <td>1686</td>
    <td>0.78</td>
    <td>0.42</td>
    <td>0.54</td>
    <td>18.73</td>
    <td>1614</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ entropy</td>
    <td><b>0.71</b></td>
    <td><b>0.34</b></td>
    <td><b>0.44</b></td>
    <td><b>23.74</b></td>
    <td>1631</td>
    <td><b>0.80</b></td>
    <td><b>0.46</b></td>
    <td><b>0.58</b></td>
    <td><b>20.38</b></td>
    <td>1683</td>
  </tr>
</table>

Table 11: Detailed experiment results of different confidence metrics as the fine-grained weight.

Instruction 1: Identify the incorrect word in the given sentence: "The group acquired the skill to ski in a suitable manner." Rewrite the identified incorrect word correctly, ensuring the new sentence adheres to proper capitalization, punctuation, and grammar rules. Include at least one synonym of the incorrect word to test the model’s understanding of context and vocabulary, and ensure the synonym is used appropriately in the context of the sentence.

Chosen Response 1: The group acquired the skill to ski in a proficient manner. The incorrect word is "suitable". A synonym for "suitable" is "appropriate", but it doesn’t fit the context of the sentence. "Proficient" is a better synonym in this context as it means skilled or expert.

Rejected Response 1: The word "acquired" seems incorrect in the given sentence. A better word to use in this context would be "learned". Rewritten sentence : "The group learned the skill to ski in a suitable manner."

Instruction 2: Rewrite the given sentence in C# syntax, using an idiom or metaphor to express disbelief, and ensure the sentence maintains its original meaning while being grammatically correct in C#. The sentence to be rewritten is: "The idea that pigs can fly seems more plausible than his explanation for being late."

Chosen Response 2: csharp\n using System;\n class Program\n {\n static void Main ( string [] args )\n {\n Console.WriteLine ( "His explanation for being late is so baffling that it's akin to saying pigs can fly , making it more believable in comparison . ) \n } \n }

Rejected Response 2: Wow , it’s truly hard to believe that pigs flying could be more plausible than my explanation for being late !

Table 12: Visualization of dynamic weights derived for chosen and rejected responses, based on our proposed calibrated entropy score. We select two samples from the datasets as an illustration.

![A code template for generating instructions from input and constraints](page_1012_1342_728_180.png)

Figure 6: The prompt template used for constraint recombination.
Please follow the examples to come up with one task with more than 5 constraints. Please list the constraints first and then provide the task.
Examples:

**Task:**
{instruction_body}
**Constraints:**
{constraint_body1}
{constraint_body2}
{constraint_body3}

**Task:**
{instruction_body}
**Constraints:**
{constraint_body1}
{constraint_body2}
{constraint_body3}
...

Figure 7: The prompt template used for self-instruct.

The following is a question used to apply constraint for generating an instruction. Please generate a new constraint question which specifies a constraint on the contrary.
Please generate the constraint question with the following format: **New Constraint**: [new constraint here]. Do not generate any other openings, closings or explanations.
**Original Constraint**: {constraint_body}

Figure 8: The prompt template used for constraint substitution.

The following is a question used to apply constraint for generating an instruction. Please generate a new constraint question which specifies a constraint deviates from the original one.
Please generate the constraint question with the following format: **New Constraint**: [new constraint here]. Do not generate any other openings, closings or explanations.
**Original Constraint**: {constraint_body}

Figure 9: The prompt template used for constraint negation.

Reviewer paper summaries: This paper introduces MuSC, a two-stage framework that improves complex instruction following ability of LLMs. Given a pre-existing or self-instructed complex instruction, MuSC first constructs preference pairs by feeding recombined instructions with complete or corrupted constraints into the LLM to obtain corresponding responses. Then token-level DPO is applied to optimize the model, where the token weight is its calibrated entropy.  A series of experiments show the effectiveness of MuSC.

The paper introduces a Multi-granularity Self-Contrastive Training (MuSC) framework to improve LLMs' ability to follow complex instructions. MuSC operates on two levels: coarse-grained, where it creates constraint-aware preference data; and fine-grained, where it optimizes token-level preferences based on model confidence. MuSC achieves significant improvement on both complex and general instruction-following benchmarks.

The paper proposes a Multi-granularity Self-Contrastive Training (MuSC) framework to improve the complex instruction alignment. Experiments are performed on multiple base models and with various strategies towards performance improvement. Findings show that the proposed approach surpasses previous self-improvement methods. The paper is written well and presents experiments with takeaways which could be useful to the community.
Summary of strengths: 1. The experiments are very extensive and solid, demonstrating the efficacy of MuSC with different settings, models,  and preference alignment algorithms. Moreover, the experiments on general datasets such as MMLU guarantee that the fundamental capability will not be harmed, strengthen the applicability of this framework.

2. Using calibrated entropy as the token weight in DPO is ingenious, avoiding the impact of other attributes such as fluency.

3.  The paper is well-writing and easy to read.

1. This paper proposes a novel token-level DPO by measuring the entropy of different tokens in positive and negative samples, achieving fine-grained DPO.
2. The effectiveness of the method is validated from multiple perspectives through experiments.

- The methodology for utilizing different constraint combinations for preference data construction is interesting and novel
- Experimental evaluation is thorough and reported results show that the proposed methodology improves over other self-alignment methods
Summary of weaknesses: 1. My main concern is that the first step of MuSC, i.e., the constrained-aware preference data construction, somehow lacks novelty since many previous works such as Suri [1] have used corrupted instruction to produce negative samples.

2.  As mentioned in the limitation section, some human evaluation would further improve the credibility of our methods

[1] Suri: Multi-constraint Instruction Following for Long-form Text Generation. Findings of ACL 2024.

Some experimental settings are not sufficiently thorough and the results are confusing.
1. In Table 4, there is no ablation of the final DPO used in the original paper on the Complex Instruction Following benchmark, while such ablation is conducted on the general ability test set.
2. It appears that the IPO results are better than the DPO results. Why not use IPO as the main method in Table 1?
3. Why are there two identical header rows in Table 5, and which results are the final? What does the gray color represent?

- While experiments seem to show valid trends across different model families, larger models are not evaluated and could provide better insights on the scalability of MuSC. Alternatively, evaluations on even smaller models (such as 2B) could provide a varied model size view into evaluations.
- Evaluation is mainly conducted with GPT4o-as-judge (as listed in the limitations section).
- While there are low expectations of toxic data present in the alignment data presented, it would be useful to provide an overall view. It would also be helpful to include an ethical considerations section which could speak to the potential impact of the application of the  proposed methodology.
Comments, suggestions and typos: 1. Is there any ablation results of "fgct" using DPO?

2. Could you further explain line 430-433: "Both the perplexity and PMI-based score underperform, as they only con432 sider the probability of the selected token instead of the whole distribution, leading to biased evaluation." Can you give a more clear example?

1. In the pre-existing instructions setting, the wizardlm dataset may not be multi-constrained, making it difficult to construct negative samples. Has additional filtering been conducted?
2. The effect of multiple iterations can be considered, i.e., using the optimized model to construct rejected samples again and performing iterations.

Table 5 - it would help if the authors could mention whether the grey text was intended, and why some rows are greyed while others are not

Line 527 - widely used
Overall assessment scores: 3

3

3.5