Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models

## Abstract

Abstract One core capability of large language models (LLMs) is to follow natural language instructions.
However, the issue of automatically constructing high-quality training data to enhance the complex instruction-following abilities of LLMs without manual annotation remains unresolved.
In this paper, we introduce AutoIF , the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification, requiring LLMs to generate instructions, the corresponding code to check the correctness of the instruction responses, and unit test samples to verify the code’s correctness.
Then, execution feedback-based rejection sampling can generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) training. AutoIF achieves significant improvements across three training algorithms, SFT, Offline DPO, and Online DPO, when applied to the top open-source LLMs, Qwen2 and LLaMA3, in self-alignment and strong-to-weak distillation settings. Our code is publicly available at https://github.com/QwenLM/AutoIF .

## 1 Introduction

The instruction-following ability of large language models (LLMs) refers to their capacity to understand, interpret, and execute commands given to them in natural language (Lou et al., ; OpenAI et al., ).
This ability is fundamental to contemporary LLMs as it enables them to leverage their underlying knowledge, interact intuitively with users (Ouyang et al., ), adapt to various requirements (Zhang et al., ), and perform complex tasks (Sun et al., ).
Misunderstandings in following instructions can lead to unintended outcomes, potentially resulting in severe consequences, particularly in critical scenarios (Zhou et al., ; Chang et al., ).

Although instruction following is crucial, scalable and reliable methods to enhance this capability of LLMs remain elusive.
Current efforts in this field are divided into manual annotation (Wei et al., ; Zhou et al., ; Jiang et al., ) and behavior imitation (Xu et al., ; Zhao et al., ).
Manual annotation involves annotators designing instructions and writing corresponding responses.
However, due to human cognition’s limitations, creating highly complex and diverse instructions is challenging, making the process difficult to scale.
Furthermore, accurately executing complex instructions can sometimes be difficult for humans (Sun et al., ; Cao et al., ), requiring multiple rounds of rigorous and costly validation (Wang et al., ; Wei et al., ).
On the other hand, behavior imitation aims to distill responses from more advanced LLMs (Taori et al., ; Peng et al., ) like GPT-4.
This approach limits models to the capabilities of the advanced LLMs from which they are distilled.
Moreover, even advanced LLMs can make mistakes, and the reliability of the distilled data cannot be guaranteed (Cui et al., ).
Consequently, models trained with this data may have a propensity to not follow instructions accurately (Zhou et al., ).

In this paper, we introduce AutoIF, the first scalable and reliable method for automatically generating instruction following training Data for Supervised Finetuning (SFT) or Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ).
The core idea of AutoIF is to use code to verify the correctness of following instructions.
Intuitively, if designed properly, a significant portion of instructions, such as “Keep your response under 20 characters in length” can be verified for correctness using code, as illustrated in Figure 1.
Therefore, the key components of AutoIF include (1) automatically generating instructions that can be verified by code, (2) automatically generating corresponding verification codes for these instructions, and (3) ensuring the reliability of the first two steps.
Specifically, we start by providing AutoIF with a small set of hand-written seed instructions.
Then, LLMs, not necessarily advanced ones, generate an augmented instruction set through self-instruct (Wang et al., ).
Next, LLMs write verification codes and unit test cases for each instruction.
Only the code that compiles correctly, passes the test cases, and back-translates to the original instruction is retained.
If an instruction does not have a corresponding code that can verify its correctness, it is discarded.
Finally, we employ LLMs to generate responses that either pass or fail the verification code using execution feedback-based rejection sampling (Yuan et al., ).
Responses that pass can be directly used for SFT, while pairs of passing and failing responses can be used to create chosen-rejected pairs for Direct Preference Optimization (DPO) (Rafailov et al., ) and other RLHF algorithms.
Moreover, once the instructions and verification code are determined, this process can be conducted on-policy, continually enhancing the instruction-following capabilities.

Through extensive experiments, we have demonstrated that AutoIF achieves significant improvements under three training algorithms—SFT, Offline DPO, and Online DPO—when applied to the top open-source LLMs, Qwen2-72B and LLaMA3-70B, in both self-alignment and strong-to-weak distillation settings.
In the IFEval benchmark, we achieved Loose Instruction (Acc.) rates of up to 88.0% with Qwen2-72B and 90.4% with LLaMA3-70B, marking the first instance of surpassing 90% accuracy.
In the FollowBench benchmark, these models also showed significant improvements, with increases of over 5% in the SSR metric (avg).
Additionally, they enabled Qwen2-7B and LLaMA3-8B to achieve average performance gains of over 4% in both benchmarks.
Replacing Qwen2-72B and LLaMA3-70B with the more advanced GPT-4 resulted in further substantial improvements.
We will open-source the SFT and DPO datasets constructed using AutoIF on Qwen2-72B, representing the first open-source complex instruction-following dataset at a scale of tens of thousands.

## 2 Related Works

Instruction-following capabilities are among the most essential features of LLMs (OpenAI et al., ; Lou et al., ), which are expected to precisely follow a broad and complex set of instructions.
Consequently, recent research has concentrated on evaluating LLMs’ instruction-following abilities in various contexts, such as verifiable (Zhou et al., ), compositional (Qin et al., ), format-related (Xia et al., ), refuting (Yan et al., ), and fine-grained instructions (Jiang et al., ).
However, a significant gap remains between open-source and proprietary closed-source LLMs.
Sun et al. () propose Conifer, which enhances the instruction-following capabilities of open-source LLMs through knowledge distillation from proprietary LLMs.
Wang et al. () use LLMs to encode instruction metadata and augment diverse instructions from this metadata, employing proprietary LLMs for quality control.
Both approaches, however, rely on proprietary LLMs for response distillation or judgment, which not only limits their potential but also subjects them to OpenAI’s terms of use (^1^11[https://openai.com/policies/terms-of-use](https://openai.com/policies/terms-of-use)).
In this work, we propose AutoIF, a more scalable and reliable method to enhance the instruction-following capabilities of LLMs.
AutoIF uses execution feedback from self-generated verification functions to provide supervision for instructions.
This allows for effective self-alignment and strong-to-weak distillation on open-source models, thereby narrowing the performance gap with proprietary LLMs.

Learning with Execution Feedback is a widely-used technique in automated alignment for tool use and coding (Cao et al., ).
These learning methods typically utilize execution feedback from tools such as code executors to provide supervision for specific tasks.
For instance, Le et al. () employ feedback from unit tests via code compilers to enhance code synthesis capabilities through reinforcement learning.
Similarly, Chen et al. () train LLMs to provide debugging suggestions as feedback to improve coding abilities.
Additionally, Qiao et al. () introduce Reinforcement Learning with execution feedback to enhance LLMs using execution results from tools.
Building on this learning paradigm, we propose a novel scalable oversight method that enables LLMs to autonomously generate verification functions and unit tests for natural language instructions, thereby applying execution feedback to enhance their instruction-following capabilities.

Figure: Figure 2: An Overview of AutoIF: A Two-Stage Automated Instruction-Following Data Synthesis Method.
Refer to caption: x2.png

## 3 AutoIF

We introduce AutoIF, an automated, scalable, and reliable method designed to enhance the instruction-following capabilities of LLMs.
In this section, we outline the preliminaries (Section 3.1), detail the two core components of AutoIF (Section 3.2, Section 3.3), and discuss various training strategies that can be seamlessly integrated with AutoIF (Section 3.4).

### 3.1 Preliminaries

Instruction-following Capabilities.
Following instructions is one of the most crucial skills in modern LLMs.
These models are expected to provide precise responses to queries containing complex instructions, which can be either atomic or compositional.
To evaluate the instruction-following capability of LLMs, we define a general instruction-following requirement as a specific task.
In this task, given an instruction $I={\{i_{j}\}}_{j=1}^{N}$ with $N$ specific constraints (e.g. “Please generate text in Shakespearean style, no more than 50 tokens” contains 2 constraints) and a specific query $x$, an LLM $\pi_{\theta}$ should generate precise response $y\sim\pi_{\theta}(y\mid x,I)$ adhering to the constraints.

Verifiable Instructions.
The complexity and diversity of instructions necessitate manual construction and verification for reliable supervision.
This practical challenge motivates us to focus initially on instructions that can be automatically verified through programs and code executors, also known as verifiable instructions (Zhou et al., ).
Specifically, for a given instruction $I$ and task-specific query $q$, there exists a verification function $f_{I}$ such that $f_{I}(y)$ returns true when the model’s response $y$ correctly follows the instruction.
We demonstrate that supervision of such instructions can be self-generated through scalable oversight with LLMs and execution feedback.
Extensive experiments in our work show that training on verifiable instructions significantly benefits the handling of other general instructions that are more complex but unverifiable with simple code snippets.

Method Overview.
AutoIF synthesizes high-quality instruction-following data through self-evolution, rejection sampling, and execution feedback.
As illustrated in Figure 2, AutoIF integrates automated data augmentation with quality verification processes, including automatically generated verification functions and back-translation instructions.
This approach enables a two-stage automated data synthesis at both the instruction (Section 3.2) and query levels (Section 3.3).
Additionally, we introduce three training strategies (Section 3.4) and explore two experimental settings (Section 4.1) to thoroughly evaluate the effectiveness and generalization of AutoIF.

### 3.2 Instruction Augmentation and Verification

We first develop verifiable instructions along with corresponding evaluation functions, using rejection sampling informed by execution feedback.

Seed Instruction Construction.
We start by handwriting a set of seed instructions, denoted as $D_{seed}$, ensuring that each instruction contains only a single atomic constraint (e.g., “Answer the words that begin with B”).
Detailed information on seed instructions is listed in Appendix A.

Self-Instruct.
Self-Instruct (Wang et al., ) is a straightforward and intuitive strategy for automated data augmentation that has garnered significant attention in the field of LLM reasoning (Xu et al., ; Zhao et al., ).
For each instruction in $D_{seed}$, we use an LLM to perform $K$ instruction rewrites, generating $D_{aug}$. We then combine the seed and augmented data sets to obtain an enhanced set of instructions, $D_{ins}=D_{seed}\cup D_{aug}$, and remove any duplicates.

Automated Quality Cross Verification.
Previous research has shown that relying solely on model-generated augmented instructions often leads to the inclusion of low-quality samples (Mumuni & Mumuni, ; Xie et al., ; Zheng et al., ).
Inspired by a series of tool execution studies, we employ an LLM to generate verification functions and test cases for each instruction. We use feedback from executing Python programs to ensure quality control.
Given the instruction set $D_{ins}$, the LLM $M$ employs a rejection sampling (Touvron et al., ; Yuan et al., ) to generate $K$ verification functions $f_{I}=\{f_{i}\}_{j=1}^{K}$ and test cases $c_{I}=\{c_{i}\}_{j=1}^{K}$ for each instruction $I$, resulting in the set $\{I,f_{I},c_{I}\}\in D_{ins}$.
We then cross-validate the quality of the instructions using the verification functions and test cases, ensuring they meet the following criteria:

- •
The verification function $f\in f_{I}$ can be successfully compiled by the Python executor.
- •
Each test case $c\in c_{I}$ achieves an accuracy rate greater than 0.5 across all verification functions.
- •
Each verification function $f\in f_{I}$ achieves an accuracy rate greater than 0.5 across all test cases.
- •
Each instruction includes at least one evaluation function and test case.

By adhering to these four conditions, we obtain the quality-filtered instruction set $\{I^{(2)},f_{I}^{(2)}\}\in D_{ins}^{(2)}$.

Back-translation Verification.
After the cross-validation stage, we obtained initially quality-verified verification functions and instructions.
To further ensure the consistency between instructions and verification functions, we introduce back-translation.
For a given pair $\{I^{(2)},f_{I}^{(2)}\}\in D_{ins}^{(2)}$, we use the LLM $M$ to back-translate the verification function $f\in f_{I}^{(2)}$ into instruction $I_{f}$.
We then treat $I$ as the *premise* and the back-translated instruction $I_{f}$ as the *hypothesis*.
Using the NLI model, we identify the semantic relationship between the two instructions.
The prediction can fall into one of three categories: *entailment*, *contradiction*, or *neutral*:

$$ $p_{\theta}(\cdot\mid q,q_{\text{aug}})=\operatorname{softmax}\left({ \operatorname{score}}_{\theta}(I,I_{f})\right),$ (1) $$

where $\operatorname{score}_{\theta}:\mathbb{R}^{k\times\ell_{I}}\times\mathbb{R}^{k
\times\ell_{I_{f}}}\rightarrow\mathbb{R}^{3}$ is a model dependent scoring function with parameters $\theta$. We filter out any instruction $I$ labeled as *contradiction*
to ensure the intent consistency.
Finally we obtain the set $\{I^{(3)},f_{I}^{(3)}\}\in D_{ins}^{(3)}$

Figure: Figure 3: Different training strategies that can be adapted with synthetic dataset generated by AutoIF.
Refer to caption: x3.png

### 3.3 Query Augmentation and Verification

Once we have obtained verified instructions and verification functions, we utilize them to create training data comprising queries and responses.

Query Reforming and Augmentation.
In the real-world application of modern chatbots, instructions are typically employed to generate constrained responses to user queries.
Therefore, creating high-quality instructions is merely the initial step toward achieving effective instruction-following capabilities.
To acquire authentic queries, as shown in the bottom part of Figure 2, we randomly selected $K$ user queries from ShareGPT (Chiang et al., ) for each instruction and concatenated them to construct the seed query dataset ${x,f_{I}^{(3)}}\in D_{q}$.
To further enhance the diversity and complexity of the input $x$, we utilized the LLM to generate $K$ responses $y_{x}=\{y_{i}\}_{i=1}^{K}$, resulting in $\{x,f_{I}^{{3}},y_{x}\}\in D_{q}$.

Instruction-following Verification.
Following the previous quality cross-verification process, we further employ verification functions to assess whether the augmented responses adhere to the constraints in input $x$.
Similarly, we require each response in $D_{q}$ to meet the following conditions:

- •
Each response must achieve an accuracy rate greater than 0.5 across all verification functions.
- •
Each input must include at least one verification function and one response.

Based on these rules, we obtain the set $(x^{(2)},f_{I}^{(3)},y^{(2)})\in D_{q}^{(2)}$.

Query Quality Verification.
Additionally, we observe that concatenated instructions and queries often conflict.
For instance, a high-quality response to the query “help me write a news article” is unlikely to comply with the instruction “please limit your answer to two words”.
Such high-level semantic inconsistencies are challenging for a simple NLI model to discern.
Therefore, we employ the LLM $M$ to assign matching scores between the instruction and query in input $x^{(2)}$ and the corresponding responses $y^{(2)}$, on a scale from 1 to 10.
We then filter out samples with a score lower than 8, constructing the final training set $D_{\text{train}}=\{x_{i},y_{i},f_{Ii}\}_{i=1}^{N}$.

### 3.4 Training Strategies

AutoIF offers multifaceted supervision for the instruction-following task, making it adaptable to various training strategies.
To thoroughly evaluate the effectiveness of AutoIF, we propose the following training approaches:

Supervised Fine-tuning (SFT).
Given $(x_{i},y_{i})\in D_{\text{final}}$, we apply the standard Supervised Fine-tuning (SFT) objective on the base model $P$ with parameters $\theta$:
$\mathcal{L}(\bm{\theta})=\sum_{(x_{i},y_{i})\in\mathcal{D}_{\text{train}}}\log
\mathbb{P_{\theta}}(y_{i}\mid x_{i})$
, where $x_{i}$ denotes the $i$-th input, consisting of a concatenated instruction and user query.

SFT + Offline DPO.
In the process of AutoIF, multiple scales of quality filtering are utilized, naturally generating a substantial number of positive and negative sample pairs.
This motivates us to obtain pairwise preference data $(x,y_{w},y_{l})$. Our preference data mining is divided into two parts:

- •
Instruction Level: During the automated quality cross-verification stage, we first extract positive samples $c_{w}$ from cases with an accuracy rate higher than 0.5 on all verification functions and negative samples $c_{l}$ from cases with an accuracy rate of 0. We then construct pairwise preference data for each instruction: $D_{\text{ins}}^{\text{pref}}\rightarrow(I,c_{w},c_{l})$.
- •
Query Level: In the query quality verification process, we similarly extract positive samples $y_{w}$ from responses with an accuracy rate higher than 0.5 on all verification functions and negative samples $y_{l}$ from responses with an accuracy rate of 0. We then construct query preference data: $D_{\text{query}}^{\text{pref}}\rightarrow(x,y_{w},y_{l})$.

Finally, we merge the two parts of the data:
$D_{\text{pref}}=D_{\text{ins}}^{\text{pref}}\cup D_{\text{query}}^{\text{pref}}$.
To further explore the potential of pairwise preference data $(x,y_{w},y_{l})\in D_{\text{pref}}$, we first perform vanilla SFT on the base model $\pi_{\theta}$ to obtain an SFT model $\pi_{\theta}^{\text{SFT}}$ as equation 3.4. Then, we apply Direct Preference Optimization (DPO) (Rafailov et al., ) on our SFT model, which can be formulated as follows:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta}^{\text{SFT}};\pi_{\text{ref}})=-\mathbb{ E}_{(x,y_{w},y_{l})\sim\mathcal{D}}[\text{log}\sigma(\beta\text{log}\frac{\pi_ {\theta}^{\text{SFT}}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\text{log} \frac{\pi_{\theta}^{\text{SFT}}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)})],$ (2) $$

where the reference model $\pi_{\text{ref}}$ is set to $\pi_{\theta}^{\text{SFT}}$ initially and remains fixed throughout training. $\beta$ is a hyperparameter and $\sigma$ is the sigmoid function.
$\mathcal{L}_{\text{DPO}}$ aims to maximize the log probability of preferred $y_{w}$ relative to the dispreferred $y_{l}$.

SFT + Iterative Online DPO.
Online training enables real-time, iterative optimization of model weaknesses.
It relies on high-quality, lightweight reward models to provide continuous supervision feedback.
In the case of AutoIF, verification functions serve as rigorous filtering standards, akin to reward models, delivering immediate feedback on model responses across training iterations.
Following offline DPO, we conduct initial SFT on the base model $\pi_{\theta}$ to derive an SFT model $\pi_{\theta}^{\text{SFT}}$ with initial instruction-following capabilities.
As depicted in Figure 3, we set the generation temperature to 0.8 and allow the SFT model to generate $K$ responses through self-sampling for each training sample, forming a response set $\{R_{1},\ldots,R_{k}\}$.
Then, we employ corresponding verification functions to assess $K$ responses, thereby constructing the online DPO dataset $D_{\text{online}}^{\text{pref}}=(x,y_{w},y_{l})$ based on average pass rates across all functions.
Finally, leveraging $D_{\textrm{online}}$, we sequentially perform DPO training on $\pi_{\theta}^{\textrm{SFT}}$. Importantly, our iterative online optimization process progressively unlocks enhanced instruction-following capabilities.

## 4 Experiment

### 4.1 Experimental Setup

Datasets & Baselines. We conduct experiments using two LLMs from the Qwen2 series (Qwen2-7B and Qwen2-72B-Instruct) and two from the LLaMA3 series (LLaMA3-8B and LLaMA3-70B-Instruct).
Please note that the AutoIF method proposed in this work has been employed in the open-source Qwen2-Instruct model. Thus, the version of Qwen2-Instruct we utilized represents an early iteration during internal development, rather than the final open-source model.
The training datasets are respectively generated from Qwen2-72B-Instruct and LLaMA3-70B-Instruct, with detailed statistics provided in Table 5.
We demonstrate the effectiveness of AutoIF by evaluating the instruction-following capabilities of models fine-tuned with self-generated datasets using AutoIF.
Additionally, we include strong open and closed-source LLM baselines such as Mixtral-8x22B and GPT-4.
For more details, refer to Appendix B.

Settings.
In our experiments, we mainly explore two experimental setups:

(1) Strong-to-Weak Distillation involves aligning a less powerful model with a stronger, well-aligned model by mimicking its generated responses. In AutoIF, we can utilize a strong model such as Qwen2-72B-Instruct for data synthesis. Subsequently, we train a less powerful model like Qwen2-7B-Instruct using this synthesized data to achieve strong-to-weak alignment.

(2) Self-Alignment: Following several self-alignment works  (Chen et al., ; Yuan et al., ), we utilize the LLM to perform the AutoIF process for synthesizing data, and then train the same model using this synthesized data.

Evaluation.
We evaluate our methods using two instruction-following benchmarks: IFEval (Zhou et al., ) and FollowBench (Jiang et al., ).
IFEval comprises 25 types of verifiable instructions across about 500 prompts.
While IFEval also focuses on verifiable instructions, extensive n-gram probing confirms no overlap between the IFEval test set and our training sets, thus eliminating any contamination concerns.
We report strict and loose accuracy metrics at both prompt and instruction levels for IFEval.
FollowBench is a fine-grained constraint-following benchmark with five levels of difficulty.
It contains diverse and open-ended instructions requiring evaluation by strong LLMs, such as GPT-4, which can fully examine the generalization of AutoIF to more general instructions not verifiable by simple code executions.
At the same time, we also evaluate our models on C-Eval  (Huang et al., ), MMLU  (Hendrycks et al., ), GSM8k  (Cobbe et al., ), and HumanEval  (Chen et al., ) to obtain a comprehensive assessment of capabilities.

**Table 1: The main results on two instruction-following and four general benchmarks. Pr. and Ins. stand for prompt and instruction levels, respectively. S and L represent strict and loose metrics for IFEval. The subscript indicates the increase in metrics compared to the corresponding backbone model. The highest accuracy for each setup is highlighted in green. Results marked with ^† are directly sourced from the original benchmarks.**
| Model | IFEval | FollowBench (SSR) | C-Eval | MMLU | GSM8k | HumanEval |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pr (S) | Pr. (L) | Ins. (S) | Ins. (L) | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 | Avg |  |  |  |  |  |
| Baselines (< 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-7B | 37.7 | 43.6 | 49.4 | 53.4 | 55.6 | 53.5 | 53.7 | 49.9 | 48.6 | 52.3 | 74.4 | 64.4 | 71.1 | 58.1 |
| Qwen2-7B(ShareGPT) | 30.9 | 33.5 | 42.4 | 45.2 | 56.1 | 52.7 | 50.8 | 45.2 | 47.9 | 50.5 | 70.2 | 59.8 | 59.4 | 52.4 |
| LLaMA3-8B | 24.6 | 26.1 | 38.1 | 39.7 | 10.0 | 10.3 | 10.5 | 14.3 | 12.7 | 11.6 | 24.2 | 38.8 | 4.5 | 0.6 |
| LLaMA3-8B(ShareGPT) | 23.7 | 26.4 | 33.8 | 37.1 | 44.0 | 40.0 | 39.6 | 33.3 | 33.6 | 38.1 | 35.2 | 44.6 | 20.5 | 38.1 |
| Mistral-7B | 23.3 | 24.6 | 38.4 | 39.6 | 40.1 | 39.7 | 37.9 | 35.7 | 36.7 | 38.0 | 38.2 | 47.6 | 20.5 | 38.4 |
| Baselines (> 10B) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-72B-Instruct | 77.1 | 80.4 | 84.4 | 86.9 | 70.2 | 66.6 | 63.5 | 58.1 | 56.3 | 62.9 | 83.8 | 80.8 | 87.9 | 73.8 |
| LLaMA3-70B-Instruct | 77.8 | 83.8 | 84.2 | 88.8 | 60.7 | 60.5 | 61.1 | 61.7 | 60.3 | 60.9 | 60.2 | 80.5 | 92.6 | 78.7 |
| Mixtral-8x22B | 41.8 | 47.3 | 55.2 | 60.0 | 63.9 | 60.0 | 58.2 | 56.2 | 55.3 | 58.7 | - | - | - | - |
| GPT-4^† | 76.9 | 79.3 | 83.6 | 85.4 | 84.7 | 77.6 | 76.2 | 77.9 | 73.3 | 77.9 | - | - | - | - |
| GPT-3.5 Turbo^† | - | - | - | - | 80.3 | 71.2 | 74.2 | 69.6 | 67.1 | 72.5 | - | - | - | - |
| Supervision Model: Qwen2-72B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Strong-to-Weak |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-7B-SFT | $\text{40.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.0}}}$ | $\text{44.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.9}}}$ | $\text{51.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.9}}}$ | $\text{55.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.0}}}$ | $\text{60.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.6}}}$ | $\text{53.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.2}}}$ | $\text{54.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.6}}}$ | $\text{49.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.0}}}$ | $\text{48.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.0}}}$ | $\text{53.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.0}}}$ | $\text{73.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.0}}}$ | $\text{64.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.0}}}$ | ${\text{74.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.0}}}}$ | $\text{58.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.2}}}$ |
| w/ Offline DPO | $\text{41.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.5}}}$ | $\text{44.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.2}}}$ | $\text{51.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.0}}}$ | $\text{56.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.8}}}$ | $\text{61.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+5.8}}}$ | $\text{54.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.0}}}$ | $\text{54.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.6}}}$ | $\text{51.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.3}}}$ | $\text{48.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.0}}}$ | $\text{54.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.7}}}$ | $\text{75.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.7}}}$ | $\text{64.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.1}}}$ | $\text{72.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.8}}}$ | ${\text{59.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.4}}}}$ |
| w/ Online DPO | ${\text{44.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+6.3}}}}$ | ${\text{46.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.0}}}}$ | ${\text{55.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+5.6}}}}$ | ${\text{57.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.5}}}}$ | ${\text{61.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+5.8}}}}$ | ${\text{56.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.3}}}}$ | ${\text{57.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.1}}}}$ | ${\text{55.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+5.5}}}}$ | ${\text{51.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.0}}}}$ | ${\text{56.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.3}}}}$ | ${\text{76.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.6}}}}$ | ${\text{64.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.4}}}}$ | $\text{72.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.2}}}$ | $\text{58.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.1}}}$ |
| \cdashline1-15   Self-Alignment |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-72B-Instruct<br>w/ Online DPO | $\pagecolor{backgreen}{\text{80.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+3.1}}}}$ | $\pagecolor{backgreen}{\text{82.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.9}}}}$ | $\pagecolor{backgreen}{\text{86.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.7}}}}$ | $\pagecolor{backgreen}{\text{88.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.1}}}}$ | $\pagecolor{backgreen}{\text{76.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+6.0}}}}$ | $\pagecolor{backgreen}{\text{69.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+3.2}}}}$ | $\pagecolor{backgreen}{\text{67.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+3.5}}}}$ | $\pagecolor{backgreen}{\text{61.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+3.5}}}}$ | $\pagecolor{backgreen}{\text{62.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+6.5}}}}$ | $\pagecolor{backgreen}{\text{67.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+4.6}}}}$ | $\pagecolor{backgreen}{\text{84.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.1}}}}$ | $\pagecolor{backgreen}{\text{81.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+0.4}}}}$ | $\pagecolor{backgreen}{\text{88.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+0.3}}}}$ | $\pagecolor{backgreen}{\text{75.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.2}}}}$ |
| Supervision Model: LLaMA3-70B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Strong-to-Weak |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaMA3-8B-SFT | $\text{28.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.1}}}$ | $\text{40.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+14.2}}}$ | $\text{41.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.3}}}$ | $\text{52.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+12.05}}}$ | $\text{46.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+36.6}}}$ | $\text{46.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+35.9}}}$ | $\text{45.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+35.4}}}$ | $\text{37.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+23.3}}}$ | $\text{41.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+28.3}}}$ | $\text{43.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+31.9}}}$ | $\text{34.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+10.3}}}$ | $\pagecolor{backgreen}{\text{45.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+6.8}}}}$ | $\pagecolor{backgreen}{\text{33.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+28.7}}}}$ | $\text{38.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+37.6}}}$ |
| w/ Offline DPO | $\text{27.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.3}}}$ | $\text{41.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+15.5}}}$ | $\text{40.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.4}}}$ | $\text{54.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+14.4}}}$ | $\text{51.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+41.9}}}$ | $\text{51.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+41.0}}}$ | $\pagecolor{backgreen}{\text{50.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+39.6}}}}$ | $\text{45.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+31.0}}}$ | $\pagecolor{backgreen}{\text{47.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+34.8}}}}$ | $\text{49.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+37.6}}}$ | $\text{36.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+12.0}}}$ | $\text{45.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+6.5}}}$ | $\text{31.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+27.4}}}$ | $\pagecolor{backgreen}{\text{38.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+37.9}}}}$ |
| w/ Online DPO | $\pagecolor{backgreen}{\text{28.8}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+4.2}}}}$ | $\pagecolor{backgreen}{\text{43.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+17.0}}}}$ | $\pagecolor{backgreen}{\text{42.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+4.1}}}}$ | $\pagecolor{backgreen}{\text{56.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+16.3}}}}$ | $\pagecolor{backgreen}{\text{54.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+44.6}}}}$ | $\pagecolor{backgreen}{\text{52.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+41.8}}}}$ | $\text{50.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+39.5}}}$ | $\pagecolor{backgreen}{\text{49.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+34.7}}}}$ | $\text{43.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+31.0}}}$ | $\pagecolor{backgreen}{\text{49.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+38.3}}}}$ | $\pagecolor{backgreen}{\text{38.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+14.0}}}}$ | $\text{45.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+6.3}}}$ | $\text{32.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+28.0}}}$ | $\text{38.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+37.8}}}$ |
| \cdashline1-15   Self-Alignment |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaMA-3-70B<br>w/ Online DPO | $\pagecolor{backgreen}{\text{80.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+2.4}}}}$ | $\pagecolor{backgreen}{\text{85.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.8}}}}$ | $\pagecolor{backgreen}{\text{86.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+2.5}}}}$ | $\pagecolor{backgreen}{\text{90.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.6}}}}$ | $\pagecolor{backgreen}{\text{71.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+10.3}}}}$ | $\pagecolor{backgreen}{\text{67.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+6.7}}}}$ | $\pagecolor{backgreen}{\text{66.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+5.1}}}}$ | $\pagecolor{backgreen}{\text{64.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+2.9}}}}$ | $\pagecolor{backgreen}{\text{63.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+3.2}}}}$ | $\pagecolor{backgreen}{\text{66.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+5.6}}}}$ | $\pagecolor{backgreen}{\text{61.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+1.4}}}}$ | $\pagecolor{backgreen}{\text{80.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+0.2}}}}$ | $\pagecolor{backgreen}{\text{92.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+0.1}}}}$ | $\pagecolor{backgreen}{\text{78.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[<br>named]{pgfstrokecolor}{rgb}{0,0.2734375,0}\text{+0.0}}}}$ |

### 4.2 Main Results

Section 4.1 reports the main results.
Overall, AutoIF substantially enhances instruction-following performance across all models, configurations (strong-to-weak distillation & self-Alignment), and training methodologies (SFT, Offline & Online DPO) on two benchmarks.
These results decisively establish the superiority of our approach. Furthermore, we have identified the following insights:

On-policy Learning is More Effective. Comparing Online DPO and Offline DPO, the model-generated online data through self-supervision demonstrates superior performance compared to offline data (Qwen2-7B, IFEval: 1.7%$\uparrow$, Followbench: 2.6%$\uparrow$). This confirms that on-policy iterative execution feedback can effectively target and enhance the model’s weaknesses.

Larger models yield greater improvements. FollowBench provides a more comprehensive instruction-following assessment than IFEval. Significantly, base models with larger parameters typically improve Followbench more than smaller models (Qwen2 72B: 4.6%$\uparrow$, LLaMA3 70B: 5.6%$\uparrow$). This underscores that models with robust foundational capabilities coupled with AutoIF, can further unlock powerful instruction-following alignment potential.

General abilities are not declined. Improving instruction following abilities without compromising other capabilities is crucial.
AutoIF notably preserves general abilities (MMLU, C-Eval), mathematical reasoning (GSM8k), and coding (Humaneval) performance across all training setups.
Surprisingly, there are even slight performance gains in on-policy settings. We attribute this preservation largely to incorporating ShareGPT data during data synthesis, highlighting AutoIF’s capability to strike a balance across diverse abilities and excel in broad applicability.

### 4.3 Quality Ablation Study

Ablation on Supervision Model. Table 3 presents the results of replacing the supervision model Qwen72B with GPT-4.
We observe that in AutoIF, a stronger supervision model (GPT-4) demonstrates more effective strong-to-weak distillation alignment, particularly evident with a performance gain of over 15% in the loose prompt in IFEval.
This is reasonable because AutoIF requires the supervision model to perform several tasks, such as text augmentation (instruction, query, and response rewriting), code generation (verification function), and quality assessment (scoring). This implies that a supervision model with stronger fundamental abilities can synthesize higher-quality data when using AutoIF.

**Table 2: Ablation study on supervision models.**
| Model | IFEval | FollowBench (SSR) |  |
| --- | --- | --- | --- |
| Prompt(L) | Instruction(L) | Avg |  |
| Qwen2-7B | 43.6 | 53.4 | 52.3 |
| Supervision Model: Qwen2-72B |  |  |  |
| +SFT | $\text{44.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+0.9}}}$ | $\text{55.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.0}}}$ | $\text{53.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.0}}}$ |
| +SFT & Offline DPO | $\text{44.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.1}}}$ | $\text{56.2}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.8}}}$ | $\text{54.0}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+1.7}}}$ |
| +SFT & Online DPO | $\text{46.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.0}}}$ | $\text{57.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.5}}}$ | $\text{56.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+4.3}}}$ |
| Supervision Model: GPT-4 |  |  |  |
| +SFT | $\text{52.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+9.3}}}$ | $\text{62.6}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+9.2}}}$ | $\text{55.1}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.8}}}$ |
| +SFT & Offline DPO | $\text{59.3}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+15.7}}}$ | $\text{68.9}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+15.5}}}$ | $\text{54.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+2.1}}}$ |
| +SFT & Online DPO | $\text{59.5}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+15.9}}}$ | $\text{69.4}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+16.0}}}$ | $\text{55.7}_{{\color[rgb]{0,0.2734375,0}\definecolor[named]{pgfstrokecolor}{<br>rgb}{0,0.2734375,0}\text{+3.4}}}$ |

Figure: Figure 4: The left two figures illustrate the quality ablation studies on instructions and queries, whereas the right two figures present the scaling analysis of SFT data and DPO pairs.
Refer to caption: x4.png

Quality Control on Instructions and Responses.
In Figure 4, we examine how varying pass rate thresholds of verification functions (indicative of data quality) affect the amount of SFT data and instruction-following performance.
As the pass rate threshold increases, the amount of SFT data decreases at the instruction level, while model performance consistently improves.
This suggests that the quality of instructions is a crucial factor influencing IF performance.
At the query level, the SFT data amount also decreases with higher pass rate thresholds. Notably, performance peaks at a pass rate of 0.8 and declines beyond 1.
This observation aligns with our expectations, indicating a trade-off between data quality and quantity.

Ablation on Specific Components. To investigate the effectiveness of various modules in AutoIF, we conduct an ablation study, as presented in Table 3.
we use w/o to denote the variant without a specific module.
The results reveal the following: (1) The performance of AutoIF declines when any quality filtering process is removed, indicating that all components are highly effective.
(2) The most significant performance drop occurs when the Cross Verification of instructions is removed, highlighting its importance over query quality verification. This underscores that a high-quality instruction set is fundamental to the AutoIF process.
(3) Eliminating the overall quality filtering process results in a more substantial performance drop than removing any single component, suggesting that quality filtering at both the instruction and query levels provides a mutually reinforcing effect.

### 4.4 Analyses

Scaling Analysis on SFT & DPO Data.
Figure 4 presents the scaling analysis of SFT and DPO data using GPT-4 as the supervision model. The results demonstrate that even with just 1/64 of AutoIF-generated SFT/DPO data, Qwen2-7B achieves impressive performance, particularly with 1/64 DPO data reaching nearly 55% in loose prompt accuracy, , an increase of 11.4% pts. This strongly verifies the high quality of AutoIF-generated data.
Further analysis reveals that IF capability steadily improves with an increase in data quantity, a scaling trend confirmed by numerous reasoning studies  (Yuan et al., ; Muennighoff et al., ).

**Table 4: Contamination analysis on SFT data generated by different supervision models. Train & Test denotes the size of corresponding set. Rephr. represents samples similar to the test sample**
| Setup | Bench. | Train | Test | Rephr. | Percentage$\downarrow$ | N-gram$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| ShareGPT | IFEval | 25K | 542 | 0 | 0.01% | 4.8% |
| Followbench | 25K | 820 | 1 | 0.01% | 2.3% |  |
| Qwen2-72B | IFEval | 10K | 542 | 2 | 0.01% | 3.5% |
| Followbench | 12K | 820 | 1 | 0.01% | 0.9% |  |
| LLaMA3-70B | IFEval | 15K | 542 | 0 | 0.01% | 2.9% |
| Followbench | 17K | 820 | 1 | 0.01% | 1.2% |  |
| GPT4 | IFEval | 25K | 542 | 0 | 0.01% | 3.6% |
| Followbench | 25K | 820 | 1 | 0.01% | 1.5% |  |

Contamination Analysis. We evaluate the contamination of the training dataset generated by AutoIF on IFEval and FollowBench.
Specifically, we employ contamination detectors from LM-Sys  (Yang et al., ), which utilize advanced chatbots to identify potentially rephrased contaminated test samples.
Additionally, we report contamination findings detected by traditional n-gram contamination algorithms.
As shown in Table 4, both contamination rates are lower than those of the ShareGPT dataset we used. This allows us to confidently assert that there is no contamination between the self-generated training samples and the test sets.
More cases can be viewed in Appendix D,

**Table 5: Data statistics and efficiency. Total denotes the total data amount without quality control.**
| Supervision | Total | SFT Data | DPO Data | Pass Rate | MBPP (Code) | IFEval |
| --- | --- | --- | --- | --- | --- | --- |
| LLaMA3-70b | 85K | 15K | 6k | 26% | 70.4 | 43.1 |
| Qwen2-72b | 123K | 10K | 4K | 28% | 73.9 | 44.7 |
| GPT4 | 210k | 25K | 15K | 34% | 87.5 | 59.3 |

Data Efficiency. Table 5 explores the relationship between model coding ability, data quality pass rate (samples with a query quality score above 8), and instruction-following capability. Surprisingly, we observe consistency in the supervision model across all three metrics.
This indicates that the execution feedback resulting from the supervision model’s coding ability substantially influences data synthesis quality and the final capability.

## 5 Conclusion

In this paper, we propose AutoIF, a scalable and automated method to enhance the instruction-following abilities of LLMs. It uses self-instruct and rejection sampling to enhance the supervisory signals of seed instructions and relies on self-generated execution feedback for quality filtering. We introduce three training strategies and two alignment settings to comprehensively analyze AutoIF. Experiments demonstrate that our method significantly improves performance across all settings in both IFEval and Followbench, with the first LLM achieving over 90% loose instruction accuracy.

## Limitations

In this paper, we propose AutoIF, a system for automated instruction augmentation and quality filtering, capable of scaling to over 10,000 instructions. While our focus is not on the construction of cross-instructions, the excellent results achieved in two instruction-following benchmarks demonstrate the generalizability of our method in handling complex instruction-following tasks. Additionally, we believe a more direct strategy would involve combining multiple simple instructions into cross-instructions, and subsequently enhancing and quality-filtering them using AutoIF. This way has the potential to further amplify the effectiveness of our method. Therefore, we consider automating and scaling cross-instruction tasks as a key direction for future research.

## Ethic Consideration

In this paper, we have fully presented the seed instruction set used by AutoIF in the Appendix. All concatenated queries are sourced from the publicly available ShareGPT dataset and have undergone multiple steps of quality filtering. Therefore, our method strives to minimize potential safety and ethical risks as much as possible. However, during the rejection sampling process, malicious prompts can lead the model to produce harmful or inappropriate outputs, which is a shared problem. Ensuring the quality of generated content in a safe and controllable manner is crucial. The application of these techniques should be guided by ethical considerations, with safeguards in place to prevent misuse and reduce the likelihood of producing harmful outcomes.

## Appendix

## Appendix A Seed Instructions

Figure 5 illustrates our hand-written seed instructions.

Figure: Figure 5: Examples of our seed instructions
Refer to caption: x6.png

## Appendix B Implementation Details

To better motivate researchers to reproduce the results, we report the detailed experimental details:

In the SFT phase, we perform full fine-tuning on Qwen2-7B and LLaMA3-8B with a learning rate of 7e-6, using a linear scheduler with 20 warm-up steps. All models are trained with DeepSpeed ZeRO Stage 3 (Rasley et al., ) and Flash-Attention 2 (Dao, ). We use a global batch size of 128, a weight decay of 0.1, and train for 3 epochs. Mixed precision training with bf16 is used, and the maximum context length is set to 8192 tokens. For Qwen2-72B and LLaMA3-70B, the global batch size is 512.

In the DPO phase, the learning rate is set to 5e-7 with a cosine scheduler and a 0.1 warm-up ratio. We use DeepSpeed ZeRO Stage 3 and Flash-Attention 2 for efficiency, with a global batch size of 64. Training utilizes a sigmoid loss function with a beta value of 0.3 and spans 2 epochs, with checkpoints every 200 steps. Mixed precision training with bf16 is employed, and the maximum context length is 4096 tokens.

We run all our experiments on NVIDIA A100 and H800 GPUs. Specifically, we train Qwen2-7B and LLaMA3-8B on 8 A100 GPUs, while Qwen2-72B-Instruct and LLaMa3-70B-Instruct on 64 H800 GPUs. Notably, we use an in-house version of Qwen2-7B without any targeted optimizations on instruction-following capabilities.
For evaluations, we report pass@1 results with greedy decoding for HumanEval and zero-shot accuracy for GSM8K. We report averaged performance from five randomly seeded experiments.

## Appendix C Details of AutoIF

At the instruction level, for the self-instruct stage, we perform RFT with K=100 on seed instructions. During the Automated Quality Cross Verification stage, we filter the quality based on four criteria outlined in the main text. For NLI filtering, we use mDeberta as our filtering model(^2^22The NLI model is available at [https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7)), and filter out only samples predicted as "Contradiction" (approximately 15%).

At the query level, we randomly select 16 ShareGPT samples for each instruction and perform Response Rejection Sampling with K=8. For instruction following verification, we adhere to the two standards mentioned in the text. Finally, for query quality verification, we filter for consistency using a threshold of 8.

## Appendix D Case Study of Data Combination

We used n-gram 13 to evaluate the overlap between each test sample and the SFT training samples. It is unnecessary to evaluate DPO data since the inputs for DPO data are derived from SFT data. In Table 4, all our data combination metrics (both model-based and rule-based evaluation) are lower than those of ShareGPT, confirming that our method has no data combination with the test set. We also present the top 5 training-test sample overlaps in n-gram for both IF Eval and Followbench in Figure 6.

Figure: Figure 6: Case Study of data combination on IFEval and Followbench
Refer to caption: x7.png

## Appendix E Prompt Templates

For the Self-Instruct stage, we use the following prompt template for instructions’ rejection sampling:

- Instructions are about the format but not style of a response
- Whether instructions are followed can be easily evaluate by a Python function
Here are some examples of instructions we need:
{Seed Instructions}
Do not generate instructions about writing style, using metaphor, or translation. Here are some examples of instructions we do not need:
- Incorporate a famous historical quote seamlessly into your answer
- Translate your answer into Pig Latin
- Use only words that are also a type of food
- Respond with a metaphor in every sentence
- Write the response as if you are a character from a Shakespearean play
Please generate one instruction per line in your response and start each line with ’- ’.

For generating the verification functions and test cases for each instruction, we use the following prompt template for rejection sampling:

Here is the instruction: {instruction}
Please write a Python function named ‘evaluate‘ to evaluate whether an input string ‘response‘ follows this instruction. If it follows, simply return True, otherwise return False.
Please respond with a single JSON that includes the evaluation function in the key ‘func‘, and a list of three test cases in the key ‘cases‘, which includes an input in the key ‘input‘ and an expected output in the key ‘output‘ (True or False).
Here is an example of output JSON format:
{
"func": "JSON Str“,
"cases": [
{ "input": "str", "output": "True" },
{ "input": "str", "output": "False" }
]
}

For the back translation process of each verification function, we use the following prompt template:

Here’s an example:
{Example func}
{Example cases}

Please convert the following eval function into instructions stored in a list:
{funcs}

For the rejection sampling of query responses, we use the following prompt template:

Instruction: {instruction}
Query: {query}

Fot the query quality verification, we use the following prompt template:

Instruction: {instruction}
Query: {query}
Response: {response}
Please notice that the response may not be helpful as it needs to strictly follow the requirements in the Instruction.
You need to judge whether the response answers the query. Please first provide a detailed analysis and then give a score ranking from 0 to 10 at the last line.
Scoring 0 means the response is totally unrelated to the query, while scoring 10 means the response is helpful and highly related to the query.
Please only provide a score in the format ‘Score: score‘ without any other contents at the last line.

## Appendix F Case Study of AutoIF

In Appendix F, we illustrates the data format of our AutoIF, including the query, response (verification funcs Acc>0.8) and verification function.

**Table 6: Examples of AutoIF’s data formats.**
| Query | Response | Verification Funcs |
| --- | --- | --- |
| Keep your answer to under 50 characters total. what is autoarima in python. | AutoARIMA automates ARIMA model selection. | [⬇](data:text/plain;base64,ZGVmIGV2YWx1YXRlKHJlc3BvbnNlOiBzdHIpIC0+IGJvb2w6CiAgICByZXR1cm4gbGVuKHJlc3BvbnNlKSA8PSA1MA==)<br>def evaluate(response: str) -> bool:<br>return len(response) <= 50 |
| Refrain from using any words that contain ’S’. i need a name for my Dungeons and Dragons world. | EternaRealm | [⬇](data:text/plain;base64,ZGVmIGV2YWx1YXRlKHJlc3BvbnNlKToKICAgIGZvcmJpZGRlbiA9ICdzUycKICAgIGZvciBjaGFyIGluIHJlc3BvbnNlOgogICAgICAgIGlmIGNoYXIgaW4gZm9yYmlkZGVuOgogICAgICAgICAgICByZXR1cm4gRmFsc2UKICAgIHJldHVybiBUcnVl)<br>def evaluate(response):<br>forbidden = ’sS’<br>for char in response:<br>if char in forbidden:<br>return False<br>return True |
| Keep your response under twenty words without sacrificing clarity. Let’s play a game shall we?. | ure, let’s play a game! What game do you have in mind? Please keep instructions simple and clear. | [⬇](data:text/plain;base64,ZGVmIGV2YWx1YXRlKHJlc3BvbnNlOiBzdHIpIC0+IGJvb2w6CiAgICByZXR1cm4gbGVuKHJlc3BvbnNlLnNwbGl0KCkpIDw9IDIwIGFuZCBsZW4ocmVzcG9uc2UpID4gMA==)<br>def evaluate(response: str) -> bool:<br>return len(response.split()) <= 20 and len(response) > 0 |
| Compose your answer using exactly 20 words. Diffrent Normal Forms. | Normal forms in databases: 1NF ensures atomic columns, 2NF eliminates non-key dependencies, 3NF removes transitive dependencies, BCNF enforces determinant restriction. | [⬇](data:text/plain;base64,ZGVmIGV2YWx1YXRlKHJlc3BvbnNlOiBzdHIpIC0+IGJvb2w6CiAgICByZXR1cm4gbGVuKHJlc3BvbnNlLnNwbGl0KCkpID09IDIw)<br>def evaluate(response: str) -> bool:<br>return len(response.split()) == 20 |
| Word your response in a five-line verse with a strict AABBA rhyme. 1.Write short notes on Decision trees.. | Decision trees, so clear and bright, Branch out to split data’s might, With nodes of questions, true or false, They sort through cases, young or old, like a versatile horse. From root to leaves, paths decide their course. | [⬇](data:text/plain;base64,ZGVmIGV2YWx1YXRlKHJlc3BvbnNlOiBzdHIpIC0+IGJvb2w6CiAgICBsaW5lcyA9IHJlc3BvbnNlLnNwbGl0KCdcbicpCiAgICBpZiBsZW4obGluZXMpICE9IDU6CiAgICAgICAgcmV0dXJuIEZhbHNlCiAgICByaHltZXMgPSBbbGluZVstMV0gZm9yIGxpbmUgaW4gbGluZXNdCiAgICByaHltZV9wYXR0ZXJuID0gcmh5bWVzWzBdID09IHJoeW1lc1sxXSA9PSByaHltZXNbMl0gIT0gcmh5bWVzWzNdID09IHJoeW1lc1s0XQogICAgcmV0dXJuIHJoeW1lX3BhdHRlcm4=)<br>def evaluate(response: str) -> bool:<br>lines = response.split(’\n’)<br>if len(lines) != 5:<br>return False<br>rhymes = [line[-1] for line in lines]<br>rhyme_pattern = rhymes[0] == rhymes[1] == rhymes[2] != rhymes[3] == rhymes[4]<br>return rhyme_pattern |

## Appendix G Baselines & Datasets

We give introductions to the LLM baselines for our instruction following.

#### LLaMA3

(Meta, ), developed by MetaAI, is the latest iteration of the LLaMA series, featuring significant upgrades. Compared to LLaMA2, LLaMA3 expands its training dataset, context length, and vocabulary, resulting in improved performance across various tasks. Enhancements in contextual understanding and language generation further distinguish LLaMA3.

#### Qwen2

(Bai et al., ), developed by Alibaba, includes five sizes: Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B. Trained on high-quality data in Chinese, English, and 27 other languages, Qwen2 excels in multilingual capabilities and shows strong performance in coding and mathematics. Additionally, it supports extended context lengths of up to 128K tokens (Qwen2-72B-Instruct), making it ideal for long texts and complex tasks.

#### Mistral-7B

(Jiang et al., ), released by Mistral AI in September 2023, leverages grouped query attention (GQA) combined with sliding window attention (SWA) to efficiently process sequences of any length, enhance inference speed, and improve throughput. It outperforms many 13B models across various tasks.

#### Mixtral-8 × \times × 7B

(Jiang et al., ) developed by Mistral AI, is the first open-source MOE large model. It is a sparse mixture of experts network and, like Mistral 7B, employs the GQA mechanism. With a smaller parameter count compared to LLaMA2-70B and GPT-3.5, it outperforms them across numerous tasks.

#### GPT Series

GPT-3.5 (OpenAI, ) and GPT-4 (Achiam et al., ), developed by OpenAI, are advanced models in the GPT series that use a three-stage reinforcement learning with human feedback (RLHF) algorithm. This enhances their instruction-following capabilities and minimizes harmful content generation. GPT-3.5 excels in text completion, translation, and summarization. Building on these strengths, GPT-4 further refines the RLHF algorithm, enhancing performance on complex instructions and making it suitable for applications ranging from academic research to industrial use.

In addition to the two Instruction-Following benchmarks introduced in the main text, we also provide a detailed overview of datasets covered in the experiments

ShareGPT refers to the multi-turn chatting histories used by Vicuna Chiang et al. (). ShareGPT includes 86K human queries and responses from ChatGPT and other chatbots. We randomly select 2w samples to train LLaMA3-8B and Qwen2-7B to obtain our baseline models: LLaMA3-8B (ShareGPT) and Qwen2-7B (ShareGPT).(^3^33Follow the set up of , we use the version from [https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) cleaned raw dataset, and follow Vicuna preprocess.).

GSM8K (Cobbe et al., ) is a mathematical dataset designed to evaluate the mathematical problem-solving abilities of language models. It consists of 8,000 diverse grade school-level math word problems, which require understanding and manipulating mathematical concepts to arrive at a correct solution. It comprises high-quality grade school math problems, with 7,473 training samples and 1,319 testing samples.

HumanEval (Chen et al., ) includes 164 unique programming challenges, each paired with approximately 9.6 test cases on average. To provide a more comprehensive evaluation of the functional accuracy of code generated by large language models, HumanEval+ substantially increases the number of test cases to an average of 774.8 per problem. In this paper, we report the Pass@1 result when applying greedy decoding.

MMLU (Hendrycks et al., ) is a benchmark designed to assess pretraining knowledge in models using zero-shot and few-shot evaluations. It includes 57 subjects across STEM, humanities, social sciences, and more, with difficulty levels ranging from elementary to advanced professional. MMLU tests both world knowledge and problem-solving skills, covering traditional disciplines like mathematics and history, as well as specialized areas such as law and ethics.

C-Eval (Huang et al., ) consists of multiple-choice questions categorized into four difficulty levels: middle school, high school, college, and professional. The questions cover 52 varied disciplines, including humanities, science, and engineering. Additionally, there is C-Eval Hard, a subset of particularly challenging topics within C-Eval that demand advanced reasoning skills. We perform an in-depth evaluation of leading language models on C-Eval, testing both English and Chinese-focused models.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 4
questions: How could the framework be extended to manage instructions that are inherently ambiguous or have multiple valid responses?

What modifications to AUTOIF would be necessary to accommodate tasks involving emotional nuance or creative outputs?

Could providing more detailed information on errors and failure rates improve the replicability and robustness of AUTOIF?

What specific criteria were used to filter low-quality instructions, and can this process be made more transparent?

What are the computational overhead implications of using execution feedback at scale, and are there any optimizations or trade-offs that can be explored?

How does AUTOIF perform in resource-constrained environments, and is there a way to quantify these trade-offs more concretely?
soundness: 3
strengths: Extensive Performance Improvements: AUTOIF leads to significant gains across various instruction-following benchmarks, such as IFEval and FollowBench. It notably achieves over 90% loose instruction accuracy on IFEval while preserving or even slightly improving performance on other key capabilities like coding and mathematical reasoning.

Comprehensive Evaluation and Ablation Studies: The research includes a thorough evaluation of AUTOIF across different models, parameter sizes, and ablation settings, demonstrating the robustness and scalability of the approach. It also explores data scaling and efficiency, highlighting the method's potential to achieve strong performance with relatively fewer data samples.

Balanced Capability Improvement: AUTOIF enhances instruction-following without compromising other model abilities, such as coding and general reasoning, making it a well-rounded solution.
summary: This work presents AUTOIF, a generic and automated framework for the generation of high-quality training data to enhance the instruction-following prowess of LLMs. The main novelty of this work is that it reduces instruction-following alignment to a verifiable process using code. AUTOIF leverages execution feedback from code verification and rejection sampling to guarantee that the generated responses meet given constraints. This technique has paved the way for preparing both positive and negative examples for algorithms using SFT and RLHF. The authors mention that AUTOIF substantially improves the performance of LLMs on many benchmarks without weakening other competencies, such as coding and mathematical reasoning.

Besides this, the paper presents an extensive experiment to show that AUTOIF improves the instruction-following accuracy of a range of LLM architectures and sizes, from Qwen2 and LLaMA3 models. The paper underlines that indeed one of the keys to good performance is iterative online training and at the same time develops the scalability and generalization potential of AUTOIF across diverse types of instructions.
weaknesses: Conceptual Rigidity: While the core contribution of this paper is insightful, the attempt to cast the instruction-following alignment into a code verification problem introduces some rigidity. This is so because in that respect, focus will be on "verifiable instructions" that can be amenable to mechanical verification by code executors. While the structure for this argument is robust in this text, it might indeed be narrow to encompass more complex or subjective tasks. The paper does not delve into the non-code-based verification methods like model-based scoring or human-in-the-loop strategies; the thrust of the discussion remains centered on the execution feedback mechanism​.

Generalization Concerns: Concentration on verifiable instructions has been made amply clear in the method section. AUTOIF framework uses code for instruction verification, which in itself inherently limits the generalization to open-ended or subjective instructions. While the paper does highlight improvements across benchmarks such as IFEval, it does not really investigate how the approach generalizes when the tasks are less structured or ambiguous. Though the second concern with respect to AUTOIF biasing the models to easily verifiable instructions might be well-taken, given the current evidence, it should be framed as a hypothesis.

Somewhat worrying lack of methodological diversity: there is heavy focus on execution feedback. Other methods are not considered, like human feedback or model-based evaluation. Still, the paper does state that execution feedback and iterative optimization work fine, so there is a possibility that with more methods added, the instruction-following capabilities could be further improved.

It is very clear that, when explaining the strengths of the method, it seems over-dependent on automation and slighting the benefits that could be derived from purposive human intervention. The consideration of benefits from a more balanced approach in complex situations is thus little. By allowing human insights to complement this framework, therefore, it would adapt to inclusivity in the given context.

Novelty: Automatic instruction verification might not be an entirely new idea. Works in this area are CodeLlama2, PLUM, LLaMA3, SelfCodeAlign, and DeepSeek-Coder-V2. In fact, research efforts such as code-based execution feedback and automated instruction alignment have been made. For example, CodeLlama2 and LLaMA3 utilized the base of code-based synthesis and validation frameworks in their architectures for better performance on top of instruction-following tasks, which underlines code execution as a natural source of feedback. Simultaneously, these related works suggest that perhaps AUTOIF, with the novelty of scalability and data synthesis, is not really a new fundamental direction in the space.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - I am curious about counter-examples where programs cannot model the constraints in the instruction; and how to recognize such cases and delegate them to human experts.
- Section 3.2: How reliable is cross-validation of test cases and verification functions? Do you see cases where cross-consistent test and verifiers are wrong together?
soundness: 3
strengths: - This paper looks at an important problem of training LLMs to precisely follow instructions
- The proposed approach is novel and seems general to most scenarios. It can be used to improve the quality of instruction-following datasets.
- This paper comes with comprehensive end-to-end evaluations as well as controlled experiments, with good insights and reasonable explanations.
summary: This paper presents AutoIF. The main idea of AutoIF is to generate constraint-following instruction data by generating validation functions that correspond to the constraints within the natural language instruction and running them to validate any prospective responses.
weaknesses: - Section 3.2: cross-verification can be weak as it essentially assumes test-function self-consistency as correctness. Not sure how reliable is self-consistency here as (i) bad tests can kill the right function and (ii) tests and functions can be broken all together result in corrupted data quality. 

----

Other minor suggestions:

- Code execution has also been widely adopted in the code domain to generate test-assured code; it is generally good to discuss more of such work (not only just CodeRL; but also RLTF, LEVER, Self-OSS-Instruct, ODEX, etc.) to connect the general research context.
- L209: It might be more accurate to cite the use of rejection sampling with "Training a helpful and harmless assistant with reinforcement learning from human feedback" in addition to/instead of the Llama 2 paper.
- L210: what does "_{j=1}" mean? Do you mean "i=1"?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. See the weaknesses above.
2. Line 107: `we validate AUTOIF’s effectiveness in both "Self Alignment" and "Weak to Strong"`. typo?
3. Will the self-alignment setting still work for smaller models like Llama3-8B?
4. What are the main technical differences from StarCoder2-Instruct, which also used the base model itself to generate instructions, code, and test cases for validation?
5. Although the IFEval and FollowBench results are impressive, these benchmarks are specifically designed to test natural language instruction following. Can improved IFEval performance be linked to more practical tasks? For instance, while benchmarks like GSM8K and HumanEval may not show additional gains, BigCodeBench / LiveCodeBench / LeetCode include complex instruction-following cases, which should ideally lead to improvement.
6. Are there analyses on how reliable the generated code and tests are? Either or both of them may be just wrong.

If my concerns are properly addressed, I will consider raising the score.
soundness: 2
strengths: 1. The technique is intuitive. IFEval was constructed using verifiable instructions to test an LLM's instruction-following ability, while this paper generates verifiable code to evaluate the instruction-response alignment.
2. AutoIF proves to be effective in the self-alignment case, where the same model is used for instruction generation and verification.
3. The evaluation covers a wide range of benchmarks and shows that AutoIF improves instruction following significantly while preserving the other general capabilities.
4. AutoIF naturally generates pair-wise preference data and can be applied to DPO, offline or online.
summary: The paper proposes AutoIF, a scalable method for generating high-quality instruction-following data for training LLMs without manual annotation. It leverages code execution to verify response quality. LLMs are used to generate instructions, produce the corresponding code, and create unit test samples for cross-validation. Execution-based rejection sampling produces data for SFT and RLHF. Applied to Qwen2 and Llama3, AutoIF enhances performance across three training algorithms (SFT, offline DPO, online DPO), achieving over 90% accuracy in IFEval’s loose instruction accuracy benchmark without sacrificing general, math, or coding abilities, highlighting its effectiveness in alignment and generalization.
weaknesses: 1. In the self-alignment setting, the evaluation is only conducted on the two >70B models. It is unclear whether the same conclusion applies to weaker base models with less than 10B parameters.
2. There are strong-to-weak and self-alignment settings, so a weak-to-strong setting should logically be included. There’s no proper justification for leaving it out.
3. There's a missing baseline where responses are validated through the LLM itself without code. Without the baseline, it's unclear how valuable the verification function and test cases are.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 2
questions: - Figure 2, improve consistency of terms in figure with 'acc' and 'Acc' being used alternatively, and the sequence of verification and func set being inverted at start.
- An example case of how a sample develops in the AutoIF pipeline would help illustrate the working even better. 
- Line 215, 232, provide reasoning and example of failing cases of why accuracy is lower. similarly for Section 4.1 talks about main results but is limited to quantitative numbers and not examples of illustrating failed cases.
- Also, most seed instruction showcase restriction over string manipulation domain only. Ensuring type of word to be present, limiting length of output and so on. There is no showcasing of diverse, novel instructions. 
- line 319, what is the value of K, for which the results have been sampled from?
soundness: 3
strengths: There is a detailed and structured pipeline of AutoIF which showcases enhanced results over multiple training strategy.
summary: The paper proposes AUTOIF, an automated, scalable, and reliable method designed to enhance the instruction-following capabilities of LLMs, using automatic code verification.
weaknesses: However, examples on failed cases and illustration of these categories of consistent failure would help understand scope of next steps and areas of improvement that has been brought about by the method. Along with the limited domain of seed instructions in terms of string manipulation only, which could be diversified.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Apart from the mentioned weaknesses, there are some additional questions:
- How do the accuracy rate thresholds in the verification steps affect the model performance?
- How does RAILF+Online DPO (e.g., OAIF [1]) compare to AUTOIF?

[1] Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., ... & Blondel, M. (2024). Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792.
soundness: 3
strengths: There are a few valuable contributions:
- The proposed AUTOIF shows that training with program-based validation can improve some instruction-following capabilities.
- The experiments are pretty extensive, where the authors benchmark the AUTOIF on four models with different sizes.
- The ablation studies are quite detailed, showing the optimal setups for AUTOIF.
summary: The authors propose AUTOIF, a training framework to improve instruction following capability via execution feedback. The authors demonstrate the efficacy of AUTOIF on four LLMs with benchmarks across various domains.
weaknesses: There are several weaknesses to be addressed:
- The idea of using programs to validate instruction-following is not novel, as it has been used in IFEval. Moreover, training with execution feedback is also not new; it has been widely used by training code LLMs [1-3].
- Based on the proposed workflow, AUTOIF is hard to scale to complex instructions, where the queries are totally quantifiable. AUTOIF cannot easily work for semantically rich instructions, which cannot be evaluated via execution. In addition, real-world scenarios that can be validated via programs may require various dependencies and APIs. AUTOIF is not scalable under these practical cases. 
- The instruction-following evaluations on IFBench and FollowBench lack practicability. These two benchmarks only contain quantifiable instructions, like word counts. The authors should consider more practical scenarios like the one in BigCodeBench [4] for the instruction-following evaluation.
- The current evaluation is English-only. The authors should evaluate their models on multilingual setups to demonstrate their generalisability.

[1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35, 21314-21328.

[2] Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., ... & Yue, X. (2024). Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658.

[3] Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J. G., & Chen, W. CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations.

[4] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., ... & Von Werra, L. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.

Meta-review: This paper proposes AUTOIF, a Self-Instruct-based approach to synthetize instruction following data that can be programmatically verified using test cases (e.g., “generate three sentences all ended with letter e”). Experiments on SFT, DPO and an online DPO setting demonstrate that AUTOIF significantly improves the base model’s instruction following performance, without compromising general math and coding skills. Notably, AUTOIF is the first to surpass 90% accuracy on IFEVAL with loose instruction accuracy.

**Strengths:**

All reviewers agree that self-supervised methods for instruction following is an important research direction and the proposed method is intuitive and effective. The experiments and ablation studies are also quite comprehensive (61cd, 7mrPm, 5kXL), “covering a wide range of benchmarks” (BmDh). The proposed method can work with both SFT and DPO-based approaches (BmDh), without compromising other model abilities (61cd).

**Weakeness:**

Perhaps the most important issue of the approach is that it is limited to instruction following problems that can be verified using code (61cd, 5kXL), therefore it is unclear whether this approach would generalize to complex tasks with “semantically rich instructions” (5kXL), or tasks with “open-ended or subjective instructions” (61cd). There are also concerns about the novelty of this approach given the vast majority of related works on synthesizing coding data using test diagnostics (61cd, 7mrP, 5kXL). However, as the first paper that generalizes similar lines of research to general-purpose instruction following, this paper already makes significant contributions to the community. Therefore, the decision is Acceptance.

Decision: Accept (Spotlight)