Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models
Authors: Guanting Dong
Keming Lu
Chengpeng Li
Tingyu Xia
Bowen Yu
Chang Zhou
Jingren Zhou
Venue: ICLR 2025 Spotlight (2025)
Keywords: Instruction Following
Large Language Models
Execution Feedback
On-policy Learning
Strong-to-Weak Distillation
Self-Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: One core capability of large language models~(LLMs) is to follow natural language instructions. However, the issue of automatically constructing high-quality training data to enhance the complex instruction-following abilities of LLMs without manual annotation remains unresolved. In this paper, we introduce AutoIF, the first scalable and reliable method for automatically generating instruction-following training data. AutoIF transforms the validation of instruction-following data quality into code verification, requiring LLMs to generate instructions, the corresponding code to verify the correctness of the instruction responses, and unit test samples to cross-validate the code's correctness. Then, execution feedback-based rejection sampling can generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) training. AutoIF achieves significant improvements across three training algorithms, SFT, Offline DPO, and Online DPO, when applied to the advanced open-source LLMs, Qwen2 and LLaMA3, in self-alignment and strong-to-weak distillation settings. Using two widely-used and three challenging general instruction-following benchmarks, we demonstrate that AutoIF significantly improves LLM performance across a wide range of natural instruction constraints. Notably, AutoIF is the first to surpass 90\% accuracy in IFEval’s loose instruction accuracy, without compromising general, math and coding capabilities. Further analysis of quality, scaling, combination, and data efficiency highlights AutoIF's strong generalization and alignment potential. Our code are available at https://github.com/QwenLM/AutoIF

Published as a conference paper at ICLR 2025
SELF-PLAY WITH EXECUTION FEEDBACK: IMPROVING INSTRUCTION-FOLLOWING CAPABILITIES OF LARGE LANGUAGE MODELS
Guanting Dong", Keming Lu, Chengpeng Li", Tingyu Xia", Bowen Yu' Chang Zhou, Jingren Zhou
Qwen Team, Alibaba Inc.
[dongguanting. dgt, 1uweming..... 1km, laachhngpengg.cpyaalliiiiieee (xiatingyu xty, yubowen. ybw, ericzhou. zc, -bb
ABSTRACT
One core capability of large language models (LLMs) is to follow natural language instructions. However, the issue automatically constructing high-quality training data to enhance the complex instruction-following abilities of LLMs without manual annotation remains unresolved. In this paper, we introduce AUTOIF, the first scalable and reliable method for automatically generating instruction- following training data. AUTOIF transforms the validation of instruction-followig data quality into code verification, requiring LLMs generate instructions, the corresponding code to verify the correctness of the instruction responses, and unit test samples to validate the code's s correctness. Then, execution feedback- based rejection sampling can generate data for Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) training. AUTOIF achieves significant improvements across three training algorithms, SFT, Offline DPO, and Online DPO, when applied to the advanced open-soure LLMs. Qwen2 and LLLMA33, in self-alignment and strong-to- weak distillation settings. Using widely-used and three challenging general instruction-followiinn benchmarks, we demonstrate that AUTOIF significantly improves LLM performance across a wide range of natural instruction constraints. Notably, AUTOIF is the first to surpass 90% accuracy in IFEval's loose instruction accuracy, without compromising general, math and coding capabilities. Further analysis of quality, scaling, combination, and data efficiency highlights AutolF's strong generalization and alignment potential. Our code are available at https:: //gi thub. cwm com/QueeLM/AutofF
Keep your response under 20 choracters in length, Are you with or Occupational English
Include least one word ending with '-ing'. What the weather today?
Insiruction
Response Today's weather sunny and the wind is blowing.
Response 1, Yes.
Response 2, The weather is sunny and it windy today.
Respouse
Response
familiar with
contains, wtw..ii..eeee....ee....e vosooosonnn words resporse split(() foz word in woeds:
def evaluste(response):
Verification
return Len(response) 20
if word.endse::t(( (areeettt"""nn"""" ing"): True
Function
Figure 1: An example of the verification function automatically assesses the adherence of responses to the instruction's 's constraints.
"ork done during internship at Qwen team, Alibaba Inc. 1 "Corrsponniig author

---

Published as a conference paper at ICLR 2025
1 INTRODUCTION
The instruction- foolowing ability of large language models (LLMs) refers to their capacity to under- stand, interpret, and execute commands given to them in natural language (Lou et al. 2023 OpenAl
jet al. 2024; Yang et al. 2024a (b. This ability is fundamental to contemporary LLMs as it enables
them to leverage their underlying knowledge, interact intuitively with users (Ouyang al. 2022).
adapt to various requirements (Zhang et al.) 2023] Lei et al.) 2023), and perform complex tasks (Sun
et al. 2024 Dong et al., 2024b) and scenarios (Lu et all [024]] Qiao et al. 2024a Zhang et al. 2024).
Misunderstandings in following instructions can lead to unintended outcomes, potentially resulting in
severe consequences, particularly in critical scenarios (Zhou et al.
2023 Chang et al. 2024).
Although instruction following is crucial, scalable and reliable methods to enhance this capability of LLMs remain elusive. Current efforts in this field are divided into manual annotation (Wei et al.
2021
Zhou et al.) 2023 Jiang et al. 2024b) and behavior imitation (Xu et al. 2023 Zhao et al.
2024 Manual annotation involves annotators designing instructions and writing corresponding
responses. However, due to human cognition's is limitations, creating highly complex and diverse instructions is challenging, making the process difficult to scale. Furthermore, accurately executing
complex instructions can sometimes be difficult for humans (Sun et al.) |024| [Cao et al., 2024b Hui
ct al. 2024), requiring multiple rounds of rigorous and costly validation (Wang et al. 2024a Wei
et al.
2024). On the other hand, behavior imitation aims to distill responses from more advanced
LLMs (Taori et al. 2023; Wang et al., 2024b Peng et al., 2023) like GPT-4 This approach limits models to the capabilities of the advanced LLMs from which they are distilled. Moreover, even
advanced LLMs can make mistakes, and the reliability of the distilled data cannot be guaranteed (Cui
ctt al. 2023). Consequently, models trained with this data may have propensity to not follow instructions accurately (Zhou et al. 2024).
In this paper, we introduce AUTOIF, the first scalable and reliable method for automatically generating instruction following training Data for Supervised Finetuning (SFT) or Reinforcement Learning from Human Feedback (RLHF) (Ouyang al.) 2022). The core idea of AUTOIF is use code to verify the correctness of following instructions. Intuitively, if designed properly, a significant portion of instructions, such as "Keep your response under 20 characters in length" can be verified for correctness using code, as illustrated in Fig. I Therefore, the key components of AUTOIF include (1) automatically generating instructions that can be verified by code, (2) automatically generating corresponding verification codes for these instructions, and (3) ensuring the reliability of the first two steps. Specifically, we start by providing AUTOIF with a small set of hand-writte seed instructions. Then, LLMss, not necessarily advanced ones, generate an augmented instruction set through self-instruct (Wang et al. 2023a Li et al., 2024b). Next, LLMs write verification codes and unit test cases for each instruction. Only the code that compiles correctly, passes the test cases, and teck-transsttes to the original instruction retained. If an instruction does not have a corresponding code that can verify its correctness, it is discarded. Finally, we employ LLMs to generate responses that either pass or fail the verification code using execution feedback-based rejection sampling (Yuan) et al. 2023). Responses that pass can be directly used for SFT, while pairs of passing and failing responses can be naturally used to create chosen-rejected pairs for Direct Preference Optimization (DPO) (Rafailov et al., 2023) and other RLHF algorithms. Moreover, once the instructions and verification code are determined, this process can be conducted on-policy, iteratively enhancing the cntructtio-following capabilities.
Through extensive experiments, we demonstate that AUTOIF significantly improves performance across three training algorithms- SFT. Offline DPO, and Online open-source L.Mss, Qwen2-72B and ILLMA3-70B, in both self-alignment and strong-to-went distillation settings. We conduct comprehensive evaluation of five general instruction-followii datasets, verfying AUTOIF's s strong general instruction alignment capabilities. Notably, first achieve Loose Instruction accuracy rates of 88.0% with Qwen2-72B and 90.4% with LLaMA3-70B on IFEval, the most widely used cnttuctiionfollowinn benchmark, while significantly preserving the LLM's coding, mathematical, and general interaction capabilities. We will open-source the SFT and DPO datasets and construction codes built with AUTOIF on Owen2-72B. marking the first large-scale, complex instruction-folloiii dataset of its kind.
when applied leading
To summarize, our contributions are as follows:
2

---

Published as a conference paper at ICLR 2025
To achieve automated, reliable improvement of LLMs' instruction-followingg with minimal human efforts, we propose AUTOIF, which first transforms instruction followiin alignment into automati- cally code verification, requiring LLMs to generate instructions, corresponding verification code, and unit test samples for cross-validation.
Based on DPO algorithms, we first regard executor feedback as natural reward model, constructing pairwise preference samples from both instruction and query aspects. We further design offline and on-policy strategies for iterative optimization of the model's weakness instruction following.
With AUTOIF, we validate AUTOIF's effectiveness in both "Self-Alignment" and "Strong-to" Weak" settings on two widely used IF benchmarks and three general IF benchmarks, especially first achieving over 90+% accuracy IFEval's Loose instruction Acc without compromising general abilities, math, and code reasoning. Further analysis on quality, scaling, combination, and data efficiency showcases AutolF's robust generalization and alignment potential.
2 RELATED WORKS
Instruction-folloving capabilities are among the most essential features of LLMs (OpenAl et al.
2024| Lou et al., (023). which are expected to precisely follow a broad and complex set of instructions. Consequently, recent research has concentrated on evaluating L.LMs' instruction-followeer in various contexts, such as verifiable (Zhou et al., 2023), compositional (Qin et al., (024a)). format- related (Xia al. 2024). refuting (Yan et al. 2024). and fine-grained instructions (Jiang al.)
abilities
2024b). However, significant gap remains between open-source and proprietary closed-source LLMs. Sun et al. (2024) propose Conifer, which enhances the instruction-folloiin capabilities of open-source LLMs through knowledge distillation from proprietary LLMs. [Wang al. (2024c) use LLMs to encode instruction metadata and augment diverse instructions from this metadata, employing proprietary LLMs for quality control. Both approaches, however, rely on proprietary LLMs for response distillation or judgment, which not only limits their potential but also subjects scalable and reliable
In this work, propose AUTOIF, capabilities of LLMs. AUTOIF uses execution feedback from self-generated verification functions provide supervision for instructions. This allows for effective self-aligne aligmmettnmnnt and strong- strong-to-wedtt te-k distillation on open-source models, thereby narrowing the performance gap with proprietary LLMs.
them to OpenAl's terms of use [ method to enhance the cntrructionfollowwiii
Learning with Execution Feedback is widely- technique in automated aligmment for tool use and coding (Cao et al. 2024a]. These learning methods typically utilize execution feedback from tools such as code executors to provide supervision for specific tasks. For instance, [e et al. (2022) employ feedback from unit tests via code compilers to enhance code synthesis capabilities through reinforcement learning. Similarly, Chen et al. (2023) train LLMs to provide debugging suggestions as feedback to improve coding abilities. Additionally, Qiao et al. (202b)) introduce Reinforcement Learning with execution feedback enhance LLMs using execution results from tools. Building on this learning paradigm, we propose a novel scalable oversight method that enables LLMs to autonomously generate verification functions and unit tests for natural language instructions, thereby applying execution feedback to enhance their instructionnfollloiing capabilities.
AUTOIF
We introduce AUTOIF, an automated, scalable, and reliable method designed to enhance the instruction-followii capabilities of LLMs. In this section, outline the preliminaries ($3.I). detail the two core components of AUTOIF ($3.2 (3.3). and discuss various training strategies that can be seamlessly integrated with AUTOIF ($.4))
3.1 PRELIMINARIES
instruction-folloiin Capabilities. Following instructions is one of the most crucial skills in modern LLMs. These models expected to provide precise responses to queries containing
https: http:://ppenaa.ccc//plllllleeees
3

---

Published as a conference paper at ICLR 2025
Stepl Instruction Augmentation and Verification
Superier Madal
Fot Nacto
Pereiantyyy
ScB-latrrut
Fhagles
Instractiass
Ores Vortfrrtte
Verifiatioo Fraection
Step2 Query Augmentation and Verification
Supetber Nookl
Fleal Intrratiin
Nuttfiation txactie
Gary Set
Dats Piler
Qety Saraa
Olvy
D-trato
Figure 2: An Overview of AUTOIF: An Automated Instruction Following Data Synthesis Method.
complex instructions, which can be either atomic or compositional. To evaluate the instruction- following capability of LLMs, we define a general mnstructionfoolowinn requirement as a specific {ip j=1 with N specific constraints (e.g. "Please generate
task. In this task, given an instruction than 50 tokens" contains 2 constraints) and specific query æ,
text in Shakespearean style, an LLM Ty should generate precise response y ~ T x, adhering to the constraints.
Verifiable Instructions. The complexity and diversity of instructions necessitate manual construction and verification for reliable supervision. This practical challenge motivates us to focus initially on instructions that can be automatically verified through programs and code executors, also known as verifiable instructions (Zhou et al.) (202)). Specifically, for a given instruction and task-specific query y. there exists verification function fi such that (y( returns true when the model's response correctly follows the instruction. We demonstrate that supervision of such instructions can be self-generated through scalable oversight with LLMs and execution feedback. Extensive experiments in our work show that training on verifiable instructions significantly benefits the handling of other general instructions that are more complex but unverifiable with simple code snippets.
Method Overview. AUTOIF synthesizes high-quality Enstruction-follvviii data through self- evolution, rejection sampling, and execution feedback. illustrated in Fig. 2, AUTOIF integrates automated data augmentation with quality verification processes, including automatically generated verification functions and back-translation instructions. This approach enables a two-stage automated data synthesis at both the instruction ($3.2) and query levels ($3.3) Additionally, we introduce three training strategies ($3.4) and explore two experimental settings ($4) to thoroughly evaluate the effectiveness and generalization of AUTOIF.
3.2 INSTRUCTION AUGMENTATION AND VERIFICATION
We first develop verifiable instructions along with corresponding evaluation functions, using rejection sampling informed by execution feedback.
Seed Instruction Construction. We start by handwriting set of seed instructions, denoted as Dseed, ensuring that each instruction contains only a single atomic constraint (e.g., "Answer the words that begin with B"). Detailed information seed instructions is listed in Appx.
Self-Instruct. Self-Instruct [Wang et al., 2023a is a straightforward and intuitive strategy for automated data augmentation that has garnered sigmificant attention in the field of LLM reasoning (Xu et al. 2023 Zhao et al. (023). For each instruction in Dseed, we use an LLM to perform K instruction rewrites, generating Daug- We then combine the seed and augmented data sets to obtain enhanced set of instructions, Dins = Dseed U Daug. and remove any duplicates.
Automated Quality Cross Verification. Previous research has shown that relying solely model-
generated augmented instructions often leads to the inclusion low-quality samples (Bai et al. 2022
Mumuni & Mumuni 2022: Dong al. 2024c Xie al. 2020 Zheng et al. 2024). Inspired by
4

---

Published as a conference paper at ICLR 2025
D SET
II) SFT Offline DPO
Apcid -Poaies
Datrai
Deerinn
Model Bunngee
Scoring
Offfiee IIPO
SPT
Blas
Base Modkl
SFT
SPI Medd
DPO
SFT
Iterative Online DPO
(Peater
Swaple
Verilaation Fasiice Scarda
Sece
Oaline EIO
Herrdten
DPO Tratsing
Figure 3: Different training strategies that can be adapted with synthetic dataset generated by AUTOIF.
series of tool execution studies, we employ an LLM to generate verification functions and test cases for each instruction. We use feedback from executing Python programs to ensure quality control. Given the instruction set Dins. the LLM . employs rejection sampling (Touvron et al. 2023 Yuan al. 2023) to generate K verification functions fr H and test 6₁ {c; for each instruction I, resulting in the set [T, cr] E Dins- We then cross-validate the quality of the instructions using the verification functions and cases, ensuring they meet the following criteria:
The verification function f fr can be successfully compiled by the Python executor.
Each test case c E ç₁ achieves an accuracy rate greater than 0.5 across all verification functions.
Each verification function f E fi achieves an accuracy rate greater than 0.5 across all test cases.
Each instruction includes least one evaluation function and case.
By adhering these four conditions, obtain the quality-filtered instruction set (112, D
Back-translation Verification. After the cross-validation stage, we obtained initially quality-verified verification functions and instructions. To further ensure the consistency between instructions and verification functions, we introduce treck-traslltiin. For a given pair (T(2), (
f(2)) E D2 we use the LLM M back-translate the verification function f E f(²) into instruction Ty- We then treat T as the premise and the back-translated instruction Ly as the hypothesis. Using the NLI model, we identify the semantic relationship between the instructions. The prediction can fall into of three categories: entailment, contradiction, or neutral:
(- q. 9aug) softmax (scorca (I, 1T)),
where scoreg Rext, x * + R is model dependent scoring function with parameters 0. We filter out any instruction labeled as contradiction to ensure the intent consistency. Finally we obtain the set (T)3), ((()) ((())
3.3 QUERY AUGMENTATION AND VERIFICATION
Once we have obtained verified instructions and verification functions, we utilize them to create training data comprising queries and responses.
Query Reforming and Augmentation. In the real-world application of modern chatbots, instructions typically employed to generate constrained responses user queries. Therefore, creating high-quality instructions is merely the initial step toward achieving effective instruction-followinn capabilities. To acquire authentic queries, as shown in the bottom part of of 2 we randomly selected K user queries from ShareGPT (Chiang et al. 2023) for each instruction and concatenated them to
construct the seed query dataset x. (( E D,, To further enhance the diversity and complexity of the input v, utilized the LLM to generate K responses yı {y: s=1- resulting in (x. f ya E D,-
5

---

Published as a conference paper at ICLR 2025
Instruction-folloving Verification. Following the previous quality cross- tross-verification process, further employ verification functions to assess whether the augmented responses adhere to the constraints in imput z. Similarly, require each response in Dy to meet the following conditions:
Each response must achieve an accuracy rate greater than 0.5 across all verification functions.
Each input must include at least one verification function and one response.
Based these rules, obtain the set (I:12, {(0) (2) E D
Query Quality Verification. Additionally, we observe that concatenated instructions and queries often conflict. For instance, high-quality response the query "help write news article" is unlikely to comply with the instruction "please limit your answer to two words". Such high-level semantic inconsistencies are challenging for simple NLI model discern. Therefore, we employ the LLM M to assign matching scores between the instruction and query in input 2(?) and the corresponding responses y( on scale from 1 to 10. We then filter out samples with score lower than 8, constructing the final training set Duain {Zi, Vis f1o)
3.4 TRAINING STRATEGIES
AUTOIF offers multifaceted supervision for the instraction-following task, making it adaptable to various training strategies. To thoroughly evaluate the effectiveness of AUTOIF, we propose the following training approaches:
Supervised Fine-tuning (SFT). Given (I,, 34) E Dinall we apply the standard Supervised Fine- tuning (SFT) objective on the base model P with parameters 0: [()) (x.,)) log Po xx))
where F; denotes the i-th input, consisting of concatenated instruction and user query.
SFT + Offline DPO. In the process of AUTOIF, multiple scales of quality filtering are utilized, naturally generating substantial number of positive and negative sample pairs. This motivates to obtain pairwise preference data (z, Vu v1). Our preference data mining is divided into two parts:
During the automated quality cross-veriiication stage, we first extract positive samples Go from cases with an accuracy rate higher than 0.5 on all verification functions and negative samples from cases with an accuracy rate of O. We then construct pairwise preference data for each instruction: Drref > Ci)
Instruction Level:
Query Level: In the query quality verification process, we similarly extract positive samples ye from responses with an accuracy rate higher than on all verification functions and negative samples y from responses with an accuracy rate of O. We then construct query preference data: DRiefy + (æ, yws 31).
Finally, we merge the two parts of the data: Ppre D.tet U Deeer To further explore the potential of pairwise preference data (z, Yws 31) E Dpecf we first perform vanilla SFT on the base model my to obtain an SFT model SFT as equation 3.4 Then, we apply Direct Preference Optimization (DPO) (Rafailov et al. 2024) our SFT model, which can be formulated as follows:
SFT (ymll))))) Glog T
SFT TØ
(yn
SPT Teet) = TE (w.... llogo (Blog (T
Tee }
(Te(( (x)
where the reference model Tee is set to SFT initially and remains fixed throughout training. B is a hyperparameter and the sigmoid function. LDPO aims to maximize the log probability of preferred yu relative to the dispreferred yt-
SFT + Iterative Online DPO. Online training enables real-time, iterative optimization of model weaknesses. It relies high-quality, lightweight reward models to provide continuous supervision feedback. the case AUTOIF, verification functions serve as rigorous filtering standards, akin to reward models, delivering immediate feedback model responses across training iterations. Following offline DPO, we conduct initial SFT on the base model mo to derive an SFT model T with initial coruution-followingg capabilities. As depicted in Fig. 3 we set the generation temperature
SPT

---

Published as a conference paper at ICLR 2025
to 0.8 and allow the SFT model to generate K responses through self self-sampling for each training sample, forming a response set {Ri...., R Then, we employ corresponding verification functions to assess K responses, thereby constructing the online DPO dataset D online = (x,, Yw- Vt) based on average pass rates across all functions. Finally, leveraging Dontine: we sequentially perform
DPO training on TH Importantly, our iterative online optimization process progressively unlocks enhanced inslroctionnfollloiing capabilities.
EXPERIMENT
Datasets & Baselines. We conduct experiments using two LLMs from the Qwen2 series (Qwen2-7B and Instruct). The training datasets are respectively generated from Qwen2-72B-Instee 70B-Instruct, with detailed statistics provided in Tab. 5) We demonstrate the effectiveness of AUTOIF by evaluating the cntrucctio-foolowing capabilities models fine-tuned with self self-generated datasets using AUToIF. Additionally, we include strong open and closed-souree LLM baselines such as Mixtral-8x22B and GPT-4. For more details, refer to Appx. SD
and two from the LLaMA3 series (LLaMA3-8B and LLaMA3-70B-
and LLaMA3-
Experimental Settings. In our experiments, we mainly explore two experimental setups:
(I) Strong-to- Ktrong-to-Weak Distillation involves aligning a less powerful model with a stronger, well- aligned model by mimicking its generated responses. In AUTOIF, we can utilize a strong model such as -ooon--7B-Innnuuee for data synthesis. Subsequently, we train a less powerful model like Qwen2-7B Owen2-7B-Instruct using this synthesized data to achieve strong-to to-ggtt alignment.
(2) Alig-Aiigmmnt: Following several self- aligmenigmmnn works (Chen et al., 2024; Yuan et al., 2024). we utilize the LLM to perform the AUTOIF process for synthesizing data, and then train the same model using this synthesized data.
Evaluation. We evaluate our methods using two widely-used instruction-following benchmarks: IFEval (Zhou al., 2023) and FollowBench (Jiang al.] (222bb as main results IFEval comprises 25 types of verifiable instructions across about 500 prompts. While IFfival also focuses on verifiable instructions, extensive n-gram probing confirms no overlap between the IFEval test set and our training sets, thus eliminating any contamination concerns. We report strict and loose accuracy metrics at both prompt and instruction levels for IFEval. FollowBench is a fine-grained constraint- following benchmark with five levels of difficulty. It contains diverse and open-ended instructions requiring evaluation by strong LLMs, such as GPT-44 which can fully examine the generalization of AUTOIF more general instructions verifiable by simple code executions. We presented specific examples in Appx. $
To explore AUTOIF on more natural instruction-follovin dataset InfoBench(Qin et al.) (0224)), the general natural instruction evaluation MT-Bench (Zheng et al., (222)) and the real-world chatbot evaluation set Arena-hard (Zheng et al., 2023) as cross domain validation. At the same time, we also evaluated our models in C- Eval (Huang al.] [002). MMLU (Hendrycks al.) [202T). GSM8k (Cobbe al., 202T). and HumanEval (Chen et al. 202 to obtain a complete capability evaluation.
scenario, further introduce the complex
4.1
MAIN RESULTS
Tab. I reports the main results. Overall, AUTOIF substantially enhances instruction-follovin performance across all models, configurations (strong-to- weak distillation & self alf-Alligmen) and training methodologies (SFT, Offline Online DPO) on two benchmarks. These results decisively establish the superiority of our approach. Furthermore, we have identified the following insights:
On-policy Learning More Effective. Comparing Online DPO and Offline DPO, the model- generated online data through self-supervision demonstrates superior performance compared to offline data (Qwen2-7B, IFEval: 1.7%t. Followbench: 2.6% This confirms that on-policy iterative execution feedback can effectively target and enhance the model's weaknesses.
Larger models yield greater improvements. FollowBench provides a more comprehensive instruction-follovinng assessment than IFEval. Significantly, base models with larger parameters
7

---

Published as a conference paper at ICLR 2025
Table 1:
The main results on two mnstructiio-following and four general benchmarks. Pr. and Ins. stand for prompt and instruction levels, respectively. S and L represent strict and loose metrics for IFEval. The subscript indicates the increase in metrics compared to the corresponding backbone model. The highest accuracy for each setup is highlighted in green Results marked with are directly sourced from the original benchmarks.
Model
Fellow (SSR)
C-Eval MMLU GSMSK HumanEval
(S)
Ins. (S)
(l.
Level
Level2
Level
Level
Level 5
Arg
Baselines 108)
Qwea2-70
37.7
43.6
49.4
53.4
55.6
53.5
53.7
19.9
48.6
52.3
74.4
64.4
71.1
58.1
Qwem2-7B(ShureGn)
30,9
33.5
42.4
45.2
56.1
52.7
50.8
45.2
47.9
50.5
70.2
59.8
59.4
52.4
LLaMA3--8
24.6
26.1
38.1
39.7
10.0
10.3
10.5
14.3
11.6
24.2
38.8
LLaMA3 SB/ShureCPP)
23.7
26.4
37.1
44.0
33.3
35.2
Mistral-7B
23.3
24,6
38.4
39.6
40.1
39.7
37.9
35.7
36.7
38.0
38.2
47.6
20.5
38.4
(sseiines 108)
Qwen2 1eem2-120-Ba Instruct
77.1
80.4
84.4
$6.9
70.2
66.5
63.5
58.1
56.1
62.9
83.8
80.8
87.9
73.8
LLaMA3-700B Indtnct
77.8
63.8
84.2
58.8
60.7
60.5
61.1
61.7
60.3
60,9
60.2
80.5
92.6
78.7
Mistral-&x228
41.8
47.3
60.0
GPT-4
76.9
79.3
83.6
85.4
84.7
77.6
762
77.9
73.3
37.9
GPT-3.5
80.3
69.6
72.5
Supervision Model: Qwen2.78
Strong-to do-
AurolF (Qwen2 7B)
40.72x0 44.5.no
51.3.1
55.4un
60.2
53.7w01
54.3w06 hoo 49 P.an
48.6200
$3.3.10
74.400
64A-n0 74.1230
SR. 58.3.0
-SFT
Offline DPO
147
214,220
5622,2
61,4,58
54.510
$43,06
1.7
a.l
72%.1.
59.5.14
Online
63
46.6.50
55.0/5.
57.9445
6L445a
56.8,33
57.8441
55
51,6/30
56.6 4.
76.0.16
64.8
72.3
58.2.0.
Self-Aligmment
AUrOlF (Qwen2-72B)
Onlline DPO
802.,.1
82.3.19
86.1,33 88.0.L.
76:2,6
69.8, 67.0,5 61.6.g. 62%,65 67.5.a6
84.9.11
81.2.a4
88.2,03
75.0.1.
Supers isson Mudel: LLaMA3-70B
Strong-do- Wook
AuroTP (LLaMA3-88)
-SFT
28.7-11 40.3.14
41443
52.2-12m 46.6.t6 46.2.15 45.9.s 37.6.313 41.00s 43..m 345.000 15.0. 33.24zs
38.2447
Offline DPO
27.933 41.6,155 4/24
14.4
51.9:419 a $1.
141 50.1 ans 45.3,310 47.5 sas
122 45.3 lss
31.9,274
38.5
Ohline DPO
28.8 43.1
42.244.
56....1 54.6,40 52. +41. 50.0.00 40.02340 48.7.51 42.9 a 58.2.140 La. 32..5220
38-4 38.4237.
Self-Allgmment
AUTOIF (LLaMA3 708)
80.2... 856.11 86.7.2, 90:411 71 01.0 67.2.a7 66.2.11 64.6.29 635,33 66.5.x
61.6.1. 80.7.m1 92.7.a1
78.7.00
SFT
typically improve Followbench more than smaller models (Qwen2 72B: 4.6%1, LLaMA3 70B: 5.6%1). This underscores that models with robust foundational capabilities coupled with AUTOIF, can further unlock powerful cntruuctionfollowinn alignment potential.
General abilities are not declined. Improving instruction following abilities without compromising other capabilities is crucial. AUTOIF notably preserves general abilities (MMLU, C-Eval), mathemati- cal reasoning (GSM8k), and coding (Humaneval) performance across all training setups. Surprisingly. there are even slight performance gains on-policy settings. We attribute this preservation largely to incorporating ShareGPT data during data synthesis, highlighting AUTOIF's capability to strike balance across diverse abilities and excel in broad applicability.
4.2 CROSS-DOMAIN VALIDATION
To verify the effectiveness of AUTOIF, con- duct generalization experiments on 3 challeng- ing inolroctionn-olloiinng datasets, As shown in Tab. 2 results show that after fine-tuning with the SFT data generated by AUTOIF, Qwen2- 7B achieved significant improvements across all three datasets. particular, when online DPO is introduced in the SFT version, the improvement become even more pronounced, with over a 6% gain on Arena-hard. We believe this may be attributed to AUTOIF's multi-step verification process, which ensures the reliability and quality of the generated instructions, allowing the aligned model to better generalize to broader instruction alignment tasks, further demonstrating AUTOIF's generalization capabilities.
Model
InfoBiench
MT-Bench
Arena Hard (winrate)
Qwen2-78
79.25
8.12
11.85
AUTOIF
+SPT
81.92 (+2.67)
8.25 (e0.13)
(+2.65)
(2.75
8:31 (40.19)
(8.56(46.71)
Table 2: Cross-domain performance on gen- eral instruction-followii foBench (Qin et al. 2024b), MT-Bench Zheng
benchmarks: In-
let al. 2023). and Arena Hard
(Li et al. 2024c).
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on supervision models.
Table 4: Ablation study on specific components.
Model
IFFval
Follow Bench (SSR)
Model
Follow
Prompi(L)
Instraction(L.)
Arg
PromptiL) Instraction(L)
Avg
Qwen2-7B
43.6
53.4
52.3
Saperiition Model: Qwen2-728
+SFT
Supervision Model: Quvn2-738
44.5.19
S5A,20
53.3-00
Qwen2-7BBSFF w/ Online DPO
46.5
57.9
56.6
+SFT Ofllnne DPO
44,741.
562.23
54.0,10
+SFT Deline
46.5.50
57.4445
66.6413
Buck-trassaaion
-08
-1.7
Mode! GPT4
Quality Verification
-1.4
-1.3
+SFT & Offline DPO
+SFT
90.3 11.3
52.9.9.3
68.9 155
62.6,922
56.1,2,
54.422.1
w/o Cross Verification
-166
3.0
-1.5
4FFT & Oalline DPO
92..159
69.A..60
55.7.34
w/o All Quality Process
-22
-3.8
-2.
Inager
The left two figures illustrate the quality ablation studies on instructions and queries, whereas the right two figures present the scaling analysis of SFT data and DPO pairs.
Figure 4:
4.3 QUALITY ABLATION STUDY
Ablation Supervision Model. Tab. 3 presents the results of replacing the supervision model Qwen72B with GPT-4. We observe that in AUTOIF, a stronger supervision model (GPT-4) demon- strates more effective strong-to-wead distillation alignment, particularly evident with a performance gain of over 15% in the loose prompt in IFEval. This is reasonable because AutolF requires the su- pervision model to perform several tasks, such text augmentation (instruction, query, and response rewriting). code generation (verification function), and quality assessment (scoring). This implies that a supervision model with stronger fundamental abilities can synthesize higher-quality data when using AUTOIF.
Ablation on Specific Components. To investigate the effectiivnesss of various modules in AUTOIF, we conduct an ablation study, as presented in Tab. 4 we use w/o to denote the variant without specific module. The results reveal the following: (1) The performance of AUTOIF declines when any quality filtering process is removed, indicating that all components are highly effective. (2) The most significant performance drop occurs when the Cross Verification of instructions removed, underscoring its importance over query quality verification. This verify that high-quality instruction set is fundamental to the AUTOIF process. (3) Eliminating the overall quality filtering process results in more substantial performance drop than removing any single component, suggesting that quality filtering at both the instruction and query levels provides a mutually reinforeing effect.
Quality Control Instructions and Responses. In Fig. 4 (left), examine how varying pass rate thresholds of verification functions (indicative of data quality) affect the amount of SFT data and instucctoonfoolowing performance. As the pass rate threshold increases, the amount of SFT data decreases at the instruction level, while model performance consistently improves. This suggests that the quality of instructions is crucial factor influencing IF performance. At the query level, the SFT data amount also decreases with higher pass rate thresholds. Notably, performance peaks at a pass rate of 0.8 and declines beyond 1. This observation aligns with our expectations, indicating a trade-off between data quality and quantity.
4.4 ANALYSES
Scaling Analysis on SFT & DPO Data. Fig. 4 (right) presents the scaling analysis of SFT and DPO data using GPT-4 as the supervision model. The results demonstrate that even with just 1/64 of AUTolF-generater SFT/DPO data, Qwen2-7B achieves impressive performance, particularly with 1/64 DPO data reaching nearly 55% loose prompt accuracy. an increase of 11.4% pts. This strongly verifies the high quality of AuTollF-generaed data. Further analysis reveals that IF

---

Published as a conference paper at ICLR 2025
Scaling Analysis of Differont Parameters IFEval
Setup
Bench.
Train Test Rephr. Percentage N-gram
Bose Model
superstad by Qwen2 72B supervial by CP74
IFEval
0.01%
4.8%
70
ShareGPP
Followbench
25K
0011%
2.3%
1Phval Inw.
60
IFEval
10K
542
2
0.01%
3.5%
Qwen2-728
50
Followbench
12K
820
0.01%
0.9%
IFEval
0.01%
2.%
LLaMA3-700 70B
Followbench
17K
820
0.01%%
1.2%
30
5:48
Owee 5-78
1oomat-138 138
Quent 2-14
GPT4
IFEval
25K
542
0
0.11%
3.6%
Owent
Followbench
0L01%
Model Purameters
Figure 5: The scaling analysis of various param- eter sizes between the base model and different supervision models on the IFEval benchmark.
Figure 6: Contamination analysis on SFT data generated by different LLMs. Rephr. represents samples similar to the test sample.
capability steadily improves with increase in data quantity, scaling trend confirmed by numerous reasoning studies (Yuan et al. 2023; Muennighoff et al. 2024).
Scaling Analysis on Model Parameters. To investigate the impact of parameter scale on instruction- following performance, we gradually increased the parameters LLMs (ranging from 1.8B to 33B) and evaluated their performance. As shown in Fig. different supervision models achieve significant improvements across various model parameter sizes. Specifically, Qwen2-72B consistently improves the all base models' Ins.(L) by 6%, while GPT-4 achieves a stable improvement of over 12%. Furthermore, across all parameter sizes, the gains from GPT-4 consistently outperform those of Qwen2-72B. These results not only confirm that AUTOIF delivers substantial and stable benefits across different base model parameter sizes, but also highlight that stronger supervision models tend to produce more powerful effects.
observe that OUTolF-generatte
SFT data by
Contamination Analysis. We evaluate the contamination of the training dataset generated by AUTOIF on IFEv and FollowBench. Specifically, we employ contamination detectors from LM-Sys (Yang et al. 2023), which utilize advanced chatbots to identify potentially rephrased contaminated test samples. Additionally, we report contamination findings detected by traditional n-gram contamination algorithms. As shown in Fig. 6 both contamination rates are lower than those of the ShareGPT dataset we used. This allows us to confidently assert that there no contamination between the self-generated training samples and the test sets.
More cases can be viewed in Appx.
Data Efficiency. Tab. 5) exxplors the relation- ship between model coding ability, data quality pass rate (samples with a query quality score above 8), and instruction-folloiing capability. Surprisingly, we observe consistency in the su- pervision model across all three metrics. This indicates that the execution feedback resulting from the supervision model's coding ability sub- stantially influences data synthesis quality and the final capability.
Supervision
Total SFT Data DPO Data
Pass Rate
MBPP (Code) IPEval
LLaMA3-706
85K
15K
6k:
26%
70.4
43.1
Qwen2-72h
123K
10K
4K
28%
73.9
44.7
GPT4
210k
25K
15K
34%
87.5
59.3
Table 5: Data statistics and efficiency. Total de- notes the total data amount without quality control.
CONCLUSION
In this paper, we propose AUTOIF, a scalable and automated method to enhance the instruction- following abilities of LLMs. It uses self-instruct and rejection sampling to enhance the supervisory signals of seed instructions and relies on self- self- gefereeeeetted execution feedback for quality filtering. We introduce three training strategies and two alignment settings to comprehensively analyze AUTOIF. Experiments demonstrate that our method significantly improves performance across all settings in both IFEval and Followbench, with the first LLM achieving over 90% loose instruction accuracy. Additionally, AUTOIF's performance improvements on three other general instruction-followt following datasets, along with results from quantitative analyses, demonstrate its generalization and scalability.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.081: 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, 7ac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862 2022. doi: 10.45550/AXXI..204.00552 https://doi org/10. 488500/aXXin 220.
URL
Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xianpei Han, Le Sun, Hongyu Lin, and Bowen Yu. Towards scalable automated alignment of 1lms: A survey, 2024a.
Boxi Cao, Keming Lu, Xinyu Lu, Jiawei Chen, Mengjie Ren, Hao Xiang, Peilin Liu, Yaojie Lu, Ben He, Xiampei Han, et al. Towards scalable automated alignment of llms: A survey. arXiv preprint arXiv:2406.01 01 01552, 2024b.
Yupeng Chang, Xu Wang, Jindong Wang. Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021a.
Mark Chen, Jerry Tworek, Heewoo Jun Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel fierbentVooss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code, 2021b.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-dehug, 2023.
11

---

Published as a conference paper at ICLR 2025
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023, https
//lmsys.orggbloog.... 2023 30133--vvvvaaaa
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ulrraeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR, 2023.
Guanting Dong, Xiaoshuai Song, Yutao Zhu, Runqi Qiao, Zhicheng Dou, and Ji-Rong Wen. Toward general instruction instructtoo-followiing alignment for retrivaal-augmented generation. CoRR, abs/2410.09584, 2024a. doi: 1044550ARXXV22410005555 arXii.
Guanting Dong. Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang. Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. Lun-W Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 177-198. Association for Computational https://doi. http://...oorrrroo111///////////
Lingiioiiiiiiniiiiiiiii Linguistics, 2024b. doi: 10.1863/V1/2044AALLLLLM acl acn--ong.
URL
Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. Progressive multimodal reasoning via active retrieval. CoRR, abs/2412. 14835, 2024c. doi: 10.48550/ARXXI.2412..44
htt/https://doi.oo org.//....rg//... 48550 a0155//axxi...o 2412. 14835
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang. Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipliie chinese evaluation suite for foundation models, 2023.
Tingfeng Hui, Lulu Zhao, Guanting Dong. Yagi Zhang, Hua Zhou, and Sen Su. Smaller language models are better instruction evolvers. CoRR, abs/2412.11231, 2024. doi: 10.48550/ARXI.24411.. 11231.
48550/arXiv 2412. 11231.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint urXiv:2401.04088. 2024a.
Yuxin Jiang, Yufei Wang, Xingshan Zeng. Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: multi-level une-ggrined constraints following benchmark for large language models, 2024b.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022.
12

---

Published as a conference paper at ICLR 2025
Shanglin Lei, Guanting Dong, Xiaoping Wang. Keheng Wang, and Sirui Wang. Instructere: Re- forming emotion recognition in conversation with a retrieval multi-task Ilms framework. CoRR. abs/2309.11911, 2023. doi: 1044850/AARXIV23309111 2309.
httpss//:///d1... org/ /ttrg10..4555 48550/
Chengpeng Li, Guanting Dong, Mingfeng Xue, Ru Peng, Xiang Wang, and Dayiheng Liu. Dotamath: Decomposition of thought with code assistance and self-correction for mathematical reasoning. CoRR, 0ttp//2470004773 2024a. doi: 1048550/ARRXI:2247..40777 URL httpps://doi.org/
18550/arXiv, 2407. 84078
Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Mugglemath: Assessing the impact of query and response augmentation on math reasoning. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 10230-10258. Asso- ciation for Computational Linguistics, 2024b. doi: 10. 18853/v1/20024 acl-aong..5555
URL
https://doi.org//
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024c. /ttp://ts://aaxii org/ abs 11939
Jiate Liu, Yigin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: reinforcement learning from unit test feedback. Trans. Mach. Learn. Res., 2023, 2023. URL
https://openreviee net wwwtteoummmeee wwtp:///oorrmmiiinnnn/nnsmi
Renze Lou, Kai Zhang, and Wenpeng Yin. A comprehensive survey on instruction following, arXiv preprint arXiv:2303. 10475, 2023.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang. Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii. Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak. Sébastien Paquet, Jennifer Robinson. Carolyn Jane Anderson, Nicolas Chapados, and al. Starcoder 2 and the stack v2: The next generation. CoRR, abs/2402.19173, 2024. doi: 10.48550/ARXIV.2402.199
URL https: //oii org/ 10. 48550/ (03550/axxii arv. 19177.1.1979
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://openreviies http:////ppereeeeeeeeeee//o//ouo http:nnee/ooofmm///-KE//EEEOOO
Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL
https://al.meta.com// mmmm://bloommeeaaaaaaaa3333 11ama-3/
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
Alhassan Mumuni and Fuseini Mumuni. Data augmentation: A comprehensive survey of modern approaches. Array, 16:100258, 2022.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida I. Wang, and Xi Victoria Lin. LEVER: learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 26106-26128 26128. PMLR, 2023. mlr, prwss//202/ni22b.... html
URL http://ceeoooonaaa.rrrrrr///////
13

---

Published as a conference paper at ICLR 2025
OpenAI. Introducing chatgpt, 2022. BBLLLnttss///oeeaa.cc/////eeed/s
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bermadett-Shapinn Shappioo, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Camphell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi. Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Lontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanic Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichro Nakano. Rajeev Nayak, Arvind Neelakantan, Richard Ngo Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish. Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeff WU, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277 2023.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma Gongque, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong. Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model
14

---

Published as a conference paper at ICLR 2025
achieve human-like mathematical reasoning? CoRR, abs/2407.01284, 2024a. doi: 10.48550/ ARXIV.2407.01284, URLhtttps://di..rgg wrt./o/.g//......1... 240.7.....4..... 2407.01288
Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback, 2024b.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024a.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models, 2024b. URL
03681
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. KDD '20, 2020.
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang. Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained instruction-follownng ability of large language models, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An cestructionfollowinn llama model, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-lim collaborative annotation through effective verification of Ilm labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2024a.
Yejie Wang, Keqing He, Dayuan Fu, Zhuoma Gongque, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang, Xunliang Cai. and Weiran Xu. How do your code 11ms perform? empowering code instruction tuning with high-quality data. CoRR, abs/2409 03810, 2024b. doi: 10.48550/ARXIV.2400000 .0009000011...0 URL hhttps://dot org/10 48550/arXiv. 2409. 03810
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023a.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution- evaluation for open-domain code generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore. December 6- 10, 2023, pp. 1271-1290. Association for Computational Linguistics, 2023b. doi: 1118663//1/20222 18653/v1/2222 Findings-emmlp.89 FINDINGS-FMNNIPP https:ttp:://di.. wwt.////....//////////1
15

---

Published as a conference paper at ICLR 2025
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T. Le, Jin Miao, Zizhao Zhang, Chen Yu Lee, and Tomas Pfister. Codeclm: Aligning language models with tailored synthetic data, 2024c.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 2021.
Jerry Wei, Chengrun Yang. Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al. Long-form factuality in large language models. arXiv preprint arXiv:2403. 18802, 2024.
Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. Fofo: A benchmark to evaluate Ilms' format-following capability, 2024.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances neural information processing systems, 33:6256 6268, 2020.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023.
Jianhao Yan, Yun Luo, and Yue Zhang. Refutebench: Evaluating refuting instruction-follooing for
large language models, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfci Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CORR, abs/2407. 10671, 2024a. doi: 0..4550/ARXIV.2447.7 10671. URL httph: org/ /00 48550/ 2407 10671
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024b. doi: http:://d/d/ org/10 48550/arXiv.... 1412.
Shuo Yang, Wei-L.in Chiang, Lianmin Zheng, Joseph E. Gonzalez, and lon Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint 2:Xi::3308.01825, 2023.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10799, 2023.
YiFan Zhang, Shanglin Lei, Runqi Qiao. Zhuoma GongQue, Xiaoshuai Song, Guanting Dong, Qiuna Tan, Zhe Wei, Peiqing Yang, Ye Tian, Yadong Xue, Xiaofei Wang, and Honggang Zhang. Multi-dimensional insights: Benchmarking real-world personalization large multimodal models. CoRR, abs/2412.12606 2024. doi: 10.4850/ARXX..2442.1050 URL htthtsht//oiiiiii. http://too.oooo///////// 18850/arXiv.2200 2412. 12606
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L. Zhang. A preliminary study of the intrinsic relationship between complexity and alignment. 2023.
16

---

Published as a conference paper at ICLR 2025
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Minghao Li, Fei Huang, Nevin L Zhang, and Yongbin Li. Tree-instruct: A preliminary study of the intrinsic relationship between complexity and alignment. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 16776-16789, 2024.
Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmenta- tion, Advances in Neural Information Processing Systems, 36, 2024.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. Utthtttss://arxiv. www.worgg//a/....///
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Luan, Denny Zhou, and Le Hou. Instruction insllwction-folloiinng evaluation for large language models, 2023.
17

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
In this paper, we propose AUTOIF, system for automated instruction augmentation and quality filtering, capable of scaling to over 10,000 instructions. While our focus is not on the construction of cross-instructions, the excellent results achieved in two cnstructioo-folloiiinn benchmarks demonstrate the generalizability of our method in handling complex instruction-followe believe more direct strategy would involve combining multiple simple instructions into cross- instructions, and subsequently enhancing and potential further amplify the effectiveness of our method. Therefore, we consider automating and scaling cross-insee coossinsttuution tasks as a key direction for future research.
tasks. Additionally,
them using AUTOIF. This way has the
B ETHIC CONSIDERATION
In this paper, we have fully presented the seed instruction set used by AUTOIF in the Appendix. All concatenated queries are sourced from the publicly available ShareGPT dataset and have undergone multiple steps of quality filtering. Therefore, our method strives to minimize potential safety and ethical risks as much as possible. However, during the rejection sampling process, malicious prompts lead the model to produce harmful inappropriate outputs, which is shared problem. Ensuring the quality of generated content safe and controllable manner crucial. The application of these techniques should be guided by ethical considerations, with safeguards in place prevent misuse and reduce the likelihood of producing harmful outcomes.
C SEED INSTRUCTIONS
illustrates our hand-written seed instructions.
Seed Instructions
Dien only wiiiaay liigs
like only apthaler (1-10
Coestrvet the reply iefggee STOP
Phopprd with nelka (57- sycteble strekrrn
Use only eventhrr tien your reply
sivies the resparme
tiee weess tuat
words Aing
Widee Moree
(1-2)
atogle suntence in suantly veech loep
Pespord with abowert
Rrspond palet
eauds conpasnn the
Autica must cignnng paice
with aaqunecs jaan
15 Transiar your eruere rsto emagl
Whtr the resporee lorble
Arowcr with eoegl only
11 convery 1000 evet woeses Regliih work
Liee woess suat hove Vivs Miree
fenour movie awrieast
Answer frre
Anower
with nest fatter abtabee
Figure 7: Examples of our seed instructions
D IMPLEMENTATION DETAILS
To better motivate researchers to reproduce the results, we report the detailed experimental details:
In the SFT phase, we perform full fine-tuning on Qwen2-7B and LLaMA3-8B 8B with learning rate of 7 using a linear scheduler with 20 warm-up steps. All models are trained with DeepSpeed ZeRO Stage 3 (Rasley et al. (020) and Flash-Attention 2 (Dao 2023). We use global batch size of 128, a weight decay 0.1, and train for epochs. Mixed precision training with 6 is used, and the maximum context length is set to 8192 tokens. For Qwen2-72B and LLaMA3-70B, the global batch size is 512.
In the DPO phase, the learning rate is set to 5e-7 with cosine scheduler and a 0. warm-up ratio. We use DeepSpeed ZeRO Stage and Aath--nttnntinn 2 for efficiency, with a global batch size of 64. Training utilizes sigmoid loss function with beta value of 0.3 and spans 2 epochs, with checkpoints every 200 steps. Mixed precision training with b is employed. and the maximum context length is 4096 tokens.
18

---

Published as a conference paper at ICLR 2025
We run all our experiments on NVIDIA A100 and H800 GPUs. Specifically, we train Qwen2-7B and LLaMA3-8B on A100 GPUs, while Qwen2-72B Dwwn2-72B-Instrucct and LLaMa3370B-Inss -on--nntt on 64 H800 GPUs. Notably, instruction- followinn capabilities. For evaluations, we report pass results with greedy decoding for HumanEval and zero-shot accuracy for GSM8K. We report averaged performance from five randomly seeded experiments.
an in-house version of Qwen2-7B without any targeted optimizations on
E DETAILS OF AUTOIF
At the instruction level, for the self-instruct stage, we perform RFT with K=100 seed instructions. During the Automated Quality Cross Verification stage, we filter the quality based on four criteria mDeberta
outlined in the main text. For NLI filtering, only samples predicted as "Contradiction" (approximately 15%).
filtering mode O and filter out
At the query level, randomly select 16 ShareGPT samples for each instruction and perform Response Rejection Sampling with K=8. For instruction following verification, we adhere to the two standards mentioned in the text. Finally, for query quality verification, we filter for consistency using a threshold of 8.
F CASE STUDY OF DATA COMBINATION
We used n-gram 13 to evaluate the overlap between each test sample and the SFT training samples. It unnecessary evaluate DPO data since the inputs for DPO data are derived from SFT data. In Fig. 6] all our data combination metrics (both model- based and rule-based evaluation) are lower than those ShareGPT, confirming that our method has no data combination with the test set. We also present the top 5 training-tes, sample overlaps in n-gram for both IF Eval and Followbench in Fig. 8
Case study
IFEVAL
On Follow Bench
Aigran
Trais date
That chts
Migrass
Trais data
That dets
ciss shat sang sang outer
cos All dasgeross
You dectas Please muphals now coceasa
You are Basslas plysicc
spice la Wappy dlrshay." Your axswer uuat
Prappred with suntance dhatiooluee every
with type slabwre cas calouirs the rotal
rndiciour problem the course Gunnar avcicsic Webe
uonaal wilswing plowes Bly
fetro sted Head
of daily cantallaaoons, they
My
whooo sverhwed??
tamphate a preckct description pawce po aesterpp stenting with PUP.
White ar repporse wordh
How mraor thet namart reqpand the wilking stgle drutoscpaore
Now propedty offeard avera Microst Auserdball eace swntenee belngg slatweend
how cauuge cudipes
redenstenters the cnly words thtt
of redly pwet.
parograph
evarpee leegth
Weuld you consiter strect carben experaavv tnchoology' Please procids use reason ixpport your opentos.
you enthr ender Flassy Bod game?l Axtive with woods that homsphore.
ruvious anleut speciemns from ao
aloobal daly good the body. than cip
Argest Your repporee tf corsaen bazy
your atarraa Whthe the resporse dite
shae snvtiisee.
eet lecrrctiinn for siegai soot, Mkx gaika
steelowes
Coe you coopge about bicttry of
Cas you wnthe can Asueciii scuipe
Could you altar a stary Jbout esuator plrpacis,
Goald pau explis to adott duseric pregremening ere, usteg TypeStrpp compasee?? the alibersiis oeree
the profectton syvthm Aeres? tiee eneety
Windows thet tias off member groupe and
reatseeingg tow eew and weedte
two repl artociir
Bek eoneders? Wel reppessee
Carl degue's style envettsss?
Wi
ctant
Can
withe decbll conlcc colee
dargugge
Figure 8: Case Study of data combination IFEval and Followbench
G PROMPT TEMPLATES
For the Self-Instruct stage, we use the following prompt template for instructions" rejection sampling:
'The ILI model is available at
https:///uugiiggaae.....///rrttt
mDeBERTa-v3 wa-base-xnli-mmaeeeeeee--
19

---

Published as a conference paper at ICLR 2025
Prompt Template of Self-Instruct Stage
You are an expert for writing instructions. Please provide (K) different instructions that meet the following requirements:
Instructions are about the format but not style of response
Whether instructions are followed can be easily evaluate by Python function
Here are some examples of instructions we need:
(Seed Instructions)
Do not generate instructions about writing style, using metaphor, or translation. Here are some examples of instructions we do not need:
Incorporate a famous historical quote seamlessly into your answer
Translate your answer into Pig Latin
Use only words that are also a type of food
Respond with a metaphor in every sentence
Write the response as if you are character from a Shakespearean play
Please generate one instruction per line in your response and start each line with
For generating the verification functions and test cases for each instruction, the following prompt template for rejection sampling:
Prompt Template for Generating Verification Functions and Cases
You are an expert for writing evaluation functions in Python to evaluate whether response strictly follows an instruction.
Here the instruction: (instruction)
Please write Python function named "evaluate" to evaluate whether input string 'response" follows this instruction. If it follows, simply return True, otherwise return False. Please respond with single JSON that includes the evaluation function in the key 'func', and list of three test cases in the key 'cases", which includes an input in the key 'input" and an expected output in the key 'output' (True or False). Here is an example of output JSON format:
"func": "JSON Str", "cases": [[ "input": "str", "output": "True" J. "imput": "str", "output": "False" }
For the back translation process of each verification function, we use the following prompt template:
Prompt Template for Back Translation
You are expert in converting Python eval function code into the corresponding instruction text. will provide the eval function code. Please strictly follow the code convert it into the corresponding instruction text.
Here's an example:
(Example func)
(Example cases)
Please convert the following eval function into instructions stored in list:
(funcs)
For the rejection sampling of query responses, the following prompt template:
Prompt Template for Response Generation
Please answer the query strictly following the instruction. Instruction: (instruction)
Query: [query)
20

---

Published as a conference paper at ICLR 2025
Fot the query quality verification, use the following prompt template:
Prompt Template for Response Generation
You are expert that is good at judging whether a response is following the instruction and query.
Instruction: (instruction]
Query: {query)
Response: fresponse)
Please notice that the response may not be helpful as it needs to strictly follow the requirements in the Instruction.
You need to judge whether the response answers the query. Please first provide a detailed analysis and then give score ranking from 0 to 10 at the last line.
Scoring O means the response is totally unrelated to the query, while scoring 10 means the response helpful and highly related the query.
Please only provide a score in the format 'Score: score' without any other contents at the last line.
H BASELINES & DATASETS
We give introductions the LLM baselines for our instruction following.
developed by MetaAI, the latest iteration of the LLaMA series, featuring significant upgrades. Compared to LLaMA2, LLaMA3 expands its training dataset, context length, and vocabulary, resulting in improved performance across various tasks. Enhancements in contextual understanding and language generation further distinguish LLaMA3.
LLaMA3
Meta 2024),
Qwen2 Baii et al. 2023), developed by Alibaba, includes five sizes: Qwen2-0.5B, Qwen2- 1.5B, Qwen2-7B, Qwen2-57B-A14hh and Qwen2-72B. Trained on high-quality data in Chinese, English, and other languages, Qwen2 excels in multilingual capabilities and shows strong performance in coding and mathematics. Additionally, it supports extended context lengths of up 128K tokens (Qwen2-72B- III--nstruct,, making it ideal for long texts and complex tasks. Thus, the version of Qwen2- Instruct, contacted the Qwen team and obtained the model weights where they did not optimize IF specifically, rather than the final open-source model.
Mistral-7B (Jiang et al. (023), released by Mistral AI in September 2023, leverages grouped query attention (GQA) combined with sliding window attention (SWA) to efficiently process sequences of any length, enhance inference speed, and improve throughput. It outperforms many 13B models across various tasks.
Mixtral-8x77 (Jiang et al. 2024a) developed by Mistral AI, is the first open-source MOE large model. It is sparse mixture of experts network and, like Mistral 7B, employs the GQA mechanism. With a smaller parameter count compared to LLaMA2-70B and GPT-3.5, it outperforms them across numerous tasks.
GPT Series GPT-3.5 (OpenAl 2022) and GPT-4 (Achiam et al. 2023), developed by OpenAl, are advanced models in the GPT series that use a three-stage reinforcement learning with human feedback (RLHF) algorithm. This enhances their mnstruction-follovin capabilities and minimizes harmful content generation. GPT-3.5 excels in text completion, translation, and summarization. Building on these strengths, GPT-4 further refines the RLHF algorithm, enhancing performance on complex instructions and making it suitable for applications ranging from academic research to industrial use.
In addition to the two Instruction- -oollwiinnn benchmarks introduced in the main text, we also provide detailed overview of datasets covered in the experiments
ShareGPT refers to the multi-turn chatting histories used by Vicuna Chiang et al. 2023). ). ShareGPT includes 86K human queries and responses from ChatGPT and other chatbots. We randomly select
21

---

Published as a conference paper at ICLR 2025
2w samples train LLaMA3-8B and Qwen2-7B to obtain our baseline models: LLaMA3-8B (ShareGPT) and Qwen2-7B (ShareGPT)
GSM8K (Cobee et al. 2021 is mathematical dataset designed to evaluate the mathematical problem-solving abilities of language models. consists of 8,000 diverse grade school-level math word problems, which require understanding and manipulating mathematical concepts arrive at correct solution. It comprises high-quality grade school math problems, with 7,473 training samples and 1,319 testing samples.
HumanEval (Chen et al. b02 includes 164 unique programming challenges, each paired with approximately 9.6 test cases average. To provide more comprehensive evaluation of the functional accuracy of code generated by large language models, HumanEval+ substantially increases the number of test cases an average of 774.8 per problem. In this paper, we report the Pass @ result when applying greedy decoding.
MMLU (Hendrycks et al., 2021) is a benchmark designed to assess pretraining knowledge in models using zero-shot and few-shot evaluations. It includes 57 subjects across STEM, humanities, social sciences, and more, with difficulty levels ranging from elementary to advanced professional. MMLU tests both world knowledge and problem-solving skills, covering traditional disciplines like mathematics and history, as well as specialized areas such as law and ethics.
C-Eval (Huang et al. 2023) consists of multiple- questions categorized into four difficulty levels: middle school, high school, college, and professional. The questions cover 52 varied disci- plines, including humanities, science, and engineering. Additionally, there is C-Eval Hard, subset of particularly challenging topics within C-Eval that demand advanced reasoning skills. We perform an in-depth evaluation of leading language models on C-Eval, testing both English and Chinese-focusee models.
MT-Bench (Zheng et al. 2023). MT-BBench is comprehensive benchmark designed evaluate the performance of multitask learning models, specifically targeting their capabilities in multi-turn dialogue and instruction-followinn tasks. This benchmark consists 80 high-quality multi-turn dialogue questions, covering eight common use cases: writing, role-playing, information extraction, reasoning, mathematics, coding, knowledge I (STEM), and knowledge II (bumanities/social sciences). MT-Bench focuses on challenging questions to better differentiate between the capabilities of various models.
Arena-Hard (Li et al. 2024c). Arena-Hard is significant dataset used to evaluate the robustness of dialogue systems, specifically designed to test a model's 's performance in handling challenging and diverse dialogue scenarios. The dataset comprises 500 carefully selected, challenging user queries that represent complex situations often encountered in real-world conversations, such as language variations, spelling errors, and grammatical mistakes.
(Qin et al. 2024b) InfoBench a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories for evaluating LLM's instruction following capabilitics.
InfoBench
I CASE STUDY OF AUTOIF
In Tab. 6 and Tab. 7 we illustrate the data format of AUTOIF, including the query, response (verification function Acc 0.8), and verification capability.
Specifically, in Tab. 6, we present the data format of AUTOIF under single constraint. Moreover, the motivation of this text is generalize more complex cssruction-followiiii abilities using AUTOIF a scalable and verifiable method-throup simple handwritten instructions. Therefore, showcase synthetic data examples of multi-atom complex instructions in Tab.
FFolow the set up of Dong et al. use the version from httpp://hugging:: http://uiinnfcc.cc//tataaaaaaaae wat..coo/aaaaa co/datasess/ Smon223148912//hhareGP wttpar__iicuuaaffifffeeeeee cleaned raw dataset, and follow Vicuna preprocess.
22

---

Published as a conference paper at ICLR 2025
Query
Response
Verification Funes
def evaluate (response: str) -> bool:
Keep your answer under characters total. what is autoarima in python.
AutoARIMA automates
return len (response)
ARIMA model selection.
def evaluate (response): forbidden 'sS' for char in response: if char in forbidden:
Refrain from using any words that contain 'S' need a name for my Dungcons and Dragons world.
EtemaRcclm
return False return True
def evaluate (response: str) -> bool: return len (response. split()) <= 20 and (eeesponss))
Keep your response under twenty words without sacrificing clarity. Let's play game shall we?.
ure, let's play game! What game do you have in mind? Please keep instructions simple and clear.
Normal forms in databases:
INF ensures atomic columns, 2NF eliminates non-key dependencies, 3NF removes transitive dependencies, BCNF enforces determinant restriction.
def evaluate (response: str) -> bool: return len (response.
Compose your answer using exactly 20 words. Diffrent Normal Forms.
split()) == 20
def evaluate (response: str) bool:
lines response.... split
Decision trees, so clear and bright, Branch out split data's might, With nodes questions, true or false, They sort through cases, young old, like versatile horse. From root leaves, paths decide their course.
'An')
if len(lines) != 5:
Word your response a five-line verse with a strict AABBA rhyme. 1. Write short notes Decision tree...
return False
rhymes [line[-]] for
line lines]
rhyme_pattern = rhymes
[0] == rhymes ==
rhymes [2] !=
rhymes [3] rhymes [4]
return rhyme_pattern
Table 6: Examples of AutolF's data formats with single constraint.
23

---

Published as a conference paper at ICLR 2025
Query
Response
Verification Funes
def evaluate (response):
words response split()
if len(words) 4 8: return False sentences [" jooin (words[i:1+4)) for in (onge lenowods)) 4)
Sunny skies expected today. High temperatures around eighty. Chance of rain low. Winds from the east. Cool evenings ahead tonight. Prepare for sunshine tomorrow. Clear conditions throughout the week. Enjoy the pleasant weather!.
Write the response a scries of four-word sentences. Answer with words that are usually used in weather forecasts.
beather_terme set
"cloudy', 'sunny', 'rain', 'now'' 'widdy" 'llear' 'fog', 'temperate' 'chilly', 'warm', cool', 'storm', 'humidity', pressure', 'visibility"
])
for sentence sentences:
for word in sentence. split(): sentence.split(): if word lower not in return False
return True
def evaluate (response)
def scrabble_sce score (word):
scores {
a 1, bb' 3, ' 3 'd':
ee': 1, 'f': ''':
4, 'i"; 1, 'j 8, 'k's
5, ' 1, 'm': 3, 'n':
10': 'p':
'q 10, 'r':
Respond with words in which the first letter is capitalized (proper nouns acronyms). Construct the reply with each word having higher Scrabble score than the previous.
1, 1s": 1, 't': 1, 'u':
Iv' 4, 'w": 4, 'x': 8, 'y':
4, 12'; 10
}
Sky Rain Wind Storm
return sum (scores get (char lower
O, 0) for char in word)
words response split()
if not all (word [0]. isupper () for word in words): return False
scrabble_scores [scrabble_score( word) for word in words]
than the previous
return all(current prev for prev,
current sip(scrabble_scores scrabble_scores[1:1
Table 7: Examples of AutolF's data formats with multiple constraints.
24

---

Published as a conference paper at ICLR 2025
J THE ILLUSTRATION OF FOLLOWBENCH'S TEST CASES
The motivation of AUTOIF is to generalize more complex instruction-follooing abilities by using a scalable and verifiable method through simple handwritten instructions. Therefore, we present the lest examples from the complex instruction-followe Followbench evaluates six dimensions, with each instruction having five levels of difficulty and comprising series of integrated tasks. Below are three features of Followbench.
evaluation set Followbench we assessed.
Six Dimensions's Tasks of Followbench: All constraints being evaluated for instruction following under the combination of various integrated tasks
1. Content Constraint: Data-to-Text Generation, Document-Level Event Argument Extraction, Document-Level Named Entity Recognition, Text Generation with Language Constraints, Open- ended Question Answering
Situation: Suggestion Generation, Role-playing. Complex Situation Reasoning
3. Style: Open-ended Question Answering
Format: text-to-Tablee Generation, Open-ended Question Answering
5. Example: 40 diverse NLP tasks
Mixed: Text Editing, Summarization, Machine Translation, Story Generation
Examples of Constraints in Six Dimensions: Each instruction's 's complexity cannot be resolved solely through surface semanties or 1-to-1 translation.
Category
Test Case Description
What, according Milton Fricdman, is the role of business in society? Additionally, analyze its influence on ethical standards socicty and identify one possible repercussion on relationships within the community. Please strengthen your argument with one relevant case study and its implications, along with citing expert opinion statistical data to support your viewpoint.
Content
Lost, found vodka, drank to forget. According to above prompt, write four-sentence story that describes a man. However, the word "man" should not appear in the story. Please write using an introspective narrative tone. You should also describe something about the bad weather.
Mixed Prompt
yesterday is Christmas Eve 1937, what would be the date four years, a month, two weeks and two days after today in MM/DD/YYYY?
Situation
How did US states get their names? Pray, respond in the writing style of Shakespeare and the elegance of the Victorian era, whilst infusing touch of humor into thy discourse, Furthermore, craft thy response with the ambiguity reminiscent of oracles of ancient Greece, leaving room for pondering and interpretation. As thou writest, channel the conciseness and vigor of Hemingway in thine articulation.
Style
Robert just called and had some more details. He talked Gay again. Sunny is OK, walked away from the wreck. It totaled her car. The airbag did not inflate so she was very lucky not to he hurt. He will report more when he gets there. Randy J.
Example
To enhance your time management skills, could you devise a method incorporating a mind map and featuring touch of alliteration in the suggestion, ensuring your answer must follow the above suggestions.
Format
Examples of Five Difficulty Levels: For one constraint, the sentence's semantic structure greatly altered at higher levels:
Similarly, IFEval is a complex instruction evaluation combining multiple instructions and remains core benchmark for foundational model instruction adherence
foll following_eval
25

---

Published as a conference paper at ICLR 2025
Difficulty
Test Case Description
Level
Identify one category from the list helow for the input text, and also infer the sentiment (positive, neutral, or negative) conveyed in the text. Your options for the category are- company, educational institution, artist, athlete, office holder, mcans of transportation, building, natural place, village, animal, plant, album, film, or written work. Michael DenDekker- Michael G. DenDekker (born July 11. 1961) is assemblyman for the state New York's 's 34th district which includes the neighborhoods of Woodside, Jackson Heights, and East Elmhurst, all in the borough/county of Queens.
Level 2
Identify one category and the sentiment conveyed (positive, neutral, or negative) in the input text, well conduct named entity recognition task to locate and highlight the important entities present. You can choose the category from the following: company, educational institution, artist, athlete, office holder, means of transportation,, building, natural place, village, animal, plant, alhum, film, or written work. Michael DenDekker Michael G. DenDekker (born July 11, 1961) is assemblyman for the state of New York's 34th district which includes the neighborhoods of Woodside, Jackson Heights, and East Elmhurst, all the borough/county of Queens.
Level 3
Analyze the provided text to pinpoint a category and the sentiment (positive, neutral, or negative) it emanates. Additionally, perform named entity recognition to emphasize notable entities and also identify the core topic discussed. Select the category from this arry: company, educational institution, artist, athlete, office holder, means of transportation, building. natural place, village, animal, plant, album, film, or written work. Michael DenDekker Michael G. DenDekker (born July 11. 1961) is an assemblyman for the state of New York's s 34th district which includes the neighborhoods of Woodside, Jackson Heights, and East Elmhurst, all the borough/county of Queens.
Level 4
Analyze the supplied text to discern a category and the sentiment it conveys (positive, neutral, or negative). Furthermore, carry out named entity recognition to highlight significant entities and determine the main theme heing discussed. In addition, perform keyword extraction underline notable terms. Choose the category from this array: company, educational institution, artist, athlete, office holder, means of transportation, building, natural place, village, animal, plant, album, film, or written work. Michael DenDekker- Michael G. DenDekker (born July 11, 1961) is an assemblyman for the state of New York's 34th district which includes the neighborhoods of Woodside, Jackson Heights, and East Elmhurst, all in the borough/county of Queens.
Level 5
Analyze the provided text to ascertain both the category and the sentiment (positive, neutral, or negative) it embodies. Additionally, conduct named entity recognition to emphasize important entities and establish the central theme. Moreover, undertake keyword extraction mark prominent words, and engage in coreference resolution identify references the same entity within the text. Select the category from this array: company, educational institution, artist, athlete, office holder, mcans of transportation, building, natural place, village, animal, plant, album, film, written work. Michael DenDekker- Michael G. DenDekker (born July 11. 1961) is an assemblyman for the state New York's 's 34th district which includes the neighborhoods of Woodside, Jackson Heights, and East Elmhurst, all in the borough/county of Queens.
Therefore, our cases and responses prove that the instruction following tasks are highly challenging, assessing the comprehensive capabilities of LLMs.
K MORE EXPERIMENT RESULTS OF AUTOIF
K.1 VALIDATION IN LONG CONTEXT INSTRUCTION-FOLLOWING SCENARIO
To validate the generalization of AUTOIF in the fields of RAG and long windows, we conduct verification experiments on the FollowRAG benchmark (Dong et al.) 2024a). As shown in Table [ AUTOIF still shows significant improvements in long text scenarios, which further validates the effectiveness of our method in woal-world challenging instruction-follooiing contexts.
26

---

Published as a conference paper at ICLR 2025
Model
NQ (IF)
NQ (RAG)
NQ (AVG)
TQ (IF)
TQ (RAG)
TQ (AVG)
Llama3-8B-SFT
15.7
59.5
37.6
15.0
76.5
Llama3-BB (AutolF)
41.3
62.4
51.9
40.3
77.6
60.0
Table 8: Performance comparison of models FollowRAG NQ and TQ benchmarks. Llama3-8B- SFT represents Llama3-8B finetuned on ShareGPT dataset and train set of NQ and TQ.
K.2 MORE SETTINGS ON Low RESOURCE SCENARIO
To validate the generalization of AUTOIF in scenarios with lighter resource consumption, we conduct experiment using Llama3-8B-instruee for self-alignment with Llama3-8h Baana-,--- which can be effectively deployed using just one GPU. Additionally, to further challenge AUTOIF's potential in more demanding scenarios, we designed Weak-to-Strong setup, enhancing Qwen2-7B with Qwen2-3B-instruck, This setup also requires only one GPU for effective deployment. As shown in Table 9 in both low-resource settings, AUTOIF consistently demonstrated stable improvements, highlighting its effectiveness.
Method
Pr. (strict)
(L)
Ins. (S)
Ins. (L)
FollowBench (Avg.)
Supervision Model: Qwen2-3B
Qwen2-7B-base
37.7
43.6
49.4
53.4
52.3
Qwen2-7B 7B (ShareGPT)
30.9
33.5
42.4
45.2
38.1
Qwen2-7B (AutolF)
40.3
46.0
53.5
56.8
53.0
Supervision Model: Llama3-8B
Llama3-8B-bnl
24.6
26.1
38.1
39.7
11.6
Llama3-8B (ShareGPT)
23.7
26.4
33.8
37.1
38.1
Llama3-8B (AutoIF)
32.5
37.7
49.2
44.2
Table The weak strong and self alignment setup on low resource scenario.
L DISCUSSION ON CODE EXECUTION WORKS
Recent advancements in code generation and verification have produced several effective approaches.
RLTF (Liu al. 2023) generates data real-time during training, utilizing multi- guantiigaanulrrtt unit test feedback to identify specific code errors, which helps improve code quality.
LEVER (Ni al. 2023) enhances this process by training verifier that assesses the correctness of programs generated by large language models (L.LMs). It evaluates the generated code based on natural language inputs, execution results, and reorders candidates using a combined score of verification and LLM probability, ensuring optimal solutions.
Dotamath (Li et al. 2024a) tackles complex mathematical tasks by decomposing them into simpler logical subtasks, leveraging code to solve these subtasks, obtaining fine-grained feedback from the code interpreter, and engaging in self-reflection and correction.
ODEX (Wang et al.) 2023b) introduces the first open-domain dataset for execution based natural language to Python code generation, featuring 945 natural language-code pairs across 79 libraries and 1,707 manually written test cases for validation. This dataset is vital for training robust models in diverse programming contexts.
Lastly, Self-OSS Instruct (Lozhkov et al. 2024) leverages context learning to enable the StarCoder2 model to autonomously generate diverse programming instructions from seed code snippets. This in- cludes concept extraction and instruction generation, fostering self-sufficient learning environment.
27

---

Published as a conference paper at ICLR 2025
Collectively, these works highlight the importance of real-time feedback, verification mechanisms, comprehensive datasets, and self-learning strategies in enhancing the quality and reliability of code generation.
M FUTURE WORK
AUTOIF, which first transforms instruction-following alignment into automatically code verification, requiring LLMs to generate instructions, corresponding verification code, and unit test samples for cross-validation. In the future, find that constructing and verifying high-level semantic instructions (such as those with emotional or creative elements) is a key direction for enhancing the LLM alignment with human instruction following. Specifically, believe there are several optimization avenues for AUTOIF to better accommodate high-level semantics:
Handwritten prompts: We can consider fine-grained emotional differences in the prompts by handwriting instructions that allow for nuanced distinctions.
Instruction rewriting phase: We establish creative principles (e.g., for emotional assistant, qualities like humor and empathy) and allow humans to iteratively optimize these principles based on the quality of generated outputs from small batches, potentially using instruction evolution techniques like AutoEval instructions [1]. Principle of LLM verification: Inspired by CAI [2], we also need to incorporate fine- tne-grained emotional differences in the verification prompts during the verification phase or creative metrics for scoring, rather than solely focusing on instruction correctness, to overcome the limitations executor-based verification that only addresses verifiable prompts.
Online/Offline DPO data construction: For creative tasks, we should avoid using executor- based success rates to construct positive and negative samples. Instead, a combination of LLM verification scores and executor-based scores should be employed to balance correctness with higher-level emotional semantics.
28


Reviews:
[{"review_id": "JIhWxR2Kd3", "reviewer": "Reviewer_61cd", "summary": "This work presents AUTOIF, a generic and automated framework for the generation of high-quality training data to enhance the instruction-following prowess of LLMs. The main novelty of this work is that it reduces instruction-following alignment to a verifiable process using code. AUTOIF leverages execution feedback from code verification and rejection sampling to guarantee that the generated responses meet given constraints. This technique has paved the way for preparing both positive and negative examples for algorithms using SFT and RLHF. The authors mention that AUTOIF substantially improves the performance of LLMs on many benchmarks without weakening other competencies, such as coding and mathematical reasoning.\n\nBesides this, the paper presents an extensive experiment to show that AUTOIF improves the instruction-following accuracy of a range of LLM architectures and sizes, from Qwen2 and LLaMA3 models. The paper underlines that indeed one of the keys to good performance is iterative online training and at the same time develops the scalability and generalization potential of AUTOIF across diverse types of instructions.", "questions": "How could the framework be extended to manage instructions that are inherently ambiguous or have multiple valid responses?\n\nWhat modifications to AUTOIF would be necessary to accommodate tasks involving emotional nuance or creative outputs?\n\nCould providing more detailed information on errors and failure rates improve the replicability and robustness of AUTOIF?\n\nWhat specific criteria were used to filter low-quality instructions, and can this process be made more transparent?\n\nWhat are the computational overhead implications of using execution feedback at scale, and are there any optimizations or trade-offs that can be explored?\n\nHow does AUTOIF perform in resource-constrained environments, and is there a way to quantify these trade-offs more concretely?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Extensive Performance Improvements: AUTOIF leads to significant gains across various instruction-following benchmarks, such as IFEval and FollowBench. It notably achieves over 90% loose instruction accuracy on IFEval while preserving or even slightly improving performance on other key capabilities like coding and mathematical reasoning.\n\nComprehensive Evaluation and Ablation Studies: The research includes a thorough evaluation of AUTOIF across different models, parameter sizes, and ablation settings, demonstrating the robustness and scalability of the approach. It also explores data scaling and efficiency, highlighting the method's potential to achieve strong performance with relatively fewer data samples.\n\nBalanced Capability Improvement: AUTOIF enhances instruction-following without compromising other model abilities, such as coding and general reasoning, making it a well-rounded solution.", "weaknesses": "Conceptual Rigidity: While the core contribution of this paper is insightful, the attempt to cast the instruction-following alignment into a code verification problem introduces some rigidity. This is so because in that respect, focus will be on \"verifiable instructions\" that can be amenable to mechanical verification by code executors. While the structure for this argument is robust in this text, it might indeed be narrow to encompass more complex or subjective tasks. The paper does not delve into the non-code-based verification methods like model-based scoring or human-in-the-loop strategies; the thrust of the discussion remains centered on the execution feedback mechanism​.\n\nGeneralization Concerns: Concentration on verifiable instructions has been made amply clear in the method section. AUTOIF framework uses code for instruction verification, which in itself inherently limits the generalization to open-ended or subjective instructions. While the paper does highlight improvements across benchmarks such as IFEval, it does not really investigate how the approach generalizes when the tasks are less structured or ambiguous. Though the second concern with respect to AUTOIF biasing the models to easily verifiable instructions might be well-taken, given the current evidence, it should be framed as a hypothesis.\n\nSomewhat worrying lack of methodological diversity: there is heavy focus on execution feedback. Other methods are not considered, like human feedback or model-based evaluation. Still, the paper does state that execution feedback and iterative optimization work fine, so there is a possibility that with more methods added, the instruction-following capabilities could be further improved.\n\nIt is very clear that, when explaining the strengths of the method, it seems over-dependent on automation and slighting the benefits that could be derived from purposive human intervention. The consideration of benefits from a more balanced approach in complex situations is thus little. By allowing human insights to complement this framework, therefore, it would adapt to inclusivity in the given context.\n\nNovelty: Automatic instruction verification might not be an entirely new idea. Works in this area are CodeLlama2, PLUM, LLaMA3, SelfCodeAlign, and DeepSeek-Coder-V2. In fact, research efforts such as code-based execution feedback and automated instruction alignment have been made. For example, CodeLlama2 and LLaMA3 utilized the base of code-based synthesis and validation frameworks in their architectures for better performance on top of instruction-following tasks, which underlines code execution as a natural source of feedback. Simultaneously, these related works suggest that perhaps AUTOIF, with the novelty of scalability and data synthesis, is not really a new fundamental direction in the space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWRXvQYD96", "reviewer": "Reviewer_7mrP", "summary": "This paper presents AutoIF. The main idea of AutoIF is to generate constraint-following instruction data by generating validation functions that correspond to the constraints within the natural language instruction and running them to validate any prospective responses.", "questions": "- I am curious about counter-examples where programs cannot model the constraints in the instruction; and how to recognize such cases and delegate them to human experts.\n- Section 3.2: How reliable is cross-validation of test cases and verification functions? Do you see cases where cross-consistent test and verifiers are wrong together?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper looks at an important problem of training LLMs to precisely follow instructions\n- The proposed approach is novel and seems general to most scenarios. It can be used to improve the quality of instruction-following datasets.\n- This paper comes with comprehensive end-to-end evaluations as well as controlled experiments, with good insights and reasonable explanations.", "weaknesses": "- Section 3.2: cross-verification can be weak as it essentially assumes test-function self-consistency as correctness. Not sure how reliable is self-consistency here as (i) bad tests can kill the right function and (ii) tests and functions can be broken all together result in corrupted data quality. \n\n----\n\nOther minor suggestions:\n\n- Code execution has also been widely adopted in the code domain to generate test-assured code; it is generally good to discuss more of such work (not only just CodeRL; but also RLTF, LEVER, Self-OSS-Instruct, ODEX, etc.) to connect the general research context.\n- L209: It might be more accurate to cite the use of rejection sampling with \"Training a helpful and harmless assistant with reinforcement learning from human feedback\" in addition to/instead of the Llama 2 paper.\n- L210: what does \"_{j=1}\" mean? Do you mean \"i=1\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HjUTaW9dTN", "reviewer": "Reviewer_BmDh", "summary": "The paper proposes AutoIF, a scalable method for generating high-quality instruction-following data for training LLMs without manual annotation. It leverages code execution to verify response quality. LLMs are used to generate instructions, produce the corresponding code, and create unit test samples for cross-validation. Execution-based rejection sampling produces data for SFT and RLHF. Applied to Qwen2 and Llama3, AutoIF enhances performance across three training algorithms (SFT, offline DPO, online DPO), achieving over 90% accuracy in IFEval’s loose instruction accuracy benchmark without sacrificing general, math, or coding abilities, highlighting its effectiveness in alignment and generalization.", "questions": "1. See the weaknesses above.\n2. Line 107: `we validate AUTOIF’s effectiveness in both \"Self Alignment\" and \"Weak to Strong\"`. typo?\n3. Will the self-alignment setting still work for smaller models like Llama3-8B?\n4. What are the main technical differences from StarCoder2-Instruct, which also used the base model itself to generate instructions, code, and test cases for validation?\n5. Although the IFEval and FollowBench results are impressive, these benchmarks are specifically designed to test natural language instruction following. Can improved IFEval performance be linked to more practical tasks? For instance, while benchmarks like GSM8K and HumanEval may not show additional gains, BigCodeBench / LiveCodeBench / LeetCode include complex instruction-following cases, which should ideally lead to improvement.\n6. Are there analyses on how reliable the generated code and tests are? Either or both of them may be just wrong.\n\nIf my concerns are properly addressed, I will consider raising the score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The technique is intuitive. IFEval was constructed using verifiable instructions to test an LLM's instruction-following ability, while this paper generates verifiable code to evaluate the instruction-response alignment.\n2. AutoIF proves to be effective in the self-alignment case, where the same model is used for instruction generation and verification.\n3. The evaluation covers a wide range of benchmarks and shows that AutoIF improves instruction following significantly while preserving the other general capabilities.\n4. AutoIF naturally generates pair-wise preference data and can be applied to DPO, offline or online.", "weaknesses": "1. In the self-alignment setting, the evaluation is only conducted on the two >70B models. It is unclear whether the same conclusion applies to weaker base models with less than 10B parameters.\n2. There are strong-to-weak and self-alignment settings, so a weak-to-strong setting should logically be included. There’s no proper justification for leaving it out.\n3. There's a missing baseline where responses are validated through the LLM itself without code. Without the baseline, it's unclear how valuable the verification function and test cases are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ep2UgeDbd7", "reviewer": "Reviewer_TNGe", "summary": "The paper proposes AUTOIF, an automated, scalable, and reliable method designed to enhance the instruction-following capabilities of LLMs, using automatic code verification.", "questions": "- Figure 2, improve consistency of terms in figure with 'acc' and 'Acc' being used alternatively, and the sequence of verification and func set being inverted at start.\n- An example case of how a sample develops in the AutoIF pipeline would help illustrate the working even better. \n- Line 215, 232, provide reasoning and example of failing cases of why accuracy is lower. similarly for Section 4.1 talks about main results but is limited to quantitative numbers and not examples of illustrating failed cases.\n- Also, most seed instruction showcase restriction over string manipulation domain only. Ensuring type of word to be present, limiting length of output and so on. There is no showcasing of diverse, novel instructions. \n- line 319, what is the value of K, for which the results have been sampled from?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "There is a detailed and structured pipeline of AutoIF which showcases enhanced results over multiple training strategy.", "weaknesses": "However, examples on failed cases and illustration of these categories of consistent failure would help understand scope of next steps and areas of improvement that has been brought about by the method. Along with the limited domain of seed instructions in terms of string manipulation only, which could be diversified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IsRw3i9Twv", "reviewer": "Reviewer_5kXL", "summary": "The authors propose AUTOIF, a training framework to improve instruction following capability via execution feedback. The authors demonstrate the efficacy of AUTOIF on four LLMs with benchmarks across various domains.", "questions": "Apart from the mentioned weaknesses, there are some additional questions:\n- How do the accuracy rate thresholds in the verification steps affect the model performance?\n- How does RAILF+Online DPO (e.g., OAIF [1]) compare to AUTOIF?\n\n[1] Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., ... & Blondel, M. (2024). Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "There are a few valuable contributions:\n- The proposed AUTOIF shows that training with program-based validation can improve some instruction-following capabilities.\n- The experiments are pretty extensive, where the authors benchmark the AUTOIF on four models with different sizes.\n- The ablation studies are quite detailed, showing the optimal setups for AUTOIF.", "weaknesses": "There are several weaknesses to be addressed:\n- The idea of using programs to validate instruction-following is not novel, as it has been used in IFEval. Moreover, training with execution feedback is also not new; it has been widely used by training code LLMs [1-3].\n- Based on the proposed workflow, AUTOIF is hard to scale to complex instructions, where the queries are totally quantifiable. AUTOIF cannot easily work for semantically rich instructions, which cannot be evaluated via execution. In addition, real-world scenarios that can be validated via programs may require various dependencies and APIs. AUTOIF is not scalable under these practical cases. \n- The instruction-following evaluations on IFBench and FollowBench lack practicability. These two benchmarks only contain quantifiable instructions, like word counts. The authors should consider more practical scenarios like the one in BigCodeBench [4] for the instruction-following evaluation.\n- The current evaluation is English-only. The authors should evaluate their models on multilingual setups to demonstrate their generalisability.\n\n[1] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35, 21314-21328.\n\n[2] Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., ... & Yue, X. (2024). Opencodeinterpreter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402.14658.\n\n[3] Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J. G., & Chen, W. CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations.\n\n[4] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., ... & Von Werra, L. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)