CFBench: A Comprehensive Constraints-Following Benchmark for LLMs
Authors: Tao Zhang, ChengLIn Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin CUI, Wentao Zhang, Zenan Zhou
Accepted at: acl25

CFBench: A Comprehensive Constraints-Following Benchmark for LLMs

Tao Zhang1*♣ Chenglin Zhu2* Yanjun Shen1* Wenjing Luo1 Yan Zhang1 Hao Liang2,3 Tao Zhang1◊ Fan Yang1 Mingan Lin1 Yujing Qiao1 Weipeng Chen1 Bin Cui2 Wentao Zhang2,3† Zenan Zhou1‡
1Baichuan Inc. 2Peking University 3Zhongguancun Academy {zhangtao2, shenyanjun, zhouzenan}@baichuan-inc.com zhuchenglin@stu.pku.edu.cn, wentao.zhang@pku.edu.cn

Abstract

The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user’s perspective. To bridge this gap, we propose CF-Bench, a large-scale Chinese Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CF-Bench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code will be made available.

1 Introduction

Large Language Models (LLMs) have become the cornerstone of numerous cutting-edge research tasks and are widely utilized in real-world scenarios (Brown et al., 2020; Chowdhery et al., 2023; Achiam et al., 2023; Touvron et al., 2023). In real-world scenarios, human instructions are inherently complex and accompanied by explicit constraints, requiring models to both understand intricate requirements and strictly comply with these constraints (Yang et al., 2023; Zhong et al., 2021; Mishra et al., 2022; Wei et al., 2021; Sanh et al., 2022). The proficiency of LLMs in comprehending requirements and adhering to natural language constraints is essential, as it ensures tasks are executed precisely and resolved perfectly according to user instructions.

The prevailing method for evaluating a model’s instruction-following ability involves using quantitative programs, human evaluators, or advanced LLMs to assess response quality across single constraints, complex problems, and finite constraints (Zhou et al., 2023a; Wang et al., 2023; Li et al., 2023; Zheng et al., 2024; Xu et al., 2023). Laskar et al. (2024) underscores the importance of evaluating data quality, highlighting the necessity for real and extensive data distribution, along with its applicability to real-world scenarios. Sun et al. (2024b)

<table>
  <tr>
    <th>Instruction</th>
    <th>Response</th>
    <th>Check List</th>
    <th>Constraints</th>
    <th>Priority</th>
    <th>Satis.</th>
  </tr>
  <tr>
    <td>Overview of the Trump Shooting Incident, include the cause, process, impact, and summary. Organize the content using numbered points and bold text. The cause should be in JSON format with event keywords: time, location, and nature. Summarize the process in three sentences. Discuss the impact from both international and domestic perspectives keeping the total content under 500 words. Finally, summarize this event in a seven-word quatrains.</td>
    <td>Overview of the Trump Shooting Incident. 1.Cause: n="time":"July 13, 2024", "location":"Butler, Pennsylvania, USA", "nature":"assassination attempt" 2.Process: a-Trump was giving a speech at the rally when the shooting occurred. b-Secret Service quickly responded and the gunman was taken away. 3.Impact: a-International: The incident shocked the world, raising concerns about US political violence. b-Domestic: US political tensions and polarization increased. Trump's support may rise. 4.Shooter Information: Details are not yet released. The FBI is investigating. 5.Seven-word quatrains: a-In former President Trump was shot at a rally, the gunman was killed by the Secret Service, shocking the world.</td>
    <td>
      <ul>
        <li>Content on Trump Shooting Incident</li>
        <li>Includes cause, process, impact</li>
        <li>Use numbered points and bold text</li>
        <li>4 Cause in JSON format: time, location, nature</li>
        <li>Process: international and domestic</li>
        <li>Total content under 500 words</li>
        <li>Seven-word quatrains</li>
      </ul>
    </td>
    <td>
      <table>
        <tr>
          <th>Semantic Content</th>
          <th>Element Content</th>
          <th>Bespoke Format</th>
          <th>Common Format</th>
          <th>Sentence Count</th>
          <th>Element Content</th>
          <th>World Count</th>
          <th>Pragmatic</th>
        </tr>
        <tr>
          <td>Primary</td>
          <td>Primary</td>
          <td>Secondary</td>
          <td>Secondary</td>
          <td>Secondary</td>
          <td>Secondary</td>
          <td>Secondary</td>
          <td>Secondary</td>
        </tr>
      </table>
    </td>
    <td>
      <table>
        <tr>
          <th>Primary</th>
          <th>Secondary</th>
          <th>Secondary</th>
          <th>Secondary</th>
          <th>Secondary</th>
          <th>Secondary</th>
          <th>Secondary</th>
          <th>Secondary</th>
        </tr>
        <tr>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
          <td>✓</td>
        </tr>
      </table>
    </td>
  </tr>
</table>

Figure 1: Sample data from CFBench. A checklist, constraint type, requirement priority, and satisfaction constitute our evaluation criteria.
also stresses that realistic evaluation metrics reflect model capabilities and guide iteration. Constraints-following evaluation faces analogous challenges, particularly within complex real-world scenarios, where data sources and contexts are diverse, and where evaluation is both subjective and arduous. Fig. 1, which addresses the aforementioned challenges, presents a sample from CFBench illustrating the Trump assassination event with different colors representing various constraint types. The instruction include multiple constraints, and the evaluation method uses a checklist to break down complex requirements into independent checkpoints, annotating constraint types and priorities. LLMs are then used to assess each checkpoint. For the English-Chinese comparison example, see Appendix Fig. 11. Consequently, we introduce two more profound challenges in constraints-following assessment.

Q1: How to construct high-quality evaluation data? Many studies focus on evaluating single constraint (Chen et al., 2022; Tang et al., 2023), lacking comprehensive analysis across diverse constraints. He et al. (2024b) examines LLM performance on complex real-world instructions but neglect constraint diversity and scenario coverage. Jiang et al. (2023) incrementally incorporate fine-grained constraints to craft multi-level instructions. However, with only 75 instances of mixed type, which risks variability due to limited data, and equating difficulty with constraint quantity oversimplifies the task. Recent work focuses on evaluating constraints combinability (Wen et al., 2024). To ensure data quality, we systematically categorize constraints by mining real-world online data and using classification, synthesis, and expert design, covering 10 primary categories and over 25 subcategories. We also cross-match these constraints with various domains and scenarios, ensuring balanced representation and expert-validated reasonableness.

Q2: How to evaluate accurately and meticulously? Evaluating LLMs’ adherence to constraints is challenging and typically involves manual, automated, and programmatic assessments using various metrics. Representative work computes outcomes for verifiable instructions using code (Zhou et al., 2023a; He et al., 2024b). Jiang et al. (2023) uses scripts and constraint-evolution paths to handle diverse challenging instructions, introducing three metrics tailored to the data’s characteristics. The DRFR method decomposes complex constraints into binary judgments, with GPT evaluating each criterion (Qin et al., 2024). Indeed, previous work has ensured the feasibility and objectivity of evaluations through various methods, but they have overlooked assessments from the user’s multiple perspectives. We deconstruct complex instructions from the user’s perspective into multiple sub-needs, categorizing them by priority and constraint type, with LLMs evaluating each checkpoint. Furthermore, a multi-dimensional evaluation criteria is proposed using three metrics from the perspectives of constraints, instructions, and requirements priority.

We introduce CFBench, a comprehensive Chinese benchmark designed to thoroughly evaluate the constraint comprehension and following capabilities of LLMs. CFBench comprises 10 primary categories and over 25 secondary subcategories organized through taxonomic and statistical methodologies. CFBench features 1,000 meticulously curated samples spanning more than 200 real-life scenarios across 20 domains and over 50 NLP tasks, enhancing the breadth and generality of the evaluation data. Additionally, we have seamlessly integrated original instructions and constraint types within each sample, paying particular attention to nuanced combinations, ensuring each constraint is credibly and coherently embedded. Our advanced evaluation methodology incorporates multi-dimensional assessment criteria, which prioritizing requirements to align LLM outputs with user needs, enhance interpretability, and facilitate iterative development. Finally, extensive experiments and exploratory discussions provide strong support for evaluation and optimization.

Overall, our contributions are mainly four-fold:

• To the best of our knowledge, we are the pioneers in systematically defining an instruction constraint framework utilizing both taxonomic and statistical methodologies.

• We introduce CFBench, a meticulously annotated, large-scale, high-quality Chinese benchmark that encompasses a broad spectrum of real-world scenarios and NLP tasks.

• We propose a multi-dimensional evaluation framework to comprehensively assess model capabilities while prioritizing user-centric needs.

• We exhaustively evaluated prominent LLMs,
uncovering significant deficiencies in constraints following and exploring performance factors and optimization strategies.

2 Related Work

2.1 Instruction Following
Fine-tuning LLMs with annotated instructional data enhances their ability to follow general language instructions (Weller et al., 2020; Sanh et al., 2022). Studies show that more complex or constrained instructions further improve this ability. For instance, six methods to create intricate instructions from a small set of handwritten seed data are proposed (Xu et al., 2023), while Mukherjee et al. (2023) elevate training data complexity by having GPT-4 (Achiam et al., 2023) generate reasoning steps for simple instructions. The latest work (Sun et al., 2024a; He et al., 2024a; Dong et al., 2024) suggests that increasing the number and variety of constraints can enhance the complexity of instructions, thereby further improving the model’s ability to follow constraint-based instructions.

2.2 Evaluation of Constraints Following
Constraints such as word count, position, topics, and content have garnered significant attention in the field of Controlled Text Generation (Yao et al., 2023; Zhou et al., 2023b). Zhou et al. (2023a) centers on assessing 25 verifiable instructions. Numerous studies have explored the adherence of LLMs to format constraints, such as complex tabular data (Tang et al., 2023) and customized scenario formats (Xia et al., 2024). Qin et al. (2024) decomposing a single instruction into multiple constraints. He et al. (2024b) gathered constraints from real-world scenarios and developed a sophisticated benchmark using detailed task descriptions and inputs. Jiang et al. (2023) progressively integrates fine-grained constraints to develop multi-level instructions, thereby enhancing complexity across six distinct types. Concurrent work (Wen et al., 2024), constructs a novel benchmark by synthesizing and refining data from the aforementioned benchmarks, with an emphasis on the combinatorial types of constraints. However, previous studies suffered from fragmented constraints, limited scenarios, and misaligned evaluation methods with user perspectives.

3 CFBench
As depicted in Fig. 2, the CFBench construction pipeline includes several key components. First, we collect and systematize constraint expressions from real-world scenarios and various NLP tasks. Using this system, we create high-quality evaluation data by combining instructions from these scenarios with advanced LLMs and manual curation. We then introduce innovative multi-perspective evaluation method. Additionally, we conduct a thorough statistical analysis and validate the quality from various angles to highlight reliability and applicability.

3.1 Constraints System
3.1.1 Constraints Collection
We amass a diverse corpus of instructions from real-world scenarios and various NLP tasks (Xia et al., 2024; Li et al., 2024) to ensure a comprehen-

![The construction pipeline and evaluation sample of CFBench. Initially, it entails the construction of the constraint system, followed by the assembly of the dataset, and culminating in the proposal of a multi-perspective user view evaluation.](page_186_120_1077_393.png)

Figure 2: The construction pipeline and evaluation sample of CFBench. Initially, it entails the construction of the constraint system, followed by the assembly of the dataset, and culminating in the proposal of a multi-perspective user view evaluation.
<table>
  <tr>
    <th rowspan="2">Split Set</th>
    <th colspan="5">Basic Info</th>
    <th colspan="10">Constraints Count</th>
  </tr>
  <tr>
    <th>Num.</th>
    <th>Len.</th>
    <th>Prim.</th>
    <th>Cons.</th>
    <th>Type.</th>
    <th>C1</th>
    <th>C2</th>
    <th>C3</th>
    <th>C4</th>
    <th>C5</th>
    <th>C6</th>
    <th>C7</th>
    <th>C8</th>
    <th>C9</th>
    <th>C10</th>
  </tr>
  <tr>
    <td>Easy Set</td>
    <td>500</td>
    <td>413</td>
    <td>1.69</td>
    <td>3.59</td>
    <td>2.83</td>
    <td>613</td>
    <td>214</td>
    <td>180</td>
    <td>170</td>
    <td>134</td>
    <td>92</td>
    <td>82</td>
    <td>95</td>
    <td>90</td>
    <td>79</td>
  </tr>
  <tr>
    <td>Hard Set</td>
    <td>500</td>
    <td>605</td>
    <td>1.98</td>
    <td>4.89</td>
    <td>3.58</td>
    <td>772</td>
    <td>345</td>
    <td>233</td>
    <td>241</td>
    <td>168</td>
    <td>122</td>
    <td>115</td>
    <td>145</td>
    <td>137</td>
    <td>81</td>
  </tr>
  <tr>
    <td>Full Set</td>
    <td>1000</td>
    <td>509</td>
    <td>1.84</td>
    <td>4.24</td>
    <td>3.20</td>
    <td>1385</td>
    <td>559</td>
    <td>413</td>
    <td>411</td>
    <td>302</td>
    <td>214</td>
    <td>197</td>
    <td>240</td>
    <td>227</td>
    <td>160</td>
  </tr>
</table>

Table 1: CF Bench Statistic. The abbreviations of 'Num.', 'Len.', 'Prim.', 'Cons.', 'Type.' denote the sample number, average instruction length, average primary requirements number, average constraint number, average constraint type number, respectively. The designations 'C1'-'C10' denote the Primary Constraint types of content, numerical, style, format, linguistic, situation, example, inverse, contradictory, and rule constraint, respectively.

sive system. Initially, we aggregate several million instructions from online logs and NLP tasks, refining these through length filtering and clustering to distill 30,000 high-quality instructions. Utilizing advanced LLM techniques, we extract and expand atomic constraints through evolutionary methods. Using LLMs, we carefully select meaningful atomic constraints, resulting in over 5000 unique constraints. Domain experts first filter out unreasonable or meaningless atomic constraints and then synthesize these into a structured framework with 10 primary categories and 25 subcategories, guided by principles of statistics, taxonomy, and linguistics.

3.1.2 Constraints System

Content constraints control the scope and depth of output content by specifying certain conditions (Zhang et al., 2023), and can be divided into lexical constraints, element constraints, and semantic constraints based on their granularity. Numerical constraints, which ensure that output content meets length and quantity requirements (Yao et al., 2023), can be classified into word-level, sentence-level, paragraph-level, document-level based on the objects involved in the planning. Stylistic constraints impart a unique flavor and color to the output, revealing the author’s traits and chosen social objectives (Tsai et al., 2021), can be subdivided into tonal, formal, audience, and authorial style constraints based on the perspective of application. Format constraints (Tang et al., 2023) standardize expression to guide the generation of complex content and can be categorized into fundamental, bespoke, and specialized scenario constraints based on their usage scenarios. Linguistic constraints (Zhou et al., 2023b) adapt to various scenarios by controlling internal features and logic, grouped into Pragmatic, Syntactic, Morphological, Phonological, and other constraints. Situation constraints (Liu et al., 2023) guide response appropriateness through background or situational parameters, can be classified into role-based, task-specific, and complex contextual constraints. Example constraints regulate new responses by leveraging the intrinsic patterns established by a limited set of samples, with an emphasis on assessing the model’s proficiency in contextual constraint learning. Inverse constraints narrow the response space through the mechanism of indirect exclusion. Contradictory constraints denote conditions that are mutually exclusive, rendering it impossible for the response to fulfill all requirements concurrently, which are prevalent in online logs and are often easily overlooked. Rule constraints define logic flows or actions and meticulously crafted to standardize the road of responses. Details are in Appendix Tab. 9.

3.2 Dataset Construction

To guarantee data quality in terms of authority and thorough coverage, we utilize a collaborative iterative methodology that synergizes expertise with the capabilities of LLMs.

3.2.1 Data Source and Selection

Real-world scenarios and NLP tasks form the foundation for CF Bench’s initial instructions. By harnessing advanced LLMs, we rigorously assess each instruction for constraint types and quantities within a predefined system, filtering out those with unreasonable or ineffective constraints. Subsequently, we balance the scenarios and constraint types, resulting in a refined set of 2,000 instructions covering all scenarios and NLP tasks. Prompts and checklist generation are in Appendix.

3.2.2 Iterative Refinement

Professional annotators carefully review and refine the data, ensuring the rationality of constraints and gold answer. If modifications are needed, instructions are revised, and LLMs generate responses
with refined evaluation criteria, repeating this process until satisfactory results are achieved. Ultimately, comprehensive support is formulated for each sample, detailing high-quality instructions, the ideal answer, specific assessment criteria, constraint types, and priority levels.

3.3 Dataset Statistics

3.3.1 Overall Statistics
Table 1 provides a statistical overview of CFBench, highlighting substantial differences between the two sets. The Hard Set has more detailed instructions, a greater variety, and a higher number of constraints, indicating higher complexity compared to the Easy Set. The table also shows the diversity and balanced distribution of primary constraint types within CFBench, outperforming other benchmarks. See the Appendix for division details.

3.3.2 Tasks and Domains Distribution
CFBench covers 20 major real-life domains and includes over 200 common scenarios and 50+ NLP tasks. Fig. 3(a) illustrates the classification of NLP tasks, including four major types: classification, generation, sequence labeling, and sentence relation, along with their corresponding specific tasks. Fig. 3(b) shows the real-life scenario-specific domain distribution, where Employment is the most prevalent category, and the other domains are relatively balanced. Our objective is to balance the real distribution with an equitable distribution. Overall, Fig. 3 illustrates that CFBench has evolved into a comprehensive and well-balanced benchmark.

3.3.3 Comparison with Other Benchmarks
As shown in Tab. 3, we thoroughly compare our benchmark with various relevant ones. In terms of size, our benchmark contains approximately twice the number of samples as others. Follow-Bench (Jiang et al., 2023) increases difficulty by adding the number of constraints, but focuses on the incremental increase of a single constraint type. ComplexBench (Wen et al., 2024) places more emphasis on the combination relationships between different constraint types, but only designs four types. IFEval (Zhou et al., 2023a) focuses on constraints that can be verified, but lacks generalization. Compared to others, CFBench provides comprehensive scenario coverage, diverse systematic constraints, numerous high-quality samples, and multidimensional evaluation. For details, see Appendix Tab. 11, including Case and Features.

3.4 Evaluation Protocol

3.4.1 Evaluation Criteria
We breaking down instructions into multiple simple, independent checkpoints to ensure evaluation accuracy, inspiration was drawn from DRFR (Qin et al., 2024). Unlike DRFR, our method emphasizes defining ideal response characteristics and critical evaluation points. The previous sections detailed the checklist generation process, a key part of our evaluation criteria. Furthermore, we employ GPT-4o, as the evaluation model. By repeatedly feeding it the instruction, test model response, and checklist with a carefully tuned prompt, we ensure that the judged response fully meets the judgement format check. This iterative process aims to maximize confidence in our evaluation. The specific evaluation prompt is in the Appendix.

3.4.2 Evaluation Metrics
Aligned with different perspectives, we define the Constraint Satisfaction Rate (CSR), Instruction Satisfaction Rate (ISR) as follows:

\[
\text{CSR} = \frac{1}{m} \sum_{i=1}^{m} \left( \frac{1}{n_i} \sum_{j=1}^{n_i} s_i^j \right)
\]
(1)

\[
\text{ISR} = \frac{1}{m} \sum_{i=1}^{m} s_i
\]
(2)

where \( s_i^j = 1 \) if the \( j \)-th constraint of \( i \)-th instruction is satisfied and \( s_i^j = 0 \) otherwise. \( s_i = 1 \) indicates that all constraints in the \( i \)-th instruction are satisfied and \( s_i = 0 \) otherwise. The requirements Priority Satisfaction Rate (PSR) is defined as follows:

\[
\text{PSR} = \frac{1}{m} \sum_{i=1}^{m} (PSR_i)
\]
(3)

Let the average score for secondary requirements be \( A \). When all primary requirements are met, \( score = 0.5 + 0.5 \times A \). If \( score > 0.8 \), then
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Easy Set</th>
    <th colspan="3">Hard Set</th>
    <th colspan="3">Full Set</th>
  </tr>
  <tr>
    <th>CSR</th><th>ISR</th><th>PSR</th>
    <th>CSR</th><th>ISR</th><th>PSR</th>
    <th>CSR</th><th>ISR</th><th>PSR</th>
  </tr>
  <tr>
    <td>o1-preview<sup>†</sup></td><td>0.926</td><td>0.806</td><td>0.844</td><td>0.814</td><td>0.462</td><td>0.592</td><td>0.870</td><td>0.634</td><td>0.718</td>
  </tr>
  <tr>
    <td>DeepSeek-V3<sup>†</sup></td><td>0.948</td><td>0.836</td><td>0.864</td><td>0.831</td><td>0.460</td><td>0.616</td><td>0.890</td><td>0.648</td><td>0.740</td>
  </tr>
  <tr>
    <td>DeepSeek-R1<sup>†</sup></td><td>0.960</td><td>0.874</td><td>0.894</td><td>0.856</td><td>0.524</td><td>0.672</td><td>0.908</td><td>0.699</td><td>0.783</td>
  </tr>
  <tr>
    <td>GPT-4o<sup>†</sup></td><td>0.956</td><td>0.868</td><td>0.888</td><td>0.816</td><td>0.438</td><td>0.582</td><td>0.886</td><td>0.653</td><td>0.735</td>
  </tr>
  <tr>
    <td>GPT-4-Turbo-20240409<sup>†</sup></td><td>0.924</td><td>0.792</td><td>0.826</td><td>0.783</td><td>0.370</td><td>0.518</td><td>0.853</td><td>0.581</td><td>0.672</td>
  </tr>
  <tr>
    <td>GPT-4-0125-Preview<sup>†</sup></td><td>0.923</td><td>0.790</td><td>0.826</td><td>0.763</td><td>0.310</td><td>0.468</td><td>0.843</td><td>0.550</td><td>0.647</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet<sup>†</sup></td><td>0.943</td><td>0.844</td><td>0.882</td><td>0.799</td><td>0.408</td><td>0.564</td><td>0.871</td><td>0.626</td><td>0.723</td>
  </tr>
  <tr>
    <td>GLM-4-0520<sup>†</sup></td><td>0.939</td><td>0.820</td><td>0.852</td><td>0.785</td><td>0.372</td><td>0.536</td><td>0.862</td><td>0.596</td><td>0.694</td>
  </tr>
  <tr>
    <td>Yi-Large<sup>†</sup></td><td>0.900</td><td>0.730</td><td>0.786</td><td>0.744</td><td>0.292</td><td>0.460</td><td>0.822</td><td>0.511</td><td>0.623</td>
  </tr>
  <tr>
    <td>MoonShot-V1-8k<sup>†</sup></td><td>0.919</td><td>0.764</td><td>0.812</td><td>0.758</td><td>0.308</td><td>0.464</td><td>0.838</td><td>0.536</td><td>0.638</td>
  </tr>
  <tr>
    <td>Llama-3-8B-Instruct*</td><td>0.656</td><td>0.300</td><td>0.356</td><td>0.562</td><td>0.122</td><td>0.238</td><td>0.609</td><td>0.211</td><td>0.297</td>
  </tr>
  <tr>
    <td>Llama-3-70B-Instruct*</td><td>0.750</td><td>0.422</td><td>0.498</td><td>0.642</td><td>0.178</td><td>0.330</td><td>0.696</td><td>0.300</td><td>0.414</td>
  </tr>
  <tr>
    <td>Vicuna-33B-V13*</td><td>0.621</td><td>0.270</td><td>0.352</td><td>0.527</td><td>0.110</td><td>0.196</td><td>0.574</td><td>0.190</td><td>0.274</td>
  </tr>
  <tr>
    <td>DeepSeek-V2-Lite-Chat</td><td>0.733</td><td>0.382</td><td>0.448</td><td>0.597</td><td>0.148</td><td>0.262</td><td>0.665</td><td>0.265</td><td>0.355</td>
  </tr>
  <tr>
    <td>Owen1.5-110B-Chat</td><td>0.905</td><td>0.724</td><td>0.792</td><td>0.730</td><td>0.276</td><td>0.438</td><td>0.818</td><td>0.500</td><td>0.615</td>
  </tr>
  <tr>
    <td>Owen2-72B-Instruct</td><td>0.944</td><td>0.836</td><td>0.880</td><td>0.791</td><td>0.342</td><td>0.530</td><td>0.867</td><td>0.589</td><td>0.705</td>
  </tr>
  <tr>
    <td>BaiChuan2-13B-Chat</td><td>0.669</td><td>0.348</td><td>0.418</td><td>0.547</td><td>0.134</td><td>0.226</td><td>0.608</td><td>0.241</td><td>0.322</td>
  </tr>
</table>

Table 2: The evaluation results of LLMs on CFBench and its splits. Notably, * stands for the model supporting mainstream languages excluding Chinese, and † represents calling through the API. The bold, underlined, and tilde denote the first, second, and third rankings, respectively.

<table>
  <tr>
    <th rowspan="2">Benchmarks</th>
    <th colspan="2">Data Quality</th>
    <th colspan="3">Evaluation</th>
  </tr>
  <tr>
    <th>Num.</th><th>Type.</th>
    <th>Syst.</th><th>Prio.</th><th>Meth.</th>
  </tr>
  <tr>
    <td>IFEval</td><td>541</td><td>4*</td><td>X</td><td>X</td><td>⊕</td>
  </tr>
  <tr>
    <td>CELLO</td><td>523</td><td>4</td><td>X</td><td>X</td><td>⊕</td>
  </tr>
  <tr>
    <td>FollowBench</td><td>820</td><td>5</td><td>X</td><td>X</td><td>⊕⊕</td>
  </tr>
  <tr>
    <td>InFoBench</td><td>500</td><td>5</td><td>X</td><td>X</td><td>⊕⊕</td>
  </tr>
  <tr>
    <td>FoFoBench</td><td>494</td><td>1</td><td>X</td><td>X</td><td>⊕</td>
  </tr>
  <tr>
    <td>ComplexBench</td><td>1150</td><td>4</td><td>✓</td><td>✓</td><td>⊕⊕</td>
  </tr>
  <tr>
    <td>CFBench</td><td>1000</td><td>10-25</td><td>✓</td><td>✓</td><td>⊕⊕</td>
  </tr>
</table>

Table 3: Detailed Comparison of Relevant Benchmarks. * represents our constraint system. 'Num.', 'Type.', 'Syst.', 'Prio.', and 'Meth.' denote the number of samples, primary constraint types, presence of a constraint system, requirement prioritization, and evaluation method.

\( PSR_t = 1 \); otherwise, \( PSR_t = 0 \), especially when any primary requirement is not met. The threshold of 0.8 is based on user feedback, reflecting tolerance for LLMs adhering to constraints. Overall, CSR, ISR, and PSR reflect different levels of user perception from multiple perspectives, including constraints, instructions, and requirement priorities.

3.5 Data Quality

3.5.1 Quality Evolution

To enhance the quality of CFBench, we invested considerable effort and financial resources. First, in the instruction generation phase, we utilized multiple advanced LLMs, such as GPT-4 and Claude, to generate diverse instructions and responses for annotator candidates. Second, we implemented a stringent manual annotation process, including annotator training, cross-validation, batch validation, expert team involvement, and iterative refinement of instruction constraints and response quality. We also ensured the objectivity, evaluability, and prioritization of checkpoints. Additionally, we balanced the data for constraint types, scenarios, and NLP task distribution. Detailed information can be found in the Appendix.

3.5.2 Quality Evaluation

To investigate CFBench’s quality, we randomly selected 100 samples for assessment. Three professional data inspectors evaluated them, resulting in high-quality rates of 94% for instructions, 94% for gold answers, and 93% for checklists (see Appendix Tab. 6). Additionally, three experts rated Owen2-7B-Instruct outputs on a 0-1 scale. The kappa coefficient between GPT-4o PSR and expert evaluations was 0.77, highlighting the effectiveness of the PSR evaluation method and metrics, even for smaller models. Details are in Appendix Tab. 7.

4 Experiment

4.1 Evaluation Settings

We evaluated 50+ top-performing models from previous benchmarks (Hendrycks et al., 2020;
Figure 4: Different mainstream models’ results under primary and secondary constraint categories.

Figure 5: Different mainstream models’ PSR results in real-world domains and NLP task types.

Cobbe et al., 2021), considering factors like model size, Chinese language support, access via API or weights, and fine-tuning with instruction data. During inference, we set the maximum generation length to 2048 and used default values for other parameters. For evaluation, we used GPT-4o as the judge model with a temperature of 0 for deterministic outputs.

4.2 Overall Result

Tab. 2 presents CFBench evaluation results for leading models. DeepSeek-R1 (DeepSeek-AI et al., 2024) leads overall, followed by GPT-4o and DeepSeek-V3 (DeepSeek-AI et al., 2025) in third. Claude-3.5-Sonnet (Anthropic, 2024) and Qwen2-72B-Instruct (Yang et al., 2024) performed well, though DeepSeek-V3 and Qwen2-72B-Instruct showed slight drops in the Full Set. The highest PSR in the Hard Set was 0.582, indicating room for improvement.

While CSR favors weaker models, ISR and PSR highlight differences in stronger models. API-accessed models like GPT-4o outperformed most open-source models, though DeepSeek-V3 and Qwen2-72B-Instruct performed well among open-source models.

<table>
  <tr>
    <th>Model</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>CFBench</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>88.7</td>
    <td>90.5</td>
    <td>0.698</td>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>88.7</td>
    <td>96.4</td>
    <td>0.691</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Instruct</td>
    <td>82.3</td>
    <td>91.1</td>
    <td>0.672</td>
  </tr>
  <tr>
    <td>DeepSeek-V2</td>
    <td>78.5</td>
    <td>79.2</td>
    <td>0.665</td>
  </tr>
  <tr>
    <td>Qwen1.5-110B-Chat</td>
    <td>80.4</td>
    <td>-</td>
    <td>0.584</td>
  </tr>
  <tr>
    <td>Qwen1.5-72B-Chat</td>
    <td>77.5</td>
    <td>79.5</td>
    <td>0.577</td>
  </tr>
</table>

Table 4: Performance Comparison on Benchmarks

4.3 Constraints-categorized Performance

To assess performance across different constraint types, we calculated satisfaction scores for the top 8 LLMs (see Fig. 4). Many models struggled with contradictory constraints, highlighting their limitations. GPT-4o excelled across various constraints, while other models alternated in leading different types. For secondary constraints, all models performed poorly in lexical, word, and sentence count constraints but did better in document count and audience style constraints. No single model consistently led across most constraint types. In summary, even the most advanced LLMs have significant room for improvement, with each model showing specific weaknesses, providing valuable insights for future iterations.

4.4 Domain and Task-categorized Performance

As depicted in Fig. 5, we evaluate performance across 21 domains and 4 major NLP task types, each with 500 examples from the two main sources of CFBench. For domain performance, employment and psychology require significant attention, while technology and recruitment are strengths for most models. For NLP tasks, GPT-4o excels in sentence relationship tasks, while Qwen2-72B-Instruct is strong in sequence labeling, likely due to its optimization for Chinese. In general, models exhibit different rankings across domains and tasks, indicating no clear absolute leader. Comprehensive improvements are needed for better constraint follow across multiple domains and tasks.

5 Discussions

5.1 Comparisons between Capabilities

Tab. 4 presents a comprehensive comparison of CFBench’s PSR with two prominent LLM evaluation benchmarks. MMLU (Hendrycks et al., 2020) focuses on knowledge proficiency, while GSM8K (Cobbe et al., 2021) emphasizes mathematical ability. GPT-4o ranks first on CFBench
but significantly lags behind, ranking third on GSM8K. Qwen2-72B-Instruct performs worse than DeepSeek-V2 on CFBench but outperforms it on MMLU. Notably, the rankings of LLMs on CF-Bench do not entirely correspond with those on the other two benchmarks, indicating that CFBench provides a novel perspective for LLMs evaluation.

5.2 Factors influencing constraints-following
Previously, we identified a significant gap in LLM constraints following performance, prompting us to further explore the influencing factors. We analyzed the impact of prompt length, number of constraints, constraint types, and primary requirements on evaluation results across five top-performing models and their average values. As shown in Fig. 6, all four factors are positively correlated with the ISR metric, with the number of constraints having the most significant effect. For PSR, the number of constraints and constraint types do not show a completely positive correlation, while the number of primary requirements has a greater influence. Users are more affected by unmet constraints when there are fewer, but become more tolerant of unmet non-primary constraints when there are many.

5.3 How to improve constraint-following ability?
In Appendix Tab. 8, we investigated methods to potentially enhance constraint following. Firstly, Supervised Fine-Tuning (SFT) significantly improves performance, with nearly all models that undergo instruction fine-tuning exhibiting substantial improvements in effectiveness, as demonstrated by the Qwen series. Secondly, model size is also an important factor, as evidenced by Qwen2-72B-Instruct showing a 40% relative PSR improvement over Qwen2-7B-Instruct. Additionally, replicating Conifer’s models (Sun et al., 2024a) reveals that fine-tuning with complex constraint instructions further enhances performance, and recent work has also been directed towards this approach (He et al., 2024a). Further exploration is intended to be pursued in future work.

![Factors Influencing Constraints-Following Performance](page_120_120_1208_480.png)

Figure 6: Factors Influencing Constraints-Following Performance

6 Conclusion

This study examines the constraints-following capabilities of LLMs. CFBench, was introduced with 1000 manually annotated samples covering more than 200 real-world scenarios and over 50 NLP tasks, encompassing a wide range of systematically defined constraint types. Each sample in CFBench includes detailed evaluation criteria, providing metrics that accurately reflect model performance and real user needs across multiple dimensions. Extensive experiments on CFBench revealed significant limitations and challenges that advanced LLMs face in following constraint instructions. Key factors and potential strategies to improve constraint following were also analyzed, and numerous insightful findings can provide valuable guidance for the optimization of LLMs’ performance. In conclusion, CFBench offers a novel perspective for evaluating LLM capabilities, providing new directions for performance assessment and improvement.
7 Limitations

7.1 Experimental Setup
This study primarily focuses on models with strong Chinese language capabilities, lacking a comprehensive survey of a broader range of English models. Additionally, while we conducted preliminary analyses on the differences in instruction-following abilities between Chinese and English, a more in-depth comparative study is absent.

7.2 Limited Exploration of Reasoning Models
Currently, deep reasoning models like R1 continue to achieve commendable results. However, there is a lack of in-depth research into these models, particularly concerning the factors that enhance their instruction-following abilities.

7.3 Evaluation Model Bias
The evaluation of models predominantly relies on GPT-4o as the judge model. Future research could explore the impact of different evaluation models on assessing the performance of other models.

8 Ethics Statement
This research adheres to the ethical guidelines set forth by the Association for Computational Linguistics (ACL). We have ensured that all data collection and experimental designs comply with privacy protection and informed consent principles, fully respecting and safeguarding the rights of all participants. Furthermore, we have evaluated the potential societal impacts of our research findings, ensuring that their application does not result in adverse effects on society.

9 Acknowledgements
This work is supported by the National Key R&D Program of China (2024YFA1014003), National Natural Science Foundation of China (92470121, 62402016), and High-performance Computing Platform of Peking University.

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. 2022. Controllable text generation with language constraints. arXiv preprint arXiv:2212.10466.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rechirou Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhizhong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushu Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuteng Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun,
T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yu Yu, Yichao Zhang, Yifan Shi, Yilang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yun Gong, Yuheng Zou, Yu-jia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. R. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxia Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Le'an Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qiaohu Zhao, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shaojing Wu, Shengfeng Ye, Shengfeng Ye, Shiron Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuteng Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zha, Yuduan Wang, Yue Gong, Yuheng Zou, Yuja He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.

Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542.

Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024a. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404.15846.

Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunze Zhou, Jiaqing Liang, and Yanghua Xiao. 2024b. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2023. Follow-bench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint arXiv:2310.20410.

Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdul- lah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, et al. 2024. A systematic survey and critical review on evaluating large language models: Challenges, limitations, and recommendations. arXiv preprint arXiv:2407.04069.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models.

Yizhi Li, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai
Zhang, et al. 2024. Cif-bench: A chinese instruction-following benchmark for evaluating the generalizability of large language models. arXiv preprint arXiv:2402.13109.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.

Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487.

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa-har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707.

Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601.

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representations.

Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao-hua Dong, Ran Lin, and Ruohui Huang. 2024a. Conifer: Improving complex constrained instruction-following ability of large language models. arXiv preprint arXiv:2404.02823.

Kun Sun, Rong Wang, Haitao Liu, and Anders Søgaard. 2024b. Comprehensive reassessment of large-scale evaluation outcomes in llms: A multifaceted statistical approach. arXiv preprint arXiv:2403.15250.

Xiangru Tang, Yiming Zong, Yilun Zhao, Arman Cohan, and Mark Gerstein. 2023. Struc-bench: Are large language models really good at generating complex structured data? arXiv preprint arXiv:2309.08963.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Al bert, Amjad Almahairi, Yasrane Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Alicia Tsai, Shereen Oraby, Vittorio Perera, Jiun-Yu Kao, Yuheng Du, Anjali Narayan-Chen, Tagyoung Chung, and Dilek Hakkani-Tur. 2021. Style control for schema-guided natural language generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 228–242.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E Peters. 2020. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361–1375.

Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. 2024. Benchmarking complex instruction-following with multiple constraints composition. arXiv preprint arXiv:2407.03978.

Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. Fofo: A benchmark to evaluate llms’ format-following capability. arXiv preprint arXiv:2402.18667.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. 2023. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129.

Shunyu Yao, Howard Chen, Austin W Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023. Collie: Systematic construction of constrained text generation tasks. arXiv preprint arXiv:2307.08689.

Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1–37.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2856–2878.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. 2023b. Controlled text generation with natural language instructions. In International Conference on Machine Learning, pages 42602–42613. PMLR.

A Appendix

A.1 Constraint System Construction

The construction of the constraint system commenced with the aggregation of data from diverse real-world scenarios and NLP tasks. This encompassed 800,000 query logs from LLM websites over the preceding six months, alongside over 300,000 data points from various NLP tasks. Instructions that were excessively long or short were filtered out, and a vector clustering deduplication algorithm was employed. This meticulous process culminated in a refined dataset comprising approximately 30,000 instructions. Subsequently, GPT was utilized to extract constraint atoms from these instructions, thereby ensuring the comprehensiveness of the constraint system. The prompt employed for GPT-4 extraction, as illustrated in Fig. 7, resulted in the identification of approximately 5,000 unique atomic constraint instructions. Three seasoned experts meticulously refined these into 1,000 significant atomic constraints. By integrating statistical analysis, classification, and linguistic principles, a hierarchical constraint system was developed using Top-Down Organization and Bottom-Up Synthesis methodologies. This system comprises 10 primary categories and 25 secondary categories. The system comprehensively categorizes all types of constraints, ensuring that nearly all specific constraints can be systematically classified within its framework. Detailed information regarding the constraint system is presented in Tab. 9.

A.2 Dataset Construction

We adopted an innovative Human-Machine Collaborative Iterative Construction approach to ensure the highest quality of data. This method involved leveraging advanced LLMs to augment original instructions with additional constraints and generate corresponding responses. These responses were meticulously reviewed for constraint validity, followed by the creation of detailed checklists for each example. Multiple experts participated in this iterative process, continuously refining the outputs by addressing issues encountered by the LLMs and re-generating or manually correcting any substandard samples. The prompts used for GPT to enhance constraints and generate checklists are illustrated in Fig. 8 and Fig. 10. Due to the limited attention given to real-life scenarios, we have meticulously organized and covered 20 domains and over 200 scenarios in our CFBench system, as detailed in Tab. 10. In the end, we gathered 1,000 high-quality data points: 500 from real-world scenarios and 500 from different NLP tasks. Specifically, we implemented the following steps to enhance data quality for manual annotations. In Tab. 5, we list the proportions of the four major NLP task categories.

<table>
  <tr>
    <th>Task Type</th>
    <th>Prop.</th>
    <th>Samples’ num</th>
  </tr>
  <tr>
    <td>Generation</td>
    <td>46%</td>
    <td>230</td>
  </tr>
  <tr>
    <td>Classification</td>
    <td>30%</td>
    <td>150</td>
  </tr>
  <tr>
    <td>Sentence Relation</td>
    <td>12%</td>
    <td>60</td>
  </tr>
  <tr>
    <td>Sequence Labeling</td>
    <td>12%</td>
    <td>60</td>
  </tr>
</table>

Table 5: Distribution of four major NLP task categories.

A.2.1 Annotator Training

We sourced annotation contractors from the public and selected 21 candidates for training by seasoned data scientists. After a one-week training program, the annotators engaged in multiple rounds of trial annotations, which were then assessed by data experts. From these assessments, the 9 annotators demonstrating the highest accuracy were selected for this dataset.

A.2.2 Cross-Validation

To reduce the likelihood of missed and incorrect annotations, we implemented an inter-annotator validation process. Three annotators independently reviewed the labeled instructions, responses, and evaluation criteria, achieving a notable agreement rate of 94%. Any discrepancies that emerged were resolved through expert adjudication, ensuring both consistency and accuracy.
Prompt Template for Atomic Constraint Extraction

You are a professional atomic constraint extractor. Your task is to extract as many atomic constraint expressions as possible from the given [Instruction].
Definition of atomic constraint expression: The smallest unit of description or constraint for the required task within the instruction.
Common types of atomic constraints include: content constraints, numerical constraints, style constraints, format constraints.

[Example]
[Instruction]: [instruction]
[Atomic constraint]: {atomic constraint 1, atomic constraint 2, ...}

[Input]
[Instruction]: [instruction]
[Your Answer]: [atomic constraint]

Figure 7: Prompt Template Atomic Constraint Extract

Prompt Template for Adding Atomic Constraints

[Task Description]
1. I currently have a seed question, but the seed questions are relatively simple. To make the instructions for the seed question more complex, I want you to add more constraints to this question.
2. You will get [Seed Question] and [Constraint References], and you can use these constraint references to increase the difficulty of the seed question.
3. [Constraint Reference] are just suggestions for constraints. When adding constraints, you should not copy, freely combine from the constraint references, or add other constraints you deem appropriate.
4. Do not delete any information from the seed question. Your task is to rewrite the seed question by adding constraints without omitting any key information from the seed question, such as reference texts.
5. Directly return the modified question (the question with added constraints), without any additional words.

[Constraint References]
1. Lexical content constraint : [Definition] [Example]
2. Word Count : [Definition] [Example]
2.5. Rule Constraint : [Definition] [Example]

[Seed Question] : [raw_question]

[Modified Question] :

Figure 8: Prompt Template for Adding Atomic constraint

A.2.3 Batch Validation

Due to the substantial size of the dataset, it was systematically divided for processing. Following a phased improvement approach, the initial batch sizes were set at 50, 100, and 200, gradually increasing to 400 for later batches. After the annotation process, 50% of the dataset was randomly selected for contractor review, while 20% of the dataset was examined by experts.

A.2.4 Data Split

We used a voting mechanism involving experts and ten models, including GPT-4o and Claude-3.5-Sonnet, to partition 10,000 CF Bench entries into ‘easy’ and ‘hard’ categories. The ‘hard’ category includes entries where multiple models struggle with PSR performance and are also challenging for humans, as verified by experts.

A.3 Evaluation Method and Metric

The state-of-the-art GPT-4o model was employed as the judge to perform binary scoring (0 or 1) for each checkpoint in the checklist. The specific evaluation prompt is illustrated in Fig. 9. The Requirement Priority-Satisfaction Ratio (PSR) was proposed as an evaluation metric that simultaneously considers the prioritization of user requirements and satisfaction levels. PSR is calculated by first ensuring that all primary requirements are met. Subsequently, the satisfaction score is determined by averaging the fulfillment of the remaining constraints to obtain \( A \). The final satisfaction score is then calculated using the formula \( 0.5 + 0.5 * A \). If the final score exceeds 0.8, PSR is set to 1. The threshold of 0.8 was established based on the average satisfaction levels derived from multiple users’ feedback on the responses to the instructions.

A.4 Quality Assessment

We employed multiple methods to validate the quality of the benchmark on a randomly selected set of 100 samples. First, we engaged three experts to independently evaluate the quality of each sample’s instruction, response, and criteria. The average quality rate determined by the three experts was consistently above 90%, as detailed in Tab. 6. To further validate the effectiveness of our proposed evaluation metric, PSR, we had the same three experts score the responses of Qwen2-7B-Instruct on these 100 cases using a 0-1 scale. Simultaneously, we utilized GPT-4o to directly score the responses, referred to as GPT-4o PSR. By calculating the kappa coefficient, we found a strong agreement between our proposed PSR evaluation metric and the human experts’ assessments. The detailed results are presented in Tab. 7. Kappa coefficient scores are interpreted as follows: below 0.2 indicates slight agreement, 0.21 to 0.40 indicates fair agreement, 0.41 to 0.60 indicates moderate agreement, 0.61 to 0.80 indicates substantial agreement, and 0.81 to 1.00 indicates almost perfect agreement.

Prompt Template for GPT-4o Judge Evaluation

I want you to act as a quality evaluator.
Evaluate [Model Answer] based on the [User Instruction], [Reference Answer], and [CheckList], scoring it either 0 or 1. Both the [Reference Answer] and the [Model Answer] respond to the [User Instruction]. The [CheckList] contains the required user instruction. Score each point in the [CheckList] as 1 if the [Model Answer] meets it, otherwise score it as 0.
Note: If the [Reference Answer] is empty, ignore it when evaluating the [CheckList] points.

[Output Requirements]
1. For each point of the [CheckList] points, output one per line, separated by "or".
2. For each line, first output the [CheckList] content, then a ' ', and finally the [Evaluation Score] (0 or 1).
3. Please directly output your evaluation without any additional content.

[Examples]
[Example1]: [Example1];
[Example2]: [Example2];

[User Instruction]: [user instruction]
[Reference Answer]: [reference answer]
[Model Answer]: [model answer]
[CheckList]: [checklist]
[Your Evaluation]: [evaluation]

Figure 9: Prompt Template for Evaluation
<table>
  <tr>
    <th>Set</th>
    <th>Instruction</th>
    <th>Gold Ans</th>
    <th>CheckList</th>
  </tr>
  <tr>
    <td>Easy Set</td>
    <td>0.96</td>
    <td>0.94</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>Hard Set</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>All Set</td>
    <td>0.94</td>
    <td>0.94</td>
    <td>0.93</td>
  </tr>
</table>

Table 6: The High-Quality Rate of 100 selected Samples

<table>
  <tr>
    <th>Set</th>
    <th>Easy Set</th>
    <th>Hard Set</th>
    <th>Full Set</th>
  </tr>
  <tr>
    <td>Avg.Expert</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>GPT-4o DS</td>
    <td>0.58</td>
    <td>0.61</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>GPT-4o PSR</td>
    <td>0.76</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>Qwen2-72B-Inst. PSR</td>
    <td>0.70</td>
    <td>0.73</td>
    <td>0.72</td>
  </tr>
</table>

Table 7: The kappa coefficient between expert evaluations and various assessment methods

A.5 Experimental Setup and Results

A.5.1 Experiment Setting

We evaluated the most popular Large Language Models (LLMs), with the majority of these models being developed by companies based in China, primarily to accommodate our CF Bench’s focus on the Chinese language. Among the 50 evaluated models, they can be categorized into two groups based on their access method: API-based and open-source weight-based models. It is worth noting that the Llama series models do not primarily support the Chinese language, which results in noticeably lower performance. Both conifer-base and conifer-test are based on the Mistral-7B foundational model. Llama-3-8B-Instruct-CN and Llama-3-70B-Instruct-CN respectively represent Llama-3-8B-Instruct-Chinese and Llama-3-70B-Instruct-Chinese, both of which have undergone Chinese SFT (Supervised Fine-Tuning). For the base models, we used a 3-shot approach to ensure a fair evaluation. The complete list of evaluated models can be found in Tab. 8.

A.5.2 Explanation of Results

GPT-4o and Claude3.5-Sonnet have demonstrated near-absolute leadership, achieving outstanding performance across various metrics and categories. Similarly, models such as GLM-4-0510, ERNIE-4-Bot-0613, ERNIE-4-Turbo-0628, DeepSeek-V2-0628, and Qwen2-72B-Instruct have also exhibited strong capabilities. Many models that support less mainstream Chinese languages performed significantly worse, which is unfair to them and only serves to illustrate their relative rankings. This also confirms that performance are highly correlated with language, especially within the scope of language constraints. From the perspective of open-source versus closed-source models, open-source models have generally achieved comprehensive success. However, Qwen2-72B-Instruct, as an open-source model, also demonstrated notable constraint-following capabilities. Regarding model size, within the Qwen series, performance metrics clearly improve with increasing model size. Additionally, models that have undergone Supervised Fine-Tuning (SFT) show significantly enhanced instruction-following capabilities. The complete evaluation results and rankings can be found in Tab. 8.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="3">Easy Set</th>
    <th colspan="3">Hard Set</th>
    <th colspan="3">Full Set</th>
  </tr>
  <tr>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
    <th>CSR</th>
    <th>ISR</th>
    <th>PSR</th>
  </tr>
  <tr><td>GPT-4o†</td><td>0.956</td><td>0.868</td><td>0.888</td><td>0.816</td><td>0.438</td><td>0.582</td><td>0.886</td><td>0.653</td><td>0.735</td></tr>
  <tr><td>GPT-4-Turbo-20240409†</td><td>0.924</td><td>0.792</td><td>0.826</td><td>0.783</td><td>0.370</td><td>0.518</td><td>0.853</td><td>0.581</td><td>0.672</td></tr>
  <tr><td>GPT-4-0125-Preview†</td><td>0.923</td><td>0.790</td><td>0.826</td><td>0.763</td><td>0.310</td><td>0.468</td><td>0.843</td><td>0.550</td><td>0.647</td></tr>
  <tr><td>GPT-3.5-Turbo-1106†</td><td>0.797</td><td>0.520</td><td>0.602</td><td>0.631</td><td>0.176</td><td>0.326</td><td>0.714</td><td>0.348</td><td>0.464</td></tr>
  <tr><td>Claude-3.5-Sonnet†</td><td>0.943</td><td>0.844</td><td>0.882</td><td>0.799</td><td>0.408</td><td>0.564</td><td>0.871</td><td>0.626</td><td>0.723</td></tr>
  <tr><td>GLM-4-0520†</td><td>0.939</td><td>0.820</td><td>0.852</td><td>0.785</td><td>0.372</td><td>0.536</td><td>0.862</td><td>0.596</td><td>0.694</td></tr>
  <tr><td>DeepSeek-V2-0628†</td><td>0.946</td><td>0.830</td><td>0.868</td><td>0.786</td><td>0.350</td><td>0.524</td><td>0.866</td><td>0.590</td><td>0.696</td></tr>
  <tr><td>ERNIE-4-Turbo-0628†</td><td>0.930</td><td>0.790</td><td>0.848</td><td>0.772</td><td>0.332</td><td>0.532</td><td>0.851</td><td>0.561</td><td>0.690</td></tr>
  <tr><td>ERNIE-4-Bot-0613†</td><td>0.929</td><td>0.792</td><td>0.832</td><td>0.779</td><td>0.338</td><td>0.518</td><td>0.854</td><td>0.565</td><td>0.675</td></tr>
  <tr><td>ERNIE-3.5-0613†</td><td>0.901</td><td>0.720</td><td>0.772</td><td>0.758</td><td>0.302</td><td>0.482</td><td>0.830</td><td>0.511</td><td>0.627</td></tr>
  <tr><td>Yi-Large†</td><td>0.900</td><td>0.730</td><td>0.786</td><td>0.744</td><td>0.292</td><td>0.460</td><td>0.822</td><td>0.511</td><td>0.623</td></tr>
  <tr><td>abab6.5-chat†</td><td>0.894</td><td>0.696</td><td>0.766</td><td>0.736</td><td>0.260</td><td>0.452</td><td>0.815</td><td>0.478</td><td>0.609</td></tr>
  <tr><td>MoonShot-V1-8k†</td><td>0.919</td><td>0.764</td><td>0.812</td><td>0.758</td><td>0.308</td><td>0.464</td><td>0.838</td><td>0.536</td><td>0.638</td></tr>
  <tr><td>Vicuna-7B-V13*</td><td>0.563</td><td>0.206</td><td>0.262</td><td>0.468</td><td>0.100</td><td>0.168</td><td>0.516</td><td>0.153</td><td>0.215</td></tr>
  <tr><td>Vicuna-33B-V13*</td><td>0.621</td><td>0.270</td><td>0.352</td><td>0.527</td><td>0.110</td><td>0.196</td><td>0.574</td><td>0.190</td><td>0.274</td></tr>
  <tr><td>Vicuna-13B-V13*</td><td>0.605</td><td>0.248</td><td>0.302</td><td>0.503</td><td>0.100</td><td>0.178</td><td>0.554</td><td>0.174</td><td>0.240</td></tr>
  <tr><td>Llama-2-7B-Chat*</td><td>0.5268</td><td>0.198</td><td>0.250</td><td>0.448</td><td>0.096</td><td>0.152</td><td>0.487</td><td>0.147</td><td>0.201</td></tr>
  <tr><td>Llama-2-13B-Chat*</td><td>0.574</td><td>0.242</td><td>0.280</td><td>0.488</td><td>0.094</td><td>0.178</td><td>0.531</td><td>0.168</td><td>0.229</td></tr>
  <tr><td>Llama-3-8B-Instruct*</td><td>0.656</td><td>0.300</td><td>0.356</td><td>0.562</td><td>0.122</td><td>0.238</td><td>0.609</td><td>0.211</td><td>0.297</td></tr>
  <tr><td>Llama-3-70B-Instruct*</td><td>0.750</td><td>0.422</td><td>0.498</td><td>0.642</td><td>0.178</td><td>0.330</td><td>0.696</td><td>0.300</td><td>0.414</td></tr>
  <tr><td>Mistral-7B-Instruct-V03*</td><td>0.227</td><td>0.072</td><td>0.086</td><td>0.148</td><td>0.008</td><td>0.022</td><td>0.188</td><td>0.040</td><td>0.054</td></tr>
  <tr><td>Conifer-Base*</td><td>0.510</td><td>0.184</td><td>0.232</td><td>0.300</td><td>0.018</td><td>0.048</td><td>0.405</td><td>0.101</td><td>0.140</td></tr>
  <tr><td>Conifer-Test*</td><td>0.559</td><td>0.215</td><td>0.255</td><td>0.328</td><td>0.102</td><td>0.156</td><td>0.443</td><td>0.159</td><td>0.206</td></tr>
  <tr><td>BaiChuan-13B-Chat</td><td>0.630</td><td>0.306</td><td>0.366</td><td>0.521</td><td>0.114</td><td>0.196</td><td>0.575</td><td>0.210</td><td>0.281</td></tr>
  <tr><td>BaiChuan2-13B-Chat</td><td>0.669</td><td>0.348</td><td>0.418</td><td>0.547</td><td>0.134</td><td>0.226</td><td>0.608</td><td>0.241</td><td>0.322</td></tr>
  <tr><td>Llama-3-8B-Instruct-CN</td><td>0.743</td><td>0.458</td><td>0.510</td><td>0.627</td><td>0.162</td><td>0.314</td><td>0.685</td><td>0.310</td><td>0.412</td></tr>
  <tr><td>Llama-3-70B-Instruct-CN</td><td>0.756</td><td>0.482</td><td>0.536</td><td>0.636</td><td>0.190</td><td>0.322</td><td>0.696</td><td>0.336</td><td>0.429</td></tr>
  <tr><td>DeepSeek-7B-Chat</td><td>0.695</td><td>0.378</td><td>0.442</td><td>0.580</td><td>0.150</td><td>0.270</td><td>0.638</td><td>0.264</td><td>0.356</td></tr>
  <tr><td>DeepSeek-V2-Lite-Chat</td><td>0.733</td><td>0.382</td><td>0.448</td><td>0.597</td><td>0.148</td><td>0.262</td><td>0.665</td><td>0.265</td><td>0.355</td></tr>
  <tr><td>DeepSeek-67B-Chat</td><td>0.802</td><td>0.516</td><td>0.578</td><td>0.662</td><td>0.180</td><td>0.350</td><td>0.732</td><td>0.348</td><td>0.464</td></tr>
  <tr><td>InternLM2-Chat-7B</td><td>0.767</td><td>0.452</td><td>0.538</td><td>0.625</td><td>0.172</td><td>0.320</td><td>0.696</td><td>0.312</td><td>0.429</td></tr>
  <tr><td>GLM-4-9B-Chat</td><td>0.885</td><td>0.678</td><td>0.742</td><td>0.742</td><td>0.288</td><td>0.450</td><td>0.813</td><td>0.483</td><td>0.596</td></tr>
  <tr><td>YI-1.5-34B-Chat</td><td>0.881</td><td>0.672</td><td>0.740</td><td>0.745</td><td>0.302</td><td>0.474</td><td>0.813</td><td>0.487</td><td>0.607</td></tr>
  <tr><td>Qwen1.5-4B</td><td>0.454</td><td>0.170</td><td>0.198</td><td>0.376</td><td>0.074</td><td>0.116</td><td>0.415</td><td>0.122</td><td>0.157</td></tr>
  <tr><td>Qwen1.5-4B-Chat</td><td>0.652</td><td>0.310</td><td>0.362</td><td>0.536</td><td>0.104</td><td>0.198</td><td>0.594</td><td>0.207</td><td>0.280</td></tr>
  <tr><td>Qwen1.5-7B</td><td>0.473</td><td>0.176</td><td>0.212</td><td>0.400</td><td>0.090</td><td>0.142</td><td>0.437</td><td>0.133</td><td>0.177</td></tr>
  <tr><td>Qwen1.5-7B-Chat</td><td>0.799</td><td>0.534</td><td>0.592</td><td>0.654</td><td>0.194</td><td>0.338</td><td>0.726</td><td>0.364</td><td>0.465</td></tr>
  <tr><td>Qwen1.5-14B</td><td>0.498</td><td>0.228</td><td>0.280</td><td>0.430</td><td>0.110</td><td>0.176</td><td>0.464</td><td>0.169</td><td>0.228</td></tr>
  <tr><td>Qwen1.5-14B-Chat</td><td>0.822</td><td>0.558</td><td>0.626</td><td>0.671</td><td>0.202</td><td>0.370</td><td>0.746</td><td>0.380</td><td>0.498</td></tr>
  <tr><td>Qwen1.5-32B</td><td>0.647</td><td>0.336</td><td>0.408</td><td>0.528</td><td>0.132</td><td>0.224</td><td>0.587</td><td>0.234</td><td>0.316</td></tr>
  <tr><td>Qwen1.5-32B-Chat</td><td>0.883</td><td>0.678</td><td>0.744</td><td>0.704</td><td>0.228</td><td>0.412</td><td>0.793</td><td>0.453</td><td>0.578</td></tr>
  <tr><td>Qwen1.5-72B</td><td>0.627</td><td>0.324</td><td>0.380</td><td>0.556</td><td>0.148</td><td>0.248</td><td>0.591</td><td>0.236</td><td>0.314</td></tr>
  <tr><td>Qwen1.5-72B-Chat</td><td>0.896</td><td>0.710</td><td>0.776</td><td>0.730</td><td>0.254</td><td>0.436</td><td>0.813</td><td>0.482</td><td>0.606</td></tr>
  <tr><td>Qwen1.5-110B-Chat</td><td>0.905</td><td>0.724</td><td>0.792</td><td>0.730</td><td>0.276</td><td>0.438</td><td>0.818</td><td>0.500</td><td>0.615</td></tr>
  <tr><td>Qwen2-0.5B-Instruct</td><td>0.446</td><td>0.150</td><td>0.172</td><td>0.393</td><td>0.070</td><td>0.110</td><td>0.419</td><td>0.110</td><td>0.141</td></tr>
  <tr><td>Qwen2-1.5B-Instruct</td><td>0.607</td><td>0.250</td><td>0.316</td><td>0.496</td><td>0.104</td><td>0.168</td><td>0.551</td><td>0.177</td><td>0.242</td></tr>
  <tr><td>Qwen2-7B</td><td>0.576</td><td>0.260</td><td>0.316</td><td>0.478</td><td>0.120</td><td>0.192</td><td>0.527</td><td>0.190</td><td>0.254</td></tr>
  <tr><td>Qwen2-7B-Instruct</td><td>0.835</td><td>0.584</td><td>0.642</td><td>0.682</td><td>0.198</td><td>0.362</td><td>0.758</td><td>0.391</td><td>0.502</td></tr>
  <tr><td>Qwen2-72B</td><td>0.711</td><td>0.424</td><td>0.484</td><td>0.568</td><td>0.170</td><td>0.274</td><td>0.640</td><td>0.297</td><td>0.379</td></tr>
  <tr><td>Qwen2-72B-Instruct</td><td>0.944</td><td>0.836</td><td>0.880</td><td>0.791</td><td>0.342</td><td>0.530</td><td>0.867</td><td>0.589</td><td>0.705</td></tr>
</table>

Table 8: The complete evaluation results and rankings of CFBench and its respective subsets. Notably, * stands for the model supporting mainstream languages excluding Chinese, and † represents calling through the API. The bold, underlined, and tilde denote the first, second, and third rankings, respectively. Llama-3-8B-Instruct-CN and Llama-3-70B-Instruct-CN respectively represent Llama-3-8B-Instruct-Chinese and Llama-3-70B-Instruct-Chinese, both of which have undergone Chinese SFT (Supervised Fine-Tuning). Both conifer_base and conifer_test are based on the Mistral-7B foundational model. For the base model, we used a 3-shot approach for generation.
<table>
  <tr>
    <th>Primary</th>
    <th>Secondary</th>
    <th>Definition</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="3">Content Constraint</td>
    <td>Lexical</td>
    <td>Mandatory use of specific terms or symbols, including their inclusion and precise placement.</td>
    <td>...must include the word "beautiful."</td>
  </tr>
  <tr>
    <td>Element</td>
    <td>Mandates for including specific elements or concepts in responses, reflecting a scenario or object.</td>
    <td>...highlights the Great Wall.</td>
  </tr>
  <tr>
    <td>Semantic</td>
    <td>Directives on thematic content, perspective, or tone, emphasizing response significance.</td>
    <td>Write a poem about London.</td>
  </tr>
  <tr>
    <td rowspan="4">Numerical Constraint</td>
    <td>Word Count</td>
    <td>Limit the number of words or tokens.</td>
    <td>A 50-word poem.</td>
  </tr>
  <tr>
    <td>Sentence Count</td>
    <td>Limit the number of sentences.</td>
    <td>... three sentences.</td>
  </tr>
  <tr>
    <td>Paragraph Count</td>
    <td>Limit the number of paragraphs.</td>
    <td>divided into 3 sections.</td>
  </tr>
  <tr>
    <td>Document Count</td>
    <td>Limit the number of documents.</td>
    <td>... list 3 articles.</td>
  </tr>
  <tr>
    <td rowspan="4">Stylistic Constraint</td>
    <td>Tone and emotion</td>
    <td>The emotional tone must adhere to standards such as seriousness, anger, joy, humor, and politeness.</td>
    <td>Write a letter in an angry and sarcastic tone.</td>
  </tr>
  <tr>
    <td>Form and style</td>
    <td>Text expression standards ensure alignment with specific stylistic criteria in both presentation and perception.</td>
    <td>Write a passage in an encyclopedic style.</td>
  </tr>
  <tr>
    <td>Audience-specific</td>
    <td>Text should be tailored to specific audiences, ensuring clarity and relevance for children, students, or specialized groups.</td>
    <td>Write a pome for a 6-year-old.</td>
  </tr>
  <tr>
    <td>Authorial style</td>
    <td>Texts should emulate the styles of authors like Shakespeare to achieve artistic effects or depth.</td>
    <td>Write a passage the style of Shakespeare.</td>
  </tr>
  <tr>
    <td rowspan="3">Format Constraint</td>
    <td>Fundamental</td>
    <td>Widely accepted and utilized standard formats, including JSON, XML, LaTeX, HTML, Table, and Markdown.</td>
    <td>Extract keywords and output in JSON format.</td>
  </tr>
  <tr>
    <td>Bespoke</td>
    <td>Protocols for information expression tailored to specific needs, including paragraphing, headings, text emphasis, examples, and bullet points.</td>
    <td>Summarize the main idea and output in unordered list format.</td>
  </tr>
  <tr>
    <td>Specialized</td>
    <td>Formatting standards tailored for specialized applications or domains.</td>
    <td>Conform to electronic medical record format.</td>
  </tr>
  <tr>
    <td rowspan="4">Linguistic Constraint</td>
    <td>Pragmatic</td>
    <td>Contextual language study, encompassing speech acts, implicature, discourse, dialects, sociolects, and language policy.</td>
    <td>Output in English, in classical Chinese style.</td>
  </tr>
  <tr>
    <td>Syntactic</td>
    <td>Sentence structure, including phrases, constituents, subordinate clauses, ba-constructions, and imperatives.</td>
    <td>Use imperatives with nouns and verb phrases.</td>
  </tr>
  <tr>
    <td>Morphological</td>
    <td>The internal structure and formation rules of words, including roots, affixes, and morphological changes.</td>
    <td>Output all content in lowercase English.</td>
  </tr>
  <tr>
    <td>Phonological</td>
    <td>Study on phonological structures: phonemes, allophones, pitch, duration, and intensity.</td>
    <td>Single-rhyme tongue twisters.</td>
  </tr>
  <tr>
    <td rowspan="3">Situation Constraint</td>
    <td>Role-based</td>
    <td>Simulating characters based on context, emulating their traits, language, and behaviors.</td>
    <td>You are Confucius, how do you decide?</td>
  </tr>
  <tr>
    <td>Task-specific</td>
    <td>Offer tailored solutions based on a nuanced understanding of situational demands.</td>
    <td>Must work from home, how to report?</td>
  </tr>
  <tr>
    <td>Complex context</td>
    <td>Reasoning and problem-solving within intricate and multifaceted contexts.</td>
    <td>4 on the left, 10 total, which from right?</td>
  </tr>
  <tr>
    <td>Example Constraint</td>
    <td>-</td>
    <td>Regulate new responses by leveraging intrinsic patterns from a limited set of samples.</td>
    <td>Example: input:xxx, output:{...}; input:xx, output?</td>
  </tr>
  <tr>
    <td>Inverse Constraint</td>
    <td>-</td>
    <td>Narrow the response space through inverse constraints and indirect exclusion.</td>
    <td>Prohibited from answer political topics.</td>
  </tr>
  <tr>
    <td>Contradictory Constraint</td>
    <td>-</td>
    <td>Mutually exclusive constraints prevent fulfilling all requirements concurrently.</td>
    <td>Write a five-character quatrain, 1000 words.</td>
  </tr>
  <tr>
    <td>Rule Constraint</td>
    <td>-</td>
    <td>Standardize the road of responses through meticulously crafted logic flows or actions.</td>
    <td>Each answer adds 1, 1+1=3, then 2+3=?</td>
  </tr>
</table>

Table 9: Constraint System of CFBench
<table>
  <tr>
    <th>Domain</th>
    <th>Scenarios List</th>
  </tr>
  <tr>
    <td>Healthcare</td>
    <td>Symptom Consultation<br>Wellness<br>Medical Education<br>Teaching Methods</td>
    <td>Diagnostic Explanation<br>Medical Info</td>
    <td>Medication Guidance<br>Guidelines Inquiry</td>
    <td>Procedures<br>Public Health</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>Academic Counseling<br>Reports<br>Market Research<br>Corporate Tax</td>
    <td>Resource Access<br>Mental Health Support<br>Interests<br>Stock Analysis</td>
    <td>Curriculum Design<br>Tutoring</td>
    <td>Communication<br>Subject Q&amp;A</td>
  </tr>
  <tr>
    <td>Finance</td>
    <td>Financial Education<br>Regulatory Analysis<br>Legal Education<br>Statute Explanation<br>Case Management</td>
    <td>Insurance Management<br>Product Development<br>Legal Consultation<br>Regulation Analysis<br>Compliance & Risk</td>
    <td>Investment Analysis<br>Corporate Financing<br>Customer Service<br>Document Review<br>IP Management</td>
    <td>Personal Finance<br>Compliance & Risk<br>Financial Reports<br>Case Analysis<br>Legal Training</td>
  </tr>
  <tr>
    <td>Legal</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Media</td>
    <td>Content Creation</td>
    <td>Information Analysis</td>
    <td>Marketing & Promotion</td>
    <td>News Reporting</td>
  </tr>
  <tr>
    <td>Tourism</td>
    <td>Travel Consultation<br>JD Writing & Analysis</td>
    <td>Itinerary Planning<br>Resume Creation</td>
    <td>Route Introduction<br>Resume Screening</td>
    <td>Interview Preparation</td>
  </tr>
  <tr>
    <td>Recruitment</td>
    <td>Interview Evaluation<br>Performance Review<br>Policy Research<br>Document Writing<br>Emergency Management<br>Purchase Planning<br>Leasing<br>Property Description<br>Renovation</td>
    <td>Career Planning<br>Public Education<br>Content Review<br>Market Trends<br>Property Valuation<br>Content Creation<br>Driving & Safety<br>Loan Calculation<br>Maintenance & Repair<br>Family<br>Social<br>Client Relations<br>Product Design<br>Data Management<br>Marketing<br>Divination<br>Spirituality<br>Goal Setting<br>Performance<br>Data Tracking<br>Shopping Decisions<br>Naming<br>Life Creations<br>TV & Film<br>Art<br>Content Creation<br>Translation<br>Customer Service<br>Reviews & Feedback<br>Recipes & Menus<br>Food Safety<br>Reviews<br>Development & Design<br>Marketing & Promotion<br>Launches<br>Product Design</td>
    <td>Service Guide<br>Business Procedures<br>Property Policies<br>Amenities<br>Sales & Marketing<br>Customer Experience<br>Insurance Evaluation<br>Friendship<br>Sexuality & Gender<br>Crisis Intervention<br>User Research<br>Cybersecurity<br>Operations<br>Feng Shui<br>Healing<br>Nutrition<br>Injury Care<br>Instant Queries<br>Recommendations<br>Q&amp;A<br>Music<br>Cultural Events<br>Office Efficiency<br>Team Collaboration<br>Marketing & Promo<br>Cooking Techniques<br>Culinary Training<br>Hardware & Peripherals<br>Operations<br>Esports & Tournaments<br>Buying Guides<br>Marketing Copy</td>
    <td>Communication Skills<br>Public Services<br>Civil Servant Training<br>Development<br>Financial Services<br>Qualifications<br>Model Consultation<br>Claims Assessment<br>Workplace<br>Life Stages<br>Public Psychology<br>Coding & Debugging<br>Computer Q&amp;A<br>UI/UX Design<br>Astrology<br>Content Review<br>Workout Plans<br>Specialized Training<br>Skincare<br>Planning<br>Literature<br>Short Videos & Live<br>Marketing<br>Collaboration<br>Food Content<br>Ingredient Prep<br>News<br>Mini Games<br>Culture & Education<br>Tips & Tricks<br>After-sales & Repairs</td>
  </tr>
</table>

Table 10: Domain and Scenarios List
<table>
  <tr>
    <th>Benchmark</th>
    <th>Case</th>
    <th>Features</th>
    <th>CFBench’s Advantages</th>
  </tr>
  <tr>
    <td>Complex Bench</td>
    <td><b>Instruction:</b> Please introduce the following painting.<br>- If the work contains any animal, the description should be in English.<br>- Otherwise, the description should be in Chinese.<br><b>Response:</b> Painting: "Mona Lisa"</td>
    <td>Focus on the <b>combination</b> of instructions, including And/Chain/Selection/Nested.<br><img src="page_420_312_180_180.png" alt="Diagram showing S_1(cond_1) branching into B_1 and B_2"></td>
    <td>CFBench supports <b>simplicity in instruction compliance, broad applicability across use cases,</b> and <b>scalability</b> for various tasks.</td>
  </tr>
  <tr>
    <td>Follow Bench</td>
    <td><b>Instruction:</b> Using following words to generate a story,<br>"needed, money, computer, bought, happy".<br><b>Response:</b> In the heart of the city, there was a research facility that desperately needed funding for a crucial project. To gather the required money, ...</td>
    <td>Focus on <b>gradual constraint escalation</b>. The more constraints per instruction, the greater the complexity.<br><img src="page_420_670_180_180.png" alt="Diagram showing Constraint Complexity increasing with Number of Constraints per Instruction"></td>
    <td>CFBench is <b>simple</b> to use, <b>resource-efficient</b>, and scalable for large models.</td>
  </tr>
  <tr>
    <td>IFEval</td>
    <td><b>Instruction:</b> Given the sentence "Two young boys with toy guns and horns." can you ask a question? Please ensure that your response is in English, and in all lowercase letters. No capital letters are allowed.<br><b>Response:</b> what are the two young boys holding?</td>
    <td>Verifiable Instructions.<br><img src="page_420_1030_180_180.png" alt="Diagram showing a magnifying glass over a question mark"></td>
    <td>CFBench is easy to implement, <b>general-purpose across models</b>, and scalable for large datasets.</td>
  </tr>
  <tr>
    <td>CFBench</td>
    <td><b>Instruction:</b> Overview of the Trump Shooting Incident, include the cause, process, aftermath, shooter information and Trump’s feedback. Organize the content using numbered points and bold text. The cause should be in JSON format with event keywords: time, location, and nature. Summarize the process in three sentences. Discuss the impact from both international and domestic perspectives, keeping the total content under 500 words. Finally, summarize this event in a seven-word quatrain.<br><b>Response:</b> 1. Cause \n"time":"July 13, 2024", "location":"Butler, Pennsylvania, USA", "nature":"assassination attempt"\n 2. Process \n-Trump was giving a speech at the rally when the shooting occurred. \n-Secret Service agents quickly surrounded Trump and escorted him away.\n 3. ...</td>
    <td><b>Comprehensive Coverage:</b> Multi-dimensional constraints;<br>Cross-task evaluation;<br>Real-world complexity.<br><b>Standardized Framework:</b> Unified metrics;<br>Cross-model compatibility.<br><img src="page_420_1390_180_180.png" alt="Diagram showing a tree structure with categories, subcategories, and constraints"></td>
    <td>Diverse instruction types, standardized evaluation, hierarchical constraints.</td>
  </tr>
</table>

Table 11: Benchmark Comparison for Different Instruction Types and Task Complexity
Prompt Template for CheckList Generation

# Task Description
You are a professional instruction and response review expert. You need to extract all possible checklist items that could affect the evaluation of the result based on the [Instruction] and [Response] within the [Constraint System]. Your output should be as detailed as possible. Use numerical numbering and "m" to list all points in the checklist.

# Constraint System
1. Lexical content constraint: {Definition} {Example}
2. Word Count: {Definition} {Example}

25 Rule Constraint: {Definition} {Example};

# Given Example
Instruction: Overview of the Trump Shooting Incident, include the cause, process, impact, aftermath, shooter information and Trump's feedback.
Organize the content using numbered points and bold text. The cause should be in JSON format with event keywords: time, location, and nature.
Summarize the process in three sentences. Discuss the impact from both international and domestic perspectives, keeping the total content under 500 words. Finally, summarize this event in a seven-word quatrain.

<table>
  <tr>
    <th>CheckList</th>
    <th>Constraints</th>
    <th>Priority</th>
    <th>Satisf.</th>
  </tr>
  <tr>
    <td>1.Content on Trump Shooting Incident</td>
    <td>Semantic Content</td>
    <td>Primary</td>
    <td></td>
  </tr>
  <tr>
    <td>2.Includes cause, process, impact</td>
    <td>Element Content</td>
    <td>Primary</td>
    <td></td>
  </tr>
  <tr>
    <td>3.Use numbered points and bold text</td>
    <td>Bespoke Format</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>4.Cause in JSON with event keywords</td>
    <td>Event Format</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>5.Process in three sentences</td>
    <td>Sentence Count</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>6.Impact: international and domestic</td>
    <td>Element Content</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>7.Total content under 500 words</td>
    <td>Word Count</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>8.Seven-word quatrain</td>
    <td>Pragmatic</td>
    <td>Secondary</td>
    <td></td>
  </tr>
</table>

Response: Overview of the Trump Shooting Incident. 1.Cause: "time": "July 13, 2024", "location": "Butler, Pennsylvania, USA", "nature": "assassination attempt"; 2.Process: n-Trump was giving a speech at the rally when the shooting occurred... n-Secret Service agents quickly surrounded Trump and escorted him away; 3.Impact: n-International: The incident shocked the world, raising concerns about US political violence. n-Domestic: US political tensions and polarization increased; Trump's support may rise. n-Shooter Information: Details are not yet released; the FBI is investigating. n.5.Seven-word quatrain: n-Fomer President Trump was shot at a rally; the gunman was killed by the Secret Service, shocking the world.

<table>
  <tr>
    <th>指令</th>
    <th>回复</th>
  </tr>
  <tr>
    <td>特朗普枪击事件概述。1.原因：(时间：“2024年7月13日”，地点：“美国宾夕法尼亚州巴特勒”，性质：“暗杀未遂”)。2.过程：特朗普在集会演讲时发生了枪击事件，秘密服务人员迅速将特朗普护送离开。3.影响：国际：该事件震惊了全世界，引发了对美国政治暴力的担忧。国内：美国的政治紧张局势和两极化加剧，特朗普的支持率可能上升。4.枪手信息：细节尚未公布，联邦调查局正在调查中。5.七个字的四行诗：前总统特朗普在集会上被枪击，枪手被秘密服务部门击毙，震惊了全世界。</td>
    <td>特朗普枪击事件概述。1.原因：(时间：“2024年7月13日”，地点：“美国宾夕法尼亚州巴特勒”，性质：“暗杀未遂”)。2.过程：特朗普在集会演讲时发生了枪击事件，秘密服务人员迅速将特朗普护送离开。3.影响：国际：该事件震惊了全世界，引发了对美国政治暴力的担忧。国内：美国的政治紧张局势和两极化加剧，特朗普的支持率可能上升。4.枪手信息：细节尚未公布，联邦调查局正在调查中。5.七个字的四行诗：前总统特朗普在集会上被枪击，枪手被秘密服务部门击毙，震惊了全世界。</td>
  </tr>
</table>

Figure 10: Prompt Template for CheckList Generation

<table>
  <tr>
    <th>Instruction</th>
    <th>cause, process, impact,</th>
    <th>Organize the content using numbered points and bold text. The cause should be in JSON format with event keywords: time, location, and nature.</th>
    <th>Summarize the process in three sentences. Discuss the impact from both international and domestic perspectives, keeping the total content under 500 words. Finally, summarize this event in a seven-word quatrain.</th>
  </tr>
  <tr>
    <th>Response</th>
    <td>Overview of the Trump Shooting Incident. 1.Cause: "time": "July 13, 2024", "location": "Butler, Pennsylvania, USA", "nature": "assassination attempt"; 2.Process: n-Trump was giving a speech at the rally when the shooting occurred... n-Secret Service agents quickly surrounded Trump and escorted him away; 3.Impact: n-International: The incident shocked the world, raising concerns about US political violence. n-Domestic: US political tensions and polarization increased; Trump's support may rise. n-Shooter Information: Details are not yet released; the FBI is investigating. n.5.Seven-word quatrain: n-Fomer President Trump was shot at a rally; the gunman was killed by the Secret Service, shocking the world.</td>
    <td></td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th>CheckList</th>
    <th>Constraints</th>
    <th>Priority</th>
    <th>Satisf.</th>
  </tr>
  <tr>
    <td>1.Content on Trump Shooting Incident</td>
    <td>Semantic Content</td>
    <td>Primary</td>
    <td></td>
  </tr>
  <tr>
    <td>2.Includes cause, process, impact</td>
    <td>Element Content</td>
    <td>Primary</td>
    <td></td>
  </tr>
  <tr>
    <td>3.Use numbered points and bold text</td>
    <td>Bespoke Format</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>4.Cause in JSON with event keywords</td>
    <td>Event Format</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>5.Process in three sentences</td>
    <td>Sentence Count</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>6.Impact: international and domestic</td>
    <td>Element Content</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>7.Total content under 500 words</td>
    <td>Word Count</td>
    <td>Secondary</td>
    <td></td>
  </tr>
  <tr>
    <td>8.Seven-word quatrain</td>
    <td>Pragmatic</td>
    <td>Secondary</td>
    <td></td>
  </tr>
</table>

Figure 11: CFbench Example: English-Chinese Comparison, Data Itself in Chinese.

Reviewer paper summaries: The paper introduces CFBench, a comprehensive Chinese benchmark designed to evaluate the constraint-following capabilities of large language models (LLMs). The CFBench comprises 1,000 curated samples, accompanied by a hierarchical constraint system. The authors propose a multi-dimensional evaluation framework that prioritizes user-centric requirements. Experiments on leading LLMs reveal significant gaps in constraint compliance, particularly for contradictory and fine-grained constraints. The benchmark’s construction emphasizes real-world applicability, expert validation, and iterative refinement, offering valuable insights for improving LLM instruction-following abilities.
Summary of strengths: 1. Comprehensive and Realistic Dataset: CFBench excels in its coverage of diverse real-world scenarios and NLP tasks, ensuring high ecological validity. The systematic constraint taxonomy, derived from authentic user instructions and expert curation, mitigate the  gap in existing benchmarks focused on fragmented or narrow constraints.

2. Innovative Evaluation Framework: The integration of multi-dimensional metrics (CSR, ISR, PSR) and requirement prioritization aligns evaluation with user perspectives, enhancing interpretability and practical relevance. 

3. Actionable Insights: The analysis of factors influencing constraint compliance (e.g., prompt length, constraint quantity) and improvement strategies (e.g., supervised fine-tuning, model scaling) provides concrete guidance for model developers.
Summary of weaknesses: 1.Limited Cross-Language Generalization: The benchmark focuses exclusively on Chinese language models, with minimal discussion of English or multilingual LLMs. This limits its applicability to global research communities and raises questions about whether findings generalize beyond Chinese-specific linguistic and cultural constraints.

2. Potential Bias in Evaluation Methodology: Heavy reliance on GPT-4o as the sole judge model introduces risks of inherent biases (e.g., preference for GPT-style outputs). 

3. Under-Explored Constraint Combinatorics: While CFBench emphasizes constraint diversity, it does not rigorously analyze how overlapping or interdependent constraints (e.g., contradictory + numerical constraints) affect model performance. This limits understanding of LLM behavior in highly complex, real-world instructions.
Comments, suggestions and typos: Method
More details about the dataset construction are needed. For example, aside from the log files, what are the specific data sources of the benchmark, and what are their corresponding sampling distributions? Since the dataset construction in the paper involves human experts, what is their specific workflow? Additionally, I am curious about how the authors determine the taxonomy of constraints belonging to the same category. For instance, I noticed that "Semantic Content" is categorized under "primary need," whereas "Element Content" falls under "secondary need." However, this categorization is not explicitly presented in Table 8.

Experiments
More SOTA models: It would be better to introduce more SOTA instruction-following models, such as Qwen2.5, Claude-3.7-Sonnet-thinking.
Mitigate the reliance on LLM-based evaluator: Actually, some of the constraints can be readily evaluated without LLM-based evaluator, e.g., Word Count. It would be better to introduce more LLM-free evaluation to reduce the evaluation cost of the benchmark.
Overall assessment scores: 3.5