Title: Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Benchmark: COMPLEXBENCH
Summary: COMPLEXBENCH is a new benchmark designed to evaluate the complex instruction-following ability of large language models, focusing specifically on the composition of multiple constraints within instructions. The authors introduce a rule-augmented LLM-based evaluation method to reliably assess model performance on complex instructions with composed constraints.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Complex instruction‑following,  natural language understanding
Phenomenon defined by authors: Yes
Phenomenon definition: LLM’s capacity to parse instructions containing several distinct constraint types and to produce an output that simultaneously meets them all.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires the LLM to generate text that satisfies multiple constraints presented in a complex instruction, where these constraints can be combined in various ways (composition types). 
Task item definition: complex instruction with multiple constraints and composition types, corresponding scoring questions to verify if the constraints, and compositions are met in a generated response.
Task definition detail: Items vary in composition (And/Chain/Selection) and constraint types.
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The instructions are constructed based on real-world application scenarios and existing instruction-following benchmarks. The analysis of composition types in real-world instructions (general and professional users) informed the taxonomy used in the benchmark.
Dataset size: 1150
Dataset metadata available: Yes
Dataset metadata detail: Task category labels (10 types) and composition/constraint annotations.​
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean and pairwise agreement figures;
Metric subscores: Yes
Metric subscores detail: Scores broken down by constraint type and composition type (e.g., Figure 6, Table 22).​
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Pairwise & question‑level agreement tables (Tables 3‑4) show evaluator's reliability.
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following