Title: FollowBench: A Multi-level Fine-grained Constraints Following
Benchmark for Large Language Models
Benchmark: FollowBench
Summary: The paper presents a benchmark called FollowBench for multi fine-grained constraint following evaluations. It asses five different constraint types (e.g. content, situation, style, format and example). The paper evaluated 13 LLMs with FollowBench which highlights weaknesses in LLMs instruction following capabilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: instruction following
Phenomenon defined by authors: Yes
Phenomenon definition: "To precisely estimate the difficulty degree to which LLMs can follow instructions"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: The task is to generate responses that satisfy all the constraints specified in the given instructions. The model must interpret multiple fine-grained constraints and produce an output that follows every constraint simultaneously. 
Task item definition:  an instruction with multiple constraints (ranging from 1 to 5)
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 820
Dataset metadata available: Yes
Dataset metadata detail: difficulty (L1-L5 based on the number of contraints)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Difficulty (L1-L5)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They have human expert annotators to assess LLM-as-a-Judge performance and they do a diversity analysis to ensure the comprehensiveness of the benchmark. 
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following