Title: FOFO: A Benchmark to Evaluate LLMs’ Format-Following Capability
Benchmark: FOFO
Summary: FOFO Is a benchmark for domain-specific format following capabilities. It evaluates a wide array of domains and subdomains across a diverse set of formats from specific medical forms to Maple. The specific examples are generated using GPT-4 and human validation.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Format following
Phenomenon defined by authors: Yes
Phenomenon definition: "precise adherence to specified formats given by humans"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task is to generate dummy data in a specified format defined by detailed instructions within a given domain. 
Task item definition: A single formatting instruction with a domain (e.g., Manufacturing), a subdomain (e.g., Optimization), and a format (e.g., "Standard Operating Procedures") with an example of the format. 
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: While following formatting instructions is real, the data is still dummy. 
Dataset size: 494
Dataset metadata available: Yes
Dataset metadata detail: domain,subdomain,format
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following