Title: Evaluating Large Language Models at Evaluating Instruction Following
Benchmark: LLMBar
Summary: This paper introduces LLMBAR, a benchmark specifically designed to evaluate how well LLM evaluators can assess instruction following in LLM outputs. This benchmark tries to evaluate whether "LLM evaluators" themselves can reliably judge how well models follow instructions.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: to evaluate how well LLM evaluators can assess instruction following in LLM outputs
Phenomenon defined by authors: Yes
Phenomenon definition: "We define it [instruction following] as the ability to correctly parse open-ended instructions and adhere to the specified requirements. This criterion relates to other desirable LLM properties, such as helpfulness."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The task requires the LLM evaluator to pick on out of two outputs that better follows a given instruction (only one is correct).
Task item definition: A single item consists of an instruction, two outputs (one that follows the instruction, one that doesn't) and a label of which is better.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 419
Dataset metadata available: Yes
Dataset metadata detail: type (natural or adversarial, instruction source, creation method
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean and for rating-based evaluations they measure "hedging rate"
Metric subscores: Yes
Metric subscores detail: natural/adverserial subsets
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They use ablation studies on the prompting strategies, they test robustness with challenging cases, they evaluate multiple LLMs, and they should that LLMBar has high human agreement rate compared to other benchmarks.
Taxonomy: Instruction Following / 
New taxonomy: General Capability / Instruction Following