Title: CriticEval: Evaluating Large-scale Language Model as Critic
Benchmark: CriticEval
Summary: CriticEval is a benchmark that measures the critique ability of large language models (LLMs) along four sub‑skills: feedback, comparison, correction (refinement), and meta‑feedback, across nine diverse task types. It supplies 3.6K human‑vetted items spanning low/medium/high/correct response qualities, provides both scalar and textual critique targets, and offers objective (correlation / accuracy / pass‑rate) and subjective (GPT‑4‑with‑reference) scoring pipelines.
Contribution: - Defines critique ability formally and decomposes it into four separable dimensions.
- Introduces the first large‑scale dataset (3,608 test items, plus dev) with reference critiques, enabling reliable GPT‑4 judging.
- Covers 9 task families (translation, chat, QA, summarization, harmlessness, two maths, two coding) and four response‑quality bands, allowing factor analysis.
- Presents extensive experiments on 35 open‑ and closed‑source LLMs, validating benchmark reliability and revealing scale trends and open‑source progress.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Critique ability of LLMs (identifying, comparing, improving, and judging responses)
Phenomenon defined by authors: Yes
Phenomenon definition: Critique ability is crucial for the self-improvement of LLMs, as it enables the effective analysis and correction of flaws in responses. This capability also facilitates a more robust framework, i.e., scalable oversight, for ensuring the AI systems remain aligned with human-desired outcomes and ethical standards.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a task input and one or two LLM responses, the model must produce a critique: (a) feedback (score + text), (b) comparison (preference + text), (c) refinement, or (d) meta‑feedback on another critique.
Task item definition: A single row consists of {instruction, responses, response_quality_labels, critique_dimension, reference_critique(s)}.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Prompts are sampled from public benchmarks; 70B‑scale LLMs generate diverse‑quality answers; GPT‑4 drafts critiques which humans review & edit.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3,608
Dataset metadata available: Yes
Dataset metadata detail: task type, critique dimension, response quality, error pattern, human scores
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric statistics: Simple mean, Spearman correlation (with p‑value < 0.05)
Metric subscores: Yes
Metric subscores detail: by critique dimension, task type, response‑quality band
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Reliability checked two ways: (1) meta‑feedback correlation of GPT‑4‑with‑reference vs. humans (ρ≈0.63); (2) ablating references drops performance ~13 points, proving reference necessity.
Taxonomy: LLM as a Judge / 
New taxonomy: General Application / LLM as a Judge