Title: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Benchmark: MT‑BenchMT‑Bench
Summary: The paper introduces MT‑Bench, a set of 80 hand‑crafted multi‑turn prompts, and Chatbot Arena, a crowdsourced head‑to‑head battle platform, to evaluate LLM assistants on open‑ended dialogue. It shows that strong LLMs such as GPT‑4 can act as reliable automatic judges of other models’ answers, matching human preferences with around 80 % agreement, and discusses biases and mitigation strategies for the “LLM‑as‑a‑judge” paradigm.


Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLM alignment with human preferences in open-ended tasks and multi-turn dialogues. Besides that, MT‑Bench covers 8 categories (writing, role‑play, extraction, reasoning, maths, coding, STEM knowledge, humanities knowledge).
Phenomenon defined by authors: Yes
Phenomenon definition: The paper states that helpfulness is judged along "helpfulness, relevance, accuracy, depth, creativity, and level of detail" and defines success as matching human preference.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: Paper mentions evaluating different aspects such as multi-turn conversational and instruction-following ability, as well as core capabilities like reasoning and math, and suggests that within helpfulness, there are multiple dimensions like accuracy, relevance, and creativity that could be analysed separately.
Task definition: The task involves evaluating the quality of responses provided by AI assistants to user questions, either through pairwise comparison or single-answer grading.
Task item definition: One dataset row = a two‑turn user conversation prompt (MT‑Bench) or a single crowd‑posed prompt (Arena) to which models must respond.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Chatbot Arena collects real users’ interactive votes but in an experimental web interface.
Dataset size: 80
Dataset size detail: MT‑Bench: 80 prompts. Chatbot Arena: around 30 000 crowd‑sourced conversations with preference votes (size varies as the platform runs). "After running Chatbot Arena for one month, we have collected around 30K votes."
Dataset metadata available: Yes
Dataset metadata detail: Category (e.g., writing, math, knowledge)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The primary evaluation metric is human preference, assessed through human ratings on MT-bench and crowdsourced votes on Chatbot Arena. Paper explores using LLMs to approximate these human preferences, employing pairwise comparison and single-answer grading. Exact match is used for specific cases like math questions where a correct answer is known. 
Metric aggregation: Simple Mean
Metric statistics: Simple mean scores, simple sum scores, inter‑annotator agreement percentages.
Metric subscores: Yes
Metric subscores detail: Results broken down by the eight prompt categories and by model pairs.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Paper analyses position, verbosity, and self‑enhancement biases, providing mitigation (swap order, reference answers) and reports failures.
Taxonomy: Alignment / Alignment
Alternate taxonomy: User Interaction
New taxonomy: General Capability / Alignment