Title: MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Benchmark: MT-Bench-101
Summary: MT-Bench-101 proposes a detailed taxonomy of multi-turn chat abilities with an associated benchmark. The overarching abilities are perceptivity, adaptibility, and interactivity. These abilities are broken down into further subskills and 13 total subtasks. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-turn chat capabilities. 
Phenomenon defined by authors: Yes
Phenomenon definition: All below are direct quotes for the sub phenomena: 
Perceptivity requires chatbots to adeptly track and use historical dialogue data to provide logical and consistent responses, encompassing the following three core abilities: Context Memory, Context Understanding, and Context Interference

Adaptability: Chatbots adjust their early responses with the user’s new requirements (Rephrasing), new conditions, and hypotheses (Reasoning) and can correct or insist on answers according to user-challenging feedback (Reflective) in user-triggered dialogue.

Interactivity: Chatbots proactively propose questions to guide the dialogue or gather information for better responses in chatbot-triggered dialogue.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: While the scope of the paper is broad, it does spend a fair amount of energy developing a taxonomy. 
Task definition: The task is to engage in multi-turn dialogue, which is then evaluated by a separate LLM according to the specific task being evaluated. The conversation history comes from a "golden context". 
Task item definition: A specified turn in a chat conversation (e.g., turn 3 in a chat about solving a reasoning problem). 
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The scenario (multi-turn conversation) is relatively ecologically valid. However, all conversations are LLM-generated.
Dataset size: 4208
Dataset size detail: The specified size is the total number of turns, which come from 1388 conversations. 
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Response format detail: Each response is part of a dialogue.
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The models are scored on the worst performing turn on the conversation level.
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: Scores are provided on a subtask level.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Partial pre-analysis.
Author validity detail: The authors specifically constructs a hierarchical taxonomy of abilities that compose the overall phenomena and do fairly large-scale human validation of the scoring. Still, no explicit mention of validity. 
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction