Title: MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
Benchmark: MT-Eval
Summary: Thie paper introduces MT-Eval, a benchmark to evaluate the multiturn conversational abilities of LLMs. Paper's analysis of interactions in LMSYS-Chat1M reveals four predominant patterns when users interact with AI assistants: Recollection, where the assistant must recall information from earlier turns; Expansion, involving the exploration of varied topics within the main subject; Refinement, where initial instructions are clarified
or revised; and Follow-up, consisting of questions based on the assistant’s previous responses. They then construct evaluation sets for each interaction type by augmenting existing datasets or creating new ones to cover real-world applications.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: LLMs' capabilities in multi-turn interactions
Phenomenon defined by authors: No
Phenomenon definition: The ability to perform coherent multi-turn interactions
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Multi-turn conversion (given a context, the model is asked to answer some questions)
Task item definition: A multi-turn query (multiple sentences)
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 168
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Difficulty 
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction