Title: MINT: EVALUATING LLMS IN MULTI-TURN INTERACTION WITH TOOLS AND LANGUAGE FEEDBACK
Benchmark: MINT
Summary: MINT extends existing benchmark to evaluate the effects of code interpreter usage and multi-turn feedback on LLM performance. It filters benchmark task to ones that benefit from feedback and multi-turn interactions and evaluates different feedback types from "lazy user" to "informative user" and with(out) tools. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning, coding, and decision-making
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Purpose notes: Each high-level phenomena is measured separately 
Task definition: The task is how performance on existing benchmarks (QA) increases when given access to GPT-4 feedback and/or code interpretor.
Task item definition: The tasks come from different benchmarks. Most are in a QA format. 
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: The tasks are sampled from 8 different benchmarks. 
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 586
Dataset metadata available: Yes
Dataset metadata detail: source dataset
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: While the expected result is often a short free response, it can be created through interaction. 
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Provided by number of turns of feedback
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: They do a partial study with actual human feedback on the benchmark tasks.
Human baseline: No
Authors discuss validity: No
Taxonomy: Agents / Coding
New taxonomy: General Application / Coding Agents