Title: MediQ: Question Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

Benchmark: MediQ
Summary: Medi-Q evaluates LLMs' ability to ask clarifying questions for improved clinical reasoning. It provides expert-authored diagnostic cases and assesses interactive decision-making.

Contribution: Introduces not just the concept of requiring proactive clarifying questions from the model, but also a pipeline to transfer any existing QA benchmark to its structure

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Interactive clinical reasoning, question-asking

Phenomenon defined by authors: Yes
Phenomenon definition: Defined as the ability to proactively ask clarifying questions to gather information for accurate medical decision-making

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a clinical vignette, decide whether to ask a follow-up question, then make a diagnosis.

Task item definition: A clinical case with simple info and a question + "patient system" simulating a patient giving follow-up answers; a correct answer/diagnosis.

Task definition detail: Task involves another LLM cosplaying as a patient volunteering information as demanded by the LLM being evaluated.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Adapted from MedQA and Craft-MD, which are simple QA benchmarks

Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: LLM acts as clinical assistant making iterative decisions and asking follow-up questions. Could probably nit the realism but in comparison to most others it's much closer to real clinical workflows.

Dataset size: 12863
Dataset metadata available: Yes
Dataset metadata detail: Annotated question helpfulness, diagnosis, case difficulty

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Extended interaction (e.g. conversation, calling an API and processing the response)
Response format detail: Open-ended Q&A + final diagnosis

Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: Includes rubric-based expert judgment and GPT-4 scoring

Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Diagnosis accuracy, question helpfulness, reliability

Metric metascoring: majority@k (majority vote over k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Human agreement on annotation; rubric validation; comparison to non-interactive version of benchmark

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine