Title: SM3: Synthetic Multi-modal Medical Text-to-Query Benchmark

Benchmark: SM3
Summary: SM3 is a synthetic benchmark for evaluating medical retrieval capabilities in LLMs. It focuses on converting diverse multimodal medical inputs into structured queries.

Contribution: Notable is the creation of 4 synthetic databases of different formats (postgresql, mongodb, neo4j, graphdb) based on the same synthetic patient data, which in turn was generated in a year-long collection process in university hospitals

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: retrieval, structured query generation

Phenomenon defined by authors: Yes
Phenomenon definition: "Medical retrieval" is framed as a model’s ability to extract structured information from multimodal clinical data in order to generate an effective search query

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task requires a model to produce a structured query based on a scenario composed of a question, medical report, and image caption.

Task item definition: Each item consists of a clinical report and question; the model must generate a structured query with fields like disease, modality, age, gender.

Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Authors crafted 4 synthetic databases of different formats (postgresql, mongodb, neo4j, graphdb) based on the same synthetic patient data & LLM-assisted variations on 400 question templates about those databases

Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The query generation is not done in real workflows but is a proxy task designed to assess medical retrieval readiness

Dataset size: 10000
Dataset metadata available: Yes
Dataset metadata detail: scenario ID, report, question, structured query (fields: disease, modality, etc.)

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: Query

Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Per-category and per-database; with and without schema; one- and 5-shot

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Human evaluation for hallucinations; comparison to real-world medical datasets

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine