Title: MedJourney: Benchmark and Evaluation of Large Language Models Over Patient Clinical Journey

Benchmark: MedJourney
Summary: MedJourney evaluates LLMs on tasks across the clinical journey of a patient. Its 12 task datasets are partially collected from other benchmarks (7) and partially newly constructed (5).

Contribution: Main contribution is mapping along the "patient journey" stages of planning, access, delivery, and ongoing care.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: clinical reasoning, medical documentation, information extraction

Phenomenon defined by authors: Yes
Phenomenon definition: "The entire workflow of the patient's clinical jounrey"

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each task is a question that simulates a step in a patient's clinical journey, eg. medical note summarization, medication reconciliation, and discharge instructions

Task item definition: Each item consists of a patient vignette (eg. past medical history, current symptoms, labs) and a task prompt (eg. generate discharge summary, answer diagnostic question).

Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Cases were built using real EMR note templates and enhanced using LLMs to simulate  patient records.

Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Tasks mimic actual medical documentation and reasoning tasks faced by clinicians

Dataset size: 120
Dataset metadata available: Yes
Dataset metadata detail: Task type, patient case ID, input note type, instruction prompt, response

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Response format detail: Freeform answers expected in medical prose; some tasks ask for structured lists or summaries. 

Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: Tasks are evaluated via GPT-4 ratings on helpfulness, correctness, and completeness; sometimes also with rubric-based human annotation.

Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Per-task performance (e.g., medication reconciliation, discharge summary) along the patient journey

Metric metascoring: consensus@k (plurality vote over k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors validate task realism by using clinical templates and evaluating with scoring emulating clinicians

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine