Title: Large Language Models are Few-Shot Clinical Information Extractors
Summary: Datasets for benchmarking LLMs on five different clinical NLP tasks: clinical sense disambiguation, biomedical evidence extraction, coreference resolution, medication status extraction, and medication attribute extraction. They show that LLMs perform well on these tasks despite not being specifically trained for the clinical domain.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: clinical information extraction
Phenomenon defined by authors: Yes
Phenomenon definition: Clinical information extraction as "the extraction of important variables trapped in clinical notes" and structuring clinical variables from unstructured text, (handling ambiguous jargon and nonstandard phrasal structure specific to clinical text).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Tasks in clinical information extraction: 1) clinical sense disambiguation, 2) biomedical evidence extraction, 3) coreference resolution, 4) medication status extraction, and 5) medication attribute extraction.
Task item definition: For example, in clinical sense disambiguation, an item consists of a clinical note and an abbreviation to be expanded.
Task source: Real task examples (e.g. GitHub issues), Expert-crafted task examples (e.g. hand-written examples)
Task source detail: Clinical text from primarily the CASI dataset (Clinical Acronym Sense Inventory) and creates new annotations on this text.
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset metadata available: Yes
Dataset metadata detail: medication type, status, relation types, antecedent-pronoun pairs
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Primary metrics used are F1 scores, accuracy, recall, and precision. Partially, micro and macro averages are reported.
Metric subscores: Yes
Metric subscores detail:  Scores for each of the five tasks. 
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Extraction
Alternate taxonomy: Medicine
New taxonomy: General Application / Extraction