Title: Revisiting De-Identification of Electronic Medical Records: Evaluation of Within- and Cross-Hospital Generalization

Summary: Benchmark for de-identification of protected health information (PHI) in Chinese electronic medical records, with a focus on cross-hospital generalization. Constructs a multi-hospital dataset and evaluates various models and domain generalization (DG) techniques to assess performance under domain shift.

Contribution: Pre-LLMs so uses the dataset - which is still a valid benchmark - as training data for a CNN and a BERT fine-tuning run.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Generalization on de-identification tasks

Phenomenon defined by authors: Yes
Phenomenon definition: Anonymization

Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Detect and remove personal health information mentions (e.g., names, locations, dates) in clinical records from three Chinese hospitals.

Task item definition: Each item is a sentence or span from an electronic medical record, with relevant tokens labeled using tags corresponding to personal data categories, but the labels hidden. The task is to recreate these token labels.

Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: Data collected from three Chinese hospitals - no idea how that got around data protection laws even with anonymization; they must be quite loose there - and hand-annotated
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: De-identification is a required step for real-world medical data sharing and lots of other data sharing contexts.

Dataset size: 500
Dataset size notes: 400
Dataset metadata available: Yes
Dataset metadata detail: Sentence and mention counts, health information category counts per dataset

Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Response format detail: Sequence output with personal data tagged
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Predicted span of tags must exactly match correct span and category

Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Per PII category (e.g., PERSON, DATE, ID)

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine