DOI: 10.5281/zenodo.15664865
title: MultiSynDS: Multilingual Synthetic Discharge Summaries Generated from Heart Failure Clinical Cases

description: Description

This dataset comprises 1 000 cardiology clinical case reports and their corresponding synthetic discharge summaries, generated automatically in English and translated into Spanish and Dutch. Original cases derive from expert-authored narratives; summaries were produced with GPT-4o-mini via zero-shot prompting, then translated into target languages. The dataset is intended for training and evaluating clinical NLP systems - particularly for named entity recognition, summarization, and cross-lingual transfer.

Dataset Composition




Language
Number of Documents
Avg. Words per Summary
Avg. Sentences


English
1,000
 540
 20


Spanish
1,000
TBD
TBD


Dutch
1,000
TBD
TBD




Data Collection & Generation



Source Cases: 1 000 anonymized cardiology case reports from AI4HF project.  

Summary Generation:



Model: GPT-4o-mini (zero-shot prompt) 

Prompt (English): Rewrite this text and structure it as a discharge summary without translation: [Clinical case text]

Prompt (Other languages):  Generate English discharge summary using the original prompt. Translate the following medical discharge summary to <lang>:[result]






Folder Structure


- original: Contains the original clinical cases in the txt subfolder and their  corresponding MeSH terms in the metadata subfolder.

- en: Discharge summaries in English generated using GPT-4o-mini.

- es: Discharge summaries in Spanish generated and transllated using GPT-4o-mini. It also includes the intermediate step, i.e., the discharge summaries in English before translation.

- nl: Discharge summaries in Dutch generated using GPT-4o-mini.


English Evaluation Metrics

Automatic (Concept-Level)

The automatic metric is a concept‐level F1 that quantifies how faithfully a generated summary preserves the medical concepts found in the original case. First, a clinical NER model (CardioNER.en) is applied to both the reference case and the generated discharge summary to extract mentions of diseases, symptoms, procedures and medications; each mention is then normalized to SNOMED CT concept identifiers via SapBERT embeddings and nearest‐neighbor lookup. For each mention, the top 10 SNOMED CT candidates are retrieved and a "positive match" is counted only when all 10 concept IDs overlap between reference and summary mentions (intersection threshold τ=10).

Across N documents, let C(n) be the set of reference concepts in document n, and Ĉ(n) the set from the summary. Concept precision and recall are then computed as:



Concept Recall    = ∑ₙ |Ĉ(n) ∩ C(n)|  / ∑ₙ |C(n)| 

Concept Precision = ∑ₙ |Ĉ(n) ∩ C(n)|  / ∑ₙ |Ĉ(n)|


 

The Concept F1 score is simply the harmonic mean of these two, providing a single, corpus‐level measure of concept coverage and fidelity.



Concept Precision: 0.48  

Concept Recall: 0.56  

Concept F1: 0.50 ± 0.1

By Entity Type (F1):



Medication: 0.78  

Disease: 0.55

Procedure: 0.45

Symptom: 0.45




Human (2 Clinicians, n = 30 cases)




Dimension
Mean Score (out of 5)
κ_w


Medical Entities Completeness
4.09
0.740


Structure - Headers
4.49
0.126


Structure - Content
4.02
0.607


Content Accuracy
4.21
0.696


Made-up Content (Hallucinations)
4.25
0.149


Overall Quality
3.63
0.419




Average of annotators scores for each dimension. Inter-annotator agreement is represented with weighted Cohen's Kappa (κ_w).

Spanish and Dutch Evaluations

Quantitative evaluation metrics for the Spanish and Dutch translations are currently under development and will be released in a future version of this dataset.

Maintenance & Contact

For questions, issue reporting or contribution guidelines, please refer to the Zenodo repository page or contact: Alberto Becerra Tomé (alberto.becerra.tome@gmail.com), Martin Krallinger (krallingermartin@gmail.com).

This work is part of the project I+D+i TED2021-129974B-C22, financed by MICIU/AEI/10.13039/501100011033 and the European Union's NextGenerationEU/PRTR.