Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation

## Abstract

Abstract Radiology reports convey detailed clinical observations and capture diagnostic reasoning that evolves over time. However, existing evaluation methods are limited to single-report settings and rely on coarse metrics that fail to capture fine-grained clinical semantics and temporal dependencies. We introduce Lunguage , a benchmark dataset for structured radiology report generation that supports both single-report evaluation and longitudinal patient-level assessment across multiple studies.
It contains 1,473 annotated chest X-ray reports, each reviewed by experts, and 80 of them contain longitudinal annotations to capture disease progression and inter-study intervals, also reviewed by experts. Using this benchmark, we develop a two-stage framework that transforms generated reports into fine-grained, schema-aligned structured representations, enabling longitudinal interpretation. We also propose LunguageScore , an interpretable metric that compares structured outputs at the entity, relation, and attribute level while modeling temporal consistency across patient timelines. These contributions establish the first benchmark dataset, structuring framework, and evaluation metric for sequential radiology reporting, with empirical results demonstrating that LunguageScore effectively supports structured report evaluation. The code is available at: https://github.com/SuperSupermoon/Lunguage

## 1 Introduction

Radiology reports play a critical role in medical diagnosis by capturing the patient’s clinical history, describing imaging findings, recording procedural steps, and noting changes over time. These reports are typically written in unstructured free-text, leading to significant variation in terminology and level of detail across radiologists. This heterogeneity complicates consistent computational interpretation and limits the development of accurate, automated systems for report generation and evaluation.

To address these challenges, structured reporting frameworks have been developed to convert free-text reports into standardized, machine-friendly formats Jain et al. (2021); Khanna et al. (2023); Wu et al. (2021); Zhang et al. (2023); Zhao et al. (2024). These representations make clinical content explicit and structured, enabling consistent and automated evaluation. While such frameworks have improved representational consistency, current evaluation methods remain fundamentally limited in two key aspects: temporal reasoning and fine-grained clinical accuracy.

Temporal reasoning is central to radiologic interpretation, as diagnoses frequently rely on comparing current and prior studies to assess whether a finding has progressed, remained stable, or newly appeared. However, most evaluation protocols Bannur et al. (2024); Huang et al. (2024); Jain et al. (2021); Khanna et al. (2023); Ostmeier et al. (2024); Smit et al. (2020); Wu et al. (2021); Yu et al. (2023a); Zhang et al. (2023); Zhao et al. (2024) assess each report in isolation, without incorporating previous findings. This makes it impossible to determine whether temporal expressions—such as “no change,” “improved,” or “new”—are appropriate. For instance, the statement “no change in pneumonia” cannot be meaningfully evaluated without confirming whether pneumonia was present in prior studies.

Fine-grained clinical accuracy is equally essential. Reliable interpretation requires preservation of detailed attributes such as precise location (e.g., “carina above 3cm”) and lesion size (e.g., “2.5 cm”). These attributes are critical for diagnostic specificity and downstream clinical decisions, yet most evaluation protocols reduce such detail. For example, the phrase “2.5 cm right upper lobe nodule with spiculated margins” may be flattened to “nodule”. The loss of granularity makes it difficult to distinguish precise from incomplete outputs.

Structured representation frameworks have partially addressed these issues by extracting clinical entities and relations from radiology reports Jain et al. (2021); Khanna et al. (2023); Wu et al. (2021); Zhang et al. (2023); Zhao et al. (2024). Some include temporal descriptors like “worsened” or “stable” Khanna et al. (2023); Wu et al. (2021). However, all remain limited to single reports and rely on explicitly stated temporal expressions, without checking consistency over time. As a result, they cannot determine whether findings align with prior studies or reflect coherent clinical trajectories. In addition, while these schemas partially improve structural representation, they often lack the clinical granularity needed for detailed diagnostic interpretation.

Recent report generation models have begun incorporating temporal inputs such as prior reports, imaging, or clinical indications Bannur et al. (2024); Zhou et al. (2024), enabling outputs that are more context-aware and temporally coherent. However, evaluation methods have not kept pace. Generated reports continue to be interpreted at separate timepoints rather than across a continuous timeline, making it difficult to assess whether models appropriately incorporated prior findings or preserved clinically important details, both in temporal and semantic dimensions.

To address these limitations, we make the following contributions. (1) We construct Lunguage, a fine-grained benchmark dataset for single and sequential structured reports. 1,473 single reports from 230 patients are annotated with 17,949 expert-validated entities and 23,307 relation–attribute pairs spanning 18 clinically grounded relation types. 80 sequential reports from 10 patients are annotated by comparing all possible observation pairs (41,122 pairs) across 3 to 14 reports per patient (1 to 1,200 days apart). These capture diagnostic reasoning through EntityGroups (identifying the same observation across multiple sequences) and TemporalGroups (grouping observations within entity groups based on their temporal relationships across studies) for longitudinal analysis. (2) Second, we develop a LLM-based extraction framework to convert free-text reports into structured format. The framework structures radiology reports into entity-relation-attribute triplets, and links them across time to form temporally coherent interpretations following the annotation schema of Lunguage. The framework demonstrates a strong agreement with human annotations, achieving an F1 score of 0.94 for entity-relation extraction, 0.86 for full triplets, 0.68 for EntityGroup, and 0.89 for TemporalGroup.
(3) Finally, we introduce LunguageScore, a clinically grounded metric quantifying both diagnostic accuracy and temporal coherence. It compares structured representations from generated reports against references, enabling assessment of clinical details and evolving diagnostic context. Our evaluation uses gold-standard structured data, but LunguageScore can extend to "silver standard" evaluation by automatically structuring both generated and reference reports when gold-standard annotations are unavailable.

Figure: Figure 1: Schema for Single and Sequential Report Structuring. The figure shows two reports from the same patient at day 10 and day 90. For the single report schema (within each report), gray solid lines connect entities to attributes, while pink and blue solid lines represent inter-entity reasoning relations (Associate, Evidence). For the sequential schema (across reports), black solid lines denote entities in the same EntityGroup (same clinical finding over time) and TemporalGroup (same diagnostic episodes), while black dashed lines show entities in the same EntityGroup but different TemporalGroups (different diagnostic episodes).
Refer to caption: /html/2505.21190/assets/figure/schema.png

## 2 Related work

##### Structuring Radiology Reports

Radiology reports encode layered clinical semantics, spanning history, imaging observations, and diagnostic impressions. Rule-based systems Wu et al. (2021); Zhang et al. (2023) achieve high precision in constrained scenarios but often struggle to generalize due to the variability of clinical language. Supervised methods Jain et al. (2021); Khanna et al. (2023); Zhao et al. (2024) using transformer-based models offer flexibility, though their effectiveness depends on the coverage and granularity of the annotation schema. More recently, prompting-based approaches have leveraged large language models (LLMs), such as GPT-4 Achiam et al. (2023) and open-source variants Touvron et al. (2023); Zheng et al. (2025), to produce structured outputs directly from free-text inputs Busch et al. (2024); Dorfner et al. (2024); Hartsock et al. (2025); Woźnicki et al. (2024). While these models exhibit strong few-shot capabilities, they may introduce issues such as hallucination, inconsistent terminology, and sensitivity to prompt design. To mitigate such variability, we incorporate a task-specific vocabulary and schema-aligned reference set to constrain output to valid clinical concepts and enhance consistency through retrieval-augmented prompting.

##### Evaluation Metrics for Radiology Report Understanding

Existing metrics fall into two main categories: lexical and model-based. Lexical metrics such as BLEU Papineni et al. (2002a), ROUGE Lin (2004), and METEOR Banerjee and Lavie (2005) rely on surface overlap and often miss clinical meaning. Model-based metrics like CheXbert Smit et al. (2020) and BERTScore Zhang et al. (2019) assess high-level similarity but lack fine-grained detail. Structure-based metrics such as RadGraph F1 Jain et al. (2021) and RaTEScore Zhao et al. (2024) improve granularity by matching clinical entities and relations. Recent work has emphasized clinical error detection. ReXVal Yu et al. (2023b) introduced expert-labeled errors, which informed RadCliQ Yu et al. (2023a), combining BERTScore and RadGraph F1 for joint lexical and semantic evaluation. LLM-based metrics like GREEN Ostmeier et al. (2024), FineRadScore Huang et al. (2024), RadFact Bannur et al. (2024) and CheXprompt Zambrano Chaves et al. (2025) further approximate expert judgments or factual correctness. However, most metrics evaluate single reports and overlook temporal consistency across exams. They also miss fine-level attributes like location, extent, or progression. In contrast, our framework supports structured, temporally aligned evaluation over patient report sequences, enabling clinically meaningful assessment across all three dimensions: semantic, structural, and temporal.

## 3 Lunguage : A benchmark for single and sequential structured reporting

We propose two complementary annotation schemas for structured understanding of radiology reports: a single-report schema capturing fine-grained interpretation within individual reports, and a sequential schema modeling patient-level diagnostic trajectories across time. Both schemas were refined with four board-certified radiologists to ensure clinical validity. Figure [1](#S1.F1) illustrates these schemas.

### 3.1 Single Structured Report: Schema and Annotation Process

We propose a schema that captures the internal structure of single reports by extracting clinically relevant information as typed entities and relations. It is designed to reflect the typical subsections of radiology reports—indication/history, findings, and impression—and supports relation extraction across sentence boundaries within each section. Notably, the indication/history section is included to preserve contextual information that influences diagnostic interpretation at the patient trajectory level.

Entities are assigned to one of six clinically grounded categories based on their derivability from chest X-ray imaging: PF (Perceptual Findings) for directly observable image features (e.g., “lung,” “opacity”); CF (Contextual Findings) for diagnoses inferred from external clinical context (e.g., “pneumonia”); OTH (Other Objects) for mentioned devices or procedures (e.g., “ET tube”); COF (Clinical Objective Findings) for structured observations from non-imaging sources (e.g., lab tests); NCD (Non-CXR Diagnosis) for diagnoses based on other modalities (e.g., “AIDS”); and PATIENT INFO for reported history or symptoms (e.g., “fever,” “cough”).

Relations capture clinical properties and inter-entity connections, often spanning multiple sentences. The schema includes diagnostic stance (DxStatus, DxCertainty); spatial and descriptive characteristics (Location, Morphology, Distribution, Measurement, Severity, Comparison); temporal dynamics (Onset, Improved, Worsened, NoChange, Placement); and contextual information (PastHx, OtherSource, AssessmentLimitations)(^1^11Abbreviations: “Dx” stands for “diagnosis” and is used in relations such as DxStatus (i.e., positive or negative finding) and DxCertainty (i.e., definitive or tentative). “Hx” in PastHx stands for “history”.). It also includes two reasoning relations: Associate (bidirectional links between related entities) and Evidence (asymmetric support from a finding to a diagnosis). For example, in “left lung opacity suggests pneumonia,” the schema identifies both Associate between opacity and pneumonia, and Evidence indicating that pneumonia is inferred from opacity. Full definitions can be found in Appendix [A.1](#A1.SS1).

##### Single Report Annotation Process

We developed a structured annotation pipeline for 1,473 reports from 230 patients in the MIMIC-CXR Johnson et al. (2019) test split to support fine-grained and clinically grounded structuring of radiology language. The pipeline comprised two stages: constructing a task-specific vocabulary and generating gold-standard structured reports (SRs), both guided by a schema representing the layered semantics of chest X-ray (CXR) reports. In the first stage, we used GPT-4 (0613)(^2^22All large language model (LLM) usage, including GPT-4, was conducted using HIPAA-compliant deployments provided by Azure and Fireworks AI.) to generate initial SRs from raw reports using schema-driven prompts. From these outputs, we extracted entity and relation attributes to build an initial vocabulary, categorized by relation type. This vocabulary was refined through systematic review by four radiologists, ensuring lexical clarity and clinical validity. The final vocabulary comprised 1,808 unique entity terms and 2,193 relation attributes, each mapped to a subcategory and, when applicable, a UMLS concept Bodenreider (2004). In the second stage, annotators manually revised the model-generated SRs using the curated vocabulary. Annotators manually reviewed all 1,473 reports section by section, with the workload equally divided among radiologists to verify every (entity, relation, attribute) triplet. This included both entity–attribute pairings and inter-entity relations, with particular attention to cross-sentence links such as Associate and Evidence. This comprehensive process yielded 17,949 entity instances and 23,307 relation instances, forming a high-quality dataset for benchmarking fine-grained information extraction and report structuring. Details of the vocabulary and annotation process are provided in Appendix [A.1.2](#A1.SS1.SSS2).

### 3.2 Sequential Structured Report: Schema and Annotation Process

Longitudinal radiology reports often exhibit lexical variation, abstraction shifts, and inconsistent phrasingMeystre et al. (2008); Wang et al. (2018). The same pathology may be described differently over time (e.g., "right opacity" vs. "focal consolidation"), complicating semantic alignment and temporal reasoning. To address this, we introduce a schema that structures reports across patient timelines through two key components:

EntityGroups identify observations that refer to the same underlying clinical finding, even when expressed using different terms, anatomical references, or levels of abstraction. Within each patient, all observation pairs are compared to detect semantic equivalence, regardless of when they appear in the timeline, whether the finding is reported as present or absent (DxStatus), or whether it is stated definitively or tentatively (DxCertainty). For example, “PICC line tip in lower SVC” and “at the cavoatrial junction” (Figure [1](#S1.F1)) may describe the same catheter tip location, reflecting inherent ambiguity in 2D imaging. Similarly, “lung volumes” reported as low on day 10 and described as “no change” on day 90 can be grouped to indicate persistent low lung volume.

TemporalGroups divide each EntityGroup into distinct diagnostic episodes based on temporal distance, shifts in status or certainty, and explicit expressions of clinical change (e.g., “worsening,” “resolved”). This approach captures clinically meaningful transitions in a patient’s condition Chapman et al. (2011); Savova et al. (2010). For example, “fever” mentioned in both the day 10 and day 90 reports (Figure [1](#S1.F1)) appears in the “history” section but occurs far apart in time; treating them as part of separate temporal groups better reflects clinical reasoning. Together, these components support fine-grained evaluation of both semantic consistency and temporal coherence in longitudinal model outputs.

##### Sequential Report Annotation Process

We annotated 80 chest X-ray reports from 10 patients among the 230 patient cohort used in the single-report annotation, to create a gold dataset for longitudinal evaluation. The same four physicians from the earlier phase participated in the annotation process, with patients equally divided among them. Each physician independently annotated their assigned patients’ reports in chronological order, identifying observations referring to the same underlying finding (EntityGroup, represented as linearized phrases combining entity and its attributes, e.g., "pleural effusion right lung increasing") and grouping them into diagnostic episodes (TemporalGroup, numbered sequentially as 1, 2, 3, etc. to distinguish separate temporal progressions) based on clinical and temporal continuity. Terminology was normalized when appropriate (e.g., aligning "right clavicle hardware" and "orthopedic side plate"), while preserving distinctions in abstraction and anatomical specificity. This process required significant effort due to the complexity of longitudinal comparison. Patients had between 3 and 14 reports, with time intervals ranging from 1 to 1,200 days. For each patient, all observation pairs—ranging from 34 to 141 per case—were compared one by one, resulting in 41,122 total comparisons. Each pair was assessed to determine whether the two observations referred to the same clinical finding, considering both meaning and timing. This detailed review was necessary to capture both consistent findings across time and clinically meaningful transitions such as resolution or recurrence. Details are provided in Appendix [A.2](#A1.SS2).

## 4 Structuring Framework for Single and Sequential Reports

We develop a two-stage framework for automatically structuring radiology reports using the same schema as our gold-standard benchmark, covering both single-report and longitudinal settings. The framework produces structured representations suitable for downstream evaluation along semantic, structural, and temporal dimensions. The framework overview can be found in Appendix [B.1](#A2.SS1)

##### (i) Single setting

To generate accurate structures from free-text, we apply corpus-guided relation extraction using a large language model (LLM). The model extracts (entity, relation, attribute) triplets aligned with our schema. While LLMs offer flexible language understanding, they can produce hallucinations and inconsistencies Busch et al. (2024); Dorfner et al. (2024); Hartsock et al. (2025); Woźnicki et al. (2024). To mitigate this, we guide the model by matching sentences against a curated vocabulary from our annotation corpus (Section [3.1](#S3.SS1)). The task spans both intra- and inter-sentential contexts, extracting triplets without templates to handle lexical variation. Prompt details and vocabulary-matching algorithm are in Appendix [B.2](#A2.SS2) and [B.3](#A2.SS3).

##### (ii) Sequential setting

Building on the structured outputs from stage (i), we use the LLM to interpret report sequences over time. To address longitudinal variability, the model performs normalization and temporal aggregation across reports. Specifically, we linearize each entity and its related attributes into flattened text, preserving their chronological order relative to the initial study (e.g., "day 0: opacity right lung", "day 30: opacity right basilar").
The LLM is provided with few-shot examples illustrating common patterns of lexical variation, abstraction shifts (e.g., descriptive to diagnostic terms), and rephrased mentions of persistent devices. Using these examples as guidance, the model then determines whether observations across time refer to the same underlying finding and whether they belong to a single temporal group. This decision is guided by semantic similarity, anatomical alignment, and temporal continuity, which is inferred by the LLM. When observations reflect recurrence after resolution or appear clinically disconnected, they are treated as distinct temporal groups.
This process generates two-fold outputs: Entity Groups and Temporal Groups, corresponding to the same concepts introduced in Section [3.2](#S3.SS2). The output format combines entity, location, and temporal pattern (e.g., "pleural effusion right lung no change") with temporal groups numbered sequentially (1, 2, 3, etc.) following the sequential schema established in Section [3.2](#S3.SS2). This approach enables faithful structuring of longitudinal narratives, capturing meaningful trajectories across diverse report sequences. Full prompt examples are provided in Appendix [B.4](#A2.SS4).

## 5 LunguageScore : A Fine-Grained Patient-Level Metric

We propose LunguageScore, a fine-grained metric that quantifies radiology report quality across semantic equivalence, temporal coherence, and attribute-level similarity. LunguageScore captures clinically meaningful distinctions in terminology ("right clavicle hardware" vs. "orthopedic side plate"), longitudinal trends (resolution vs. decrease), and detailed attributes such as size (2.3 cm vs. 3.0 cm). It integrates these dimensions into a single similarity score that contrasts the (sequence of) candidate report(s) against the (sequence of) reference report(s), enabling patient-level evaluation.

##### Evaluation Principles.

LunguageScore is grounded in three clinical principles: semantic sensitivity captures concept-level equivalence across linguistic variation Meystre et al. (2008); Wang et al. (2018); temporal coherence ensures alignment with clinical timelines for assessing disease progression Chapman et al. (2011); Savova et al. (2010); and structural granularity evaluates fine-grained attributes critical for diagnosis Demner-Fushman et al. (2009); Pons et al. (2016). These principles enable clinically faithful evaluation suitable for real-world deployment.

##### Evaluation Method.

Each patient is associated with a sequence of $T$ structured reports. The metric operates at the patient level and supports both single-report ($T=1$) and sequential-report ($T>1$) evaluations. In the single-report setting, evaluation is based on semantic and structural alignment, while in the sequential-report setting, temporal alignment is additionally incorporated to assess consistency across longitudinal disease trajectories. Formally, LunguageScore evaluates similarity between predicted and gold reference sets of structured report findings as follows.

For each patient, we compare all predicted and gold reference findings across the entire sequence of reports. Let $\mathcal{S}^{\text{pred}}={(S_{1}^{\text{pred}},\dots,S_{T}^{\text{pred}})}$ and $\mathcal{S}^{\text{gold}}={(S_{1}^{\text{gold}},\dots,S_{T}^{\text{gold}})}$ denote the predicted and gold sequences for a given patient, where each $S_{t}^{(\cdot)}$ is the set of all structured findings at the t-th study. Pairwise similarity is computed over every possible pair of findings, pooled across all timepoints:

$$ $(f^{\text{pred}},f^{\text{gold}})\in\left(\bigcup_{t_{p}=1}^{T}S_{t_{p}}^{\text{pred}}\right)\times\left(\bigcup_{t_{g}=1}^{T}S_{t_{g}}^{\text{gold}}\right).$ (1) $$

Each pair of findings is assigned a composite similarity score that captures alignment across semantic, temporal, and structural similarity dimensions, as defined below:

$$ $\text{MatchScore}(f^{\text{pred}},f^{\text{gold}})=\text{Semantic}\cdot(\text{Temporal if }T>1)\cdot\text{Structural}\,.$ (2) $$

Semantic similarity determines whether two findings express the same underlying clinical concept. For semantic representation, we use different approaches for single versus sequential reports: in the single-report setting ($T=1$), each finding is simply represented as a linearized phrase derived from the entity and all its associated attributes (e.g., "opacity"-"left lung"-"nodular"-"slightly increased"). However, in the sequential-report setting ($T>1$), where findings need to be tracked across time, we utilize the EntityGroup (see Section [4](#S4)) for representation. This approach allows lexically divergent but conceptually identical findings to be treated as semantically aligned across multiple reports.
Cosine similarity is computed between contextual embeddings of these semantic representations using domain-specific clinical BERT models (MedCPT Jin et al. (2023) and BioLORD Remy et al. (2024)) chosen for their ability to capture semantic variability in chest X-ray reports. We use the average of cosine similarities computed from both models to improve robustness. Model selection details are provided in Appendix [C.3](#A3.SS3).

$$ $\text{Semantic}(f^{\text{pred}},f^{\text{gold}})=\text{cosine}(\text{Embed}(f^{\text{pred}}),\text{Embed}(f^{\text{gold}}))$ (3) $$

Temporal similarity is defined only when $T>1$ and captures alignment across timepoints.
It ensures that findings are not only semantically similar but also temporally coherent with the patient’s disease progression.
To prevent matches across unrelated timepoints, LunguageScore prioritizes findings that occur in the same study timepoint $t$ and TemporalGroup.
Temporal alignment receives the maximum score (= 1) when both study timepoint $t$ and TemporalGroup match, and a reduced score when only one matches, for example, when a predicted finding belongs to the correct TemporalGroup but appears in a different study.
Final scores are computed using equal weights:

$$ $\text{Temporal}(f^{\text{pred}},f^{\text{gold}})=w_{S}\cdot\mathbf{1}[\text{S}(f^{\text{pred}})=\text{S}(f^{\text{gold}})]+w_{G}\cdot\mathbf{1}[\text{G}(f^{\text{pred}})=\text{G}(f^{\text{gold}})]\,.$ (4) $$

where S refers to the study timepoint $t$, G refers to the TemporalGroup of findings across time, and equal weights ($w_{S}=w_{G}=0.5$) are used in our implementation.

Structural similarity evaluates individual attributes (e.g. Location, Measurement…) between predicted and gold reference findings, enabling fine-grained comparison. Each attribute is assigned a normalized weight $w_{\text{attribute}}$ based on its clinical importance, as determined by experts, reflecting its role in decision making (see Appendix [C.1](#A3.T1)). Similarity is computed as:

$$ $\text{Structural}(f^{\text{pred}},f^{\text{gold}})=\sum_{\text{attribute}}w_{\text{attribute}}\cdot\text{sim}(f^{\text{pred}}[\text{attribute}],f^{\text{gold}}[\text{attribute}])\,,$ (5) $$

where $\text{sim}(\cdot)$ returns 1 for exact matches on binary attributes(^3^33Binary attributes: DxStatus (positive/negative) and DxCertainty (definitive/tentative)) and cosine similarity for non-binary attributes(^4^44Non-binary attributes include: Location, Severity, Onset, Improved, Worsened, Placement, NoChange, Morphology, Distribution, Measurement, Comparison, PastHx, OtherSource, AssessmentLimitations) using the average of MedCPT and BioLORD contextual encoders. This ensures that evaluation captures both overall correctness and clinically critical attribute accuracy.

##### Set-level matching with partial credit.

We can compute the combined MatchScore by multiplying semantic, temporal, and structural similarity scores (Equations [3](#S5.E3)-[5](#S5.E5)), as shown in Equation [2](#S5.E2). We then perform optimal bipartite matching between predicted findings $i$ and gold reference findings $j$ using MatchScore $s_{ij}$
as edge weights, giving us sets of matched pairs $\{(f^{(pred)}_{m},f^{(gold)}_{n})\}$, unmatched predicted findings $\{f^{(pred)}_{u}\}$, and unmatched gold reference findings $\{f^{(gold)}_{v}\}$. Matched pairs contribute similarity $s_{mn}$ to true positives (TP), with residual $(1-s_{mn})$ assigned to false positives (FP) and negatives (FN). Unmatched findings incur penalties based on their most similar finding:

$$ $\text{TP}=\sum_{(m,n)}s_{mn},\text{FP}=\sum_{(m,n)}\left(1-s_{mn}\right)+\sum_{u}\left(1-\max_{j}s_{uj}\right),\text{FN}=\sum_{(m,n)}\left(1-s_{mn}\right)+\sum_{v}\left(1-\max_{i}s_{iv}\right)\,.$ (6) $$

This formulation supports partial credit based on alignment strength. Full credit is awarded only when a finding fully aligns semantically, temporally, and structurally. Partial matches contribute proportionally to evaluation scores, and when one set contains more findings than the other, the extra findings remain unmatched and are penalized as either FPs or FNs. This scoring scheme enables nuanced evaluation that distinguishes between minor misalignments and complete misses. The final F1 score can be computed from these TP, FP and FN counts using the standard formula. Additional examples illustrating the metric are provided in Appendix [C.2](#A3.SS2).

## 6 Experiments

We conduct three sets of experiments to evaluate our approach from complementary perspectives: (1) the performance of the proposed structuring framework, (2) the diagnostic utility of LunguageScore as a single-report evaluation metric, and (3) the ability of LunguageScore to benchmark performance of various single- and longitudinal-report generation models.

### 6.1 Structuring framework validation

We first assess the structuring framework on Lunguage, our benchmark of 1,473 chest X-ray reports from 230 patients. Each patient has 1 to 15 imaging studies, with a subset of 10 patients selected for full longitudinal trajectories. Reflecting the progressive nature of clinical interpretation, we evaluate the framework in two stages: (i) single-report structuring, which measures the model’s ability to extract localized semantic relations, and (ii) temporal inference, which assesses whether findings are consistently aligned and appropriately organized into clinical episodes across time.

**Table 1: Performance of various models under zero-shot and 5-shot settings. Left: single-report performance. Right: sequential reasoning performance. Best scores per block are bolded.**
|  |  | Single setting | Sequential setting |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | entity-relation | entity-relation-attribute | Entity Grouping | Temporal Grouping |  |  |  |  |  |  |  |  |
| Shot | Model | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R |
| Zero | GPT-4.1 | 0.91 | 0.83 | 1.00 | 0.78 | 0.79 | 0.77 | – | – | – | – | – | – |
| Qwen3 | 0.73 | 0.58 | 1.00 | 0.62 | 0.53 | 0.75 | 0.50 | 0.42 | 0.65 | 0.83 | 0.85 | 0.82 |  |
| Deepseek-v3 | 0.87 | 0.76 | 1.00 | 0.76 | 0.72 | 0.80 | 0.41 | 0.32 | 0.76 | 0.79 | 0.86 | 0.75 |  |
| Llama4-Maverick | 0.81 | 0.68 | 1.00 | 0.69 | 0.64 | 0.76 | 0.35 | 0.25 | 0.77 | 0.60 | 0.88 | 0.47 |  |
| 5-shot | GPT-4.1 | 0.94 | 0.88 | 1.00 | 0.86 | 0.86 | 0.86 | 0.68 | 0.77 | 0.65 | 0.89 | 0.86 | 0.93 |
| Qwen3 | 0.92 | 0.85 | 1.00 | 0.84 | 0.83 | 0.85 | 0.62 | 0.57 | 0.71 | 0.84 | 0.86 | 0.84 |  |
| Deepseek-v3 | 0.93 | 0.88 | 1.00 | 0.86 | 0.85 | 0.86 | 0.66 | 0.63 | 0.75 | 0.85 | 0.88 | 0.84 |  |
| Llama4-Maverick | 0.94 | 0.88 | 1.00 | 0.86 | 0.86 | 0.85 | 0.52 | 0.38 | 0.87 | 0.62 | 0.90 | 0.48 |  |

##### Single setting

We evaluate the model’s ability to generate accurate structured representations from individual reports by comparing predicted (entity, relation, attribute) triplets against expert annotations in Lunguage. Using micro-averaged precision, recall, and F1 scores at both the entity–relation and full triplet levels, we assess our prompting strategy on GPT-4.1 Achiam et al. (2023) and several recent open-source LLMs Liu et al. (2024); Touvron et al. (2023); Zheng et al. (2025), all evaluated under the same framework configuration described in Section [4](#S4). As shown in Table [1](#S6.T1), all models achieve perfect recall and F1 scores 0.92-0.94 for entity–relation extraction with 5-shot prompting, and 0.84-0.86 F1 for full triplet extraction. Increasing the number of few-shot examples leads to further gains, highlighting the robustness of the framework despite the complexity of the schema. Additional analyses, including comparisons with and without vocabulary guidance, 10-shot prompting results, and qualitative examples, are provided in Appendix [B.5](#A2.SS5).

##### Sequential setting

The second stage evaluates how well models group temporally distributed findings into clinically meaningful categories. This grouping task presents challenges due to subtle semantic distinctions in medical terminology. For example, "heart size" and "mediastinal silhouette" might require different groupings despite both relating to cardiac imaging—"heart size" focuses on dimensions (potentially grouping with "cardiomegaly") while "mediastinal silhouette" concerns shape, and a patient could simultaneously have cardiomegaly with a normal mediastinum.
Using micro-averaged F1 scores for evaluation, we found that zero-shot prompting yielded limited results, with GPT-4.1 often producing invalid outputs. Performance improved significantly with five-shot prompting, where most models achieved F1 scores above 0.6 for entity grouping (GPT-4.1 reached 0.68), and temporal grouping showed even stronger results.
Although our strict grouping criteria may result in lower F1 scores when semantically similar concepts fall into different groups, this doesn’t compromise the final clinical evaluation. When these grouped entities are later used in LunguageScore (Section [5](#S5), Equation [3](#S5.E3)), the semantic similarity calculation ensures that related concepts still receive appropriately high similarity scores, thereby preserving clinical validity despite strict initial grouping boundaries. Additional analyses are available in Appendix [B.6](#A2.SS6).

### 6.2 Metric Validation with ReXVal

**Table 2: Kendall Tau and Pearson correlation coefficients (with 95% CIs) between single-report metrics and the total number of radiologist-annotated errors in each report, across the ReXVal dataset.**
| Metric | Kendall Tau | Pearson |
| --- | --- | --- |
| BLEU | -0.39 (-0.27, -0.48) | -0.53 (-0.44, -0.61) |
| BERTScore | 0.50 (-0.42, -0.58) | -0.63 (-0.55, -0.70) |
| GREEN | -0.63 (-0.56, -0.69) | -0.73 (-0.67, -0.78) |
| 1/FineRadScore | -0.69 (-0.63, -0.74) | -0.75 (-0.70, -0.80) |
| RaTEScore | -0.52 (-0.44, -0.59) | -0.63 (-0.56, -0.70) |
| LunguageScore | -0.58 (-0.51, -0.64) | -0.69 (-0.63, -0.74) |

We validate the diagnostic utility of LunguageScore on the ReXVal dataset Yu et al. (2023b), a benchmark with 200 MIMIC-CXR report pairs which were annotated by 6 radiologists, designed to evaluate the alignment between scoring of automated metrics and that of radiologists. Since this benchmark does not include sequential reports, we apply only the single-report version of LunguageScore (i.e., semantic and structural alignment). We compare our metric against the following established alternatives: BLEU Papineni et al. (2002b), BERTScore Zhang et al. (2019), GREEN Ostmeier et al. (2024), FineRadScore Huang et al. (2024), and RaTEScore Zhao et al. (2024). For further details on the settings we used to run each metric, we refer to Appendix [D](#A4).
Table [2](#S6.T2) shows the Kendall Tau and Pearson correlation between each single-report level metric and the total number of errors (both significant and insignificant) identified by radiologists, across all reports in the ReXVal dataset. A more negative correlation indicates stronger alignment with radiologist assessments. Note that we invert FineRadScore to align its direction with other metrics.
We also report 95% confidence intervals, calculated via bootstrapping with 1,000 resamples with replacement of the 200 reports.

Our proposed metric outperforms the other structure- and/or semantics-based metric (BLEU, BERTScore, and RaTEScore) but does not surpass the LLM-derived scores (FineRadScore and GREEN) in terms of correlation with human experts. Nevertheless, it achieves performance close to GREEN and FineRadScore, which were explicitly designed to align with the ReXVal error taxonomy. In contrast, our metric is based solely on semantic and structural alignment between the findings in each report, without access to predefined error types. We further explore inter-metric correlations in Appendix [D](#A4), showing that LunguageScore correlates highly with all other metrics.

### 6.3 Benchmarking single-report and sequential report generation models

We further validate LunguageScore by comparing it against existing evaluation methods across multiple report generation models, assessing its ability to capture clinically meaningful differences at both the single-report and patient-level scales. To this end, we benchmark the performance of four generative models: MAIRA-2 Bannur et al. (2024), Medversa Zhou et al. (2024), RGRG Tanida et al. (2023) and Cvt2distilgpt2 Nicolson et al. (2023).

##### Radiology report generation

All evaluated models require frontal chest X-ray images. Of 80 studies in our sequential dataset, 13 lacked frontal images in MIMIC-CXR, limiting analysis to 67 studies. We used only these studies to ensure comparability across evaluations. For MAIRA-2, we included lateral images when available, while other models received only frontal views. MAIRA-2, RGRG, and Cvt2distilgpt2 generated findings sections, while Medversa produced both findings and impressions, which we combined into complete reports.
Note that only MAIRA-2 was trained to incorporate prior studies, and we explored two settings: standard (using true reference reports from prior studies) and cascaded (using previously MAIRA-2-generated reports as prior context). Further details can be found in Appendix [E](#A5).

##### Single-report setting

In the single-report setting, we compare generated reports with ground truth references on a study-by-study basis across 67(^5^55Whenever no frontal image was available for a study, we were not able to generate a report. These studies are therefore excluded from the sequential analysis, leaving gaps in the sequence of reports that might influence the final result. This occurred for 5 out of 10 patients.) studies. Reference reports combine findings and impression sections. Table [3](#S6.T3) shows performance across various metrics, including our new LunguageScore. For LunguageScore, we use our annotated reports as ground truth structured resources and compare them with outputs from the structuring process in Section [4](#S4). MAIRA-2 (standard setting) clearly outperforms all other models, demonstrating the value of longitudinal context even when evaluated at single-report level. The cascaded setting slightly underperforms compared to standard, as it can drift off course when building upon previously generated reports.

**Table 3: Structured radiology report generation results with 95% confidence intervals.**
|  | Single-report setting | Sequential setting |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | RaTEScore | GREEN | 1/FineRadScore | LunguageScore | LunguageScore |
| Medversa Zhou et al. (2024) | 0.499 (0.47, 0.53) | 0.314 (0.26, 0.37) | 0.170 (0.14, 0.20) | 0.409 (0.38, 0.44) | 0.410 (0.37, 0.45) |
| Cvt2DistilGPT2 Nicolson et al. (2023) | 0.436 (0.41, 0.47) | 0.240 (0.19, 0.29) | 0.152 (0.12, 0.18) | 0.367 (0.34, 0.40) | 0.371 (0.33, 0.41) |
| RGRG Tanida et al. (2023) | 0.479 (0.45, 0.51) | 0.266 (0.23, 0.30) | 0.131 (0.11, 0.15) | 0.406 (0.38, 0.43) | 0.391 (0.36, 0.42) |
| MAIRA-2 Bannur et al. (2024) (standard) | 0.518 (0.49, 0.54) | 0.325 (0.28, 0.37) | 0.193 (0.15, 0.24) | 0.429 (0.40, 0.46) | 0.432 (0.41, 0.46) |
| MAIRA-2 Bannur et al. (2024) (cascade) | 0.504 (0.48, 0.53) | 0.299 (0.25, 0.34) | 0.161 (0.13, 0.19) | 0.419 (0.39, 0.45) | 0.416 (0.38, 0.45) |

##### Sequential Setting

We use the same reports as in the single-report setting but include the history (i.e., indication) section in addition to findings and impression, as it provides essential context for understanding the patient’s trajectory over time.
We evaluate all models in this setting because radiology reports are inherently longitudinal, describing findings across multiple imaging studies. Even models trained on single image-report pairs should produce temporally coherent outputs if each report is properly grounded in the image.
As shown in Table [3](#S6.T3), MAIRA-2, explicitly designed for sequential generation, achieves the highest performance. MedVersa, which additionally uses the history section as input, ranks second. In contrast, models that do not use the history section (CVT2DistilGPT2, RGRG) perform worse. Notably, CVT2DistilGPT2 improves slightly in this setting, while RGRG’s performance declines, revealing differences in temporal coherence.
Our sequential LunguageScore uniquely captures such weaknesses in longitudinal consistency, highlighting its value in evaluating clinically realistic reporting behavior. Section [D](#A4) provides further analysis of the metric’s error sensitivity in both settings.

## 7 Conclusion, Limitations and Future Directions

This work introduces a comprehensive framework for evaluating radiology reports, grounded in Lunguage, a fine-grained benchmark for single and sequential structured reports. We propose a two-stage LLM-based structuring framework and LunguageScore, a novel metric reflecting clinical attributes across semantic, temporal, and structural dimensions.
Limitations: Our study has several important limitations. First, our sequential dataset includes only 10 patients due to labor-intensive annotation, necessitating larger-scale datasets. Second, cross-validation by multiple radiologists is needed to ensure robustness. Third, our framework requires performance improvements in handling complex temporal relationships.
Future Directions: Advancing patient-centered reporting necessitates integration of structured EHR data beyond chest X-rays. Current image-based generation approaches struggle with context-rich sections like patient history. Models lacking access to such contextual signals remain fundamentally limited in longitudinal reasoning and diagnostic continuity, highlighting the need for broader integration with EHR data in future research.

## References

- Achiam et al. [2023]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Alsentzer et al. [2019]
Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.
Publicly available clinical BERT embeddings.
In *Proceedings of the 2nd Clinical Natural Language Processing Workshop*, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
doi: 10.18653/v1/W19-1909.
URL [https://www.aclweb.org/anthology/W19-1909](https://www.aclweb.org/anthology/W19-1909).
- Banerjee and Lavie [2005]
Satanjeev Banerjee and Alon Lavie.
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.
In *Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization*, pages 65–72, 2005.
- Bannur et al. [2024]
Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, et al.
MAIRA-2: Grounded radiology report generation.
*arXiv preprint arXiv:2406.04449*, 2024.
- Bodenreider [2004]
Olivier Bodenreider.
The unified medical language system (umls): integrating biomedical terminology.
*Nucleic acids research*, 32(suppl_1):D267–D270, 2004.
- Busch et al. [2024]
Felix Busch, Lena Hoffmann, Daniel Pinto Dos Santos, Marcus R Makowski, Luca Saba, Philipp Prucker, Martin Hadamitzky, Nassir Navab, Jakob Nikolas Kather, Daniel Truhn, et al.
Large language models for structured reporting in radiology: past, present, and future.
*European Radiology*, pages 1–14, 2024.
- Chapman et al. [2011]
Wendy W Chapman, Prakash M Nadkarni, Lynette Hirschman, Leonard W D’avolio, Guergana K Savova, and Ozlem Uzuner.
Overcoming barriers to nlp for clinical text: the role of shared tasks and the need for additional creative solutions, 2011.
- Demner-Fushman et al. [2009]
Dina Demner-Fushman, Wendy W Chapman, and Clement J McDonald.
What can natural language processing do for clinical decision support?
*Journal of biomedical informatics*, 42(5):760–772, 2009.
- Dorfner et al. [2024]
Felix J Dorfner, Liv Jürgensen, Leonhard Donle, Fares Al Mohamad, Tobias R Bodenmann, Mason C Cleveland, Felix Busch, Lisa C Adams, James Sato, Thomas Schultz, et al.
Comparing commercial and open-source large language models for labeling chest radiograph reports.
*Radiology*, 313(1):e241139, 2024.
- Gu et al. [2020]
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.
Domain-specific language model pretraining for biomedical natural language processing, 2020.
- Hartsock et al. [2025]
Iryna Hartsock, Cyrillo Araujo, Les Folio, and Ghulam Rasool.
Improving radiology report conciseness and structure via local large language models.
*Journal of Imaging Informatics in Medicine*, pages 1–12, 2025.
- Huang et al. [2024]
Alyssa Huang, Oishi Banerjee, Kay Wu, Eduardo Pontes Reis, and Pranav Rajpurkar.
Fineradscore: A radiology report line-by-line evaluation technique generating corrections with severity scores.
*arXiv preprint arXiv:2405.20613*, 2024.
- Jain et al. [2021]
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al.
Radgraph: Extracting clinical entities and relations from radiology reports.
*arXiv preprint arXiv:2106.14463*, 2021.
- Jin et al. [2023]
Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu.
Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval.
*Bioinformatics*, 39(11):btad651, 2023.
- Johnson et al. [2019]
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.
Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.
*Scientific data*, 6(1):317, 2019.
- Khanna et al. [2023]
Sameer Khanna, Adam Dejl, Kibo Yoon, Steven QH Truong, Hanh Duong, Agustina Saenz, and Pranav Rajpurkar.
Radgraph2: Modeling disease progression in radiology reports via hierarchical information extraction.
In *Machine Learning for Healthcare Conference*, pages 381–402. PMLR, 2023.
- Lee et al. [2020]
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.
Biobert: a pre-trained biomedical language representation model for biomedical text mining.
*Bioinformatics*, 36(4):1234–1240, 2020.
- Lin [2004]
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In *Text summarization branches out*, pages 74–81, 2004.
- Liu et al. [2024]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Liu et al. [2025]
Xiaohong Liu, Hao Liu, Guoxing Yang, Zeyu Jiang, Shuguang Cui, Zhaoze Zhang, Huan Wang, Liyuan Tao, Yongchang Sun, Zhu Song, et al.
A generalist medical language model for disease diagnosis assistance.
*Nature Medicine*, pages 1–11, 2025.
- Meystre et al. [2008]
Stéphane M Meystre, Guergana K Savova, Karin C Kipper-Schuler, and John F Hurdle.
Extracting information from textual documents in the electronic health record: a review of recent research.
*Yearbook of medical informatics*, 17(01):128–144, 2008.
- Nicolson et al. [2023]
Aaron Nicolson, Jason Dowling, and Bevan Koopman.
Improving chest x-ray report generation by leveraging warm starting.
*Artificial intelligence in medicine*, 144:102633, 2023.
- Ostmeier et al. [2024]
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al.
Green: Generative radiology report evaluation and error notation.
*arXiv preprint arXiv:2405.03595*, 2024.
- Papineni et al. [2002a]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pages 311–318, 2002a.
- Papineni et al. [2002b]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th annual meeting of the Association for Computational Linguistics*, pages 311–318, 2002b.
- Pons et al. [2016]
Ewoud Pons, Loes MM Braun, MG Myriam Hunink, and Jan A Kors.
Natural language processing in radiology: a systematic review.
*Radiology*, 279(2):329–343, 2016.
- Rao et al. [2024]
Vishwanatha M Rao, Serena Zhang, Julian N Acosta, Subathra Adithan, and Pranav Rajpurkar.
Rexerr: Synthesizing clinically meaningful errors in diagnostic radiology reports.
In *Biocomputing 2025: Proceedings of the Pacific Symposium*, pages 70–81. World Scientific, 2024.
- Rao et al. [2025]
Vishwanatha M Rao, Serena Zhang, Julian N Acosta, Subathra Adithan, and Pranav Rajpurkar.
Rexerr-v1: Clinically meaningful chest x-ray report errors derived from mimic-cxr (version 1.0.0).
*Physionet*, 2025.
doi: https://doi.org/10.13026/9dns-vd94.
- Remy et al. [2024]
François Remy, Kris Demuynck, and Thomas Demeester.
Biolord-2023: semantic textual representations fusing large language models and clinical knowledge graph insights.
*Journal of the American Medical Informatics Association*, 31(9):1844–1855, 2024.
- Savova et al. [2010]
Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute.
Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications.
*Journal of the American Medical Informatics Association*, 17(5):507–513, 2010.
- Smit et al. [2020]
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren.
Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert.
*arXiv preprint arXiv:2004.09167*, 2020.
- Tanida et al. [2023]
Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert.
Interactive and explainable region-guided radiology report generation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 7433–7442, 2023.
- Touvron et al. [2023]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
- Wang et al. [2018]
Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, et al.
Clinical information extraction applications: a literature review.
*Journal of biomedical informatics*, 77:34–49, 2018.
- Woźnicki et al. [2024]
Piotr Woźnicki, Caroline Laqua, Ina Fiku, Amar Hekalo, Daniel Truhn, Sandy Engelhardt, Jakob Kather, Sebastian Foersch, Tugba Akinci D’Antonoli, Daniel Pinto dos Santos, et al.
Automatic structuring of radiology reports with on-premise open-source large language models.
*European Radiology*, pages 1–12, 2024.
- Wu et al. [2021]
Joy T Wu, Nkechinyere N Agu, Ismini Lourentzou, Arjun Sharma, Joseph A Paguio, Jasper S Yao, Edward C Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, et al.
Chest imagenome dataset for clinical reasoning.
*arXiv preprint arXiv:2108.00316*, 2021.
- Yu et al. [2023a]
Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, et al.
Evaluating progress in automatic chest x-ray radiology report generation.
*Patterns*, 4(9), 2023a.
- Yu et al. [2023b]
Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, EKU Fonseca, Henrique Lee, Zahra Shakeri, Andrew Ng, et al.
Radiology report expert evaluation (rexval) dataset, 2023b.
- Zambrano Chaves et al. [2025]
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, et al.
A clinically accessible small multimodal radiology model and evaluation metric for chest x-ray findings.
*Nature Communications*, 16(1):3108, 2025.
- Zhang et al. [2023]
Mengliang Zhang, Xinyue Hu, Lin Gu, Tatsuya Harada, Kazuma Kobayashi, Ronald Summers, and Yingying Zhu.
Cad-chest: Comprehensive annotation of diseases based on mimic-cxr radiology report.
*(No Title)*, 2023.
- Zhang et al. [2019]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
*arXiv preprint arXiv:1904.09675*, 2019.
- Zhao et al. [2024]
Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.
Ratescore: A metric for radiology report generation.
*arXiv preprint arXiv:2406.16845*, 2024.
- Zheng et al. [2025]
Xingyu Zheng, Yuye Li, Haoran Chu, Yue Feng, Xudong Ma, Jie Luo, Jinyang Guo, Haotong Qin, Michele Magno, and Xianglong Liu.
An empirical study of qwen3 quantization.
*arXiv preprint arXiv:2505.02214*, 2025.
- Zhou et al. [2024]
Hong-Yu Zhou, Subathra Adithan, Julián Nicolás Acosta, Eric J Topol, and Pranav Rajpurkar.
A generalist learner for multifaceted medical image interpretation.
*arXiv preprint arXiv:2405.07988*, 2024.

## Appendix A Lunguage Details

##### Dataset preparation

Lunguage aims to support patient-level evaluation of chest X-ray reports by modeling longitudinal diagnostic scenarios. To this end, we curated a benchmark dataset from the official test split of MIMIC-CXR, including all 1,473 reports corresponding to 230 patients. Each patient had between 1 and 15 imaging studies.

We followed the official MIMIC-CXR preprocessing protocol to extract structured text from each report. Specifically, we parsed the history (including “Indication”), findings, and impression sections. The history/indication field provides contextual information relevant to diagnostic reasoning, such as presenting symptoms (e.g., “fever,” “fatigue,” “cough”) or evaluation intents (e.g., “rule out pneumonia”). In contrast, the findings and impression sections describe image-based observations and interpretations.

Section-level coverage across the dataset is summarized as:

- •
History (i.e., Indication): 1,362 reports (92.5%)
- •
Findings: 1,224 reports (83.1%)
- •
Impression: 1,015 reports (68.9%)

Among the reports, 767 contained both findings and impression sections, 457 had findings only, 248 had impression only, and 1 contained only a history section. We excluded infrequently occurring sections such as comparison (often containing anonymized metadata using placeholders like “__”), and technique (e.g., “AP view”), as these appeared in fewer than 5% of cases and were not directly relevant to diagnostic content.

To preserve diagnostic integrity and linguistic variability, we retained all reports in their original form without content filtering. This includes templated reports (e.g., “No acute cardiopulmonary process”) and incomplete notes. All reports were annotated using our schema-based pipeline with no preprocessing beyond section parsing. Structured reports were constructed while preserving raw textual expressions to ensure alignment with the source language used by radiologists.

Figure: Figure A.1: Distribution of the number of imaging studies per patient in Lunguage. Skyblue bars indicate the number of patients for each trajectory length (i.e., number of chest X-ray studies), reflecting the single-report annotation coverage. Salmon bars represent the subset of patients whose reports are also annotated at the longitudinal level. Values above the bars show the number of patients per group (n =), and for salmon bars, the number of patients with sequential annotations. The legend summarizes the total number of patients and reports included at each annotation level.
Refer to caption: /html/2505.21190/assets/figure/datastat.png

### A.1 Single-report Schema: Entity and Relation Definition

Lunguage represents each radiology report as a structured collection of (entity, relation, attribute) triplets. This schema is designed to encode the diagnostic content of reports in a form that supports structured analysis, longitudinal reasoning, and machine-readable interpretation. It captures both observable features from chest X-ray (CXR) images and additional contextual elements embedded in clinical narratives.

##### Entity Types

Entities represent clinically meaningful units such as findings, diagnoses, objects, or background context. Each entity is assigned one of six mutually exclusive Cat (category) labels, depending on whether it originates from the CXR image or external clinical sources.

Chest X-ray Findings are entities that can be directly visualized on the chest X-ray or inferred through image-based interpretation, possibly with minimal supporting context. These form the core of radiologic description and are divided into the following types:

- •
PF (Perceptual Findings): Visual features that are explicitly visible in the image and correspond to anatomical or pathological structures (e.g., “opacity”, “pleural effusion”, “pneumothorax”). These are the most direct and objective form of image evidence.
- •
CF (Contextual Findings): Diagnoses that require interpretation of visual findings in light of limited contextual knowledge (e.g., “pneumonia”, “congestive heart failure”). These may involve reasoning beyond the image but still rely primarily on radiographic evidence.
- •
OTH (Other Objects): Non-anatomic elements such as medical devices, surgical hardware, or foreign materials visible on the image (e.g., “endotracheal tube”, “central venous catheter”, “foreign body”). These often require placement verification or complication monitoring.

Non Chest X-ray Findings are entities that cannot be determined from the image alone and must be inferred from patient history, clinical documentation, or other diagnostic modalities:

- •
COF (Clinical Objective Findings): Structured clinical measurements or physical findings derived from sources such as laboratory tests or vital signs (e.g., “elevated white cell count”, “low oxygen saturation”). These provide objective support for contextual interpretation.
- •
NCD (Non-CXR Diagnosis): Diagnoses that originate from non-CXR modalities (e.g., CT, MRI, serology) and are either mentioned for completeness or used to explain findings (e.g., “stroke”, “AIDS”).
- •
PATIENT INFO: Historical or subjective patient information, such as symptoms or clinical background, that contributes to interpretation (e.g., “fever”, “history of malignancy”, “recent trauma”).

Each entity is additionally annotated with the following attributes that define its diagnostic interpretation within the report:

- •
DxStatus: Indicates whether the entity is considered present or absent in the current study. This label is determined from report language and includes implications from stability or change. For example, “resolved effusion” is annotated as Positive, while “unchanged opacity” is Positive unless the prior state was normal, in which case it is Negative.
- •
DxCertainty: Reflects the level of confidence expressed by the radiologist, labeled as either Definitive or Tentative. Typical cues include phrases like “suggests”, “cannot exclude”, or “possibly indicative of”, all leading to a tentative label.

##### Relation Types

Relations describe either attributes of a single entity or clinically relevant links between multiple entities. All relations must be grounded in the report text and can span across sentences within the same section.

1. Diagnostic Reasoning
These relations connect semantically and clinically related entities. They encode the logic behind diagnostic interpretation.

- •
Associate: A bidirectional, non-causal relationship between entities that co-occur or are conceptually linked (e.g., “opacity” $\leftrightarrow$ “consolidation”). When Evidence is used, a corresponding Associate is also required in the reverse direction.
- •
Evidence: A unidirectional relation in which a finding supports a diagnosis (e.g., “pneumonia” $\rightarrow$ “opacity”).

2. Spatial and Descriptive Attributes
These relations describe intrinsic visual characteristics of an entity as observed within a single chest X-ray image. Unlike temporal attributes, these do not require comparison with prior studies. Instead, they provide descriptive detail that refines the interpretation of a finding or object in terms of location, form, extent, intensity, and symmetry.

- •
Location: Specifies the anatomical or spatial position of the entity (e.g., “right upper lobe”, “carina above 3 cm”). An entity may have multiple location labels, annotated as a comma-separated list (e.g., “right upper lobe, suprahilar”). Location applies to both disease findings and device placements (e.g., “fragmentation” of “sternal wires”).
- •
Morphology: Describes the shape, form, or structural appearance of the entity (e.g., “nodular”, “linear”, “reticular”, “confluent”). Morphological terms help differentiate types of opacities or identify characteristic patterns of pathology.
- •
Distribution: Refers to the anatomical spread or pattern of the entity (e.g., “focal”, “diffuse”, “multifocal”, “bilateral”). This helps characterize whether the finding is localized or widespread, and whether it follows typical anatomical distributions.
- •
Measurement: Captures quantitative properties such as size, count, or volume (e.g., “2.5 cm”, “few”, “multiple”). These descriptors are typically numerical or ordinal and assist in severity grading or follow-up comparison.
- •
Severity: Reflects the degree of abnormality or clinical impact, often based on radiologic intensity or extent (e.g., “mild”, “moderate”, “severe”, “marked”).
- •
Comparison: Indicates asymmetry or difference across anatomical sides or regions within the same image (e.g., “left greater than right”, “right lung appears denser”). This is distinct from temporal comparison and only refers to spatial contrasts visible in the current image.

3. Temporal Change
These relations capture how an entity has changed over time by comparing the current study to previous imaging or known clinical baselines. Temporal attributes are essential for longitudinal interpretation and reflect disease progression, treatment response, or clinical stability. Unlike static descriptors, these attributes require temporal context and often imply clinical decision points.

- •
Onset: Indicates the timing or duration of a finding as described in the report (e.g., “acute”, “subacute”, “chronic”, “new”). These descriptors suggest whether a condition has recently appeared or has been long-standing.
- •
Improved: Signals that a finding has regressed or resolved compared to a prior state (e.g., “resolved effusion”, “decreased consolidation”). It is typically associated with positive treatment response or natural recovery.
- •
Worsened: Indicates that the condition has progressed, increased in extent, or become more severe over time (e.g., “enlarging opacity”, “increased pleural effusion”). This is often associated with disease progression or complications.
- •
No Change: Describes a finding that has remained stable since a prior study (e.g., “unchanged opacity”, “persistent nodule”). Although these are annotated as Positive by default, they are marked as Negative if the prior state was normal (i.e., continued absence of disease).
- •
Placement: Applies specifically to entities labeled as OTH (devices). It describes both the position (e.g., “in expected position”, “malpositioned”) and temporal actions involving the device (e.g., “inserted”, “withdrawn”, “removed”). This attribute is crucial for monitoring device-related interventions over time.

4. Contextual Information This category captures auxiliary information that influences the interpretation of findings but is not a primary descriptor of the radiologic appearance. These relations provide critical contextual cues—such as modality constraints, patient factors, or historical references—that support diagnostic interpretation. While not visual in the conventional sense, they are essential for accurately situating radiologic findings within the broader clinical scenario.

- •
Past Hx: Refers to the patient’s prior medical or surgical history that contextualizes current findings (e.g., “status post lobectomy”, “known tuberculosis”). These mentions often justify or explain current observations or exclude certain diagnoses.
- •
Other Source: Indicates that part of the reported information is derived from modalities other than chest X-ray (e.g., “seen on CT”, “confirmed on MRI”). This distinction is important when findings cannot be visualized directly on the image being interpreted.
- •
Assessment Limitations: Describes technical or procedural factors that constrain the radiologist’s ability to interpret the image accurately (e.g., “poor inspiration”, “rotated patient position”, “limited view due to overlying hardware”). These limitations help qualify the certainty or completeness of the report’s conclusions.

#### A.1.1 Task-specific Vocabulary Construction

To systematically capture the range of descriptive, temporal, spatial, and contextual attributes found in radiologic reporting, we constructed a structured vocabulary of relation terms based on all schema-defined relation types instantiated in Lunguage. To initiate this process, we first applied GPT-4 to a subset of reports to produce initial structured outputs, from which we extracted candidate terms for each relation type. These candidate vocabularies were then manually reviewed and refined by relation category to ensure clinical accuracy, coverage, and consistency.

The primary goals of this process were: (1) to ensure consistency in how lexical expressions are mapped to relation categories, (2) to develop clinically meaningful subcategories within each relation type, and (3) to normalize lexical expressions for downstream applications such as search, reasoning, and integration with structured knowledge resources.

Importantly, our vocabulary only includes relation types that correspond to lexically explicit attributes in the text. We excluded four relation types—Evidence, Associate, DxStatus, and DxCertainty—which, while critical to the annotation schema, are not represented as direct lexical expressions. Evidence and Associate describe reasoning links between entities, often inferred across sentences. DxStatus and DxCertainty encode interpretive stance (e.g., presence vs. absence, tentative vs. definitive) and require contextual reading of the sentence. As these relation types are derived from pragmatic interpretation rather than explicit phrases, they fall outside the scope of vocabulary-level normalization.

For the remaining relation types, we extracted all unique values that were directly linked to entities during annotation. Each relation type was reviewed independently by four board-certified physicians to verify accurate categorization, eliminate inconsistencies, and normalize redundant expressions. We further organized each relation into subcategories reflecting finer-grained semantic distinctions that align with radiologic conventions. For example, among the 543 Location terms, we identified 277 unique anatomical paths grouped under higher-level systems: respiratory (229), musculoskeletal (82), cardiovascular (73), and others. Likewise, Morphology (218 terms) was divided into shape and structure (116), texture and density (63), and smaller classes such as condition.

Temporal progression was captured through Onset (60), Improved (120), Worsened (108), and No Change (138), each of which was subtyped into graded interpretations (e.g., “moderate improvement”, “minimal worsening”). Device-related metadata were structured under Placement (78), which includes terms for positional accuracy (e.g., “malpositioned”) and procedural changes (e.g., “removed”, “repositioned”). Additional relation types included Measurement (147 terms across size, quantity, and normality), Severity (89), Distribution (37), and Comparison (46).

We also captured auxiliary contextual information that, while potentially observable on imaging, typically reflects non-primary or supportive elements in interpretation. This includes Assessment Limitations (296 terms), categorized into four major types: evaluation limitations (143), patient-related limitations (72), field-of-view limitations (55), and technical limitations (26). Other categories include Other Source (56), which marks references to non-CXR modalities (e.g., CT, MRI), and Past Hx (41), which captures historical clinical references.

The resulting vocabulary includes 14 relation types derived from lexical evidence, each organized into coherent subtypes that reflect the nuances of radiologic description. Normalized forms were retained as preferred terms, and inconsistent variants were removed. Although formal UMLS mapping was not enforced—given that many of the relation terms lie outside conventional ontologies—we ensured lexical consistency and clinical interpretability to support future integration efforts. This curated vocabulary enables fine-grained modeling of chest X-ray reports and ensures that structured annotations reflect a clinically grounded and internally consistent taxonomy of radiologic language, aligned with the conventions of routine diagnostic documentation.

#### A.1.2 Single Annotation Details

To construct a clinically reliable gold-standard dataset, we implemented a structured annotation pipeline that reviewed and refined the initial triplets generated by GPT-4 (0613). Unlike the vocabulary construction phase—which focused on individual terms without considering report context—this stage involved section-by-section review of all structured outputs in each report to ensure contextual accuracy and logical consistency.

All 1,473 chest X-ray reports in Lunguage were divided evenly among annotators. Each annotator independently reviewed approximately one-quarter of the dataset, ensuring balanced coverage and minimizing reviewer bias across the annotated corpus. Within each report, annotators examined the structured outputs across the history/indication, findings, and impression sections. The goal was to verify whether the extracted (entity, relation, attribute) triplets accurately captured the meaning of the source text and aligned with the predefined schema.

This review explicitly included schema elements that require contextual interpretation and cannot be evaluated at the lexical level alone—namely, DxStatus, DxCertainty, Associate, and Evidence. These attributes reflect interpretive judgments, such as identifying when an “opacity” supports a diagnosis of “pneumonia” or whether two entities should be linked through an associative relation. Annotators verified whether such relations were correctly inferred from the surrounding text and whether the attributes assigned to each entity (e.g., presence, uncertainty, temporal change) matched the narrative context.

To support this process, we developed a custom annotation interface (Figure [A.2](#A1.F2)) that displayed the original report text alongside GPT-4’s predicted triplets and an editable table of structured fields. Each sentence in the report was paired with its associated annotations, including entity category, relation type, and all relevant attributes. Annotators could directly add, edit, remove, or merge entries to reflect clinically accurate interpretations. For example, terms like “ground glass opacity”—which could be mistakenly split—were merged into a single PF (perceptual finding) entity based on how radiologists commonly use the phrase. Annotation was conducted separately for each section (history, findings, impression), and the interface supported sentence-level review within each section to ensure consistent entity–relation mappings when terms appeared across multiple sentences.

As a result of this process, the finalized gold dataset includes 17,949 validated entities and 23,307 relation instances. These annotations reflect both explicit descriptive attributes and contextually inferred diagnostic relationships, providing a robust benchmark for evaluating schema-based information extraction systems in chest radiograph interpretation.

Figure: Figure A.2: Annotation interface used during gold dataset construction. Annotators reviewed GPT-4-generated triplets per report section and refined the entity–relation structure to ensure schema correctness and contextual validity.
Refer to caption: /html/2505.21190/assets/figure/streamlit.png

### A.2 Sequential Annotation Details

In contrast to the single-report structuring phase, which focused on refining schema-based annotations within individual reports, the sequential annotation phase aimed to assess the longitudinal consistency of entity-level interpretations across temporally ordered reports from the same patient. This required global comparisons across all sections—history, findings, and impression—integrating entity–relation triplets into clinically coherent sequences.

Unlike earlier phases that processed each report independently, this step involved exhaustive pairwise comparisons of all annotated expressions across time. Annotators judged whether lexically distinct phrases referred to the same underlying clinical entity by examining radiological terminology, anatomical location, temporal modifiers (e.g., “resolving”, “unchanged”), and diagnostic specificity. Expressions identified as referring to the same finding were grouped together; otherwise, they were assigned to separate entity groups.

To further structure these entity groups, we assessed whether each represented a single episode of care or multiple distinct episodes. This required examining the temporal order and interval between observations. Intervals were computed using the StudyDate metadata from MIMIC-CXR, and episode boundaries were assigned based on temporal coherence—considering factors such as time gaps, patterns of resolution or worsening, and recurrence of findings.

For example, a progression from “moderate left effusion” (day 0) to “small effusion” (day 14) and “trace effusion” (day 45) was treated as a single resolving episode. However, a subsequent “moderate effusion” on day 180 was regarded as a separate episode, while all entities assigned to either episode are grouped into the same Entity Group. Similarly, “right lower lobe opacity” followed by “resolving infiltrate” was interpreted as one episode, whereas a new “opacity” on day 150 initiated a different episode. This process was applied to 80 chest X-ray reports from 10 patients, yielding longitudinal annotations that capture consistent entity grouping across lexical variations and clinically coherent organization of episodes based on temporal reasoning.

To better characterize the annotation results, we summarize the distribution of entity groupings and temporal episodes in Table [A.1](#A1.T1). The columns report:

- •
# Reports: The total number of reports per patient sequence.
- •
Entity Group Distribution: The number of findings assigned to each entity group (#Group), after normalization and longitudinal reasoning. Some groups consist of a single unique expression, while others aggregate multiple semantically related terms.
- •
Temporal Group Distribution: The number of findings assigned to each temporal group (#Group), where each group represents a distinct clinical episode.

**Table A.1: Distribution of entity groups and temporal groups across annotated patient sequences.**
| Subject ID | # Reports | Entity Group Distribution (#Group:Count) | Temporal Group Distribution (#Group:Count) |
| --- | --- | --- | --- |
| p10274145 | 5 | 1:19, 2:11, 3:2, 4:3 | 1:33, 2:2 |
| p10523725 | 9 | 1:36, 2:6, 3:3, 4:2, 5:2, 7:2 | 1:47, 2:2, 3:1, 6:1 |
| p10886362 | 10 | 1:26, 2:3, 3:6, 4:4, 6:1, 7:1, 9:1, 13:1 | 1:39, 2:4 |
| p10959054 | 7 | 1:31, 2:6, 3:2, 4:2, 5:1, 6:1, 9:1 | 1:37, 2:5, 3:2 |
| p12433421 | 13 | 1:49, 2:6, 3:10, 5:1, 7:1, 17:1 | 1:66, 2:2 |
| p15321868 | 6 | 1:24, 2:5, 3:2, 4:1, 5:2 | 1:32, 2:2 |
| p15446959 | 5 | 1:29, 2:7, 3:3, 4:2 | 1:37, 2:4 |
| p15881535 | 3 | 1:17, 2:2, 3:2, 5:1 | 1:20, 2:2 |
| p17720924 | 8 | 1:30, 2:8, 3:5, 4:1, 5:1 | 1:41, 2:2, 4:2 |
| p18079481 | 14 | 1:34, 2:10, 3:3, 4:2, 6:3, 7:3, 8:1 | 1:43, 2:10, 3:3 |

Across the 10 patients in the sequential evaluation phase, the number of temporal groups assigned to a single entity group ranged from 1 to 6, indicating that some findings were observed in multiple distinct clinical episodes over time. Likewise, the number of distinct entity groups varied significantly. Most entity groups consisted of a single mention, but some aggregated up to 17 lexically different expressions. For example, subject p12433421 exhibited the most diverse entity grouping, with 17 distinct phrases all referring to variations of pleural effusion (e.g., “effusion,” “pleural effusion,” “pleural effusion left”) unified under one normalized cluster. Similarly, subject p10523725 had the highest number of temporal groups (6) within a single entity group, driven by repeated mentions of dyspnea across non-contiguous timepoints. These results highlight the complexity and variability of radiologic expression in longitudinal reporting, and underscore the necessity of models and metrics capable of robustly handling both semantic variation and episodic continuity in time-aware clinical tasks.

## Appendix B Framework details

### B.1 Overview

Figure: Figure B.1: Overview of our end-to-end pipeline. We begin with gold-standard structured reports (Lunguage) created by radiologists. Candidate free-text reports are generated by a report model and structured via our two-stage framework: (1) schema-aligned extraction (Framework (Single)), and (2) longitudinal grouping and normalization (Framework (Sequential)). Candidate and gold outputs are aligned by entity and temporal groups, and evaluated using LUNGUAGESCORE across semantic, temporal, and structural dimensions. Std. timepoint denotes the acquisition date of each chest X-ray study.
Refer to caption: /html/2505.21190/assets/figure/overview.png

##### Framework Overview and Evaluation Setup.

Figure [B.1](#A2.F1) presents a complete overview of our pipeline, integrating the three core contributions of this study: the construction of the LUNGUAGE benchmark, the development of a two-stage LLM-based structuring framework, and the design of LUNGUAGESCORE, a clinically grounded evaluation metric.

We begin with gold-standard annotations encompassing both single-report structures and longitudinal sequences. Our two-stage framework first applies schema-aligned extraction to derive entity–attribute–relation triplets from free-text inputs (Framework (Single)), and subsequently performs longitudinal normalization and temporal grouping across studies to identify consistent findings and clinically coherent episodes (Framework (Sequential)).

After this process, the structured candidate report is compared to the gold-standard annotations of the reference report using fine-grained matching that incorporates semantic similarity, temporal coherence, and structural attribute alignment. These dimensions are jointly assessed by LUNGUAGESCORE, which computes similarity scores based on the full set of extracted and grouped triplets.

### B.2 Single Setting Prompt

Figure: Figure B.2: Prompt template used for single-report structuring of chest X-ray findings. The model receives section-wise input sentences along with vocabulary-based candidate spans and is instructed to extract relations and attributes.
Refer to caption: /html/2505.21190/assets/figure/single_prompt.png

### B.3 Vocabulary Matching Algorithm

To improve consistency in entity extraction and reduce hallucinations in schema-based structuring, we implemented a vocabulary-guided span matching algorithm (see Appendix [A.1.1](#A1.SS1.SSS1) for details on vocabulary construction). This algorithm processes each section of the radiology report (e.g., findings) to identify candidate entity spans by directly matching contiguous token sequences against entries in a schema-defined vocabulary, without normalization such as lowercasing or punctuation removal. Each sentence is evaluated independently, and multiple overlapping matches are retained—e.g., “left lung” may correspond to both PF and Location.

Importantly, the matched vocabulary spans are not assumed to constitute a complete or authoritative set of entities. Instead, they serve as reference cues for the LLM, which remains responsible for the final relation extraction. The LLM is expected to leverage the matched terms as guidance while retaining the flexibility to identify additional entities or values not covered by the vocabulary. This design accommodates incompleteness in the vocabulary and enables the model to make context-sensitive inferences based on both the prompt and observed patterns in the data.

The matching algorithm is summarized below:

Figure: Algorithm 1 Span-Based Vocabulary Matching

This procedure constrains entity recognition to schema-aligned expressions, allowing the LLM to focus on inferring relational structure rather than determining precise span boundaries. By anchoring extraction to predefined lexical targets, it reduces ambiguity and ensures consistent treatment of clinically equivalent yet lexically variable expressions.

### B.4 Sequential Setting Prompt

Figure: Figure B.3: Prompt template provided to the LLM for sequential structuring of radiologic findings. The model is instructed to group terms referring to the same clinical observation and to identify episode boundaries based on time intervals and progression patterns. Grouping and temporal disambiguation criteria are embedded in the prompt, following the structured annotation protocol.
Refer to caption: /html/2505.21190/assets/figure/sequential_prompt.png

### B.5 Single Setting Analysis

**Table B.1: Ablation results of GPT-4.1 under varying prompt-shot configurations and vocabulary matching. We report precision (P), recall (R), and F1 scores for both entity-relation pair extraction and complete triplet extraction tasks.**
|  |  | entity-relation | entity-relation-attribute |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Shot | Vocab Usage | F1 | P | R | F1 | P | R |
| Zero | No | 0.79 | 0.65 | 1.00 | 0.52 | 0.65 | 0.44 |
| Yes | 0.92 | 0.85 | 1.00 | 0.78 | 0.80 | 0.77 |  |
| 5-shot | No | 0.93 | 0.87 | 1.00 | 0.84 | 0.85 | 0.83 |
| Yes | 0.94 | 0.89 | 1.00 | 0.87 | 0.87 | 0.86 |  |
| 10-shot | No | 0.94 | 0.88 | 1.00 | 0.86 | 0.86 | 0.85 |
| Yes | 0.96 | 0.91 | 1.00 | 0.89 | 0.90 | 0.87 |  |

We conducted an ablation study to quantify the individual and combined effects of vocabulary matching and in-context demonstrations on single-report structuring. Using 80 radiology reports from 10 patients, previously annotated for sequential evaluation, this subset enabled consistent evaluation across controlled input conditions.

Six configurations were tested by varying two factors: (1) whether span-to-category alignment via vocabulary matching was applied, and (2) the number of in-context examples provided in the prompt (0, 5, or 10). Vocabulary matching involved matching contiguous text spans against a predefined lexicon and retrieving all associated schema categories, ensuring lexical consistency and reducing ambiguity in span interpretation, as described in Appendix [B.3](#A2.SS3). In-context demonstrations consisted of structured examples retrieved from the gold set of structured reports using BM25 retrieval, based on textual similarity to the input report. These examples illustrate appropriate usage of entity types and relations under the schema.

As shown in Table [B.1](#A2.T1), vocabulary matching consistently enhanced performance across all prompt configurations. Under the zero-shot setting, incorporating vocabulary guidance raised the triplet-level F1 score from 0.52 to 0.78, and the entity-relation F1 from 0.79 to 0.92. When five in-context demonstrations were provided, the triplet F1 increased further—reaching 0.84 without vocabulary and 0.87 with vocabulary. The highest accuracy was achieved by combining both components: the 10-shot setting with vocabulary matching attained a triplet F1 of 0.89.

These results indicate that vocabulary matching and in-context demonstrations offer complementary benefits. Vocabulary alignment improves lexical grounding and category consistency, while prompting with examples strengthens structural fidelity across varying linguistic expressions. Together, they establish a robust configuration for producing schema-compliant structured outputs from free-text radiology reports.

To illustrate the qualitative impact of vocabulary matching and prompt-based demonstrations, we examined example outputs across configurations with and without these components. In the sentence “there is no focal consolidation”, the model without vocabulary and prompt guidance extracted “focal consolidation” as the entity, conflating the modifier and the core clinical concept. In contrast, all other configurations correctly identified “consolidation” as the schema-aligned entity. A similar pattern was observed in “there are no new focal opacities concerning for pneumonia”, where the no-guidance setup extracted “focal opacities”, whereas guided configurations yielded the correct entity “opacities”.

These examples underscore the importance of explicitly aligning model outputs to a predefined schema. Linguistically valid but structurally inconsistent extractions can hinder downstream applications, where precise interpretation and reliable information linkage are essential. By providing lexical anchoring through vocabulary and structural demonstrations via prompts, our approach ensures that model predictions are not only accurate but also semantically coherent and clinically usable.

### B.6 Sequential Setting Analysis

We qualitatively evaluated model behavior in the sequential setting by analyzing entity grouping outputs over time. Using longitudinal chest X-ray reports from a single patient, we assessed how well the predicted entity groupings aligned with gold-standard annotations. As illustrated in Figure [B.4](#A2.F4), the patient underwent three imaging studies at 0, 1292, and 1591 days from the initial scan, enabling a detailed examination of temporal consistency in entity tracking.

Most clinical observations were consistently grouped across both annotations. For instance, three lexical variants—orthopedic side plate right clavicular unchanged, right clavicle hardware, and internal fixation hardware—were all correctly assigned to the same entity group in both the gold standard and the model output. Although the representative phrase differed (orthopedic side plate… in the reference versus right clavicle hardware in the prediction), the group identity was preserved, indicating successful recognition of referential equivalence across timepoints.

Nevertheless, several discrepancies emerged. One involved two temporally separated mentions of pneumonia, which were grouped together in the gold annotations but split into separate groups in the model output. This divergence arose because the model treated the findings differently based on their diagnostic status (e.g., resolved vs. new). Such behavior suggests that the model failed to fully comply with the grouping principle that emphasizes radiological identity over contextual modifiers like status or timing.

Another deviation was observed in the handling of evolving descriptions related to opacity in the right cardiophrenic sulcus. Whereas the gold annotations grouped temporally related expressions (e.g., opacity…interval resolution) into a single entity, the model assigned each instance to a separate group. This highlights the model’s limited ability to incorporate temporal continuity cues such as “improving” or “resolving” when constructing entity-level associations.

Despite these localized inconsistencies, the overall grouping performance remained robust. In many cases, LunguageScore reported high similarity scores between predicted and reference structures, indicating that the model preserved the essential semantic structure even when precise grouping boundaries differed slightly. These findings support the reliability of our sequential annotation approach for tracking clinically meaningful entities over longitudinal report timelines.

Figure: Figure B.4: Entity grouping results for patient p15881535 based on sequential chest X-ray reports, comparing human-annotated gold-standard groupings (rows) with GPT-4.1 model predictions (columns). Each numbered cell corresponds to an individual finding, expressed as a linearized phrase that combines the entity and its attributes. While group labels may vary slightly in wording, alignment is assessed based on 1:1 row-to-column correspondence. Among 34 total findings, the gold standard forms 22 groups and the model predicts 25; excluding 3 grouping mismatches, the results show strong agreement, illustrating the model’s adherence to temporal and semantic grouping criteria.
Refer to caption: /html/2505.21190/assets/figure/p15881535_entity_mapping.png

## Appendix C LunguageScore Details

### C.1 Attribute Weights of LunguageScore

To reflect the clinical importance of structured attributes in radiology reports, LUNGUAGESCORE applies attribute-specific weights when measuring similarity between predicted and reference structures. Each comparison is performed at the level of relational triplets, jointly assessing both temporal and structural alignment. For structural attributes, we assign weights based on expert consensus from the four board-certified radiologists who participated in the data annotation process, reflecting each attribute’s diagnostic significance. Although the initial weights are unnormalized, they are rescaled such that their total contribution sums to 1.0 during evaluation (see Table [C.1](#A3.T1)).

For the sequential setting, temporal alignment contributes a fixed weight of 1.0, divided equally between two components: whether the predicted and reference findings belong to the same study timepoint (0.5), and whether they fall within the same temporal group (0.5).

Although our schema includes inferential relations such as Associate and Evidence, these are intentionally excluded from the evaluation metric. Such relations capture diagnostic reasoning—e.g., linking “opacity” as supporting evidence for “pneumonia”—but do not directly reflect the correctness of factual information. Scoring them would conflate interpretive inference with structural accuracy. Instead, our metric focuses on clinically grounded descriptors and attributes that define the diagnostic content of the report. Future extensions may consider integrating reasoning-based relations in settings that explicitly target causal or explanatory fidelity.

**Table C.1: Weights used in LUNGUAGESCORE for evaluating structural similarity. Temporal weights apply only in the sequential setting, while structural attribute weights reflect the diagnostic importance of each relation type. All values are normalized such that their respective groups (temporal or structural) sum to 1.0 during evaluation.**
| Temporal Weights | Value |
| --- | --- |
| Study Timepoint | 0.5 |
| Temporal Group | 0.5 |

### C.2 LunguageScore examples

##### Single-Report Assessment

To illustrate how LUNGUAGESCORE evaluates structured prediction quality in the single-report setting, we present detailed examples of pairwise comparisons between predicted and gold-standard structured reports. As detailed in Section [5](#S5) in the main text, each comparison is decomposed into two complementary components:

- •
Semantic Score: Computed as the cosine similarity between embedded linearized entity phrases. These phrases are formed by concatenating free-text attributes, including Location, Morphology, Distribution, Measurement, Severity, Onset, Improved, Worsened, No Change, and Placement. This representation captures the semantic content of the entity and its descriptive qualifiers, allowing similarity to be measured in an integrated manner.
- •
Structural Score: A weighted sum of attribute-wise comparisons. Categorical attributes (DxStatus and DxCertainty) are scored in binary fashion (1.0 for exact match, 0.0 otherwise), while all other attributes are evaluated via cosine similarity of their embeddings. The relative importance of each attribute is determined by expert-defined weights (see Table [C.1](#A3.T1)).

The final similarity between a predicted and reference finding is calculated as the product of the semantic and structural scores:

$$ $\text{{Total Score}}=\text{Semantic Score}\times\text{Structural Score}$ $$

Note: Entity refers to the linearized phrase comprising the core entity and its attributes. Avg. Cosine indicates cosine similarity averaged over MedCPTJin et al. (2023) and BioLORD23Remy et al. (2024) embeddings of the phrases. Weights shown in the table reflect unnormalized values; the final Structural Score is computed by normalizing the weighted sum by the total weight of all included attributes. For a more formal explanation of the scoring method, we refer to Section [5](#S5) in the main text.

Example 1: Moderate Match with Attribute-Level Divergence

| Attribute | GT Value | Pred Value | Match Type | Score | Weight |
| --- | --- | --- | --- | --- | --- |
| Entity | effusions bilateral small | pleural effusion left-sided pleural small stable | Avg. Cosine | 0.743 | — |
| DxStatus | positive | positive | Exact match | 1.00 | 0.50 |
| DxCertainty | definitive | definitive | Exact match | 1.00 | 0.10 |
| Location | bilateral | left-sided pleural | Avg. Cosine | 0.54 | 0.20 |
| Severity | small | small | Exact match | 1.00 | 0.15 |
| Improved | — | stable | Avg. Cosine | 0.00 | 0.15 |

Example 2: Partial Match with Location and Severity Differences

| Attribute | GT Value | Pred Value | Match Type | Score | Weight |
| --- | --- | --- | --- | --- | --- |
| Entity | opacification left retrocardiac | pleural effusion left moderate | Avg. Cosine | 0.447 | — |
| DxStatus | positive | positive | Exact match | 1.00 | 0.50 |
| DxCertainty | definitive | definitive | Exact match | 1.00 | 0.10 |
| Location | left retrocardiac | left | Avg. Cosine | 0.60 | 0.20 |
| Severity | — | moderate | Avg. Cosine | 0.00 | 0.15 |

Example 3: Strong Match with Minor Lexical Variants

| Attribute | GT Value | Pred Value | Match Type | Score | Weight |
| --- | --- | --- | --- | --- | --- |
| Entity | opacity right lung base | opacity right lower lung base stable | Avg. Cosine | 0.842 | — |
| DxStatus | positive | positive | Exact match | 1.00 | 0.50 |
| DxCertainty | definitive | definitive | Exact match | 1.00 | 0.10 |
| Location | right lung base | right lower lung base | Avg. Cosine | 0.95 | 0.20 |
| Improved | — | stable | Avg. Cosine | 0.00 | 0.15 |

##### Sequential-Report Assessment

To clarify how LUNGUAGESCORE computes similarity in the sequential setting, we present illustrative examples comparing gold-standard and predicted findings. Each score is computed from three components:

- •
Semantic Score: In the sequential-report setting, semantic similarity is computed between ENTITYGROUP representations, which group together lexically variable but conceptually equivalent findings observed at different timepoints.
- •
Temporal Score: Value of 1.0 if both findings appear in the same study timepoint and in the same temporal group, or 0.5 if they belong to the same broader temporal group but from different studies, or vice versa. If neither matches, the score is 0.
- •
Structural Score: Weighted average of attribute-level matches (exact for binary attributes, cosine similarity for textual ones).

The overall similarity score is computed as:

$$ $\text{Total Score}=\text{Semantic Score}\times\text{Temporal Score}\times\text{Structural Score}$ $$

**Table C.2: Examples of LUNGUAGESCORE computations in the sequential setting. Each row compares a predicted finding against the corresponding ground-truth reference. Total Score is computed as the product of semantic similarity, temporal alignment, and structural accuracy. Time denotes the study timepoint, and TG indicates the assigned temporal group.**
|  | GT | Prediction | Explanation | Total (Sem × Temp × Str) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Case | EntityGroup | Time | TG | EntityGroup | Time | TG |  |  |
| 1 | pleural effusion subpulmonic moderate | 2 | 1 | pleural effusion right subpulmonic layering moderate stable | 2 | 1 | Minor semantic variation in anatomical modifiers and progression terms | 0.68 (0.82 × 1.0 × 0.83) |
| 2 | hilar contours stable | 3 | 1 | hilar contours unchanged | 3 | 1 | Semantically equivalent; lexical variation in stability descriptor | 0.90 (0.93 × 1.0 × 0.97) |
| 3 | atelectasis left lower lobe mild-to-moderate | 1 | 1 | atelectasis left lower lobe unchanged | 2 | 1 | Different timepoints (0.5), severity term vs. stability term mismatch | 0.35 (0.92 × 0.50 × 0.76) |
| 4 | PICC mid SVC | 2 | 1 | left PICC mid SVC | 1 | 1 | Core entity match with modifier discrepancy; higher specificity in prediction; different timepoints | 0.45 (0.90 × 0.50 × 1.00) |
| 5 | hilar contours unchanged | 2 | 1 | cardiomediastinal silhouette unchanged | 3 | 1 | Semantically related anatomical terms; timepoint mismatch (0.5) | 0.34 (0.68 × 0.50 × 1.00) |

##### Final Scoring and Interpretability

LUNGUAGESCORE calculates a Total Score for each matched pair of predicted and reference findings by combining semantic similarity and structural alignment. In the single-report setting, the total score is defined as the product of cosine similarity over linearized entity phrases and a weighted score of attribute-level matches. In the sequential setting, the metric further incorporates a temporal alignment factor, distinguishing between exact study-time matches and broader temporal group continuity.

These component-wise scores are then aggregated across matched pairs to compute the overall F1 metric, as detailed in Section [5](#S5). Crucially, each comparison yields interpretable diagnostics: the semantic score quantifies lexical alignment of free-text descriptors; the structural score exposes attribute-level agreement or divergence; and in longitudinal contexts, the temporal score reveals whether grouping decisions respect continuity over time.

By exposing this granularity, LUNGUAGESCORE not only delivers a robust scalar evaluation, but also supports nuanced error analysis—highlighting which components of a model’s output (e.g., misassigned severity, incorrect timing, lexical drift) most strongly influenced final performance. This interpretability makes the metric especially valuable to understand model’s behavior.

### C.3 Clinical BERT Model Selection

We considered multiple clinical BERT models for computing contextual semantic embeddings. The candidate models we compared were BioLORD Remy et al. (2024), BiomedBERT Gu et al. (2020), MedCPT Jin et al. (2023), BioClinicalBERT Alsentzer et al. (2019), ClinicalBERT Liu et al. (2025) and BioBERT Lee et al. (2020). To decide which models to use in the semantic similarity step of LunguageScore, we conducted an experiment over ReXVal, a subset of the MIMIC-CXR test set encompassing 50 randomly selected studies. We structured each individual study according to our framework described in Section [4](#S4)(i), and then generated all linearized phrases derived from entity–location–attribute triplets for both the reference report and the candidate report. We then used each candidate BERT embedding model to generate an embedding for each phrase, and computed the pairwise cosine similarity for all pairs of phrases (one from the reference report and one from the candidate report). Figure [C.1](#A3.F1) shows the distribution of this similarity score for the different BERT embedding models. We find that BiomedBERT, BioClinicalBERT, ClinicalBERT and BioBERT lack variety, always scoring pairs of phrases as highly related. BioLORD manages to capture the most diversity in semantic similarity, followed by MedCPT. For this reason, we choose to use both BioLORD and MedCPT to calculate semantic similarity, by taking the average over both models.

Figure: Figure C.1: Distribution of pairwise cosine similarity scores for different BERT embedding models, calculated between pairs of embedded linearized phrases taken from the ReXVal datset.
Refer to caption: /html/2505.21190/assets/figure/emb_models.png

## Appendix D Metric Validation

##### Metric Implementation Details

Whenever not further specified, we used default settings for all the metrics as provided by their respective libraries. For BLEU, we use the implementation provided in the huggingface/evaluate library. For BERTScore, we also use the implementation from the huggingface/evaluate library, with distilroberta-base as an embedding model. For GREEN, we use StanfordAIMI/GREEN-radllama2-7b as a language model. For FineRadScore, we use GPT-4 as a language model, which responds with a list of errors each linked to a severity level. To turn this into a score, we associate each severity level with a number, and sum these scores, forming FineRadScore as proposed by Huang et al. (2024). In our tables, we report 1/FineRadScore, inverting the total sum to ensure that a higher score is associated with higher quality. For RaTEScore, we use their default weight matrix. Note that in their own comparison with ReXVal, the authors used a custom weight matrix trained specifically for long reports instead of the default, explaining the slight discrepancy between their reported Kendall Tau correlation with ReXVal radiologists and the one we report in Table [2](#S6.T2).

##### ReXVal Analysis

To assess the consistency of our metric with established evaluation standards, we conducted a correlation analysis across the ReXVal benchmark, which includes expert-annotated radiology reports and associated error counts. Specifically, we computed pairwise Pearson correlations between all single-report metrics over the ReXVal dataset. As presented in Figure [D.1](#A4.F1), our metric exhibits strong positive correlations with BLEU (0.73), BERTScore (0.77), GREEN (0.84), RaTEScore (0.77), and 1/FineRadScore (0.73). Notably, among all evaluated metrics, our score achieves the highest average correlation across all pairwise comparisons, indicating strong alignment with multiple evaluation perspectives and suggesting broader generalizability.

Furthermore, Figure [D.2](#A4.F2) illustrates the linear relationship between each metric and the number of radiologist-identified errors per ReXVal report. Although 1/FineRadScore shows the highest overall correlation, its relationship with error counts is not consistently linear, especially when the number of errors is low. In these cases where distinguishing between high-quality outputs is most crucial, its ability to make fine-grained distinctions is limited. In contrast, our metric not only maintains strong correlation but also demonstrates stable linear responsiveness across the full error range, underscoring its robustness and reliability as a clinically aligned evaluation measure.

Figure: Figure D.1: Pairwise Pearson correlations between our metric (LunguageScore), and the metrics BLEU, BERTScore, GREEN, 1/FineRadScore and RaTESCore.
Refer to caption: /html/2505.21190/assets/figure/ReXVal_correlation.png

Figure: Figure D.2: Scatter plot illustrating the correlation between the total number of errors identified by radiologists per report, and each of the single-report metrics, including our LunguageScore. $r$ indicates the Pearson correlation as reported in Table [2](#S6.T2).
Refer to caption: /html/2505.21190/assets/figure/scatter.png

##### Error Sensitivity Analysis with ReXErr Rao et al. ( 2024 )

To assess the error sensitivity of our metric across diverse failure types in radiology report generation, we use the ReXErr-v1 dataset Rao et al. (2025), which contains synthetic reports with systematically injected clinical errors. These errors are categorized into content addition, context-dependent, and linguistic quality types, covering a broad spectrum of realistic mistakes. We focus on the subset of ReXErr aligned with our sequential structured report dataset, comprising 57 MIMIC-CXR reference reports paired with corresponding error-injected versions. Each manipulated report contains three injected errors, drawn from 12 defined error categories using a context-sensitive sampling method.

For each pair, we extract the Findings and Impression sections and evaluate them independently using our single-report LunguageScore, along with established alternatives: GREEN, FineRadScore, and RaTEScore. Figure [D.3](#A4.F3) displays the score distributions for each of the 12 error types, relative to the average score across the subset. Our metric demonstrates differentiated sensitivity across error types, with notably larger penalizations for false predictions, incorrect negations, and changes in severity—reflecting its alignment with clinically meaningful deviations.

Figure: Figure D.3: Distribution of the scores for each of the twelve error types in ReXErr, relative to the average score across the 57 ReXErr reports.
Refer to caption: /html/2505.21190/assets/figure/rexerr_green.png

##### Sequential Sensitivity Analysis

We further assessed the sensitivity of LunguageScore to clinically meaningful disruptions in temporal coherence by constructing a synthetic evaluation set in which longitudinal progression cues were deliberately inverted. Specifically, we selected 8 patient sequences from our sequential-report dataset that contained explicit temporal descriptors—such as improved or worsened—and manually reversed these attributes to simulate a contradiction in the clinical trajectory. For example, a statement like “the previously seen right lower lobe opacification has decreased substantially” was changed to “increased substantially,” thereby inverting its semantic implication. Two patient sequences that lacked any such temporal expressions were excluded.

Both the single-report and sequential variants of LunguageScore were applied to these perturbed sequences. To quantify the metric’s responsiveness, we introduce the Effect Rate, which captures the average score reduction per flipped attribute:

$$ $\text{Effect Rate (\%)}=\frac{1-\text{score}}{\#\text{flipped attributes}}\times 100$ $$

A perfect score of 1.0 indicates complete semantic and structural agreement with the gold standard. Deviations from this ideal reflect the metric’s sensitivity to reversed temporal directionality. The normalization by the number of flipped attributes allows us to measure the per-attribute impact on the similarity score.

**Table D.1: Effect Rate for each manipulated patient sequence. W/I denotes the number of worsened/improved attributes flipped.**
| Patient ID | # Attr. (W/I) | Single Score | Effect Rate (S, %) | Sequential Score | Effect Rate (Seq, %) |
| --- | --- | --- | --- | --- | --- |
| p10274145 | 5 (0/5) | 0.981 | 0.38 | 0.979 | 0.42 |
| p10523725 | 3 (1/2) | 0.989 | 0.37 | 0.987 | 0.43 |
| p10886362 | 8 (5/3) | 0.983 | 0.21 | 0.979 | 0.26 |
| p10959054 | 13 (9/4) | 0.967 | 0.25 | 0.963 | 0.28 |
| p12433421 | 15 (8/7) | 0.968 | 0.21 | 0.971 | 0.19 |
| p15321868 | 2 (1/1) | 0.982 | 0.90 | 0.988 | 0.60 |
| p15881535 | 1 (0/1) | 0.992 | 0.80 | 0.992 | 0.80 |
| p18079481 | 10 (2/8) | 0.976 | 0.24 | 0.980 | 0.20 |

While the absolute Effect Rates are relatively small (typically below 0.5%), they scale proportionally with the number of flipped attributes, indicating that LunguageScore reliably captures the semantic impact of trend reversals. Notably, even sequences with a single flipped term exhibited pronounced per-attribute degradation, highlighting the metric’s granularity and responsiveness. These results affirm that LunguageScore can effectively detect inconsistencies in longitudinal directionality, even when the surface fluency of the report remains intact.

## Appendix E Synthetic Report Generation Details

##### MAIRA-2 Bannur et al. ( 2024 )

At the input, we feed in a frontal chest X-ray image for the current study. If there is no frontal available for the patient, we do not generate a report. If there are multiple frontals, we randomly choose one. We also pass along a random lateral chest X-ray image for the current study, should it be available. MAIRA-2 additionally accepts the indication, technique and comparison sections. We therefore input the history for the current study in the “indication” field, if there is one. For the comparison, we input “Chest radiography dated _.” if there is a previous study, to comply with the anonymised dates in the MIMIC-CXR dataset. We do not input a technique, since this field could not be reliably extracted for the MIMIC-CXR test set. We explore two distinct ways for including prior information in the generation setup. In the standard setting, we input the ground truth reference report that is available for the previous study. This report is structured following the template “INDICATION: <prior_history> COMPARISON: <prior_comparison> FINDINGS: <prior_findings> IMPRESSION: <prior_impression>.”, where <prior_history>, <prior_impression> and <prior_findings> are all taken from the previous study’s ground truth reference report, and substituted by “N/A” if they are missing. If there is no prior study, the prior report field is set to “None”. If the previous study was the first one in the sequence, then <prior_comparison> is set to “N/A”, otherwise it is set to “Chest radiograph dated _.” In the cascaded setting, <prior_findings> is set to the findings report that was generated in the previous study (if there is one, otherwise the prior report field is set to “None”), while <prior_impression> is left blank (because MAIRA-2 only generates findings), and the other inputs remain the same. In both settings, we input the frontal view from the prior study, if there is one, and if there are multiple options, we choose the same one that was used to generate the previous report. We ask MAIRA-2 to generate the findings section for the current study, using their default settings, without grounding.

##### RGRG Tanida et al. ( 2023 ) and CVT2DistilGPT2 Nicolson et al. ( 2023 )

For both models, we input the current frontal image, once again choosing a random one when there are multiple ones, and foregoing generation when there are none. For the CVT2DistilGPT2 model, we use the variant that was trained on MIMIC-CXR. We use the default setup as suggested on the model’s Github pages. Both models generate radiological “findings” as the full report, outputting no specific “impression” section.

##### Medversa Zhou et al. ( 2024 )

Next to the current frontal image, we also fill in the additional input fields expected by Medversa, which are context, prompt, modality and task. For context, we follow the template “Age: None. Gender: None. Indication: <current_history>”. For <current_history>, we pass along the “history” section of the reference report, should it be available, and otherwise we set it to “None”. The modality and task are set to “cxr” and “report generation” respectively. All language generation parameters are left as default. The prompt is set to “Can you provide a report of <img0> with findings and impression?”. Note that this is the only model with the ability to generate an impression section, and it will therefore naturally have an advantage over the other models when we compare it to the reference report, where both the findings and impression section are included based on their availability in the ground truth.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: * The paper introduces LUNGUAGE, the benchmark specifically designed for both single-report and longitudinal structured radiology report evaluation. It addresses a key gap in temporal and schema-aligned medical language evaluation.
* The proposed LUNGUAGESCORE enables structured, interpretable, and temporally aware evaluation of generated reports, addressing the limitations of coarse or purely lexical metrics.
summary: This paper presents LUNGUAGE, a benchmark for structured and longitudinal radiology report generation. It includes 1,473 expert-annotated chest X-ray reports, with 80 capturing disease progression over time. The authors propose a two-stage framework for converting reports into schema-aligned representations and introduce LUNGUAGESCORE, a metric that evaluates entity, relation, and attribute consistency across time. This work enables fine-grained and temporally-aware assessment of radiology report generation.
weaknesses: I have previously reviewed an earlier version of this work, and I acknowledge that the authors have addressed several prior concerns, particularly in terms of annotation quality and clinical relevance. However, one major concern still remains:

While the annotation quality and clinical depth are commendable, the overall dataset scale—particularly the limited number of annotated reports and longitudinal cases—remains insufficient to support the benchmark’s claimed generalizability and broader impact. The small number of patients and annotated reports restricts the diversity of pathologies, temporal patterns, and linguistic variations, making it difficult to serve as a comprehensive resource for training or evaluating robust, general-purpose longitudinal models.

Given that this benchmark is positioned as a potential standard for evaluating structured report generation and temporal consistency, I would have expected either significantly broader coverage or stronger justification that the current scope is sufficient.

I would also be interested in seeing how other reviewers weigh the dataset’s scale versus its clinical design, to better understand the consensus on its impact and readiness for adoption.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I fully appreciate the substantial effort behind this work and its significance for longitudinal report evaluation. However, a few details remain unclear to me:

- In single-report evaluation, it is common for reports to already contain longitudinal comparison cues (e.g., “no change,” “unchanged,” etc.). How does the method interpret and evaluate such explicitly comparative attributes when assessing a single report in isolation?
- The semantic similarity metric is somewhat confusing. Are you using the fully structured phrases (e.g., “opacity–left lung–nodular–slightly increased”) directly as input for embeddings, or are you using the original unstructured free-text sentences? If structured phrases, essentially sequences of discrete terms, are embedded directly, is the resulting semantic similarity truly meaningful? In contrast, structural similarity appears more reliable and interpretable in this context.
- In the bipartite matching step, does the approach account for one-to-many alignments? For instance, a single phrase in the reference report might correspond to two or more phrases in the generated report. If such cases arise, how does the method handle them?
- The paper mentions grouping certain entities. Do the authors account for view-specific groupings? Some findings are only observable in specific radiographic views (e.g., PA vs. lateral), and longitudinal sequences may occasionally miss certain views altogether. How does the proposed method handle and evaluate such view-dependent and potentially incomplete scenarios?
- In longitudinal evaluation, some follow-up reports may be overly concise and omit entities previously mentioned (e.g., a finding described in an earlier report is absent in a later one without explicit negation). How does the method treat such omissions? Could this affect the final evaluation scores, especially if the absence is interpreted as a negative finding or simply ignored?

Suggestions for Future Work:

- While the current focus is on textual structuring, it would be a valuable extension to ground each structured phrase with corresponding bounding boxes in the chest X-ray images. This would enable more fine-grained, vision-language alignment and enhance clinical utility.
- Given the benchmark’s small size, its main role is evaluation. It would be more impactful if the authors could leverage their automated structuring pipeline to scale up this dataset for training more robust chest X-ray report generation models. But I also have a concern, if without expert validation, automated expansion may introduce hallucinations. Although the paper notes high consistency when leveraging domain-specific lexicons, medical applications demand high reliability, and unverified synthetic labels could negatively impact downstream model training. It is another challenge that warrants careful consideration.

Overall, I  hope authors will continue to expand and refine this work, which can benefit the communities.
soundness: 3
strengths: - The paper introduces LUNGUAGE, a longitudinal and structured benchmark for chest X-ray report generation. It offers fine-grained report structuring and pays specific attention to the evaluation of longitudinal reports. The dataset has been reviewed by radiologists, enhancing its clinical reliability.

- The authors release their structuring algorithm, prompting templates, and the expert-validated vocabulary, which collectively facilitate standardized, automated, and reproducible extensions of this work.

- The paper introduces LUNGUAGESCORE, integrates semantic, structural, and temporal dimensions into a unified scoring framework. This design enables more effective discrimination among models, especially in terms of their temporal consistency.
summary: This paper introduces LUNGUAGE, a fine-grained structured and longitudinal evaluation benchmark tailored for chest X-ray reports. The authors design and implement an automatic structuring framework aligned with the proposed schema, enabling entity and relation extraction and grouping across both single and longitudinal multi-report settings. The framework demonstrates relatively high agreement with human annotations. Furthermore, the paper proposes LUNGUAGESCORE, an interpretable evaluation metric that jointly assesses semantic fidelity, structural consistency, and temporal coherence, thereby establishing a systematic foundation for the evaluation of longitudinal radiology reports generation.
weaknesses: - The overall dataset size in this work is relatively small. Although a key contribution is the focus on longitudinal report sequence evaluation, the number of such longitudinal sequences is limited, which makes the evaluation somewhat insufficient. While the authors acknowledge this limitation in the conclusion, I would strongly encourage them to expand the benchmark with more cases if feasible.

- The study primarily uses the MIMIC-CXR dataset and its subsets, which exhibit a fairly homogeneous reporting style. The evaluation would be more convincing if it were extended to include an additional, distinct dataset with different reporting conventions.

- The evaluation relies on the LLM’s performance. Minor errors in entity extraction, relation identification, or grouping can cascade, since MatchScore = Semantic × (Temporal if T > 1) × Structural, leading to unstable or unreliable results.

- Several concerns regarding specific details, as outlined in **Questions**.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See Weakness
soundness: 3
strengths: 1. This paper focuses on radiology report evaluation through the extraction of structured knowledge. In addition to single-report evaluation, the framework introduces a sequential evaluation paradigm, which has been largely underexplored in previous research.

2. The paper provides a detailed pipeline for constructing an evaluation benchmark, which can serve as a valuable resource for future research in radiology report analysis.

3. The proposed evaluation framework is comprehensive, covering a broad range of aspects, from lexical to semantic similarity. Moreover, the involvement of domain experts in annotating the structured radiology reports ensures the reliability and clinical validity of the dataset.
summary: This paper presents a benchmark for structured and sequential chest x-ray interpretation. It addresses two major limitations in prior works, i.e., temporal reasoning across longitudinal reports (e.g., "no change" or "worsening" must be validated against prior findings) and 
fine-grained clinical accuracy, including attributes like location, size, and morphology, which are often lost in coarse metrics.
weaknesses: 1. Despite the comprehensive evaluation design, the structure extracted from radiology reports is primarily represented as triples in a knowledge graph. However, extensive research has already explored this representation. A higher-level view of structured reports should encompass not only triples but also observations, findings, and clinical statuses. Therefore, entity-level evaluation alone may be insufficient to capture the full complexity of radiology reports. ref: [1]

2. The LUNGUAGESCORE combines both lexical and semantic similarity. However, before applying this metric, one must extract triples from reports using external tools (LLMs in this paper). While these models demonstrate good performance in triple extraction, it might be worth exploring an alternative approach, i.e., using LLM-as-a-judge for computing LUNGUAGESCORE. This approach is simple, effective, and relies only on prompting. Moreover, scoring semantic similarity through embeddings may not fully align with radiologists’ subjective preferences (as shown in Table 2).

3. The structured evaluation closely resembles KG-based evaluation, yet its performance in single-report settings is comparable to that of RadGraph, suggesting limited improvement in this aspect.

4. Regarding temporal evaluation at the semantic level, GREEN can also detect comparisons with prior findings (e.g., “(f) Omitting a comparison detailing a change from a prior study”). According to Table 2, the proposed metric appears to underperform GREEN in this regard.

[1] Automated Structured Radiology Report Generation. https://arxiv.org/pdf/2505.24223

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. In the 3.1, there are several kinds of entities defined. Is this categorization referred from any medical textbook or handbook, or created by yourself? Any domain experts confirm that this category is reasonable? Can an entity be categorized into more than one kind?

2. In the sequential report structure, can one clinical finding in the Day 10 report be associated with more than one clinical finding in the Day 90 report? In addition, is many-to-many association between clinical findings in different sequential reports possible? If possible, how can your proposed schema represent that?
soundness: 3
strengths: A benchmark dataset of sequential radiology report is proposed, where many kinds of clinical entities and relationships are annotated. Dataset construction is described with details.
summary: A benchmark dataset of sequential radiology report is proposed, containing 1,473 reports from 230 patients, including 10 patients with sequential reports. A set of clinical entities and relationships are annotated. A new evaluation metric is proposed for generated report evaluation. 

The dataset construction details are presented, but the size of the dataset is small.
weaknesses: 1. More experimental results are needed for testing the temporal-aware report generation models on the proposed dataset.

2. The benchmark dataset is in small-scale. As the sequential interpretation claimed as the novelty or contributions, only 10 patients in the proposed dataset contains sequential reports (longitudinal trajectories).


Decision: Reject