Libra: Leveraging Temporal Images for Biomedical Radiology Analysis
Authors: Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho
Accepted at: acl25

Libra: Leveraging Temporal Images for Biomedical Radiology Analysis

Xi Zhang, Zaiqiao Meng*, Jake Lever, Edmond S. L. Ho
School of Computing Science
University of Glasgow
{X.Zhang.6}@research.gla.ac.uk
{Zaiqiao.Meng, Jake.Lever, Shu-Lim.Ho}@glasgow.ac.uk

Abstract
Radiology report generation (RRG) requires advanced medical image analysis, effective temporal reasoning, and accurate text generation. While multimodal large language models (MLLMs) align with pre-trained vision encoders to enhance visual-language understanding, most existing methods rely on single-image analysis or rule-based heuristics to process multiple images, failing to fully leverage temporal information in multi-modal medical datasets. In this paper, we introduce Libra, a temporal-aware MLLM tailored for chest X-ray report generation. Libra combines a radiology-specific image encoder with a novel Temporal Alignment Connector (TAC), designed to accurately capture and integrate temporal differences between paired current and prior images. Extensive experiments on the MIMIC-CXR dataset demonstrate that Libra establishes a new state-of-the-art benchmark among similarly scaled MLLMs, setting new standards in both clinical relevance and lexical accuracy. All source code and data are publicly available at: https://github.com/X-iZhang/Libra.

1 Introduction
Radiology reports are critical for biomedical radiology analysis, offering structured summaries of imaging studies such as chest X-rays (CXR). Commonly divided into sections like Findings, Impression, Indication, Technique, Comparison, and History (Ganeshan et al., 2018), these reports guide diagnostic and therapeutic decisions (Najjar, 2023). However, manually generating such reports is both complex and time-consuming. Automating radiology report generation (RRG) holds great promise for alleviating radiologist burnout, increasing efficiency, and improving communication (Zhang et al., 2020b). Despite this, the intricate nature of medical imaging demands precise and detailed documentation, making RRG a challenging task.

Figure 1: Examples of hallucinations in RRG using the MLLM (MAIRA-1 (Hyland et al., 2024)). (a) Single-image case: spurious references to nonexistent prior studies. (b) Temporal image case: inaccurate interpretation of temporal changes when integrating prior studies.

Recent advances in Multimodal Large Language Models (MLLMs), such as LLaVA (Liu et al., 2023) and InstructBLIP (Dai et al., 2023), have demonstrated potential in vision-language tasks. However, their performance diminishes in specialised biomedical contexts due to the significant domain shift between general-purpose and medical image-text data (Tu et al., 2023). These models often lack fine-grained detail for medical imaging tasks, resulting in surface-level understanding akin to layperson interpretations. While continued pre-training on medical datasets and domain-specific fine-tuning (e.g., LLaVA-Med (Li et al., 2023a), MedBLIP (Chen et al., 2023b)) improve performance, they still cannot fully capture the complexities of medical image analysis (Xiao et al., 2024).

One critical gap in the current MLLM-based approaches is their limited ability to incorporate temporal context, which is pivotal in clinical practice. Radiologists routinely compare current imaging results with prior studies to identify temporal changes, a process crucial for understanding dis-
ease progression and guiding treatment decisions. Indeed, the MIMIC-CXR database (Johnson et al., 2019b) reveals that 67% of patients had at least two studies over time, underscoring the need for temporal reasoning. However, most MLLMs designed for RRG tasks focus on single-image analysis, neglecting this temporal dimension (Zhang et al., 2024c). As illustrated in Figure 1, MAIRA-1 (Hyland et al., 2024) introduces hallucinated prior references in single-image analysis and misinterprets temporal changes when integrating prior studies.

1 Although recent models like MedVersa (Zhou et al., 2024) and MAIRA-2 (Bannur et al., 2024) have introduced multi-image processing, they do not explicitly model or extract temporal differences. Instead, they rely on inserting visual tokens from different studies at specific points within textual inputs and delegate the reasoning task to the LLM. Similarly, Banerjee et al. (2024) and Chaves et al. (2024) leverage GPT-4V (OpenAI et al., 2024) to eliminate hallucinated references to prior studies in the dataset but lacks dedicated mechanisms for modelling temporal progression. Additionally, existing MLLMs often rely on embeddings from the last or penultimate layer of the image encoder (Chen et al., 2023a; Zhang et al., 2024a), primarily capturing global features. However, RRG tasks require fine-grained details2 (Sloan et al., 2024), which a single-layer embedding often cannot fully represent (Jiang et al., 2024). To tackle these limitations, we enhance MLLM temporal awareness for RRG tasks by addressing two main challenges:

• Designing robust MLLM architectures that seamlessly handle prior study references in RRG.

• The scarcity of effective feature alignment projectors in MLLMs capable of handling the high-granularity requirements of downstream tasks.

To overcome these gaps, we propose Libra (Leveraging Temporal Images for Biomedical Radiology Analysis), a novel temporal-aware framework tailored for RRG tasks. Libra employs a pre-trained visual transformer encoder, RAD-DINO (Pérez-García et al., 2024), to generate robust image features, which are then refined using a new projector crafted for the temporal awareness, before being fed into the medical large language model (LLM), Meditron (Chen et al., 2023c). Through a two-stage training strategy, Libra aligns temporal visual features with the text embedding space, improving temporal coherence in RRG.

Our modular approach integrates state-of-the-art open-source pre-trained models for medical image and text processing while introducing a dedicated temporal-aware adapter to align visual and textual modalities within the embedding space. This paper makes the following contributions:

o Libra, a temporal-aware MLLM designed to model temporal references and mitigate temporal hallucinations in RRG tasks.

o Temporal Alignment Connector (TAC), comprising the Layerwise Feature Extractor (LFE) and Temporal Fusion Module (TFM), which extracts high-granularity image features from multiple encoder layers and integrates temporal references from the prior study when available.

o Extensive evaluation on the MIMIC-CXR dataset, achieving state-of-the-art results on average among similarly scaled MLLMs, with case analysis illustrating Libra’s architectural benefits.

2 Libra

2.1 Model Architecture

Our Libra model follows the standard architecture of MLLMs, such as LLaVA (Liu et al., 2023), comprising an image encoder, a text decoder and a connector module to map visual features into the text embedding space. Figure 2 shows the overall architecture of Libra. Specifically, we utilise a frozen biomedical image encoder, i.e. RAD-DINO (Pérez-García et al., 2024), a visual transformer extensively pre-trained on medical scans using the DINOv2 image-only self-supervised learning approach (Oquab et al., 2024). The text encoder is deployed by Meditron-7B (Chen et al., 2023c), which builds on Llama-2 and is further pre-trained on specialised medical corpora.

To effectively connect the image encoder and LLM, we design a novel Temporal Alignment Connector (TAC) tailored to capture and integrate temporal information from paired images taken at different time points. Meanwhile, when no prior image is available, we employ a dummy prior image, which is simply a copy of the current image, to mitigate spurious references to nonexistent scans, as shown in Figure 2 (bottom). This design enables Libra to effectively manage temporal data (e.g., stable, improved, worsening) and enhances its ability to generate accurate and coherent radiology reports.

1 Appx. A covers related work; Appx. B outlines our research objectives (e.g., the definition of temporal information).
2 E.g., severity and temporal progression of findings.
Figure 2: The overall architecture of Libra. The core component, the Temporal Alignment Connector (TAC), processes paired temporal images to enhance temporal reasoning. TAC consists of two key modules: (a) the Layerwise Feature Extractor (LFE), which aggregates multi-layer image features from the image encoder, and (b) the Temporal Fusion Module (TFM), which aligns the extracted features and integrates temporal differences before feeding them into the LLM. When no prior image is available, a dummy prior image is used to support temporal modelling, mitigate hallucinations, and prevent spurious references to nonexistent prior studies.

2.2 Temporal Alignment Connector
To address the challenges of integrating temporal information and aligning high-granularity visual features for RRG tasks, TAC bridges the image encoder and the LLM. It processes visual features from two temporal snapshots to produce a unified representation sensitive to temporal changes. As shown in Figure 2 (right), TAC includes two key components: the Layerwise Feature Extractor, which extracts high-granularity image representations, and the Temporal Fusion Module, which integrates temporal references from the prior study.

2.2.1 Layerwise Feature Extractor
To leverage abundant image feature representations encoded by a pre-trained image encoder, we extract image patch token features of all the hidden layers for a given pair of input images. By default, the RAD-DINO image encoder (Pérez-García et al., 2024) has 12 hidden layers and processes 518 × 518 input images into 14 × 14 patches, generating 1,369 patch token sequences per hidden layer. Rather than relying on a single global feature token (e.g., the [CLS] token), we collect same-dimensional patch embeddings from each layer per image, denoted as \( E_{img}^{i} \in \mathbb{R}^{N \times D_{img}} \), where \( N = 1,369 \) is the number of patch tokens and \( D_{img} \) is the embedding dimension of the image encoder.

Then, these embeddings are concatenated across all layers as \( E'_{img} = \{E_{i}^{img}\}_{i=1}^{n} \), where \( n \) is the number of hidden layers. Drawing from the progressive compression strategy in VGG (Simonyan and Zisserman, 2015), our Layerwise Feature Extractor (LFE) reduces dimensionality across layers while preserving critical information. First, we utilise Squeeze-and-Excitation (SE) Networks (Hu et al., 2019), which construct informative features by integrating both spatial and channel-wise information within local receptive fields at each layer. The SE block is applied to obtain calibrated feature representations, using GELU (Hendrycks and Gimpel, 2023) as the activation function.

Next, we employ a specialised pointwise convolution module to align the feature spaces across different layers, using a depthwise 2D convolution with filters and stride of 1, without bias. The compressed features are represented as \( A_{img} \sim Conv2d^{k}_{j}(SE^{k}_{j}(E'_{img})) \), where \( k \) is the original layer number and \( j \) is the layer number after compression. Following the size-reduction pattern of convolutional layers in VGG, the image features are compressed according to \( \{k, j\} \in \{12, 6, 3, 1\}^3 \). Through three stages of progressive compression,

3Since RAD-DINO has 12 hidden layers, the prime factorisation chain provides the factors as {12, 6, 3, 1}.
we obtain the final patch-level representation:
\[
A'_{img} = Conv2d_6^{12}(SE_6^{12}(E'_{img}))
\] (1)
\[
A''_{img} = Conv2d_8^{6}(SE_8^{6}(A'_{img}))
\] (2)
\[
A_{img} = Conv2d_1^{3}(SE_1^{3}(A''_{img}))
\] (3)

For simplicity, we use \( LFE(\cdot) \) to denote the above three stages of compression, which project a given input image \( E'_{img} \) into its feature representation of the fixed dimension, \( A_{img} \in \mathbb{R}^{1 \times N \times D_{img}} \):
\[
A_{img} = LFE(E'_{img})
\] (4)

By progressively refining each image’s representations through multiple stages, the LFE generates a unified and compact feature set suitable for temporal alignment. This design ensures that both high-granularity and global context are retained, as illustrated in (\( a \)) of Figure 2.

2.2.2 Temporal Fusion Module
The Temporal Fusion Module (TFM) is inspired by the transformer decoder and is designed to integrate temporal information by leveraging prior images as auxiliary context. It takes as input a paired set of compressed features from both the current and prior images, denoted as \( A^{\text{curr}}_{img} \) and \( A^{\text{prior}}_{img} \), respectively, which are obtained after processing through the LFE. The temporal fusion process is defined as:
\[
Z_{img} = TFM(A^{\text{curr}}_{img}, A^{\text{prior}}_{img})
\] (5)
where TFM learns to weigh the current image using prior image features, refining the representation to enhance temporal awareness. The resulting feature sequence, \( Z_{img} \in \mathbb{R}^{N \times d} \), serves as the input to the LLM, where \( N \) is the number of patch tokens and \( d \) is the hidden dimension of the LLM. This process encapsulates the temporal evolution of the patient’s condition, allowing the language model to generate accurate and contextually aware radiology reports.

Prior Image Prefix Bias
The dataset contains samples with and without a prior image. When a prior image is not available, we set \( A^{\text{prior}}_{img} = A^{\text{curr}}_{img} \). However, this “dummy prior image” is indistinguishable from a true prior in raw features. To differentiate it, we add a trainable bias, as \( b_{\text{prior}} \).

Following the attention scaling techniques for adjusting hidden space degrees of freedom with a chi-square distribution (Vaswani et al., 2017), a nonlinear scaling function amplifies higher similarity values. The cosine similarity between the current and prior images is scaled with an exponent of \( \frac{4}{\sqrt{d}} \), where \( d \) is the LLM hidden size:
\[
b'_{\text{prior}} = b_{\text{prior}} \cdot \left( \frac{\cos(A^{\text{curr}}_{img}, A^{\text{prior}}_{img}) + 1}{2} \right)^{\frac{4}{\sqrt{d}}}
\] (6)
\[
A^{\text{prior}}_{img} = A^{\text{prior}}_{img} + b'_{\text{prior}}
\] (7)

This nonlinear scaling emphasises high similarity values, modulating the influence of prior image features. When no prior image is available, the high similarity score ensures that the effect of the dummy prior is adequately represented. This adjustment prevents samples with a dummy prior image from undergoing redundant rounds of parallel multi-head self-attention during subsequent propagation through the transformer blocks, in Figure 2.

Transformer Block
The Transformer Block in TFM follows the standard Transformer design but is optimized for handling temporal image pairs. It consists of multi-head self-attention (\( Self\ Attn \)), multi-head cross-attention (\( Cross\ Attn \)), and two multi-layer perceptron (\( MLP \)) sub-layers. As illustrated in (\( b \)) of Figure 2. The paired (\( A^{\text{curr}}_{img}, A^{\text{prior}}_{img} \)) are processed with layer normalization (\( LN \)) and residual connections:
\[
T^{\text{self}}_{\text{curr}} = LN(A^{\text{curr}}_{img} + Self\ Attn(A^{\text{curr}}_{img}, A^{\text{curr}}_{img}))
\] (8)
\[
T^{\text{self}}_{\text{prior}} = LN(A^{\text{prior}}_{img} + Self\ Attn(A^{\text{prior}}_{img}, A^{\text{prior}}_{img}))
\] (9)
\[
T^{\text{cross}}_{img} = LN(T^{\text{curr}}_{\text{img}} + Cross\ Attn(T^{\text{curr}}_{\text{img}}, T^{\text{prior}}_{\text{img}}))
\] (10)
\[
T^{\text{out}}_{img} = LN(A^{\text{img}}_{img} + MLP_{\text{attn}}(T^{\text{cross}}_{img}))
\] (11)
\[
Z_{img} = MLP_{\text{final}}(T^{\text{out}}_{img})
\] (12)
where \( MLP_{\text{attn}} \) is a simple neural network composed of two fully connected layers with GELU as the activation function. After that, the features are processed through \( MLP_{\text{final}} \), a straightforward neural network consisting of four fully connected layers with the same activation function, but with hidden dimensions matching those of the LLM.

2.3 Prompt Design
To enhance Libra’s ability to perceive temporal changes and integrate medical information in RRG, we design a structured prompting strategy, consisting of a system prompt and a clinical prompt, as shown in Figure 2 (left). The system prompt enables the LLM to recognise temporal variations, while standard report sections (\( Indication\ History \), \( Comparison \), \( Technique \)) are integrated into the clinical prompt (see Appx. C for a detailed example).

The full prompt is: “Provide a detailed description of the findings in the radiology image. Following clinical context:{...}.” There are datasets, e.g. MIMIC-CXR (Johnson et al., 2019b), where the report sections are unavailable. For these datasets, we set the prompt as follows: “Provide a detailed description of the findings in the radiology image.” After tokenising and embedding prompts, the refined image features (\( Z_{img} \)) are inserted between the system prompt and clinical prompts.
2.4 Temporal-aware Training
Libra focuses on frontal-view images, either posterior-anterior (PA) or anterior-posterior (AP), and targets the Findings sections of RRG, as these contain the most direct clinical observations. It employs a two-stage training strategy, inspired by recent MLLM fine-tuning techniques (McKinzie et al., 2024), to progressively learn visual feature alignment and temporal information extraction.

Temporal Feature Alignment In the first stage, the visual encoder and LLM weights are frozen, while the TAC is trained. This stage focuses on Findings and Impression generation from paired images and performing CXR-related visual question answering (VQA) tasks to extract high-quality image representations and capture temporal changes.

Downstream Task Fine-tuning In the second stage, we apply Low-Rank Adaptation (LoRA) (Hu et al., 2021) to fine-tune the LLM on the Findings generation task, while keeping the visual encoder and TAC weights frozen. LoRA achieves performance comparable to full fine-tuning at a substantially lower computational cost. The detailed training configuration, including hyperparameter setup and computational cost, is provided in Appx. D.

3 Experiments

3.1 Task and Dataset
Task Description We focus on generating the Findings section of radiology reports for frontal CXRs, ensuring a fair comparison with prior work. The Findings section provides radiologists’ observations, encompassing both normal and abnormal findings. While additional sections like Indication and Technique primarily serve as routine records (e.g., clinical history or specific physician requests), they also assist the model in understanding temporal changes across images. Hence, we incorporate clinical instructions about the current image as prompts to guide Libra to complete the RRG task.

The most common CXR is frontal views, either PA or AP. Although lateral views are occasionally used to supplement anatomical assessments (Islam et al., 2023), they are excluded in this study to maintain consistency with previous research on RRG tasks, such as Chaves et al. (2024) and Hyland et al. (2024). Both current and prior images in our experiments exclusively utilise single frontal views.

Dataset Description Libra is trained and evaluated using the MIMIC-CXR dataset (Johnson et al., 2019b) and its derivative datasets, including Medical-Diff-VQA (Hu et al., 2023) and MIMIC-Ext-MIMIC-CXR-VQA (Bae et al., 2023). All datasets are split according to the official labels to prevent data leakage. Detailed dataset descriptions and preprocessing steps are in Appx. E.

Following the dataset scaling law utilised in multi-stage MLLM fine-tuning methods (Zhu et al., 2023), we adopt a two-stage training strategy, as noted in Sec. 2.4. The first stage trains TAC on ~1.2M CXR-image text pairs from MIMIC-CXR and its derivatives, including Findings, Impression, and VQA tasks, enabling it to learn CXR token distributions and image-text relationships. The second stage fine-tunes the model on downstream tasks, refining the LLM to align high-granularity CXR features with the Findings section of reports.

Beyond Findings section generation, the first stage incorporates Impression section and VQA tasks. The Impression section, which summarises diagnoses and proposes further investigations (Babar et al., 2021), facilitates alignment between CXRs and their textual descriptions. We use the same system and clinical prompts as for Findings, replacing ‘Findings’ with ‘Impression’. For VQA, the system prompts remain unchanged, while clinical prompts are adapted to address medical-specific questions, guiding caption generation. These VQA tasks refine the MLLM’s biomedical vocabulary usage and strengthen image-text alignment.

3.2 Evaluation Metrics
We evaluate the generated reports using lexical and radiology-specific metrics, adhering to established protocols. Lexical metrics include ROUGE-L (Lin, 2004), BLEU-{1, 4} (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and BERTScore (Zhang et al., 2020a). Radiology-specific metrics include RadGraph-F1 (Jain et al., 2021), RGer (Delbrouck et al., 2022a), F1-CheXpert (Irvin et al., 2019), CheXbert vector similarity (Smit et al., 2020a), and RadChiQ (Yu et al., 2022) version 0.

These clinical metrics typically emphasise the accuracy of medical findings, prioritising the detection of clinically relevant entities. However, they do not evaluate the model’s ability to capture temporal information. Therefore, we introduce the temporal entity F1 score (\( F_{1\text{temp}} \)) to assess this aspect. In particular, the temporal entity F1 score specifically measures the accuracy of entities related to progression over time described in the report 4.

4Full metric descriptions, including \( F_{1\text{temp}} \), are in Appx. F.
<table>
  <tr>
    <th>Metric</th>
    <th>LLaVA-Med<sup>‡</sup></th>
    <th>CheXagent<sup>‡</sup></th>
    <th>GPT-4V<sup>‡</sup></th>
    <th>Med-PaLM M</th>
    <th>LLaVA-Rad</th>
    <th>MAIRA-1</th>
    <th>Libra (%)</th>
  </tr>
  <tr>
    <th colspan="8">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.6</td>
    <td>21.5</td>
    <td>13.2</td>
    <td>27.5</td>
    <td>30.6</td>
    <td>28.9</td>
    <td>36.2 (+18.3%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>35.4</td>
    <td>16.9</td>
    <td>16.4</td>
    <td>32.3</td>
    <td>38.1</td>
    <td>39.2</td>
    <td>51.2 (+30.6%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>14.9</td>
    <td>4.7</td>
    <td><b>17.8</b></td>
    <td>11.5</td>
    <td>15.4</td>
    <td>14.2</td>
    <td>24.3 (+36.5%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>35.3</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>33.3</td>
    <td>48.7 (+38.0%)</td>
  </tr>
  <tr>
    <th colspan="8">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>19.1</td>
    <td>–</td>
    <td>–</td>
    <td><b>26.7</b></td>
    <td>–</td>
    <td>24.3</td>
    <td>32.4 (+21.3%)</td>
  </tr>
  <tr>
    <td>RRG<sub>R</sub></td>
    <td>23.8</td>
    <td>20.5</td>
    <td>13.2</td>
    <td>–</td>
    <td>29.4</td>
    <td>29.6</td>
    <td>36.9 (+25.0%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>(L)</sub></td>
    <td>3.30</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>3.10</td>
    <td>2.76 (-11.0%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>36.9</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>44.0</td>
    <td>46.3 (+5.2%)</td>
  </tr>
  <tr>
    <th colspan="8">CheXpert-F1:</th>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>42.7</td>
    <td>39.3</td>
    <td>35.5</td>
    <td>53.6</td>
    <td><b>57.3</b></td>
    <td>55.7</td>
    <td>55.3 (-3.4%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>26.9</td>
    <td>24.7</td>
    <td>20.4</td>
    <td>39.8</td>
    <td>39.5</td>
    <td>38.6</td>
    <td>40.2 (+1.1%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>43.9</td>
    <td>41.2</td>
    <td>25.8</td>
    <td><b>57.9</b></td>
    <td>57.4</td>
    <td>56.0</td>
    <td>58.9 (+1.8%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>36.3</td>
    <td>34.5</td>
    <td>19.6</td>
    <td>51.6</td>
    <td>47.7</td>
    <td>47.7</td>
    <td>52.6 (+2.0%)</td>
  </tr>
</table>

Table 1: Findings generation performance on the MIMIC-CXR test split. ‡ denotes results from Chaves et al. (2024), while “–” indicates missing data. The best performances in bold, and the second-best scores are underlined. Metrics where lower values are better are marked with ‘↓’. Percentage (%) shows improvement over the best existing model.

Temporal Entity F1  Building on the work of Bannur et al. (2023), we set a reward list comprising common radiology-related keywords indicative of temporal changes. Temporal entities are then extracted from both the ground truth (\( E_{gt} \)) and the generated reports (\( E_{gr} \)) without applying stemming or lemmatization, preserving the precision of temporal descriptions. After extraction, we compute precision (\( P_{temp} \)) and recall (\( R_{temp} \)), which are subsequently used to calculate the \( F1_{temp} \), defined as the harmonic mean of precision and recall (Van Rijsbergen, 1974), also known as the F1 score.

\[
F1_{temp} = (1 + \beta^2) \cdot \frac{P_{temp} R_{temp}}{\beta^2 P_{temp} + R_{temp}}
\] (13)

\[
P_{temp} = \frac{|E_{gt} \cap E_{gr}| + \epsilon}{|E_{gt}| + \epsilon}
\] (14)

\[
R_{temp} = \frac{|E_{gt} \cap E_{gr}| + \epsilon}{|E_{gr}| + \epsilon}
\] (15)

where \( \epsilon \) is a small value, set to a default of \( 1 \times 10^{-10} \), to prevent division by zero (it is also added to the numerator for special cases where no temporal entities are present in the ground truth).

3.3 Baselines

While the MIMIC-CXR dataset provides an “official” test split, strict comparisons with prior studies are challenging due to differences in inclusion criteria and pre-processing steps. For instance, Yu et al. (2022) and Jeong et al. (2023) included only one image per study, resulting in a test set of 1,597 samples, while Tanida et al. (2023) followed the Chest ImaGenome split (Wu et al., 2021). Such variations in test set distributions can significantly impact the reported results (Park et al., 2024). To ensure fairness, we use a widely adopted test set focused on frontal-view CXRs 5, aligned with previous studies such as MAIRA-1 (Hyland et al., 2024) and LLaVA-Rad (Chaves et al., 2024).

Recent concurrent work, such as M4CXR (Park et al., 2024), employs multi-turn chain-of-thought prompting (Wei et al., 2023) for report generation, which differs from our task setup. Additionally, we do not compare with MAIRA-2 (Bannur et al., 2024), a model designed for grounded radiology report generation incorporating lateral views and prior study reports for each subject within the input prompt. Bannur et al. (2024) emphasises a positive transfer between this distinct task setup and standard RRG, which falls beyond our study’s scope. For comparison of the latest concurrent and non-LLM-based models, see Appx. G.1 and Appx. G.2.

Considering these factors, we compared our model with state-of-the-art models, including LLaVA-Med (Li et al., 2023a), CheXagent (Chen et al., 2024b), GPT-4V (OpenAI et al., 2024), Med-PaLM M (Tu et al., 2023), LLaVA-Rad and MAIRA-1. Table 1 presents the results. As many of these models are not publicly available, we report their evaluation results from the original sources.

3.4 Results

From Table 1, Libra 6 achieves competitive results across most traditional lexical and clinical metrics, excelling in ROUGE-L, BLEU, METEOR, and RadGraph-based scores. It also leads in the radiologist-aligned RadCliQ metric and CheXbert vector similarity. In the CheXpert classification, it attains the highest Macro-F1 scores and a competitive Micro-F1. Overall, Libra demonstrates robust performance in RRG by effectively leveraging temporal information, with only minor gaps in select clinical metrics. These results highlight the effectiveness of its TAC in capturing temporal contexts and generating clinically relevant radiology reports.

5The test set includes 2,461 frontal-view samples.

6Libra was tested on single-image inputs without priors for fair comparison with models lacking temporal modelling.
<table>
  <tr>
    <th>Metric</th>
    <th>Libra-I</th>
    <th>w/o TFM</th>
    <th>w/o LFE</th>
    <th>w/o PIPB</th>
    <th>w/o TAC</th>
  </tr>
  <tr>
    <th colspan="6">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.56</td>
    <td>27.33 (-0.85%)</td>
    <td>27.21 (-1.27%)</td>
    <td>27.43 (-0.48%)</td>
    <td>26.17 (-5.04%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>34.84</td>
    <td>34.17 (-1.92%)</td>
    <td>34.21 (-1.82%)</td>
    <td>34.60 (-0.67%)</td>
    <td>33.03 (-5.20%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>11.51</td>
    <td>11.13 (-3.33%)</td>
    <td>11.11 (-3.47%)</td>
    <td>11.43 (-0.73%)</td>
    <td>10.02 (-12.98%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>35.50</td>
    <td>35.06 (-1.24%)</td>
    <td>34.96 (-1.52%)</td>
    <td>35.28 (-0.62%)</td>
    <td>33.98 (-4.28%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>55.87</td>
    <td>55.60 (-0.49%)</td>
    <td>55.49 (-0.69%)</td>
    <td>55.74 (-0.23%)</td>
    <td>54.63 (-2.22%)</td>
  </tr>
  <tr>
    <th>Fl<sub>temp</sub></th>
    <th>26.63</th>
    <th>25.96 (-2.51%)</th>
    <th>26.21 (-1.57%)</th>
    <th>26.58 (-0.18%)</th>
    <th>25.39 (-4.65%)</th>
  </tr>
  <tr>
    <th colspan="6">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>22.52</td>
    <td>22.20 (-1.42%)</td>
    <td>22.03 (-2.19%)</td>
    <td>22.35 (-0.74%)</td>
    <td>21.51 (-4.48%)</td>
  </tr>
  <tr>
    <td>RGEr</td>
    <td>27.32</td>
    <td>26.89 (-1.59%)</td>
    <td>26.72 (-2.19%)</td>
    <td>27.09 (-0.84%)</td>
    <td>25.97 (-4.96%)</td>
  </tr>
  <tr>
    <td>RadClIQ<sub>0</sub> (↓)</td>
    <td>3.10</td>
    <td>3.12 (-0.65%)</td>
    <td>3.12 (-0.65%)</td>
    <td>3.11 (-0.32%)</td>
    <td>3.15 (-1.61%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>42.02</td>
    <td>41.57 (-1.07%)</td>
    <td>41.37 (-1.54%)</td>
    <td>41.92 (-0.24%)</td>
    <td>40.93 (-2.59%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>52.48</td>
    <td>51.74 (-1.42%)</td>
    <td>51.68 (-1.53%)</td>
    <td>52.13 (-0.67%)</td>
    <td>51.13 (-2.57%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>36.87</td>
    <td>36.04 (-2.25%)</td>
    <td>36.12 (-2.03%)</td>
    <td>36.14 (-1.97%)</td>
    <td>35.85 (-2.76%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>56.63</td>
    <td>55.37 (-2.23%)</td>
    <td>55.79 (-1.49%)</td>
    <td>55.87 (-1.34%)</td>
    <td>54.51 (-3.74%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>49.33</td>
    <td>47.76 (-3.18%)</td>
    <td>47.82 (-3.06%)</td>
    <td>47.98 (-2.75%)</td>
    <td>47.22 (-4.28%)</td>
  </tr>
</table>

Table 2: Results of ablation experiments for the Temporal Alignment Connector. ‘↓’ indicates that lower is better. Values in (%) indicate the percentage decrease compared with the Libra-I.

4 Ablation Studies

We conducted ablation studies on Libra’s key components, evaluating module and dataset configurations. All experiments were performed on the MIMIC-CXR test split for the Findings generation, with prior images included by default and consistent hyperparameters during training and inference.

Does the Temporal Alignment Connector improve model performance? To evaluate the impact of TAC on Libra’s performance in RRG, we used a model initialised with the RAD-DINO (Pérez-García et al., 2024) image encoder, TAC, and Meditron-7b (Chen et al., 2023c) as the LLM. The baseline (<i>Libra-I</i>) was conducted by fine-tuning only the TAC for the <i>Findings</i> generation task. As shown in Table 2, we performed ablation studies by progressively removing different TAC components, including the Temporal Fusion Module (TFM), Layerwise Feature Extractor (LFE), Prior Image Prefix Bias (PIPBI), and the entire TAC.

Removing TFM restricted the model to single-image processing, akin to LLaVA (Liu et al., 2023), but with a four-layer MLP for aligning image features with the LLM’s hidden dimensions. Without LFE, the model used the penultimate layer of the encoder. Removing PIPB excluded the mechanism for differentiating true and dummy prior images. Finally, removing the entire TAC left the model reliant solely on the image encoder and LLM.

The results indicate that removing any TAC submodule leads to performance declines across all metrics compared to <i>Libra-I</i>. TFM removal caused a notable drop in the F1<sub>temp</sub> score (↓>2%), highlighting its role in capturing temporal information. LFE removal significantly decreased RadGraph-related scores, underscoring its importance in extracting detailed image features. PIPB removal impacted clinical metrics more than lexical metrics, indicating its role in enhancing clinical relevance. Complete TAC removal led to substantial declines in all metrics, demonstrating its critical role in integrating image details and temporal information. The evaluation confirms that TAC plays a vital role in improving Libra’s ability to generate high-quality, temporally aware radiology reports.

For additional ablation studies exploring TAC’s contributions, including its impact under general-domain and radiology-specific pre-trained models, its performance after the second training stage, its robustness under extended fine-tuning and diverse conditions, an analysis of whether incorporating temporal information improves Libra’s performance in RRG tasks, refer to Appx. H.1–H.6.

Are additional Impression and VQA datasets necessary during the feature alignment? To assess the impact of incorporating additional datasets during the first stage of training, we compared a model (<i>Libra-f</i>) trained solely on the <i>Findings</i> data with Libra, which also used <i>Impression</i> and VQA data for feature alignment, as shown in Table 3.

After the first stage, Libra outperformed <i>Libra-f</i> in lexical metrics but showed a slight decline in clinical scores. This decline stems from VQA tasks emphasizing fine-grained, grounded descriptions rather than holistic findings. VQA focuses on individual symptoms, whereas <i>Findings</i> integrates multiple normal and abnormal observations, affecting F1<sub>temp</sub> by reducing identified temporal entities.
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="2">Stage: 1</th>
    <th colspan="2">Stage: 2</th>
  </tr>
  <tr>
    <th>Libra-f</th>
    <th>Libra</th>
    <th>Libra-f</th>
    <th>Libra</th>
  </tr>
  <tr>
    <th colspan="5">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.56</td>
    <td>27.27<sup>▼</sup></td>
    <td>35.31</td>
    <td>36.66<sup>▲</sup></td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>34.84</td>
    <td>41.24<sup>▲</sup></td>
    <td>49.92</td>
    <td>51.25<sup>▲</sup></td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>11.51</td>
    <td>13.59<sup>▲</sup></td>
    <td>23.05</td>
    <td>24.54<sup>▲</sup></td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>35.50</td>
    <td>39.44<sup>▲</sup></td>
    <td>47.99</td>
    <td>48.90<sup>▲</sup></td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>55.87</td>
    <td>56.00<sup>▲</sup></td>
    <td>61.28</td>
    <td>62.50<sup>▲</sup></td>
  </tr>
  <tr>
    <td>F1<sub>temp</sub></td>
    <td>26.63</td>
    <td>24.80<sup>▼</sup></td>
    <td>33.52</td>
    <td>35.34<sup>▲</sup></td>
  </tr>
  <tr>
    <th colspan="5">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>22.52</td>
    <td>20.45<sup>▼</sup></td>
    <td>30.77</td>
    <td>32.87<sup>▲</sup></td>
  </tr>
  <tr>
    <td>RGEr</td>
    <td>27.32</td>
    <td>25.19<sup>▼</sup></td>
    <td>35.44</td>
    <td>37.27<sup>▲</sup></td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0</sub> (.)</td>
    <td>3.10</td>
    <td>3.31<sup>▲</sup></td>
    <td>2.83</td>
    <td>2.72<sup>▲</sup></td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>42.02</td>
    <td>35.33<sup>▼</sup></td>
    <td>45.32</td>
    <td>46.85<sup>▲</sup></td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td colspan="4"></td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>52.48</td>
    <td>43.63<sup>▼</sup></td>
    <td>54.11</td>
    <td>55.87<sup>▲</sup></td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>36.87</td>
    <td>25.68<sup>▼</sup></td>
    <td>37.16</td>
    <td>40.38<sup>▲</sup></td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>56.63</td>
    <td>49.75<sup>▲</sup></td>
    <td>58.76</td>
    <td>60.07<sup>▲</sup></td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>49.33</td>
    <td>40.40<sup>▼</sup></td>
    <td>51.99</td>
    <td>53.75<sup>▲</sup></td>
  </tr>
</table>

Table 3: Ablation results for dataset configurations. ▲ denotes improvement, while ▼ indicates decline.

In the second stage, fine-tuning on Findings restored balance, further improving performance. These results indicate that additional datasets enhance Libra’s RRG ability, while second-stage fine-tuning ensures well-rounded report generation.

5 Performance Analysis

We qualitatively assess Libra’s ability to generate temporally consistent radiology reports.

Cases without Prior Image As shown in Figure 3 (a), Libra produced detailed descriptions beyond the ground truth, identifying “sternal wires” and their type. This demonstrates its capability to deliver clinically relevant information without spurious referencing nonexistent prior studies.

Cases with Prior Image In Figure 3 (b), new abnormalities such as pleural effusion and pneumonia appeared in the current image. Without a prior image, Libra correctly described the present findings but did not infer disease progression, avoiding spurious references while still suggesting further investigations. When the prior image was considered, Libra effectively captured these progressive changes, provided detailed descriptions, and explicitly referenced the comparison. This facilitated a clear understanding of temporal changes and more accurate descriptions of disease progression.

Evaluating Temporal Consistency To assess temporal reasoning, we swapped image order, using the prior image as the current image and vice versa. The generated report then reflected an improved patient condition, aligning with the reversed input sequence but contradicting the ground truth of the original current image. Notably, the report closely resembled the original description of the prior image, as shown at the bottom of Figure 3 (b). This indicates that Libra can effectively adapt to both temporal contexts, generating accurate and contextually consistent reports that simulate the conditions of standard clinical practice.

Figure 3: Radiological symptoms, while temporal changes are in red. Key highlights presented in bold. Heatmap analysis is available in Appx. I.

![Radiological symptoms, while temporal changes are in red. Key highlights presented in bold.](page_1012_186_388_388.png)

6 Conclusion

In this study, we introduced Libra, a temporal-aware multimodal large language model tailored for chest X-ray report generation tasks. Libra employs a two-stage training framework, leveraging a radiology-specific image encoder and language model connected via the Temporal Alignment Connector, enabling seamless integration of visual and textual modalities. Trained solely on the open-access MIMIC-CXR dataset (Johnson et al., 2019b), Libra demonstrates notable performance gains across key metrics compared to similarly scaled models. Through qualitative and quantitative analysis, we showed that Libra effectively utilises temporal relationships between current and prior scans, addressing challenges such as hallucinations in referencing prior studies. This highlights Libra’s ability to generate clinically accurate and temporally consistent radiology reports, setting a new paradigm for multimodal medical AI research.

Future work will focus on expanding Libra’s clinical applicability by incorporating diverse imaging modalities and enhancing temporal reasoning capabilities, and extending it in an agentic way.
Limitations

Despite Libra’s ability to model temporal paired images for radiology report generation (RRG), certain limitations remain. First, Libra relies on single prior images for temporal modelling, whereas clinical practice often involves multiple prior scans with varied intervals and angles. Extending the model to handle multiple temporally sequenced images remains an open challenge. Second, our study is based on a single-source dataset with inherent biases in patient demographics and imaging protocols, which may limit generalizability across broader clinical settings. Lastly, while Libra is designed for CXR-based RRG, its applicability to other imaging modalities (e.g., CT, MRI) and integration with structured medical knowledge remains unexplored. For a detailed discussion of these limitations and future directions, see Appx. J.

Ethics Statement

This work presents Libra, a model designed to enhance radiology report generation by integrating temporal and visual information. While Libra has the potential to improve clinical workflows, reduce radiologist workload, and enhance diagnostic consistency, its deployment must be approached with caution to ensure ethical and responsible use.

Our research exclusively utilises the publicly available and “de-identified” MIMIC-CXR dataset (Johnson et al., 2019b) under its licensing constraints and in accordance with official guidelines, ensuring adherence to ethical and privacy standards under the CITI “Data or Specimens Only Research” certification. By relying solely on open datasets, we prioritise transparency and reproducibility, aligning with best practices in ethical AI research.

This work is intended to support, not replace, medical professionals, ensuring it serves as a complementary tool within clinical practice. While the societal implications are largely positive, further validation across diverse patient populations and healthcare systems is necessary to address potential biases inherent in the dataset. Additionally, it is crucial to mitigate the risks of over-reliance on AI systems, which could inadvertently undermine human oversight or exacerbate healthcare disparities.

Future efforts will aim to extend the model’s capabilities to encompass multiple imaging modalities and broader datasets, ensuring greater generalisability, fairness, and adaptability across diverse clinical settings.

References

Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering . In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6077–6086, Los Alamitos, CA, USA. IEEE Computer Society.

Zaheer Babar, Twan van Laarhoven, Fabio Massimo Zanzotto, and Elena Marchiori. 2021. Evaluating diagnostic content of ai-generated radiology reports of chest x-rays. Artificial Intelligence in Medicine, 116:102075.

Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric I-Chao Chang, Tackeun Kim, and Edward Choi. 2023. Ehrrqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Preprint, arXiv:2310.18652.

Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, and Pranav Rajpurkar. 2024. Direct preference optimization for suppressing hallucinated prior exams in radiology report generation. Preprint, arXiv:2406.06496.

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.

Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Anton Schwaighofer, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, Felix Meissen, Mercy Ranjit, Shaury Srivastav, Julia Gong, Fabian Falck, Ozan Oktay, Anja Thieme, Matthew P. Lungren, Maria Teodora Wetscherek, Javier Alvarez-Valle, and Stephanie L. Hyland. 2024. Maira-2: Grounded radiology report generation. Preprint, arXiv:2406.04449.

Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, and Ozan Oktay. 2023. Learning to exploit temporal structure for biomedical vision-language processing. Preprint, arXiv:2301.04558.

Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayá. 2020. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis, 66:101797.

Yiming Cao, Lizhen Cui, Lei Zhang, Fuqiang Yu, Zhen Li, and Yonghui Xu. 2023. Mmtm: Multi-modal memory transformer network for image-report consistent medical report generation. Proceedings
of the AAAI Conference on Artificial Intelligence, 37(1):277–285.

Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, Hany Awadalla, Julia Gong, Houdong Hu, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Yu Gu, Cliff Wong, Mu Wei, and 8 others. 2024. Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation. Preprint, arXiv:2403.08002.

Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiuyi Chen, Jing Shi, Shuang Xu, and Bo Xu. 2023a. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38–56.

Qiuqiu Chen, Xinyue Hu, Zirui Wang, and Yi Hong. 2023b. Medblip: Bootstrapping language-image pre-training from 3d medical images and texts. Preprint, arXiv:2305.10799.

Weixing Chen, Yang Liu, Ce Wang, Jiarui Zhu, Shen Zhao, Guanbin Li, Cheng-Lin Liu, and Liang Lin. 2024a. Cross-modal causal intervention for medical report generation. Preprint, arXiv:2303.09117.

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakaheirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. 2023c. Meditron-70b: Scaling medical pretraining for large language models. Preprint, arXiv:2311.16079.

Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021. Cross-modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5904–5914, Online. Association for Computational Linguistics.

Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439–1449, Online. Association for Computational Linguistics.

Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2022. Generating radiology reports via memory-driven transformer. Preprint, arXiv:2010.16056.

Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blanckenmeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, Emily B. Tsai, Andrew Johnston, Cameron Olsen, Tanishq Mathew Abraham, Sergios Gatidis, Akshay S. Chaudhari, and Curtis Langlotz. 2024b. Chexagent: Towards a foundation model for chest x-ray interpretation. Preprint, arXiv:2401.12208.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, and 48 others. 2022. Palm: Scaling language modeling with pathways. Preprint, arXiv:2204.02311.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructclip: Towards general-purpose vision-language models with instruction tuning. Preprint, arXiv:2305.06500.

Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluetghen, Emily Tsai, Omar Almus, and Curtis Langlotz. 2022a. Improving the factual correctness of radiology report generation with semantic rewards. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4348–4360, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluetghen, Emily Tsai, Omar Almus, and Curtis P. Langlotz. 2022b. Improving the factual correctness of radiology report generation with semantic rewards. Preprint, arXiv:2210.12186.

Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2016. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–310.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805.

Xinpeng Ding, Yongqiang Chu, Renjie Pi, Hualiang Wang, and Xiaomeng Li. 2024. HiA: Towards Chinese Multimodal LLMs for Comparative High-Resolution Joint Diagnosis . In proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, volume LNCS 15012. Springer Nature Switzerland.

Emil Fischer. 1894. Einfluss der configuration auf die wirkung der enzyme. Berichte der deutschen chemischen Gesellschaft, 27(3):2985–2993.
Dhakshinamoorthy Ganeshan, Phuong-Anh Thi Duong, Linda Probyn, Leon Lenchik, Tatum A McArthur, Michele Retrouvey, Emily H Ghobadi, Stephane L Desouches, David Pastel, and Isaac R Francis. 2018. Structured reporting in radiology. Academic radiology, 25(1):66–73.

Tiancheng Gu, Dongnan Liu, Zhiyuan Li, and Weidong Cai. 2023. Complex organ mask guided radiology report generation. Preprint, arXiv:2311.02329.

Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). Preprint, arXiv:1606.08415.

Wenjun Hou, Yi Cheng, Kaishuai Xu, Wenjie Li, and Jiang Liu. 2023a. RECAP: Towards precise radiology report generation via dynamic disease progression reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2134–2147. Singapore. Association for Computational Linguistics.

Wenjun Hou, Kaishuai Xu, Yi Cheng, Wenjie Li, and Jiang Liu. 2023b. Organ: Observation-guided radiology report generation via tree reasoning. Preprint, arXiv:2306.06466.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. 2019. Squeeze-and-excitation networks. Preprint, arXiv:1709.01507.

Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M. Summers, and Yingying Zhu. 2023. Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 4156–4165, New York, NY, USA. Association for Computing Machinery.

Zhongzhen Huang, Xiaofan Zhang, and Shaoting Zhang. 2023. Kiut: Knowledge-injected transformer for radiology report generation. Preprint, arXiv:2306.11345.

Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Mercy Ranjit, Anton Schwaighofer, Fernando Pérez-García, Valentina Salvatelli, Shaury Srivastav, Anja Thieme, Noel Codella, Matthew P. Lungren, Maria Teodora Wetscherek, Ozan Oktay, and Javier Alvarez-Valle. 2024. Maira-1: A specialised large multimodal model for radiology report generation. Preprint, arXiv:2311.13668.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Preprint, arXiv:1901.07031.

S. K. M Shakekul Islam, MD Abdullah Al Nasim, Ismail Hossain, Md Azim Ullah, Kishor Datta Gupta, and Md Monjur Hossain Bhuiany. 2023. Introduction of medical imaging modalities. Preprint, arXiv:2306.01022.

Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. Preprint, arXiv:2106.14463.

Jaehwan Jeong, Katherine Tian, Andrew Li, Sina Hartung, Fardad Behzadi, Juan Calle, David Osayande, Michael Pohlen, Subathra Adithan, and Pranav Rajpurkar. 2023. Multimodal image-text matching improves retrieval-based chest x-ray report generation. Preprint, arXiv:2303.17579.

Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. 2024. From clip to dino: Visual encoders shout in multi-modal large language models. Preprint, arXiv:2310.08825.

Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. 2019a. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. Preprint, arXiv:1901.07042.

Alistair E W Johnson, David J Stone, Leo A Celii, and Tom J Pollard. 2018. The mimic code repository: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association, 25(1):32–39.

Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. 2019b. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317.

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau mann, Hofung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Preprint, arXiv:2306.00890.

Mingjie Li, Bingqian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang. 2023b. Dynamic graph enhanced contrastive learning for chest x-ray report generation. Preprint, arXiv:2303.10323.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Fenglin Liu, Shen Ge, and Xian Wu. 2021a. Competence-based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3001–3012, Online. Association for Computational Linguistics.

Fenglin Liu, Xian Wu, Shen Ge, Wei Fan, and Yuexian Zou. 2021b. Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13748–13757, Los Alamitos, CA, USA. IEEE Computer Society.

Fenglin Liu, Changchang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. 2021c. Contrastive attention for automatic chest X-ray report generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 269–280, Online. Association for Computational Linguistics.

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest x-ray report generation. Preprint, arXiv:1904.02633.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.

Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3242–3250.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dutter, Dhruvi Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, and 13 others. 2024. Mm1: Methods, analysis and insights from multimodal llm pre-training. Preprint, arXiv:2403.09611.

Xin Mei, Rui Mao, Xiaoyan Cai, Libin Yang, and Erik Cambria. 2024. Medical report generation via multimodal spatio-temporal fusion. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 4699–4708, New York, NY, USA. Association for Computing Machinery.

Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz, and Dan Jurafsky. 2021a. Improving factual completeness and consistency of image-to-text radiology report generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288–5304, Online. Association for Computational Linguistics.

Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P. Langlotz, and Dan Jurafsky. 2021b. Improving factual completeness and consistency of image-to-text radiology report generation. Preprint, arXiv:2010.10042.

Reabal Najjar. 2023. Redefining radiology: a review of artificial intelligence integration in medical imaging. Diagnostics, 13(17):2760.

Aaron Nicolson, Jason Dowling, and Bevan Koopman. 2023. Improving chest X-ray report generation by leveraging warm starting. Artificial Intelligence in Medicine, 144:102633.

Aaron Nicolson, Jason Dowling, and Bevan Koopman. 2024. Longitudinal data and a semantic similarity reward for chest x-ray report generation. Preprint, arXiv:2307.09758.

Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer. 2021. Progressive transformer-based generation of radiology reports. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2824–2832, Punta Cana, Dominican Republic. Association for Computational Linguistics.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Maxime Oquab, Timothée Darcel, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, and 7 others. 2024. Dinov2: Learning robust visual features without supervision. Preprint, arXiv:2304.07193.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.

Jonggwon Park, Soobum Kim, Byungmu Yoon, Jihun Hyun, and Kyoyun Choi. 2024. M4cxt: Exploring multi-task potentials of multi-modal large language models for chest x-ray interpretation. Preprint, arXiv:2408.16213.
Linus Pauling, Robert B. Corey, and H. R. Branson. 1951. The structure of proteins: Two hydrogen-bonded helical configurations of the polypeptide chain. Proceedings of the National Academy of Sciences, 37(4):205–211.

Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Nassir Navab, and Matthias Keicher. 2023. Radiolog: A large vision-language model for radiology report generation and conversational assistance. Preprint, arXiv:2311.18681.

Fernando Pérez-García, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Matthew P. Lungren, Maria Wetscherek, Noel Codella, Stephanie L. Hyland, Javier Alvarez-Valle, and Ozan Oktay. 2024. Radino: Exploring scalable medical image encoders beyond text supervision. Preprint, arXiv:2401.10815.

Han Qin and Yan Song. 2022. Reinforced cross-modal alignment for radiology report generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 448–458, Dublin, Ireland. Association for Computational Linguistics.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. Preprint, arXiv:1910.02054.

Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-Critical Sequence Training for Image Captioning . In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179–1195, Los Alamitos, CA, USA. IEEE Computer Society.

Santosh Sanjeev, Fadillah Adamyah Maani, Arsen Abzhanov, Vijay Ram Papineni, Ibrahim Almakky, Bartlomiej W. Papiez, and Mohammad Yaqub. 2024. Tibix: Leveraging temporal information for bidirectional x-ray and report generation. Preprint, arXiv:2403.13343.

Francesco Dalla Serra, Chaoyang Wang, Fani Deligianni, Jeffrey Dalton, and Alison Q O’Neil. 2023. Controllable chest x-ray report generation from longitudinal representations. Preprint, arXiv:2310.05881.

Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. Preprint, arXiv:1409.1556.

Phillip Sloan, Philip Clatworthy, Edwin Simpson, and Majid Mirmehdi. 2024. Automated radiology report generation: A review of recent advances. IEEE Reviews in Biomedical Engineering, page 1–20.

Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Parreek, Andrew Ng, and Matthew Lungren. 2020a. Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500–1519, Online. Association for Computational Linguistics.

Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Parreek, Andrew Y Ng, and Matthew P Lungren. 2020b. Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint arXiv:2004.09167.

Xiao Song, Xiaodan Zhang, Junzhong Ji, Ying Liu, and Pengxu Wei. 2022. Cross-modal contrastive attention model for medical report generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2388–2397, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2023. Interactive and explainable region-guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7433–7442.

Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornblith, David Fleet, Philip Mansfield, Sushant Prakash, Renee Wong, Sunny Virmani, and 13 others. 2023. Towards generalist biomedical ai. Preprint, arXiv:2307.14334.

Cornelis Joost Van Rijsbergen. 1974. Foundation of evaluation. Journal of Documentation, Volume 30(Issue 4):365–373.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164.

Fuying Wang, Shenghui Du, and Lequan Yu. 2024. HerGen: Elevating radiology report generation with longitudinal data. Preprint, arXiv:2407.15158.

Jun Wang, Abhir Bhalerao, and Yulan He. 2022. Crossmodal prototype driven network for radiology report generation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXV, pages 563–579. Springer.

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, and Ronald M. Summers. 2018. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. Preprint, arXiv:1801.04334.
Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023. Metrtransformer: Radiology report generation by transformer with multiple learnable expert tokens. Preprint, arXiv:2304.02211.

James D Watson and Francis HC Crick. 1953. Molecular structure of nucleic acids: a structure for deoxyribose nucleic acid. Nature, 171(4356):737–738.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.

Joy T. Wu, Nkechinyere N. Agu, Ismini Lourentzou, Arjun Sharma, Joseph A. Paguio, Jasper S. Yao, Edward C. Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, Leo A. Celi, and Mehdi Moradi. 2021. Chest imagenome dataset for clinical reasoning. Preprint, arXiv:2108.00316.

Hanguang Xiao, Feizhong Zhou, Xingye Liu, Tianqi Liu, Zhipeng Li, Xin Liu, and Xiaoxuan Huang. 2024. A comprehensive survey of large language models and multimodal large language models in medicine. Information Fusion, 117:102888.

Shuxin Yang, Xian Wu, Shen Ge, S. Kevin Zhou, and Li Xiao. 2022a. Knowledge matters: Chest radiology report generation with general and specific knowledge. Medical Image Analysis, 80:102510.

Shuxin Yang, Xian Wu, Shen Ge, S. Kevin Zhou, and Li Xiao. 2022b. Radiology report generation with a learned knowledge base and multi-modal alignment. Preprint, arXiv:2112.15011.

Di You, Fenglin Liu, Shen Ge, Xiaoxia Xie, Jing Zhang, and Xian Wu. 2021. Aligntransformer: Hierarchical alignment of visual regions and disease tags for medical report generation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III, page 72–82, Berlin, Heidelberg. Springer-Verlag.

Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y. Ng, Curtis P. Langlotz, Vasantha Kumar Venugopal, and Pranav Rajpurkar. 2022. Evaluating progress in automatic chest x-ray radiology report generation. medRxiv.

Chenlu Zhan, Yu Lin, Gaoang Wang, Hongwei Wang, and Jian Wu. 2024. Medm2g: Unifying medical multi-modal generation via cross-guided diffusion with visual invariant. Preprint, arXiv:2403.04290.

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024a. Vision-language models for vision tasks: A survey. Preprint, arXiv:2304.00685.

Ke Zhang, Hanliang Jiang, Jian Zhang, Qingming Huang, Jianping Fan, Jun Yu, and Weidong Han. 2024b. Semi-supervised medical report generation via graph-guided hybrid feature consistency. Trans. Multi., 26:904–915.

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Matthew P. Lungren, Tristan Naumann, and 2 others. 2024c. Biomedclip: a multimodal biomedical foundation model pre-trained from fifteen million scientific image-text pairs. Preprint, arXiv:2303.00915.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675.

Yixiao Zhang, Xiaossong Wang, Ziyue Xu, Qihang Yu, Alan Yuille, and Daguang Xu. 2020b. When radiology report generation meets knowledge graph. Preprint, arXiv:2002.08277.

Hong-Yu Zhou, Subathra Adithan, Julián Nicolás Acosta, Eric J. Topol, and Pranav Rajpurkar. 2024. A generalist learner for multifaceted medical image interpretation. Preprint, arXiv:2405.07988.

Hong-Yu Zhou, Chenyu Lian, Liansheng Wang, and Yizhou Yu. 2023. Advancing radiograph representation learning with masked record modeling. Preprint, arXiv:2301.13155.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Preprint, arXiv:2304.10592.
Appendix Contents

A  Related Work .................................................. 16
    A.1  Radiology Report Generation ................................. 16
    A.2  LLM-based Model ............................................. 16
    A.3  non-LLM-based Model ......................................... 16
    A.4  Radiological Image Representation ........................... 17

B  Research Objectives ............................................ 17
    B.1  Temporal Information ....................................... 17
    B.2  Research Aims ............................................... 17
    B.3  Research Scope .............................................. 17

C  Prompt Example ................................................ 18

D  Training Configuration ......................................... 18

E  Datasets Description ........................................... 19

F  Evaluation Metrics ............................................. 20
    F.1  Lexical Metrics ............................................ 20
    F.2  Clinical Metrics ........................................... 20
    F.3  Temporal Entity F1 ......................................... 21

G  Analysis of Concurrent Work and Non-LLM-based Models ............ 21
    G.1  Discussion on Performance with Radiology Foundation Models ... 21
    G.2  Discussion on Performance with non-LLM-based Models .......... 22

H  Additional Ablation Studies .................................... 23
    H.1  Impact of Temporal Information on Libra in RRG ............... 23
    H.2  Impact of the Temporal Alignment Connector under General-Domain Pre-trained Models ... 24
    H.3  Impact of the Temporal Alignment Connector After the Second-Stage Fine-tuning .... 24
    H.4  Robustness Evaluation of the Temporal Alignment Connector .... 25
    H.5  Impact of Radiology-Specific Pre-trained Models on Libra ...... 25
    H.6  Incremental Component Analysis ............................. 26

I  Heatmap Analysis and Temporal Feature Representation ............. 27

J  Extended Discussion on Limitations ............................. 29
A  Related Work

A.1 Radiology Report Generation

Radiology report generation (RRG) aims to address the long-tail distribution of observations in chest X-rays (CXRs) and produce fine-grained descriptions of clinical findings, making it a key objective in automated medical imaging analysis (Wang et al., 2018).

Early RRG systems relied on recurrent neural networks (RNNs) (Liu et al., 2019), which have since been largely replaced by transformer-based architectures (Miura et al., 2021b; Chen et al., 2022), including large language models (LLMs) such as PaLM (Chowdhery et al., 2022) and Vicuna-7B (Chiang et al., 2023). These models excel at language generation, offering substantial improvements in fluency and factual accuracy.

To further enhance clinical accuracy, some methods incorporate reinforcement learning (RL) to optimise for task-specific rewards, such as capturing “clinically relevant” features (Liu et al., 2019; Irvin et al., 2019) or maintaining logical consistency (Miura et al., 2021a; Delbrouck et al., 2022a). However, these approaches often rely on external tools like CheXbert (Smit et al., 2020a) or RadGraph (Jain et al., 2021), adding complexity to the optimisation process.

Recent advancements in LLMs have shown that plain auto-regressive language modelling can achieve strong performance in RRG tasks. However, RL-based objectives and task-specific optimisations remain complementary, offering additional opportunities for improvement. Research on leveraging temporal information in RRG tasks can be broadly categorised into LLM-based and non-LLM-based methods, each presenting distinct advantages and challenges.

A.2 LLM-based Model

LLM-based models have achieved significant success in the RRG task, primarily due to advancements in visual instruction tuning (Liu et al., 2023). Structurally, these models (Li et al., 2023a; Chaves et al., 2024; Hyland et al., 2024; Zhou et al., 2024; Park et al., 2024) typically consist of an image encoder and an adapter that connects the encoder’s outputs to the LLM. Most existing adapters use single-layer hidden representations (e.g., the last or penultimate layer) from pre-trained image encoders, limiting their ability to integrate features from multiple images effectively.

In end-to-end training, LLM-based models handle multiple image inputs by concatenating them with textual prompts, forming a composite input to the LLM. For instance, the input format is often structured as “<Current Image Placeholder> + <Prior Image Placeholder> + <Prompt>”. However, this approach provides limited guidance on the relationship between the images within the prompt. Ding et al. (2024) proposed the High-Resolution Instruction-Aware Adapter (HIA) to refine image-text representations, improving the model’s ability to follow textual prompts with multiple images. While this enhances instruction adherence, it does not explicitly model relationships between paired images.

In contrast to this vanilla approach, Libra explicitly models temporal relationships in paired images through its Temporal Alignment Connector (TAC). Instead of simply concatenating images in the LLM’s latent space, TAC leverages all hidden-layer features from the image encoder to provide richer feature representations. By directly modelling temporal dynamics, Libra enables more precise and context-aware radiology report generation.

A.3 non-LLM-based Model

Non-LLM-based models typically employ transformer encoder-decoder architectures or their variants, which often require separate training for individual modules. These approaches handle “single-” and “double-” image inputs by symbolically differentiating tasks and employing distinct architectures tailored for each input type. Additionally, they frequently incorporate extra information such as prior reports, symptom labels, and knowledge graphs.

For instance, Serra et al. (2023) uses symbolic alignment in its Longitudinal Projection Module along with a separately trained BERT-based (Devlin et al., 2019) text generator. RECAP (Hou et al., 2023a) implements a two-stage training process: classification tasks followed by report generation, leveraging a transformer encoder-decoder with symbolic task differentiation. TiBiX (Sanjeev et al., 2024) incorporates causal attention layers and learnable padding tokens to handle cases without prior images, while BioViL-T (Bannur et al., 2023) is a self-supervised vision-language training framework that features a CNN–Transformer hybrid multi-image encoder trained jointly with a BERT-based text model.

On one hand, the difference in model parameter sizes, and on the other, as LLM-based models gen-
erally outperform other types of models (i.e. non-LLM-based) in the RRG task, papers on non-LLM-based models or those using small language models (SLMs) typically do not compare their methods with LLM-based approaches. Nonetheless, we conducted comparisons and discussions to reaffirm this observation, as detailed in Appx. G.2.

A.4 Radiological Image Representation

Radiology-specific pre-trained image encoder models are essential for RRG tasks due to the unique characteristics of radiological images, which fall outside the distribution of general-domain image models (Pérez-García et al., 2024).

Several notable advancements have been made in this domain. Zhou et al. (2023) proposed Masked Record Modeling (MRM), a unified framework combining self-supervision with radiology report supervision to enhance radiograph representation learning. Similarly, BioViL-T (Bannur et al., 2023) employs a CNN-Transformer hybrid architecture to model multimodal relationships and leverage temporal structures for tasks such as disease progression classification and report generation. In addition, BiomedCLIP (Zhang et al., 2024c) is a multimodal biomedical foundational model pretrained across diverse biomedical tasks.

RAD-DINO (Pérez-García et al., 2024) is a medical image encoder that employs a pure image-based self-supervised learning approach from DINOv2 (Oquab et al., 2024) for continuous pretraining, focusing exclusively on image data to avoid the limitations of text supervision. Recent works have extensively applied RAD-DINO to RRG tasks, including MAIRA-2 (Bannur et al., 2024) and M4CXR (Park et al., 2024). Notably, Pérez-García et al. (2024) demonstrated that RAD-DINO outperforms other image encoders in RRG tasks.

Building on this evidence, our model incorporates RAD-DINO as its image encoder to ensure high-quality radiological image representations, providing a robust foundation for downstream RRG tasks.

B Research Objectives

B.1 Temporal Information

Temporal changes are critical for understanding disease progression. In radiology, paired images and their corresponding reports document subtle evolutions of symptoms over time. This temporal information is often captured by comparing current scans with prior ones to highlight symptom evolution or newly identified findings.

The relative positioning of scans within the timeline determines the extent of temporal information. Therefore, the relative timing between scans is key: when the prior scan is recent, reported changes tend to be minimal; conversely, an older prior scan reveals more pronounced differences.

Importantly, while temporal context enriches the diagnostic narrative, it does not alter the factual observations present in the current scan—it merely provides additional layers of interpretative insight.

B.2 Research Aims

This study aims to enhance radiology report generation (RRG) by effectively incorporating temporal information into the modelling process. In clinical practice, chest X-ray (CXR) analysis often depends on comparing the current scan with the prior image to capture disease progression and evolution. Our primary objective is to leverage these temporal cues to generate more accurate, context-aware radiological reports that faithfully reflect both stable conditions and clinically significant changes.

Unlike previous LLM-based models (discussed in Appx. A.2), which depend on the LLM to infer temporal information solely from text, our approach explicitly models temporal relationships at the architectural level. Inspired by the principle “structure determines function” (Fischer, 1894; Pauling et al., 1951; Watson and Crick, 1953), we introduce the Temporal Alignment Connector (TAC), a dedicated module designed to capture temporal dynamics. Details are provided in Sec. 2.2.

B.3 Research Scope

This study focuses on frontal chest X-rays, treating each examination per image while incorporating a single prior image as an auxiliary input when available. Rather than modelling patient-level longitudinal history, our goal is to generate a report for the current image while leveraging temporal information from one preceding scan. To ensure fairness in benchmarking, Libra was evaluated on single-image inputs without priors (see Table 1). Yet, temporal information remains implicitly present through several factors:

• Explicit temporal states (e.g., “stable” or “unstable”) are frequently described in reports.
• Latent temporal progression exists in datasets, as prior studies influence diagnostic phrasing.
• The absence of a prior image itself constitutes a temporal scenario, representing an extreme case where the patient’s condition is assumed stable due to a lack of comparative reference.

Our model can effectively handle scenarios with limited temporal information in the RRG task. For instance, in a case where a patient has two scans taken just milliseconds apart, the current and prior images would be nearly identical, as no pathological changes would manifest within such a short interval. This extreme scenario demonstrates how the model handles clinical practice under limited temporal information. In such cases, the correct diagnosis for this minimal interval would be that the patient’s condition is “stable”; our model should then generate a report reflecting this stability. When no prior image is available, we employ a dummy prior image (a copy of the current image) to maintain input consistency and mitigate spurious references to nonexistent priors.

However, in clinical practice, patients often undergo multiple prior scans, sometimes from different orientations, providing a more complex temporal context. This lies beyond the scope of our current study, and a detailed discussion of such scenarios is provided in Appx. J.

C Prompt Example

We selected examples from the MIMIC-CXR (Johnson et al., 2019b) dataset and synthesised them using GPT-4 (OpenAI et al., 2024) to ensure ethical compliance, as illustrated in Table 4. Following the rule-based approach by Hyland et al. (2024), we extracted key sections from the report of the current image. Each example combines a fixed system prompt with a dynamic clinical prompt tailored to the current scan. We utilised four clinical instructions from the original report: {Indication}, {History}, {Comparison}, and {Technique}. In contrast, MAIRA-2 (Bannur et al., 2024), which incorporates prior image reports, our approach focuses exclusively on the current image’s context, maintaining a clear distinction from prior study information of the report.

<table>
  <tr>
    <th>Original Radiology Report</th>
  </tr>
  <tr>
    <td><i>EXAMINATION:</i> Chest (Portable AP)<br><i>INDICATION:</i> Dyspnea and cough, right-sided back pain.<br><i>HISTORY:</i> Intubation with pulmonary edema.<br><i>COMPARISON:</i> Chest radiographs on ___ and CT chest without contrast on ___.<br><i>TECHNIQUE:</i> Portable upright chest radiograph.<br><i>FINDINGS:</i> In comparison with the prior study, there are diffuse bilateral pulmonary opacifications, more prominent on the right. These findings could indicate severe pulmonary edema, but superimposed pneumonia or developing ARDS cannot be excluded. Monitoring and support devices are appropriately positioned.<br><i>IMPRESSION:</i> Diffuse bilateral pulmonary opacities, more pronounced on the right, consistent with severe pulmonary edema. Superimposed pneumonia or evolving ARDS remains a possibility. Recommend clinical correlation and continued monitoring.</td>
  </tr>
</table>

<table>
  <tr>
    <th>Prompt Content</th>
  </tr>
  <tr>
    <td>[System prompt]: {<br>The assistant specialised in comparing Chest X-ray images, identifying differences, and noting temporal changes.}<br>+ <Image Representation Placeholder> +<br>[Clinical prompt]: {<br>Provide a detailed description of the findings in the radiology image. Following clinical context:<br>Indication: Dyspnea and cough, right-sided back pain.<br>History: Intubation with pulmonary edema.<br>Comparison: Chest radiographs on ___ and CT chest without contrast on ___.<br>Technique: Portable upright chest radiograph.}</td>
  </tr>
</table>

Table 4: Examples of Libra’s system and clinical prompts for <i>Findings</i> section generation in RRG task.

computational infrastructure equipped with A6000 GPU (48GB of memory) and using DeepSpeed optimization (Rajbhandari et al., 2020) with ZeRO-2 for stage 1 and ZeRO-3 for stage 2, and BF16 precision is enabled.

A cosine learning rate scheduler is employed, starting with a warm-up phase of 0.03. In the first stage of training, we run for 1 epoch (~385 hours) with a learning rate of \( 2 \times 10^{-5} \). In the second stage, the model is trained for 3 epochs (~213 hours) at the same learning rate. The LoRA (Hu et al., 2021) parameters are set to \( r = 128 \) and \( alpha = 256 \). The final checkpoint for all runs is selected based on the observation of the minimum loss on the evaluation dataset throughout the training process.

Note: Prior works, especially in the medical domain, typically employ full model fine-tuning for RRG tasks. However, due to hardware constraints, we can only adopt a lightweight training technique for parameter-efficient adaptation. As a result, our approach may underperform full model fine-tuning strategies in the second stage, despite maintaining computational efficiency.
<table>
  <tr>
    <th>Dataset</th>
    <th>Task Type</th>
    <th># Samples</th>
    <th>% Has Prior</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Train (%)</th>
    <th>Valid (%)</th>
    <th>Test (%)</th>
    <th>Train</th>
    <th>Valid</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>MIMIC-CXR</td>
    <td>Findings</td>
    <td>162 955 (13.43%)</td>
    <td>1286 (0.88%)</td>
    <td>2461 (2.78%)</td>
    <td>58.43</td>
    <td>60.11</td>
    <td>86.03</td>
  </tr>
  <tr>
    <td></td>
    <td>Impression</td>
    <td>199 548 (16.45%)</td>
    <td>1671 (1.14%)</td>
    <td>2343 (2.64%)</td>
    <td>64.85</td>
    <td>67.09</td>
    <td>85.49</td>
  </tr>
  <tr>
    <td rowspan="6">Medical-Diff-VQA</td>
    <td>Difference</td>
    <td>131 563 (10.85%)</td>
    <td>16 372 (1.17%)</td>
    <td>16 389 (1.848%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Abnormality</td>
    <td>116 394 (9.59%)</td>
    <td>14 512 (9.90%)</td>
    <td>14 515 (16.37%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Presence</td>
    <td>124 654 (10.28%)</td>
    <td>15 549 (10.61%)</td>
    <td>15 523 (17.51%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>View</td>
    <td>44 970 (3.71%)</td>
    <td>5696 (3.89%)</td>
    <td>5599 (6.31%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Location</td>
    <td>67 187 (5.54%)</td>
    <td>8510 (5.81%)</td>
    <td>8496 (9.58%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Level</td>
    <td>53 728 (4.43%)</td>
    <td>6722 (4.59%)</td>
    <td>6846 (7.72%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Type</td>
    <td>22 067 (1.82%)</td>
    <td>2709 (1.85%)</td>
    <td>2702 (3.05%)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td rowspan="7">MIMIC-Ext-MIMIC-CXR-VQA</td>
    <td>Presence</td>
    <td>109 455 (9.02%)</td>
    <td>26 153 (17.84%)</td>
    <td>4566 (5.15%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Anatomy</td>
    <td>37 952 (3.13%)</td>
    <td>10 210 (6.96%)</td>
    <td>1963 (2.21%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Attribute</td>
    <td>49 948 (4.12%)</td>
    <td>13 111 (8.94%)</td>
    <td>2578 (2.91%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Abnormality</td>
    <td>60 692 (5.00%)</td>
    <td>16 109 (10.99%)</td>
    <td>3199 (3.61%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Size</td>
    <td>16 000 (1.32%)</td>
    <td>4000 (2.73%)</td>
    <td>705 (0.80%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Plane</td>
    <td>7992 (0.66%)</td>
    <td>1992 (1.36%)</td>
    <td>386 (0.44%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Gender</td>
    <td>7992 (0.66%)</td>
    <td>1992 (1.36%)</td>
    <td>396 (0.45%)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>Multi-type</td>
    <td>1 213 097 (100%)</td>
    <td>146 594 (100%)</td>
    <td>88 669 (100%)</td>
    <td>64.73</td>
    <td>49.09</td>
    <td>83.67</td>
  </tr>
</table>

Table 5: Datasets used for training and evaluating Libra include statistics on the proportion of samples that contain prior images. The first stage uses the full dataset, while the second stage fine-tunes for downstream tasks.

E Datasets Description

MIMIC-CXR (Johnson et al., 2019b) This is a large, publicly accessible dataset comprising 377,110 DICOM images across 227,835 studies, each accompanied by a radiology report (Johnson et al., 2019b). For images, we use the commonly available JPEG files from MIMIC-CXR-JPG (Johnson et al., 2019a), rather than the original DICOM files, and we preprocess the dataset to exclude non-AP/PA scans. For each report, we extract the Findings, Impression, Indication, History, Comparison, and Technique sections using rule-based heuristics supported by the official MIMIC code repository (Johnson et al., 2018).

For the Findings section generation task, studies without extractable Findings are discarded, while other missing sections are permitted. The same approach is applied to the Impression section generation task. In all our experiments, we adhere to the official MIMIC-CXR dataset split.

Meanwhile, we retrieve prior images by following the chronological order of studies as indicated by the official labels, selecting the closest prior study as the reference image. It is important to note that, to prevent data leakage between the train, validation, and test sets, prior images are retrieved only from within the same split.

Medical-Diff-VQA (Hu et al., 2023) This dataset is a derivative of the MIMIC-CXR dataset, focused on identifying differences between pairs of main and reference images. The data split adheres to the original labelling, ensuring no data leakage occurs. In total, this dataset comprises 700,703 question-answer pairs derived from 164,324 main-reference image pairs. As shown in Table 5, the questions are divided into seven categories: abnormality, location, type, view, presence, and difference.

Each pair consists of a main (current) image and a reference (prior) image, both taken from different studies of the same patient. The reference image is always selected from an earlier visit, with the main image representing the later visit. Of the seven question types, the first six types focus on the main image, while the “difference” questions involve both images.

MIMIC-Ext-MIMIC-CXR-VQA (Bae et al., 2023) This dataset extends MIMIC-CXR for VQA tasks tailored to CXRs. It includes questions generated from 48 unique templates covering seven content types: presence, anatomy, attribute, abnormality, size, plane, and gender, as shown in Table 5. Each template was developed with the guidance of board-certified medical experts to ensure clinical relevance, addressing both standard medical VQA content and more complex logical scenarios. In total, the dataset consists of 377,391 unique entries. Since annotations are based on single images, the current image serves as a dummy prior image for all entries in our experiment.

For this study, we carefully selected datasets that provide complete reports and temporal information (i.e., prior images) to ensure alignment with our research objectives (see Appx. B) for the RRG task. After thoroughly evaluating other datasets, we found them unsuitable for the following reasons:
CheXpert (Irvin et al., 2019) This dataset includes annotated scans with label-specific annotations rather than full medical reports. While useful for training image encoders or annotation models, it is not appropriate for the RRG task, which requires complete diagnostic reports.

PadChest (Bustos et al., 2020) Although it includes reports and corresponding prior images, its reports are in Spanish, placing cross-language training beyond the scope of our model.

IU-Xray (Demner-Fushman et al., 2016) This dataset lacks patient-level metadata and prior study information, which is critical for our focus on temporal information in chest X-rays.

Chest ImaGenome Dataset (Wu et al., 2021) Although derived from MIMIC-CXR (Johnson et al., 2019b), it does not follow the official split, raising concerns about potential data leakage between training, validation, and test sets.

Meanwhile, the following two datasets were processed using GPT-4 (OpenAI et al., 2024) to eliminate hallucinated references to prior exams. While this prevents erroneous comparisons, it also removes essential temporal information originally present in the reports, potentially affecting tasks that rely on temporal reasoning.

LLaVA-Rad MIMIC-CXR Dataset (Chaves et al., 2024) This dataset was refined using GPT-4 (OpenAI et al., 2024) through a structured text-cleaning pipeline. The process involved: (1) correcting typographical errors and split words, (2) removing redundant or repeated phrases to improve clarity, (3) eliminating explicit temporal references (e.g., “Compared to the prior study, no significant interval change was noted”) to ensure the report focuses exclusively on the current image, and (4) restructuring content into standardised sections, including Indication, Findings, and Impression.

ReXPref-Prior Dataset (Banerjee et al., 2024) A modified version of MIMIC-CXR (Johnson et al., 2019b) in which GPT-4 (OpenAI et al., 2024) systematically removes all references to prior exams from both the Findings and Impression sections. While this adjustment prevents spurious prior-study references, it also eliminates crucial temporal context, limiting its suitability for applications requiring longitudinal assessment of disease progression.

F Evaluation Metrics

F.1 Lexical Metrics
We employed standard natural language generation metrics to quantify the overlap between generated and reference reports. Specifically, ROUGE-L (Lin, 2004) measures the length of the longest common subsequence between the generated and reference reports. BLEU-\{1,4\} (Papineni et al., 2002) calculates n-gram precision and applies a brevity penalty to discourage overly short predictions. METEOR (Banerjee and Lavie, 2005), computes the weighted harmonic mean of unigram precision and recall, with an additional penalty for fragmenting consecutive word sequences. Finally, we report BERTScore (Zhang et al., 2020a), which leverages pre-trained contextual embeddings from BERT (Devlin et al., 2019) to match words in candidate and reference sentences based on cosine similarity. We used default parameters for all of these evaluation metrics.

F.2 Clinical Metrics
For radiology-specific metrics, we used as many of the same evaluation scores as possible from previous studies (Tu et al., 2023; Hyland et al., 2024; Bannur et al., 2024; Chaves et al., 2024), including the following:

RadGraph-based metrics RadGraph model (Jain et al., 2021) is designed to parse radiology reports into structured graphs. These graphs consist of clinical entities, which include references to anatomy and observations, as well as the relationships between these entities. This structured representation enables a more detailed and systematic analysis of radiology reports, facilitating downstream tasks such as information extraction, report generation, and clinical decision support.

These include RadGraph-F1 (Jain et al., 2021), which computes the overlap in entities and relations separately and then reports their average. And a variant of it, RGER (Delbrouck et al., 2022b), which matches entities based on their text, type, and whether they have at least one relation7.

CheXpert F1 This set of metrics utilizes the CheXbert automatic labeler (Smit et al., 2020a) to extract “present”, “absent”, or “uncertain” labels for each of the 14 CheXpert pathologies (Irvin et al., 2019) from the generated reports and their

7RGER is implemented as F1RadGraph with reward=partial by the radgraph package.
<table>
  <tr>
    <th>Ground Truth</th>
    <th>Candidate</th>
    <th>ROUGE-L</th>
    <th>RadGraph-F1</th>
    <th>\( F_{1_{\text{temp}}} \)</th>
  </tr>
  <tr>
    <td>Compare with prior scan, pleural effusion has <span style="color:red">worsened</span>.</td>
    <td>The pleural effusion has progressively <span style="color:red">worsened</span> since previous scan.</td>
    <td>0.47</td>
    <td>0.86</td>
    <td><b>1.0</b></td>
  </tr>
  <tr>
    <td></td>
    <td>The pleural effusion is noted again on the current scan.</td>
    <td>0.22</td>
    <td>0.80</td>
    <td><b>0.0</b></td>
  </tr>
</table>

Table 6: Evaluation of candidate reports using the Temporal Entity F1 score (\( F_{1_{\text{temp}}} \)). Descriptions of temporal changes are marked.

corresponding references. In line with prior work, we report CheXpert-F1 for all 14 classes, as well as for the 5 most common findings in CXR reports, referring to these as “[Macro/Micro]-F1-[5/14]”.

CheXbert vector similarity We also employ CheXbert vector similarity (Yu et al., 2022), which calculates the cosine similarity between the embeddings of the generated and reference reports after processing them through the CheXbert model (Smit et al., 2020a).

RadCliQ In addition, we utilise RadCliQ (Radiology Report Clinical Quality) (Yu et al., 2022), a composite metric that combines RadGraph-F1 and BLEU scores in a linear regression model to estimate the number of errors that radiologists are likely to detect in a report. To maintain consistency with previous research, we use version 0 of it.

Both the CheXbert vector similarity, RadCliQ_0, and RadGraph-F1 metrics are calculated using the code released by Yu et al. (2022).

F.3 Temporal Entity F1

We introduced \( F_{1_{\text{temp}}} \), a metric specifically designed to detect temporal entities reflecting changes over time. Unlike traditional lexical or radiology-specific metrics, \( F_{1_{\text{temp}}} \) evaluates the quality of temporal information in radiology reports.

As shown in Table 6, the differences in lexical (ROUGE-L (Lin, 2004)) and clinical (RadGraph-F1 (Jain et al., 2021)) metrics between the two candidates are relatively smaller compared to the \( F_{1_{\text{temp}}} \) score. This demonstrates that Temporal Entity F1 effectively captures and evaluates the quality of temporal information in radiology reports, distinguishing it more accurately than other standard metrics in the context of temporal information descriptions.

G Analysis of Concurrent Work and Non-LLM-based Models

G.1 Discussion on Performance with Radiology Foundation Models

As shown in Table 7, these models belong to the category of radiology foundation models.

DaDialog (Pellegrini et al., 2023) is a conversational MLLM designed for a broad range of dialogue-based medical assistance tasks. To enhance structured findings extraction, it employs the publicly available CheXbert model (Smit et al., 2020b) to extract symptom labels from scans, facilitating a structured representation of findings.

MedVersa (Zhou et al., 2024) and M4CXR (Park et al., 2024) support a diverse set of tasks, including medical report generation, visual grounding, and visual question answering. These models aim to provide general-purpose multimodal medical assistance by leveraging vision-language pre-training strategies.

MAIRA-2 (Bannur et al., 2024) specialises in grounded radiology report generation, which differs from traditional report generation tasks by requiring explicit image-level localization of findings and symptoms. Grounded radiology reporting, as defined by Bannur et al. (2024), structures the report as a list of sentences, where each sentence: (1) is linked to zero or more spatial image annotations, and (2) describes at most a single finding from an image. To support this task, MAIRA-2 introduces a custom dataset, explicitly designed to provide structured annotations aligning textual descriptions with spatial regions of interest in radiological images. This approach contrasts with conventional RRG models that generate unstructured free-text reports.

It is worth noting that the inference sets differ slightly across these models. Additionally, all these models leverage supplementary radiology information, such as lateral view scans, prior study reports, or both (as detailed in Appx. A.2), to enhance their performance in radiology-related tasks.
<table>
  <tr>
    <th>Metric</th>
    <th>RaDialog</th>
    <th>MedVersa</th>
    <th>MAIRA-2</th>
    <th>M4CXR</th>
    <th>Libra (%)</th>
  </tr>
  <tr>
    <th colspan="6">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>31.6</td>
    <td>–</td>
    <td><b>38.4</b></td>
    <td>28.5</td>
    <td>36.7 (-4.4%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>39.2</td>
    <td>–</td>
    <td>46.5</td>
    <td>33.9</td>
    <td>51.3 (10.3%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>14.8</td>
    <td>17.8</td>
    <td>23.4</td>
    <td>10.3</td>
    <td>24.5 (4.7%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>–</td>
    <td>–</td>
    <td>42.0</td>
    <td>–</td>
    <td>48.9 (16.4%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>–</td>
    <td>49.7</td>
    <td>–</td>
    <td>–</td>
    <td>62.5 (25.8%)</td>
  </tr>
  <tr>
    <th colspan="6">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>–</td>
    <td>28.0</td>
    <td><b>34.6</b></td>
    <td>21.8</td>
    <td>32.9 (-4.9%)</td>
  </tr>
  <tr>
    <td>RG<sub>ER</sub></td>
    <td>–</td>
    <td>–</td>
    <td><b>39.7</b></td>
    <td>–</td>
    <td>37.6 (-5.3%)</td>
  </tr>
  <tr>
    <td>RadCliQ(1.)</td>
    <td>–</td>
    <td>2.7</td>
    <td>2.6</td>
    <td>–</td>
    <td>2.7 (-3.8%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>–</td>
    <td>46.4</td>
    <td>50.6</td>
    <td>–</td>
    <td>46.9 (-7.3%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>39.2</td>
    <td>–</td>
    <td>58.5</td>
    <td><b>60.6</b></td>
    <td>55.9 (-7.8%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>–</td>
    <td>–</td>
    <td><b>42.7</b></td>
    <td>40.0</td>
    <td>40.4 (-5.4%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>–</td>
    <td>–</td>
    <td>58.9</td>
    <td><b>61.8</b></td>
    <td>60.1 (-2.8%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>–</td>
    <td>–</td>
    <td>51.5</td>
    <td>49.5</td>
    <td>53.8 (4.5%)</td>
  </tr>
</table>

Table 7: Findings section generation performance of Libra and the latest concurrent work. The best performances are highlighted in bold, and the second-best scores are underlined. ‘-’ indicates that lower values are better. ‘–’ indicates missing data. The percentage (%) indicates the improvement over the best existing model.

Despite these considerations, Libra achieves the highest scores on most lexical metrics, including BLEU-{1, 4}, METEOR, and BERTScore, while trailing slightly behind MAIRA-2 on ROUGE-L. In clinical metrics, Libra predominantly ranks second, just behind the best-performing model. For clinical metrics, Libra consistently ranks second, just behind the top-performing model. In metrics that evaluate medical entities and their relationships, such as RadGraph-F1, RG<sub>ER</sub>, and RadCliQ, Libra also ranks second. Similarly, Libra comes second in the CheXbert vector embedding score. However, in the CheXpert metrics, Libra ranks first in Macro-F1 for the 5-class subset, with only a slight dip in the Micro-F1 score for the 14-class subset.

Incorporating lateral images and prior study reports could enhance clinical scores. Additionally, strategies like chain-of-thought reasoning and grounded report generation further improve performance in RRG tasks. Looking ahead, we plan to develop model architectures that can automatically adapt to multiple tasks and diverse scenarios, enabling more efficient handling of additional radiological information.

G.2 Discussion on Performance with non-LLM-based Models

To compare with non-LLM-based models, we selected evaluation metrics commonly used in these studies. These include BLEU-{1, 2, 3, 4} (Papineni et al., 2002), METEOR (MTR) (Banerjee and Lavie, 2005), and ROUGE-L (R-L) (Lin, 2004). For clinical metrics, we report CheXbert (Irvin et al., 2019), Precision (P), Recall (R), and F<sub>1</sub>.

Baseline For performance evaluation, we compare our model with the following baselines: ST (Vinyals et al., 2015), ATT2IN (Rennie et al., 2017), ADAATT (Lu et al., 2017), TopDown (Anderson et al., 2018), R2Gen (Chen et al., 2020), R2GenCMN (Chen et al., 2021), M<sup>2</sup>TR (Noorolahzadeh et al., 2021), CMCL (Liu et al., 2021a), PPKED (Liu et al., 2021b), AlignTransformer (You et al., 2021), CA (Liu et al., 2021c), LKBMA (Yang et al., 2022b), KnowMAT (Yang et al., 2022a), XPRONET (Wang et al., 2022), CMM-RL (Qin and Song, 2022), RAMT (Zhang et al., 2024b), CMCA (Song et al., 2022), KiUT (Huang et al., 2023), DCL (Li et al., 2023b), MMTN (Cao et al., 2023), METrans (Wang et al., 2023), ORGAN (Hou et al., 2023b), COMG (Gu et al., 2023), BioViL-T (Bannur et al., 2023), RGRG (Tandia et al., 2023), RECAP (Hou et al., 2023a), CvT2DistilGPT2 (Nicolson et al., 2023), VLCI (Chen et al., 2024a), TiBiX (Sanjeev et al., 2024), MedM2G (Zhan et al., 2024), MS-TF (Mei et al., 2024), HERGen (Wang et al., 2024), CXRMeta (Nicolson et al., 2024).

To ensure fairness, Libra also utilizes prior images, aligning with other models that leverage prior images or additional information. As demonstrated in Table 8, Libra, similar to other LLM-based models, consistently outperforms non-LLM-based models. This advantage is largely attributed to advancements in LLMs and visual instruction tuning (Liu et al., 2023), enabling multimodal large language models (MLLMs) to achieve superior performance in RRG tasks.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">Lexical Metrics</th>
    <th colspan="3">Clinical Metrics</th>
  </tr>
  <tr>
    <th>B-1</th>
    <th>B-2</th>
    <th>B-3</th>
    <th>B-4</th>
    <th>MTR</th>
    <th>R-L</th>
    <th>P</th>
    <th>R</th>
    <th>F<sub>1</sub></th>
  </tr>
  <tr><td>ST<sup>†</sup></td><td>29.9</td><td>18.4</td><td>12.1</td><td>8.4</td><td>12.4</td><td>26.3</td><td>24.9</td><td>20.3</td><td>20.4</td></tr>
  <tr><td>ATT2IN<sup>‡</sup></td><td>32.5</td><td>20.3</td><td>13.6</td><td>9.6</td><td>13.4</td><td>27.6</td><td>32.3</td><td>23.9</td><td>20.4</td></tr>
  <tr><td>ADAATT<sup>‡</sup></td><td>29.9</td><td>18.5</td><td>12.4</td><td>8.8</td><td>11.8</td><td>26.6</td><td>26.8</td><td>18.6</td><td>18.1</td></tr>
  <tr><td>TopDown<sup>‡</sup></td><td>31.7</td><td>19.5</td><td>13.0</td><td>9.2</td><td>12.8</td><td>26.7</td><td>32.0</td><td>23.1</td><td>23.8</td></tr>
  <tr><td>R2Gen</td><td>35.3</td><td>21.8</td><td>14.5</td><td>10.3</td><td>14.2</td><td>27.0</td><td>33.3</td><td>27.3</td><td>27.6</td></tr>
  <tr><td>R2GenCMN</td><td>35.3</td><td>21.8</td><td>14.8</td><td>10.6</td><td>14.2</td><td>27.8</td><td>34.4</td><td>27.5</td><td>27.8</td></tr>
  <tr><td>XPRONET</td><td>34.4</td><td>21.5</td><td>14.6</td><td>10.5</td><td>13.8</td><td>27.9</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>CMCL</td><td>34.4</td><td>21.7</td><td>14.0</td><td>9.7</td><td>13.3</td><td>28.1</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>PPKED</td><td>36.0</td><td>22.4</td><td>14.9</td><td>10.6</td><td>14.9</td><td>28.4</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>AlignTransformer</td><td>37.8</td><td>23.5</td><td>15.6</td><td>11.2</td><td>15.8</td><td>28.3</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>CA</td><td>35.0</td><td>21.9</td><td>15.2</td><td>10.9</td><td>15.1</td><td>28.3</td><td>35.2</td><td>29.8</td><td>30.3</td></tr>
  <tr><td>LKBMA</td><td>38.6</td><td>23.7</td><td>15.7</td><td>11.1</td><td>--</td><td>27.4</td><td>42.0</td><td>33.9</td><td>35.2</td></tr>
  <tr><td>M<sup>†</sup>TR</td><td>37.8</td><td>23.2</td><td>15.4</td><td>10.7</td><td>14.5</td><td>27.2</td><td>24.0</td><td>42.8</td><td>30.8</td></tr>
  <tr><td>KnowMAT</td><td>36.3</td><td>22.8</td><td>15.6</td><td>11.5</td><td>--</td><td>28.4</td><td>45.8</td><td>34.8</td><td>37.1</td></tr>
  <tr><td>RAMT</td><td>36.2</td><td>22.9</td><td>15.7</td><td>11.3</td><td>15.3</td><td>28.4</td><td>38.0</td><td>34.2</td><td>33.5</td></tr>
  <tr><td>CMM-RL</td><td>38.1</td><td>23.2</td><td>15.5</td><td>10.9</td><td>15.1</td><td>28.7</td><td>34.2</td><td>29.4</td><td>29.2</td></tr>
  <tr><td>CMCA</td><td>36.0</td><td>22.7</td><td>15.6</td><td>11.7</td><td>14.8</td><td>28.7</td><td>44.4</td><td>29.7</td><td>35.6</td></tr>
  <tr><td>KiUT</td><td>39.3</td><td>24.3</td><td>15.9</td><td>11.3</td><td>16.0</td><td>28.5</td><td>37.1</td><td>31.8</td><td>32.1</td></tr>
  <tr><td>DCL</td><td>--</td><td>--</td><td>--</td><td>10.9</td><td>15.0</td><td>28.4</td><td>47.1</td><td>35.2</td><td>37.3</td></tr>
  <tr><td>MMTN</td><td>37.9</td><td>23.8</td><td>15.9</td><td>11.6</td><td>16.1</td><td>28.3</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>METrans</td><td>25.0</td><td>16.9</td><td>12.4</td><td>15.2</td><td>--</td><td>29.1</td><td>36.4</td><td>30.9</td><td>31.1</td></tr>
  <tr><td>ORGAN</td><td>38.6</td><td>25.6</td><td>17.2</td><td>12.3</td><td>16.2</td><td>29.3</td><td>41.6</td><td>41.8</td><td>38.5</td></tr>
  <tr><td>COMG</td><td>36.3</td><td>23.5</td><td>16.7</td><td>12.4</td><td>12.8</td><td>29.0</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>MedM2G</td><td>41.2</td><td>26.9</td><td>17.9</td><td>14.2</td><td>--</td><td>30.9</td><td>--</td><td>--</td><td>--</td></tr>
  <tr><td>CvT2DistilGPT2</td><td>39.2</td><td>24.5</td><td>16.9</td><td>12.4</td><td>15.3</td><td>28.5</td><td>35.9</td><td>41.2</td><td>38.4</td></tr>
  <tr><td>RGRG</td><td>37.3</td><td>24.9</td><td>17.5</td><td>12.6</td><td>16.8</td><td>26.4</td><td>46.1</td><td>47.5</td><td>44.7</td></tr>
  <tr><td>BioViL-T</td><td>--</td><td>--</td><td>--</td><td>9.2</td><td>--</td><td>29.6</td><td>--</td><td>--</td><td>17.5</td></tr>
  <tr><td>VLCI</td><td>40.0</td><td>24.5</td><td>16.5</td><td>11.9</td><td>15.0</td><td>28.0</td><td>48.9</td><td>34.0</td><td>40.1</td></tr>
  <tr><td>TiBiX</td><td>32.4</td><td>23.4</td><td>18.5</td><td>15.7</td><td>16.2</td><td>33.1</td><td>30.0</td><td>22.4</td><td>25.0</td></tr>
  <tr><td>RECAP</td><td>42.9</td><td>26.7</td><td>17.7</td><td>12.5</td><td>16.8</td><td>28.8</td><td>38.9</td><td>44.3</td><td>39.3</td></tr>
  <tr><td>MS-TF</td><td>43.6</td><td>27.5</td><td>18.4</td><td>12.9</td><td>17.7</td><td>30.5</td><td>--</td><td>--</td><td>41.1</td></tr>
  <tr><td>CXRMate</td><td>--</td><td>--</td><td>--</td><td>7.9</td><td>--</td><td>26.2</td><td>43.8</td><td>34.9</td><td>35.7</td></tr>
  <tr><td>HERGen</td><td>39.5</td><td>24.8</td><td>16.9</td><td>12.2</td><td>15.6</td><td>28.5</td><td>41.5</td><td>30.1</td><td>31.7</td></tr>
  <tr><th>Libra</th><th>51.3</th><th>38.0</th><th>30.0</th><th>24.5</th><th>48.9</th><th>36.7</th><th>59.7</th><th>52.5</th><th>55.9</th></tr>
</table>

Table 8: Findings Generation Performance of Libra and non-LLM-based Models. The best performances are highlighted in bold, and the second-best scores are underlined. † denotes results from Chen et al. (2021), and “–” indicates missing data. These results are taken from the best performances reported in their original papers.

H Additional Ablation Studies

H.1 Impact of Temporal Information on Libra in RRG

Temporal information is embedded in paired images and referenced in the corresponding radiology reports, capturing changes over time through references to prior symptoms and their progression, as discussed in Appx. B.1. As shown in Table 5, **86%** of the test data includes prior images, providing a solid foundation for evaluating the impact of temporal information.

During training, Libra integrates the ability to perceive and utilise temporal information into its architecture. To evaluate whether Libra effectively leverage temporal information during inference, we assess its performance using prior images when available as references to determine their impact on the overall capability.

In Table 9, the inclusion of prior images sub-

<table>
  <tr>
    <th>Metric</th>
    <th colspan="2">Libra</th>
  </tr>
  <tr>
    <th></th>
    <th>w/o prior</th>
    <th>w/ prior (%)</th>
  </tr>
  <tr>
    <th colspan="3">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>36.17</td>
    <td>36.66 (+1.35%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>51.20</td>
    <td>51.25 (+0.10%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>24.33</td>
    <td>24.54 (+0.86%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>48.69</td>
    <td>48.90 (+0.43%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>61.94</td>
    <td>62.50 (+0.90%)</td>
  </tr>
  <tr>
    <th colspan="3">F<sub>1temp</sub></th>
  </tr>
  <tr>
    <td></td>
    <td>32.72</td>
    <td>35.34 (+8.00%)</td>
  </tr>
  <tr>
    <th colspan="3">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>32.42</td>
    <td>32.87(+1.39%)</td>
  </tr>
  <tr>
    <td>RGE<sub>ER</sub></td>
    <td>36.92</td>
    <td>37.57(+1.76%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0.1</sub></td>
    <td>2.76</td>
    <td>2.72 (+1.45%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>46.31</td>
    <td>46.85 (+1.17%)</td>
  </tr>
  <tr>
    <th colspan="3">CheXpert-F1:</th>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>55.25</td>
    <td>55.87 (+1.12%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>40.15</td>
    <td>40.38 (+0.57%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>58.93</td>
    <td>60.07 (+1.93%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>52.61</td>
    <td>53.75 (+2.17%)</td>
  </tr>
</table>

Table 9: Ablation results for Libra without (w/o) and with (w/) the prior image. Values in (%) indicate the percentage improvement.
<table>
  <tr>
    <th>Metric</th>
    <th>Libra-b</th>
    <th>w/o TFM</th>
    <th>w/o LFE</th>
    <th>w/o PIPB</th>
    <th>w/o TAC</th>
  </tr>
  <tr>
    <th colspan="6">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.26</td>
    <td>26.80 (-1.69%)</td>
    <td>26.57 (-2.53%)</td>
    <td>27.00 (-0.95%)</td>
    <td>24.58 (-9.83%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>34.94</td>
    <td>33.61 (-3.81%)</td>
    <td>33.68 (-3.61%)</td>
    <td>34.47 (-1.35%)</td>
    <td>31.40 (-10.13%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>11.74</td>
    <td>10.97 (-6.56%)</td>
    <td>10.94 (-6.81%)</td>
    <td>11.57 (-1.45%)</td>
    <td>8.89 (-24.28%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>35.37</td>
    <td>34.50 (-2.46%)</td>
    <td>34.30 (-3.03%)</td>
    <td>34.93 (-1.24%)</td>
    <td>32.41 (-8.37%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>55.51</td>
    <td>54.97 (-0.97%)</td>
    <td>54.75 (-1.37%)</td>
    <td>55.26 (0.45%)</td>
    <td>53.07 (-4.40%)</td>
  </tr>
  <tr>
    <th>F1<sub>temp</sub></th>
    <th>24.77</th>
    <th>23.54 (-4.97%)</th>
    <th>24.00 (-3.11%)</th>
    <th>24.68 (-0.36%)</th>
    <th>22.52 (-9.08%)</th>
  </tr>
  <tr>
    <th colspan="6">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>21.67</td>
    <td>21.06 (-2.81%)</td>
    <td>20.73 (-4.34%)</td>
    <td>21.35 (-1.48%)</td>
    <td>19.77 (-8.77%)</td>
  </tr>
  <tr>
    <td>RG<sub>ER</sub></td>
    <td>26.28</td>
    <td>25.45 (-3.16%)</td>
    <td>25.14 (-4.34%)</td>
    <td>25.84 (-1.67%)</td>
    <td>23.74 (-9.67%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0.1</sub></td>
    <td>3.17</td>
    <td>3.20 (-0.95%)</td>
    <td>3.22 (-1.58%)</td>
    <td>3.18 (-0.32%)</td>
    <td>3.27 (-3.15%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>39.58</td>
    <td>38.74 (-2.12%)</td>
    <td>38.37 (-3.06%)</td>
    <td>39.49 (-0.23%)</td>
    <td>37.56 (-5.10%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>49.06</td>
    <td>47.68 (-2.81%)</td>
    <td>47.57 (-3.04%)</td>
    <td>48.40 (-1.35%)</td>
    <td>46.57 (-5.08%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>33.07</td>
    <td>31.60 (-4.45%)</td>
    <td>31.78 (-3.90%)</td>
    <td>31.78 (-3.90%)</td>
    <td>31.27 (-5.44%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>54.55</td>
    <td>52.14 (-4.42%)</td>
    <td>52.94 (-2.95%)</td>
    <td>53.10 (-2.66%)</td>
    <td>50.72 (-7.02%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>47.24</td>
    <td>44.28 (-6.27%)</td>
    <td>44.39 (-6.04%)</td>
    <td>44.68 (-5.42%)</td>
    <td>43.48 (-7.96%)</td>
  </tr>
</table>

Table 10: Results of ablation experiments for the Temporal Alignment Connector. ‘↓’ indicates that lower is better. Values in (%) indicate the percentage decrease compared with the Libra-b.

stantially enhances Libra’s performance across all metrics. Notably, clinical scores exhibit greater improvements compared to lexical scores, underscoring the importance of temporal information in generating high-quality medical reports beyond merely improving linguistic fluency.

The F1<sub>temp</sub> score shows the most substantial improvement, with an increase of **8%**, highlighting Libra’s capability to effectively leverage temporal changes provided by prior images. These results validate the role of temporal information in enhancing the quality of the generated *Findings* section and improving Libra’s overall performance.

**H.2 Impact of the Temporal Alignment Connector under General-Domain Pre-trained Models**

Domain-specific pre-trained models (i.e., RAD-DINO (Pérez-García et al., 2024) and Meditron (Chen et al., 2023c)) inherently incorporate domain-specific knowledge, such as phrasing conventions, pronoun usage, and even temporal information embedded in the training corpus. To isolate the structural impact of TAC, we used a general-domain image encoder (DINOv2 (Oquab et al., 2024)) and a LLM (Vicuna-7B-v1.5 (Chiang et al., 2023)), allowing the structural enhancements of TAC to be observed more directly.

We replicated the first ablation setup from Sec. 4. We first conducted a baseline experiment, referred to as *Libra-b*, by fine-tuning only the adapter for the *Findings* generation task. As shown in Table 10, we then conducted ablation studies by sequentially removing different components from the model, including the Temporal Fusion Module (TFM), Layerwise Feature Extractor (LFE), Prior Image Prefix Bias (PIPb), and the entire TAC. Removing TFM restricts the model to processing only the current image, using a configuration similar to LLaVA (Liu et al., 2023), but with a four-layer MLP to align the image feature with the LLM’s hidden dimensions. Notably, without TFM, the model cannot process prior images or dummy prior images, and is limited to only the current image as input. Without LFE, the model follows the LLaVA setup, using the penultimate layer of the image encoder to process single or paired images.

The ablation results are consistent with those observed using domain-specific models, as presented in Table 2. Removing any TAC submodule led to declines across all metrics. Specifically, removing TFM caused a notable drop in the F1<sub>temp</sub> score (↓>4%), emphasising its role in capturing temporal information. The absence of LFE significantly reduced RadGraph-related scores, demonstrating its importance for detailed image feature extraction. PIPB removal primarily impacted clinical metrics, while removing the entire TAC resulted in substantial declines across all metrics. These findings reaffirm the critical role of TAC in integrating image details and temporal information effectively.

**H.3 Impact of the Temporal Alignment Connector After the Second-Stage Fine-tuning**

To further evaluate the impact of the Temporal Alignment Connector (TAC) on Libra’s performance, we followed the setup of the first ablation
<table>
  <tr>
    <th>Metric</th>
    <th>Libra-2</th>
    <th>w/o TFM</th>
    <th>w/o LFE</th>
    <th>w/o PIPB</th>
    <th>w/o TAC</th>
  </tr>
  <tr>
    <th colspan="6">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>35.31</td>
    <td>35.16 (-0.42%)</td>
    <td>35.09 (-0.64%)</td>
    <td>35.23 (-0.23%)</td>
    <td>34.41 (-2.55%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>49.92</td>
    <td>49.44 (-0.97%)</td>
    <td>49.47 (-0.90%)</td>
    <td>49.75 (-0.34%)</td>
    <td>48.61 (-2.63%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>23.05</td>
    <td>22.67 (-1.66%)</td>
    <td>22.65 (-1.75%)</td>
    <td>22.97 (-0.35%)</td>
    <td>21.51 (-6.70%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>47.99</td>
    <td>47.69 (-0.62%)</td>
    <td>47.62 (-0.77%)</td>
    <td>47.84 (-0.31%)</td>
    <td>46.95 (-2.16%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>61.28</td>
    <td>61.13 (-0.24%)</td>
    <td>61.07 (-0.34%)</td>
    <td>61.21 (-0.12%)</td>
    <td>60.60 (-1.12%)</td>
  </tr>
  <tr>
    <th>F1<sub>temp</sub></th>
    <th>33.52</th>
    <th>33.10 (-1.27%)</th>
    <th>33.25 (-0.79%)</th>
    <th>33.49 (-0.09%)</th>
    <th>32.73 (-2.36%)</th>
  </tr>
  <tr>
    <th colspan="6">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>30.77</td>
    <td>30.55 (-0.72%)</td>
    <td>30.43 (-1.10%)</td>
    <td>30.65 (-0.40%)</td>
    <td>30.07 (-2.27%)</td>
  </tr>
  <tr>
    <td>RGE<sub>R</sub></td>
    <td>35.44</td>
    <td>35.16 (-0.79%)</td>
    <td>35.05 (-1.10%)</td>
    <td>35.29 (-0.42%)</td>
    <td>34.55 (-2.51%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0</sub> (.)</td>
    <td>2.83</td>
    <td>2.84 (-0.35%)</td>
    <td>2.84 (-0.35%)</td>
    <td>2.85 (-0.71%)</td>
    <td>2.85 (-0.71%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>45.32</td>
    <td>45.08 (-0.53%)</td>
    <td>44.97 (-0.77%)</td>
    <td>45.27 (-0.11%)</td>
    <td>44.73 (-1.30%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>54.11</td>
    <td>53.73 (-0.70%)</td>
    <td>53.70 (-0.76%)</td>
    <td>54.00 (-0.20%)</td>
    <td>53.41 (-1.30%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>37.16</td>
    <td>36.74 (-1.13%)</td>
    <td>36.78 (-1.02%)</td>
    <td>36.79 (-1.00%)</td>
    <td>36.64 (-1.40%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>58.76</td>
    <td>58.10 (-1.12%)</td>
    <td>58.32 (-0.75%)</td>
    <td>58.36 (-0.68%)</td>
    <td>57.65 (-1.89%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>51.99</td>
    <td>51.16 (-1.60%)</td>
    <td>51.19 (-1.54%)</td>
    <td>51.27 (-1.38%)</td>
    <td>50.87 (-2.15%)</td>
  </tr>
</table>

Table 11: Results of ablation experiments for the Temporal Alignment Connector after the second stage. '↓' indicates that lower is better. Values in (%) indicate the percentage decrease compared with the Libra-2.

study in Sec. 4. After the first stage of alignment, the model underwent a second stage of fine-tuning. This stage was designed to optimise the model’s performance on the Findings section generation task by leveraging the aligned visual and textual features learned during the initial stage.

In this phase, we applied Low-Rank Adaptation (LoRA) Hu et al. (2021) to fine-tune the pre-trained LLM (Meditron Chen et al. (2023c)), while keeping the visual encoder (RAD-DINO Pérez-García et al. (2024)) and TAC weights frozen. The baseline for this experiment is Libra-2 (in Table 11), which is derived from Libra-1 (in Table 2) after undergoing LoRA fine-tuning.

We conducted ablation studies by progressively removing different TAC components, including TFM, LFE, the Prior Image Prefix Bias (PIPb), and the entire TAC. Results consistently showed declines across all metrics compared to Libra-2, mirroring the trends observed in Sec. 4. This reinforces that the performance improvements brought by TAC are stable and unaffected by changes in training stages. It further confirms that TAC has embedded the capability to process temporal information within the model.

H.4 Robustness Evaluation of the Temporal Alignment Connector

To evaluate the robustness of the Temporal Alignment Connector (TAC), we introduced an additional round of LoRA fine-tuning to induce overtraining. Following the setup in Appx. H.3, after integrating the first LoRA weights, a new set of LoRA adapters was reinitialised for the LLM and trained for one epoch under the same second-stage fine-tuning configuration. The baseline for this experiment is Libra-3 (as shown in Table 12), which is derived from Libra-2 (illustrated in Table 11) following this additional fine-tuning step.

The results reveal that, compared to Libra-2, Libra-3 exhibits minimal changes in lexical scores, while clinical scores decline due to overfitting caused by the additional fine-tuning. Notably, the CheXpert (Smit et al., 2020a) (Macro-F1-[5/14]) scores exhibit the most influential reduction.

Despite this decline, ablation studies confirm that TAC’s performance improvements remain robust, unaffected by variations in training strategies. This resilience stems from TAC’s ability to capture and retain temporal image representations during the initial training phase, which are preserved through subsequent fine-tuning.

These findings underscore TAC’s reliability as a critical component for temporal information processing in RRG tasks. It ensures stability even under diverse training conditions.

H.5 Impact of Radiology-Specific Pre-trained Models on Libra

Aligning radiology images with textual information is a key challenge in RRG tasks. To demonstrate the benefits of using radiology-specific pre-trained models for more accurate feature representation and improved MLLM performance, we initialised a Libra model with RadDINO, the TAC, and Meditron-7b, conducting the first stage of training, denoted as Libra-1 (This is consistent with the baseline setup of the previous ablation study in Sec. 4). Then we replaced the image encoder and LLM with their general-domain counterparts, DI-
<table>
  <tr>
    <th>Metric</th>
    <th>Libra-3</th>
    <th>w/o TFM</th>
    <th>w/o LFE</th>
    <th>w/o PIPB</th>
    <th>w/o TAC</th>
  </tr>
  <tr>
    <th colspan="6">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>35.58</td>
    <td>35.53 (-0.14%)</td>
    <td>35.51 (-0.21%)</td>
    <td>35.55 (-0.08%)</td>
    <td>35.28 (-0.86%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>49.54</td>
    <td>49.38 (-0.32%)</td>
    <td>49.39 (-0.30%)</td>
    <td>49.48 (-0.11%)</td>
    <td>49.10 (-0.88%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>23.61</td>
    <td>23.48 (-0.55%)</td>
    <td>23.47 (-0.58%)</td>
    <td>23.58 (-0.12%)</td>
    <td>23.07 (-2.28%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>47.61</td>
    <td>47.51 (-0.21%)</td>
    <td>47.49 (-0.26%)</td>
    <td>47.56 (-0.10%)</td>
    <td>47.26 (-0.73%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>61.54</td>
    <td>61.49 (-0.08%)</td>
    <td>61.47 (-0.11%)</td>
    <td>61.52 (-0.04%)</td>
    <td>61.31 (-0.37%)</td>
  </tr>
  <tr>
    <th>F1<sub>temp</sub></th>
    <th>33.51</th>
    <th>33.37 (-0.42%)</th>
    <th>33.42 (-0.27%)</th>
    <th>33.50 (-0.03%)</th>
    <th>33.24 (-0.79%)</th>
  </tr>
  <tr>
    <th colspan="6">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>29.82</td>
    <td>29.75 (-0.24%)</td>
    <td>29.71 (-0.37%)</td>
    <td>29.78 (-0.13%)</td>
    <td>29.59 (-0.76%)</td>
  </tr>
  <tr>
    <td>RGE<sub>R</sub></td>
    <td>35.60</td>
    <td>35.51 (-0.26%)</td>
    <td>35.47 (-0.37%)</td>
    <td>35.55 (-0.14%)</td>
    <td>35.30 (-0.84%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0</sub> (↓)</td>
    <td>2.91</td>
    <td>2.92 (-0.34%)</td>
    <td>2.92 (-0.34%)</td>
    <td>2.91 (−)</td>
    <td>2.93 (-0.68%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>44.77</td>
    <td>44.69 (-0.18%)</td>
    <td>44.65 (-0.26%)</td>
    <td>44.75 (-0.04%)</td>
    <td>44.57 (-0.45%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>52.45</td>
    <td>52.33 (-0.23%)</td>
    <td>52.32 (-0.25%)</td>
    <td>52.41 (-0.08%)</td>
    <td>52.22 (-0.44%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>30.77</td>
    <td>30.65 (-0.38%)</td>
    <td>30.66 (-0.34%)</td>
    <td>30.67 (-0.33%)</td>
    <td>30.63 (-0.47%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>54.42</td>
    <td>54.22 (-0.38%)</td>
    <td>54.28 (-0.25%)</td>
    <td>54.30 (-0.23%)</td>
    <td>54.08 (-0.63%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>44.58</td>
    <td>44.34 (-0.54%)</td>
    <td>44.35 (-0.52%)</td>
    <td>44.37 (-0.46%)</td>
    <td>44.26 (-0.72%)</td>
  </tr>
</table>

Table 12: Results of ablation experiments for the Temporal Alignment Connector with additional LoRA fine-tuning after the second stage. ‘↓’ indicates that lower is better. Values in (%) indicate the percentage decrease compared with the Libra-3.

<table>
  <tr>
    <th>Metric</th>
    <th>Libra-1</th>
    <th>w/o RadDINO</th>
    <th>w/o Meditron</th>
    <th>w/o RadDINO+Meditron</th>
  </tr>
  <tr>
    <th colspan="5">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>27.56</td>
    <td>27.66 (0.36%)</td>
    <td>27.29 (-0.98%)</td>
    <td>27.26 (-1.09%)</td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>34.84</td>
    <td>35.32 (1.38%)</td>
    <td>34.91 (0.20%)</td>
    <td>34.94 (0.29%)</td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>11.51</td>
    <td>12.56 (9.12%)</td>
    <td>11.61 (0.87%)</td>
    <td>11.74 (2.00%)</td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>35.50</td>
    <td>35.65 (0.42%)</td>
    <td>35.53 (0.08%)</td>
    <td>35.37 (-0.37%)</td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>55.87</td>
    <td>55.89 (0.04%)</td>
    <td>55.58 (-0.52%)</td>
    <td>55.51 (-0.64%)</td>
  </tr>
  <tr>
    <th>F1<sub>temp</sub></th>
    <th>26.63</th>
    <th>25.53 (-4.13%)</th>
    <th>24.78 (-6.95%)</th>
    <th>24.77 (-6.98%)</th>
  </tr>
  <tr>
    <th colspan="5">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>22.52</td>
    <td>22.11 (-1.82%)</td>
    <td>23.13 (2.71%)</td>
    <td>21.67 (-3.77%)</td>
  </tr>
  <tr>
    <td>RGE<sub>R</sub></td>
    <td>27.32</td>
    <td>26.72 (-2.20%)</td>
    <td>27.53 (0.77%)</td>
    <td>26.28 (-3.81%)</td>
  </tr>
  <tr>
    <td>RadCliQ<sub>0</sub> (↓)</td>
    <td>3.10</td>
    <td>3.13 (-0.97%)</td>
    <td>3.08 (0.65%)</td>
    <td>3.17 (-2.26%)</td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>42.02</td>
    <td>40.78 (-2.95%)</td>
    <td>41.94 (-0.19%)</td>
    <td>39.49 (-6.02%)</td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>52.84</td>
    <td>51.55 (-2.44%)</td>
    <td>51.45 (-2.63%)</td>
    <td>49.06 (-7.15%)</td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>36.87</td>
    <td>34.58 (-6.21%)</td>
    <td>37.20 (0.90%)</td>
    <td>33.07 (-10.31%)</td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>56.63</td>
    <td>55.00 (-2.88%)</td>
    <td>55.39 (-2.19%)</td>
    <td>54.55 (-3.67%)</td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>49.33</td>
    <td>47.26 (-4.20%)</td>
    <td>47.62 (-3.47%)</td>
    <td>47.24 (-4.24%)</td>
  </tr>
</table>

Table 13: Ablation results for radiology-specific pre-trained models in Libra. ‘↓’ indicates that lower is better. Values in (%) indicate the percentage improvement compared to Libra-c.

NOv2 and Vicuna-7B-v1.5, respectively. Finally, we replaced both components, which is also referred to as Libra-b (in Table 10).

As shown in Table 13, substituting radiology-specific pre-trained models with general-domain models resulted in a notable decline in clinical scores, while the impact on lexical scores was minimal. Notably, replacing the radiology-specific image encoder caused a more pronounced decline in clinical metrics compared to replacing the language model. This suggests that accurate medical image representation provides greater benefits in RRG tasks, indicating the importance of incorporating domain-specific knowledge into pre-trained models to enhance Libra’s performance.

H.6 Incremental Component Analysis

We conducted an incremental study to evaluate the effectiveness of each component in Libra’s architecture. Starting with a baseline model similar to LLaVA—comprising a pre-trained CLIP image encoder, a randomly initialised four-layer MLP adapter, and Vicuna-7B-v1.5 as the LLM—we trained the adapter on the Findings section generation task.

Improvements were introduced incrementally, as summarised in Table 14. First, we replaced the image encoder with DINOv2. Next, we incorporated the LFE (prefix module of TAC) and subsequently added the TFM (suffix module), completing the TAC connector. We then replaced the image encoder and LLM with RAD-DINO and Meditron,
<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="6">Stage 1: Temporal Feature Alignment</th>
    <th rowspan="2">Stage 2 Libra</th>
  </tr>
  <tr>
    <th>Initial</th>
    <th>DINO</th>
    <th>LFE</th>
    <th>TFM</th>
    <th>RAD-DINO</th>
    <th>Meditron</th>
    <th>Dataset</th>
  </tr>
  <tr>
    <th colspan="8">Lexical:</th>
  </tr>
  <tr>
    <td>ROUGE-L</td>
    <td>23.77</td>
    <td>24.58</td>
    <td>26.57</td>
    <td>27.26</td>
    <td>27.29</td>
    <td>27.56</td>
    <td>27.27</td>
    <td><b>36.66</b></td>
  </tr>
  <tr>
    <td>BLEU-1</td>
    <td>31.48</td>
    <td>31.40</td>
    <td>33.68</td>
    <td>34.94</td>
    <td>34.91</td>
    <td>34.84</td>
    <td>41.24</td>
    <td><b>51.25</b></td>
  </tr>
  <tr>
    <td>BLEU-4</td>
    <td>8.41</td>
    <td>8.89</td>
    <td>10.94</td>
    <td>11.74</td>
    <td>11.61</td>
    <td>11.51</td>
    <td>13.59</td>
    <td><b>24.54</b></td>
  </tr>
  <tr>
    <td>METEOR</td>
    <td>32.1</td>
    <td>32.41</td>
    <td>34.3</td>
    <td>35.37</td>
    <td>35.53</td>
    <td>35.50</td>
    <td>39.44</td>
    <td><b>48.90</b></td>
  </tr>
  <tr>
    <td>BERTScore</td>
    <td>52.76</td>
    <td>53.07</td>
    <td>54.75</td>
    <td>55.51</td>
    <td>55.58</td>
    <td>55.87</td>
    <td>56.00</td>
    <td><b>62.50</b></td>
  </tr>
  <tr>
    <th colspan="8">F1<sub>temp</sub></th>
    <th>21.60</th>
    <th>22.52</th>
    <th>24.00</th>
    <th>24.77</th>
    <th>24.78</th>
    <th>26.63</th>
    <th>24.80</th>
    <th><b>35.34</b></th>
  </tr>
  <tr>
    <th colspan="8">Clinical:</th>
  </tr>
  <tr>
    <td>RadGraph-F1</td>
    <td>18.58</td>
    <td>19.70</td>
    <td>20.73</td>
    <td>21.67</td>
    <td>23.13</td>
    <td>22.52</td>
    <td>20.45</td>
    <td><b>32.87</b></td>
  </tr>
  <tr>
    <td>RG<sub>ER</sub></td>
    <td>23.05</td>
    <td>23.74</td>
    <td>25.14</td>
    <td>26.28</td>
    <td><u>27.53</u></td>
    <td>27.32</td>
    <td>25.19</td>
    <td><b>37.57</b></td>
  </tr>
  <tr>
    <td>RadChiQu(J)</td>
    <td>3.35</td>
    <td>3.26</td>
    <td>3.22</td>
    <td>3.17</td>
    <td><u>3.08</u></td>
    <td>3.10</td>
    <td>3.31</td>
    <td><b>2.72</b></td>
  </tr>
  <tr>
    <td>CheXbert vector</td>
    <td>35.59</td>
    <td>37.94</td>
    <td>38.37</td>
    <td>39.49</td>
    <td>41.94</td>
    <td><u>42.02</u></td>
    <td>35.33</td>
    <td><b>46.85</b></td>
  </tr>
  <tr>
    <td><i>CheXpert-F1:</i></td>
    <td colspan="8"></td>
  </tr>
  <tr>
    <td>Micro-F1-14</td>
    <td>44.75</td>
    <td>46.57</td>
    <td>47.57</td>
    <td>49.06</td>
    <td>51.45</td>
    <td><u>52.48</u></td>
    <td>43.63</td>
    <td><b>55.87</b></td>
  </tr>
  <tr>
    <td>Macro-F1-14</td>
    <td>25.13</td>
    <td>31.27</td>
    <td>31.07</td>
    <td>33.07</td>
    <td>37.20</td>
    <td>36.87</td>
    <td>25.68</td>
    <td><b>40.38</b></td>
  </tr>
  <tr>
    <td>Micro-F1-5</td>
    <td>45.97</td>
    <td>50.72</td>
    <td>52.94</td>
    <td>54.55</td>
    <td>55.39</td>
    <td><u>56.63</u></td>
    <td>49.75</td>
    <td><b>60.07</b></td>
  </tr>
  <tr>
    <td>Macro-F1-5</td>
    <td>36.55</td>
    <td>43.48</td>
    <td>44.39</td>
    <td>47.24</td>
    <td>47.62</td>
    <td><u>49.33</u></td>
    <td>40.40</td>
    <td><b>53.75</b></td>
  </tr>
</table>

Table 14: Results of ablation experiments for key components of Libra on Findings section generation performance. * indicates our initialised model. ′ denotes component replacement. + signifies structural addition. ′ represents dataset configuration. The best performances are highlighted in bold, and the second-best scores are underlined. ‘↓’ indicates that lower is better.

respectively. The dataset for the first stage was expanded, and final fine-tuning was conducted for downstream tasks to produce Libra.

With each enhancement, the model’s performance improved, demonstrating the critical role of each component. Notably, the addition of the TFM during the alignment stage provided the most significant improvement, showcasing its ability to capture temporal information, which is essential for the RRG task.

However, data expansion in the first stage led to improved lexical scores but a slight decline in clinical metrics, likely due to the VQA task’s focus on fine-grained grounded information rather than holistic report generation, as mentioned in Sec. 4. This shift also affected the F1<sub>temp</sub> score, as temporal entities are often linked to specific symptoms. These declines were subsequently addressed through second-stage fine-tuning, resulting in overall improved performance.

Evaluation of Libra’s Temporal Awareness
Another approach to investigating the model’s ability to capture temporal information is to evaluate it separately within the test split based on the presence or absence of prior images, in Table 15.

With the addition of the TFM, the model exhibited temporal awareness. It is worth noting that, for the first time, the F1<sub>temp</sub> score of samples with prior images surpassed those without, and this trend persisted through subsequent optimisations. This indicates that the structural enhancements have resulted in a sustained improvement in the model’s temporal perception capabilities. An effective example is in Sec. 5.

![Line graph showing F1_temp score for Libra with and without prior image](page_1012_1342_388_282.png)

Table 15: Results of ablation experiments for Libra on the F1<sub>temp</sub> score. Among 2,461 test samples, 2,117 include a prior image, while 344 do not.

I Heatmap Analysis and Temporal Feature Representation

The heatmap in Figure 4 corresponds to the example in (a) of Figure 3, where no prior image was used as a reference. It illustrates the clear differences in feature representations across layers of the RAD-DINO (Pérez-García et al., 2024) image
Figure 4: Heat map visualisation of image representations from different image encoder layers and the Temporal Alignment Connector (TAC), up-sampled using a Gaussian filter. Warm colours (red, yellow) indicate regions with higher weight allocations in the intermediate outputs of the “hidden-state” within the model blocks, while cool colours (blue, green) represent regions with lower weight.

Figure 5: Heat map visualisation of image representations from the Temporal Alignment Connector (TAC), up-sampled using a Gaussian filter. The arrows (→) represent the direction of temporal information, pointing from the prior image to the true current image. Warm colours (red, yellow) indicate regions with higher weight allocations in the intermediate outputs of the “hidden-state” within the model blocks, while cool colours (blue, green) represent regions with lower weight.

encoder. The shallow layers primarily capture the overall lung structure, while the deeper layers focus on specific disease regions.

After passing through the Layerwise Feature Extractor (LFE), the image feature representations assign higher weights to larger symptom regions, achieving finer granularity. Following the Temporal Alignment Connector (TAC), the model integrates the weighted dummy prior image, producing a uniform feature distribution that reflects temporal information. This indicates no significant changes compared to the prior study and facilitates smoother image feature representations for downstream text generation by the LLM.

The heatmap in Figure 5 corresponds to the example in (b) of Figure 3, where a prior image is provided. After processing through the LFE, the model captures fine-grained feature representations in symptom areas. When processed by the TAC, these features are integrated with the differences between the two images, effectively reflecting temporal information, as demonstrated in TAC-1 (top) of Figure 5.

When the image order is swapped, treating the prior image as the current image, the LFE output remains unchanged. However, comparing TAC-2 (bottom of Figure 5) and TAC-1 outputs reveals significant differences in lung feature representations. This highlights the model’s directional temporal perception and confirms that the TAC module effectively encodes temporal information from different time points, while the LFE focuses solely on image features without temporal encoding.

This behaviour aligns with the design of the TAC, where residual connections prioritise the current image as the main modality and the prior image as the auxiliary. Swapping the image order changes the main modality, altering the temporal state of symptoms in the generated report, such as reversing descriptions from “improving” to “worsening,” as discussed in Sec. 5.
J  Extended Discussion on Limitations

While our work represents a step forward in leveraging temporal information for radiology report generation, it also has several limitations that warrant further exploration.

Handling Multiple Prior Scans  Our current model is designed to process a single prior scan alongside the current scan. While this approach aligns with standard clinical workflows, which typically prioritise the most recent prior study for comparisons, it overlooks scenarios where multiple prior scans could offer a richer temporal perspective. For instance, analysing a sequence of images spanning an extended period could provide deeper insights into gradual disease progression. Future efforts should focus on extending our framework to incorporate multiple prior scans efficiently, enabling a more nuanced understanding of temporal patterns in clinical data.

Temporal Information Beyond Image Comparisons  Currently, our model captures temporal information through paired image comparisons and corresponding textual reports. However, clinical assessments often draw upon a broader context, including historical notes, laboratory results, and other longitudinal patient data. Expanding our approach to integrate these diverse temporal data sources could facilitate a more holistic understanding of disease trajectories and patient history, significantly enhancing clinical applicability.

Sparse Temporal Data Challenges  In cases where prior scans are unavailable or minimally informative (e.g., taken within a short interval), our “dummy prior image” provides a workaround. However, the model’s ability to interpret and generate meaningful outputs under these constraints may still be limited. Future research could focus on synthesising or imputing temporal context to enhance performance under these constraints.

Computational Complexity  The use of temporal alignment mechanisms and multi-layer feature integration increases computational demands, posing challenges for deployment in resource-constrained environments. Future optimisation efforts should focus on reducing computational overhead while maintaining performance.

Generalisability Across Modalities and Datasets
Our study is limited to frontal-view chest X-rays and the MIMIC-CXR dataset (Johnson et al., 2019b). The applicability of our approach to other imaging modalities (e.g., CT, MRI) and datasets (e.g., CheXpert (Irvin et al., 2019), PadChest (Bustos et al., 2020)) remains unexplored. Future studies should assess the model’s generalisability to a broader range of datasets and imaging contexts.

Based on the identified limitations, we outline the following directions:

• Develop frameworks for integrating multiple prior scans with dynamic temporal reasoning to better capture longitudinal changes.
• Expand the model to incorporate multi-modal imaging and textual data for more comprehensive diagnostic insights.
• Investigate the integration of diverse temporal data sources, such as electronic health records (EHRs), to enhance clinical applicability.
• Exploring lightweight model architectures for faster inference while maintaining high performance.

These advancements aim to address the current limitations while broadening the applicability of temporal-aware multimodal models in radiology and other clinical domains.

Reviewer paper summaries: This study proposes Libra, a perceptual multimodal large language model tailored for the chest X-ray report generation tasks. Libra adopts a two-stage training framework, utilizing a radiology-specific image encoder and a language model connected by a temporal alignment connector to achieve seamless integration of visual and textual modalities. Libra demonstrates significant performance gains on key metrics.

This paper introduces Libra, a temporal-aware MLLM designed to automate radiology report generation by integrating prior imaging data to model temporal progression. The core contribution is the Temporal Alignment Connector, which combines a Layerwise Feature Extractor and Temporal Fusion Module to align visual and textual modalities across time. The authors evaluate Libra on the MIMIC-CXR dataset, comparing it against baselines including non-LLM models and LLM-based approaches.
Summary of strengths: The idea of Leveraging temporal images in radiology report generation is interesting.
The experiments are comprehensive and complete.

- Temporal reasoning is critical and somewhat new in radiology, and the paper identifies a clear gap in existing MLLM-based approaches.

- The comparison against 20+ models, including recent LLMs (e.g., MAIRA-1, Med-PaLM), strengthens the empirical rigor.
Summary of weaknesses: The overall novelty of the proposed Libra model is somewhat limited. Biomedical radiology analysis with temporal image studies has been conducted. The handling of temporal image input in the Libra model has room for improvement. The model is designed to use a single prior image for temporal modeling, which is a significant limitation considering that in clinical practice.  In detail,
When using paired temporal data each time, is the report of the most recent image used as the label during training? Is this data efficiency relatively low?
For single image data, I understand that the corresponding report part should not contain comparison information. This conflicts with the system prompts of the model and will affect the model's instruction-following ability.

- The TAC architecture’s core components (multi-layer feature aggregation, cross-attention) are incremental adaptations of established techniques rather than novel contributions. Layerwise feature extraction mirrors prior work in medical vision-language models (e.g., BioViL-T), yet the cross-attention mechanism used in TFM lacks domain-specific adaptations (e.g., modality-specific scaling).

- The model evaluation is constrained to a single dataset (MIMIC-CXR).

- A comparison with recent state-of-the-art models like MAIRA-2 is missing, which incorporates lateral views and grounded reasoning.

- The use of temporal alignment mechanisms and multi-layer feature integration increases computational demands. I would like to see a comparison of training costs and inference latency.
Comments, suggestions and typos: None

- While the author emphasizes "temporal hallucination mitigation," the evaluation focuses on static metrics (e.g., BLEU, ROUGE) rather than explicit temporal reasoning.
Overall assessment scores: 3.5

3
Metareview score: 3
Metareview: This paper introduces a temporal-aware multimodal large language model Libra for radiology report generation. Libra integrates paired chest X-ray images through the proposed Temporal Alignment Connector, which extracts high-granularity visual features across layers and models temporal progression via cross-attention. Built upon strong existing models—namely, the Meditron language model and the RAD-DINO image encoder—Libra demonstrates strong performance on the MIMIC-CXR dataset across both lexical and clinical metrics, and introduces a new Temporal Entity F1 score to assess temporal reasoning.
Summary of reasons to publish: 1. All three reviewers agree that the paper's target problem of modeling temporal changes between chest X-rays is important, and the idea of temporal reasoning is interesting.
2. The paper presents extensive comparisons against over 20 models, including specialized medical LLMs and MLLMs.