AIRI at RRG24: LLaVa with specialised encoder and decoder (2024)

Abstract: We present a new approach to generating the “Findings” and “Impression” sections in the chest X-rays radiology reports, developed as part of the shared radiology task at BioNLP 2024. By integrating a DINOv2 vision encoder trained on medical data with specialized biomedical large language model using the LLaVA framework, our method addresses complex medical semantics and diverse findings in imaging. We use datasets from PadChest, BIMCV-COVID19, CheXpert, OpenI, and MIMIC-CXR. The evaluation metrics demonstrate our method’s effectiveness and the potential for automating the generation of radiology reports.

text: Proceedings of the 23rd Workshop on Biomedical Language Processing, pages 603–607 August 16, 2024. ©2024 Association for Computational Linguistics
603

heading: 1 Introduction
text: The automatic generation of radiology reports from chest X-rays is a challenging and significant task in the field of biomedical natural language processing (BioNLP). The growing volume of medical imaging data and the limited number of radiologists necessitate the development of robust automated systems to assist in report generation. Such systems not only have the potential to improve clinical workflow efficiency but also to ensure consistency and comprehensiveness in radiological interpretations.
In recent years, advancements in deep learning and natural language processing have paved the way for innovative approaches to tackle this task. The new approaches typically involve the integration of convolutional neural networks (CNNs) or visual transformers for image feature extraction with recurrent neural networks (RNNs) or transformers for text generation (Selivanov et al., 2023). Despite the progress, challenges such as
∗V.S., M.M., and D.U. contributed equally.
capturing complex medical semantics, handling diverse imaging findings, and ensuring the clinical accuracy of generated reports remain.
This paper explores a new method for generating the Findings and Impression sections of radiology reports from chest X-rays. Our approach is to combine a vision encoder, self-trained on medical data, with specialized biomedical LLM for text generation, using LLaVA framework. This work was done as a part of Radiology Report Generation ahared task at BioNLP 2024 Workshop (Xu et al., 2024) using the data provided by the organizers. The metrics were calculated using the ViLMedic platform (Delbrouck et al., 2022b).

heading: 2 Data

heading: 2.1 Training and validation data
text: The data from 5 datasets where combined to create the competition training and validation datasets: PadChest(Bustos et al., 2020), BIMCVCOVID19(Vayá et al., 2020), CheXpert(Chambon et al., 2024), OpenI(Demner-Fushman et al., 2012) and MIMIC-CXR(Johnson et al., 2019). The training and validation sets are grouped by study but not by subjects. The official language of PadChest and BIMCV-COVID19 is Spanish, and their reports have been translated using GPT-4 by the shared task organizers.
The data consists of radiology studies, each containing one or more chest X-ray images in various projections. Each study also includes Impression and Finding texts. Some studies have only the Impression or only the Findings section, while others have both.

heading: 2.2 Testing Data
text: The studies in the test sets are unseen studies provided by organizers. Public test sets for impression and findings contain both study images and ground truth texts while private test set contains only images.

heading: 2.3 Data preprocessing
text: Due to technical limitations, we only used the first two images from each study. Studies with only one image were not further processed. For studies with more than one image, the first two images were stitched together horizontally. No additional preprocessing was applied to the texts.

heading: 3 Evaluation metrics
text: In the evaluation of radiology report summarization systems, several metrics are commonly used to assess the performance and accuracy of the generated summaries. These metrics ensure that the summaries produced by the models are not only syntactically and semantically correct but also factually accurate. The metrics used in this competition where BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019), F1CheXbert (Smit et al., 2020), and F1-RadGraph (Delbrouck et al., 2022a).

heading: 3.1 BLEU (Bilingual Evaluation Understudy)
text: BLEU-4: This metric is widely used for evaluating machine translation systems. It measures the precision of n-grams in the generated summary by comparing it to one or more reference summaries.
BLEU-4 specifically considers 4-gram overlaps, providing a robust measure of how many 4-grams in the generated text appear in the reference texts. However, it does not account for recall or the contextual meaning of words.

heading: 3.2 ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
text: ROUGE-L: ROUGE is predominantly used for evaluating automatic text summarization. ROUGE-L measures the longest common subsequence (LCS) between the generated summary and the reference summary. This metric emphasizes recall by capturing the longest sequence of words that appear in both the generated and reference summaries, thus reflecting the ability of the summary to include important information.

heading: 3.3 BERTScore
text: BERTScore: This metric computes the similarity between the generated and reference texts using pre-trained BERT embeddings. It calculates a similarity score for each token in the candidate sentence with each token in the reference sentence. BERTScore accounts for the semantic meaning of words, making it more robust against synonyms and paraphrasing compared to BLEU and ROUGE.

heading: 3.4 F1-CheXbert
text: F1-CheXbert: This metric evaluates the factual correctness of the generated summaries with a focus on specific medical conditions mentioned in radiology reports. CheXbert is a specialized tool designed to extract medical observations from radiology reports. The F1 score is calculated based on the precision and recall of these extracted observations, ensuring that the generated summaries accurately reflect the medical conditions described in the reference summaries.

heading: 3.5 F1-RadGraph
text: F1-RadGraph: Similar to F1-CheXbert, this metric evaluates the factual correctness of the summaries using the RadGraph dataset. RadGraph focuses on extracting entities and the relations between them from radiology reports. The F1-RadGraph score measures the accuracy of these extractions, comparing the generated summaries to the reference summaries to ensure that the critical entities and their relationships are accurately captured.
These metrics collectively provide a comprehensive evaluation framework for radiology report summarization systems. BLEU and ROUGE focus on the surface-level n-gram overlaps, while BERTScore provides a deeper semantic evaluation. F1-CheXbert and F1-RadGraph ensure the factual accuracy of medical details, which is crucial for clinical applications.

heading: 4 Methods and Results
text: We used the LLaVA model (Liu et al., 2024) with a DINOv2 encoder (Oquab et al., 2023) and OpenBio-LLM-8B (Ankit Pal, 2024) as a text decoder. The whole pipeline was implemented using HuggingFace’s transformers (Wolf et al., 2020) and trl (von Werra et al., 2020) libraries.
For image encoding we used a DINOv2 Model with the following parameters:
• Model: ViT-base 14, initialized from torch.hub’s dinov2 vitb14 • Patch size: 14 • Number of parameters: 86M • Time and Resources: 4xA100 80GB GPUs,
Training Total Time: 2 days • Dataset: MIMIC-CXR Train, downsampled
to 518 px • Batch size per GPU: 50 • Base Learning Rate: 0.001
For text generation, we used OpenBioLLM-8B, an open-source language model designed specifically for the biomedical domain.
• Training type: LoRA on LLM’s Attention matrices (r=64, alpha=16) + MM projector • Architecture: OpenBio-LLM-8B + in-house DINOv2 trained on MIMIC-CXR Train • Time and Resources: 5 epochs, 8xH100 80GB GPUs, DeepSpeed Zero-3; Training Total Time: 2 days • Batch size per GPU: 8, gradient accumulation: 2 • Base Learning Rate: 0.001, cosine schedule, warmup: 0.15 • Optimizer: Adam
Vanilla approach to fine-tune LLaVA model with language model unfreezed resulted in rapid overfitting, thus we opted for PEFT methods (Mangrulkar et al., 2022), namely LoRA (Hu et al., 2022).
We used the same model for generating both impression and findings, using different prompts: either ”Write findings for this X-ray.” or ”Write impression for this X-ray.”.
We used the following system prompt, inspired by LLaVA-Med (Li et al., 2024):”You are a large language and vision assistant. You are designed to assist human with a variety of medical visual content and clinical research tasks using natural language. Follow the instructions carefully and provide clinically valid answers.”
Our results on hidden test sets are presented in Table 4 and Table 5.
Our relatively simple model demonstrates strong performance in generating radiology reports. We attribute this success to the use of a specialized image encoder and a specialized large language model. Future improvements can be realized by employing larger models and fully using the available image data, which would likely enhance the competition metrics of the generated reports.

heading: Future Work
text: Future improvements can be realized by employing larger models and fully using the available image data, which would likely enhance the competition metrics of the generated reports