CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset

## Abstract

Abstract X-ray image-based medical report generation (MRG) is a pivotal area in artificial intelligence which can significantly reduce diagnostic burdens and patient wait times. Despite significant progress, we believe that the task has reached a bottleneck due to the limited benchmark datasets and the existing large models’ insufficient capability enhancements in this specialized domain. Specifically, the recently released CheXpert Plus dataset lacks comparative evaluation algorithms and their results, providing only the dataset itself. This situation makes the training, evaluation, and comparison of subsequent algorithms challenging. Thus, we conduct a comprehensive benchmarking of existing mainstream X-ray report generation models and large language models (LLMs), on the CheXpert Plus dataset. We believe that the proposed benchmark can provide a solid comparative basis for subsequent algorithms and serve as a guide for researchers to quickly grasp the state-of-the-art models in this field. More importantly, we propose a large model for the X-ray image report generation using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning. Extensive experimental results indicate that the autoregressive pre-training based on Mamba effectively encodes X-ray images, and the image-text contrastive pre-training further aligns the feature spaces, achieving better experimental results.
Source code can be found on https://github.com/Event-AHU/Medical_Image_Analysis .

## 1 Introduction

X-ray image based Medical Report Generation (MRG) is a critical research problem in artificial intelligence, which targets describing the findings or impressions from the given X-ray data using natural language. The successful implementation of this task can significantly reduce the diagnostic burden on physicians, decrease patient wait times, and foster the positive application of artificial intelligence. However, the path to progress in this direction is not smooth sailing, there remain formidable challenges that need to be overcome. The challenging issues include image interpretation, data annotation, heterogeneity issues, consistency and standardization of reports, diversity and variability of diseases, interpretability of algorithms, etc. How to address these challenges further and improve the quality of medical report generation remains an urgent research problem.

After revisiting the mainstream algorithms of X-ray image medical report generation, we find that datasets like IU X-ray and MIMIC-CXR are widely used for the training and evaluation of report generation models. However, the IU X-ray only contains 7,470 images and 3,955 radiology reports samples, which is rather limited, especially in the large model era. The recently released CheXpert Plus dataset [] is a large-scale dataset for the X-ray report generation, however, they did not release comparative methods, making it difficult for subsequent algorithms to conduct experiments and comparisons on this dataset. Therefore, we conduct a comprehensive benchmarking of existing open-sourced mainstream X-ray report generation models, Large Language Models (LLMs), and Vision-Language Models (VLMs), termed CXPMRG-Bench, on the newly released CheXpert Plus dataset, as shown in Fig. 1. The completion of this work can also help researchers identify which large models and algorithms are currently leading in the field of X-ray report generation.

On the other hand, most mainstream algorithms follow the encoder-decoder framework which usually adopts the vision encoder (e.g., ResNet [], Transformer []) to process the given X-ray data and a text decoder (e.g., LSTM [], GRU [], Transformer []) for report generation. Along with the development of pre-trained LLM and VLM, the quality of medical reports is enhanced significantly. There are already some researchers who exploit the pre-training for the X-ray report generation. For example, Wang et al. [] propose high-definition X-ray vision models using context-aware masked auto-encoder. CXR-CLIP [] is a new pre-training method that generates more image-text pairs and introduces contrastive loss to enhance the discriminative power of images and texts, effectively learning features in the CXR domain. PTUnifier [] proposes a simple and effective method that utilizes visual and textual prompt pools to make the model compatible with different types of inputs, thereby unifying the advantages of fusion encoders and dual encoders.
However, we believe these models may be limited by the following issues:
Firstly, the Transformer vision backbone brings huge computational costs $\mathcal{O}(N^{2})$, which is not hardware friendly;
Secondly, many X-ray models are pre-trained in a single stage, which may constrain their overall performance. As pure X-ray images are abundant and readily collectible, paired X-ray and report data are relatively scarce. Failing to utilize these visual data resources would be a significant missed opportunity.

To address the issues mentioned above, in this work, we exploit multi-stage pre-training for the X-ray image MRG task and propose the MambaXray-VL large model, including self-supervised autoregressive generation and Xray-report contrastive learning, and supervised fine-tuning on each downstream report generation datasets, as shown in Fig. 2. Specifically speaking, we first partition and feed the X-ray image into the Mamba network to predict the next tokens based on previous context tokens in an autoregressive generation manner. This will enhance the vision perception ability of X-ray significantly using the relatively low-cost Mamba network ($\mathcal{O}(N)$). For the second stage, we feed the paired X-ray image and corresponding reports into the Mamba vision backbone and text encoder (Bio_ClinicalBERT [], Llama2 []) for contrastive learning. It will align the X-ray image and reports using the pre-trained feature space. After that, we conduct supervised fine-tuning on each downstream X-ray report generation dataset to achieve higher performance by feeding the X-ray image into the pre-trained Mamba vision backbone network and LLM decoder network. Extensive experiments on three MRG benchmark datasets demonstrate that our pre-trained MambaXray-VL model achieves state-of-the-art performance.

To sum up, the contributions of this paper can be summarized as the following three aspects:

1). We conduct a comprehensive benchmark for the newly released CheXpert Plus dataset [], termed CXPMRG-Bench, which covers 19 mainstream X-ray medical report generation algorithms, 14 large language models, and 2 vision-language models. To the best of our knowledge, this benchmark is the first large-scale evaluation of the CheXpert Plus dataset, providing subsequent researchers in the field of X-ray report generation with important reference and comparison criteria.

2). We propose a new pre-trained large model, termed MambaXray-VL, which adopts the Mamba as the vision encoder and the large language model as the text decoder. Unlike conventional complex Transformer vision models, our Mamba architecture, which employs a multi-stage pre-training strategy, has also achieved state-of-the-art performance.

3). We extend our research to a broader scope by conducting experiments on the IU X-ray and MIMIC-CXR datasets. We perform analytical experiments and visualizations to deepen the understanding of our MambaXray-VL model’s performance and its capabilities in generating X-ray medical reports, thereby enhancing the robustness and generalizability of our findings across different datasets.

The rest of this paper is organized as follows:
In section 2, we review the related works to this paper including X-ray medical report generation, pre-trained large models, and state space model.
We introduce the pre-trained MambaXray-VL large model for the X-ray medical report generation in section 3.
After that, we introduce the CXPMRG-Bench benchmark on the CheXpert Plus dataset in section 4.
The experimental configurations and analysis are described in section 5.
Finally, we conclude this paper and propose possible research directions in section 6.

## 2 Related Work

In this section, we will review the related works on X-ray Medical Report Generation, Pre-trained Large Models, and State Space Models. More works can be found in the following surveys [, , ].

### 2.1 X-ray Medical Report Generation

In recent years, X-ray medical report generation has garnered increasing attention. To enhance model performance, researchers have pursued various improvements in different directions.
Specifically, DCL [] introduces a Dynamic Graph at the visual features of medical images, leveraging knowledge to strengthen the feature representation of these images. RGRG [] takes a novel approach by using object detection methods to extract lesion regions and then generating text based on these extracted regions, ultimately combining all the text to form the final report. HERGen [] discovers the historical information between medical reports, treating all reports of a patient as a temporally ordered whole. This approach effectively integrates the temporal and causal information of the reports.
R2GenGPT [] replaces the decoder part of the traditional medical report generation framework with a more powerful large language model, achieving improved performance.
R2GenCSR [] is a recently proposed LLM-based framework for X-ray MRG which employs the Mamba as the visual backbone and retrieves contextual samples from the training set to enhance feature representation and discriminative learning.

It is evident that the vision encoders used in these models are all conventional networks pre-trained on ImageNet []: DCL [] employs ViT [], RGRG [] uses ResNet50 [], HERGen [] utilizes CvT [], R2GenGPT [] incorporates SwinTransformer [], and R2GenCSR [] leverages VMamba [].
These encoders, pre-trained on non-medical X-ray images, exhibit certain limitations when extracting features from medical X-ray images. In contrast, our proposed MambaXray-VL is pre-trained on millions of datasets and has a natural advantage in the extraction of features from medical images, especially in the task of medical report generation.

### 2.2 Pre-trained Large Models

The pre-trained language models, vision models, and vision-language models are widely exploited in nowadays. Currently, the widely used MAE [] (Masked Autoencoders) is a self-supervised learning method for computer vision, known for its scalability and simplicity.
Recently, Apple’s team proposed AIM [], a series of vision models using autoregressive objectives for pretraining, inspired by large language models, demonstrating similar scaling properties.
ARM [] is a new self-supervised visual representation learning method based on AIM [] and Mamba []. Through the autoregressive generation based pre-training, the visual capabilities of the Mamba model can be significantly enhanced, outperforming other training strategies in terms of both efficiency and performance.
CLIP [] (Contrastive Language-Image Pre-Training) jointly trains image and text encoders using contrastive learning. The key idea is to enable the model to understand and process multi-modal data (images and text) through joint training.
Inspired by these works, our newly proposed MambaXray-VL utilizes autoregressive generation based pre-training, and CLIP pre-training can achieve better results on medical report generation.

### 2.3 State Space Model

Since its introduction in 2017, Transformer [] has quickly become the preferred model framework for researchers due to its strong performance. However, as the model scales and sequences become longer, its limitations have surfaced. One major drawback is the quadratic growth in computational complexity of the self-attention mechanism with increased context length. Mamba [] addresses these issues by using Selective State Space Models (SSMs) to improve traditional state space models and incorporating a hardware-aware parallel algorithm for recurrent operations.
Vim [] (Vision Mamba) is the first SSM model adapted for vision tasks. It uses positional embeddings and bidirectional state space models to achieve high performance, particularly on high-resolution images. VMamba [] extends Mamba by providing a global receptive field with linear complexity. MambaMLP [] is a new architectural component based on Mamba, designed to enhance feature mixing and representation learning by combining Mamba with an MLP, thereby improving performance on visual tasks. The new SSD (State Space Duality) algorithm proposed by Mamba-2 [] can fully utilize matrix multiplication units on modern hardware, making it 2-8 times faster than the vanilla Mamba.
The successful applications of the Mamba in many computer vision tasks [, , ] inspired us to adapt it to the pre-trained X-ray large model for medical report generation.

## 3 MambaXray-VL Large Model

In this section, we will first give an overview of our proposed MambaXray-VL large model, then, we will dive into the details of the proposed multi-stage training strategy. Finally, we highlight some implementation details worth noting in the pre-training phase.

Figure: Figure 2: An overview of our proposed MambaXray-VL pre-training framework. It contains three training stages, i.e., Mamba-based autoregressive generation, Xray-report based contrastive learning, and supervised fine-tuning. Specifically, the first phase mainly aims to make full use of larger-scale X-ray visual data to obtain a better visual backbone network (this paper chooses the low-complexity Mamba model). The second phase uses image-text contrastive loss to align X-ray images with medical reports. The third phase can fine-tune on various medical report generation datasets to obtain more refined X-ray report generation results. Note that the layers or modules with fire/snow symbols denote the parameters that are tuned/frozen in the training phase.
Refer to caption: extracted/5891527/figures/framework.jpg

### 3.1 Overview

As shown in Fig. 2, we propose a new multi-stage pre-training strategy for the X-ray image medical report generation, including self-supervised autoregressive generation, Xray-report contrastive learning, and supervised fine-tuning. The key insight of multi-stage pre-training instead of joint training is that the aligned Xray-report data are limited, but there are more publicly available X-ray images. Thus, we first pre-training a large-scale vision backbone network on the X-ray images using the Mamba layers, due to a better balance between the computational cost and accuracy. More importantly, we adopt the autoregressive generation to achieve self-supervised learning on the X-ray image. It performs similar or better than the widely used MAE (Masked Auto-Encoder) pre-training strategy for this task. Then, we transfer the Mamba vision backbone to the second stage, i.e., Xray-report contrastive learning. Specifically, we feed the paired data into the pre-trained Mamba vision backbone and language encoder for the vision-language feature extraction. This stage will project the vision and language representations into a shared feature space to bridge the vision-semantic gaps. Finally, we conduct supervised fine-tuning on the training subset of downstream datasets for the X-ray medical report generation.

### 3.2 Multi-Stage Pre-training

As illustrated in Fig. 2, our proposed MambaXray-VL large model contains three training stages which will be introduced in the following paragraphs respectively.

$\bullet$ Stage #1: Auto-regressive Generation for Mamba Vision Encoder Pre-training.
To make full use of existing X-ray images, we conduct self-supervised learning to obtain a strong vision backbone network. Different from the widely used MAE (Masked Auto-Encoder)-based framework, in this work, we find that the autoregressive generation based framework works similar or even better for the X-ray images, inspired by the success of autoregressive generation in ChatGPT [], GPT-4 [], and ARM [].
Let’s denote the X-ray image as $\mathcal{I}\in\mathbb{R}^{192\times 192\times 3}$, we first partition it into non-overlapping image patches $\mathcal{P}_{i}\in\mathbb{R}^{16\times 16\times 3},i=\{1,2,...,N\}$ and project them into visual tokens $\mathcal{T}_{i}\in\mathbb{R}^{1024},i=\{1,2,...,N\}$ using a convolutional layer (kernel size $16\times 16$). Here, N is 144 when the resolution of the input X-ray image is set as $3\times 192\times 192$. Then, we feed the visual tokens into the Vim [] backbone network for feature extraction whose complicity $\mathcal{O}(N)$ is much lower than the widely used Transformer $\mathcal{O}(N^{2})$. The key operation of Vim is the Mamba block (a specific variation of State Space Model []), as shown in Fig. 2. The visual tokens will first be normalized and fed into the SSM and scan branches. The outputs will be multiplied and added with residual connections. The SwiGLU [] is adopted to further process output features before being fed into subsequent Mamba blocks. Finally, an MLP layer is adopted for token reconstruction using the auto-regressive generation loss function.

The objective of autoregressive pre-training is to predict the probability of the next token one by one based on the given corpus $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},...,\mathcal{T}_{n}\}$, which can be written as:

$$ $\displaystyle p(\mathcal{T})=\prod_{i=1}^{n}p(\mathcal{T}_{i}|\mathcal{T}_{1}, ...,\mathcal{T}_{i-1},\Theta).$ (1) $$

We can find that the likelihood of each token $\mathcal{T}_{i}$ is computed based on the context of all the proceeding tokens $\{\mathcal{T}_{1},...,\mathcal{T}_{i-1}\}$. Thus, the loss function used for stage 1 can be formulated as follows:

$$ $\displaystyle\mathcal{L}_{AR}=\sum_{i=1}^{n-1}|Vim([\mathcal{T}_{1},..., \mathcal{T}_{i}])-\mathcal{T}_{i+1}|^{2}.$ (2) $$

$\bullet$ Stage #2: Xray-Report Contrastive Learning.
We adopt the Mamba vision backbone network from the first stage and conduct contrastive learning on the paired Xray-report samples. This will further align the dual modalities as validated in the CLIP []. In our implementation, we randomly sample a mini-batch and feed the X-ray images and medical reports into the Vim backbone and the language model (Bio_ClinicalBERT [], Llama2 []) and compute the cosine similarity between the paired and unpaired samples:

$$ $\displaystyle\mathcal{L}_{CTL}=Similarity(Vim(\mathcal{I}_{i}),LM(\mathcal{R}_ {j})),$ (3) $$

where $i$ and $j$ are the index of the X-ray image and report annotation.

$\bullet$ Stage #3: Supervised Fine-tuning.
After the two pre-training stages, we conduct supervised fine-tuning on the training subset of X-ray image medical report generation. Similar to the first stage, we partition the given X-ray image into non-overlapping patches and project them into visual tokens. Then, the pre-trained Vim backbone network is used for the feature extraction. We concatenate the visual tokens and generation prompt as the input of a large language model for high-performance medical report generation.

In this stage, we adopt the negative log-likelihood as the loss function, i.e.,

$$ $\displaystyle\mathcal{L}_{NLL}=-\sum_{i=1}^{T}logp_{\theta}(y_{i}|Prompt,[y_{1 },...,y_{i-1}]),$ (4) $$

where $\theta$ denotes the trainable parameters and $T$ is the number of words the large language model predicted.
$Prompt$ is the instruction prompt which is “Generate a comprehensive and detailed diagnosis report for this chest X-ray image.” used in our experiments.

### 3.3 Implementation Details

$\bullet$ Pre-training Stage.
Both MambaXray-VL-Base and MambaXray-VL-Large were pre-trained for 100 epochs, with batch sizes set at 256 and 128, respectively. The base learning rate, based on a batch size of 256, was set to 1.5e-4. We adopted a cosine decay schedule with a warm-up for 5 epochs and used the AdamW [] optimizer with a weight decay of 0.05. The resolution of input images is resized to 192 $\times$ 192 in the pre-training phase.

In the second stage, we utilized a vision-text contrastive learning pre-training method to train MambaXray-VL, enabling alignment to the text feature space. Specifically, we used a dataset of 480,000 image-text pairs, composed of publicly available datasets from MIMIC-CXR [], CheXpert Plus [], and IU-Xray []. Inspired by ARM [], we used a unidirectional scanning approach in the first stage that fits the autoregressive generation to achieve more efficient pre-training. In the second stage, we extend the scanning block to four copies in order to improve the performance of the model. During this stage, we chose to pre-training for 50 epochs, with a batch size set to 192. The visual encoder was Vim [], loaded with weights from the first stage of pre-training, while the text encoder was Bio_ClinicalBERT [], both encoders were set to be trainable. We employed the same optimizer as in the first stage, but the input image size was changed to 224 $\times$ 224.

**Table 1: Experimental Results on the CheXpert Plus Dataset using Mainstream Medical Report Generation Algorithms. B4, R, M, and C is short for BlEU-4, ROUGE-L, METEOR, CIDEr, respectively. P, R, and F1 is short for Precision, Recall, F1 score, respectively. min is short for minutes. The Param listed in this table denotes the parameters needed to be tuned in the training phase. The best result is highlighted in bold, and the second-best result is underlined.**
| Index | Algorithm | Publish | Encoder | Decoder | B4,      R,      M,     C | P,      R,      F1 | Time (min) | Param (M) | Code |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #01 | R2GenRL [] | ACL22 | Transformer | Transformer | 0.035, 0.186, 0.101, 0.012 | 0.193, 0.229, 0.196 | 44.33 | 59.87 | URL |
| #02 | XProNet [] | ECCV22 | Transformer | Transformer | 0.100, 0.265, 0.146, 0.121 | 0.314, 0.247, 0.259 | 6.3 | 62.35 | URL |
| #03 | MSAT [] | MICCAI22 | ViT-B/16 | Transformer | 0.036, 0.156, 0.066, 0.018 | 0.044, 0.142, 0.057 | 5.72 | 141.10 | URL |
| #04 | ORGan [] | ACL23 | CNN | Transformer | 0.086, 0.261, 0.135, 0.107 | 0.288, 0.287, 0.277 | 46.66 | 67.50 | URL |
| #05 | M2KT [] | MIA21 | CNN | Transformer | 0.078, 0.247, 0.101, 0.077 | 0.044, 0.142, 0.058 | 22.5 | 69.07 | URL |
| #06 | TIMER [] | CHIL23 | Transformer | Transformer | 0.083, 0.254, 0.121, 0.104 | 0.345, 0.238, 0.234 | 26.5 | 79.28 | URL |
| #07 | CvT2DistilGPT2 [] | AIM23 | Transformer | GPT2 | 0.067, 0.238, 0.118, 0.101 | 0.285, 0.252, 0.246 | 13.93 | 128 | URL |
| #08 | R2Gen [] | EMNLP20 | Transformer | Transformer | 0.081, 0.246, 0.113, 0.077 | 0.318, 0.200, 0.181 | 110.05 | 83.5 | URL |
| #09 | R2GenCMN [] | ACL21 | Transformer | Transformer | 0.087, 0.256, 0.127, 0.102 | 0.329, 0.241, 0.231 | 66.08 | 67.70 | URL |
| #10 | Zhu et al. [] | MICCAI23 | Transformer | Transformer | 0.074, 0.235, 0.128, 0.078 | 0.217, 0.308, 0.205 | 10.03 | 85.95 | URL |
| #11 | CAMANet [] | IEEE JBH23 | Swin-Former | Transformer | 0.083, 0.249, 0.118, 0.090 | 0.328, 0.224, 0.216 | 23.08 | 43.22 | URL |
| #12 | ASGMD [] | ESWA24 | ResNet-101<br>Transformer | Transformer | 0.063, 0.220, 0.094, 0.044 | 0.146, 0.108, 0.055 | 87.37 | 277.41 | URL |
| #13 | Token-Mixer [] | IEEE TMI23 | ResNet-50 | Transformer | 0.091, 0.261, 0.135, 0.098 | 0.309, 0.270, 0.288 | 17.54 | 104.34 | URL |
| #14 | PromptMRG [] | AAAI24 | ResNet-101 | Bert | 0.095, 0.222, 0.121, 0.044 | 0.258, 0.265, 0.281 | 108.45 | 219.92 | URL |
| #15 | R2GenGPT [] | Meta-Rad.23 | Swin-Transformer | Llama2 | 0.101, 0.266, 0.145, 0.123 | 0.315, 0.244, 0.260 | 77.8 | 90.9 | URL |
| #16 | WCL [] | EMNLP21 | Transformer | Transformer | 0.084, 0.253, 0.126, 0.103 | 0.335, 0.259, 0.256 | 24.08 | 81.29 | URL |
| #17 | R2GenCSR [] | arXiv24 | VMamba | Llama2 | 0.100, 0.265, 0.146, 0.121 | 0.315, 0.247, 0.259 | 31.2 | 91.7 | URL |
| #18 | VLCI [] | arXiv24 | Transformer | Transformer | 0.080, 0.247, 0.114, 0.072 | 0.341, 0.175, 0.163 | 123.71 | 91.46 | URL |
| #19 | Wang et al. [] | arXiv24 | ViT | Llama2 | 0.064, 0.220, 0.110, 0.059 | 0.175, 0.099, 0.078 | 10.82 | 358.80 | URL |
| #20 | MambaXray-VL-B | Ours | MambaXray-VL | Llama2 | 0.105, 0.267, 0.149, 0.117 | 0.333, 0.264, 0.273 | 50.66 | 57.31 | URL |
| #21 | MambaXray-VL-L | Ours | MambaXray-VL | Llama2 | 0.112, 0.276, 0.157, 0.139 | 0.377, 0.319, 0.335 | 55.18 | 202.32 | URL |

$\bullet$ Fine-tuning Stage.
In the downstream fine-tuning stage, we tested the model’s performance on three different public datasets. On the IU-Xray [] dataset, we set the maximum training epochs to 30 and the batch size to 20. The visual encoder used was Vim [], loaded with weights from the second stage of pretraining, while the large language model was Qwen-1.5-1.8B [], with max_length set to 60 and a validation frequency of 1, meaning we validated after each training epoch. On the MIMIC-CXR [] and CheXpert Plus [] datasets, we set the maximum training epochs to 6 and the batch size to 18. The visual encoder remained unchanged, while the large language model used was Llama2-7B [], with max_length set to 100 and a validation frequency of 0.5, meaning we validated at both the end of each training cycle and after the training was complete. We froze the large language model and trained only the visual encoder and the visual mapper layer, by following the R2GenGPT [].

## 4 CXPMRG-Bench

In this paper, we benchmark the newly released CheXpert Plus dataset for the X-ray image based medical report generation. The mainstream MRG algorithms and large language models are listed in the following subsections. For the experimental results, please refer to Table 1, Table 2, and Fig. 1.

### 4.1 Mainstream MRG Algorithms

For the mainstream X-ray image MRG algorithms, as shown in Table 1, we train and test 21 open-sourced algorithms from the year 2020 to the year 2024. These models adopt the
CNN (ORGan [], M2KT [], ASGMD [], Token-Mixer [], PromptMRG []),
Transformer (R2GenRL [], XProNet [], MSAT [], TIMER [], CvT2DistilGPT2 [], R2Gen [], R2GenCMN [], Zhu et al. [], CAMANet [], R2GenGPT [], WCL [], VLCI [], Wang et al. []), and
Mamba (R2GenCSR [], MambaXray-VL-B, MambaXray-VL-L) as their vision backbone network, and utilize the LSTM, Transformer based model as the decoder network. Note that, the MambaXray-VL-B and MambaXray-VL-L are two models proposed in this paper which will be introduced in the next section.

When reproducing these X-ray based MRG models, we found that some algorithms use truncated ground truth for comparison, which we believe may not accurately reflect the true evaluation results. Therefore, we abandoned the truncation mechanism and used the complete ground truth for result evaluation, making the obtained results more accurate and reliable.

### 4.2 LLMs for MRG

We evaluate a total of 16 open-source LLMs, as shown in Table 2, including Vicuna-7B [], QWen1.5-7B [], QWen2-7B-Instruct [], InternLM-7B [], Llama2-7B [], Llama2-13B [], Llama3-8B [], Llama3.1-8B [], GPT2-Medium [], Orca 2-7B [], Orca 2-13B [], DeepSeek-LLM-7B-Chat [], Yi-1.5-6B-Chat [], Yi-1.5-9B-Chat []. Note that part of the LLMs is selected from open-llm-leaderboard (^1^11[https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)) and integrated with R2GenGPT [] model by replacing the Llama2 language decoder with corresponding LLMs. In our implementation, we keep the visual encoder SwinTransformer unchanged for a fair comparison. In addition, we also test two pre-trained vision-language large models, i.e., InternVL-2 [] and MiniCPM-V2.5 [], to check whether a better performance can be obtained, as shown in Table 2.

### 4.3 Evaluation Results

[Mainstream MRG Models] As shown in Table 1, there are five MRG models which achieve a higher B4 metric, i.e., the XProNet [] (0.100), R2GenGPT [] (0.101), R2GenCSR [] (0.100), and our newly proposed MambaXray-VL-B and MambaXray-VL-L which achieves 0.105, and 0.112, respectively. It is intuitive to find that the large language model Llama2 works well for the MRG task. For F1 in the clinical metric, the top-5 models are our newly proposed MambaXray-VL-L (0.335), Token-Mixer [] (0.288), PromptMRG [] (0.281), ORGan [] (0.277) and our proposed MambaXray-VL-B (0.273). From these results, we can find that our proposed multi-stage pre-training strategy is rather effective in the disease-aware perception of the MRG.

[LLM/VLM based MRG Models] As shown in Table 2, we also report the performance of existing widely used LLMs by replacing the Llama2 based on the R2Gen-GPT framework (SwinTransformer is adopted as the vision backbone network). It is easy to find that the Vicuna-V1.5 [] released in the year 2023 achieves the best B4 metric and the InternLM [] performs the best on the F1 clinical metric.
For the two vision-language models we evaluated, i.e., the InternVL-2 and MiniCPM-V2.5, we can find that their results are not as good as other LLM-based models, although they have similar parameters. These results demonstrate that the vision-language models pre-trained on natural image-pairs may have large gaps with the X-ray medical images. Compared with the mainstream MRG models reported in Table 1, the LLM-based MRG achieves better results than regular language decoders which demonstrates the effectiveness of pre-trained LLMs.

[Efficiency & Parameters] From the perspective of running efficiency, we test these models on a server with A800 GPUs (80GB). Note that, we set the batch size as large as possible to make full use of the GPU memory. As a result, we can find that MSAT [] and XProNet [] are the first two algorithms that only need 5.72 and 6.3 minutes for the testing subset. R2Gen [], PromptMRG [], and VLCI [] are relatively slow and need more than 100 minutes on the testing subset of CheXpert Plus dataset. For the LLM-based MRG reported in Table 2, we can find that Yi-1.5 [] with 6.1B and 8.8B achieves better efficiency which needs 43.66 and 48.50 minutes for the testing.
From the Fig. 1 and Table 1, we can find that the ASGMD [], PromptMRG [], Wang et al. [], and our MambaXray-VL-L contains the most parameters (larger than 200M) needed to be tuned in the training phase. However, we can find that our model runs faster than these large models which only need 55.18 minutes. It fully validated the efficiency of our proposed framework for the X-ray image based medical report generation.

**Table 2: Experimental Results of Medical Report Generation on the CheXpert Plus Dataset using different LLMs and VLMs based on R2Gen-GPT. The symbol $\dagger$ indicates that the model is a VLM. The Param listed in this table denotes the parameters of LLM/VLM.**
| Index | LLM/VLM | Year | B4 | R | M | C | P | R | F1 | Time (min) | Param | Code |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #01 | Vicuna-V1.5 [] | 2023 | 0.104 | 0.272 | 0.160 | 0.202 | 0.334 | 0.258 | 0.276 | 72.00 | 6.7B | URL |
| #02 | Qwen-1.5 [] | 2024 | 0.098 | 0.262 | 0.139 | 0.139 | 0.303 | 0.233 | 0.241 | 154.25 | 7.7B | URL |
| #03 | Qwen-2 [] | 2024 | 0.100 | 0.270 | 0.142 | 0.159 | 0.313 | 0.269 | 0.261 | 103.33 | 7.6B | URL |
| #04 | InternLM [] | 2024 | 0.063 | 0.207 | 0.136 | 0.104 | 0.307 | 0.274 | 0.284 | 294.00 | 7.3B | URL |
| #05 | Llama-2 [] | 2023 | 0.102 | 0.267 | 0.157 | 0.179 | 0.315 | 0.244 | 0.260 | 77.78 | 6.7B | URL |
| #06 | Llama-2 [] | 2023 | 0.101 | 0.269 | 0.160 | 0.214 | 0.321 | 0.254 | 0.267 | 116.42 | 13.0B | URL |
| #07 | Llama-3 [] | 2024 | 0.077 | 0.220 | 0.121 | 0.134 | 0.306 | 0.232 | 0.222 | 130.00 | 8.0B | URL |
| #08 | Llama-3.1 [] | 2024 | 0.075 | 0.221 | 0.121 | 0.136 | 0.295 | 0.251 | 0.242 | 110.00 | 8.0B | URL |
| #09 | GPT2-Medium [] | 2019 | 0.063 | 0.198 | 0.104 | 0.067 | 0.358 | 0.186 | 0.165 | 57.33 | 354M | URL |
| #10 | Orca-2 [] | 2023 | 0.103 | 0.270 | 0.161 | 0.199 | 0.330 | 0.251 | 0.271 | 177.33 | 6.7B | URL |
| #11 | Orca-2 [] | 2023 | 0.100 | 0.266 | 0.159 | 0.187 | 0.317 | 0.242 | 0.257 | 108.66 | 13.0B | URL |
| #12 | Deepseek-LLM [] | 2024 | 0.096 | 0.268 | 0.137 | 0.150 | 0.336 | 0.256 | 0.253 | 201.30 | 6.9B | URL |
| #13 | Yi-1.5 [] | 2024 | 0.091 | 0.263 | 0.131 | 0.136 | 0.322 | 0.229 | 0.226 | 43.66 | 6.1B | URL |
| #14 | Yi-1.5 [] | 2024 | 0.096 | 0.269 | 0.138 | 0.155 | 0.336 | 0.241 | 0.243 | 48.50 | 8.8B | URL |
| #15 | InternVL-2^$\dagger$ [] | 2023 | 0.058 | 0.188 | 0.112 | 0.085 | 0.196 | 0.127 | 0.132 | 108.50 | 8.0B | URL |
| #16 | MiniCPM-V2.5^$\dagger$ [] | 2024 | 0.046 | 0.177 | 0.085 | 0.076 | 0.254 | 0.152 | 0.122 | 51.50 | 8.4B | URL |

## 5 Experiments

### 5.1 Dataset

In the first stage of autoregressive pre-training, we used about 1.27 million medical chest X-ray images proposed in the work []. In the second stage of image-text contrastive learning pre-training, we used a combination of training data from the MIMIC-CXR [], CheXpert Plus [], and IU X-ray [] datasets, totaling 480k image-report pairs. Note that the CheXpert Plus dataset used here consists of images and impressions, not the image and findings combination used in the third stage. We strictly excluded any testing samples used in the third stage, resulting in a total of 210k image-impression pairs. In the third stage, We evaluate the performance of our model on three datasets, including IU X-Ray [], MIMIC-CXR [], and CheXpert Plus [] dataset. A brief introduction to these datasets is given below.

$\bullet$ IU X-ray Dataset  [] (^2^22[https://iuhealth.org/find-medical-services/x-rays](https://iuhealth.org/find-medical-services/x-rays)) published in 2016 is one of the most frequently used publicly available medical image datasets for medical report generation. It contains 7,470 images and 3,955 radiology reports, with each report associated with either frontal or both frontal and lateral view images. Each report is divided into four sections: Indication, Comparison, Findings, and Impression. For a fair comparison, we used the same dataset split protocol as R2GenGPT [], dividing the dataset into training, testing, and validation sets with a ratio of 7:1:2.

$\bullet$ MIMIC-CXR Dataset  [] (^3^33[https://physionet.org/content/mimic-cxr/2.0.0/](https://physionet.org/content/mimic-cxr/2.0.0/)) is one of the largest publicly available chest X-ray datasets, containing free-text radiology reports. These records from 2011-2016 include 377,110 radiographic images and 227,835 radiology reports collected from 65,379 patients at the Beth Israel Deaconess Medical Center Emergency Department in Boston, Massachusetts. For fair comparison, we used the same dataset split protocol as R2GenGPT, with 270,790 samples for training the model, and 2,130 and 3,858 samples for validation and testing sets, respectively.

$\bullet$ CheXpert Plus Dataset  [] (^4^44[https://github.com/Stanford-AIMI/chexpert-plus](https://github.com/Stanford-AIMI/chexpert-plus)) is a new radiology dataset designed to enhance the scale, performance, robustness, and fairness of deep learning models in the field of radiology. This dataset includes 223,228 chest X-rays (in DICOM and PNG formats), 187,711 corresponding radiology reports (de-identified and parsed into 11 sections), de-identified demographic data from 64,725 patients, 14 chest pathology labels, and RadGraph [] annotations. For a fair comparison, we followed the dataset split protocol used in R2GenCSR [] which adopted Findings as the ground truth and split the training/validation/testing subset based on the ratio 7:1:2. The training subset with 40,463 samples, the validation subset with 5,780 samples, and the testing subset with 11,562 samples.

### 5.2 Evaluation Metric

For the X-ray medical report generation, we evaluate the model using widely used natural language generation (NLG) metrics, including CIDEr [], BLEU [], ROUGE-L [], and METEOR []. More in detail,
CIDEr [] evaluates text through TF-IDF weighted n-gram matching, placing greater emphasis on the importance of words;
BLEU [] evaluates text quality through n-gram matching;
ROUGE-L [] evaluates text using the longest common subsequence;
METEOR [] improves upon BLEU by considering synonyms and word order.

To measure the accuracy of descriptions for clinical abnormalities, we also report Clinical Efficacy (CE) metrics. CE metrics require the use of the CheXPert [] toolkit to first extract labels from predictive reports and ground truth, and then to compare the presence status of important clinical observations to capture the diagnostic accuracy of the generated reports. We use Precision, Recall, and F1 to evaluate model performance for clinical efficacy metrics.

**Table 3: Comparison of our model’s performance on the IU X-ray and MIMIC-CXR datasets. The symbol $\dagger$ indicates that we follow the R2Gen annotation using Findings and evaluate with our method, as their report modifies the ground truth to an Impression concatenated with Findings. The best result is highlighted in bold, and the second-best result is underlined.**
| Dataset | Methods | Publication | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE-L | METEOR | CIDEr |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| IU X-Ray | R2Gen [] | EMNLP 2020 | 0.470 | 0.304 | 0.219 | 0.165 | 0.371 | 0.187 | - |
| R2GenCMN [] | ACL-IJCNLP 2021 | 0.475 | 0.309 | 0.222 | 0.170 | 0.375 | 0.191 | - |  |
| PPKED [] | CVPR 2021 | 0.483 | 0.315 | 0.224 | 0.168 | 0.376 | 0.187 | 0.351 |  |
| AlignTrans [] | MICCAI 2021 | 0.484 | 0.313 | 0.225 | 0.173 | 0.379 | 0.204 | - |  |
| CMCL [] | ACL 2021 | 0.473 | 0.305 | 0.217 | 0.162 | 0.378 | 0.186 | - |  |
| Clinical-BERT [] | AAAI 2022 | 0.495 | 0.330 | 0.231 | 0.170 | 0.376 | 0.209 | 0.432 |  |
| METransformer [] | CVPR 2023 | 0.483 | 0.322 | 0.228 | 0.172 | 0.380 | 0.192 | 0.435 |  |
| DCL [] | CVPR 2023 | - | - | - | 0.163 | 0.383 | 0.193 | 0.586 |  |
| R2GenGPT^$\dagger$ [] | Meta Radiology 2023 | 0.465 | 0.299 | 0.214 | 0.161 | 0.376 | 0.219 | 0.542 |  |
| PromptMRG [] | AAAI 2024 | 0.401 | - | - | 0.098 | 0.160 | 0.281 | - |  |
| BootstrappingLLM [] | AAAI 2024 | 0.499 | 0.323 | 0.238 | 0.184 | 0.390 | 0.208 | - |  |
| MambaXray-VL-Base | Ours | 0.479 | 0.322 | 0.236 | 0.179 | 0.388 | 0.215 | 0.508 |  |
| MambaXray-VL-Large | Ours | 0.491 | 0.330 | 0.241 | 0.185 | 0.371 | 0.216 | 0.524 |  |
| MIMIC-CXR | R2Gen [] | EMNLP 2020 | 0.353 | 0.218 | 0.145 | 0.103 | 0.277 | 0.142 | - |
| R2GenCMN [] | ACL-IJCNLP 2021 | 0.353 | 0.218 | 0.148 | 0.106 | 0.278 | 0.142 | - |  |
| PPKED [] | CVPR 2021 | 0.360 | 0.224 | 0.149 | 0.106 | 0.284 | 0.149 | 0.237 |  |
| AlignTrans [] | MICCAI 2021 | 0.378 | 0.235 | 0.156 | 0.112 | 0.283 | 0.158 | - |  |
| CMCL [] | ACL 2021 | 0.344 | 0.217 | 0.140 | 0.097 | 0.281 | 0.133 | - |  |
| Clinical-BERT [] | AAAI 2022 | 0.383 | 0.230 | 0.151 | 0.106 | 0.275 | 0.144 | 0.151 |  |
| METransformer [] | CVPR 2023 | 0.386 | 0.250 | 0.169 | 0.124 | 0.291 | 0.152 | 0.362 |  |
| DCL [] | CVPR 2023 | - | - | - | 0.109 | 0.284 | 0.150 | 0.281 |  |
| R2GenGPT^$\dagger$ [] | Meta Radiology 2023 | 0.408 | 0.256 | 0.174 | 0.125 | 0.285 | 0.167 | 0.244 |  |
| PromptMRG [] | AAAI 2024 | 0.398 | - | - | 0.112 | 0.268 | 0.157 | - |  |
| BootstrappingLLM [] | AAAI 2024 | 0.402 | 0.262 | 0.180 | 0.128 | 0.291 | 0.175 | - |  |
| MambaXray-VL-Base | Ours | 0.420 | 0.264 | 0.180 | 0.129 | 0.283 | 0.162 | 0.206 |  |
| MambaXray-VL-Large | Ours | 0.422 | 0.268 | 0.184 | 0.133 | 0.289 | 0.167 | 0.241 |  |

### 5.3 Comparison with SOTA Algorithms

**Table 4: Component analysis of the key modules in our framework on MIMIC-CXR and CheXpert Plus dataset. The symbol $\dagger$ indicates that we are using the Base version of the model, while the others are the Large versions. Vim-IN1K indicates the use of weights pre-trained on ImageNet-1K; Vim-PTD indicates the use of weights pre-trained on 1.27 million X-ray images; MAE represents the Masked Auto-encoders pre-training framework; ARG represents the Auto-regressive Generation pre-training framework; CTL represents the contrastive learning loss between images and text; SFT represents supervised fine-tuning. B4, R, M, and C represents BLEU-4, ROUGE-L, METEOR, and CIDEr, respectively.**
| Index | Vim-IN1K | Vim-PTD | MAE | ARG | CTL | SFT | MIMIC-CXR | CheXpert Plus |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B4 | R | M | C | B4 | R | M | C |  |  |  |  |  |  |  |
| $\#01$ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 0.125 | 0.285 | 0.167 | 0.244 | 0.101 | 0.266 | 0.145 | 0.123 |
| $\#02$ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | 0.104 | 0.260 | 0.141 | 0.154 | 0.094 | 0.257 | 0.140 | 0.104 |
| $\#03$ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | 0.130 | 0.286 | 0.162 | 0.224 | 0.089 | 0.247 | 0.134 | 0.089 |
| $\#04$ ^$\dagger$ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | 0.108 | 0.264 | 0.144 | 0.170 | 0.090 | 0.249 | 0.132 | 0.103 |
| $\#05$ ^$\dagger$ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | 0.121 | 0.280 | 0.161 | 0.224 | 0.093 | 0.254 | 0.138 | 0.102 |
| $\#06$ ^$\dagger$ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | 0.129 | 0.283 | 0.162 | 0.206 | 0.105 | 0.267 | 0.149 | 0.117 |
| $\#07$ | ✓ | ✗ | ✗ | ✓ | ✗ | ✓ | 0.105 | 0.258 | 0.139 | 0.143 | 0.082 | 0.236 | 0.126 | 0.080 |
| $\#08$ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | 0.130 | 0.286 | 0.162 | 0.224 | 0.089 | 0.247 | 0.134 | 0.089 |
| $\#09$ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | 0.133 | 0.289 | 0.167 | 0.241 | 0.112 | 0.276 | 0.157 | 0.139 |

$\bullet$ Results on IU X-ray Dataset.
As shown in Table 3, it can be seen that both our MambaXray-VL-Base and MambaXray-VL-Large exhibit excellent performance on the IU X-ray dataset. Among them, the MambaXray-VL-Large model is at the SOTA level on BLEU-2 (B2), BLEU-3 (B3), and BLEU-4 (B4) metrics with scores of 0.330, 0.241, and 0.185, respectively. This result indicates the superiority of our method over other report generation methods. However, on some other metrics such as BLEU-1 (B1), ROUGE-L (R), METEOR (M), and CIDEr (C), our method does not achieve optimal performance. This reflects the need to improve the generalization of our method on other datasets.

**Table 5: Comparison of the text encoders used in the second stage on the MIMIC-CXR and CheXpert Plus datasets.**
| LLM | MIMIC-CXR | CheXpert Plus |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BLEU-4 | ROUGE-L | METEOR | CIDEr | BLEU-4 | ROUGE-L | METEOR | CIDEr |  |
| Baseline | 0.125 | 0.285 | 0.167 | 0.244 | 0.101 | 0.266 | 0.145 | 0.123 |
| Llama2 [] | 0.122 | 0.276 | 0.157 | 0.211 | 0.066 | 0.233 | 0.124 | 0.043 |
| Bio_ClinicalBERT [] | 0.133 | 0.289 | 0.167 | 0.241 | 0.112 | 0.276 | 0.157 | 0.139 |

$\bullet$ Results on MIMIC-CXR Dataset.
As shown in Table 3, our method also demonstrates outstanding performance on the MIMIC-CXR dataset, surpasses all other advanced report generation methods, and achieves the most advanced level in several common indicators (e.g., BLEU-1, BLEU-2, BLEU-3, and BLEU-4). Specifically, our method improves the BLEU-4 metric by 6% compared to R2GenGPT. Encouragingly, we achieved favorable results for two of the three remaining metrics, ROUGE-L and METEOR, with scores of 0.289 for ROUGE-L and 0.167 for METEOR, which again demonstrates the superior performance of our model. In the CIDEr metric, our model achieved a score of 0.241, indicating that MambaXray-VL still has room for improvement.

$\bullet$ Results on CheXpert Plus Dataset.
As shown in Table 1, our model MambaXray-VL-Large achieves state-of-the-art performance in all evaluation metric species. These include NLG evaluation metrics and CE evaluation metrics. In detail, for the NLG metrics, our scores on BLEU-4, ROUGE-L, METEOR, and CIDEr are 0.112, 0.276, 0.157, and 0.139, respectively. For the CE metrics, our scores on Precision (P), Recall (R), and F1-score (F1) are 0.377, 0.319, and 0.335, respectively. These experimental results fully demonstrate the superior performance of our model on the CheXpert Plus dataset. In terms of efficiency, our method took 55.18 minutes to complete the testing subset of the CheXpert Plus dataset with a parameter size of 202.32M, showing its effectiveness and efficiency in processing X-ray images.

### 5.4 Ablation Study

Figure: Figure 3: X-ray images and their corresponding ground-truths, along with the output of our model and R2GenGPT model generation reports on the MIMIC-CXR dataset. Matching sentences in our report are highlighted in yellow, R2GenGPT matching sentences are highlighted in cyan, and sentences matching by both models are highlighted in pink.
Refer to caption: extracted/5891527/figures/0.png

$\bullet$ Effectiveness of Autoregressive Generation for Pre-training on X-ray Image?
As shown in Table 4, we first compare the autoregressive generation (ARG) pre-training with the Masked Auto-Encoder (MAE) pre-training. From the $\#02$ and $\#03$ rows, it can be seen that the results achieve 0.130/0.089 on the BLEU-4 metric of the MIMIC-CXR and CheXpert Plus datasets, respectively. Note that the ARG pre-training method outperforms the MAE on all metrics, with a +45% (i.e., (0.224-0.154)/0.154) improvement on CIDEr compared to MAE. The ARG-based pre-training achieves similar performance compared with MAE-based pre-training on the CheXpert Plus dataset.

$\bullet$ Effectiveness of Xray-Report Contrastive Learning.
In addition, we further explored the impact of contrastive learning (CTL) on the final performance. The experimental results in the $\#05$ and $\#06$ rows of Table 4 demonstrate its effectiveness. After introducing the CTL loss, we find that the results on the MIMIC-CXR and CheXpert Plus datasets have all received improvement. More in detail, it improves the ROUGE-L metric by over +5% on the CheXpert Plus dataset. These experiments demonstrate the positive effect of the CTL loss we used in the pre-training stage.

$\bullet$ Comparison between ViT and Mamba using Autoregressive Generation.
As shown in the $\#01$ and $\#09$ rows of Table 4, the $\#01$ row uses a visual coder based on the Transformer architecture, while the last row uses a visual coder with auto-regressive pre-training of the Mamba architecture. It can be clearly observed that the encoder based on the Mamba architecture achieves better performance in the vast majority of metrics, both on the MIMIC-CXR and CheXpert Plus datasets, especially on BLEU-4 for the MIMIC-CXR data, where the Mamba architecture improves by +6% compared to the Transformer architecture. However, on the MIMIC-CXR dataset, the metric CIDEr does not score significantly better than the Transformer architecture. Overall, this series of experiments is sufficient to demonstrate the effectiveness of the auto-regressive pre-trained visual coder based on the Mamba architecture.

$\bullet$ Clinical-BERT vs Llama2 in Xray-Report Contrastive Learning.
In this work, we test two models for contrastive learning in the second stage, i.e., the Bio_ClinicalBERT [] and Llama2 []. As shown in Table 5, the experimental results on both MIMIC-CXR and CheXpert Plus datasets all demonstrate that the Bio_ClinicalBERT [] achieves a better performance for the X-ray report generation. We think this may be caused by the fact that the Bio_ClinicalBERT [] is an LLM pre-trained using medical data, while the Llama2 [] is pre-trained using common text data and sensitive to parameter tuning. This experiment inspired us to consider pre-training large language models using medical data in future works.

$\bullet$ Analysis on Different Configurations of Mamba Vision Encoder.
Intuitively, the large version of the Mamba model has better generalization and robustness compared to the base version, as it has deeper network layers or higher feature dimensions. As shown in Table 4, we can see that the results in lines $\#7$, $\#8$, and $\#9$ (Vim-large) are significantly better than lines $\#4$, $\#5$, and $\#6$ (Vim-base). Meanwhile, our Vim-large achieved optimal performance in experiments after equipping all modules. Thus, it is obvious that the larger version of Vim has a more stable performance on both MIMIC-CXR and CheXpert Plus datasets.

$\bullet$ Does VLMs Pre-trained using Natural Image-Text Samples Ready for the X-ray Report Generation? In this paper, we also conduct supervised fine-tuning on the CheXpert Plus dataset using Vision-Language Models (VLMs), including InternVL-2 [] and MiniCPM V2.5 []. We replace the vision and language backbone network of R2Gen-GPT using the VLMs to adapt them for the X-ray image based report generation task. As illustrated in Table 2, we can find that the performance of the two models is not as good as the compared models. These experiments demonstrate a large gap between pre-training on the natural and X-ray images. In our future works, we consider further adapting the pre-trained VLMs using natural images to the X-ray image domain to achieve a better performance.

### 5.5 Visualization

As shown in Fig. 3, we give some examples to illustrate the effectiveness of our proposed MambaXray-VL model for the X-ray image based report generation.
For specific X-ray images, we compared ground truth with the report generated by the MambaXray-VL model and the report generated by the R2GenGPT model. The X-ray images we chose contain both front and side views, normal images, and images containing lesion areas, enabling a more comprehensive and rational visualization. For a more intuitive visualization, we have highlighted the parts that match the ground truth. The yellow highlighted area is the part of the report generated by our model that matches the ground truth, and the blue highlighted area is the part of the report generated by the R2GenGPT model that matches the ground truth. The pink highlighted area is the portion of the report generated by both our model and the R2GenGPT model that matches the ground truth. It is clear that the report generated by our model is closer to the real report than the report generated by the R2GenGPT model, which indicates that our model is effective.

### 5.6 Limitation Analysis

This paper provides a comprehensive benchmark for the X-ray image based medical report generation, which covers the mainstream MRG models and LLMs. The LLMs evaluated in this work focus on 7B and 13B which is hardware friendly, and the LLMs with more parameters are not discussed due to the limited computational resources. On the other hand, there are still many Vision-Language Models (VLMs) developed for natural images that are not benchmarked, due to the limited performance of the X-ray image-based medical report generation.

## 6 Conclusion and Future Works

In this work, we propose to benchmark the CheXpert Plus dataset by re-training the mainstream X-ray report generation models and large language models. This benchmark will help identify which large models and algorithms are leading in this domain, significantly promoting academic progress and technological development. In addition, we also propose a new Mamba-based vision-language large model for the X-ray image based medical report generation. It involves three pre-training stages which make full use of auto-regressive generation loss, Xray-report contrastive learning, and supervised fine-tuning. We validate the effectiveness of our proposed pre-trained large model on IU X-ray, MIMIC-CXR, and CheXpert Plus datasets. From the newly built benchmark, we can find that the current large language models still perform poorly on the report generation task.

In our future works, we will consider introducing structured knowledge graphs into the large language model to guide the report generation. In addition, fine-grained X-ray image patch mining guided by the medical report may be another idea worthy of study. We leave them as the future works.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1- Provide a better description of the technical contribution 

2- Elaborate more (preferrable theoretically) why an auto-regressive approach is better than a masked auto-encoder in this problem.

3- I suggest benchmarking with a few recent approaches as shown in the weaknesses section (point 3).
soundness: 3
strengths: 1- The paper provides a comprehensive comparison and benchmark of the CheXpert Plus dataset, which was recently published.

2- The paper aggregates multiple technical components and shows how these results outperform many existing methods.

3- Ablation studies are extensive and sound.

4- The paper is simple to follow and read.
summary: The paper presents a thorough benchmarking of several X-ray report generation models and LLMs on the CheXpert Plus dataset. It argues that this benchmark provides a solid comparative framework for future algorithms. Additionally, the paper introduces a modified version of the vision Mamba (Vim) for X-ray image report generation. The paper incorporates Mamba-based self-supervised autoregressive generation, X-ray report contrastive learning, and supervised fine-tuning in their method. The paper presents extensive experimental results and shows that the Mamba-based autoregressive pre-training encodes X-ray images well. Ablation studies demonstrate results under different scenarios.
weaknesses: 1- The technical novelty of the work is limited. However, the amount of work performed in the paper is significant.

2- The justification provided why an autoregressive method works better than other methods e.g., masked auto-encoder is rather limited. 

3- Some of the benchmark methods presented in Table 1 are relatively old and several newer methods were not included. For instance, the below papers were public by June 2024 (when the dataset was made public). 
https://arxiv.org/pdf/2311.18681
https://openaccess.thecvf.com/content/CVPR2024/papers/Bu_Instance-level_Expert_Knowledge_and_Aggregate_Discriminative_Attention_for_Radiology_Report_CVPR_2024_paper.pdf
https://openaccess.thecvf.com/content/CVPR2024/papers/Yan_AHIVE_Anatomy-aware_Hierarchical_Vision_Encoding_for_Interactive_Radiology_Report_Retrieval_CVPR_2024_paper.pdf
https://arxiv.org/abs/2406.04449

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The paper benchmarks the newly released CheXpert Plus dataset for the X-ray image based medical generation, could the research describe CheXpert Plus dataset build process in detail?
soundness: 2
strengths: The paper is written well and easy to unserstand. The structure of the paper is commendabe, and each section is logically organized. The paper finds the weaknesses of the benchmark dataset in MRG, and uses the latest dataset to conduct a comprehensive benchmark test. It presents Mamba based auto regressive pre-training for improving performance and reducing computing costs.
summary: The paper conducts a comprehensive benchmark for the newly released CheXpert Plus dataset, named CXPMRG-Bench. Then, it presents a pre-trained large model, termed MambaXray-VL, which includes three stages: self-supervised auto-regressive generation for Mamba Vision encoder pre-training, contrastive learning and supervised fine-tuning. Through this method, the experiment is extended into IU X-ray and MIMIC-CXR datasets, testing the performance and capabilities of MambaXray-VL in generating X-ray reports.
weaknesses: Despite the paper's clarity, several imprecise arguments and overstatements necessitate revision and clarification:
* Compared to other algorithms, the advantages of the innovative MambaXray-VL design in medical report generation will be expressed better if there is a visual representation. The model can be segmented to show the multi-stage visual training process, such as data preprocessing, feature extraction, model training, and validation. Alternatively, the performance comparison of MambaXray-VL model with other models can be shown in the form of bar charts or line charts (such as accuracy, recall rate, F1-score, etc.) to highlight the advantages and effectiveness of this model.
* CheXpert Plus dataset can solve the disadvantage of joint training in limited dataset, and also increase efficiency. Therefore, I hope the article can explain in more detail the reasons for choosing multi-stage pre-training rather than joint training, and give more credible and specific advantages in multi-stage pre-training.
* Different algorithms have different degrees of operation on lable truncation, or do not use truncation, so when comparing the performance of the algorithm in this paper with other algorithms, does the uniformity of truncation be considered? Did the experiment compare truncation with no truncation on its impact on model training?

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: Could the authors elaborate on any inherent limitations of the CheXpert Plus dataset and discuss how these might impact the benchmark results? For example, are there any biases or data distribution issues that could influence model performance?

Could the authors provide more detailed results or ablations for each pre-training stage (Stage 1 and Stage 2)? How does each stage individually contribute to the final performance of MambaXray-VL?

Could the authors provide more insights into the selection process for hyperparameter configurations? Were there specific tuning processes or criteria used to arrive at the final values?

How does MambaXray-VL compare with other state-of-the-art (SOTA) models in terms of computational efficiency and accuracy? Are there specific metrics or settings where it particularly outperforms or underperforms?

The paper emphasizes the 3D modeling approach of MambaXray-VL, but how does it compare with traditional 2D models for similar tasks? Are there specific scenarios where 3D models outperform 2D, or vice versa?

The authors mention plans to enhance the model with knowledge graphs and domain-specific LLMs. Could they share more details on how these additions might be implemented or what challenges they foresee in integrating such components?
soundness: 2
strengths: The work offers a novel benchmark, CXPMRG-Bench, on the newly released CheXpert Plus dataset, representing a significant addition to the resources for X-ray-based MRG. This benchmark fills an existing gap by providing a comprehensive, standardized evaluation of MRG models, which previously faced challenges due to the lack of available comparative methods.

The introduction of the MambaXray-VL model with a multi-stage pre-training approach is innovative. By combining self-supervised autoregressive generation, X-ray-report contrastive learning, and supervised fine-tuning, the paper proposes a unique training pipeline that is highly tailored to medical image processing requirements.

The significance of this paper lies in its potential to standardize evaluations in the field of MRG, a domain where objective and comparative assessments have been historically challenging due to limited datasets and benchmark standards.

By addressing the limitations in computational costs and training stages of previous models, the proposed MambaXray-VL model could streamline the development of high-performance MRG systems that are more efficient and accessible, thus lowering the barrier for implementing these solutions in clinical settings.

The work’s impact extends to future research directions, as it suggests integrating structured medical knowledge and specialized LLMs for further improvements, signaling potential advancements in both performance and interpretability for medical applications.
summary: The paper focuses on advancing the field of automated medical report generation (MRG) from X-ray images, a crucial task that can reduce diagnostic workload and improve patient care. This work addresses limitations in existing MRG datasets and models by introducing a comprehensive benchmark using the CheXpert Plus dataset. This benchmark, called CXPMRG-Bench, evaluates 19 X-ray report generation algorithms, 14 large language models (LLMs), and 2 vision-language models (VLMs) to establish a comparative framework for MRG algorithms.

The paper also introduces the MambaXray-VL model, which utilizes a novel multi-stage pre-training approach with self-supervised autoregressive generation, contrastive learning, and supervised fine-tuning, leading to significant improvements in encoding and aligning X-ray image features with text data. Extensive experiments demonstrate that MambaXray-VL outperforms existing models across multiple metrics, setting a new benchmark for MRG performance on CheXpert Plus, IU X-ray, and MIMIC-CXR datasets. The paper concludes with plans to enhance model performance by integrating knowledge graphs and expanding pre-training on medical-specific data.
weaknesses: The paper could provide a more balanced perspective by discussing the limitations of the CheXpert Plus dataset and how they might affect benchmark results.

The paper does not present the separate performance results of Stage 1 (autoregressive learning) and Stage 2 (contrastive learning), nor does it compare these with other models in terms of regression/comparative learning effectiveness.

There is a lack of detailed information regarding the selection of hyperparameter configurations used in training.

Although the paper includes benchmark comparisons, it would benefit from a deeper analysis of MambaXray-VL’s computational efficiency and accuracy compared to state-of-the-art (SOTA) models.

The paper focuses on 3D modeling, but a comparative analysis with existing 2D models for similar tasks could better highlight the specific strengths and limitations of the 3D approach.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: Please refer to Weaknesses
soundness: 2
strengths: 1. This paper provides a comprehensive benchmark for the newly released CheXpert Plus dataset, covering 19 medical report generation algorithms, which is worth encouraging.
2. The authors support their claims with both qualitative and quantitative results, adding credibility to their findings.
3. The paper is well-written, with a clear and logical structure, making it accessible and easy to understand.
summary: This paper examines X-ray image-based medical report generation (MRG) by benchmarking existing MRG and large language models on the CheXpert Plus dataset, with the aim of supporting future research. Besides, the authors also propose a multi-stage pre-training strategy that integrates Mamba-based self-supervised autoregressive generation, X-ray-report contrastive learning, and supervised fine-tuning. The framework is evaluated on three public datasets: CheXpert Plus, MIMIC-CXR, and IU-Xray. However, the study primarily compiles existing models without introducing major innovations.
weaknesses: 1. Limited contribution: Benchmarking existing methods on a new dataset is not sufficient to justify publication in ICLR. Additionally, the pre-training-based framework proposed for report generation is not novel, as similar approaches have already been explored in works like CXR-CLIP and PTUnifier. Moreover, the effectiveness of the introduced Mamba network is not adequately validated (see point 4 for details).
2. Unfair comparison: The comparisons in Tables 1 and 3 are potentially biased. The proposed framework uses significantly more training data than the baselines, including 1.27 million images in the first stage and three datasets for pre-training in the second stage, whereas most of the baseline methods only train on their own datasets. This raises uncertainty about whether the performance gains are due to the proposed model or the use of additional data.
3. Lack of comparison with VL pre-training-based methods: Although the authors employ a pre-training framework, they do not compare their approach with other existing vision-language pre-trained methods for report generation, such as CXR-CLIP and PTUnifier.
4. Insufficient validation of Mamba network: The effectiveness of the Mamba network is not convincingly demonstrated in Table 4. Simply comparing rows 01 and 09 does not validate Mamba's effectiveness, as row 09 differs from row 01 not only in network structure but also in the use of additional data and pre-training strategies. A fair comparison of Transformer, CNN, and Mamba under identical conditions is needed.
5. Inconsistent performance of ARG Module: In Table 4, the proposed ARG outperforms existing MAE on MIMIC-CXR but shows a performance drop on the CheXpert Plus dataset, raising questions about the consistency and effectiveness of this module. 
6. Table 3 lacks a comparison of clinical efficacy metrics, which would provide valuable insights into the real-world applicability of the models.
7. In the third stage, the authors only use "image and findings" and omit "impressions," which is an essential part of medical reports. The rationale behind this decision is unclear.