LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation
Authors: Suhyeon Lee
Won Jun Kim
Jinho Chang
Jong Chul Ye
Venue: ICLR 2024 poster (2024)
Keywords: large language model
multimodal
medical imaging
chest X-ray
bidirectional
instruction-tuning
vision-question answering
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We present a state-of-the-art multimodal LLM for chest X-ray understanding and generation, developed using a method that builds upon the transformer+VQGAN architecture and adapts it for instruction-finetuning of an LLM pretrained only on text.

Abstract: Following the impressive development of LLMs, vision-language alignment in LLMs is actively being researched to enable multimodal reasoning and visual input/output. This direction of research is particularly relevant to medical imaging because accurate medical image analysis and generation consist of a combination of reasoning based on visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge between image processing (encoding or generating) networks and LLMs; but presumably, in order to achieve maximum reasoning potential of LLMs on visual information as well, visual and language features should be allowed to interact more freely. This is especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language-based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, we build upon this approach and develop a method for instruction-tuning an LLM pre-trained only on text to gain vision-language capabilities for medical images. Specifically, we leverage a pretrained LLM’s existing question-answering and instruction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to a given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our LLM-CXR trained in this approach shows better image-text alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks.

Published as a conference paper at ICLR 2024
LLM-CXR: INSTRUCTION-FINETNEED LLM FOR CXR IMAGE UNDERSTANDING AND GENERATION
Suhyeon Lee", Won Jun Kim", Jinho Chang & Jong Chul Ye Korea Advanced Institute of Science & Technology, {suhyeon. lee, wonjun, jinhojak515, jong. Long-yej@kann weaayyee@kkks.t... ac.
ABSTRACT
Following the impressive development of LLMs, vision-lanuuage alignment in LLMs is actively being researched to enable multimodal reasoning and visual input/output. This direction of research is particularly relevant to medical imaging because accurate medical image analysis and generation consist of reasoning based on a combination of visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge hetween image processing (encod- ing or generating) networks and LLMs; but presumably, in order achieve maximum reasoning potential of LLMs on visual information as well as language, image and text features should be allowed to interact more freely. This especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language- based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, instruction-tuninn tunionnttunnngggg an LLM pre-trained only on text to gain vision- laiion-annuuge capabilities for medical images. Specifically, we leverage pretrained LLM's existing question- answering and isstuction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our model, alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks.
build upon this approach and develop method for
trained in this approach shows better image-text
INTRODUCTION
The last few years have seen remarkable development in the field of Large language models (LLMs). L.MMs are considered a different class of Al models because of their ability to flexibly understand/generate natural language and perform language-
reasoning, allowing them to generalize to a variety of given tasks without the need to be explicitly trained for them. As a next step, methods to enable the input of visual information alongside language in LLMS (OpenAl, 2023 Liu et al. 2023 Alayrac et al.. 2022 Liet al. 2023) as well as methods that output images from LLMs Koh et al. 2023a b) are being actively developed. These models have great potential to be particularly useful in the medical domain, as working with medical images such chest X-rays (CXRs) requires the ability to understand context, perform reasoning, and communicate conclusions in both image and text forms. The first generation of medical multimodal LLMs has begun emerge recently (Moor al. 2023 Thawkar al. 2023
Xu et al. 2023a
The main challenge in developing these models is achieving alignment between the pretrained language features of LLMS and the newly introduced image features without catastrophic forgetting of their language abilities. This is a more difficult challenge in the domain of medical images compared to natural images because the model needs distinguish subtle differences in images or even parts of images
"These authors contributed equally this work.
In order to comply with the MIMIC-CXR data usage license (Johnson et al. 2019a). all CXR images
presented in the Figures 3 4 [ are replaced with similar CXR's from the Indiana University chest X-ray
dataset (Demner-Fushman al. 2016); and the presented MIMIC text reports paraphrased.

---

Figure 1: (a) Example of previous work that indirectly implements multimodal bidirectional L.M by connecting pretrained image encoder or image generation model to a pretrained LLM with a mapping layer. (b) Example of previous work that implements multimodal bidirectional non-LLM transformer with Published as a conference paper at ICLR 2024
(a)
Pre trained VOGAN
edaptor Inage gonerative
fot itattar
#
LLM
hutoccionn pooge
lpper D
Asrparac
adaptor module
Instruction Fine-tuning
Image captioning
Give me brief
Loft fung pneumanto,
Visual question answering
No, there pinural effucsion.
chere
pleural effisiin??
LLM-CXR
MRT medical imaging
Natural lanpuage comprehunsion
technique that
Transformer
What MP??
decodar
Image generation
Generate CXR image for the following
diognasir, left lung
VQ-GAN trained from scratch (i.e. without learned language features). (c) To enable direct multimodal feature interaction LLMs pre- pre-trained with text, our method implements (b) through LLM--fccifiic instruction fine-tuning scheme.
(e.g., pneumonia vs. pulmonary edema on CXR) and then provide accurate text descriptions image generations. Natural images tend to be more diverse than medical images, and each one can be described by a broad range of statements. Medical images, on the other hand, require highly specific yet comprehensive descriptions making the space for correct answers much smaller and more complex. This means that a medical multimodal LLM requires more intimate mapping between textual and visual features.
Currently, the most popular approach to map visual features to and from an I.I is to train an 'adapter net- work' to act as a mapping layer that translates the output of pretrained image encoder network to form that can be understood by an LLM (Alayrac et al. 2022 Li al.. 2023 Zhu et al.) (023) or to connect the output of an LLM to an saangeggnerrating network to output images (Koh et al. 2023a b). In these approaches,
LLMs are frozen to prevent forgetting their language and reasoning capabilities. These multimodal LLMs have demonstrated impressive capabilities in vision-language tasks such as image captioning, zero-shot clas- sification, visual question and answering (VQA), image generation, and image retrieval. However, with this common approach of using adapter networks, vision-language alignment may be limited as the adapter net- work serves an information bottleneck that can hinder the interplay between visual and language features.
To better bridge the gap between image and text, we take inspiration from the field vision-language pertaining (VLP) with non-LLM transformers, where there has already been lot of work on treating images and text in the same token embedding space. Most prominent is the approach that tokenizes images using VQ-GAN (Esser al. 2021) VAE (Van Den Oord al. 2017)) and generates sequences of both text tokens and image tokens using an autoregressive transformer decoder (Zhang et al. 2021] Lu et al. 2022 Wang al. 2022a Lee et al. 2023). These previous works fit well with our view that for better image-text alignment, models should be able to process images and text equally without separate adapter bottleneck. Moreover, the fine details of the CXR images such as texture are important in medical diagnosis, making the tokens from local image features from VQ encodings preferable targets for alignment than the global descriptions. Hence, in this work, we take advantage of the widely used architectural component VQ-GAN seamlessly integrate the image-text token space without requiring any structural modifications to the underlying base LLM.
Building this foundation, propose method for achieving better image- alignment in LLMs for CXR image understanding and generation by leveraging an LLM's built-in instruction followwng abilities. Specifically, seek to teach the model visual information by giving it diverse instructions surrunndin
2

---

Published as a conference paper at ICLR 2024
CXR image analysis and generation and then using the outputs to finetune the LLM. As such, one of our main contributions is the development of this instruction instruction-fimetuiing method that has been tailored to be suitable for an treaady-rainedd LLM to expand its capabilities to input and output images (tokenized by VQ- GAN) without modification of model structure or objectives. An important distinction from previous work such as (Zhang et al. 2021 [Lu et al. [2022] Wang al. [022a] Lee al. (2023) is that while non-LLM VLP transforners were trained from scratch without a previous understanding of language- meaning there was no concem of forgetting nor the opportunity to take advantage of language understanding- our contribution a method that takes pretrained LLM and adds bidirectional multimodal capabilities by simple instruction- finetuning process designed for LLMs (further detailed in Section 2 Through this novel approach, we produce finetuned LLM proficient in bidirectional, multimodal tasks capable of CXR-to-repon generation, report-to-CXR generation, and CXR-related VQA. We show that this model has state-of-the-ari image-text understanding and generative capabilities by demonstrating that it outperforms previously developed models in each of these tasks even though the other models were specifically designed for only a subset of the tasks.
2 LLM-CXR
2.1 CLINICAL SETEMMATTOO-PREERVVIIE CXR TOKENIZATION
For the tokenization process of the images, we used VQ-GAN (Esse et al. 2021 a widely used approach (Wang et al. 2022a Lu et al. 2022 Zhang et al. 2021 Lee et al. 2023) for tokenizing images in multimodal generation models with transformers. More specifically, we utilize the quantized latent space of the VQ-GAN model trained on the image domain. VQ-GAN consists of frozee encoder :(:):RCXHXI) CxHWW - (1,2,..., King) decoder D(-): : CERKonga xns that contains Kimg codes. With this VQ-GAN, it is possible to obtain tokenized image 2E [mng) of length dz. As shown in Figure [(c), this allows us to freely convert images into tokens and then back to images similar to an autoencoder (Kramer 1991). Furthermore, the tokenized images contain more localized information in each token, making them suitable for medical diagnosis purposes where localized and texture information is also critical. The VQ-GAN remains frozen during the training of the LLM. Its sole purpose is to encode and decode images, facilitating their input to and output from the LLM similar tokenizers for text. Consequently, the LLM operates with images in the form of these image tokens, both for input and output processes.
Kimg) jdh - C and codebook
However, the original VQ-GAN's reconstruction objective during training only consists of L1 loss and LPIPS loss (Zhang et al. 2018). which causes loss of clinically important information such as characteristics of microscopie lesions in the information bottleneck formed by the quantization process. Therefore, to minimize the loss of such important but subtle information in CXRs, we present another contribution: an additional dime-simallll feature L.2 reconstruction loss extracted from the CXR encoder model of the TorchXRay Vision (Cohen et al. 2022) library that is used when training the CXR tokenization leads to
VQ-GAN for image tokenization. This clinical anformationn-peserrvin performance improvement in both the report-to- CX- task and the CXR-to-report task.
2.2 EXPANDING LLM'S ' TOKEN EMBEDDING SPACE
Non-LLM transformers are trained with both images and text from the beginning. Training multimodal LLMs has a subtle but crucial difference: LLMs are already trained on text, and this text- training is too expensive to be done again during multimodal training. Thus, the goal is to confer visual capabilities using a fine-tuning process to an LLM previously trained only on the text in a way that the newly introduced visual information is in line with the pre-existing language information. To place image tokens in the same embedding space as text tokens without losing the language abilities of the LLM, we treated the process of adding image tokens to the model for fine-tuning the same as the technique of increasing the special token in the vocabulary (Le., token type) in language model finetuning for the image retrieval and generation (Koh et al. 2023b a). Concretely, if the LLM's original embedding table was IR Krert xd, in which the embedding dimension is de, then the embedding table is expanded to R (K K Kmng) Xm The existing elements are retained and used as initial values for fine-tuning, while the newly expanded parts are initialized randomly. The entire embedding table is trainable during the fine-tuning process.
3

---

Published as a conference paper at ICLR 2024
2.3 DATA AUGMENTATION WITH SYNTHETIC VQA
Text reports for CXRs contain a comprehensive, detailed description of the CXR image in question. While these image-text pairs can be used as-is to achieve vision-language alignment, this training can be enhanced by taking the text report and generating visual questions and answers (VQAs) that can be asked about CXR images. This is not only a way to further enhance vision-language alignment but also an important way to ensure that the natural language interaction capability is maintained in the model. We show that utilizing this multi-task instruction-tuning approach improves performance on all fronts (VQA, text-based image generation, and image-based text generation).
"What is the most likey of the subtle opacity the right lung base?"
Early preumonia the most lively touse of the subble opecity at the right king base...
REPORT:
opacity the right ling base, most likely due to early anoumonis, There pleural effusion mothora, The mediastinal sthouette bloterra hila apocar narmal.."
any effusion??
No, there is no pleural effuston present in the chest X0 ray.
LlaMaZ
"Is there anoumathorax""
No, there no pneumcthoraa present the chest X-ray.
"What is the apacarance of the indiastinal slit ouette and hla?"
mediastinal sihoouttt and hila appcar normal"
Figure 2: Examples of VQA generated from a CXR text report. f
We use LlaMa 2 (L.l 2-1ma-1133
(Touvron et al. 2023) to generate questions and answers about a chest X-ray as shown in Figure 2 Specifically, about 200,000 CXRs that were labeled the MIMIC-CXR-IPG dataset to be positive for one or more lesions were selected, and LlaMa 2 was prompted to generate a few questions for each CXR. The prompt used to generate these VQAs and more examples of generated VQAs are included in Appendix [B
2.4 MMAGE-TEXT BIDIRECTIONAL INSTRUCTION FINE-TUNING
Taking inspiration from previous methods for non-LLM transformers' multimodal generative meth- ods (Wang et al. 2022a
al. 2022 Zhang et al. 2021 Lee et al. 2023). we adopt and transform this technique into an instruction finetuning (Wang et al. 2022b Wei et al. (021]) scheme suitable for LLMs pretrained on a large text corpus. Since this process is simply fine-tuning process for LLM, no structural or objective changes are made to LLM other than the expansion of the token embedding table, and no additional networks are required. The template for custruction-finetuiitt uses the template used by the Alpaca family (Taori et al. 2023 Databricks] 2023). Appendix Cls the template of the prompt for instruction fine-tuning from the Alpaca family which consists of Instruction, Input, Response sections.
During the fine-tuning process, the LLM is optimized according the objective function that outputs a response based on the instruction-ings input pairs in an autoregressive manner. Note that this is an instruction- naiigg scheme, inheriting but distinct from the training of non-LLM transformer multimodal generation methodologies. The advantage of this scheme will be covered in more detail later in Section 2.4.1
The tasks used for fine-tuning are categorized into four main types: 1) natural language instruction- following (NL-IF), 2) report-to-CXR generation, 3) CXR-to-report generation, and 4) CXR-based vision question answering (CXR-VQA). These are the four primary task types categorized based on input and output modalities; but for the model, they form a rich training environment with a wide spectrum of tasks that are distinguished by Instruction. NL-IF and CXR-VQA training examples provide multi-dimensional tasks so that the model can learn intricately aligned visual and textual features and generalization of tasks instructed in natural language. The report-to-CX. and EXR-to-report generation tasks are used in high volume during training and are important for vision-language alignment, but it must be noted that they are merely two tasks among many. Through the use of Instructions to specify tasks, we add several unseen multimodal task capabilities to the base LLM without overwriting existing language- based interactive capabilitics. It also enables simpler yet more general user interaction compared to the cxisting non-LLM multimodal bidirectional generation models discussed above, they only be queried for certain tasks using predefined tokens, while LL-M-bssed models enable queries based on natural language instructions and thus the possibility of generalizing zero-shot tasks (Wei et al. 2021 Wang et al. 20226).
NL-IF task. We initialize the base LLM with weights of a pretrained instruction-followw LLM
(Databricks 2023). To minimize the risk of forgetting of language proficiency during the fine-tuning
4

---

Published as a conference paper at ICLR 2024
process, concumently engage in cnntructton-followwin tuning using the same NL-IF dataset used to instruction-tune this base LLM.
This is a task that aims to generate CXR images that match the Inpui radiology report as Response. The Instructions for this task are randomly sampled from 10 versions similar the instructions in the example below, The LLM directly outputs image tokens in the same way as text tokens due to the utilization of the expanded token space encompassing both text and image tokens. Therefore, CXR image generation does not require an additional network or text-to-ing to-image generative model (e.g. stable diffusion) as seen in Wu et al. 2023b); Koh et al. (2023a). Below is an example instruction/input pair used to instruct the LLM to generate CXR image.
Report-to-CXR generation.
#4# Instruction: Generate a caest X-ray image that corresponds to the entered free-text radiology reports for the chest X-zay image. Input: Bilateral, diffuse, confluent pulmonary opacities. Disferential diagnoses include severe pul edema ARDS or hemorrhage.
#4# Responss: <vQ032 VQ015 vQ124 VQ054 VQ032>
CXR-to-report generation. In this task, tokenized CXR images are the Inpul. Note that in our model, the image is not processed through a separate network trained with paired vssion-language data as in Li et al. (2023); Liu et al. (2023); the tokenized image is directly into the LLM, and the LLM itself learns top of its language capabilities. It can then be instructed to output corresponding radiology report to given image as Response. Instructions are also randomly sampled from versions similar to the example below. The following snippet is an example instruction for the CXR-to-report task.
vIsual information
Generate radiology reports for the entered CXR image. Input: <V0071 VQ057 402 ... VQ122 VQ002> #*# Response: No acute cardiopulmonary process.
### Instruction ion:
CXR- VQA task. In this task, questions about an image are given as Instructions, and the model generates an appropriate Response. Questions are about the CXR images given as Input. This not only trains the model to gain VQA capabilities but also improves the performance in the other vision-language tasks as well (i.e., enhancement of vision- languaee aligmment). Below is example instruction for the CXR-VE VQA task.
#*# Instruction: What is the size of the pleural effusions?
Input: <VQ121 VQ720 V0002 V0005 V0428> ### Response: The bilateral pleural effusions are moderate to large.
2.4.1 TRAINING OBJECTIVE
The training objective is to generate the entire target paragraph which consists of Instruction, Input, and Response in an autoregressive manner. However, similar to general GPT pre- training (Radford et al., 2018 2019: Brown et al. 2020). the loss is only applied to the tokens generated after the response key (i.e. ### Response: )). following an instmatiion-tuiing scheme (Taori et al. 2023 Databricks 2023
Specifically, for tokenized training paragraph [21, -3i--, .................... where æ denotes Instruction and Input sections and y denotes Response section, the training loss is given by:
Linstruct logp(y/())
Σ Dogp(y/bi-1191ee
Fn, Ing 21).
(1)
Note that this objective is different from the one used to train non-LLM transformers such as in Lee al. (2023) which uses Ijooint =- - logp( (z. We hypothesize that instruction-tuning through the of conditional loss mitigates overfitting to the fine-tuning dataset, particularly when working with limited data, thus promoting a better learning environment that encourages the model to expand its understanding from its pre-existing features rather than memorizing new features.
2.4.2 TWO-STAGE TINE-TUNING
Similar to previous works Chambon et al. (002 Xu et al. (2023b), we place our focus on frontal view (i.e., AP and PA) images and the Impression sections of corresponding reports they the most relevant to making diagnoses and more amenable to straightforward comparison the results. Additionally, we minimize references prior studies within reports (as we will use just one image at time during inference
5

---

Published as a conference paper at ICLR 2024
and it would not make sense to have reports that refer to prior studies) by prining the MIMIC-CXR-JPG dataset to only include the first study for each subject.
Drawing inspiration from mulli-stage training techniques commonly used in recent LLM fine-tuning methods (Zhu et al. 2023), we follow similar two-stage training approach. the first stage, unfiltered high-volume data (i.e., all CXR image-report pairs and findings/impression sections in the MIMIC- dataset) is used to train the model on the entire distribution of new image tokens and the general relationship between image and text tokens. In the second stage, model performance is further enhanced using the pruned dataset (i.e., using only frontal view images, the first study for each patient, and impression sections of text reports) that provides stronger signal for -sioonlanguage alignment.
Implementation details are provided in the Appendix A
EXPERIMENTS
We compare the performance LLM-CXR against similar contemporary models across all tasks performed by LLM-CXR. For CXR-to-repoy generation, we compare results with UniXGen (Lee) al. 2023), XrayGPT (Thawkar et al.) 2023), RadFM Wu al. (2023a), IFC[Debrou et al. (2022) and R2Gen Chen et al. (2020): for CXR- VQA, with XrayGPT (Thawkar al. 2023), RadFM al. (2023a). and ELIXR (Xu ct al. 2023al: and for text-to-CXR generation, with UniXGen (Lee et al. (023) and
RoentGen (Chambon et al. 2002).. Note that only LLM-CXR is capable of performing all threc tasks. For
further details regarding the ablation experiments on the design our method, please refer to Appendix
3.1 CXR-TO-REPORT GENERATION TASK
We use each model to generate text radiology reports for CXR images in the MIMIC-CXR-JPG dataset using LLM-CXR (Figure B). UniXGen (Lee et al. 2023), XrayGPT (Thawkar et al. 2023). RadFM Wu al. 2023a), Delbrouck et al. (2022), and R2Gen Chen et al. (2020). In order to quantify how well the generated reports reflect the clinically significant radiologic information in the CXR images, use the CheXpert-labed labeler [Irvin et al. 2019), which is rule-based natural language processing tool that reads a text report for CXR and extracts whether the report mentions the presence or absence of significant radiologic findings (e.g., edema, pleural effusion, etc.), and compare the extracted labels with that of ground-truth reports.
Input CXR
Generated Ground-truth
"1. There moderate large left pleural effusion. Right pleural effusion is similar prior.
"Left pleural effusion with overlying atelectasis. opacity may due combination pleural effusion and stelectasis... Mild pulmonary
"Cardiac silhouette enlarged but there vascular congestion. Opacification in retrocardiac region consistent with volume loss in left lower difficult superimposed pneumonia...
"No evidence cardiopulmonary disease."
congestion.
"Bilateral pleural effusion."
"Mild pulmmnnar edems, pulmonary vascular congestion and small to moderate left pleural effusion."
"1. Mild cardiomegaly. 2. Left lobe consolidation could stelectasis, pneumonia, or aspiration'.
"No acuite intrathoracik process"
Figure 3: Examples of text report generation for given CXR image with LLM-CXR, While the generated reports use different wording from the ground-truth reports, LLM-CXR is able to generate reports that capture the gist the contents of the CXR, demonstrating alignment vision-language features within the model. In addition, similar to real CXR reports, LLM-CXR often proposes valid causes for certain findings (e.g., suggesting aspiration as the cause of consolidation), demonstrating language-bass based reasoning ability characteristic of L.LMs.

---

Published as a conference paper at ICLR 2024
Is lung consolication or pneumonia this image?
Yes, worrisom for pnnumonia and/or aspiration.
What location the lung consolication or pneumonia, present?
The lung consalidation or preummnii is located in the right lower lobe.
Does the patient have lung opacity?
Yes, the patient has moderate pulmmnnay edema.
What abnormalities are seen within lungs?
Vascular congestion and associated patchy airspace
disease suggesting
niggestingg.ae aspiration or pneummnia..
Figure 4: Examples of VQA with LLM-CXR, LLM-CXR understands questions given in natural language and is able answer with relevant findings.
To quantify the similarity between generated reports and ground-truth reports, we measured AUROC/F1 (Table [ and Jaccard similarity index (Table 2 between labels of clinical significance (e.g. pneumonia. cardiomegaly) extracted from both generated and ground-truth reports using the CheXpert labeler (Irvin al. 2019). Note that LLM-CXR and XrayGPT utilize input images with a resolution of 256 and 224x224 pixels respectively. In contrast, the original UniXGen model (UniXGen-512) uses images at a higher resolution of 512x512 pixels. Thus for comparison, we conduct experiments for UniXGen twice once with its native 512px resolution and once again with 256px resolution images (used for LLM-CXR) upsampled to 512px.
In terms of both metrics, our model demonstrates superior performance when compared to competitors operating at the same resolution. Notably, there is no substantial performance gap observed when compared to UniXGen and RadFM, which operate with images at higher resolution of 512x512 512 pixels. LLM-CXR (3B parameters) exhibits stronger accuracy than XrayGPT (based on the Vicuna-7B model). For completeness, we also measure the natural language generation performance metrics: BLEU, ROUGE-L, METEOR, CIDEr; however, because these metrics are less informative in the setting of medical image understanding where precise language is employed, we present them in the Appendix D (Table
Table 1: CXR-to-report generation AUROC and F1
A
Atel.
Cusl.
Protx. Edema
Eff.
Pna.
Cmmel.
Les.
Frac. Opac. ECm. NoF. P.O.
Dew. Miero
Macro
Weighted
RadFM
0.587
0.498
0.633
0.657
0,504
0.611
0.516
0.498
0.514
0,502
0.666
0,499
0.638
0.556
0.596
UniXGen-512
0,570
0.533
0.519
0.615
0,682
0.526
0.645
0.501
0.498
0.555
0.510
0.676
0.498
0,740
0.668
0.576
0.628
IFCC
0.479
0.508
0.486
0.504
0.496
0.486
0.545
0,518
0.498
0.497
0.463
0.497
0.499
0.494
0.543
0.497
0.498
R2Gen
0,501
0.485
0.504
0,500
0.503
0.502
0.505
0,510
0,500
0.501
0.511
0.494
0.500
0.498
0,542
0.501
0.500
UniXGen-255
0.518
0,542
0,513
0,510
0.513
0.499
0.519
0.511
0.575
0.528
0.540
XPT
0.551
0,506
0,511
0.590
0.595
0.519
0.570
0.511
0,499
0.553
0.539
0.592
0.490
0.646
0.617
0.548
0.577
LLM-CXR
0,558
0.517 0.496
0.619
0.641
0.509 0.577
0.506
0.494
0.537
0.505
0.677
0.498
0.640
0.628
0.555
0,597
F1
Atel.
Cnsl.
Pmnts.
Edema
E.I.
Prn.
Cmgl.
Les.
Frac. Opac. ECm.
Nol:
P.O.
Dev.
Miiroo
Macro
Weighted
RadFM
0,325
0,024
0.018
0,404
0.494
0.034
0,387
0.065
0.000
0.177
0.026
0,524
0,000
0.38]
0.370
0.204
0.341
UniXGen 512
0.298
0.116 0.064
0.374
0.530
0.121
0.23
0.014
0,000
0.317
0.049
0.532
0.000
0.586
0.413
0.245
0.398
IPCC
10.19
0,083
0.020
0,205
0,312
0.006
0.270
0.068
0.000
0.323
0,042
0.200
0.000
0,292
0.220
0.141
0.225
R2Gen
0.168
0.073
0.129
0.034
0.043
0.000
0.051
0.201
0.113
UniXGen-256
0.146
0.072
0.083
0.226
0.215
0.072
0.176
0,055
0.000
0.282
0.047
0.411
0.092
0.367
0.262
0.160
0.243
XPT
0.279
0.065
0.049
0.334
0.404
0.110
0.347
0.058
0.016
0.352
0.076
0.371
0.000
0,470
0.326
0.200
0.330
LLM-CXR
0.272 0.081
0.013
0.382
0,464
0.084
0.327
0.036
0.000
0.278
0.035
0.535
0,000
0.453
0.360
0.211
0,350
Table CXR-to -e-oot generation Jaccard similarity index!
1
Micro
Macro
Weighted
No mention
Possible
Negative
Positive
RadH-M
0.4894
0.2367
0.5568
0.6525
0.0123
0.0552
0,2270
UniXGen-512
0.4716
0.2520
0.5464
0,6323
0.0423
0.0733
0,2602
IRCC
0.4141
0.1935
0.4877
0,5835
0.0027
0.0643
0.1237
R2Gen
0.4038
0,1988
0.4842
0.5797
0.0337
0.0701
0.1116
UniXGenn2505 256
0.5993
0,2438
0.6236
0.7480
0.0317
0.04499
0.1501
XIa
0.4107
0.2206
0.4934
0.5773
0.0413
0.0690
0.1948
LLM-CXR
0.6092
0.2699
0.6420
0.7585
0.0483
0.0530
0.2198
Bold in report-to-CXR task result tables indicates the best performance among models at the same resolution with M-CX (256x256). If the highest performance achieved the 512, this is separately underlined.
7

---

Published as a conference paper at ICLR 2024
3.2 CXR- VQA TASK
We adopt the VQA performance assessment framework of ELIXR (Xu et al. 2023b) which, in summary, asks about the presence, location, and severity of certain lesions or findings for each CXR image and marks each answer as 0 (the answer is incomect, internally inconsistent, or irrelevant). (correct), or 0.5 (partially correct or not quite correct but reasonable explanation for the CXR image). First, as done in ELIXR, we randomly select cight cases that are labeled in MIMIC- with the following diagnoses: 'No Finding'. 'Pneumothorax', 'Pleural Effusion', 'Edema', 'Consolidation' OR 'Pneumonia" (considered a single unified class), and 'Lung Lesion'. We use the same questions and grading rubric used in Xu et al.) (2023b). Note that the reported scores for ELIXR are taken from the paper Xu et al. (2023b) as their model not publicly available, but the scores for XrayGPT and our model (LLM-CXR) were measured by the authors using open-sourced checkpoints.
Table 3: Accuarcy of the CXR-VQA
VQA task by topic and label diagnosis.
ELIXR
Xuu et al.
2023a does
not report its VQA accuracy by label diagnosis.
Accuracy t
All
Presence
Location
Size, severity, type
ELIXR
54.8%
64.5%
41.0%
25.0%
XrayGPT
25.2%
27.4%
21.9%
20.3%
RadFM
32.7%
34.5%
31.3%
20.8%
LLM-CXR
44.8%
41.3%
50.0%
62.5%
Accuracy
All
Cnsl/Pna.
Edema
Lsn.
NoF.
Elf.
Pmtx.
XrayGPT
25.2%
25.0%
26.25%
17.2%
42.5%
20.0%
18.8%
RadFM
32.7%
34.4%
31.3%
40.6%
61.3%
26.3%
23.8
LLM-CXX
44.8%
39.1%
53.8%
50.0%
71.3%
53.8%
22.5%
An example of VQA performed by LLM-CXR is shown in Figure 4 Accuracies of VQA in comparison with other multimodal LLMs capable of CXR reading are shown in Table 3 ELIXR (Xu et (2023a) uses PaLM-2 as the base LLM and uses the framework of BLIP-2 (Li et al. 2023) (i.e. Q-fommer) for achieving vision-language alignment while XrayGPT Thawkar et al. (023)) uses a Vicuna-7B as the base LLM and uses the MedCLIP image encoder Wang et ( (2022c) and a linear mapping layer for vision-language alignment. Because the VQA task requires a general understanding of language, this task can only be done by the larger, LLM-based models. LLM-CXR holds promise even amongst bigger models.
3.3 REPORT-TO-CXL CENERATION TASK
Because our instruction ttinintttnngggn includes image generation tasks, LLM-CXR is also able to generate matching chest X-rays when given text report (Figure We measure the quality of generated images with FID (Table [6] in Appendix DD. and we measure vision-language alignment (i.e, how well the text used to guide image generation is reflected the generated image) by calculating the AUROC/FI (Table against the original CXR images in MIMIC-CXR-JPGs with a pretrained CXR lesion classifier network (Cohen al.] (022)) specifically, densenet res224 -a1ll. We compare our generated CXRs to RoentGen (Chambon et al. 2022), a stable diffusion- model that generates CXRs based on text descriptions, and UniXGen (Lee al. 2023). a bespoke non-LLM transforme--ee based model trained from scratch to generate CXR images and reports.
As shown in Figure 5 L.LM-XRR is able to reflect lesion characteristics, location, and severity in its generated CXR images. Quantitatively, FID indicates that LLMM-XR generates images closer to real CXR images than UniXGen or RoentGen. With regards to alignment with input text in the generated images, AUROC/F1 indicates that LLM-CXR generates images that are most aligned with input texts.
Table 4: CXR generation AUROC and Fl.
AUROC1
Atel.
Cusl.
Pintx.
Edema
E.f.
Pna.
Cmgl.
Les.
Frac.
Opae. ECm..
Micro
Maero Weighted
RoentGinn
0.7661 0.7535
0.8078
0,7084
0.8169 0.6054 0.7780 0,6283
0.7294
0.7061
0.7013
0.7055
UniXGen
0.7982
0,7509 0,6640
0.7876
0.7725
0.7065 0,7610 0,7200 0.7121 7L7867 0.7893
0.7435
0.7499
0,7518
LLM-CXR
0.8054
0.8263 0.7540 088111
0.8155
0.7722 0.7846 0.7852 0.7896 0,8311 0.8335
0.7907
0.7980
0.7991
F1
Atel
Cusl.
Puntx.
Edema
E.I.
Pua.
Cmngl.
Les.
Frac.
Opac.
ECm.
Micro
Macro
Weighted
RocntGen
0.8113
0.7286 0,7110 0.2954
0,7619 0.2501 0,7639 0,2677 0.6580 0,7781
0.7066
0.6578
0.6121
0.6298
UniXGien
0.8648 0.6913 0.4981
0,7378
0.7008 0.7213 0.7598
0.5606
0.6424 0.7794
0.7958
0.7164
0.7046
0.7082
LLM-CXRR
0,8777
0,8283 0,7024
0.8061
0,8183
0.7529 0,8372
0.7678
0.7753 0,8342
0.8274
0.8065
0.8022
0.8054
8

---

Published as a conference paper at ICLR 2024
[a] (b) Different locations
Generated CXRs
Input:
"No cardiopulmenayy process"
"Midd pulmonary edema?"
"Right pleursl offucton"
"Left pleural effusion"
"Bilateral affusion"
boema:""
Figure 5: CXR images generated with LLM-CXR using radiology reports input. (a) Normal CXRs. (b) Words such as "severe" and "mild" allow for the generation different severities lesions. (c) Specification of the location of lesions using words such as 'left', 'right", and 'bilateral'.
4 CONCLUSION
Multimodal LLMs have great potential to assist in the field diagnostic radiology as they can reason about visual information and express their findings in natural language images understandable by medical pro- fessionals. However, major challenge is achieving sufficient vision-language alignment in these pretrained LLMs. Most work on vision- langugge aligmment in LLMs has focused developing adapter networks to connect an image-processee processing network and a pretrained, frozen LLM. However, this approach has so far fallen short in achieving the level of vision-language alignment needed accurately describe medical im- ages and is prone to hallucinations despite the use of L.LMs with relatively high numbers of parameters. In method L.LMs that enables them
this work, we proposed a different approach, an cnttucctionfinetumne to understand and generate visual information, that shows more promise in achieving better-aligned vision and language features. We leveraged the uuguuage-understandim capability of LLMs to provide a complex training environment that induces the incorporation of visual features into its language features and shows that can lead to better visual understanding and generation with LLMs even with a much smaller model.
5 LIMITATIONS AND FUTURE WORK
There are a few limitations this study that could be improved upon. Most importantly, while our method shows better performance than other larger models, there is still much room to be improved in the alignment visual and language features. For example, generated CXR reports still contain false positives (i.e., they mention findings that not actually present) and miss diagnoses. This problem could be further mitigated in the future by strengthening the alignment of images and text reports within the model by employing other vision-language techniques, improving the quality/quantity of the training data, using larger LLMs. For instance, the radiology reports of the MIMIC dataset often refer to previous imaging studies which were unhelpful and act noise instead of signal in our framework; we anticipate that using each patient's is CXR scans longitudinally, i.e., using the timestamps of each study, can help improve the quality of generated results by properly incorporating this information into the training process.
Furthermore, in our framework, 256x 256 image is translated into 256 image tokens. As a consequence, the resulting sequence contains relatively long token sequences compared to text, resulting latency ranging from 30 to 60 seconds for image generation tasks and about 10 seconds for text generation tasks with the consumer GPU, Although our model presumably has faster inference times than larger models, it still cannot claim real- cal-time responsiveness. potential avenue for improvement is adopting techniques that enable dynamic tokenization of images (Jin al. (2023). as opposed to using fixed-length tokens. This approach could potentially alleviate the latency issues and pave the way for more responsive real-time applications.

---

Published as a conference paper at ICLR 2024
ETHICS
Two important ethical issues at the intersection between medicine and Al are safety and privacy.
Al models will probably play increasingly larger roles in healthcare systems. It is important that they are adopted in a way that improves patient safety and reliability of the systems already in place. LLMs have known issues with hallucinations and biases, which may be propagated when put into use without proper supervision. While communicative models like this one will potentially serve as a crucial interface between AT systems and human medical professionals so that such problems can be avoided, systems still need to be put in place so as to keep potential biases and hallucinations in check. There will also need to be continuous work to improve and scrutinize these models.
Furthermore, while our model is trained on the deidentified, publicly available MIMIC dataset, generative Al models such as ours raise concerns about privacy as institutions have the potential develop these models with private patient data. When these multimodal LLMs reach the proficiency to be used in real clinics and become immediately valuable, regulations and technological security measures must be put in place to prevent breaches of patient privacy.
We hope that our model will serve as a step forward in developing reliable Al systems for healthcare.
REPRODUCIBILITY
The pretrained models and datasets we use are all publicly available (Databricks' dolly-v2-3b, Imagenet- pretrained VQ-GAN, and MIMC-CXR-PGG,, We release all code and model checkpoints upon publication along with step-by-step guidance to reproduce the methods explained in Section 2 so that anyone can reproduce our results")
ACKNOWLEDGMENNS
This research was supported by the National Research Foundation of Field-oriented Technology Development Project for Customs Administration funded by the Korean government (the Ministry of Science & ICT and the Korea Customs Service) through the National Research Foundation (NRF) of Korea under Grant PRR2002M3I1A10907 Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry, and Energy, the Ministry of Health Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711137899, KMDF_PR RMDFFPPR-0200001 _0015); Culture, Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture. Sports and Tourism in 2023; and Institute of Information & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea goverInen(MMSIT, Ministry Science and ICT) (No. 2022-0-00984 Development of Artificial Intelligence Technology for Personalized Plug-and- ala- Explanation and Verification Explanation).
NRF2021M331AA109995
REFERENCES
Ccan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.067455 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are sho-shot leamnrr.. Advances in neural information processing systems, 33:1877-1901,, 2020.
Pierre Chambon, Christian Bluethgen, Bean-B Delbrouck, Rogier Van der Sluijs, Malgorzata Polacin, Juan Manuel Zambrano Chaves, Tanishg Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz, and
http:/giihubbccoo httpea///giichubb.mmm//y//22222
10

---

Published as a conference paper at ICLR 2024
Akshay Chaudhari. Roentgen: vision-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211.22737, 2022.
Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, November 2020.
Joseph Paul Cohen, Joseph D. Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Matteo Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, and Hadrien Bertrand. TorchXRay Vision: A library of chest X-ray datasets and models. In Medical Imaging with Deep
Learning, 2022. (ttp/ttts////tttta
com/ httmm//mmmmee//orrcca//a/ssss///
Databricks. Free dolly: Introducing the world's first truly open instruction-uuned Ilm. 2023.
Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis P. Langlotz. Improving the factual correctness of radiology report generation with semantic rewards, 2022.
Dina
Mare D. Kohli, Mare B. Rosenman, Sonya E. Shooshan, Laritza Rodriguez, Sameer Antani, George R. Thoma, and Clement McDonald. Preparing collection of radiology examinations for distribution and retrieval. 2016.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Thirty-Third AAAI Conference on Artificial Intelligence, 2019.
Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.0460 046.9, 2023.
Alistair EW Johnson, Tom Pollard, Seth Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientificc duta, 6(1):317, 2019a.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horn., Mimic-cxr-jpg. a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.0704 2019b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint 2rXiv:2305.17216 2023a.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models images for multimodal inputs and outputs, 2023b.
Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-24 243, 1991.
Hyungyung Lee, Wonjae Kim, Jwa Kim, Tackeun Kim, Jihang Kim, Leonard Sunwoo, and Edward Choi. Unified chest x-ray and radiology report generation model with multi-view chest x-rays. arXiv preprint arXiv:2302. 12172, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
11

---

Published as a conference paper at ICLR 2024
Ilya Loshchilov and Frank Hutter. axx0:1771.05100,20000 2017.
Decoupled weight decay regularization. arXiv preprint
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.00 200.:22000.8916,, 2022.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: multimodal medical few-shot learner. arXiv preprint arXiv:2307 15189, 2023.
OpenAL. Introducing chatgpt. http://openniccommbeegchhhug 2022.
OpenAL. Gpt-4 technical report, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooiing llama model. https://g wtmp..t.ooocoooo.o...oooommmmm.t htttmm/tttutuuee wtt.stsswwwa///atoorrrt.ro a paca 2023.
Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint 27X11:2306.07971, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequenee learning framework. In International Conference Machine Learning, pp. 23318-23340. PMLR, 2022a.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022b.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text, 2022c.
Jason Wei. Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot leamerrs arXiv preprint arXiv:2109.01652, 2021.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.024000 2023a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 1lm. arXiv preprint arXiv:2309.05519 2023b.
Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng. Atilla Kiraly, Sahar Kazemzadeh, Zakkai Melamed, Jungyeon Park, Patricia MacWilliams, Yun Liu, Chuck Lau, Precti Singh, Christina Chen, Mozziyar Etemadi, Sreenivasa Raju Kalidindi, Kat Chou, Greg Corrado, Shravya Shetty, Daniel Tse, Shruthi Prabhakara, Daniel Golden, Rory Pilgrim, Krish Eswaran, Andrew Sellergren, and Yossi Matias. Elixr: Towards general purpose x-ray artificial intelligence system through alignment large language models and radiology vision encoders. arxiv, 2023a. URL rxi org/ abs / /200 01317
12

---

Published as a conference paper at ICLR 2024
Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Attila Kiraly, Sahar Kazemzadeh, Zakkai Melamed, et al. Elixr: Towards a general pupose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders. arXiv preprint arXiv:2308.01317 2023b.
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vilge Unified generative pre- pre-training for bidirectional languae generation. arXiv preprint arXiv:2112.15283, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In CVPR, 2018.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
A IMPLEMENTATIONS DETAILS
Dataset We used MIMIC-CXR v2.0.0 (Johnson et al. 2019a) as our dataset CXR-report pairs. The data set consists of 377.110 CXRs from 227,835 radiology studies. The train-test split used the standard split of MIMIC-CXR-JPC (Johnson et al. 2019b). The test set sizes before and after pruning are 368,960 and 70,403, respectively. The test set was created using only AP/PA views from the raw data set excluding the train set, with a total of 3,530. The original images are files of various sizes, but the images were converted into JPEGs of square 256x 256 images.
Training VQ-GAN We trained the VQ-GAN (Esser et al. 2021 on 256x256 MIMIC-XXI train data starting from the pretrained weight and configuration the imgeeeet Soannet_f1r 6_106_1112220011 1024 model. For clinical information-preserviig preserving loss, 1024-dimensional features were extracted with TorchXRay BaayhXRRaVisio''ss Vision's (Cohen model of the target image and the reconstructed image, and then the L2 distance of the two features was used. At this time, the loss multiplied by 100 the weight and added to the total loss. The number of indices the codebook King is 1024 and the dimension of the codebook embedding n. is 256. Since 256x254 image encoded and quantized with 16x 16 matrix by encoder and quantizer, the dimension of the quantized latent vector (i.e, image tokens) of the image de is 256 by flattening it. Model training was performed for 590k steps with the Adam (Kingma & Ba 2014) optimizer, with a batch size of 2 and a learning rate of 4,5e-6.
ct al. 2022) densenet 1osseeett21l-1 1e1--e22224-a2
We used the dolly-v2 121y--2--3 3b (Databricks) 2023) model, which fine-tuned for the coanuctioo-followiing task based on the GPT-NeoX (Black al. 2022) architecture, base model. The model has a total of 2.8 billion parameters and has 50821 token types (Kiext). We have extended additional 1024 (Kamg) new image tokens should be available. For each image token from the VQ-GAN encoder, the value obtained by adding Kecart to each image token value is input to the L.MM as token ID.. If the token output from the model is treated as an image token if the ID is greater than or equal to Krezt, and cach Keent subtracted and input to the VQ-GAN decoder.
Fine-tunning LLM
the number of entries in the token embedding table to 51845 (Krent King).
Model training was performed with a learning rate of 5e-6 and a batch size of 16 using the AdamW (Loshchilov & Hutter (207)) optimizer all stages. Stage uses 76k steps as epochs and stage uses 9k steps as epoch for trainin. Training took about 14.5 hours for stage I and about 1.5 hours for stage -erooo, report-to-CXR, CXR- CXR-VQA, and NI.- IF in the data set are [30%, 30%, 20%, 20%) and [21 %, 21 %, 63%, 5%), for stage and stage 2, respectively.
2 using NVIDIA 100 40GB 8.. The proportions
B SYNTHETIC VQA GENERATION
We locally used LlaMa 2 (Ll 2 1amma--11b-chaa 1h---chhhhhhhh Touvron al. (203) to generate the synthetic VQAs. Radiology reports used were filtered from the MIMIC-CXR dataset through the method mentioned in the text, and total of 27,322 were used. The following prompt used to generate about questions and answers using a chest X-ray report in MIMIC-CRR.
You radiologist asking questions about caast X-ray image. You always give long, detailed explanations as answers.
You are given a chest X-ray, delimited by triple backticks.
Create some questions about the chest x-ray, each question followed by an appropriate answer.
Output your questions and answers in JSON format like: [[ [[["question": "<question>>", "answer"; "<answerl>"),, {("question" "<question2>"" "answer": "<answer2>> ...].
[zeport]
14

---

Published as a conference paper at ICLR 2024
Note, questions about any change from the last or previous chest X-ray scans or CTs. Questions are also never
about future plans; questions always focus on the chest X-ray itself. Answers are very detailed and
include explanations without repeating words that the question.
Here are the questions and answers:
We generated total of 126, 26,795 question-answer pairs using the described method. All question-answer pairs from stage I, along with 75,100 pairs randomly extracted in stage 2, were employed for training purposes. The train-test split followed the MIMIC-CXR official split to which the source report belongs. Following are some examples of generated VQAs:
Q: What
is the most likely cause of the aubtle opacity at the right lung base? A: Early pneumonia is the most likely cause of the subtle opacity at the right lung base, as there is no pleural effusion or pneumothorax.
Q: Is there any pleural effusion?
A: No, there is no pleural effusion present in the chest X-ray.
Q: is there any pneumothorax?
A: No, there is no pneumothorax present in the chest X-ray.
What is the appearance the mediastinal silhouette and hila? A: The mediastinal silhouette and hila appear normal
chest X-ray,
Q: What is the impression based on the chest X-ray?
A: The impression based on the chest X-ray is aubtle opacity the right lung base, which could represent early pneumonia,
C TEMPLATE FOR LLM INSTRUCTIONS
Following is the template of the prompt for instruction fine-tuning from the Alpaca family which consists of Instruction, Input, Response sections. This template is employed consistently both the instruction-folloiin tuning process and the inference process. During inference, the model functions to predict tokens following the response key (### Response:).
Below is an instroctoon that describes task. Write a response that appropriately completes the request.
### Instruction:
(instruction)
Input:
(input)
### Response:
(response)
### End
D ADDITIONAL EXPERIMENT RESULTS
Additional experimental results that were not covered the main text due to space constraints are shown in Table and Table
15

---

Published as a conference paper at ICLR 2024
Table 5: CXR-to-Repot NLG Metries.
BLEU1
BLEU2
BLEU3
BLEU4
METEOR
ROUGE_L
CIDEr
RadFM
0.1344
0.0650
0,0362
0.0221
0.0867
0.1064
0.0264
UniXGen-512
0.1471
0.0736
0,0403
0.0233
0.0813
0.1331
0.1413
IFCC
0.0636
0.0268
0.01066
0.0040
0.0809
0.0673
0.0058
R2Gen
0.0898
0.0359
0,0134
0.0822
0.0734
0.0104
UniXGen-256
0.1287
0,0570
0.0264
0.0137
0,0725
0.0893
0.0399
XrayGPT
0.1002
0.0417
0.0178
0,1038
0.0907
0.0139
LLM-CXR
0.0920
0.0459
0.0260
0.0154
0.0690
0.1618
0.5248
Table 6: FID of generated CXRs from reports
FID
inception-v3-20488
txv-all-1024 +
UniXGen
78.19 42.38
7,894
RoentGen
6.039
LLM-CXR
22.75
0.7136
E ABLATION STUDIES AND DISCUSSION
We conducted comprehensive ablation study to provide rigorous justification for the design choices made in our method. This ablation analysis specifically focused on the tasks and was evaluated using the evaluation metrics those outlined in the main text.
and report-to-CXX
In this ablation study, we systematically removed one element at a time from our method to assess its impact. The factors subjected ablation included the clinical information- trreeraiion-peservvingg loss (CIP loss), simultaneous training of the CXR-VE CXR-VQA task (CXR-VQA) the use the entire dataset by the additional training (stage tr.) instead of using only pruned dataset, and the adoption of instruction tuning loss Linstruct during fine-tuning as opposed to the joint loss Ljoins (instruct tr.).
Table 7: CXR-to-report generation AUROC and FI.
AUROC
F1 f
Micro
Macro
Weighted
Micro
Macro
Weighted
LLM-CXR
0.6285
0,5553
0.5969
0,3604
0,2113
0.3504
CIP loss
0.6170
0.5495
0,5853
0,3418
0.1994
0.3260
CXR-VQA
0.6143
0.5492
0,5846
0.3378
0.1945
0.3173
stagel tr.
0,5770
0.5242
0.5435
0.2659
0.1347
0.2253
instruct 0.6071
0.5422
0.5745
0.3203
0.1927
0.3161
The findings from our CXR-to-report ablation study (Table 7) highlight the positive contributions of all ablation factors toward enhancing the model's performance. Notably, the incorporation of stage 2 training, which involves initially training with the full dataset, and the generation of the CXR-VQA dataset through augmentation, alongside simultaneous CXR-VQA task training, led to a substantial improvement in the alignment between images and reports within the CXR-to-report task.
This improvement can be attributed to several factors. Firstly, the significantly increased volume of image-report pairs during stage training enabled more accurate learning of common meporrreport relationships, even though the additional dataset may not be directly related to our final tasks. Secondly, the CXR-VQA task provided direct supervision in comprehending and answering specific image characteristics, in contrast to the report generation task, where information about single image is
3 Txv-all-1083 is measured from the 1024-dim features of densenetl 21teeee22-1re2-2----- res2224--l from Cohen et al. 2022). Because we inevitably used our specific composition of MIMIC-CRR evaluation datasets for common evaluation across multiple models and tasks, the results may differ from those reported.
16

---

Published as a conference paper at ICLR 2024
distributed and represented. Consequently, these results suggest that the capacity to understand and respond to images acquired through the VQA task not only enhanced performance within the VQA task but also improved the overall quality of report generation.
Table 8: Report-to-CXR generation AUROC and Fl.
AUROC1
F1
Micro
Macro
Weighted
Micro
Macro
Weighted
LLM-CXR
0.8065
0.8025
0.8054
0.7907
0.7980
0.7991
CIP loss
0.8227
0.8196
0.8223
0.7890
0.7971
0.7977
CXR-VQA
0.7505
0.7433
0.7465
0.7841
0.7871
0.7876
stagel tr.
0.7213
0.7125
0.7167
0.7217
0.7224
0.7231
instruct tr.
0,5839
0,5751
0,5808
0.6040
0.5959
0.5965
In the case of the report-to-CXR task (Table most design choices led to unambiguous improvements. However, the addition of the clinical information-preserving (CIP) loss shows conflicting effects by decreasing AUROC whilst increasing the Fl-score. This may be due to better performance for minority classes with smaller support (e.g. pneumonia, enlarged cardiomediastinum) but lower performance for the more common majority classes (e.g. normal, pleural effusion). Taking into account that the incorporation of the CIP loss resulted in a substantial performance boost in the CXR-to co--oott task, we view this as favorable trade-off within the constraints of the limited capacity of a small model such dolly-v2-3b.
Table 9: FID of generated CXRs from reports.
FID
inception-y3-2048
txv-all-1024
LLM-CXR
22.75
0.714
CIP loss
20.93
0,931
CXR-VQA
32.55
1.539
stagel
32,05
1.226
instruct tr.
21.51
1.477
We also measure FID score for generated CXR images (Table 9). All design choices lead a decrease (improvement) in the FID score when measured using a CXR-specific classifier network (txv-all- 1024). Certain techniques increase (worsen) the FID score measured using the Inception- V3 network. This most likely due to the fact that our framework generates images that closely match the distribution of real CXR images so that features extracted from a network trained only on natural images such as Inception-V V3 cannot distinguish the subtleties in the generated images. Therefore, FID scores measured using txv-all- 1024 are more appropriate for image quality assessment, and thus all techniques employed in the final model (LLM-CXR) can be interpreted increasing generated CXR image quality.
F INSTRUCTIONS FOR MULTIMODAL TASKS
For the diversity of instructions, in the process of training and inference, one instruction is randomly sampled and used from the list 10 instructions below. The instructions were modulated to 10 using OpenAl's ChatGPT OpenAl 2022) from the basic instruction.
ReRto- task
Generate free-text radiology reports for the entered chest X-ray images.
Use the entered chest X-ray images to create corresponding free-text radiology reports.
Based on the entered chest X-ray images, produce free-text radiology reports.
Create free-text radiology reports that correspond to the entered chest X-ray images.
Utilize the entered chest X-ray images to generate corresponding free-text radiology reports.
Generate free-text radiology reports in accordance with the entered chest X-ray images.
17

---

Published as a conference paper at ICLR 2024
Use the entered chest X-ray images to create accurate free-text radiology reports.
Produce free-text radiology reports that match the entered chest X-ray images.
Create free-text radiology reports that are consistent with the entered chest X-ray images.
Utilize the entered chest X-ray images to generate comprehensive free-text radiology reports.
Report-to-CXR task
Generate chest X-ray image that corresponds to the entered free-text radiology reports for the chest X-ray image.
Use the free-text radiology reports for the chest X-ray image to produce a corresponding chest X-ray image.
Utilize the entered free-text radiology reports for the chest X-ray image to create a corresponding chest X-ray image.
Create chest X-ray image that matches the free-text radiology reports entered for the chest X-ray image.
Produce a chest X-ray image that consistent with the free-text radiology reports entered for the chest X-ray image.
Based on the free-text radiology reports for the chest X-ray image, generate a corresponding chest X-ray image.
Use the free-text radiology reports entered for the chest X-ray image create corresponding chest X-ray image.
Generate a chest X-ray image that is in accordance with the free-text radiology reports for the chest X-ray image entered.
Create chest X-ray image that corresponds to the free-text radiology reports entered for the chest X-ray image.
Utilize the entered free-text radiology reports for the chest X-ray image produce corresponding chest X-ray image.
G TO-OR- MORE EXAMPLES
Semantic descriptions of pathologies Radiology reports describe the semantic features of pathologies they appear on the CXR scan. The most common descriptions involve location and severity, Here show that our model incorporates features described in radiology reports when generating corresponding CXR images (Figure G).
Artificial devices Artificial devices are frequently captured in CXR images and reports. They have semantic features that are different from physiologic or pathologic features. We show that our model has leamned to generate the general appearance of these devices (Figure
18

---

Published as a conference paper at ICLR 2024
(e)
candloputmonory
process.
'Mii bilttral
pulmonary vnlimo"
brlotoro!
"Sevee right
pulmonary offvsion
(+) Wild left puimonary effoii"
Figure 6: CXRs generated for different descriptions of pathologies. The model is able to accurately capture different levels of severity in the generated CXRs (b, e) and generate lesions in specified locations (d, e).
19

---

Published as a conference paper at ICLR 2024
"No ocote
cordioputmonary
poolined innough lot trarsvenous
epproach rnd into the dgnn ventricte ond uppropriate? ?
"ACD commect postior."
endiotucheal
measures
asproxtmately Jcrs above the caring."
Wiight
cotricer
covostriel janrtine"
Figure 7: CXRs generated for radiology reports describing various foreign bodies. Reports normal CXRs (a) and of large features such pacemakers AICDs (b, realistically reflected in the generated images. Reports describing smaller, more detailed features such as endotracheal tubes and central venous catheters represented in the generated CXRs but less accurately (d, e), with deterioration in image quality around the neighborhood of the described feature (e.g. the trachea is not cleanly generated when the input report describes an endotracheal tube in (d)) imperfect representation of feature itself (e.g., venous catheters are generated but show missing parts in (e)).
20

---

Published as a conference paper at ICLR 2024
H CXR-TO-REPORT: MORE EXAMPLES
Radiologist's Impression (GT)
Generated Impression (ours)
(a)
"...Low lung volumes. Areas of otelectasis at both left and right lung basis. No pleural effusions. No pneumonia"
"Low lung volumes with patchy and streaky opocities most likely reflective of atelectasis. Infection aspiration cannot be excluded in the correct clinical setting..."
(b)
"Moderately severe pulmonary edemo..."
"Moderate pulmonary edemo."
(c)
"Mild pulmonary edema stable ... Cardiomegaly is accentuated by the low lung volumes."
"Enlorged cordiac silhouette and interstitial pulmonary edema."
(d)
"Increased markings bilaterolly may be due to the combination of underlying pulmonary fibrosis ond moderate pulmonary edemo, superimposed infectious process cannot be excluded."
"Slight increase in multifocol opacities which could be due to infection or mild pulmonary edema."
(e)
"Moderate cardiomegaly with AICD unchanged position. No evidence of congestive heart failure pneumonia."
"Pacer device leads terminating in the expected location of right otrium and right ventricle."
Figure 8: Generated reports contain not only diagnoses but also descriptions of pathologies present in CXR images such 'low lung volumes', and 'patchy and streaky opacities' (a). While the exact wording may differ from the ground-truth text report, generated reports are able to often capture the gist of the findings the CXR images ("moderately severe" vs. "moderate" in (b); "cardiomegaly" vs. "enlarged cardiac silhouette" in (c)). Suggestions for potential pathologic processes that underlie the findings in the CXR also align with ground-tue truth reports (d). Generated reports also note the presence of artificial devices such pacemakers (here, AICD recognized pacemaker the distinction relies finer details that would require further training to reliably distinguish (e)).
21


Reviews:
[{"review_id": "1yzlaEsLKP", "reviewer": "Reviewer_3hh7", "summary": "This study investigates the issue of multi-modal data alignment in large language models, using medical images and reports. \n\nIn terms of contributions, the authors propose a bidirectional reasoning generation mechanism that encodes images into tokens, allowing large language models to process and generate both text and images. In model training, a two-stage fine-tuning method is utilized, which not only captures the latent feature distribution of images but also imposes constraints on the model's representation of high-quality samples. \n\nOverall, this paper offers new perspectives on the processing of multi-modal data in large language models, demonstrating their value in the field of medical image processing.", "questions": "1. This paper demonstrates the potential application of LLMs in the medical field, but the content of the study extends beyond generating diagnostic reports from radiological images. I hope the author could further clarify in the introduction whether there are specific application scenarios for this research.\n\n2. In '2.1 CLINICAL INFORMATION-PRESERVING CXR TOKENIZATION', the author mentions '...causes loss of clinically important information such as characteristics of microscopic lesions...'. However, in this study, generating image tokens is key to aligning images with text during LLM fine-tuning. Is the mere use of L2 reconstruction loss sufficient to effectively reduce the loss of clinical information? Could there be an enhancement of features in the design of the module network structure? I think this is a very important challenge in this paper, and I hope the difference in experimental results before and after the introduction of L2 reconstruction loss can be explained.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author demonstrates the potential of large language models in radiological diagnostics, offering greater flexibility than previous methods in generating diagnostic reports or creating images.\n\n2. Inspired by VQ-GAN, the author ingeniously encodes images into tokens and integrates these image tokens into the large language model (LLM) for fine-tuning, achieving alignment of language and image features in the feature space.\n\n3. The author defines four tasks in this paper, particularly emphasizing the use of the CHATGPT API for Visual Question Answering (VQA), which further strengthens the association between images and text in the feature space. This part of the design is very interesting.\n\n4. During training, the dataset is cleansed, with initial learning of image latent features using low-quality data; the LLM is fine-tuned using the cleansed data.", "weaknesses": "1. The experiments in this paper were conducted using only one dataset. As the author mentioned, the quality of the dataset used was limited, necessitating adjustments in the training strategy.\n\n2. The paper lacks visualization of the alignment of images and text in the feature space. Particularly when using VQA for data augmentation, the questions posed by GPT contain rich prior information, focusing the text more on the key lesion areas in the images. I believe that appropriate visualization is necessary to demonstrate the effectiveness of incorporating VQA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYFDQagnpl", "reviewer": "Reviewer_xBeK", "summary": "The manuscript introduces an instruction-tuning technique geared towards amplifying the image comprehension and generative capabilities of a text-exclusive LLM for CXR imagery interpretation and generation. The outcomes indicate a top-tier performance in both image-text understanding and generative capacities, surpassing earlier models in the field.", "questions": "Methodology Concerns:\n1. How does the instruction-finetuning method differentiate itself from existing large-scale medical models that employ a similar strategy?\n\nComparative Analysis:\n1. Is the comparison between LLM-CXR and the open-sourced versions of XrayGPT and UniXGen made on a consistent dataset?\n2. Why hasn't the paper compared its methodology with existing LLM+instruction tuning models, such as Med-PaLM and Med-PaLM 2?\n3. Are there reasons for not exploring contemporary methodologies in report generation, like \"Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation, CVPR 2023\"?\n\nEvaluation Metrics:\nWhy were the AUROC scores for report generation only presented for six categories instead of the standard 14 categories?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method of instruction-finetuning effectively elevates the LLM's capability to intricately map vision-image modalities. Consequently, it markedly excels in tasks like CXR-to-Report generation, CXR-VQA, and Report-to-CXR generation, outshining the open-sourced versions of models such as XrayGPT and UniXGen.", "weaknesses": "1. The paper's premise relies on the fine-tuning of LLM across multiple instruction tuning tasks. This approach isn't particularly novel, as numerous large-scale medical models adopt a similar strategy but with a broader functional range. Both the innovation in method and the paper's applicative contribution seem to be lacking.\n2. The LLM-CXR, when compared with the open-sourced versions of XrayGPT and UniXGen, appears to be unfair. It's noted that these models aren't trained on a consistent instruction tuning dataset. It's recommended that a uniform dataset is utilized for training before making such comparisons. Furthermore, there's an evident absence of comparison with existing LLM+instruction tuning models, such as Med-PaLM and Med-PaLM 2.\n3. The paper's comparative methods across various understanding and generative tasks aren't comprehensive. For instance, in the realm of report generation, several contemporary methodologies exist, such as \"Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation, CVPR 2023.\"\n4. The evaluation metrics used for report generation seem lacking in depth. For instance, within the paper (table 1), only the AUROC scores across six categories are reported, while prior research typically reports the average Precision/Recall/F1-score across all 14 categories.\n5. The paper's writing style isn't fluid and contains multiple errors, hindering smooth reading and comprehension. For instance, repetitive phrases like \"… generate these VQAs as shown as shown in …\" and grammatical mistakes like \"During the fine-tuning process …\" detract from the paper's clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QNBuA1wLeW", "reviewer": "Reviewer_FRze", "summary": "The paper delves into enhancing Large Language Models (LLMs) with vision-language capabilities, specifically targeting medical imaging like chest X-rays (CXR). Recognizing that current \"adapter network\" methods might limit the deep integration of visual and language features, the authors propose a novel \"instruction-finetuning\" method. Drawing from vision-language pretraining (VLP) techniques, they tokenize images using VQ-GAN, facilitating the generation of combined text and image sequences. Rather than building a new model, they finetune a pretrained LLM with diverse CXR-related instructions. This approach allows the LLM to understand and generate visual data without structural modifications. Their finetuned LLM showcases proficiency in tasks such as translating CXRs to reports and vice versa, and performing CXR-specific visual question answering. The model not only outperforms specialized models in these tasks but also demonstrates the potential of seamlessly integrating visual and language abilities in LLMs for medical applications.", "questions": "If the authors address the areas of improvement, I can reconsider my assessment.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strengths:\n1. **Novel Approach**: The paper introduces a novel \"instruction-finetuning\" method, which is a significant departure from the prevalent \"adapter network\" techniques. This innovative approach allows for a more intimate integration of visual and language features in LLMs.\n\n2. **Leveraging Existing LLMs**: Instead of starting from scratch, the authors smartly utilize the inherent instruction-following abilities of pretrained LLMs. This approach is efficient and maximizes the potential of existing models.\n\n3. **Broad Application**: The finetuned LLM exhibits versatility in handling a range of tasks, from converting CXRs to reports, generating CXRs from textual reports, to CXR-specific visual question answering. This breadth of application showcases the model's potential in real-world medical scenarios.\n\n4. **Outperformance**: The paper demonstrates that their model surpasses other specialized models in various tasks. This comparative analysis underscores the efficacy of their approach.\n\n5. **Seamless Integration**: By using VQ-GAN for image tokenization, the authors ensure a smooth integration of image and text token spaces without necessitating structural changes to the base LLM. This seamless integration is crucial for practical implementations.", "weaknesses": "Areas for improvement:\n\n1. **Potential for Catastrophic Forgetting**: As with any endeavor to expand a pretrained model's capabilities, there's the inherent risk of the model losing or diminishing its foundational skills—known as catastrophic forgetting. While the paper's intent is to preserve and build upon the LLM's language capabilities, it doesn't extensively address measures taken to prevent this potential degradation.\n\n2. **Adapter Network Dismissal**: The paper critiques the prevalent \"adapter network\" approach but doesn't provide a comprehensive empirical comparison. A deeper, side-by-side evaluation highlighting performance, adaptability, and computational efficiency would offer a clearer picture of why their \"instruction-finetuning\" method is superior or preferable.\n\n3. **VQ-GAN Limitations**: Using VQ-GAN for image tokenization introduces its own set of challenges. VQ-GANs, while powerful, can sometimes produce artifacts or representations that aren't entirely faithful to the original image. The paper doesn't discuss how it mitigates or addresses these potential shortcomings, leaving room for questioning the quality or accuracy of the generated content.\n\n4. **Lack of Diverse Evaluation**: The paper's evaluation, though rigorous, might benefit from a more diverse set of metrics. For instance, qualitative evaluations or user studies involving medical professionals could provide insights into the model's practical utility. Comparisons with human diagnostic capabilities might also offer a benchmark for the model's proficiency.\n\n5. **RoentGen AUC and FID numbers**: The authors need to address the discrepancy between the AUC and FID numbers provided in their paper and those in the RoentGen paper. It should be clarified whether these numbers were reproduced by the authors or cited from the RoentGen study, and if there are differences, the paper should explain the reasons behind these.\n\n6. **Minor edits**:\n\na. Grammar throughout the text requires attention for better readability, as exemplified by the construction of the second sentence in section 2.2.\n\nb. The term \"txv-all-1024\" utilized in the text should be clearly defined. It is presumed to refer to the 1024-dimensional outputs from a densenet-121 model trained on chest X-ray classification, but this assumption needs confirmation in the paper for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TNPLXVrCdU", "reviewer": "Reviewer_DPiA", "summary": "This work developed an instruction-finetuning method to integrate visual information into out-of-the-box LLMs, which can be used for bidirectional multi-modal CXR tasks, such as CXR report generation, VQA, and report-to-CXR generation.", "questions": "- How scalable is the tokenization and fine-tuning process for larger medical images (e.g., 3D CT/MR scans)?\n- How interpretable are the model's decisions, especially given the clinical context where explanations for predictions are crucial?\n- The comparison with other related methods is missed. https://github.com/chaoyi-wu/RadFM", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The image is tokenized by VQ-GAN, ensuring that clinical information is preserved. It is an efficient use of existing resources and knowledge by expanding pre-trained LLM's embedding space to include image tokens.\n- The bidirectional instruction fine-tuning maintains the integrity of the LLM's structure and objectives while expanding its capabilities, which have many applications in the field.", "weaknesses": "The interpretability and scalability are not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lO3ryqdRrM", "reviewer": "Reviewer_MXmf", "summary": "The authors proposed an image-text alignment framework for chest X-ray images and report pairs based on LLM models. In addition to the existing vision adapter, the authors reconstruct the adapted vision representation back to the images.  Furthermore, VQA pairs (as image-instruction-answer) are generated using GPT3.5 from the associated reports as a form of data augmentation for the image report pairs.  The reconstruction module is pre-trained as a VQGAN and then frozen when tuning the VQA instructions. The MIMIC-CXR dataset is employed here for the experiments. Superior results of the proposed model are reported in comparison to previous LLM models in report generation and VQA. However, the presented work suffers from several critical flaws that are detailed below.", "questions": "see weaknesses", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ Tackling the image-text alignment problem in medical imaging, which is not well-researched yet\n+ The manuscript is overall easy-to-follow", "weaknesses": "- The authors claim the proposed bidirectional LLM is different from previous ones, as illustrated in Figure 1. However, I found the difference between them (a) and (c) is really minor. The encoder and decoder parts in the VQGAN are indeed equivalent to the vision adapter and image generative model, as listed in (a). Therefore, it is a bit overclaimed that it is novel to introduce bidirectional reconstruction tasks(both image-to-text and text-to-image) in the pre-training.\n- The motivation for training an image-text aligned model is not clearly introduced and justified. First, it will be helpful to discuss how this model could be applied to the downstream clinical tasks. Then, the designed experiments only demonstrate the performance of the proposed method on these instruction tuning tasks, i.e., report generation based on images and VQA. It is hard to appreciate the benefit of adopting such a pre-trained and then SFTed model in practical use without demonstrating downstream applications. There are many chest X-ray benchmarks, and datasets are commonly used for the evaluation of pre-trained models, e.g., disease classifications and localizations. \n- The metrics utilized in the experiments for most tasks are not the commonly used ones, e.g., AUCROC/F1 and Jaccard similarity index for report generation, FID alone for image generation, and accuracy alone for VQA. \n- In the results of report generation, only LLM-based methods are compared. How about a dozen of those SOTA  methods in chest X-ray reporting? IU X-ray is another dataset commonly used for the evaluation of report generation performance. It will be helpful to report results on that as well, mainly when used as a cross-domain evaluation dataset.  \n- I am not sure why the upsampling is performed for the image data as described in section 3.1 since the X-ray images are much larger than this resolution. How this upsampling process will affect the results?\n- In section 2.2, the authors mentioned that the image tokens are parts of expanded embedding tables. I wonder how big K_img should be?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)