SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chunyuan Li*, Cliff Wong*, Sheng Zhang*, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao
Microsoft https://aka.ms/llava-med
Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, fine-tuning LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.
Parallel image-text data is abundantly available in the general domain, such as web images and their associated captions. Generative pretraining has proven effective to leverage this parallel data for self-supervised vision-language modeling, as demonstrated by multimodal GPT-4 [35] and open-sourced efforts such as LLaVA [27]. By instruction-tuning models to align with human intents based on multimodal inputs, the resulting large multimodal models (LMMs) exhibit strong zero-shot task completion performance on a variety of user-oriented vision-language tasks such as image understanding and reasoning, paving the way to develop general-purpose multimodal conversational assistants [22, 2, 23, 10].
While successful in the general domains, such LMMs are less effective for biomedical scenarios because biomedical image-text pairs are drastically different from general web content. As a result, general-domain visual assistants may behave like a layperson, who would refrain from answering
biomedical questions, or worse, produce incorrect responses or complete hallucinations. Much progress has been made in biomedical visual question answering (VQA), but prior methods typically formulate the problem as classification (e.g., among distinct answers observed in the training set) and are not well equipped for open-ended instruction-following. Consequently, although conversational generative AI has demonstrated great potential for biomedical applications [20, 33, 19], current investigations are often limited to unimodal text.
In this paper, we present Large Language and Vision Assistant for BioMedicine (LLaVA-Med), a first attempt to extend multimodal instruction-tuning to the biomedical domain for end-to-end training of a biomedical multimodal conversational assistant. Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9]. Most recently, large-scale biomedical VL learning has been made possible by the creation of PMC-15M [51], a broad-coverage dataset with 15 million biomedical image-text pairs extracted from PubMed Central<sup>1</sup>. This dataset is two orders of magnitude larger than the next largest public dataset, MIMIC-CXR [16], and covers a diverse image types. Inspired by recent work in instruction-tuning [37, 27], LLaVA-Med uses GPT-4 to generate diverse biomedical multimodal instruction-following data using image-text pairs from PMC-15M, and fine-tune a large biomedical-domain VL model [27] using a novel curriculum learning method.
Specifically, our paper makes the following contributions:
- Biomedical multimodal instruction-following data. We present a novel data generation pipeline to create diverse (image, instruction, output) instances, by sampling biomedical image-text pairs from PMC-15M and using GPT-4 to create instructions from the text alone (which becomes the intended output). This requires zero manual annotations and creates an extremely diverse visual instruction-following dataset by piggybacking on PMC-15 that covers the full spectrum of research findings over biomedical images.
- LLaVA-Med. We propose a novel curriculum learning method for adapting LLaVA [27] to the biomedical domain using our self-generated biomedical multi-modal instruction-following dataset. Specifically, we first fine-tune LLaVA to align biomedical vocabulary using the image-text pairs as is (with the generic instruction that simply asks for a description of the image). We then continue training the model using our self-generated instruction-following data to learn open-ended conversational semantics. In this way, we were able to train LLaVA-Med in less than 15 hours with eight A100s. Our empirical study validates the effectiveness of domain-specific instruction-tuning, and reveals best practice and interesting findings for adapting multimodal conversational assistant to high-value verticals. On well-established biomedical VQA datasets, fine-tuning LLaVA-Med often outperforms supervised state-of-the-art (SoTA).
- Open-source. To facilitate research in biomedical multimodal learning, we will release the following assets to the public: the biomedical multimodal instruction-following dataset and the codebase for data generation and model training.
Biomedical Chatbots. Inspired by ChatGPT [34]/GPT-4 [35] and the success of open-sourced instruction-tuned large language models (LLMs) in the general domain, several biomedical LLM chatbots have been developed, including ChatDoctor [49], Med-Alpaca [13], PMC-LLaMA [47], Clinical Camel [1], DoctorGLM [48], and Huatuo [46]. They are initialized with open-sourced LLM and fine-tuned on customized sets of biomedical instruction-following data. The resulting LLMs emerge with great potential to offer assistance in a variety of biomedical-related fields/settings, such as understanding patients' needs and providing informed advice.
To our knowledge, Visual Med-Alpaca [42] is the only existing multimodal biomedical chatbot that accepts image inputs. Though Visual Med-Alpaca and the proposed LLaVA-Med share a similar input-output data format, they differ in key aspects: (i) Model architectures. LLaVA-Med is an end-to-end neural model and Visual Med-Alpaca is a system that connect multiple image captioning models with a LLM, using a classifier to determine if or which biomedical captioning model is responsible for the image. The text prompt subsequently merges the converted visual information with the textual query, enabling Med-Alpaca to generate an appropriate response. (ii) Biomedical
<sup>1</sup>https://www.ncbi.nlm.nih.gov/pmc/
instruction-following data. While Visual Med-Alpaca is trained on 54K samples from limited biomedical subject domains, LLaVA-Med is trained a more diverse set.
Biomedical Visual Question Answering. An automated approach to building models that can answer questions based on biomedical images stands to support clinicians and patients. To describe existing biomedical VQA methods, we make a distinction between discriminative and generative methods. For discriminative methods, VQA is treated a classification problem: models make predictions from a predefined set of answers. While discriminative methods yield good performance, they deal with closed-set predictions [14], and require mitigation when a customized answer set is provided in at inference [24, 51, 9]. The discriminative formulation is suboptimal towards the goal of developing a general-purpose biomedical assistant that can answer open questions in the wild. To this end, generative methods have been developed to predict answers as a free-form text sequence [5, 30, 44]. Generative methods are more versatile because they naturally cast the close-set questions as as special case where candidate answers are in language instructions.
Data-Centric Paradigms. LLaVA-Med is similar to prefix tuning of language models (LMs) in [44] in that a new trainable module connects frozen image encoder and causal LM. In [44], a three-layer MLP network is used to map the visual features into a visual prefix, and the pre-trained LM are GPT2-XL [40], BioMedLM [45] and BioGPT [31], with size varying from 1.5B to 2.7B. By contrast, LLaVA-Med uses a linear projection and a 7B LM [8, 43]. Most importantly, we undertake data-centric paradigm, while all existing methods are model-centric. [44] focuses efforts on exploring various modeling choices. Our main contributions instead comprise proposing a novel data generation method that uses GPT-4 to self-instruct biomedical multimodal instruction-following data using freely-available broad-coverage biomedical image-text pairs extracted from PubMed Central [51].
3 Biomedical Visual Instruction-Following Data
There are a lack of multimodal biomedical datasets to train an instruction-following assistant. To fill this gap, we create the first dataset of its kind from widely existing biomedical image-text pairs, through a machine-human co-curation procedure. It consists of two sets, concept alignment and instruction-following, which are used at different training stages, described in Section 4.
Biomedical Concept Alignment Data. For a biomedical image  $\mathbf{X}_{\mathrm{v}}$  and its associated caption  $\mathbf{X}_{\mathrm{c}}$ , we sample a question  $\mathbf{X}_{\mathrm{q}}$ , which asks to describe the biomedical image. With  $(\mathbf{X}_{\mathrm{v}}, \mathbf{X}_{\mathrm{c}}, \mathbf{X}_{\mathrm{q}})$ , we create a single-round instruction-following example:
$$
\text {H u m a n}: \mathbf {X} _ {\mathrm {q}} \mathbf {X} _ {\mathrm {v}} <   \text {S T O P} > \backslash \mathrm {n} \text {A s s i s t a n t}: \mathbf {X} _ {\mathrm {c}} <   \text {S T O P} > \backslash \mathrm {n} \tag {1}
$$
Depending on the length of caption, the question that is sampled either asks to describe the image concisely or in detail. Two lists of questions are provided in Appendix A. In practice,  $25\%$  of captions have length less than 30 words in PMC-15M [51], and thus 30 words is used as the cutoff point to determine which list to choose. We sample 600K image-text pairs from PMC-15M. Though this dataset only presents one-single task instructions, i.e., image captioning, it contains a diverse and representative set of biomedical concept samples from the original PMC-15M [51].
Biomedical Instruction-Tuning Data. To align the model to follow a variety of instructions, we present and curate diverse instruction-following data with multi-round conversations about the provided biomedical images, by prompting language-only GPT-4. Specifically, given an image caption, we design instructions in a prompt that asks GPT-4 to generate multi-round questions and answers in a tone as if it could see the image (even though it only has access to the text). Sometimes the image caption is too short for GPT-4 to generate meaningful questions and answers. To provide more context regarding the image, we also create a prompt that includes not only captions but also sentences from the original PubMed paper that mentions the image. We also manually curate few-shot examples in the prompt to demonstrate how to generate high-quality conversations based on the provided caption and context. See Appendix A.2 for the prompt and few-shot examples. To collect image captions and their context, we filter PMC-15M to retain the images that only contain a single plot. From them, we sample 60K image-text pairs from the five most common imaging modalities: CXR (chest X-ray), CT (computed tomography), MRI (magnetic resonance imaging), histopathology, and gross (i.e., macroscopic) pathology. We then extract sentences that mention the image from the
Biomedical Visual Instruction-Following Example
Figure 4: Contrast-enhanced CT scan of the chest for patient #1. A large cavitating lesion was identified in the right upper lobe with pleural involvement and likely extension into the chest wall. In addition, right hilar lymphadenopathy was suspected. This was the primary malignant lesion in this patient.
Sentences with in-line figure mentions:
- Computerized tomography (CT) scans of the chest (Figure 4) were obtained.
- (Figure 4) and demonstrated a large cavitating lesion in the posterior aspect of the right upper lobe with probable pleural involvement likely representing the primary lesion.
User: Where is the lesion located?
Assistant: The lesion is located in the posterior aspect of the right upper lobe of the lung.
User: Is it extending into the chest wall?
Assistant: Yes, the CT scan suggests that the large cavitating lesion in the right upper lobe is likely extending into the chest wall.
User: Can you explain more about the cavitating lesion?
Assistant: Certainly. A cavitating lesion refers to an abnormal mass or growth within the lung tissue that has a cavity or hollow space inside it. This cavity can be formed due to the necrosis (death) of the tissue inside the lesion, leading to the breakdown and disintegration of the affected tissue. Cavitating lesions can be caused by various factors, such as infections (like tuberculosis or fungal infections), autoimmune diseases, or malignancies (cancer). In the context of the CT scan provided, the large cavitating lesion in the right upper lobe of the lung is suggestive of a malignant (cancerous) growth. The cavity within the lesion may be a result of the rapid and uncontrolled growth of cancer cells, which can lead to the destruction of surrounding healthy tissue. It's important to note that further diagnostic tests, such as a biopsy, would be required to confirm the malignancy and determine the specific type of cancer involved.
Figure 1: An instance of our GPT-4 generated instruction-following data. Top: The figure and caption were extracted from a PubMed Central full-text article [38], along with the corresponding citances (mentions of the given figure in the article). Bottom: The instruction-following data generated by GPT-4 using the text only (caption and citances). Note that the image is not used to prompt GPT-4; we only show it here as a reference.
original PubMed paper as additional context to the caption, inspired by the observations that external knowledge helps generalization [21, 28].
An example of instruction-following data is shown in Figure 1 shows, and the data statistics is shown in Figure 2. We have produced three versions of instruct data when iteratively improving the data quality: (i)  $60K$ -IM. The aforementioned dataset that considers inline mentions (IM) as the context. (ii)  $60K$ . A dataset of similar size (60K samples) without IM in self-instruct generation. (iii)  $10K$ . A smaller dataset (10 samples) without IM. They are used to ablate our data generation strategies and their impact on trained LLaVA-Med in experiments.
4 Adapting Multimodal Conversational Models to the Biomedical Domain
We employ LLaVA, a general-domain multimodal conversation model [27], as the initial general-domain LM, and continuously train the model to the biomedical domain. The same network architecture is utilized, where a linear projection layer connects the vision encoder and the language model. For LLaVA-Med model training, we use a two-stage procedure, illustrated in Figure 3.
(a) Instruction
(b) Responses
(c) Frequencies of images and QA pairs on the five domains.
Figure 2: The data statistics of biomedical multimodal instruction-following data: (a,b) The root verb-noun pairs of instruction and responses, where the inner circle of the plot represents the root verb of the output response, and the outer circle represents the direct nouns. (c) The distribution of images and QA pairs on the five domains, one image is shown per domain. The domain example images are from [3, 36, 4, 32, 50].
Stage 1: Biomedical Concept Feature Alignment. To balance between concept coverage and training efficiency, we filter PMC-15M to 600K pairs. These pairs are converted to instruction-following data using a naive expansion method: instructions simply presents the task of describing the image. For each sample, given the language instruction and image input, we ask the model to predict the original caption. In training, we keep both the visual encoder and LM weights frozen, and only update the projection matrix. In this way, the image features of vast novel biomedical visual concepts can be aligned to their textual word embeddings in the pre-trained LM. This stage can be understood as expanding the vocabulary of aligned image-text tokens to the biomedical domain. This stage is critical when training LLaVA-Med from customized vision encoder and LLM, where LLaVA initialization is not available. However, when training LLaVA-Med by initializing from LLaVA, we find that this stage can be either skipped to save compute cost, or consider the strategy of removing  $\mathbf{X}_{\mathbf{q}}$  in (1) and tuning LLM for higher performance. Please see the discussion in Appendix C.2.
Stage 2: End-to-End Instruction-Tuning. We only keep the visual encoder weights frozen, and continue to update both the pre-trained weights of the projection layer and LM. To train the model to follow various instructions and complete tasks in a conversational manner, we develop a biomedical chatbot by fine-tuning our model on the biomedical language-image instruction-following data collected in Section 3. As demonstrated in the experiments to be described later, the LLaVA-Med model at this stage is able to not only be served as a biomedical visual assistant to interact with users, but also achieve good zero-shot task transfer performance when evaluated on well-established biomedical VQA datasets.
Discussion. We discuss four favorable properties/implications of LLaVA-Med: (i) Affordable development cost. Instead of scaling up data/model for the best performance, we aim to provide affordable and reasonable solutions with low development cost: it takes 7 and 8 hours for stage 1 and 2 on 840G A100 GPUs, respectively (see Table 5 for detailed numbers). (ii) A recipe for many domains.
Figure 3: LLaVA-Med was initialized with the general-domain LLaVA and then continuously trained in a curriculum learning fashion (first biomedical concept alignment then full-blown instruction-tuning). We evaluated LLaVA-Med on standard visual conversation and question answering tasks.
Though this paper focuses on biomedical domains, the proposed adaptation procedure is generalizable to other vertical domains such as gaming and education, where novel concepts and domain knowledge are needed to build a helpful assistant. Similar to the don't stop pre-training argument in [12], we consider a scalable pipeline to create domain-specific instruct data from large unlabelled data, and advocate don't stop instruction-tuning to build customized LMM. (iii) Low serving cost. While the model size of general LMM can be giant and serving cost can be prohibitively high, customized LMM has its unique advantages in low serving cost. (iv) Smooth Model Adaptation. Alternatively, the network architecture allows us to initialize the vision encoder from BioMedCLIP [51], or initialize the language model from Vicuna [8], which may lead to higher performance. However, adapting from LLaVA smooth adaptation as a chatbot, where model's behaviors transit from layperson to a professional assistant that is able to provide helpful domain-specific response.
We conduct experiments to study two key components, the quality of the produced multimodal biomedical instruction-following data, and performance of LLaVA-Med. We consider two research evaluation settings: (1) What is the performance of LLaVA-Med as an open-ended biomedical visual chatbot? (2) How does LLaVA-Med compare to existing methods on standard benchmarks? To clarify, throughout the entire experiments, we only utilize the language-only GPT-4.
5.1 Biomedical Visual Chatbot
To evaluate the performance of LLaVA-Med on biomedical multimodal conversation, we construct an evaluation dataset with 193 novel questions. For this test dataset, we randomly selected 50 unseen image and caption pairs from PMC-15M, and generate two types of questions: conversation and detailed description. The conversation data is collected using the same self-instruct data generation pipeline as for the 2nd stage. Detailed description questions were randomly selected from a fixed set [27] of questions to elicit detailed description responses.
We leverage GPT-4 to quantify the correctness of the model answer to a question when given the image context and caption. GPT-4 makes a reference prediction, setting the upper bound answer for the teacher model. We then generate response to the same question from another LMM. Given responses from the two assistants (the candidate LMM and GPT-4), the question, figure caption, and figure context, we ask GPT-4 to score the helpfulness, relevance, accuracy, and level of details of the responses from the two assistants, and give an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. GPT-4 is also asked to provide a comprehensive explanation the evaluation, for us to better understand the models. We then compute the relative score using GPT-4 reference score for normalization.
The results are reported in Table 1. LLaVA-Med with Stage-1 training alone is insufficient as a chatbot, as it loses its ability to follow diverse instructions, though biomedical concept coverage is improved. LLaVA-Med with the full two-stage training consistently outperforms the general domain LLaVA, and training with larger instruct data (from 10K to 60K samples) leads to higher performance. When inline mentions are considered in self-instruct, the generated data 60K-IM slightly improves the chat ability. The results demonstrate the effectiveness of the strategies in biomedical instruction-following data collection as well as the value of dataset assets. Overall, for the best LLaVA-Med, it matches the  $50.2\%$  performance of GPT-4. Note that GPT-4 generates response by considering
(Question Count)	Question Types	Domains	Overall
Conversation (143)	Description (50)	CXR (37)	MRI (38)	Histology (44)	Gross (34)	CT (40)	(193)
LLaVA	39.4	26.2	41.6	33.4	38.4	32.9	33.4	36.1
LLaVA-Med
Stage 1	22.6	25.2	25.8	19.0	24.8	24.7	22.2	23.3
10K	42.4	32.5	46.1	36.7	43.5	34.7	37.5	39.9
60K	53.7	36.9	57.3	39.8	49.8	47.4	52.4	49.4
60K-IM	55.1	36.4	56.2	40.4	52.7	51.8	50.1	50.2