SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Performance comparison of multimodal chat instruction-following abilities, measured by the relative score via language GPT-4 evaluation.
Visual input example, Biomedical image from [6]
Caption: Chest x-ray (CXR) on Day 2 of admission post-intubation (yellow line showing the level of the endotracheal tube). Rapidly worsening ground-glass opacities are seen throughout the lungs with relative subpleural sparing (red arrows)
Source: https://www.ncbi.nlm.nih.gov/PMC/articles/PMC7398722/
User	What is shown in this image?
LLaVA-Med	The image is a chest X-ray (CXR) that shows bilateral patchy infiltrates, which are areas of increased opacity in the lungs. These infiltrates can be indicative of various lung conditions, such as infections, inflammation, or other lung diseases.
User	Are there any devices implanted in this patient?
LLaVA-Med	Yes, it appears that the patient has several devices implanted. These devices may include endotracheal tubes, central venous catheters, and/or pulmonary artery catheters. These devices are typically used for monitoring and managing various medical conditions, as well as for providing support during surgical procedures.
User	What is shown in this image?
LLaVA	The image shows a ribcage with wires coming out of it, indicating that it is likely a medical illustration or diagram.
User	Are there any devices implanted in this patient?
LLaVA	Yes, there are several devices protruding from the patient's chest, which are likely medical devices used for treatment or monitoring.
User	What is shown in this image?
Language GPT-4	The image shows a chest x-ray taken on Day 2 of a patient's admission after intubation. You can see the endotracheal tube marked by a yellow line, and rapidly worsening ground-glass opacities throughout the lungs. There is also relative subpleural sparing, which is indicated by red arrows.
User	Are there any devices implanted in this patient?
Language GPT-4	Yes, there is an endotracheal tube implanted in the patient, as indicated by the yellow line in the image.
Table 2: Example comparison of medical visual chat and reasoning capabilities. The language-only GPT-4 is considered as the performance upper bound, as the golden captions and inline mentions are fed into GPT-4 as the context, without requiring the model to understand the raw image.
ground-truth caption and golden inline mentions, without understanding the images. Though not a fair comparison between LMMs and GPT-4, GPT-4 is a consistent and reliable evaluation tool.
Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations. While we agree with the existence of the bias, we believe that GPT-4 is a meaningful and consistent measurement in our settings. We always compare the answers from the candidate model against the GPT-4's answers when computing the relative score, and the ranking of the resulting numbers is consistent, though the numbers themselves might be biased to GPT's answers. The conclusions of our ablation study can be obtained by considering the ranking. Further, by taking the self-enhancement bias into consideration for fairness, we expect that LLaVA-Med actually performs even closer to GPT-4 than the current numbers indicate.
In Table 2, we provide examples on the biomed visual conversations of different chatbots. LLaVA-Med precisely answers the questions with biomedical knowledge, while LLaVA behaves like a
Method	VQA-RAD	SLAKE	PathVQA
Ref	Open	Closed	Ref	Open	Closed	Ref	Open	Closed
Supervised finet-tuning results with our own experiment runs
LLaVA	50.00	65.07	78.18	63.22	7.74	63.20
LLaVA-Med (From LLaVA)	61.52	84.19	83.08	85.34	37.95	91.21
LLaVA-Med (From Vicuna)	64.39	81.98	84.71	83.17	38.87	91.65
LLaVA-Med (BioMed CLIP)	64.75	83.09	87.11	86.78	39.60	91.09
Representative & SoTA methods with numbers reported in the literature
VL Encoder-Decoder [5]	71.49	82.47	71.49	85.61
Q2ATransformer [30]	79.19	81.20	54.85	88.85
Prefix T. Medical LM [44]	84.30	82.01	40.00	87.00
PubMedCLIP [9]	60.10	80.00	78.40	82.50
BiomedCLIP [51]	67.60	79.80	82.05	89.70
M2I2 [24]	66.50	83.50	74.70	91.10	36.30	88.00
Table 3: Comparison with prior state-of-the-art supervised methods. For open-set questions, we report the recall for our free-form text generation method in column Open. For closed-set questions, we report the accuracy in column Closed. Bold indicates LLaVA-Med achieves new SoTA. For open-ended questions, prior methods still formulate the problem as classification among distinct answers in the training set, which may overestimate their generalizability as these datasets are unusual in that the test answers are almost always present in training..
layperson, who hallucinate based on commonsense. Since the multimodal GPT-4 is not publicly available, we resort to language-only GPT-4 for comparison. We feed golden captions and inline mentions into GPT-4 as the context, it generates knowledgeable response through re-organizing the information in the conversational manner.
5.2 Performance on Established Benchmarks
Datasets and Evaluation Metrics. We train and evaluate LLaVA-Med on three biomedical VQA datasets. The data statistics are summarized in Table 8 in Appendix. For the closed-set questions, we report the accuracy/percentage of the ground-truth tokens that appear in the generated sequences. For open-set questions, we use recall to evaluate the ratio that ground-truth tokens appear in the generated sequences. In the literature, the open-set problem is formulated as an closed-set setting, where the unique training answers are considered as the answer candidates, from which the models can select to predict answers for testing questions. Since we do not provide any constraint for the responses to open-set questions, our formulation is closer to open-set nature, but is intrinsically harder.
Comparisons with SoTA. We compare LLaVA-Med with the general domain LLaVA and existing representative methods in Table 3. First, All LLaVA-Med variants outperform LLaVA. While the difference of language model initialization from LLaVA or Vicuna is minor, the initialization of vision encoder from BioMed CLIP is slightly better than from general-domain CLIP. Second, the fine-tuning performance of LLaVA-Med is higher than supervised SoTA on the closed-set questions on VQA-RAD and PathVQA. This validates LLaVA-Med's strong ability in following instruction to complete biomedical tasks, when clear instructions are provided (e.g., yes or no). Third, for open-set questions, LLaVA-Med achieves SoTA on SLAKE, while its performance is limited on other datasets, especially compared with existing methods. This is perhaps because the open-set biomedical questions can be ambiguous without constraining their excepted answer options. Meanwhile, evaluation of free-form text prediction for medical VQA remains as an open research problem that we are actively exploring.
Ablation Studies. To study the impact of our curated instruction data and hyper-parameters in the training pipeline, we report the performance of different model variants in Table 4 (a). Several findings are confirmed: (i) LLaVA-Med consistently outperforms LLaVA by a large margin, indicating the effectiveness of our biomedical domain-specific adaptation. The performance gaps on zero-shot are larger than that in fine-tuned settings, showing that LLaVA-Med is clearly a better option than LLaVA when deploying one model for various scenarios in the wild. (ii) Training longer in Stage 1 improves zero-shot transfer, but Stage 1 alone is not sufficient, because the single image captioning
LLaVA-Med Model Variants	VQA-RAD	SLAKE	PathVQA	Average
Instruct	Stage 1	Stage 2	FT	Open	Closed	Open	Closed	Open	Closed
CLIP Vision Encoder [39], 7B Language Model
0	1	0	0	15.27	12.50	18.55	13.46	6.26	13.51	13.26
0	3	0	0	15.33	15.44	23.61	15.38	6.35	14.74	15.14
10K	1	3	0	25.79	57.35	31.50	51.68	8.49	59.66	39.08
10K	3	3	0	28.44	59.56	22.63	43.99	5.40	52.67	35.45
10K	1	3	1	36.39	55.88	71.64	56.49	25.50	82.87	54.79
10K	1	3	3	18.59	55.51	78.60	63.46	34.02	86.94	56.19
60K	1	1	0	29.80	55.15	38.08	50.00	11.70	59.66	40.73
60K	1	3	0	29.67	60.29	35.53	53.85	11.76	53.20	40.72
60K	1	3	1	22.63	58.09	72.75	54.33	24.19	71.60	50.60
60K	1	3	3	54.12	64.71	79.33	64.90	17.18	71.37	58.60
60K-IM	1	1	0	29.67	61.40	38.44	52.40	11.41	56.24	41.59
60K-IM	1	3	0	28.23	61.40	39.17	52.16	12.30	54.05	41.22
60K-IM	1	3	1	28.61	56.25	70.58	54.57	11.17	59.19	46.73
60K-IM	1	3	3	55.50	66.54	80.57	64.18	35.88	89.15	65.30
60K-IM	1	3	9	66.26	80.88	82.30	84.86	37.59	91.54	73.90
60K-IM	1	3	15	61.53	84.19	83.08	85.34	37.95	91.21	73.88
60K-IM	1	3	18	61.37	81.25	84.24	83.17	37.88	91.39	73.22
CLIP Vision Encoder [39], 13B Language Model
60K-IM	1	3	0	31.66	61.40	37.71	49.76	11.34	49.63	40.25
60K-IM	1	3	9	64.58	77.94	84.97	85.58	38.82	92.39	74.05
BioMed CLIP Vision Encoder [51], 7B Language Model
60K-IM	1	3	0	37.84	60.66	39.73	54.33	11.65	49.07	42.21
60K-IM	1	3	9	64.75	83.09	87.11	86.78	39.60	91.09	75.40
LLaVA	0	0	0	20.74	59.19	26.82	50.24	8.74	45.65	35.23
LLaVA	0	0	3	50.00	65.07	78.18	63.22	7.74	63.20	54.57
(a) Ablation studies with varying number of training epochs at different stages. 60K-IM indicates the instruct data generated with inline mentions. The gray rows are zero-shot performance of LLaVA-Med trained with different instruct data, they are selected to show in Table 3.
LLaVA-Med Model Variants	VQA-RAD	SLAKE	PathVQA	Average
Description	Module	Stage 1	Stage 2	FT	Open	Closed	Open	Closed	Open	Closed
Training without Stage-1
N/A	N/A	0	3	0	23.28	62.13	36.50	56.01	7.73	58.12	40.63
Training with Stage-1
Yes	Projection	1	3	0	25.79	57.35	31.50	51.68	8.49	59.66	39.08
Yes	LLM	1	3	0	26.23	52.33	37.62	52.48	9.38	58.88	39.49
No	LLM	1	3	0	26.88	56.16	35.23	55.78	9.39	63.27	40.97
(b) The impact of Stage-1 training. All jobs are initialized with LLaVA. "Description" indicates whether the description text is included in input in Stage-1. "Module" indicates the trainable module in Stage-1. The numbers in green cells show the effectiveness of Stage-1 training, though the average scores are similar.
Table 4: Quantitative results on three established biomedical VQA datasets. "Instruct" is the instruct dataset, "Stage 1" and "Stage 2" provides the number of training epochs for each stage, "FT" is Fine-Tuning.
Stage 1	Stage 2	VQA-RAD	SLAKE	PathVQA
1	3	Instruct	1	3	1	3	1	3	1	3
6.8	19.4	10K	0.6	1.8	0.3	0.6	0.6	1.0	1.0	2.5
60K	2.6	8.0
Table 5: Running time (hours) for 1 and 3-epoch training, with batch size 128 on eight A100 GPUs.
instruction in Stage 1 may encourage the model to lose its ability in follow diverse instructions. (iii) Instruction-following data in Stage 2 is critical, and the performance is generally improved, when the instruct data amount increases from 10K to 60K. The 60K-IM data provides the best averaged zero-shot and fine-tuned performance, respectively, validating the effectiveness of considering inline mention as external knowledge in data creation.  $(iv)$  Fine-tuning longer on downstream datasets till 9 epochs benefits the performance, especially on checkpoints with 3-epoch training in Stage 2. Increasing language model size from 7B to 13B improves the overall zero-shot performance and fine-tuned performance. We suggest practitioners to choose the appropriate quality-cost trade-off, by referring to the running time in Table 5.  $(v)$  When downstream samples are available, fine-tuning itself provides the largest performance gain (54.57-35.23=19.34). However, by training with high quality instruct data such as 60K-IM in the Stage-2, we can further boost performance significantly (65.30-54.57=10.73). Stage-2 itself is not as effective as direct fine-tuning on downstream tasks, meanwhile Stage 1, 2 and fine-tuning are all required for the best performance.  $(vi)$  We study the confidence interval (CI) by running the same experiment configuration three times, and report the standard derivation (std) in Table 10 (b). Note it is infeasible to provide CIs for all experiments due to the large number of jobs. Though the std of averaged results of three datasets are small, there is some evidence that results on one single dataset might be statistically significant. We suggest the users with resource to run multiple jobs using the released code to draw more rigorous conclusions for experiments of interest & importance, to alleviate the limitation from the lack of CIs.
Impact of Stage-1. We consider the more strategies to train Stage-1 in addition to tuning the linear projection layer only. Please see the representative results in Table 4 (b), and the detailed discussed in Section C.2. It yields higher average performance at the early epochs such as epoch 1, when training LLaVA-Med from LLaVA using Stage-2 only, without Stage-1. As the training continues to epoch 3 or more, all training methods perform similarly measured by the average scores. However, training with Stage-1 consistently provides higher performance than training without Stage-1 on the PathVQA dataset, which indicates the Stage-1 can benefit certain biomedical domains, when related additional knowledge is learned. Our suggestions on the necessity of Stage-1 training are  $(i)$  If LLaVA-Med is trained with a customized vision encoder or LLM that are not included in LLaVA (i.e., no LLaVA checkpoint is available), Stage-1 is critical in aligning the multimodal feature space, and yield good performance.  $(ii)$  If LLaVA-Med is trained by initializing from LLaVA, the Stage-1 training is optional. In this case, it is more cost-efficient to skip Stage-1 and train Stage-2 only, which can quickly provide good performance on the vertical domains with less cost. However, for scenarios with a large number of in-domain image-text pairs that pre-trained LLaVA does not have much related knowledge, we suggest adding the Stage-1 training on the in-domain pairs: The best strategy in this case is full-model fine-tuning of the LLM, and removing the instruction text of describing the image.
We present LLaVA-Med, a large language-and-vision model for the biomedical domain. To create this model, we create high-quality biomedical language-image instruction-following dataset using a self-instruct approach to build a data curation pipeline using language-only GPT-4 and external knowledge. LLaVA-Med demonstrates strong excellent chat abilities with domain knowledge, and outperforms previous supervised SoTA on three VQA datasets on certain metrics with subsequent fine-tuning.
While we believe that LLaVA-Med represents a significant step towards building a useful biomedical visual assistant, we note that LLaVA-Med is limited by hallucinations and weak in-depth reasoning common to many LMMs. We discuss the limitations of LLaVA-Med and the utilization of GPT-4 API for data generation pipeline in Section B. Future work is directed toward improving quality and reliability of LLaVA-Med. We hope the LLaVA-Med recipe can inspire the applications of training large language-and-vision assistant to more vertical domains.
[1] Clinical Camel. https://wanglab.ml/clinical_camel.html, 2023. 2
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We build a large-scale multi-modal biomedical instruction tuning dataset.
(b) Did you describe the limitations of your work? [Yes] See "Comparisons with SoTA" and "Conclusions".
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes we discussed the safety and ethical filtering policy of our data generation process in section 3.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read it and our dataset does not contain sensitive items.
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See project website https://aka.ms/llava-med and Appendix for dataset links
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Method & Experiment
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] Error bars are not most applicable in our case, instead we report comprehensive ablation results in section 5.2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See section 5.2
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See https://aka.ms/llava-med.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See project website https://aka.ms/llava-med and Appendix for dataset links.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We follow the copyright of the original data source from PubMedCentral..
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We respect the personally identifiable in the images such as the right to be portrayed, which is the responsibility of the copyright owner.
A Data
A.1 Open-sourced Medical Visual Instruction-Following Datasets
Training. The Stage-1 data follows CC BY NC 4.0 license. As described in Section 3, we create three versions of datasets for our biomedical visual instruction tuning in the 2nd stage.
- 10K: https://hanoverprod.blob.core.windows.net/public.med_llava/finetune_pmc/finetune_postprocess_caption_10k.json
- 60K: https://hanoverprod.blob.core.windows.net/public/med_llava/binetune_pmc/binetune_postprocess)caption cleansed_60k.json
- 60K-IM: https://hanoverprod.blob.core.windows.net/public/med_llava/binetune_pmc/binetune_postprocess)caption_im cleansed_60k.json
Evaluation. As described in Section 5.1, to evaluate the biomedical chat ability, we create an evaluation set.
https://hanoverprod.blob.core.windows.net/public.med_llava/multimodal chat eval/qa_50_images.jsonl
Images. The image url paths can be seen in the files:
- Training:
https://hanoverprod.blob.core.windows.net/public.med_llava/images/finetune_image_url.jsonl
- Evaluation:
https://hanoverprod.blob.core.windows.net/public.med_11ava/images/eval_image_url.jsonl
A.2 Prompts
Instructions for brief image description. The list of instructions used to briefly describe the image content are shown in Table 6. They present the same meaning with natural language variance.
"Describe the image concisely."
- "Provide a brief description of the given image."
- "Offer a succinct explanation of the picture presented."
- "Summarize the visual content of the image."
- "Give a short and clear explanation of the subsequent image."
- "Share a concise interpretation of the image provided."
- "Present a compact description of the photo's key features."
- "Relay a brief, clear account of the picture shown."
- "Render a clear and concise summary of the photo."
- "Write a terse but informative summary of the picture."
- "Create a compact narrative representing the image presented."
Table 6: The list of instructions for brief image description.
Instructions for detailed image description. The list of instructions used to describe the image content in detail are shown in Table 7. They present the same meaning with natural language variance.
Self-instruct prompts. The prompts used to generate medical instruction following data are shown in Figure 4 and Figure 5.
- "Describe the following image in detail"
- "Provide a detailed description of the given image"
"Give an elaborate explanation of the image you see"
- "Share a comprehensive rundown of the presented image"
- "Offer a thorough analysis of the image"
"Explain the various aspects of the image before you"
- "Clarify the contents of the displayed image with great detail"
- "Characterize the image using a well-detailed description"
- "Break down the elements of the image in a detailed manner"
- "Walk through the important details of the image"
- "Portrait the image with a rich, descriptive narrative"
- "Narrate the contents of the image with precision"
- "Analyze the image in a comprehensive and detailed manner"
- "Illustrate the image through a descriptive explanation"
"Examine the image closely and share its details"
- "Write an exhaustive depiction of the given image"
Table 7: The list of instructions for detailed image description.
Prompting GPT-4 to generate medical visual instruction-following data
messages = [ {"role": "system", "content": ""You are an AI assistant specialized in biomedical topics.
You are provided with a text description (Figure Caption) of a figure image from a biomedical research paper. In some cases, you may have additional text (Figure Context) that mentions the image. Unfortunately, you don't have access to the actual image.
Below are requirements for generating the questions and answers in the conversation:
- Avoid quoting or referring to specific facts, terms, abbreviations, dates, numbers, or names, as these may reveal the conversation is based on the text information, rather than the image itself. Focus on the visual aspects of the image that can be inferred without the text information.
- Do not use phrases like "mentioned", "caption", "context" in the conversation. Instead, refer to the information as being "in the image."
- Ensure that questions are diverse and cover a range of visual aspects of the image.
- The conversation should include at least 2-3 turns of questions and answers about the visual aspects of the image.
- Answer responsibly, avoiding overconfidence, and do not provide medical advice or diagnostic information. Encourage the user to consult a healthcare professional for advice."""}\}
]   
for sample in fewshot_samples: messages.append({"role":"user", "content":sample['context']}) messages.append({"role":"assistant", "content":sample['response']})   
messages.append({"role":"user", "content":query})
Figure 4: messages we use to prompt GPT-4 to generate medical visual instruction-following data. Manually curated few-shot examples are included in the prompt, where each example has input sample['context'] and output sample['response']. Please see Figure 5 for one of the few-shot examples.
One of the few-shot examples used in the prompt
Example input:
Figure Caption:
Figure 2: Chest X-ray. Cardiomegaly with diffuse bilateral interstitial infiltrates and a right-sided pleural effusion.
Figure Context:
- His respiratory status continued to worsen and on hospital day number 14, he required intubation and mechanical ventilation for hypoxemic respiratory failure (Figure 2).
Example output:
User: Which side of the lung has a pleural effusion?
Assistant: The right side of the lung has a pleural effusion.
User: According to the CT image, does the patient has cardiomegaly?
Assistant: The image appears to be a chest x-ray. However, it indicates that the patient has cardiomegaly, which means an enlarged heart. Additionally, there are diffuse bilateral interstitial infiltrates and a right-sided pleural effusion present in the image.
User: Additional Info: The patient's respiratory status continued to worsen and on hospital day number 14, requiring intubation and mechanical ventilation for hypoxemic respiratory failure.
Assistant: Thank you for providing additional information about the patient's condition. Based on the information you provided, it appears that the patient's respiratory status continued to worsen and on hospital day number 14, they required intubation and mechanical ventilation for hypoxemic respiratory failure. This suggests that the patient's lung function has significantly declined, and they may be experiencing respiratory distress. It is important for the medical team to closely monitor the patient's condition and provide appropriate treatment to manage their symptoms and improve their lung function.
Figure 5: One of the few-shot examples used in our prompt to construct medical visual instruction-following data. "Figure Context" in the input provides additional context by including sentences from the corresponding PubMed paper that mention the figure.
B More Discussions of LLaVA-Med
B.1 Limitations of LLaVA-Med