LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 16:59:31

Abstract: Multimodal Large Language Model (MLLM) has recently garnered significant attention as a prominent research focus. By harnessing the capability of powerful Large Language Model (LLM), it facilitates the transition of conversational generative AI from unimodal text to performing multimodal tasks. This blooming development has begun to significantly impact the medical field. However, visual language models in the general domain lack sophisticated comprehension required for medical visual conversations. Even some models specifically tailored for the medical domain often produce answers that tend to be vague and weakly related to the visual contents. In this paper, we propose a fine-grained and adaptive visual language model architecture for Chinese medical visual conversations through parameter-efficient tuning. Specifically, we devise a fusion module with fine-grained vision encoders to achieve enhancement for subtle medical visual semantics. Then we note data redundancy that is common in medical scenes but ignored in most prior works. In cases of a single text paired with multiple figures, we utilize weighted scoring with knowledge distillation to adaptively screen valid images mirroring text descriptions. For execution, we leverage a large-scale Chinese ultrasound multimodal dataset obtained first-hand from the hospital database. We create instruction-following data based on text derived from doctors, which ensures professionality and thus contributes to effective tuning. With enhanced architecture and quality data, our Large Chinese Language and Vision Assistant for Ultrasound (LLaVA-Ultra) shows strong capability and robustness to medical scenarios. On three medical visual question answering datasets, LLaVA-Ultra surpasses previous state-of-the-art models on various metrics.

Main paper: 
 related work: 
 Research of Multimodal Large Language Models (MLLMs) [4, 41, 45] is an emerging hotspot currently. The key concept is to leverage pretrained Large Language Models (LLMs) to incorporate information from other modalities, such as vision, for performing multimodal tasks like video understanding [31, 51] and embodied agent [37, [53] [54] [55] . Remarkable works such as BLIP-2 [18] , LLaVA [21] and LLAMA-Adpater [48] demonstrate prominent generative capabilities including visual question answering and image captioning [8, 9] . These advancements have expanded the research of AI into new fields and hold promising prospects for further development [22] . Medical multimodal chatbots. Inspired by the aforementioned successes in the general domain, explorations into MLLMs have progressively transitioned into the medical domain, exemplified by models like LLaVA-Med [17] and Med-PaLM M [35] . They utilize medical datasets to perform instruction tuning for MLLMs initialized from the general domain and thus create potential applications in medical scenarios, such as medical VQA. For instance, LLAVA-Med utilizes instruction-tuning data [27] generated from the PMC-15M [49] dataset to train a multimodal medical chatbot. Consequently, it extends LLaVA to the medical domain.Although LLaVA-Med and our proposed LLaVA-Ultra share a similar base model LLaVA, they differ significantly: (𝑖) Model architecture. LLaVA-Med is based on the base model LLaVA without significant modifications. However, we propose effective enhancements to the model structure to adapt the characteristics of medical data. It primarily focuses on improving comprehension of subtle visual semantics and adapting to data redundancy in medical scenarios. Thus, our LLaVA-Ultra improves effectiveness and robustness in medical applications. (𝑖𝑖) Data source and characteristic. LLaVA-Med relies on rougher internet-based data PMC-15M [49] , whereas our model utilizes professional detailed data first-hand sourced from the hospital. It incorporates data redundancy and data similarity and is more relevant to real-world healthcare scenarios, placing higher requirements on modeling capabilities than the former. LLaVA-Med's data is in English, while ours is in Chinese. Other commonly used datasets such as MedICaT [32] , ROCO [26] and MIMIC-CXR [14] exhibit similar limitations as above.Chinese medical chatbots. Medical assistants are emerging in the Chinese language field as well. Notably, BenTsao [38] integrates Chinese medical knowledge bases into both the training and inference phases of LLMs, culminating in the development of a medical chatbot in the Chinese language. This approach mirrors the construction method adopted by most existing Chinese medical chatbots, such as MedicalGPT [44] , HuatuoGPT [47] , DoctorGLM [43] and XrayGLM [40] . However, to the best of our knowledge, XrayGLM stands as the only existing multimodal Chinese medical chatbot capable of processing image inputs. Nevertheless, it relies on relatively coarse data and primarily focuses on chest X-rays, thus lacking the breadth and diversity of data. Meanwhile, the presented model response examples are diagnostics mainly for images without lesions, lacking results for diverse medical images. And it fails to produce more detailed answers. In contrast, our proposed model, LLaVA-Ultra, offers substantial enhancements in these respects. 
multimodal conversational model in chinese medical domain 3.1 ultrasound concept feature alignment: 
 We employ a network architecture that is based on the multimodal conversation model LLaVA, incorporating a projection module to link the visual encoder with the language model. The model parameters are initialized with weights from LLaVA-Med in the English medical domain, followed by finetuning [39] using medical domain instructions derived from our Chinese ultrasound dataset. For each paired sample, given textual instructions 𝑋 𝑞 and image inputs 𝑋 𝑣 , we ask the model to produce answers 𝑋 𝑎 related to the original caption 𝑋 𝑐 , such as giving a diagnosis or describing visual contents. The instruction tuning process can be formulated as follows,EQUATIONwhere 𝜃 is the network parameters to be optimized. Following the approach of LLaVA-Med, we conduct training in two stages. The first stage involves aligning ultrasound images with corresponding medical concepts using simple instructions. In the second stage, we utilize diverse instructions generated by GPT-3.5 [23] to enable our model to address free-form conversations. During training, we freeze the visual encoder and most of the LLM weights and update the parameters of the projection layers, LoRA [10] , and our designed modules for enhancement. Through iterative optimization, the model assimilates a substantial volume of new ultrasound visual information and aligns it with medical textual concepts. As a result, it can serve as an ultrasound visual chatbot.Most existing MLLMs utilize the CLIP series [7, 29, 33] as visual modules, incorporating features from deep layers that represent the global context as inputs to the LLM. However, it may result in less fine-grained visual perception in MLLMs. Additionally, the improvement of MLLMs is hindered by the limitations of the visual branch [19, 30, 42] , primarily due to the unbalanced scales of visual and language models (e.g. ViT-Large-300M vs. LAMA-7B/13B) [13] . Therefore, there is a need for visual enhancement in MLLMs, especially in the medical domain where image information is subtle. To address this issue, we integrate the Segment Anything Model (SAM) [16] , effective in capturing finer-grained features, as an extra visual encoder and further incorporate it through a fusion strategy, as shown in Fig. 2 . Specifically, the input images undergo processing by both the CLIP and SAM (ViT-Large-based) encoders 𝐹 1 , 𝐹 2 to derive visual features and then align them with the linguistic feature space of the LLM through corresponding projection modules 𝑃 𝜃 1 , 𝑃 𝜃 2 . We combine the resultant features 𝐻 1 , 𝐻 2 using a learnable weight parameter 𝛼 for feature fusion as follows,𝐻 1 = 𝑃 𝜃 1 (𝐹 1 (𝑋 𝑣 )), 𝐻 2 = 𝑃 𝜃 2 (𝐹 2 (𝑋 𝑣 )), 𝐻 𝑣 = 𝛼 • 𝐻 1 + (1 -𝛼) • 𝐻 2 .(EQUATIONIt enables an appropriate balance between the two typical visual features. The fused feature 𝐻 𝑣 enriches with detailed local information, such as the texture of lesion areas. It is subsequently concatenated with the instruction tokens to serve as input for the LLM, thereby enhancing the fine-grained visual perception of the MLLM. Discussion. (𝑖) Medical domain adaptation. This is particularly crucial for medical images, where features like lesion areas are often more subtle compared to natural images. (𝑖𝑖) Parameterefficient scheme. It validates the feasibility of visual enhancement in a parameter-efficient way for multiple frozen visual encoders. (𝑖𝑖𝑖) Model extension. In addition to SAM, it can be generalized to explore the application value of other vision models for MLLMs.Data redundancy is commonly encountered in clinical scenes, where there is a group of images corresponding to the same text but only some images are valid. To effectively deal with this scenario while balancing computational cost, we devise an adaption module. For such a paired instance, we calculate the weight scores of the grouped 𝑘 images based on the features obtained from the visual-language projection. Then we sample the image with the highest score as the valid one that best matches the specifics of the text. Specifically, we design two adaptive sampling strategies, as shown in Fig. 2: (𝑎) Feature scoring strategy. We use the projected image feature 𝐻 𝑣 as the weight score, as the optimization of the projection modules is related to the image-text alignment during training. Since 𝐻 𝑣 contains multiple tokens ℎ, each focusing on different content, we avoid treating them equally, such as by simply summing or averaging. Instead, we employ a set of learnable parameters 𝑤 to calculate a weighted average as a score. And we sample the image with the highest score as the valid one to utilize in this group, i.e., where 𝑑 is the number of channels in feature 𝐻 𝑣 . Thus, these weights can be progressively optimized during training to focus on tokens that are more expressive of the relevance of the image to the text.EQUATION(𝑏) Attention scoring strategy. The strategy above indirectly learns to align during training. However, in our dataset, each text instance contains sufficient information to directly match the text with the most suitable image by comparing the text with each image's features. For instance, if the instruction's question pertains to the diagnosis, we can utilize the descriptive textual ultrasound findings. We leverage the first 𝑁 LLM layers to perform cross attention in Visual Transformer (ViT) [6, 36] of each image feature in the paired instance and the same text of ultrasound findings 𝐻 𝑡 , i.e.,EQUATIONThe obtained scores directly reflect the relevance of each image to this same text. Considering the absence of additional textual knowledge during inference, we treat these attention scores as pseudo labels and calculate the cross-entropy loss with feature scores 𝑠 𝑓 𝑒𝑎 above after normalization, i.e.,EQUATIONBy optimizing the weight parameter 𝑤 with this knowledge, we can better prioritize tokens and thus screen the image that strongly correlates with the text. Discussion. (𝑖) Fitting medical scenarios. Our method leverages redundant medical data, which is highly relevant and practical for physicians' diagnostic processes in real-world scenarios. (𝑖𝑖) Computational efficiency. Instead of captioning all images corresponding to the same text and selecting the most accurate results, we offer a computationally low-cost approach. We directly select feature scores or use a small-scale attention module to screen effective images before feeding them into the LLM. Even with the employment of the redundancy adaptation module and the additional visual encoder mentioned above, it only takes 60 hours on 4 48G A40s for training. (𝑖𝑖𝑖) Data utilization. In the second screening method above, we leverage the textual information of ultrasound observations, a previously untapped resource. It often directly reflects the image's information more intuitively than diagnostic results, thus better identifying valid images. Additionally, the ultrasound observations text and the subsequent diagnostic results input into the LLM are strongly interconnected. The former serves as the surface-level description, while the latter offers a summary and in-depth characterization of the former. This intrinsic relationship facilitates the extraction of more detailed and profound medical semantics during learning. (𝑖𝑣) Domain adaptation. Although the focus of this paper is on the ultrasound domain, it can be generalized to other medical imaging modalities, such as CT, CXR, and MRI. There are also requirements for fine-grained analysis and cases of data redundancy in these domains. It is possible to utilize features and knowledge of these domains to construct the corresponding assistants similar to our proposed method. 
professional ultrasound multi-modal data 4.1 ultrasound multi-modal data: 
 There is a lack of Chinese medical datasets to perform finetuning for MLLMs. To fill this gap, we present a first attempt to utilize a large-scale Chinese multimodal ultrasound hospital dataset and it has the noteworthy following characteristics: (𝑖) First-hand source and diversity. Our dataset is directly sourced from the hospital database. It consists of over 20k medical text descriptions paired with 170k ultrasound images with more than 20 examination sites, such as heart, thyroid, breast, uterus, prostate, etc. Such large-scale first-hand Chinese medical data has rarely been achieved in previous works. (𝑖𝑖) Professionality. It contains comprehensive and detailed clinical text such as examination sites, medical histories, ultrasound observations, diagnosis, etc. Professional doctors offer all the content, ensuring data reliability, which is rarely realized in prior datasets. It makes our work a valuable contribution to applying clinical information to scientific research. (𝑖𝑖𝑖) Challenges within ultrasound modality. Medical imaging modalities used in existing models typically include chest X-ray (CXR), computed tomography (CT), and magnetic resonance imaging (MRI). It's relatively possible for even a layman to identify the body parts in these images, such as the head and chest. However, when confronted with ultrasound, its imaging characteristics make it difficult for non-physicians to achieve this task. This inherent professional barrier poses a greater challenge for MLLMs to learn medical semantics like a layman. (𝑖𝑣) Fine granularity. Our data contains many samples with high similarity due to the hospital source, a feature frequently absent in existing medical datasets. Thus, it places a higher requirement on fine-grained medical understanding. (𝑣) Respect for medical reality. Datasets like PMC-15M used by LLaVA-Med typically feature paired instances where a single image corresponds to a text. However, it is often inconsistent with clinical practice where data redundancy exists. Our first-hand dataset addresses this by capturing many instances where one text is paired with multiple images stemming from frames in the same ultrasound video. For example, when the text describes a lesion, only images of frames scanned to the lesion are considered valid for mirroring the text, while those without lesion display are invalid. Yet it lacks intuitive labels for validity determination. Hence, it challenges the model to distinguish valid images for reasoning and holds practical relevance.Inspired by LLaVA-Med, we generate Chinese ultrasound instructionfollowing data as Fig. 3 . For one caption 𝑋 c paired with 𝑘 images 𝑋 v , we create an instruction-following example with question 𝑋 q :Human : 𝑋 q 𝑋 1 v ... 𝑋 𝑘 v <STOP>\n Assistant : 𝑋 c <STOP>\n Due to the diverse types of captions and sampled questions, the instructions require answers related to examination sites, ultrasound observations, or medical diagnoses. To validate data rationality, we produce two versions of instruction data: (𝑖) Our ultrasound dataset that considers examination sites as cues in questions. (𝑖𝑖) A dataset of similar size without sites mentioned in questions, while expecting the model to state it in answers. They are utilized in experiments to evaluate their impact on trained LLaVA-Ultra. Source: hospital Language: Chinese Figure 3 : An example of our GPT-3.5 [23] generated instruction-following data. Top: A professional multimodal instance from our Chinese ultrasound hospital dataset. There exists data redundancy where a text corresponds to multiple images, but only those mirroring textual descriptions are valid (e.g., display lesions mentioned in the text). Bottom: The instruction-following data generated by GPT-3.5 using the textual descriptions.Examination 
experiments 5.1 implementation details: 
 Datasets. Besides our large-scale hospital ultrasound dataset, we evaluate models on two open-source medical visual question and answer (Med-VQA) datasets, as illustrated in Tab. 1: (𝑖) SLAKE [20] is an English-Chinese bilingual dataset and contains 642 images and over 7000 Q&A pairs covering 12 diseases and 39 organs, with CT, MRI, and X-Ray imaging modalities. The Q&As span various topics such as diagnosis, anatomical structure, and lesion location. Presently, SLAKE serves as a crucial benchmark for VQA tasks in the medical domain for its diversity. It has been utilized for evaluation purposes in significant medical multimodal large language model works, such as LLaVA-Med, Med PaLM M, and PMC-VQA. (𝑖𝑖) OpenI [5, 12] is a chest radiograph dataset from Indiana University Hospital. XrayGLM [40] has preprocessed these unstructured data and translated the English reports into Chinese using ChatGPT. This can provide support in the case of insufficient open-source Chinese multimodal medical data. Our experiments adopt this obtained set comprising 6,436 images and 3,218 reports.Evaluation Metrics. Based on LLaVA-Med's evaluation metrics, we've made further improvements. We tokenize model-generated answers and ground-truth separately, then calculate metrics. For open-set questions, we report the Exact Match (EM) score, F1 score, Precision, Recall and Bilingual Evaluation Understudy (BLEU) score [25] . Notably, BLEU score utilizes precise matching of 4-grams of sequences to evaluate text generation results. We adjust the weights of n-grams and obtain the scores with 1-gram, 2-gram, 3-gram and the uniform set for comprehensive evaluation in accuracy and fluency. In SLAKE, we extra report LLaVA-Ultra's results in CT, MRI, and X-Ray subsets and utilize accuracy scores for the closed-set.Experiment Details. LLaVA-Ultra is initialized with weights from LLaVA-Med in modules included in the latter. It utilizes CLIP-ViT-L/14 and extra SAM-ViT-L as visual encoders and LLaMA-13B as LLM. We use the linear projection for multimodal connection. To ensure a fair comparison, we maintain parameters of compared baseline models in accordance with the respective papers or codes and train until loss functions converge. Data preprocessing ways are also the same. Experiments employ 4 48GB NVIDIA A40s with PyTorch. Models are optimized by Adam [15] with learning rate 1𝑒 -3 and batch size 16. We provide the comparisons between LLaVA, LLaVA-Med, and our LLaVA-Ultra when employed as a medical visual chatbot.For qualitative comparisons shown in Fig. 4 , LLaVA model for the general domain struggles with medical tasks, highlighting the gap between domains. While tailored for the medical domain, LLaVA-Med still inadequately addresses Chinese ultrasound scenes. It often focuses solely on textual medical concepts in the questions and offers vague and invalid answers that do not meet the questioner's needs. They tend to rely more on medical knowledge learned early from the LLMs rather than effectively incorporating medical visual features. This results in responses that display weak correlations with the input images and even exhibit inaccuracy. It implies the flaws in their model structures. In contrast, our LLaVA-Ultra model The liver was full in shape, with a smooth envelope, and a cystic dark area was seen in the liver, about 0.70 cm in size, with a thin and smooth wall, good internal translucency, and enhanced posterior echogenicity; the rest of the parenchymal echogenicity was enhanced, dense, and poorly distributed, and the vascular network was poorly defined. The left and right intrahepatic bile ducts were not significantly dilated. The shape and size of the gallbladder were normal, the wall of the gallbladder was smooth without thickening, the bile in the lumen was well transmissible, No abnormal echoes were seen inside the gallbladder. The internal diameter of the common bile duct was not significantly widened, and there was no obvious abnormality in the lumen. The spleen was normal in shape and size, and the echoes in the spleen were uniform and small. The pancreas was normal in shape and size, with homogeneous internal echogenicity, no obvious dilatation of the main pancreatic duct, and no obvious foci of occupancy.This ultrasound is from color Doppler ultrasound, left heart function measurement, ventricular wall motion analysis, tissue Doppler imaging This ultrasound is from an ultrasound scan of the heart. This ultrasound image is from a heart exam.It is from ultrasound of liver, gallbladder, pancreas and spleen . It is from a pregnancy test, specifically a fetal ultrasound. This ultrasound is from the stomach.The heart had normal internal diameters in all atria, normal internal diameter of the left atrium, and no significant abnormalities in the morphology, echogenicity, or activity of the aortic and pulmonary valves. The internal diameter of each atrium was normal. The morphology, echo, and activity of the mitral and tricuspid valves were not significantly abnormal. The thickness of the interventricular septum and the posterior wall of the left ventricle was normal with retrograde motion. No abnormal left ventricular wall activity was seen, EF: 65%. There was no continuous interruption of the cardiac structures, and no obvious liquid dark areas were seen in the pericardial cavity. CDFI:A small amount of regurgitant signal was seen in the tricuspid orifice and mitral orifice. TDI:Diastolic E/A at the level of the mitral annulus was <1.0. . The shape and size of the liver were within normal range, the peritoneum was smooth . A cystic dark area was seen in the liver, the vascular network was not clear, and the left and right intrahepatic bile ducts were not obviously dilated. The shape and size of the gallbladder were normal, the wall of the gallbladder was smooth without thickening, the bile in the lumen was well transmissible, and no obvious abnormal echoes were seen in the gallbladder. The internal diameter of the common bile duct was not significantly widened, and there was no obvious abnormality in the lumen. The spleen was normal in shape and size, and the echoes in the spleen were uniform and small. The pancreas was normal in shape and size, with homogeneous internal echoes, no obvious dilatation of the main pancreatic duct.The image is an ultrasound image of the liver showing its different lobes in what appears to be a three-dimensional view of . This type ultrasound scan provides detailed information about the structure and function of the liver and can be used for diagnostic purposes. The image does not show any abnormalities in the liver or its blood vessels, which indicates that the liver appears to be healthy. The presence of the gallbladder suggests that the bile ducts are functioning properly and the overall image provides a clear view of the normal anatomy and physiology of the liver.. This diagram of a liver, gallbladder, pancreas, and spleen ultrasound shows some medical information, including the shape and location of the liver, gallbladder, pancreas, and spleen. This information can help doctors diagnose and treat a patient's disease, as well as assess a patient's health.Mild tricuspid, mitral, and aortic regurgitation A color image showing the left ventricle of the heart, part of the electrocardiogram, along with a fetal heartbeat monitor. This indicates that the image is relevant for medical diagnosis and heart health monitoring. The electrical activity of the heart is displayed in a polar plot.The medical diagnosis of this diagram of cardiac color Doppler ultrasound, left heart function measurements, ventricular wall motion analysis, and tissue Doppler imaging may be to assess the function and health of the heart. Doppler ultrasound is a noninvasive cardiac test that provides information about the structure of the heart. Left heart function measurement measures function of heart's circulation to assess normalcy and function of the heart.The image is an ultrasound scan of a fetal examination, showing the fetal pole, placenta, and umbilical cord.. Based on this picture of liver, gallbladder, pancreas, and spleen ultrasound, the medical diagnosis may include the following: 1. Liver: possible hepatitis, cirrhosis, and hepatic fibrosis. 2. gallbladder: possible cholecystitis, gallbladder stones, and other diseases. 3. pancreas: possible pancreatitis, pancreatic cancer, and other diseases. 4. spleen: possible splenitis, splenic carcinoma, and other diseases. It should be noted that this is only a preliminary medical diagnosis. . 2 and Fig. 5 , which may benefit from its textual knowledge acquired during LLM initialization. This indicates that its training on the Chinese dataset does not significantly compromise this part of English knowledge [46] . We also report results in CT, MRI, and X-Ray subsets on SLAKE besides the overall average values. Our model pretrained on ultrasound modality shows effective adaptability across diverse imaging modalities. It highlights LLaVA-Ultra's robustness and versatility in accomplishing downstream multimodal tasks after simple finetuning.To assess the validity of the model's components, we conducted comparative experiments on several aspects shown in Tab. 3. Visual enhancement. To prove the necessity of visual enhancement, we replace our dual visual encoders and their fusion module with the original single CLIP encoder and observe the decrease of all metrics in Tab. 3. This verifies the necessity of strengthening the visual branch of MLLMs and underlines the effectiveness of our feature fusion strategy. The integration of SAM features enables LLaVA-Ultra to extract finer-grained visual semantics, which is a crucial aspect of handling subtle information in medical scenarios.Data redundancy adaptation. We remove our data redundancy adaptation module, the metrics in Table . 3 show a significant reduction. This highlights the importance of addressing data redundancy, which is prevalent in real medical scenarios but less noticed. In previous works, when multiple images correspond to the same text, such text is often assigned to each image. It results in mapping the images that don't mirror the specific text, i.e. redundant data, and the valid images to a similar feature representation. This hinders the model from learning accurate medical semantics and cross-modal alignment. As shown in the scores, LLaVA-Ultra full model can address this issue effectively with our adaption strategy. Specifically, our attention scoring strategy (Func.b) leverages rich textual data for feature alignment and thus achieves better scores compared to the simpler feature scoring (Func.a).Data construction. We modified the instruction data by removing the cues of the examination site from the question. Instead, we ask a generalized question like "give the diagnosis of this ultrasound image". Results in Tab. 3 indicate a slight decrease in evaluation metrics within acceptable limits. This demonstrates the effectiveness of limited cues for the model to learn specific medical knowledge, as well as the robustness of our model structure.Although LLaVA-Ultra shows impressive capabilities in Chinese medical multimodal understanding, it still has some limitations, including: 1) Its performance is hindered by the scale of the pretrained vision models. 2) Our large-scale medical dataset has not yet included more comprehensive labels e.g., segmentation to allow our model to engage further enhancement in visual perception. 
conclusion: 
 We propose LLaVA-Ultra, a large Chinese language and vision assistant for the ultrasound domain. To achieve this, we create a high-quality visual-language instruction-following dataset from the large-scale professional ultrasound database of hospitals. More importantly, we improve the conventional visual language model structure by performing visual enhancement and data redundancy adaptation. It enables LLaVA-Ultra to fit the needs of fine-grained medical information and practical clinical scenarios, thus producing high-quality responses in medical visual conversations. On three medical VQA datasets, LLaVA-Ultra outperforms previous SOTA in various metrics, demonstrating its effectiveness and robustness.  
: 
 self-supervised finetuning with instruction-following data, pretrained Large Language Models (LLMs) [3, 34, 52 ] can be adapted to unseen tasks, resulting in improved zero-shot performance. This simple yet powerful method has extended to the multimodal field. It gives rise to Multimodal Large Language Models (MLLMs) [1, 11, 45] that excel in visual language tasks such as image reasoning and further forms user-oriented multimodal conversation assistants.Based on the success of MLLMs in the general domain, similar initiatives have emerged for medical applications [2, 28, 50] . Current research is gradually transitioning from unimodal text to incorporating multimodal medical data. However, despite their effectiveness in general tasks, MLLMs often struggle in medical contexts. This results in inaccuracy or refusal to provide answers when faced with medical questions. It is partially due to the scarcity and difficulty in accessing parallel medical image-text data, unlike the diverse internet data available for the general domain. LLaVA-Med [17] alleviates this issue to some extent by creating multimodal medical instruction-following data for finetuning. However, it still struggles to provide more detailed correct responses. Sometimes its answers tend to be vague, focusing more on medical concepts in textual questions rather than in-depth image analysis. From the data perspective, it utilizes image-text data extracted from PubMed public papers, which may be coarser and less cross-modal matched than those from primary sources. Notably, it ignores data redundancy common in clinics shown in Fig. 1 as many other previous works. This puts a need for model enhancement and adaption for the fine-grained medical domain. Additionally, there is little exploration of extensive Chinese data in the medical multimodal field.In this paper, we propose a Large Chinese Language and Vision Assistant for Ultrasound (LLaVA-Ultra), an end-to-end trained medical multimodal chatbot. To our knowledge, it is the first attempt to extend multimodal finetuning to the Chinese medical domain based on a large-scale dataset. Its domain-specific pretraining has shown effectiveness for medical vision-language (VL) tasks. Specifically, our paper makes the following contributions:• Enhancement for medical adaptation. To meet the needs of subtle medical images, we leverage an extra fine-grained Segment Anything Model (SAM) [16] encoder to jointly extract visual semantics with the CLIP encoder. A followed fusion module can effectively integrate these two typical features and thus achieve visual enhancement for better multimodal alignment. Moreover, for data redundancy common in medical scenarios, we design an adaptive sampling module with weighted scoring and knowledge distillation to automatically screen valid information. It improves the model's robustness and thus ensures correct responses in complicated practical medical scenarios. • High-quality medical parallel data. We present a novel data sourcing pipeline to collect a large-scale Chinese ultrasound multimodal dataset first-hand from the hospital database. It covers professional content provided by doctors for ultrasound examinations of many body parts. We sample pairs of around 170k ultrasound images and 20k clinical texts and generate multi-modal instructions with GPT-3.5 [23] for medical instruction-tuning. • LLaVA-Ultra. Contributed by the robust architecture as well as fine-grained professional data, LLaVA-Ultra shows the best practice in the Chinese medical domain. Trained in only 60 hours with 4 48GB A40s, it provides detailed answers relevant to visual content in medical conversations. On our ultrasound hospital dataset and public medical visual question answering (VQA) datasets, LLaVA-Ultra outperforms prior state-of-the-art (SOTA). 
 
 Appendix:  


Decision: Thanks for submitting the paper to ACM Multimedia. The reviewers felt that the application area was very interesting and the new approach and the considered dataset can spur future growth in this research space. The reviewers also had questions about the validation approach. Many of the reviewer concerns were allayed with the rebuttal. The paper is inherently multimodal and will generate interest at the conference. An acceptance is recommended.

Reviews: review: paper_summary: paper_summary: The paper presents LLaVA-Ultra, a multimodal large language model specifically designed for Chinese medical visual conversations, particularly focusing on ultrasound images. This model aims to address the limitations of current visual language models in the medical domain by providing more precise and relevant responses to medical visual queries. The model utilizes a fusion module with fine-grained vision encoders and adapts to data redundancy in medical scenarios through weighted scoring and knowledge distillation. The authors demonstrate the effectiveness of LLaVA-Ultra on three medical visual question-answering (VQA) datasets, demonstrating comparable or superior performance to existing models.
strength_weakness: strength_weakness: 1. The paper is written clearly and is easy to understand.
2. The research addresses a crucial gap in the field of medical visual language models, specifically in the Chinese medical domain. By developing a model that can accurately interpret and respond to visual medical data, this work has the potential to significantly impact medical diagnostics and education. 
3. The authors conducted experiments on three medical visual question-answering datasets, including a large-scale Chinese ultrasound dataset sourced directly from hospitals. The results consistently demonstrate the model’s superior performance compared to previous state-of-the-art models, validating the effectiveness of their approach.
limitations: limitations: 1. During model training, both CLIP encoder and SAM encoder are frozen. Since both models are trained on natural images rather than medical images, will this affect model performance?

2. There are too few comparison methods in the experiment. It is recommended to add the comparison with other medical MLLMs, such as Med-PaLM

3. The results in bold in Table.2 are a bit confusing. It would be helpful to clarify the criteria for highlighting these results.

4. The BLEU indicator is a relatively mechanical comparison of the accuracy of n-grams, which is not suitable for evaluating the generation results of LLM. It is recommended to add the evaluation results of metrics such as Meteor and CIDEr.

5. Has LLaVA-Med been fine-tuned on the US-Hospital Chinese dataset? If it has not been fine-tuned, the comparison seems a bit unfair. In other words, how to prove that the model structure of LLAVA-Ultra is better than LLaVA-Med?
paper_summary: paper_summary: This article presents a Large Chinese Language and Vision Assistant for question answering on a Chinese ultrasound medical dataset. The innovation lies in integrating a Segment Anything Model into the visual encoder. In cases where a single text is paired with multiple figures, they employ weighted scoring with knowledge distillation to adaptively screen valid images reflecting text descriptions.
strength_weakness: strength_weakness: - The article addresses a medical visual question-answering task on an untapped Chinese dataset.
- The model shows partial superiority in Chinese performance compared to SOTA.
limitations: limitations: - The article only shows partial superiority in Chinese performance compared to SOTA; in Table 2, it does not surpass SOTA on English datasets.
- The article does not utilize semantics-based evaluation metrics.
- As the instructions are generated by GPT, I wonder about the difficulty of converting the whole dataset into English. 
- Which module of your approach is adapted to Chinese?
paper_summary: paper_summary: This paper present LLaVA-Ultra, a LVM assistant for ultrasound understanding. LLaVA-Ultra adopts similar methods to LLaVA to bring the ultrasound images into training but also introduces new domain-specific modules such as the redundancy adaptation. The model was trained by 170k ultrasound images paired with 20k clinical texts from an in-house dataset, which is further processed by GPT-3.5 to produce instruction-following conversations like LLaVA-Med. The proposed model showed improved performance on several benchmarks and the authors also presented interesting analyses comparing different models.
strength_weakness: strength_weakness: 1. Overall, multi-modal learning for medical images is an interesting and important task.
2. A new model, LLaVA-Ultra, is proposed with some novel domain-specific components in addition to the vanilla LLaVA.
3. A potentially useful dataset is also contributed.
4. Writing is mostly clear.
limitations: limitations: 1. The evaluations only use automatic metrics. However, they might not correlate well with expert evaluations. As such, the authors should conduct larger-scale human evaluation on the new model.
2. It is not clear whether the dataset will be publicly available.
3. The compared baselines are pretty basic without proper referring to other studies using these datasets. For example, https://dl.acm.org/doi/abs/10.1145/3581783.3611830 should also be mentioned or compared.
4. The authors should discuss more on how a model trained by ultrasound images is supposed to perform well on medical VQA datasets involving other modalities (e.g., X-ray).
5. It seems suboptimal to use a fixed CLIP/SAM encoder. Have you tried continuing pre-training one encoder using your paired data with contrastive learning?