# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 1 Introduction  
Visual question answering (VQA) is a process that provides meaningful information from images to a user based on a given question. With the rapid advancements in computer vision (CV) and naturalanguage processing (NLP), medical visual question answering (Med-VQA) has attracted much attention. The Med-VQA model seeks to retrieve accurate answers by fusing clinical visuals and inquiries. The Med-VQA model can aid in medical diagnosis, automatically extract data from medical images, andower medical professionals' training costs. Additionally, the Med-VQA system has many benefits for the medical industry. Here are a few illustrations:  
* **Diagnosis and treatment:** By offering a quick and precise method for analyzing medical images, Med-VQA can help medical practitioners diagnose and treat medical disorders. Healthcare experts canearn more about the patient's condition by inquiring about medical imaging (such as X-rays, CT scans, and MRI scans), which can aid in making a diagnosis and selecting the best course of therapy. In addition, it wouldreduce the doctors' efforts byetting the patients obtain answers to the most frequent questions about their images. * **Medical education:** By giving users a way toearn from medical images, Med-VQA can be used to instruct medical students and healthcare workers. Students canearn how to assess and understand medical images--a crucial ability in the area of medicine--by posing questions about them. * **Patient education:** By allowing patients to ask questions about their medical photos, Med-VQA can help them better comprehend their medical issues. Healthcare practitioners can improve patient outcomes by assisting patients in understanding their problems and available treatments by responding to their inquiries regarding their medical photos. * **Research:** argc collections of medical photos can be analyzed using Med-VQA to glean insights that can be applied to medical research. Researchers can better comprehend medical issues and create new remedies by posing questions regarding medical imaging and examining the results. Although much Med-VQA research has been accomplished, it requires much enhancement due to public data and data sizeimitations to achieve practical usage. There are a few public data available: the VQA-RAD [1], VQA-Med 2019 [2], VQA-MED 2020 [3], SLAKE dataset [4], and Diabetic Macular Edema (DME) dataset [5]. Only the VQA-RAD and SLAKE datasets are manually generated and validated by clinicians. In addition, they have more question diversity among all medical VQA datasets. The DME dataset is manually generated but not validated by specialists. Several models have been developed to solve this problem. These models rely on four types of methods: joint embedding approaches [6; 7; 8], attention mechanisms [9; 10; 11; 12; 13], composition models [14; 15; 16; 17; 18], and knowledge base-enhanced approaches [19; 20; 21; 1]. In Med-VQA, VGGNet [22], ResNet [23], and the ensemble of vision pre-trained models [24; 25] are the vision features extraction methods widely used, whileSTM [26], Bi-LSTM [27], and BERT [28] are the text features extraction mainly utilized.ately, most models have aimed to use attention mechanisms to align between the text and image features [10; 29; 30; 31]. In addition, vision andanguage (V + L) pre-trained models, such as visualBERT [32], VilBERT [33], UNITER [34], and CLIP [35]. Researchers claimed that Med-VQA requires more text information about images to facilitate the classification task for the model. Therefore, they utilize image captioning generation to give the model extra information about the image [36]. Regarding the state-of-the-art (SOTA), the multi-model structure's complex has been increased to obtain better performance than previous models. This is because deepearning is a black-box method and the VQA multi-model has three main parts: text phase, vision, phase, and fusion, in which no researchers have proven that a method used in each part is the most significant in VQA behavior.