# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 1 Introduction
For example, utilizing VGGNet to obtain vision features in a model achieved high performance, but it does not guarantee that another model with the same structure will achieve as high performance as the first model's due to the difference in the hyper-parameters configuration. Medical VQA is still at an early stage and needs more exploration to develop efficient models for practical application, which requires a profound understanding of theimitations in this field. A big issue in Med-VQAies in dataimitation. Although most medical VQA researchers employed a CNN pre-trained vision model to solve the dataimitation, this does not reduce the variation between the feature scales of visual entities with a high pixel resolution and the text words. Using transformers for vision manipulates the images as text and reduces this variation [37]. Therefore, this paper uses the transformer for the text and vision feature extraction process. In addition, text transformers, such as BERT and BioBERT, are widely used for textual feature extraction, but the ELECTA transformer (Efficientlyearning an Encoder that Classifies Token Replacements Accurately), which outperforms the SOTA text transformer, is not utilized. Various researchers utilize ensembleearning to obtain the efficiency of different models. [PERSON] et al. [38] designed another method called model soup, which has three types: uniform, greedy, andearnable. The greedy soup achieved the highest performance among the three types. The authors discovered that the SOUPS architecture offers several advantages over traditional ensemble models, including improved accuracy, generalization, interpretability, and more efficient computation. The computation cost for ensemble \(k\) models is \(O(k)\), whereas the computation cost of greedy soup is \(O(1)\). Therefore, in this paper, the proposed model uses the greedy soup method to obtain those advantages. Overall, the contributions of this paper can be summarized as follows:  
* Since the ELECTRA-base transformer that is pre-trained on aarge corpus of text data using contrastive estimation and outperforms other text transformers, such as BERT and BioBERT, is not utilized yet in medical VQA for the textual feature extraction, the proposed model exploits the ELECTRA-base transformer to extract text features from the question. * The proposed model aims to solve the issue of thearge feature variation between the question and the image. Therefore, it exploits theast vision transformer, swin-base-patch4-window7-224. The SWIN transformer utilized in this model to extract visual features from the image is used for the first time in medical VQA. * The extracted visual and textual features are combined and fed into an MLP for classification. * The proposed model is fine-tuned based on the model parameters that achieved the highest validation accuracy and the greedy soup approach to show the significant impact of the greedy soup method. * The performance of theast two models are compared with the model by [PERSON] et al. [5], which we denoted as the SOTA because it is the unique research on this dataset. * The model based on the greedy soup fine-tuning technique is optimized to select the bestearning rate and batch size. The remainder of this paper is organized as follows. The related work is summarized in Section 2. Section 3 presents the proposed methods, while Section 4 discusses and analyzes the model performance, utilized dataset, and environment setup. Section 5 provides the conclusions and future research direction.