# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
[5] evaluated the SOTA models with different question categories: overall, which is the whole dataset; grade, which refers to the questions about the diabetic macula edema grade; whole, which is a question about the existence of exudates in the whole image; macula, which refers to the fovea in the dataset; and region, which is a question about specific region in the image. The research follows the same evaluation to make a fair comparison with the SOTA model. Table 16 compares the SOTA and the nine models outperforming it based on the accuracy evaluation metric. All models based on greedy soup outperform the SOTA performance in terms of overall, whole, grade, and macula. The SOTA achieved higher region accuracy than the greedy soup model trained in 128 batch size and \(1.0\times 10^{-4}\) learning rate with a slight difference of 0.04, where the SOTA model achieved 83.12%, and the gs-128-\(1.0\times 10^{-4}\) model achieved 83.12%. Therefore, seven fusion models based on greedy soup exceed the SOTA model. Even the bv-32-\(1.0\times 10^{-4}\) model achieved the highest grade accuracy; its overall accuracy is less than five fusion models based on greedy soup, which are gs-16-\(1.0\times 10^{-4}\), gs-32-\(1.0\times 10^{-4}\), gs-32-\(9.0\times 10^{-5}\), gs-32-\(8.0\times 10^{-5}\), and gs-32-\(1.0\times 10^{-5}\), and it has the same accuracy as gs-32-\(2.0\times 10^{-4}\). Figure 16 shows the comparison of the SOTA model and the proposed nine models in terms of overall, whole, grade, region, and macula. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Overall** & **Grade** & **Whole** & **Macula** & **Region** \\ \hline SOTA [5] & 83.49 & 80.69 & 84.96 & 87.18 & 83.16 \\ bv-32-\(1.0\times 10^{-4}\) & 84.97 & **84.73** & 90.84 & 85.29 & 83.22 \\ bv-16-\(1.0\times 10^{-4}\) & 85.74 & 79.39 & 90.84 & 83.21 & 86.27 \\ gs-128-\(1.0\times 10^{-4}\) & 84.13 & 80.92 & 90.08 & 88.55 & 83.12 \\ gs-32-\(1.0\times 10^{-5}\) & 85.74 & 83.21 & 90.08 & 87.79 & 85.19 \\ gs-32-\(1.0\times 10^{-4}\) & 86.8 & 83.21 & **92.37** & **90.84** & 85.95 \\ gs-16-\(1.0\times 10^{-4}\) & **87.41** & 82.44 & 88.55 & 87.02 & **88.02** \\ gs-32-\(2.0\times 10^{-4}\) & 84.97 & 83.97 & 87.02 & 89.31 & 84.20 \\ gs-32-\(9.0\times 10^{-5}\) & 85.81 & 82.44 & 89.31 & 88.55 & 85.40 \\ gs-32-\(8.0\times 10^{-5}\) & 86.04 & 80.92 & 89.31 & 89.32 & 85.84 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: The result comparison with the SOTA. Figure 16: The comparison between the SOTA model and the proposed models. ## 5 Conclusions

Visual question answering (VQA) is a task that involves generating or predicting an answer to a question about visual images in human language. This task combines two branches of AI, namely, NLP and computer vision, and is an active field of research. However, in the medical field, VQA is still in its early stages and requires extensive efforts and exploration to become practically useful. This paper presents two VQA models in the medical field. The models utilized the most current transformers for vision and language. Using a transformer instead of the visual CNN pre-trained model reduces the variation between the textual and visual features. ELECTRA-base and SWIN-base are utilized for textual and visual feature extraction. The model enhances the SOTA performance. The first model, the best-value-based model, fine-tuned the parameters based on the model that achieved the highest validation accuracy during training. The second model, the greedy-soup-based model, fine-tuned the parameters based on the greedy soup technique, where the parameters are the average of parameters that significantly impact the validation accuracy during the training. On the other hand, the model fused three best-value-based models with different training step numbers, leading to different models' weights. The best-value-based model achieved 84.97% accuracy, which outperformed the SOTA accuracy, which is 83.49%. Using greedy soup to select the model parameters enhances the model performance from 84.97% to 86.8%.