# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
Since the dataset is unbalanced, weighted average precision, recall, and F1-score were calculated. The greedy-soup-based model outperformed the best-value-based model in all calculated performances. The greedy-soup-based model was optimized for batch size and learning rate with values 16, 32, 64, 128 and 8.0 \(\times\) 10\({}^{-5}\), 9.0 \(\times\) 10\({}^{-5}\), 1.0 \(\times\) 10\({}^{-4}\), 2.0 \(\times\) 10\({}^{-4}\), 3.0 \(\times\) 10\({}^{-4}\), respectively. The best model performance was achieved when the batch size was 16 and the learning rate was 1.0 \(\times\) 10\({}^{-4}\), with an accuracy of 87.41%. The other two models that exceeded the SOTA accuracy were the model with batch size 32 and learning rate 1.0 \(\times\) 10\({}^{-5}\) and the model with batch size 128 and learning rate 1.0 \(\times\) 10\({}^{-4}\), with accuracy values of 85.74 and 84.13, respectively. Furthermore, the accuracy for question types and locations were calculated separately and compared with the SOTA. Since there is no rule for hyper-parameters values, an optimization technique, such as a genetic algorithm, can significantly improve the model performance by selecting the optimal ones. Collecting large datasets in the medical field for VQA is complex and requires a long time because specialists are needed to validate the question-answer pairs and the images. In future work, we aim to collect and validate vast data with the help of specialists. Conceptualization S.A.-H.; methodology, S.A.-H. and M.E.B.M. ; software, S.A.-H.; validation, S.A.-H. and A.A.; formal analysis, S.A.-H.; investigation, S.A.-H.; resources, S.A.-H. and A.A.; data curation, S.A.-H.; writing--original draft preparation, S.A.-H.; writing--review and editing, M.E.B.M. ; visualization, S.A.-H.; supervision, M.E.B.M. and S.A.-A. ; project administration, A.A.; funding acquisition, A.A. All authors have read and agreed to the published version of the manuscript. This work was funded by the Deanship of Scientific Research at King Saud University through the initiative of DSR Graduate Students Research Support (GSR). Not applicable. The dataset we used in the experiment is available at [[https://zenodo.org/record/6784358](https://zenodo.org/record/6784358)]([https://zenodo.org/record/6784358](https://zenodo.org/record/6784358)) (accessed on 1 April 2023). The authors would like to thank King Saud University and the College of Computer and Information Sciences. Additionally, the authors would like to thank the Deanship of Scientific Research at King Saud University for funding and supporting this research through the initiative of DSR Graduate Students Research Support (GSR). This work was supported in part by KSU and the Center for Complex Engineering Systems (jointly between MIT and KACST). ## Conflicts of Interest: The authors declare no conflict of interest. Figure A2: The model training loss using different learning rates and batch size = 32. ## References

* [1] [PERSON]; [PERSON]; [PERSON]; [PERSON] [PERSON] Visual7w: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-28 June 2016; pp. 4995-5004. * [2] [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] [PERSON]; [PERSON] [PERSON] VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019. In proceeding of Working Notes of CLEF 2019, Lugano, Switzerland, 9-12 September 2019. * [PERSON] et al. (2020) [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON]; [PERSON] Overview of the VQA-Med Task at ImageCLEF 2020: Visual Question Answering and Generation in the Medical Domain. In Proceedings of the CLEF 2020--Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, 22-25 September 2020; pp. 1-9. * [PERSON] et al. (2021) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering. In Proceedings of the 2021 IEEE 18 th International Symposium on Biomedical Imaging (ISBI), Nice, France, 13-16 April 2021; pp.