# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
They trained the models with the Adam optimization function with a learning rate of \(1.0\times 10^{-4}\) and 64 batch size for 100 epochs with an early stop if there is no validation accuracy enhancement for 20 epochs. The proposed model is configured with a learning rate value of 0.0001, the default value for the AdamW optimization function, and a batch size value of 32, which is usually used in the field. In both techniques, the model outperforms the SOTA performance, which has an accuracy of 83.49%, where it achieves an accuracy of 84.97% and 86.8% for the two models, respectively. Table 2 presents the precision, recall, F1-score, macro average, and weighted average accuracy for each class in the testing dataset for both models. In addition,they present the overall accuracy. The result appears that the model with the greedy soup fine-tuning technique outperforms the traditional one. According to the confusion matrix for both models in Figure 8, it can be clearly seen that the greedy soup model outperforms the best-value-based model in predicting '0' and 'no' answers, whereas the best-value-based model outperforms it in predicting '1' and 'yes' answers. For the answer of grade equal to '0', although the greedy-soup-based model predicts correct answers more than the best-value-based model, the latter achieves 100% precision, whereas the greedy-soup-based model achieves 95.12% precision. This result shows that the best-value-based model can predict that other answers are not '1', but at the same time, it can not always predict the answer '1' as '1', and that is why the recall of the answer '1' in the greedy-soup-based model is higher than that in the best-value-based model. Overall, since the dataset is unbalanced, the macro average and weighted average performance are more precise than those metrics, which compute the performance ignoring the variance between the answers numbers in the dataset. Therefore, comparing the two methods based on the macro average and weighted average performance is considered. The average macro precision, recall, and F1-score are 81.38 vs. 79.97, 83.88 vs. 81.08, and 8123 vs. 79.99 for the best-value-based and greedy-soup-based models, respectively. On the other hand, the average weighted precision, recall, and F1-score are 85.98 vs. 87.29, 84.97 vs. 86.80, and 85.15 vs. 86.93 for the best-value and greedy-soup-based models, respectively. Figure 9 presents the validation performance for both model fine-tuning techniques. The best-value-based model achieves higher average macro performance than the greedy-soup-based models, whereas the greedy-soup-based model achieves higher weighted average performance and overall accuracy than the first model. Therefore, the greedy-soup-based model is more significant than the best-value-based model for two reasons. The first reason is that the weighted average metric considers the unbalanced \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Instances No.** \\ \hline
0 & 0.9512 & 0.7959 & 0.8667 & 49 \\
1 & 0.4091 & 0.6000 & 0.4865 & 15 \\
2 & 0.8971 & 0.9104 & 0.9037 & 67 \\ no & 0.9057 & 0.8569 & 0.8806 & 650 \\ yes & 0.8354 & 0.8906 & 0.8621 & 530 \\ accuracy & & & 0.8680 & 1311 \\ macro avg & 0.7997 & 0.8108 & 0.7999 & 1311 \\ weighted avg & 0.8729 & 0.8680 & 0.8693 & 1311 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The result of models using the greedy soup fine-tuning—batch size = 32, \(\text{lr}=1.0\times 10^{-4}\). Figure 8: The model confusion matrix for models with different fine-tuning method. dataset and gives each class weight based on how much that class appears. The second reason is that accuracy is the metric computed on the SOTA med-VQA. The model is optimized based on learning rate and batch size hyper-parameters.