# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
\begin{table}
\begin{tabular}{c c c c c} \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Instances No.** \\ \hline
0 & 0.8936 & 0.8571 & 0.8750 & 49 \\
1 & 0.4615 & 0.4000 & 0.4286 & 15 \\
2 & 0.8732 & 0.9254 & 0.8986 & 67 \\ no & 0.8738 & 0.8523 & 0.8629 & 650 \\ yes & 0.8242 & 0.8491 & 0.8364 & 530 \\ accuracy & & & 0.8497 & 1311 \\ macro avg & 0.7853 & 0.7768 & 0.7803 & 1311 \\ weighted avg & 0.8497 & 0.8497 & 0.8495 & 1311 \\ \hline \end{tabular}
\end{table} 
Table 5: The result of models using the greedy soup fine-tuning—batch size = 32, lr = \(2.0\times 10^{-4}\). Figure 10: The confusion matrices of models with different learning rates and batch size = 32. Figure 11: _Cont._ \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Answer**} & \multicolumn{3}{c}{**Beat-Value-Based Model**} & \multicolumn{3}{c}{**Greedy-Soup-Based Model**} & \multirow{2}{*}{**Samples No.**} \\ \cline{2-2} \cline{5-8}  & **Precision** & **Recall** & **F1-Score** & **Precision** & **Recall** & **F1-Score** & \\ \hline
0 & 1.000 & 0.7755 & 0.8736 & 0.9512 & 0.7959 & 0.8667 & 49 \\
1 & 0.4444 & 0.8000 & 0.5714 & 0.4091 & 0.6000 & 0.4865 & 15 \\
2 & 0.9242 & 0.9104 & 0.9173 & 0.8971 & 0.9104 & 0.9037 & 67 \\ no & 0.9029 & 0.8154 & 0.8569 & 0.9057 & 0.8569 & 0.8806 & 650 \\ yes & 0.7976 & 0.8925 & 0.8424 & 0.8354 & 0.8906 & 0.8621 & 530 \\ Accuracy & & & 0.8497 & & & 0.8680 & 1311 \\ macro avg & 0.8138 & 0.8388 & 0.8123 & 0.7997 & 0.8108 & 0.7999 & 1311 \\ weighted avg & 0.8598 & 0.8497 & 0.8515 & 0.8729 & 0.8680 & 0.8693 & 1311 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: The result of the best-value model fine-tuned based on the best validation accuracy—batch size = 32, lr = \(1.0\times 10^{-4}\) weight. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Samples No.** \\ \hline
0 & 0.9730 & 0.7347 & 0.8372 & 49 \\
1 & 0.3684 & 0.4667 & 0.4118 & 15 \\
2 & 0.8400 & 0.9403 & 0.8873 & 67 \\ no & 0.9089 & 0.7985 & 0.8501 & 650 \\ yes & 0.7849 & 0.9019 & 0.8393 & 530 \\ accuracy & & & 0.8413 & 1311 \\ macro avg & 0.7750 & 0.7684 & 0.7652 & 1311 \\ weighted avg & 0.8515 & 0.8413 & 0.8422 & 1311 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: The result of the the model with batch size = 128 and lr = \(1.0\times 10^{-4}\). Figure 12: The confusion matrices of models with different batch sizes and learning rate = \(1.0\times 10^{-4}\). An experiment of the best-value model with the same hyper-parameters (16 batch size and \(1.0\times 10^{-4}\) learning rate) of the best model based on the fusion of three models with the same seed (42) is conducted to show the effectiveness of the model based on the weights fusion of multiple fine-tuned models. Table 13 shows the performance of the best-value model with batch size 16 and learning rate \(1.0\times 10^{-4}\). In addition, Figure 14 shows the confusion matrix and validation accuracy. The metrics used for the comparison are accuracy, which is the metric usually used in medical VQA, and weighted average, precision, recall, and F1-score, as a result of the unbalanced dataset with a massive variance of the example number in each class. Table 14 shows the results of four models, two best-value models and two fused models. Figure 15 presents the comparison chart showing how the greedy soup model can produce more significant results than the model based on the best validation performance with the same computation cost \(O(1)\) for the investigated problem. The greedy soup model with batch size 32 and learning rate \(1.0\times 10^{-4}\) enhances the performance of the best-value model with the same batch size and learning rate values by 1.83%, 1.31%, 1.83%, and 1.78% for accuracy, weighted average precision, weighted average recall, and weighted average F1-recall, respectively. The models trained with batch size 16 and learning rate \(1.0\times 10^{-4}\) have enhanced accuracy, weighted average precision, weighted average recall, and weighted average F1-recall.