# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
Since no rule indicates the best values assigned for the hyper-parameters, we select the best learning rate from the learning rate set \(1.0\times 10^{-5}\), \(8.0\times 10^{-5}\), \(9.0\times 10^{-5}\), \(1.0\times 10^{-4}\), \(2.0\times 10^{-4}\), \(3.0\times 10^{-4}\), and \(1.0\times 10^{-3}\). This set is selected based on the default value of AdamW and experiments. The AdamW default value is \(1.0\times 10^{-4}\). The selecting technique we follow starts from the default learning rate of AdamW then decreases the value by 0.00001 and trains the model. We repeat decreasing the value until there is no model performance enhancement for two sequential values compared to the model performance in the default value learning rate. We follow the same method by increasing the learning rate value by 0.0001 until there is no significant model performance for two sequential values. This technique produces \(8.0\times 10^{-5}\), \(9.0\times 10^{-5}\), \(1.0\times 10^{-4}\), \(2.0\times 10^{-4}\), and \(3.0\times 10^{-4}\). Then the edge values are considered as well, which are \(1.0\times 10^{-5}\) and \(1.0\times 10^{-3}\). The model is trained for all learning rates with a batch size of 32. The selection of the best learning rate is based on overall accuracy. The model accuracies for the seven experiments with learning rates of \(1.0\times 10^{-5}\), \(8.0\times 10^{-5}\), \(9.0\times 10^{-5}\), \(1.0\times 10^{-4}\), \(2.0\times 10^{-4}\), \(3.0\times 10^{-4}\), and \(1.0\times 10^{-3}\) are 85.74%, 86.04%, 85.81%, 86.8%, 84.97%, 66.67%, and 64.61%, respectively. The learning rate of value \(1.0\times 10^{-4}\) is selected because it achieved the highest performance. The results of the other metrics for all experiments are presented in Tables 3-9. The confusion matrices present the number of each answer predicted correctly or mistakenly, which are shown in Figure 10. The accuracy chart for validation is shown in Figure 11, whereas the validation loss and Train loss is in Figures A1 and A2. \begin{table}
\begin{tabular}{c c c c c} \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Instances No.** \\ \hline
0 & 0.6622 & 1.0000 & 0.7967 & 49 \\
1 & 0.0000 & 0.0000 & 0.0000 & 15 \\
2 & 0.9636 & 0.7910 & 0.8689 & 67 \\ no & 0.7727 & 0.5231 & 0.6239 & 650 \\ yes & 0.5822 & 0.8151 & 0.6792 & 530 \\ accuracy & & & 0.6667 & 1311 \\ macro avg & 0.5961 & 0.6258 & 0.5937 & 1311 \\ weighted avg & 0.6925 & 0.6667 & 0.6581 & 1311 \\ \hline \end{tabular}
\end{table} 
Table 6: The result of models using the greedy soup fine-tuning—batch size = 32, lr = \(3.0\times 10^{-4}\). \begin{table}
\begin{tabular}{c c c c c} \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Instances No.** \\ \hline
0 & 0.7119 & 0.8571 & 0.7778 & 49 \\
1 & 0.0000 & 0.0000 & 0.0000 & 15 \\
2 & 0.8194 & 0.8806 & 0.8489 & 67 \\ no & 0.7634 & 0.4815 & 0.5906 & 650 \\ yes & 0.5623 & 0.8170 & 0.6662 & 530 \\ accuracy & & & 0.6461 & 1311 \\ macro avg & 0.5714 & 0.6073 & 0.5767 & 1311 \\ weighted avg & 0.6743 & 0.6461 & 0.6346 & 1311 \\ \hline \end{tabular}
\end{table} 
Table 7: The result of the best-value model with batch size = 32 and lr = \(1.0\times 10^{-3}\). \begin{table}
\begin{tabular}{c c c c c} \hline
**Answer** & **Precision** & **Recall** & **F1-Score** & **Instances No.** \\ \hline
0 & 0.8000 & 0.8980 & 0.8462 & 49 \\
1 & 0.8000 & 0.2667 & 0.4000 & 15 \\
2 & 0.8592 & 0.9104 & 0.8841 & 67 \\ no & 0.8819 & 0.8615 & 0.8716 & 650 \\ yes & 0.8349 & 0.8585 & 0.8465 & 530 \\ \hline \end{tabular}
\end{table} 
Table 8: The result of the best-value model with batch size = 32 and lr = \(1.0\times 10^{-5}\).