# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method
### Dataset

The Diabetic Macular Edema (DME) dataset [5] is generated automatically from the Indian Diabetic Retinopathy Image Dataset (IDRiD) [81], and the e-Ophta dataset [82] is used. The dataset has 679 images with 13,470 question-answer pairs distributed into 433, 112, 134 images with 9779, 2380, and 1311 for the train, validation, and testing dataset, respectively. The dataset has questions about hard exudates, optic discs, and exudates? grades. The distinction between grades is based on the presence of a hard exudate at various sites on the retina. Specifically, grade 0 means no hard exudate at all, grade 1 means there is a hard exudate in the periphery of the retina (i.e., central fovea and radius one papilla diameter), and grade 2 if there is a hard macular exudate. The dataset has the original images and masks that specify a specific region of the image. The masks have to be applied to the images before training. Figure 7 shows examples of images and related question-answer pairs from the dataset. Table 1 shows the distributed of classes in the train, validation, and testing datasets. ### Evaluation Metrics

We calculate the model accuracy, precision, recall, F1-score, macro accuracy average, and weighted accuracy average to measure the model performance and compare it with the SOTA models. The equation of each metric is shown in the following: \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Yes** & **No** & **0** & **1** & **2** & **Total** \\ \hline Train & 4713 & 4639 & 166 & 41 & 220 & 9779 \\ Validation & 1151 & 1123 & 39 & 8 & 59 & 2380 \\ Test & 530 & 650 & 49 & 15 & 67 & 1311 \\ \hline Total & 6394 & 6412 & 254 & 64 & 346 & 13,470 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Number of instances per answer for each part of DME dataset. Figure 7: Examples of DME dataset images and related QA pairs, where the yellow circle is an optic disk, the cutine circle is the macula, and x is the fovea- the center of the macula. * **Accuracy:** Accuracy is a commonly used performance metric for classification problems that measures the proportion of correctly classified samples out of the total number of samples. It is calculated using the equation: \[\text{Accuracy}=\frac{\text{True Positives + True Negatives}}{\text{True Positives + False Positives + True Negatives + False Negatives}}\] (21) where True Positives are the number of positive samples that are correctly identified as positive by the model, True Negatives are the number of negative samples that are correctly identified as negative by the model, False Positives are the number of negative samples that are incorrectly identified as positive by the model, and False Negatives are the number of positive samples that are incorrectly identified as negative by the model. * **Precision:** Precision, also known as a positive predictive value, measures the proportion of the predicted true positive samples out of all predicted positive samples. The precision metric's importance is giving an indication of whether the model has aow false positive rate based on a high precision score. It is calculated using the equation: \[\text{Precision}=\frac{\text{True Positives}}{\text{True Positives + False Positives}}\] (22)
* **Recall or Sensitivity:** Recall, also known as sensitivity or true positive rate, measures the proportion of the predicted positive samples out of the actual positive samples. The recall metric is critical in applications such as medical diagnosis, where it is essential to minimize false negatives. It is calculated using the equation: \[\text{Recall}=\frac{\text{True Positives}}{\text{True Positives + False Negatives}}\] (23)
* **F1-score:** The F1 -score is the harmonic mean of precision and recall that measures the model's accuracy in correctly identifying positive samples. The F1-score is a valuable metric for evaluating classification models, especially when the dataset is imbalanced, and the goal is to ensure that the model performs well on both positive and negative samples.