# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### SOUPS Fusion Method  
* **Greedy soups** The greedy soups technique is used to fuse the parameters of multiple fine-tuned models, where the model is trained, and validation accuracy is calculated several times during training; the final parameters will be the average of the best \(k\) models' parameters based on the fine-tuned model weights that significantly improve the validation performance by averaging the weights of the new fine-tuned model with theist of previous models weights. The proposed model utilizes this technique to maximize its performance. Let \(M=\{m_{1},m_{2},\ldots m_{n}\}\) and \(\theta=\{\theta_{0},\theta_{1},\ldots,\theta_{n}\}\) denote the set of models and their parameters, respectively.et \(sg=\{\theta_{0},\theta_{1},\ldots,\theta_{k}\}\) and \(Mk=\{m_{k1},m_{k2},\ldots,m_{kk}\}\) be soup ingredients or the parameters for the considered \(Mk\) models. Each time \(i\) the validation is computed, the model \(m_{i}\) is considered if \(valAcc(m_{i})>min(m_{k1},m_{k2},\ldots,m_{kk})\). The \(Mk\) models are sorted decreasingly. From the \(Mk\) models and their \(sg\) parameters, the model is considered for fusion if, for each time \(i\) to \(k\), the \(valAcc(sg_{i-1}\cup\{\theta_{i}\})\) is greater than \(valAcc(sg_{i-1})\).et \(c-sg\) denote the considered \(j\) models. The final model parameters \(\theta^{\prime}\) are the average of the \(c-sg\) models' parameters, and they are calculated using the equation: \[\theta^{\prime}=\frac{1}{j}\sum_{i=1}^{j}\theta_{i}\] (19) The proposed model is set to be based on the best three models, \(k=3\), where the validation is calculated in the middle and end of each epoch. Algorithm 1 shows the algorithm of greedy soup to fuse the three models with different hyper-parameters (number of steps). Figure 6 presents the overall structure of the greedy soup technique to fuse three models. * **The best value** The fine-tuning based on the best accuracy value selects the model that achieves the highest score. It is represented using the equation: \[f(x,argmax,ValAcc(\theta_{i}))\] (20)```
0: number of epoch epoch and greedy soup grade \(k\)
0: model parameters \(\theta\)
1:\(valAcc\leftarrow\{\}\)
2:\(ingredients\leftarrow\{\}\)
3:for\(i\gets 1\) to epochdo
4:if not mid(epoch) then
5: train model
6:endif
7: Compute the \(val-accuracy(model)\)
8: train model
9:if length(\(ingredients)<3\)then
10:\(ingredients\gets ingredients\cup\{model(\theta)\}\)
11:\(valAcc\gets valAcc\cup val-accuracy(model)\)
12:elseif\(val-accuracy(model)>min(valAcc)\)then
13:remove(\(valAcc,min(valAcc)\))
14:remove(\(ingredients,min(model(\theta))\))
15:\(ingredients\gets ingredients\cup\{model(\theta)\}\)
16:endif
17:endfor
18: decreasing order of \(ValAcc\) with its appropriate ingredients
19:\(ingredients\_soup\leftarrow\{\}\)
20:for\(i\gets 1\) to \(k\)do
21:if\(ValAcc(\text{average}(ingredients\_soup\quad\cup\quad\{ingredients_{i}\}))\geq ValAcc(\text{average}(ingredients\_soup))\)then
22:\(ingredients\_soup\gets ingredients\_soup\cup\{ingredients_{i}\}\)
23:endif
24:endfor
25:return\(\text{average}(ingredients\_soup)\)
```

**Algorithm 1** Greedy_Soup_Model

## 4 Experiment

### Environment Setup

The models are trained on a Google Colab premium account with NVIDIA A100-SXM4-40 GB (Nvidia Corporation, Santa Clara, CA, USA ) and 80 GB RAM. For the optimization function, AdamW is used with \(1.0\times 10^{-3}\) and 0.9 weight decay andearning rate decay, respectively. The model optimized to select the best batch size andearning rate among the values \(1.0\times 10^{-3}\), \(3.0\times 10^{-3}\), \(2.0\times 10^{-3}\), \(1.0\times 10^{-4}\), \(9.0\times 10^{-5}\), \(8.0\times 10^{-5}\), \(1.0\times 10^{-5}\) and \(16\), \(32\), \(64\), \(128\), respectively. The model was trained for ten epochs in all experiments and evaluated using the validation set twice per epoch. Furthermore, it sets up to stop training if there is no enhancement for ten sequential times. Figure 6: The Greedy Soup Model.