# An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
## 3 Methodology
### ELECTRA Transformer  
The proposed model utilizes ELECTRA, a textual pre-trained transformer model trained on aarge corpus of text for textual featurization [77]. The main advantage of ELECTRA is designing a novel pre-training approach called contrastive estimation. Unlike other pre-training approaches, such as maskedanguage modeling used in BERT, ELECTRA trains the model to distinguish between real and replaced tokens in an input sentence. This method encourages the model toearn more effectively from the surrounding context and improves its ability to capture the nuances of theanguage,eading to better performance on downstream tasks. Furthermore, the ELECTRA transformer offers several advantages over other transformer models. Apart from improving efficiency, it has faster times, robustness to noise and adversarial attacks, and is adaptable to a wide range of naturalanguage processing tasks [77]. Figure 5 shows the replaced token detection overview. The contrastive estimation can be expressed with the following equations:  
Given an input sentence \(x\), a set of replaced sentences \(x^{\prime}\), and a binaryabel \(y\) (where \(y=1\) if \(x^{\prime}\) is a true replacement of \(x\), and \(y=0\) otherwise), the contrastiveeoss function, which is a type of cross-entropy (CE), is defined as:  
\[{}_{CE}=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y }_{i}) \tag{5}\]  
where \(N,y_{i}\), and \(\hat{y}_{i}\) are the number of training examples, the trueabel of example \(i\), and the predictedabel of example \(i\), respectively. The predictedabel \(\hat{y}_{i}\) is computed as:  
\[\hat{y}_{i}=\frac{\exp(f(x_{i},x_{i}^{\prime}))}{\exp(f(x_{i},x_{i}^{\prime})) +\sum_{j\
eq i}\exp(f(x_{i},x_{j}^{\prime}))} \tag{6}\]  
where the similarity score between the input sentence \(x_{i}\) and the corresponding replaced sentence is \(x_{i}^{\prime}\)\(f(x_{i},x_{i}^{\prime})\). The similarity score is computed using a neural network based on the concatenated input, replaced sentences as input, and produces a scalar output. The denominator in the predictedabel expression normalizes the scores across all replaced sentences for a given input sentence. During training, the model is presented with pairs of input sentences containing real tokens and replaced tokens. The model is trained to predict whether the tokens replaced in the second sentence were replaced based on the surrounding context. This method encourages the model to effectivelyearn from theanguage and improve its understanding of the nuances of theanguage. The ELECTRA transformer consists of 12 ayers in its transformer encoder. Eachayer contains 12 attention heads and a hidden size of 768. The model total parameters is 110 million. The input to the ELECTRA-base transformer is a sequence of tokens, which are first passed through an embeddinglayer to obtain a sequence of dense vectors. The transformer encoderayers then process these vectors to extract contextualized representations of the input text. Theength of the features produced by the ELECTRA-base transformer is 512. The feature extraction processes are described as follows:  
Figure 5: Replacement of the token detection overview. * Token Embedding: The input text \(T\) is tokenized, and each token is mapped to its corresponding embedding vector. The token embeddings are denoted as \(X\). \[X=\text{TokenEmbedding}(T)\] (7)
* Encoderayers: The ELECTRA transformer uses a stack of encoderayers to process the token embeddings and extract contextualized representations of the input text. Each encoderayer consists of a self-attention mechanism followed by a feedforward network. \[X^{\prime}=\text{Encoder}(X)\] (8)
* Maskedanguage Modeling: During pre-training, a subset of the input tokens are randomly masked, and the model is trained to predict the original tokens based on the masked tokens. The masked tokens are denoted as \(M\). \[M=\text{Mask}(T)\] (9)
* Masked Token Prediction: The output of the encoderayers is used to predict the original tokens based on the masked tokens.