## 2 Materials and Methods
### MGFGF-BiLSTM-EGP
The probability of co-occurrence between words \(i\) and \(j\) is then calculated using the formula:

\[P_{ij}=\frac{X_{ij}}{\sum_{k}X_{ik}} \tag{1}\]

Here, \(P_{ij}\) indicates the likelihood of word \(j\) appearing within the context of word \(i\). The GloVe model refines word vectors to accurately reflect these co-occurrence probabilities, aiming to minimize the following objective function:

\[J=\sum_{i,j=1}^{V}f\big{(}X_{ij}\big{)}\left(w_{i}^{T}\widetilde{w_{j}}+b_{i} +\widetilde{b_{j}}-\log\big{(}X_{ij}\big{)}\right)^{2} \tag{2}\]

In this equation, \(w_{i}^{T}\) and \(w_{j}\) are the word vectors for words \(i\) and \(j\), respectively, while \(b_{i}\) and \(b_{j}\) are the corresponding bias terms. The logarithm of the co-occurrence count between the words, \(\log\big{(}X_{ij}\big{)}\), and the weighting function \(f\big{(}X_{ij}\big{)}\) are used to reduce the influence of rare co-occurrences, defined as:

\[f\big{(}X_{ij}\big{)}=\left\{ \begin{array}{ll}\big{(}\frac{X_{ij}}{X_{max}} \big{)}^{\alpha}&\text{if }X_{ij}<X_{max}\\ 1&\text{otherwise}\end{array} \right. \tag{3}\]

In this context, \(X_{max}\) and \(\alpha\) are hyperparameters that shape the weighting function, ensuring that both frequent and rare co-occurrences are appropriately weighted. As shown in Figure 4, for each sentence \(X=\{c_{1},c_{2},\dots,c_{n}\}\) with n tokens, each token \(c_{i}\) is represented by combining the character with GloVe word vectors. The set \(W_{c}=\{w_{i}\mid c_{i}\in w_{i}\}\) represents the collection of word vectors \(w_{i}\) for words containing the

Figure 3: Character encoder framework. character \(c_{i}\). To balance the influence of word vectors according to their frequencies, logarithmic scaling is applied to the word frequencies:

\[f_{j}^{\prime}=\log\bigl{(}1+f_{j}\bigr{)} \tag{4}\]

where \(f_{j}^{\prime}\) is the logarithmically scaled frequency of word \(j\), and \(f_{j}\) is the original frequency. The adjusted word vector for word \(j\) is then calculated as:

\[w_{j}^{\prime}=f_{j}^{\prime}w_{j} \tag{5}\]

Here, \(w_{j}^{\prime}\) is the weighted word vector, and \(w_{j}\) is the original word vector. The integration of character vectors with weighted word vectors is achieved through a multi-head attention mechanism. The query, key, and value vectors are first calculated as follows:

\[Q=W_{Q}h_{i} \tag{6}\]

\[K_{j}=W_{K}w_{j}t \tag{7}\]

\[V_{j}=W_{V}w_{j}t \tag{8}\]

where \(Q\) represents the query vector, \(K_{j}\) is the key vector, and \(V_{j}\) is the value vector. The matrices \(W_{Q},\ W_{K}\), and \(W_{V}\) are the projection matrices for the query, key, and value, respectively. The attention mechanism computes the attention weights to evaluate the relevance of each key-value pair to the query. \[\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{9}\]

where \(d_{k}\) is the dimension of the key vector, and the softmax function normalizes the similarity scores, converting them into a probability distribution. A weighted summation of the value \(\text{Attention}\bigl{(}QW_{Qi},\ K_{j}W_{Ki},\ V_{j}W_{Vi}\bigr{)}\) vectors using the computed attentional weights yields an output vector for each header:

\[\text{head}_{i}=\sum_{j=1}^{n}\text{Attention}\bigl{(}QW_{Qi},K_{j}W_{Ki},V_{ j}W_{Vi}\bigr{)} \tag{10}\]

Figure 4: Word encoder framework. where \(n\) is the length of the input sequence, and \(W_{Qi},W_{Ki},W_{Vi}\) are the projection matrices for the \(i\)-th attention head. \(\text{Attention}(QW_{Qi},K,W_{Ki},W_{Vi})\) is the attention weight calculated using the query matrix \(QW_{Qi}\) and the key matrix \(K_{j}W_{Ki}\). The corresponding value matrix \(V_{j}W_{Vi}\) is weighted and summed up. These probabilities are used to weigh the value vectors, determining their contribution to the final output. The multi-head attention mechanism allows the model to focus on different parts of the input sequence simultaneously. By employing multiple attention heads, the model captures a wider range of relationships within the data.