# Smart Contract Vulnerability Detection Based on Hybrid Attention Mechanism Model
## 4 Our Approach
### Hybrid Attention Mechanism
#### 4.3.2 Multi-Head Attention Encoder  
Whether RNN, LSTM and GRU, or their variant structures, can only be calculated from left to right or from right to left in sequence when calculating the text. This mechanism will lead to two problems. For one thing, the calculation of timestep \(t\) depends on the calculation result at time \(t-1\), which will limit the parallel calculation ability of the model; for another, various gate mechanisms of LSTM and GRU can solve the issue of long sequence dependency to a certain extent, but the performance will drop sharply when encountering a particularly long sequence. In 2017, Google emphasized the role of attention mechanism and proposed the Transformer model. Transformer is a Seq2 Seq model [58], which mainly applies to complete tasks such as machine translation. The biggest difference from LSTM and GRU is that the Transformer model does not need to be processed sequentially according to the order of the words, and can train all the words at the same time, which greatly improves the degree of parallelism and increases computational efficiency. Moreover, Transformer introduces Positional Encoding to understand the position of each word in the sentence, not just the semantics of the word itself. Figure 11 depicts the Transformer encoder. The input sequence first undergoes word embedding and position encoding to obtain a word embedding matrix \(X\) that contains position information. The two dimensions of \(X\) represent word embedding dimension and sequence length respectively. Feature analysis is then performed in parallel through eight attention heads, where each attention head has three weight matrices, namely \(W_{q}\), \(W_{k}\) and \(W_{v}\). For the word vector \(x_{t}\) for a word, the Query vector \(q_{t}\), the Key vector \(k_{t}\), and the Value vector \(v_{t}\) can be obtained by calculating it with three weight matrices. After concatenating the three vectors of all words, the query matrix Q, key matrix K, and the value matrix V are, respectively, acquired. To obtain the attention weight of a certain word, the query vector \(q_{t}\) of the word multiplied by the key matrix V makes the scores of all words in the sequence for the current word. After dividing each score by the square root of the key vector dimension, use softmax to obtain the probability, then multiply it with the corresponding word value vector, and finally accumulate all the value vectors to obtain the output vector \(z_{t}\) of the current word. Figure 11: Transformer Encoder. The multi-head attention encoder in this paper no longer involves the recurrent neural network, but utilizes two steps to acquire the feature vector. The first step is to obtain the positional encoding information of each word. To capture the order of words in a code fragment, the feature \(p_{t}\) of position embedding is introduced in this paper. It can add the position information of each word in the word vector. Then, we do linear transformation to the word vector \(x_{t}\) and position embedding for each word. The specific process are expressed in the following equation. \[p_{t}^{i}=\left\{ \begin{array}{c}sin\left(\frac{t}{10000^{\frac{T}{d}}} \right)ifi=2k\\ cos\left(\frac{t}{10000^{\frac{T}{d}}}\right)ifi=2k+1\end{array} \right. \tag{14}\]  
\[x_{t}=(x_{t}+p_{t})W_{p}+b_{p} \tag{15}\]  
where \(t\) is the position of the word, \(i\) is the dimension position in the word vector, and \(d\) is the output dimension. Self-Attention mechanism is called intra-attention, whose query vector (\(Q\)), key value (\(K\)) and value (\(V\)) all come from the same input, while the query vector in traditional attention mechanism generally comes from the outside. Self-Attention uses the attention mechanism to dynamically generate weights of different connections, so as to better learn distance dependence. The core of the self-attention mechanism lies in the dot product attention. Its calculation method is shown as follows.