# Smart Contract Vulnerability Detection Based on Hybrid Attention Mechanism Model
## 4 Our Approach
### Hybrid Attention Mechanism
#### 4.3.1 Single-Head Attention Encoder  
Recurrent neural network (RNN) is a kind of neural network, which is improved from feed forward neural network [55]. RNNs employ its internal state (memory) to process variable-length input sequences, but such a structure also has defects. The processing ability for long sequences is poor, and gradients are prone to disappear. In addition to that, the time and computational cost required for RNN training are enormous. To solve these problems, long short-term memory neural network (LSTM) [56] and gated recurrent unit neural network (GRU) [57] are proposed. Long short-term memory neural network is a variant of RNN. Yet unlike RNNs, LSTMs have feedback connections. A common LSTM cell is composed of a cell, an input gate, an output gate, and a forget gate. Similar to LSTM, GRU is also a solution to the vanishing gradient in RNN. In comparison with LSTM, the experimental effect presented by GRU has almost no change, but the structure of GRU is simpler. Compared with LSTM, GRU has only two gate functions: update gate and reset gate, which new state information combined with the previous memory unit is determined by the reset gate, and which information of the previous memory unit needs to be stored in the current timestep is controlled by the update gate. The structure diagram of GRU model is shown in Figure 9.  
Where \(z_{t}\) is update gate, \(r_{t}\) means reset gate, \(h_{t-1}\) indicates the status information of the previous moment, \(h_{t}\) is current status information, and \(\sigma\) represents the sigmoid function that converts data into a value in the range of 0 to 1 and acts as a gated signal. The specific calculation process are as follows:  
\[r_{t}=\sigma(W_{r}\cdot[h_{t-1},x_{t}]) \tag{2}\]  
\[z_{t}=\sigma(W_{z}[h_{t-1},x_{t}]) \tag{3}\]  
\[z_{t}=\sigma(W_{z}[h_{t-1},x_{t}]) \tag{4}\]  
\[\widetilde{h_{t}}=\tanh(W_{h}[r_{t}\times h_{t-1},x_{t}]) \tag{5}\]  
\[h_{t}=(1-z_{t})*h_{t-1}+z_{t},\widetilde{h_{t}} \tag{6}\]  
where \(W_{r}\),\(W_{z}\),\(W_{\widetilde{h_{t}}}\) are the parameters learned through training, and is the candidate hidden layer state, which is only related to the input \(x_{t}\) and the hidden state \(h_{t-1}\) of the previous layer. The disadvantage of the traditional GRU neural network is that it can only read the input source code information from a single direction. However, the actual situation needs to fully consider the feature representation of the front and back source codes. To address this problem, this paper adopts BiGRU neural network to capture the vector features from the forward and the backward directions. The principle of BiGRU is shown in Figure 10.  
Figure 9: The basic structure of Gated Recurrent Unit. \(GRU\) gains the input vectors \((x_{1},x_{2},\ldots x_{T})\) from the forward direction, and compute the forward hidden layer state corresponding to each vector, i.e., \(\left(\overset{\rightarrow}{h_{1}},\overset{\rightarrow}{h_{2}},\ldots, \overset{\rightarrow}{h_{T}}\right)\) and vice versa. The forward and backward hidden layer states of the \(t\)th input vector can be specifically expressed as:  
\[\overset{\rightarrow}{h_{t}}=GRU(h_{t-1},x_{t}) \tag{7}\]  
\[\overset{\leftarrow}{h_{t}}=GRU\left(\overset{\leftarrow}{h_{t-1}},x_{t}\right) \tag{8}\]  
After that, the concatenation operation is performed to obtain the final hide layer state of output, as shown in the following Equation. \[r_{t}=\sigma(W_{r}\cdot[h_{t-1},x_{t}]) \tag{2}\]0  
The attention mechanism stems from the human brain's ability to perceive new things, allocating more attention to what is important and less attention to what is not. By assigning different probability weights to the hidden layer units of neural networks with the help of the attention mechanism, more favorable features can be extracted. Attention connects two different parts by automatically weighing and transforming the data, highlighting the key points to show better performance.