# Smart Contract Vulnerability Detection Based on Hybrid Attention Mechanism Model
## 5 Experiments and Results
### Experimental Results  
Result for answering RQ1: To answer whether our proposed model can be improved by incorporating critical code fragment, we conducted two experiments on the SmartWild dataset. It can be seen that the number of positive (i.e., contain vulnerability) and negative samples is heavily imbalanced. Therefore, we limited the generation of negative examples in extracting code fragments from the source code. For evaluation, we used 1864 samples from SmartWild and all smart contracts from Sbcurated. In total, the number of testing set was 2000. We built two datasets, including the original contracts set and the extracted code fragments set. Table 4 shows that the experimental results with code fragments extraction are better than source code as a model input. Obviously, there is a significant improvement in each of the metrics. In particular, Precision and Recall is improved by 3.2% and 7.08%. These results prove that the extracted code fragments can improve the effectiveness of the model. Conceptually, we infer that the original source code of smart contracts contains some information unrelated to vulnerabilities, and the code fragments with data flow dependency and control flow dependency between statements pay more attention to the key point of vulnerability and learn the relevant vulnerable features better. Result for answering RQ2: To prove the effectiveness of our proposed method, we focused on the performance comparison from two categories. First, we chose several representative deep learning models based on data, which include RNN, LSTM, GRU, and so on. The comparison results are summarized in Table 5. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Number(P \(\mid\)N)** & **Accuracy** & **Precision** & **Recall** & **F1** \\ \hline HAM (Unsliced) & 2000(1183 \(\mid\)817) & 84.57\% & 86.38\% & 84.56\% & 85.46\% \\ \hline HAM (code fragment) & 2000(1064 \(\mid\)936) & 88.96\% & 89.58\% & 91.64\% & 90.51\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: The effect of code fragment on vulnerability detection. As can be seen from Table 5, it can be obviously concluded that the HAM outperforms other baseline methods by a large margin in terms of all the four metrics. In particular, HAM achieves a high F1-score of 94.04%, an accuracy of 93.36%, and a recall of 96.64%. An improvement may stem from the fact that we consider data flow and control flow in smart contracts. One can see that the Recall rate and F-score of BiLSTM and BiGRU are higher than the corresponding LSTM and GRU. We think the reason is that in the field of smart contract vulnerability detection, the effect of bidirectional recurrent neural network is better than that of unidirectional recurrent neural network. This is because during the training process of the neural network model, the next output of the unidirectional neural network is only affected by the previous output, whereas the output of the bidirectional neural network is not only affected by the forward output, but also by the backward output. The output of the bidirectional neural network is jointly determined by the output of several units before and after. However, the HAM exploits a single-head attention encoder combined with a bidirectional network and a multi-head attention encoder, which can better perform feature learning. We further visualize the quantitative results of Table 5 in Figure 12a-e. Specially, Figure 12a-e present the comparison results of reentrancy, arithmetic vulnerability, unchecked return value, timestamp dependence and tx.origin vulnerability detection, respectively. Among them, the seven rows from front to back represent model RNN, LSTM, GRU, BiLSTM, BiGRU, BiGRUAt, and our proposed method HAM. For each row in the Figures, accuracy, precision, recall, and F1 score are respectively presented from left to right.