# Toward Vulnerability Detection for Ethereum Smart Contracts Using Graph-Matching Network
## 5 Evaluation
### Experimental Setup  
We implemented prototypes of _ASGVulDetector_ and _BASGVulDetector_ in Python. The AST of a smart contract in source code was parsed by solc-ast ([[https://github.com/iamdefinitelyahuman/py-solc-ast](https://github.com/iamdefinitelyahuman/py-solc-ast)]([https://github.com/iamdefinitelyahuman/py-solc-ast](https://github.com/iamdefinitelyahuman/py-solc-ast)) accessed on 21 July 2021), while the machine learning models were powered by the PyTorch library ([[https://github.com/pytorch/pytorch](https://github.com/pytorch/pytorch)]([https://github.com/pytorch/pytorch](https://github.com/pytorch/pytorch)) accessed on 18 May 2021). The computer we used for model training and vulnerability detection was a desktop with Intel Core i7 CPU, 16 GB memory, and NVIDIA Tesla T4 GPU, running on Ubuntu 20.04 operating system. We collected a dataset of 5735 open-source smart contracts from SmartBug [37], SolidFI [38], and other labeled contracts, which contains the four types of vulnerabilities, i.e., reentrancy, timestamp dependency, block info dependency, and Tx.Origin. These labeled contracts were randomly divided into a training set and a testing set, with a 70-30 split. In experiments for _BASGVulDetector_, these smart contracts were compiled to bytecodes beforehand. The threshold of similarity for the GMN model to detect vulnerability was set to 0.85, and other parameters of training models were determined according to the experiment conducted in Section 5.4.  
Since our two approaches support different scenarios, i.e., source code and bytecode, we evaluated them separately. For the purpose of comparisons, in experiments for _ASGVulDetector_, we selected three state-of-the-art vulnerability detection tools, i.e., _SmartCheck_[24], _Slither_[25], and _DR-GCN_[27], all of which belong to the static analysis category and are applicable for the source-code scenario. As aforementioned, _SmartCheck_ converts Solidity code to a XML-based intermediate representation, and detects vulnerability by checking XPath patterns. _Slither_ follows the same idea but defines its own intermediate representation, i.e., SlitherIR, and combines data flow and taint tracking to analysis the code. Among the above competitors, _DR-GCN_ is the most similar work to our approaches. It constructs a graph with three types of nodes and four types of edges, and uses GNN for vulnerability detection. In experiments for _BASGVulDetector_, we selected three other vulnerability detection tools, i.e., _ContractFuzzer_[11], _Oyente_[15], and _Securify_[21], which support bytecode and belong to the categories of fuzzing, symbolic execution, and formal verification, respectively. _ContractFuzzer_[11] instruments EVM and performs fuzz testing on smart contracts. _Oyente_[15] formalizes the semantics of Ethereum smart contracts and provides a symbolic execution tool to detect bugs. _Securify_[21] describes all compliance and violation patterns in a designated domain-specific language, and conducts formal verification on smart contracts. Comparisons with the above methods are expected to validate the effectiveness and advantages of our approaches.