# HinPhish: An Effective Phishing Detection Approach Based on Heterogeneous Information Networks
## 5 Experiments Evaluation and Comparing
### Experiment 4: Comparison of HinPhish with Machine Learning-Based Methods  
As mentioned earlier, many studies have applied different features, including URLs, webpages, or features provided by third-party services, in order to detect phishing websites by machine learning algorithms. In this section, we reproduce three representative works [18; 45; 46]. In order to better compare with the proposed method, we used the same data set. It is worth noting that some webpages were no longer accessible, especially given that the survival time of phishing websites is typically very short. Thus, we extracted the webpage features from the already saved webpage files in order to better ensure the consistency of data characteristics. Table 6 lists the performances of the considered methods. The three works all used a few typical URL features and page features, such as domain length, https token, and hyperlinks in the page. In addition, [45; 46] also integrated the characteristics of some third-party services, such as WHOIS, PageRank, and Google Index. They differed from HinPhish, in that [45; 46] processed the features into binary features according to statistical laws, while HinPhish retains non-binary features. From the experimental results, it can be seen that all of the methods achieved satisfactory detection results. On our data set, among the three methods, that of [45] was the worst-performing, with an accuracy of 0.9154, while [46] had an accuracy of 0.9797. Our proposed approach performed slightly better, achieving an accuracy of 0.9856. In addition, we also provide the ROC curves of the considered methods and HinPhish, as shown in Figure 9. As they all showed an excellent performance, the curves quickly rose to a very high true positive rate. We carried out a ten-fold cross validation in all the experiments, so the ROC curves  
Figure 8: Comparison of HinPhish with Search Engine-based Methods. are very smooth. Among all the methods, HinPhish had the highest AUC value, indicating that it had the best performance. This further confirms our previous conclusion. It is worth mentioning that [PERSON] et al. [18] proposed that adding URL length as a feature can significantly improve the detection accuracy, reaching more than 0.99. We observed a similar situation on our data set; that is, consideration of the URL length can improve the accuracy of classification. Further exploration found that such results may have been caused by the bias of the data set. In Figure 10, we show the frequency curves of the URL lengths of the legitimate and phishing websites in our data set. In addition, the 500,000 URLs obtained from the pages of Alexa top 10 K revealed the frequency distribution of URL length of more general legitimate websites, marked as \"Extend\" in the image. Obviously, in our data set, there was a large difference between the URL length distribution of legitimate websites and phishing websites, which also explains why using the URL length as a feature significantly improved the accuracy. At the same time, we also observed a large difference in URL length distribution between our legal websites and more general legal websites. In fact, the difference in URL length between legal websites and phishing websites is not so obvious. In other words, there is a deviation between our data set and general actual websites, in terms of URL length. Therefore, we did not incorporate the implementation of [18], considering the URL length, into our comparison. The considered machine learning-based methods can achieve quite good performances; however, they also have some insurmountable drawbacks. First, these methods need to acquire some features from third-party services. This requires a large number of query requests, which may increase the network burden. Even worse, these features have become increasingly difficult to acquire, due to an increased awareness of privacy protection. Moreover, for these detection methods using clear features, attackers can make targeted changes to bypass detection.