# Investigating the Influence of Feature Sources for Malicious Website Detection
## 5 Experimental Results--GAWAIN
### Model Performance  
The final results of training on each feature category with XGBoost and the rest of the models can be found in Table 4. Note that the feature categories are described in Table 1. We also designated a combination of lexical, host-based and content-based features as _Baseline_ category. Most of these features are part of other studies, and therefore their unity should carry the most relevant information for the problem. The accuracy of these feature sets was evaluated in two ways. The first is Individual accuracy, which is the test accuracy after training XGBoost on each feature category separately. The second is combined accuracy, which is the test accuracy after training on each feature category combined with the preceding feature category in Table 4. For example, under the content embeddings--PCA 10, that feature is studied in combination with Image embeddings--PCA (the best performing highlighted feature), URL embeddings--PCA 10 (for the same reason mentioned earlier), and the baseline. The combination of features follows a cascade style additions, that is, once we have the established the best performance for feature categories \(f_{1},\ldots,f_{k}\) together, we proceed to add the next category \(f_{k+1}\). For example, in Table 4, the _Host-based_ combined accuracy refers to combination of _Lexical + Host-based_ features, then _Content-based_ features are added on top of _Lexical + Host-based_ combination. The Longformer results are a batch of experiments where each one is added separately on top of _Baseline_ combination, with the highlighted score selected as the representative score for the batch, and in this case it is _PCA 10_ variation. This score _URL embeddings [Longformer] PCA 10_ is added to the _Baseline_ before moving to the next batch of scores involving image embeddings. The process is repeated until we have no feature categories left, with the random order of embeddings to add. Inspecting the Individual accuracy result in Table 4, the top feature categories are the _URL embeddings_ (both Longformer and Distilbert) and _Content-based_ with test accuracy of more than 76%. We see that from two distinct sources we are able to achieve the similar score, where _Content-based_ features should contain the necessary information for more precise classification, in principle, as opposed to the URL related categories. This does align with certain studies showing the importance of pure URLs as the main information carrier. This is exemplified by the results where the URL embeddings have slightly higher performance compared to the manually engineered _Lexical_ features. We also notice that embedding the content itself loses a lot of performance, and that embeddings themselves are not capturing enough context that we are able to extract manually, which is the opposite outcome when comparing against the _Lexical_ category. \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline \hline
**Parameter Name** & **Assigned Value** \\ \hline Max Depth & \{3, 4, 5, 6, 7, 8, 9, 10\} \\ \hline Minimum child weight & \{1, 5, 10\} \\ \hline Number of estimators & \{100, 120, 150, **165**, 180, 200, 300, 1000\} \\ \hline Colsample by tree & \{0.6, 0.8, **1.0**\} \\ \hline Learning rate & \{0.1, 0.2, **0.3**, 0.4, 0.5\} \\ \hline Tree method & Auto, **Exact**, Approx, Hist, GPU-hist \\ \hline Booster & Gbtree, Gblinear, **Dart** \\ \hline Gamma & \{1, 10\({}^{-1}\), 10\({}^{-2}\), 10\({}^{-4}\), 10\({}^{-6}\), 10\({}^{-8}\), **10\({}^{-10}\)**} \\ \hline \hline \end{tabular}
\end{table} 
Table 3: XGBoost hyperparameters and tested values. Looking at the combined results, the _Baseline_ combination captures most of the information from the features and sets the bar at 83.11%. This number represents the performance of the most commonly used features in other studies combined, and it immediately stands out that the result is quite a margin away from other published results (above 90%).