# DNS Firewall Based on Machine Learning
## 5 Data Analysis
### Machine Learning: Classification
To run each ML algorithm, we created a function that receives the dataset containing the best nine features using each one of the three feature selection/importance methods mentioned earlier in this section. To run the cross-validation process with the dependent and independent features the cross_validate function from SkLearn [48] was used. which allows to \"evaluate metric(s) by cross-validation and also record fit/score times\". To evaluate the effectiveness of the ML models we chose common metrics as follows:

* Accuracy: The accuracy is the ratio between the correctly predicted observations versus the total number of observations;
* Precision: The precision is the ratio between the correctly predicted positive observations and the total predicted positive observations;
* Recall: The recall is the ratio between the correctly predicted positive observations and all observations in the actual class (0 or 1);
* F1 score: The F1 score is the average of precision and recall and takes into account the false positives and false negatives;* Score time: Corresponds to the time for the test of the estimator on each cross-validation iteration. The source code used to evaluate the feature importance and the ML algorithms is publicly available in [49]. ## 6 Results

For consistency of results, the computational performance must be taken into account. The experiments were performed on a machine with the following characteristics:

* Processor: Intel Core i7-8565U @ 1.80 Ghz;
* Memory: SDRAM DDR4-2400 16 GB (16 GB \(\times\) 1);
* Graphics Processor: Intel UHD Graphics 620;
* Storage: 512 GB PCIe Gen 3 \(\times\) 4;
* Read up to 3100 MB/s;
* Write up to 2800 MB/s. The results for each feature importance method are described in the Tables 1-3. As mentioned in the previous section, the test mode selected was 10-fold-cross-validation and the number of features selected was nine. The features vary for each feature selection/importance method (feature importance, univariate selection and RFE). Table 1 presents the classification results achieved when considering the features with higher importance ( 'NumericRatio', 'ConseantRatio', 'NumericSequence', 'HasSPFInfo', 'TXTDnsResponse', 'DomainLength', 'VowelRatio', 'CreationDate' and 'StrangeCharacters') determined using the feature importance method. Considering the feature importance method, the ML algorithm with the highest accuracy (95%), recall (93%), and F1 score (94%) was the KNN. Although this algorithm has good results, the time to predict is above 1 s (1793), losing to the CART algorithm with a score time of 0.01 s. Furthermore, the precision was slightly higher for the CART algorithm. Table 2 presents the results when the univariate feature selection method was used. The top nine features achieved by this method were: 'HasSPFInfo', 'TXTDnsResponse', 'CreationDate', 'NumericRatio', 'LastUpdateDate', 'HttpResponseCode', 'ConseantRatio', 'MXDnsResponse' and 'StrangeCharacters'. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Algorithm** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **Time (sec)** \\ \hline SVM & 0.912 & 0.949 & 0.872 & 0.898 & 3.087 \\ \hline LR & 0.916 & 0.949 & 0.878 & 0.905 & 0.014 \\ \hline LDA & 0.908 & 0.950 & 0.861 & 0.894 & 0.012 \\ \hline KNN & 0.951 & 0.965 & 0.936 & 0.946 & 1.793 \\ \hline CART & 0.947 & 0.967 & 0.926 & 0.939 & 0.011 \\ \hline NB & 0.903 & 0.947 & 0.851 & 0.887 & 0.012 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Results using the Feature Importance feature selection method. From the results presented in Table 2, the algorithm with the best result was the CART. This algorithm shows a validation accuracy of 95%, with a precision of 96%. The recall reached 94% and the F1 score 95%. The lowest score time (0.013) is shared by the CART, LDA, and NB algorithm. Table 3 presents the results achieved by using the features with higher importance according the RFE method.