# DNS Firewall Based on Machine Learning
## 5 Data Analysis
### Machine Learning: Classification  
The ML process was divided into two steps: feature selection/importance and ML algorithm classification. In the first step, after the normalization of the dataset, we used three different methods to assess the features that have more weight for the analysis. The feature selection process allows a better understanding of the data, better understanding of the model, and reduces the number of input features while maximizing the classification accuracy results. There are numerous methods of feature selection/importance for different scopes. Considering the state of the art, our dataset, and type of analysis, we explored the following feature selection methods:  
* Feature importance: This method uses the Extra Trees Classifier algorithm to fit and return the importance of the features;
* Univariate selection: Using a mathematical approach based on statistics this method works by picking up the best features on univariate statistical tests;RFE: According to [31], this method fits a model and deletes the feature (or features) with the lowest importance until the specified number of features is reached. As described in [32], the behavior and performance of a feature selection algorithm depends strongly on the classification rule, sample size, and feature/label distribution. The feature importance method makes use of the Extra Trees Classifier algorithm to fit and return the importance of the features. The Code Snippet 1 was used to identify the most significant features. ```
importpandasaspd fromsklearn.ensembleimportExtraTreesClassifier ####Parameters:#X-Independentvariables#y-Dependentvariable#feature_number-predefinednumberoffeaturestobeselected## deffeature_importance(X,y,feature_number): model=ExtraTreesClassifier() model.fit(X,y) feat_importances=pd.Series(model.feature_importances_, index=X.columns) returnfeat_importances.nlargest(feature_number).axes[0].values
```  
Code Snippet 2: Univariate Selection function  
```
importpandasaspd fromsklearn.feature_selectionimportSelectKBest####Parameters:#X-Independentvariables#y-Dependentvariable#feature_number-predefinednumberoffeaturestobeselected



this case the Decision Tree Regressor was used) that will iterate and seek for the lowest feature importance and exclude it. The recursive method stops when the number of features left is equal to the desired number of features selected. After the definition of the parameters, the model is fitted and transformed using the method transform that allows reducing \(X\) to the selected features. Similarly to the previous method, a data-frame was created to save each attribute and the associated importance. The code is presented in the Code Snippet 3. ```
importpandasaspd fromsklearn.treeimportDecisionTreeRegressor fromsklearn.model_selectionimportStratifiedKFold fromsklearn.feature_selectionimportRFECV ###Parameters:#X-Independentlyvariables #y-Dependentvariable #feature_number-predefinednumberoffeaturestobeselected # defrfe_cross_validation(X,y,feature_number): rfecv=RFECV(estimator=DecisionTreeRegressor(),step=1, cv=StratifiedKFold(10),verbose=1, min_features_to_select=feature_number, n_jobs=4) rfecv.fit(X,y) rfecv.transform(X) dest=pd.DataFrame() dest['attr']=X.columns dest['importance']=rfecv.estimator_.feature_importances_ dest=dset.sort_values(by='importance',ascending=False) returndset.attr.head(feature_number).values
```

The identification of the most significant features is an important step in the analysis especially when the number of features is high. It is necessary to remove the features with less importance to feed the ML algorithm with only the most important features, to avoid over-fitting, to decrease the training time, and to maximize the accuracy results. Once the most significant features are identified, the next step is the application of ML algorithms to determine the classification accuracy, precision, recall, F1 score, and the score time (seconds) achieved per algorithm. The algorithms used for the evaluation are presented below.