# A Comprehensive Analysis of Accuracies of Machine Learning Algorithms for Network Intrusion Detection

Anurag Das, Samuel A. Ajila, Chung-Horng Lung

► To cite this version:

Anurag Das, Samuel A. Ajila, Chung-Horng Lung. A Comprehensive Analysis of Accuracies of Machine Learning Algorithms for Network Intrusion Detection. 2nd International Conference on Machine Learning for Networking (MLN), Dec 2019, Paris, France. pp.40-57, 10.1007/978-3-030-45778-5_4. hal-03266472

HAL Id: hal-03266472

https://inria.hal.science/hal-03266472

Submitted on 21 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A Comprehensive Analysis of Accuracies of Machine Learning Algorithms for Network Intrusion Detection

Anurag Das, Samuel A. Ajila, Chung-Horng Lung

Department of Systems and Computer Engineering, Carleton University
1125 Colonel By Drive, Ottawa K1S 5B6, Ontario Canada

anuragdas@cmail.carleton.ca; {ajila, chung}@sce.carleton.ca

**Abstract.** Intrusion and anomaly detection are particularly important in the time of increased vulnerability in computer networks and communication. Therefore, this research aims to detect network intrusion with the highest accuracy and fastest time. To achieve this, nine supervised machine learning algorithms were first applied to the UNSW-NB15 dataset for network anomaly detection. In addition, different attacks are investigated with different mitigation techniques that help determine the types of attacks. Once detection was done, the feature set was reduced according to existing research work to increase the speed of the model without compromising accuracy. Furthermore, seven supervised machine learning algorithms were also applied to the newly released BoT-IoT dataset with around three million network flows. The results show that the Random Forest is the best in terms of accuracy (97.9121%) and Naïve Bayes is the fastest algorithm with 0.69s for the UNSW-NB15 dataset. C4.5 is the most accurate one (87.66%), with all the features considered to identify the types of anomalies. For BoT-IoT, six of the seven algorithms have a close to 100% detection rate, except Naïve Bayes.

**Keywords:** Network Intrusion Detection, Supervised Learning, UNSW-NB15 dataset, BoT-IoT dataset

## 1 Introduction

Due to the massive growth of computer networks and its many applications, the number of network flows has increased tremendously. The considerable large number of traffic flows leads to a massive amount of data, which eventually leads to the vulnerability of the data and network as a whole. One of the many challenges of cybersecurity research is to identify intrusion/anomaly in the traffic flow. A network Intrusion Detection System (IDS) is one of the solutions to detect such attacks before they compromise the network. An IDS monitors the normal traffic flows and identifies its characteristics or patterns. If a new flow does not follow the same characteristics, it might be an anomaly. Hence, an IDS may help identify even detect unknown attacks. Note that this paper uses intrusion and anomaly interchangeably.

This research is an experimental investigation of nine machine learning algorithms on the dataset UNSW-NB15 (released in November 2015) [1] and seven machine algorithms on the recently released BoT-IoT dataset (released in November 2018) [2]. This research intends to discuss the following questions:

1. *Network Intrusion Detection:* How effective is it to detect network intrusion based on the traffic flow features present in datasets using different machine learning techniques?

2. *Types of Intrusion Classification:* Different cyberattacks can be stopped by different mitigation techniques. Hence classification of attack is as important as the detection of
attacks. How effective can the classification of the types of attacks be from different features of network traffic flows present in datasets?

3. *Accuracy of models:* Which machine learning model has the highest accuracy for classifying the network anomalies for the selected datasets?

4. *Efficiency of models:* Which machine learning model is efficient for detecting network intrusion without compromising on accuracy? The earlier an attack is detected, the less harm it can generate on the network. Furthermore, by selecting a fewer number of features from the complete dataset, we can reduce the computation time a machine learning model takes to build.

The main contributions of the paper are: Firstly, comparing the accuracy and the time to build in the evaluation of network intrusion detection of the UNSW-NB15 dataset using nine machine learning techniques. Secondly, using the same nine machine learning techniques and nine different features selections, we compared and evaluated the performance of the various methods to identify the types of network intrusions in the UNSW-NB15 dataset. Thirdly, we analysed and evaluated the accuracy of and time to build seven machine learning techniques on the newly released BoT-IoT dataset. The premise upon which this research is based is to synthesis the previous research works on the UNSW-NB15 dataset [1–9]. Some (if not all) of related research works only used one or two machine learning algorithms to analyse the dataset, and in some cases do not even identify the different anomalies.

The rest of this paper is structured as follows. Section II presents background information about different supervised learning algorithms. Section III gives a literature review of previous research works and the different feature selection methods used in this paper. Section IV presents the two datasets used in this research. Section V describes the methodology and the three sets of experiments. Section VI provides the results and discussion and the conclusion is given in section VII.

# 2 Background

## 2.1 Supervised Learning Algorithms

Machine learning is the study of algorithms that can learn complex relationships or patterns from empirical data and make accurate decisions [10]. Machine learning can be classified into supervised learning, semi-supervised learning, and unsupervised learning. Supervised learning deduces a functional relationship from training data that generalizes well to the whole dataset. In contrast, unsupervised learning has no training dataset and the goal is to discover relationships between samples or reveal the latent variables behind the observations [11]. Semi-supervised learning falls between supervised and unsupervised learning by utilizing both labeled and unlabeled data during the training phase [10]. Among the three categories of machine learning, supervised learning is the best fit to solve the
prediction problem in the auto-scaling area [11]. Therefore, this research focuses on supervised learning.

After conducting an in-depth search and review of research papers that have previously used the UNSW NB15 dataset, we selected nine machine learning algorithms that appear frequently in different papers [1], [3], [7], and [8].

*Random Tree* is an algorithm with a random number of features at each node, and it is used for classification [12]. This algorithm is very fast, but it suffers from overfitting. To overcome overfitting, Random Forest is used with this algorithm. We used the WEKA [13] implementation of this algorithm in which the Random Tree classifier constructs a tree that considers K random chosen attributes at each tree node. There is no pruning and has an option to estimate classifier probabilities (or target mean in the case of regression) based on a hold-out set (i.e. back fitting). We set the seed to be 1, that is, the random number seed used for selecting attributes.

*Random Forest* is an ensemble learning algorithm which can be applied on classification as well as a regression problem [12]. In this technique, lots of decision trees are produced at training time. For a regression problem, the mean is considered, and for the classification problem, the mode is used. Random Forest was designed to combat the overfitting problem in the random tree. Random Forest is a classifier for constructing a “forest” of random trees.

*Bayesian Networks* (WEKA Bayes Net) [13] - These networks show the probabilistic relations between different features with the target attribute (one which is to be classified) [12]. In this research, this algorithm is used to calculate the probability of different features with an impact on the anomaly. The dual nature of a Bayesian network makes learning a Bayesian network a two stage processes: first learn a network structure, then learn the probability tables. All Bayes network algorithms implemented in Weka assume that all variables are discrete finite and no instances have missing value [14]. In general, Bayes Network learning uses various search algorithms and quality measures [13]. In our case, we used the K2 search algorithm and the SimpleEstimator for the estimate function [13].

*Naive Bayes* - These are traditional classifiers and they are based on the Bayes theorem of independent relation between the features [12]. Although it is an old technique, this algorithm is still highly scalable and it can be used to build the fastest model for large dataset such as UNSW NB15. These classifiers are family of simple probabilistic classifiers with strong (naïve) independence. The assumption here is that the features of measurement are independent of each other.

*k-Nearest Neighbours (k-NN)* - k-NN is an algorithm which can be used for both regression and classification [12]. The model consists of training k closest samples in the feature space. In classification, the class having the maximum number of k nearest neighbours is chosen. Weights are assigned to nearer neighbours that contribute more to the result compared to the ones that are farther away. It is an instance-based learning algorithm where all the calculations are deferred until the final regression/classification. Hence it is known as a lazy algorithm. This algorithm is called “IBk” in Weka [13]. It selects an appropriate value of k based on cross-validation and can also do distance weighting.
**C4.5** - It is a decision tree-based classifier [12]. It is an extension of the classical ID3 algorithm from the same author - Ross Quinlan [15]. It uses information entropy and gain for decision making. On the Weka platform [13], it is called J.48, which is an implementation of C4.5 in Java. J.48 generates a pruned (or unpruned) C4.5 decision tree. The seed in this classifier is used for randomizing the data when reduced error pruning is used.

**REPT** - Reduced Error Pruning Tree (REPT) is a fast decision tree based on the C4.5 algorithm and it can produce classification (for discrete outcome) or regression trees (for continuous outcome). It builds a regression/decision tree using information gain/variance and prunes it using reduced-error pruning with back-fitting). [12], [13]. Missing values are replaced by breaking down the corresponding instances.

**RIPPER** - Repeated incremental pruning to produce error reduction (RIPPER) is an inductive rule-based classifier which is the optimized version of incremental reduced error pruning [12]. It uses incremental reduced-error pruning and a somewhat complicated global optimization step. It makes rules for all features. Depending on the satisfaction of those rules, a network flow is classified as normal or an anomaly. On the Weka platform, it is called Jrip [13]. Generally, the Weka Jrip implements a propositional rule learner.

**PART** (Partial Decision Tree) - Here rules are made according to the features of the past observations and classification of whether the data is an anomaly or normal is done according to the rules [12]. The Weka [13] implementation builds a partial C4.5 decision tree in each iteration and makes the “best” leaf into a rule.

These nine algorithms are either tree-based or partial tree or forest (a collection of trees) or networks (a form of tree). We have set the “seed” to 1 and the batch size to 100 where needed.

# 3 Literature Review

A number of research efforts [1–8] [12] have been conducted for network anomaly or intrusion detection using the UNSW-NB15 dataset. These approaches have certain limitations. Some research papers considered one or two machine learning algorithms. For instance, only the research works in [4] and [12] use more than one machine learning technique on the UNSW-NB15 dataset. Furthermore, the following research works: [4], [5], [6] and [12] do not identify the types of attacks. They only detect if a flow is normal or an anomaly. In addition, the research work in [12] does not adopt a feature selection method. Research works in [1–3] and [7–8] do classify the attack types, but they only investigate a single machine learning technique.

In this paper, we investigate the detection, the types of attacks, and make a comparison between the effectiveness of nine different machine learning techniques as well as the impact of different feature selection techniques on those machine learning algorithms. Our
research, like [1] and [5], also does a benchmark of time taken for various machine learning techniques when applied together with specific feature selection methods.

The authors of [1] divided their network intrusion detection model into 3 stages. In the first stage, they applied Correlation-based feature selection on all the 45 features (as shown in Table 1) along with the genetic search. They used a statistical filter-based feature selection method on the complete dataset. Once they obtained the best features from stage 1, they applied a wrapper-based filter on those selected features only. The machine learning algorithm used in the wrapper-based filter was Random Forest. At the end of stage 2, the authors identified five best features, namely, sbytes, tcprrt, synack, dmean and response_body_len. In stage 3, they used the Random Forest classifier to detect the anomaly. They were able to improve the accuracy of the model from 94.70% to 99.63%. One problem in this approach is that only 5 out of 45 features were finally considered.

<table><caption>**Table 1.** Features for UNSW-NB15</caption><thead><tr><td>Feature Number</td><td>Feature Name</td><td>Feature Number</td><td>Feature Name</td></tr></thead><tbody><tr><td>1</td><td>id</td><td>23</td><td>dtcpb</td></tr><tr><td>2</td><td>dur</td><td>24</td><td>dwin</td></tr><tr><td>3</td><td>proto</td><td>25</td><td>tcprrt</td></tr><tr><td>4</td><td>service</td><td>26</td><td>synack</td></tr><tr><td>5</td><td>state</td><td>27</td><td>ackdat</td></tr><tr><td>6</td><td>spkts</td><td>28</td><td>Smean</td></tr><tr><td>7</td><td>dpkts</td><td>29</td><td>dmean</td></tr><tr><td>8</td><td>sbytes</td><td>30</td><td>trans_depth</td></tr><tr><td>9</td><td>dbytes</td><td>31</td><td>response_body_len</td></tr><tr><td>10</td><td>rate</td><td>32</td><td>ct_srv_src</td></tr><tr><td>11</td><td>sttl</td><td>33</td><td>ct_state_tls</td></tr><tr><td>12</td><td>dttl</td><td>34</td><td>ctdst_ltm</td></tr><tr><td>13</td><td>sload</td><td>35</td><td>ctdst_dport_ltm</td></tr><tr><td>14</td><td>dload</td><td>36</td><td>ctdst_sport_ltm</td></tr><tr><td>15</td><td>sloss</td><td>37</td><td>ctdst_src_ltm</td></tr><tr><td>16</td><td>dloss</td><td>38</td><td>is_ftp_login</td></tr><tr><td>17</td><td>sinpkt</td><td>39</td><td>ct_ftp_cmd</td></tr><tr><td>18</td><td>dinpkt</td><td>40</td><td>ctflw http_mthd</td></tr><tr><td>19</td><td>sjit</td><td>41</td><td>ct_srv_ltm</td></tr><tr><td>20</td><td>djit</td><td>42</td><td>ct_srv_dst</td></tr><tr><td>21</td><td>swin</td><td>43</td><td>is_SMS_ips_ports</td></tr><tr><td>22</td><td>stcpb</td><td>44</td><td>attack_cat</td></tr><tr><td></td><td></td><td>45</td><td>label</td></tr></tbody></table>
The authors of [2] were the original authors of UNSW-NB15. Their method for feature selection has three parts: feature conversion, feature reduction and feature normalization. After all the three steps, they selected the best features which are *Dttl*, *synack*, *swin*, *dwin*, *ct_state_tls*, *ct_src_ltm*, *ct_srv dst*, *Sttl*, *et_dst_sport_ltm*, and *Djit*. Association rule mining was also used to find features which are not correlated to each other, but highly correlated to the target attribute that the authors wanted to predict. They ranked all the 43 features (excluding id). From the ranking, they selected the top 25% (i.e., top 11 out of 43) features. Furthermore, Independent Component Analysis (ICA) was used to find the best features in [7].

The authors of [3] used the Weka tool [14] to select the optimal features. They used *CfsSubsetEval* (attribute evaluator) + *GreedyStepwise* method and *InfoGainAttributeEval* (attribute evaluator) + *Ranker* method. The classifier Random Forest was used to evaluate the accuracy. The combination of features which generated the highest accuracy was selected. The five selected features are *service*, *sbytes*, *sttl*, *smean*, *ct_dst_sport_ltm*. Their test accuracy was around 83% for anomaly type classification.

The authors of [5] used a pure statistical filter-based subset evaluation (FBSE) method of correlation-based feature selection to detect Denial of Services (DoS) attacks. The final features selected are F7, F10, F11, F12, F18 and F32 (see Table 1). They used Artificial Neural Network to detect the attacks and obtained an accuracy of 81.34%. Their false alarm rate was quite high with 21.13%.

The authors of [6] trained a deep learning model on the entire dataset for 10-fold cross-validation. The most important features were then selected using the Gedeon method [9]. Gedeon method selects features which are unique from one another even if the information they provide is minor. They discarded the features which generate huge amount of redundant information. The accuracy obtained from the proposed model was 98.99% with a low false alarm rate of 0.56%.

The authors of [8] used Genetic Algorithm to find the best features. They used Support Vector Machine (SVM) to check the accuracy of the selected features.

# 4 Datasets

Two datasets have been selected for our experimental validation. They are UNSW-NB15 and BoT-IoT, which are described in the following subsections.

## 4.1 UNSW-NB15

This dataset was created by Moustafa and Slay [10]. The UNSW-NB15 dataset is a mixture of real normal traffic flow and synthetic attacks. The types of attacks and the number of each attack in the testing dataset are shown in Fig. 1. The testing dataset has
82,332 records (37,000 normal and 45,332 anomalies) and 45 attributes or features (see TABLE 1)

<figure><img src="image_2.png" /><figcaption>Fig. 1. Distribution of anomalies (attack types) in the UNSW-NB15 training dataset</figcaption></figure>

## 4.2 The BoT-IoT Dataset

One of the original authors of UNSW-NB15 was also involved in creating the BoT-IoT dataset [11], as depicted in Fig. 2, in the Cyber Range Lab of UNSW Canberra Cyber Center. This dataset is a combination of standard and botnet traffic (hence the name). Attack distribution in the training dataset is depicted in Fig. 2. The training dataset has 2,934,817 records. The features used for the experiments were the top 10 features [11] selected by the creators of this dataset.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Distribution of anomalies (attack types) in BoT-IoT training dataset</figcaption></figure>

# 5 Methodology and Experiments

The specifications of the system environment for our experiments are shown in Table 2. Weka [14] tool was used for detection (training and testing), and the RStudio for data preprocessing with the R programming language. Microsoft Excel is used for data visualization. Three different experiment sets were conducted, which are described as follows.

<table><caption>Table 2. HARDWARE SPECIFICATIONS</caption><tr><td>Processor</td><td>Intel(R) Xeon(R) @ 2.50 GHz (2 processors)</td></tr><tr><td>RAM</td><td>32.0 GB</td></tr><tr><td>Operating System</td><td>Windows 7 - 64 bit OS</td></tr><tr><td>Architecture</td><td>Microarchitecture - Ivy Bridge, Multiprocessor (2 Processors)</td></tr></table>

**Experiment set 1:** This set of experiments is to detect if a network flow is normal or an anomaly using supervised learning techniques. The nine machine learning algorithms described above have been evaluated and compared for accuracy (%) and the time taken to build the model (in seconds). The dataset used for this experiment set is the UNSW-NB15 training dataset.

**Experiment set 2:** In this set of experiments, the type of network attacks is also identified using the nine supervised learning algorithms for validation. Further, eight different feature selection techniques and the complete dataset (making a total of nine different feature sets) together with the nine different machine learning algorithms making
81 different combinations of feature selection methods and machine learning techniques to identify the types of attack. The dataset used is the UNSW-NB15 training dataset.

**Experiment set 3:** The types of network attacks are identified using seven supervised machine learning algorithms on the ten best features pre-selected by the authors of the BoT-IoT training dataset [11].

In addition, 10-fold cross-validation has been adopted [13] in our experiments. The standard way of predicting the error rate of a learning technique given a single, fixed sample of data is to use stratified 10-fold cross-validation. The dataset is divided randomly into 10 parts in which the class is represented in approximately the same proportions as the full dataset. Each part is held out in turn and the learning scheme trained on the remaining nine-tenths; then its error rate is calculated on the holdout set. In the end, the average of all the iterations is calculated. As all the values have been tested at least once, this step helps in avoiding overfitting. Why 10? Previous extensive works in the domain have shown that the number 10 is about the right number of folds to get the best estimate of error.

## 5.1 Experiment set 1 - Supervised Learning on UNSW-NB15 to detect the anomaly

Our first experiment-set used nine supervised learning algorithms on the UNSW-NB15 training dataset to detect the anomaly. The methodology in this experiment is that all the attributes (i.e. features) are considered and all nine machine learning algorithms are used to build anomaly detection models. The machine learning algorithms are Random Forest, Random Tree, Bayes Network, Naive Bayes, k-NN, C4.5, Reduced Error Pruning Tree, RIPPER and PART. The experimental results are presented in Table 3.

<table><caption>**Table 3.** Experiment I Results</caption><thead><tr><th>Machine Learning Algorithms</th><th>Accuracy (%)</th><th>False Positive Rate (%)</th><th>Precision (%)</th><th>Recall (%)</th><th>Time to build the model (s)</th></tr></thead><tbody><tr><td>Random Forest</td><td>97.9121</td><td>2</td><td>97.9</td><td>97.9</td><td>57.25</td></tr><tr><td>Random Tree</td><td>96.1036</td><td>4</td><td>96.1</td><td>96.1</td><td>0.93</td></tr><tr><td>Bayes Network</td><td>81.6961</td><td>17.2</td><td>82.7</td><td>81.7</td><td>4.93</td></tr><tr><td>Naive Bayes</td><td>76.1952</td><td>21.4</td><td>79.1</td><td>76.2</td><td>0.69</td></tr><tr><td>k-nearest neighbours</td><td>93.4691</td><td>6.5</td><td>93.5</td><td>93.5</td><td>1.51</td></tr><tr><td>C4.5</td><td>97.3194</td><td>2.7</td><td>97.3</td><td>97.3</td><td>15.63</td></tr><tr><td>REPT</td><td>97.068</td><td>2.9</td><td>97.1</td><td>97.1</td><td>3.43</td></tr><tr><td>RIPPER</td><td>96.7582</td><td>3.2</td><td>96.8</td><td>96.8</td><td>185.36</td></tr><tr><td>PART</td><td>97.3109</td><td>2.7</td><td>97.3</td><td>97.3</td><td>53.69</td></tr></tbody></table>
As presented in Table 3, in terms of accuracy, Random Forest is the most accurate anomaly detection model with 97.91% closely followed by C4.5 (97.3194%), then PART (97.3109). Naïve Bayes is the least accurate considering the nine algorithms. Judging by the false positive rates and the precision (Table 3), there are little or no significant statistical differences between the following algorithms in terms of accuracy – Random Forest, Random Tree, C4.5, REPT, RIPPER, and PART. In terms of speed, that is, time to build, Naïve Bayes is the fastest model with 0.69 seconds although the least accurate. Random Tree is equally fast with a build time of just 0.93 seconds and gives a high accuracy of 96.10%. Table IV shows the confusion matrix of the Random Forest.

<table><caption>**Table 4.** Confusion Matrix of the Random Forest Algorithm</caption><thead><tr><th colspan="2">Classified as</th><th></th></tr></thead><tbody><tr><td>Normal</td><td>Anomaly</td><td></td></tr><tr><td>36354</td><td>646</td><td>Normal</td></tr><tr><td>1073</td><td>44259</td><td>Anomaly</td></tr></tbody></table>

The diagonal in green indicates the correct classification. There are total of 37,000 normal flows and 45,332 anomalies in the UNSW-NB15 training dataset. Out of 37,000 normal, 36,354 were classified normal (98.25%) correctly, but 646 observations were classified anomaly incorrectly. Out of 45,332 anomalies, 1073 were classified normal incorrectly, but 44,259 were classified anomaly (97.63%) correctly.

## 5.2 Experiment set 2 - Supervised Learning on UNSW-NB15 to detect the anomaly type

Experiment set 2 was conducted to identify not only if a network traffic flow is normal or an anomaly, but also the types of anomaly. The objective is to support an appropriate action to mitigate it. The methodology for Experiment-set 2 is depicted in Fig. 3.

This experiment set 2 has nine sub-experiments with different feature sets. The experiments are based on methods previously published in the literature [1-8]. The features for the nine sub-experiments are described as follows
<figure><img src="image_4.png" /><figcaption>Fig. 3. Experiment 2 Methodology</figcaption></figure>

1. All the features of the UNSW-NB15 training dataset are considered.

2. FBSE+WBSE - Features selected by FBSE and wrapper WBSE. This experiment uses features suggested in [1].

3. **PCA + Feature Normalization** - Features selected by principal component analysis (PCA) and feature normalization are considered. This experiment is based on the features selected by [2].

4. **Weka Feature Selection** - Optimal features of UNSW-NB15 were selected using the Weka tool. This method was proposed in [3].

5. **Association Rule Mining** - Feature selection based on association rule mining. This experiment is based on work done in [4].

6. **Correlation based Feature Selection** - Pure statistical feature selection method based on correlation was used to select features. This experiment is based on [5], [16].

7. **Deep Learning with 10-fold CV** - The authors of [6] selected the best features which generated the highest accuracy for their deep learning model after 10-fold cross-validation [17–19]. These features were used in this sub-experiment.

8. **ICA** - Features used in this experiment were based on ICA proposed in [7].

9. **Genetic Algorithm with SVM** - Feature selection was done using the genetic algorithm. The classifier used in the genetic algorithm to check the highest accuracy was SVM. This methodology was proposed in [8].
<table><caption>**Table 5.** ACCURACIES OF DIFFERENT ALGORITHMS FOR DIFFERENT FEATURE SETS FOR UNSW-NB15</caption><thead><tr><th>Feature Sets</th><th>Random Forest</th><th>Random Tree</th><th>Bayes Network</th><th>Naive Bayes</th><th>k-NN</th><th>C4.5</th><th>REPT</th><th>RIPPER</th><th>PART</th></tr></thead><tbody><tr><td>All Features</td><td>87.08</td><td>84.17</td><td>65.28</td><td>46.16</td><td>80.62</td><td><strong>87.66</strong></td><td>86.62</td><td>80.24</td><td>87.05</td></tr><tr><td>FBSE+WBSE</td><td>82.85</td><td>80.92</td><td>74.25</td><td>17.95</td><td>76.59</td><td>82.56</td><td>82.33</td><td>76.67</td><td>82.30</td></tr><tr><td>PCA + Feature Normalization</td><td>85.85</td><td>83.48</td><td>71.55</td><td>41.87</td><td>81.27</td><td>85.78</td><td>85.13</td><td>79.58</td><td>85.65</td></tr><tr><td>Weka Feature Selection</td><td>82.99</td><td>82.85</td><td>74.55</td><td>57.57</td><td>82.5</td><td>83</td><td>82.8</td><td>76.44</td><td>82.9</td></tr><tr><td>Association Rule Mining</td><td>77.70</td><td>75.18</td><td>62.72</td><td>51.07</td><td>77.04</td><td>78.22</td><td>77.99</td><td>72.69</td><td>77.83</td></tr><tr><td>Correlation based Feature Selection</td><td>74.31</td><td>71.93</td><td>60.32</td><td>43.05</td><td>73.59</td><td>74.58</td><td>74.28</td><td>71.15</td><td>74.3</td></tr><tr><td>Deep Learning with 10-fold CV</td><td>85.57</td><td>83.8</td><td>66.55</td><td>56.59</td><td>84.17</td><td>86.16</td><td>85.10</td><td>79.24</td><td>85.91</td></tr><tr><td>ICA</td><td>85.68</td><td>83.59</td><td>74.27</td><td>37.87</td><td>80.12</td><td>85.31</td><td>84.86</td><td>77.6</td><td>85.03</td></tr><tr><td>Genetic Algorithm with SVM</td><td>77.35</td><td>74.18</td><td>69.68</td><td>35.83</td><td>71.67</td><td>76.05</td><td>76.80</td><td>71.86</td><td>75.78</td></tr></tbody></table>

Table 5 shows the experimental results. C4.5 generates the best accuracy for All Features (87.66%) followed by Random Forest (87.08%) then comes PART (87.05%) and the rest are REPT (86.62%), Random Tree (84.18%), k-NN (80.62%), Bayes Network (65.28%) and finally Naïve Bayes (46.16%).

Four algorithms (Random Forest, C4.5, REPT, and PART) are on par (roughly 83%) for FBSE+WBSE [2] and the rest are below 80% with Naïve Bayes as the worst (17.955%).

In the case of PCA + Feature Normalization [2], Random Forest produces the best accuracy (85.85%), followed by C4.5 (85.78%), PART (85.65%), REPT (85.13%), then Random Tree, k-NN, RIPPER and finally Bayes Network.

For Weka Feature Selection [3], six algorithms have roughly the same accuracies of around 83%. The maximum accuracy for Association Rule Mining [4] is 78.22% for C4.5 and the minimum is 51.07% for Naïve Bayes.

The accuracy results for Correlation based Feature Selection [5] ranges from 74.58% for C4.5 to 43.05% for Naïve Bayes. Random Forest has the best accuracy for ICA [7].

All the accuracy measurements for the Genetic Algorithm with SVM [8] are below 80% with the highest as 77.35% for Random Forest and the minimum is 35.83% for Naïve Bayes.

In general, Random Forest came out to be the top in terms of accuracy for the feature sets for UNSW-NB15. This is closely followed by C4.5.
## 5.3 Experiment set 3 - Supervised Learning on BoT-IoT to detect the anomaly type

Bot-IoT training dataset [11] was used for experiment-set 3. It has around 3 million values. Only the top 10 features according to the original dataset authors were selected for this experiment set. Unfortunately, the Weka tool crashed for k-NN and RIPPER algorithms. It was unable to build models for these algorithms. This probably is due to the size of the dataset. Table 6 shows the results for the seven remaining algorithms and almost all the algorithms have around a 100% detection rate, except Naïve Bayes. These results are like that of the authors of [10]. Random Tree has high accuracy with a minimal build time of 96.98 seconds. Table 7 shows the confusion matrix of Random Tree for the BoT-IoT training dataset with 10-fold Cross-Validation.

<table><caption>**Table 6.** Accuracies of Different Algorithms for Different Feature Sets for BoT-IoT</caption><thead><tr><th>Algorithms</th><th>Accuracy %</th><th>False Positive %</th><th>Precision %</th><th>Recall %</th><th>Time to build (seconds)</th></tr></thead><tbody><tr><td>Random Forest</td><td>99.99</td><td>0</td><td>100</td><td>100</td><td>5628.96</td></tr><tr><td>Random Tree</td><td><strong>99.9937</strong></td><td>0</td><td>100</td><td>100</td><td><strong>96.98</strong></td></tr><tr><td>Naïve Bayes</td><td><strong>73.4121</strong></td><td>2.8</td><td>73.4</td><td>71.1</td><td><strong>11.46</strong></td></tr><tr><td>C4.5</td><td>99.99</td><td>0</td><td>100</td><td>100</td><td>448.57</td></tr><tr><td>REPT</td><td>99.99</td><td>0</td><td>100</td><td>100</td><td>205.96</td></tr><tr><td>Bayes Network</td><td>99.6</td><td>0.2</td><td>99.6</td><td>99.6</td><td>228.6</td></tr><tr><td>PART</td><td>99.99</td><td>0</td><td>100</td><td>100</td><td>1210.6</td></tr></tbody></table>

<table><caption>**Table 7.** Confusion Matrix for BoT-IoT</caption><thead><tr><th colspan="6">Classified as</th></tr><tr><th>Normal</th><th>DDoS</th><th>DoS</th><th>Reconnaissance</th><th>Theft</th><th></th></tr></thead><tbody><tr><td>349</td><td>1</td><td>6</td><td>12</td><td>2</td><td>Normal</td></tr><tr><td>2</td><td>1541262</td><td>48</td><td>3</td><td>0</td><td>DDoS</td></tr><tr><td>4</td><td>54</td><td>1320083</td><td>6</td><td>1</td><td>DoS</td></tr><tr><td>14</td><td>13</td><td>12</td><td>72878</td><td>2</td><td>Reconnaissance</td></tr><tr><td>1</td><td>0</td><td>0</td><td>3</td><td>61</td><td>Theft</td></tr></tbody></table>

The green diagonal shows the correct classification. As illustrated in Fig. 2, there are total of 370 normal flows, 1,541,315 DDoS, 1,320,148 DoS, 72,919 Reconnaissance, and 65 Theft in the BoT-IoT dataset. From the 370 normal flows, 349 (94.32%) were classified correctly. Out of 1,541,315 DDoS, 1,541,262 (99.99%) were classified correctly. For DoS 1,320,083 (99.9%) of 1,320,148 were classified correctly. 61 (93.85%) out of 65 theft were identified successfully.
# 6 Results and Discussions

In Experiment-set 1, nine machine learning algorithms have been evaluated and compared on UNSW-NB15 for network intrusion detection. In addition, the types of network intrusion are identified as well.

In Experiment-set 2, 81 (nine machine learning algorithms with eight different feature selection methods and all features together) different techniques have been compared. To the best of our knowledge, this is the first time all these nine methods are compared for the same dataset.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Comparison of accuracy for different machine learning algorithms</figcaption></figure>

Benchmarking has also been conducted on time taken to build each model [20–25].

Fig. 4 displays the results of the comparison of different machine learning algorithms for anomaly detection on all the UNSW-NB15 dataset features. Random Forest provides the best accuracy (97.91%) but very costly in terms of computational time (57.25 sec).

According to Fig. 5, Naive Bayes is the fastest algorithm for all the features. Random Tree is the most optimal algorithm with a high accuracy of 96.10% and second fastest with a build time of only 0.93 s.

Fig. 6 is the comparison of all the different feature selection methods used from the existing literature that are applied to Random Forest to detect the anomaly.

When all features are used, Random Forest generates the best accuracy. For feature selection, PCA + Feature Normalisation, Deep Learning + 10-fold cross-validation and ICA all are valid regarding high accuracy.
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Comparison of build time for different machine learning algorithms</figcaption></figure>

According to Fig. 7, FBSE + WBSE, Weka feature selection is the fastest with around 19 seconds to build using the Random Forest algorithm. PCA + Feature Normalisation is a balanced approach for feature selection. It has the second highest accuracy with 85.85% and third fastest with 26.32 Seconds.

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Comparison of accuracies for feature selection methods on the Random Forest algorithm</figcaption></figure>

According to Fig. 8, almost all the algorithms have around 100% detection accuracy for the BoT-IoT. The exception is Naïve Bayes with 73.4121%.

Fig. 9 shows that the time taken to build the model is very high for Random Forest. Naïve Bayes has the shortest time.
<figure><img src="image_8.png" /><figcaption>Fig. 7. Comparison of build time for feature selection methods on the Random Forest algorithm</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Comparison of accuracies for different machine learning algorithms on BoT-IoT dataset</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9. Comparison of computational time for different machine learning algorithms on BoT-IoT dataset</figcaption></figure>
# 7 Conclusion and Future Work

We asked four questions at the beginning of this paper. The answers or the explanation to these questions are examined below.

**Question 1 - How effective is it to detect network intrusion based on the traffic flow features present in datasets using different machine learning techniques?**

As shown in Table 3 and Fig. 4, different machine learning algorithms give different accuracies for the UNSW-NB15 dataset. Firstly, this shows that the choice of a machine learning technique for anomaly detection is crucial based on the dataset. Secondly, there is a set of machine learning algorithms that have similar “accuracies” judging by the false positive rates and precisions which means that if speed is important one can choose a machine learning technique among others based on time to build the model.

**Question 2 - How effective can the classification of the types of attacks be from different features of network traffic flows present in datasets?** Table 5 shows the results of using the nine algorithms for nine different feature selection sets (81 different sub experiments). From this table, we can see that the set “All Features” shows “better accuracy” compared to other feature selections supposedly contains the best features. These results are somewhat different from previous research works (for example in papers [2] to [8]) that used feature selections. The only conclusion we can make here is that if timing (i.e., speed) is not important, then it may be better to use all the features if an efficient algorithm is used.

**Question 3 - Which machine learning model has the highest accuracy for classifying the network anomalies for the selected datasets?** To answer this question, there is no single machine learning algorithm that can be tagged as “the best” for classifying anomalies. However, based on the “nature” of the data and, these two datasets (UNSW-NB15 and BoT-IoT) the tree-based algorithms appear to perform better than non-tree based. In Table 3, other than k-NN; Bayes Network and Naïve Bayes are less accurate compared to the rest of the algorithms.

**Question 4 - Which machine learning model is efficient for detecting network intrusion without compromising on accuracy?** So far as this research is concerned and based on the dataset (i.e., UNSW-NB15), the best algorithm that does not compromise on accuracy is Random Tree (Table 3) with 96.9121% accuracy and a time of 0.93 seconds. The second is REPT with 97.3109% and 3.43 seconds. Other algorithms (Random Forest, PART, RIPPER, etc.) have higher accuracies but not that efficient in terms of build time.

In conclusion, in this research, supervised learning techniques were used to detect anomaly in the UNSW-NB15 dataset. Although Random Forest has the best accuracy (97.9121%) and Naïve Bayes is the fastest, Random Tree is the most optimal algorithm with an accuracy of 96.10% and the second fastest with a build time of only 0.93 seconds. Furthermore, supervised learning techniques were used to classify the types of attacks as well. C4.5 is the most accurate one (87.66%) with all the features considered. Eight different types of feature selection methods were used from existing literature to investigate the accuracy and timing of each model. PCA + Feature Normalisation [2] is a balanced
approach for feature selection; it has the second highest accuracy with 85.85% and the third fastest with 26.32 seconds. Supervised Learning techniques were used on the BoT-IoT dataset to classify anomaly types. Random Tree is an optimal algorithm with almost perfect accuracy, and it is the second fastest one with just 96.98 seconds taken to build a model for 3 million network flows.

For future work, various feature selection methods can be applied to supervised learning algorithms. Weka can also be used for more feature selection methods. GPUs or distributed systems can be used to ease the computation burden. Unsupervised Learning algorithms can be applied to the BoT-IoT datasets. Deep learning models like Convolutional Neural Networks and Recurrent neural networks can be trained and compared to traditional machine learning algorithms regarding accuracy and speed.

## References

[1] N. Moustafa, J. Slay and G. Creech, "Novel Geometric Area Analysis Technique for Anomaly Detection using Trapezoidal Area Estimation on Large-Scale Networks", IEEE Transactions on Big Data, pp. 1-1, 2017.

[2] N. Koroniotis, N. Moustafa, E. Sitnikova, B. Turnbull, "Towards the Development of Realistic Botnet Dataset in the Internet of Things for Network Forensic Analytics: Bot-IoT Dataset", https://arxiv.org/abs/1811.00701, 2018.

[3] T. Janarthanan and S. Zargari, "Feature selection in UNSW-NB15 and KDDCUP'99 datasets", Proc. of IEEE 26th Int'l Symp. on Industrial Electronics, 2017.

[4] N. Moustafa; J. Slay, "A hybrid feature selection for network intrusion detection systems: Central points", Proc. of the 16th Australian Information Warfare Conf., Nov. 2015.

[5] M. Idhammad, K. Afdel and M. Belouch, "DoS Detection Method based on Artificial Neural Networks", Int'l Journal of Advanced Computer Science and Applications, 8 (4), 2017.

[6] M. Al-Zewairi, S. Almajali and A. Awajan, "Experimental Evaluation of a Multi-layer Feed-Forward Artificial Neural Network Classifier for Network Intrusion Detection System," Proc. of Int'l Conf on New Trends in Computing Sciences, 2017, pp. 167-172.

[7] H. Gharaee and H. Hosseinvand, "A new feature selection IDS based on genetic algorithm and SVM," Proc. of the 8th Int'l Symp. on Telecommunications, 2016, pp. 139-144.

[8] N. Moustafa, and J. Slay, "UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)." Proc. of IEEE Military Communications and Information Systems Conf. (MilCIS), 2015.

[9] N. Moustafa and J. Slay, "The evaluation of network anomaly detection systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set," Information Security Journal, vol. 25, no. 1, pp. 18-31.

[10] A. Y. Nikravesh, S. A. Ajila, C.-H. Lung, "An Autonomic Prediction Suite for Cloud Resource Provisioning", Journal of Cloud Computing, 6 (3), Feb 2017, pp. 1-20.
[11] S. A. Ajila and A. A. Bankole, "Using Machine Learning Algorithms for Cloud Prediction Models in a Web VM Resource Provisioning Environment," Trans. on Machine Learning and Artificial Intelligence, 4 (1), pp 29-51, 2016.

[12] S. Marsland, *Machine Learning: An Algorithmic Perspective*, 2nd Edition, Chapman and Hall/CRC, 2014.

[13] Weka, https://www.cs.waikato.ac.nz/ml/weka/ Access June, 2019.

[14] R. R. Bouckaert, *Bayesian Network Classifiers in Weka* for Version 3-5-7, My 12, 2008, https://www.cs.waikato.ac.nz/~remco/weka.bn.pdf, Access June 2019

[15] J.R. Quinlan, "Induction of Decision Trees", *Machine Learning*, 6 Kluwer Academic Publishers, Boston, 1986.

[16] H. Nguyen, K. Franke and S. Petrovic, "Improving Effectiveness of Intrusion Detection by Correlation Feature Selection," Proc. of Int'l Conf. on Availability, Reliability and Security, 2010, pp. 17-24.

[17] N. Shone, T. N. Ngoc, V. D. Phai and Q. Shi, "A Deep Learning Approach to Network Intrusion Detection," IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 1, pp. 41-50, Feb. 2018.

[18] M. S. Pervez and D. M. Farid, "Feature selection and intrusion classification in NSL-KDD cup 99 dataset employing SVMs," Proc. of Int'l Conf on Software, Knowledge, Information Management and Applications, 2014, pp. 1-6.

[19] H. T. Nguyen, K. Franke and S. Petrovic, "Towards a Generic Feature-Selection Measure for Intrusion Detection," Proc. of Int'l Conf on Pattern Recognition, 2010, pp.1529-1532.

[20] A. Zainal, M. A. Maarof and S. M. Shamsuddin, "Feature Selection Using Rough Set in Intrusion Detection," TENCON IEEE Region 10 Conference, Hong Kong, 2006, pp. 1-4.

[21] Z. Muda, W. Yassin, M. N. Sulaiman and N. I. Udzir, "Intrusion detection based on K-Means clustering and Naïve Bayes classification," Proc. of 7th Int'l Conf on Information Technology in Asia, 2011, pp. 1-6.

[22] S. Kumar and A. Yadav, "Increasing performance Of intrusion detection system using neural network," Proc. of IEEE Int'l Conf on Advanced Communications, Control and Computing Technologies, 2014, pp. 546-550.

[23] B. Ingre and A. Yadav, "Performance analysis of NSL-KDD dataset using ANN," Proc. of Int'l Conf on Signal Processing and Communication Eng. Systems, Guntur, 2015, pp. 92-96.

[24] T. Garg and S. S. Khurana, "Comparison of classification techniques for intrusion detection dataset using WEKA," Proc. of Int'l Conf on Recent Advances and Innovations in Engineering, 2014, pp. 1-5.

[25] N. Paulauskas and J. Auskalnis, "Analysis of data pre-processing influence on intrusion detection using NSL-KDD dataset," Proc. of Open Conf. of Electrical, Electronic and Information Sciences (eStream), 2017, pp. 1-5.