## 6 Performance Evaluation
### Machine Learning Algorithms' Implementation and Validation
#### 6.2.2 Experimental Results  
To evaluate the efficiency of our models, we employed the use of Area Under the Curve-Receiver Operating Characteristics (AUC-ROC). The AUC-ROC was selected because our model tries to classify between two classes (anomalous and benign), and the AUC-ROC has the ability to perform binary classification. The ROC plots a graph of the True Positive Rate (TPR) against the False Positive Rate (FPR) at various thresholds. Both TPR and FPR are defined in the Equations (4) and (5):  
\[\text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{4}\]  
\[\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}} \tag{5}\]  
where _TP_ stands for true positive, _FN_ stands for false negative, _FP_ stands for false positive, and _TN_ stands for true negative  
\[\text{AUC}=\int_{0}^{1}\text{TPR}(t)\,d\text{FPR}(t) \tag{6}\]  
The AUC measures the area under the ROC and provides a value that shows how well the model is able to distinguish between positive and negative classes. When the value is equal to 1, it has perfectly classified the object; instead, when the value is equal to 0.5, it performs a random guess; finally, when the value is less than 0.5, it performs worse than a random guess, and we can assume the model is confused and not learning any useful pattern. The AUC is calculated using the equation in Equation (6), where \(t\) is the decision threshold.  
The datasets used for the evaluation of the proposed algorithm (HD-DNCAE) include benchmark data such as the NSL-KDD [40], UNSW-NB15 [41], and CIC-IDS2017 [42]. The NSL-KDD dataset is based on the data captured in the DARPA'98 Intrusion Detection System (IDS) evaluation program and has been the most widely used dataset for the evaluation of anomaly detection methods. UNSW-NB15 is a comprehensive dataset for network intrusion detection systems, which contains raw network packets and includes nine different attacks. The CIC-IDS2017 dataset contains benign data and the most up-to-date common attacks, which resemble true real-world data. It also includes the results of the network traffic analysis with labelled flows based on the time stamp, source, and destination IPs, source and destination ports, protocols, and attacks. These datasets were used as a baseline for the benchmarking of the outcomes of the proposed model.  
Finally, network-related datasets have been collected from the Livorno ICT stack as well as from the underlying machine-to-machine platform (namely Mobius oneM2M [43]), which is used to collect and store data coming from the IoT monitoring network deployed in the Livorno seaport (e.g., weather conditions, noise and pollution levels, etc.). The main parameters that are taken into consideration include IP address and port, protocol, timestamps, flow duration and rate, packet count and length, inter-arrival times, and several TCP tags.  
The table in Figure 10 shows the results of the DNCAE algorithm from the work in [3]. Figures 11 and 12 show the result of the HD-DNCAE algorithm using two different optimisers. From a simple glance, the DNCAE performs better in most of the one-class classifiers across the datasets. However, when we look closely, we can see that in areas where the DNCAE performs poorly, the HD-DNCAE mostly has doubled its accuracy. When we take the Local Outlier Factor (LOF), HD-DNCAE outperforms DNCAE across all the datasets.  
Figure 11: HD-DNCAE (SGD with momentum) AUC score table.  
Figure 10: DNCAE AUC score table.  
Figure 13 shows the best scores obtained across the datasets for each of the algorithms. As we can see from the plot, HD-DNCAE performs similarly to DNCAE while not requiring setting the number of clusters in advance. This brings us closer to the zero-touch approach for our security module being deployed inside the Livorno ICT stack.