# Intrusion Detection and Prevention in CoAP Wireless Sensor Networks Using Anomaly Detection
## 4 Experimental Evaluation
### A Multi-Class Problem Approach  
The results obtained from the analysis of the pcap files are constituted by observations, consisting of arrays of features organized as a rectangular matrix. Such observations are submitted to data pre-processing, dimensionality reduction and classifier training. The proportion of the number of NORMAL vs. erroneous observations (thus, resultant from the misbehaving node) is of 2 to 1, thus for each two NORMAL observations we observe one erroneous observation. We split the classification problem into two problems. The first considers the four attacks and the normal behavior of the node, allowing a security manager to know the type of misbehavior present in the topology, if any. The second problem was approached from a binary problem perspective, and in this case the goal of the security manager is simply to detect the presence of an intrusion in the network. The two types of problems may yield different results, since the classifier will have in consideration that, for the binary class problem, the ERRONEOUS label is a combination of all the four labels of misbehavior. Moreover, the LDA feature extraction algorithm has in consideration the labels of the observations, which are different for the multi and the binary class problems. We employed two feature extraction algorithms, Principal component analysis (PCA) and Linear discriminant analysis (LDA). For both algorithms, we trained classifiers regarding the scoring metric of _Accuracy_. For PCA, the data is pre-processed with a Standard Scaler, mean 0 and standard deviation 1. The features of the data are reduced by PCA to 3 components with whitening, and the method is set to auto. Next, a grid search for the best parameters of the SVM classifier is performed, with a K-fold of 3 and setting the scoring metric to _Accuracy_. In this process we employed the following SVM Kernels: Linear, RBF, polynomial and sigmoidal. The obtained results are illustrated in Figures 5 and 6. For each considered Kernel, the best parameters and accuracy results are found by a grid search under the previously described conditions. The results obtained in this process are presented in Table 1.  
As we can observe, PCA does not yield very good results while, nonetheless, RBF and sigmoidal kernels gave the best results on PCA in terms of accuracy, as shown in Table 1, with \(\sim\)51% for the RBF kernel and \(\sim\)62% for the sigmoidal kernel. Even with the previous two best kernels with PCA, there were cases of 0% of TP, being WRONG ACCEPT for the RBF kernel and DoS ACK for the sigmoidal kernel. According to the ROC curves of the previous best kernels, RBF has a higher recall and F_Measure on average than sigmoidal, scoring 76% for the DoS ACK attack, as illustrated in Figure 5b, versus the 21% of the same kernel, as in Figure 6b. The best result for sigmoidal kernel versus RBF kernel, in terms of F_measure and Recall, was with the WRONG ACCEPT attack, scoring 80%, versus the 48% of RBF kernel. In our next evaluation, LDA is used with the method of singular value decomposition and 6 components. Again, the data is pre-processed with a Standard Scaler, mean 0 and standard deviation 1. The results are illustrated in Figures 7 and 8. For each kernel, the best parameters and accuracy results are found by a grid search under the previously described conditions. The results obtained in this process are presented in Table 2.  
As can be observed, the results show that LDA is better than PCA in terms of clustering the data according to each of the classes. By applying grid search with the same kernels, the polynomial kernel of degree 1 reaches a total accuracy of \(\sim\)93%, which corresponds to a linear kernel, versus the second best kernel, which is sigmoidal, with \(\sim\)75% of accuracy, as is visible in Table 2. In fact, sigmoidal kernel gave TPs of 0% for all attacks except DoS FREQ, with an almost 50% of average for the remaining attacks.