## 5 Performance Comparison
### Comparison to Traditional ML Algorithms  
In this section, we compare the performance of our (_CNN3 + AE_) pipeline to well-known ML algorithms such as _Support Vector Machine (SVM)_[71], _Random forest (RF)_, and _K-nearest neighbors (KNN)_. The performance of a shallow learner is also explored using a _one-dimensional Convolutional Neural Network (1D CNN)_.  
SVM is a supervised ML algorithm proposed in 1995 [72], where an optimal hyperplane is used to separate classes in the data space. SVM proved to be a successful method to attack a two-class classification task when extracting well-representative features from data. RF was first introduced by [PERSON] [73]. It is an ensemble learning method for classification, where a multitude of decision trees are trained to output the label of the classes, and the predicted class is determined by majority voting. The KNN algorithm is a nonparametric, lazy learning algorithm that outputs each input's class based on the majority of its nearest neighbors (proximity is measured by feature similarity) [74]. We tested these algorithms under all channel scenarios and SNR values to compare them to our proposed classifier. 1D CNN was chosen as a shallow benchmark learner as it enables frame-level investigation, and its use had been explored for audio recognition and Natural Language Processing (NLP). 1D CNN has been used with raw waveform and usually combined with a _Recurrent Neural Network_ (RNN) in audio applications [75]. The convolution layer's kernel size in our benchmark 1D CNN is set to 3, and 24 filters were used with a ReLU activation. The soft-max activation function was used to classify the protocols. The input for SVM, KNN, RF, and 1D CNN consisted of the noisy spectrogram images of size \(128\times 128\), i.e., each image had \(16,384\) features; for the 1D CNN, this was shaped as a \((16384,1)\) array. Each dataset was split to 80% for training, and the remaining 20% is for testing. In each training session, the data were shuffled, and 33% of it was used for cross-validation to tune the hyperparameters of the model whenever this could apply. Each classifier's performance was evaluated using accuracy, recall, precision, and F-score [76]. Figure 8 shows the empirical cumulative distribution function (CDF) of the overall accuracy for the shallow classifier on our datasets across all the channel scenarios. The overall accuracy is the ratio of correctly predicted observation to the total observations. The accuracy of the SVM is the lowest, while KNN and RF achieve 20%. 1D CNN gives an accuracy of 20% in most of the datasets and 60% in a few datasets, depending on the channel scenario and SNR levels. Figure 7: Confusion matrices for CNN3+AE in some representative SNR levels and channel scenarios: (a) SNR 5 dB and _scenario 5_; (b) SNR 15 dB and _scenario 3_; (c) SNR 20 dB and _scenario 2_. The recall (i.e., sensitivity) is shown in Figure 8b. The precision across all datasets under various noise conditions and SNR values is shown in Figure 8c. The overall CDF of F-score is illustrated in Figure 8d across all the 25 datasets. The performance of our (CNN3 + AE) is much higher than the other compared ML algorithms in terms of accuracy, recall, precision, and F1 score. The highest accuracy achieved by our developed model is 100% across many datasets such as a dataset with SNR 5 dB and _scenario 3_, 5 dB and _scenario 4_, and 5 dB and _scenario 4_ as shown in Figure 9. The minimum accuracy is 55% for a dataset with the AWGN channel and SNR level equal to 0 dB (More features are recovered at 0 dB that were reported lost in [47].). Figure 8: The overall performance cumulative distribution function (CDF) across a range of channel scenarios and signal-to-noise ratio (SNR) values: (**a**) Accuracy, (**b**) Recall, (**c**) Precision, and (**d**) F1. Figure 9: The overall accuracy for the traditional ML vs. the proposed approach across a range of channel scenarios and SNR values.