## 6 Evaluation
### Results and Discussion
#### 6.2.5 Training and Testing Samples  
In general, it is widely known that using a small size of training dataset will cause a machine learning model to perform poorly due to underfitting of the training dataset. On the other hand, if the training dataset is too large, it will cause overfitting of the training data. Similarly, too few test data may result in a high variance estimation of the model performance [42]. Therefore, to ensure that the cause estimation algorithm can be generalized when applied to a newly acquired scan response sample, we filter the training samples that are used in building the MM-LDA model by removing scan response samples with high RTT variance in each scenario. For this, the training sample is divided into multiple slices. To pick training sets that can be used for building a new model, each training datum in the slice is selected by a random selection and is used to build a model. Any training datum of a model that failed to outperform the initial model performance threshold (e.g., achieving a 95% correct estimation) is dropped from the training set used in building the model. Thereafter, the training and test samples were selected from the first set of simulation samples for each scan response scenario using a conventional split of 80%/20% for training/test split, respectively. The selected training samples from the filtered training data of each scan response scenario are used to train the model, and the test set, which is not used to train the model, is used to estimate how well the model will generalize when applied to newly acquired scan response data.  
Figure 10 shows the correct estimation ratio of the scan failure/delay of training and test data samples. The first set of results shows the correct estimation ratio of the proposed MM-LDA filtered for the training and test data, respectively. In the second set of results, we compare the performance of the proposed MM-LDA filtered on the training and test data with the performance of other conventional machine learning classifiers. As shown in Figure 10a, the proposed MM-LDA with filtered training set achieves an average of 98% correct estimation of the cause of scan failure/delay (i.e., normal--1.0, SNR insufficient--0.91, and NW congestion--1.0 correct estimation ratio).  
Similarly, an average of 97% correct estimation (i.e., normal--1.0, SNR insufficient--0.94, and NW congestion--1.0 correct estimation ratio) is confirmed for the test set, as shown in Figure 10b. In comparison to other conventional machine learning algorithms, as shown in Figure 10c,d, we confirm that LR, NB, and SVM classifiers perform poorly in the estimation of both the training and test data, with estimation results below 70%. On the other hand, KNN and CART achieve high estimation of both the training and test data with 95% and 98% estimation results, respectively. However, only the MM-LDA with the filtered training set is able to generalize new scan response where all the conventional methods fail (see Section 6.2.6).  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Scan RTT** & **Scan Jitter** & **Scan Maximum RTT** & **Scan Minimum RTT** \\ \hline Scan RTT & 1 & 0.054 & 0.869 & 0.988 \\ Scan jitter & 0.054 & 1 & 0.371 & \(-\)0.040 \\ Scan maximum RTT & 0.869 & 0.371 & 1 & 0.817 \\ Scan minimum & 0.988 & \(-\)0.040 & 0.817 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Correlation coefficients of scan response of the training data.