## 4 Experimental Results and Discussion
### Performance Analysis  
The performance of the proposed method can be depicted by the following values: precision for the correctly identified type of PQD, \(P_{\text{occur}}\); recall for the correctly identified type of PQD, \(R_{\text{occur}}\)  
Figure 11: Accuracy comparison with different training percentages. (**a**) classification accuracy with feature heuristic selection; and, (**b**) classification accuracy without feature selection. Figure 10: Identification accuracy results with different numbers of selected features. (**a**) Training result only by [PERSON]’s scoring method; and, (**b**) training result by [PERSON]’s score and the permutation entropy method. precision for the incorrectly identified type of POD, \(P_{\text{other}}\); and, recall for the incorrectly identified type of PQD, \(R_{\text{other}}\). \[ \begin{array}{l}P_{defect}=\frac{TP}{TP+FP}\\ R_{defect}=\frac{TP+FN}{TN}\\ P_{other}=\frac{FP}{TP+TN}\\ A{\text{ccuracy}}=\frac{TP+TN}{TP+FP+TN+FN}\times 100\%\end{array}  \tag{14}\]  
As shown in Equation (14), TP represents the number of true positive instances, which means that the estimated PQD type is correctly classified. FP represents the number of false positive instances, which means that another PQD is classified incorrectly as the estimated PQD type. FN represents the number of false negative instances, which means that the estimated PQD type is incorrectly classified as another PQD type. TN represents the number of true negative instances, which means that another PQD type is correctly identified as the corresponding PQD. Table 3 presents the confusion matrix of the PQD classification for the validation data sets. The classification results achieve high accuracy, as observed from the confusion matrix. However, there are still some mistakes due to the similar characteristics of some PQD types. For instance, two C4 (Interrupt Disturbances) samples are incorrectly assigned to C2 (Sag). Meanwhile, three C2 samples are also incorrectly assigned to C4, because the formula definitions of C2 and C4 are the same, and the only difference between these two classes is parameter \(\alpha\). As referred to above in Table 2, parameter \(\alpha\) randomly varies over a range. To be specific, for C2 (sag), the range of parameter \(\alpha\) is from 0.1 to 0.9. For C4 (interrupt) and the range of parameter \(\alpha\) is from 0.9 to 1. When parameter \(\alpha\) is selected to be near for both classes, the features that were extracted from the signal are also similar. In this circumstance, it would lead to an incorrect classification. The results for C11 (Sag with harmonics) and C13 (Interruption with harmonics) also verify this situation. The receiver operating characteristic (ROC) curve is adopted to depict the tradeoff relationship between the true positive rate and the false positive rate to evaluate the performance of the proposed classifier, as shown in Figure 12. Ten OVR classifiers were constructed due to the ten PQD types since the \"one-versus-rest\" (OVR) structure was utilized for multiclass classification in this paper. Subsequently, ten ROC curves were drawn to represent the true positive rate versus the false positive rate at various threshold values of \(\theta\). To be specific, the threshold varied from 0 to 1 with a step of 0.02. By giving the threshold \(\theta\), the average values of the true positive rate and false positive rate were calculated. The areas of the four ROC curves from the C1 class to the C10 class were 0.979, 0.958, 0.949, 0.963, 0.957, 0.978, 0.947, 0.965, 0.986, and 0.987. For the PQD classification, a large false positive rate would cause false alarms, which would result in unexpected or even mistaken strategy responses for power grids. More seriously, a large false positive rate would lead to worse consequences in the distribution generation systems with high penetration.