## 5 Results
### Machine Learning-Based Fault Presence and Type Detection Models  
The performance result of the ML-based fault presence detection algorithm is indicated by testing with 20 recorded data from the testbed as given in Figure 14. The various ML-based models and feature extraction methods are used in the test process to find the best option for fault detection. It should be re-emphasized that the training model is trained with the 46 kV substation simulation environment's dataset, created based on another substation, and has never been used in the testbed. However, the results are shown using a test dataset from the designed testbed. Further, each combination of ML algorithms and feature extraction techniques is tested with all recorded data from all meters or relays. To clarify, a single model is employed for the entire test dataset; however, we increase the variation of the single model by applying different ML algorithms and feature extraction techniques to identify the best ML-based fault presence and type detection model. As shown in Figure 14, the prediction accuracy of ML-based fault presence detection is presented in two perspectives: one is ML algorithms, and the other is feature extraction methods. In terms of ML algorithms' performance, CNN and SVM models provide the maximum prediction accuracy results compared to DT and RFC. Due to the constant tree structure that is not flexible, the prediction accuracy rates of DT and RFC are low for this application. Another reason for this performance decrease on DT and RFC is that the training dataset created using the simulated data of 46 kV substation was never used in the testbed. Figure 13: Results of the envelope-based fault region detection algorithm with various fault types and relays/meters (fault regions are indicated with red lines, as estimated by the designed method). In the training process of SVM, the polynomial equation was evaluated to find optimal border between four labels (normal, motor, switching, and fault). The SVM method indicates the best performance prediction accuracy in whole feature extraction methods because each label has unique specific behavior that easy to separate by SVM. The CNN processes patterns and is typically used with image data (2-dimensional data). If the features cannot provide some obvious pattern, the performance of CNN is decreased. As given in Figure 14, the performance of CNN with raw data is 0% because the pattern on the data may break down due to the 2-dimensional conversion of raw data (the vector raw data are split and combined to obtain matrix data suitable for the CNN input). However, the CNN provides promising accuracy with the combination of the amplitude and phase features (AP), frequency domain (FFT), hidden patterns (ACF), and the combination of the time and frequency domains (PSD). The WT is well-known method to find any frequency change in the time domain, which helps to figure out the start/end points of a fault; however, it cannot provide distinct features for fault presence detection application as seen in Figure 14. Accordingly, each ML method cannot achieve significant performance from the WT-based feature extraction technique. Consequently, the CNN and SVM algorithms with the AP, FFT, ACF and PSD techniques provide the best performance for fault presence detection application. In Table 7, the classification layer 1 in the proposed model is analyzed with different performance metrics: precision, recall, and F1-Score. Herein, the precision shows how often ML models do correct prediction; recall indicates the frequency of correct label identification; the F1-Score shows the overall performance by considering both the precision and recall performance metrics. Also, each performance metric is indicated between 1 and 0, where 1 is the best and 0 is the worst. As presented in Table 7, each feature extraction technique is investigated with various ML algorithms. The results reveal that DT and RFC algorithms are worst at each feature extraction technique.