## 4 Results from Case Studies
### Validation of Synchrophasor Applications Using Dynamic Grid Simulations
#### 4.3.2 Results from Synchrophasor Disturbance Classification
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline \# of hidden dimensions \(Q\) & 15 \\ \# of feature dimensions \(P\) & 15 \\ Optimizer & rmsprop [69] \\ Learning rate & 0.01 \\ Batch size & 50 \\ Maximum \# of epochs & 1000 \\ \hline \hline \end{tabular}
\end{table} 
Table 12: Hyperparameters used for the _disturbance classification module_. Additionally, early stopping was used to limit the number of training epochs and prevent overfitting. Table 13 shows the accuracy and F1-score results of the training, validation and test datasets for a single training run. Appendix A gives additional information for calculating both metrics. The validation accuracy and F1-score are quite low compared to the training and validation test due to the small size of the validation dataset. The test performance is between the training and the validation values as a result of the high number of samples. Additionally, the training and validation instances cannot be excluded from the test dataset, because of the different sampling rates. Figure 20 shows the normalized confusion matrix for the true and predicted class labels from the training dataset. Very high accuracies can be observed for the conventional and renewable power plant outages as well as the load losses. A few misclassified training instances occur for line trips and short circuits of Line 32. Also in other cases, it can be seen that a correct differentiation between line trips and short circuits represents a big challenge for the PMU based disturbance classification task. One possible reason could be the short duration of these events, which makes a classification difficult for samples with a large time lag to the start of the disturbance. Similar conclusions arise from the receiver operating characteristic (ROC) analysis of the training and validation predictions shown in Figure 21. In accordance with the previous findings, accuracy drops between training and validation can be observed for line outages (minimum AUC value: 0.97) and short circuits (minimum AUC value: 0.99). Other disturbance classes are affected only marginal. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metric** & **Training** & **Validation** & **Test** \\ \hline Accuracy (\%) & 97.92 & 89.81 & 94.80 \\ F1-score (macro) (\%) & 97.91 & 89.01 & 94.73 \\ F1-score (micro) (\%) & 97.92 & 89.81 & 94.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Accuracies and F1-scores for training, validation and test predictions. Figure 20: Normalized confusion matrix for training predictions. In further investigations, the embedding functions are compared with each other for different numbers of hidden dimensions, learning rates and optimizers. The training, validation and test performance results are given in Figure 22. Highest accuracies can be achieved in case of the local parametric attention model (\"local scoring\") with 95.12% (training) and 90.89% (test) as well as the non-parametric attention model using cosine similarity (\"cosine attention\") with 94.58% (training) and 91.96% (test). In case of no embedding with 90.54% (training) and 86.53% (test), the last hidden state \(\mathit{h}^{T}\) is used as the feature vector. In general, attention based embedding functions increase the test accuracy about 4-5%. Feedforward neural network based embedding functions show very low accuracies on the training (max. 85.88%), validation (max. 78.94%) and test datasets (max. 79.89%) and even perform worse than the \"no embedding\" variant. This could be due to the vectorization of the hidden state matrix ignoring the dependencies between the hidden state vectors. In contrast to that, attention-based embeddings are more effective to summarize the information across the time steps but keep the number of hidden dimensions unchanged. This can lead to performance degradations when calculating distances between the high-dimensional feature vectors. Furthermore, some parameters of the _base scenario_ are varied to evaluate their influence on the classification accuracies.