## 4 Discussion
### Validation of the Results by Confusion Matrix Performance Evaluation and ROC Curves
The alarmed space-time windows are very low (i.e., around one or a few percent) for all the investigations with an alarm-window time of 90 days, and they are between 5.1% and 19.1% for the analyses performed in a time window of 500 days. Both the investigated time windows show alarmed times below the 50% threshold--in some cases, lower than 1%--which provide further evidence of the non-chance nature of the detected anomalies in relation to the following earthquakes. The accuracy is similar to that of a previous work performed on _Swarm_ electromagnetic data and earthquakes but using a machine learning approach [40]. Furthermore, the overall performances reflected the previous considerations, with the scores for the magnetic field being generally better than those for the electron density. Further, better scores were always yielded when the frequency content of the signal was investigated, confirming that it is fundamental to take into account the signal frequency of the possible electromagnetic ionospheric precursor in future studies. In order to further statically check the predicting capability of the magnetic field and electron density anomalies extracted in this work as the best candidates for earthquake precursors by WSC, the Receiving Operating Characteristic (ROC) graph has been calculated. ROC is a well-established method to test a precursor candidate, e.g., [41], and it is a graph where the false positive rate is conventionally reported on the horizontal axis and the hit rate is reported on the vertical one. The diagonal from (0%, 0%) to (100%, 100%) represents the random case, while the upper-left corner (0%, 100%) is an ideal case that predicts all the events without any false alarm. A good prediction is required to be as far from the diagonal toward the ideal case as possible. In our case, we extract all the anomalies identified as statistically significant in this work, i.e., the pre-earthquake concentrations that show a statistical coefficient \(d\geq\) 1.5 and \(n\geq\) 4.0, as previously used in this work and first defined by [PERSON] et al. [29]. In this section, other concentrations that are different from the absolute maximum one that passed this statistical criterion have been also taken into account. Collecting all the anomalies from the magnetic field and electron density _Swarm_ data and all the techniques, we produced a new dataset of the best anomalies that contains 4831 _Swarm_ anomalous signals after cleaning the overcounted ones. With the new dataset of the best WSC anomalies, the \"Confusion matrix\" and ROC diagram have been evaluated and reported in Table 2 and Figure 5, respectively. The step time of calculus has been set to be equal to the alerted time, and the window following the one with any _Swarm_ electromagnetic anomaly is alerted if it contains any anomaly; otherwise, it is not alerted. Comparing Table 2 with Table 1, it is possible to fully confirm that the selections made in this paper extracted the statistically best pre-earthquake candidates permitted to reduce the false positive rate and increased the hit rate. The ROC curve in bold blue in Figure 5 is obtained after testing several alerted times from 10 days (lower left side) to 2500 days (upper side). Two black arrows have underlined the two alerted times of 90 and 500 days, and they correspond to the one used in Table 2 and are mostly selected in this paper. In the same figures, the Area Under the Curve (AUC) has also been represented as a black inclined hatch for the random case (i.e., the one under the diagonal equal to 50% for the definition) and as a red inclined hatch for the real best _Swarm_ anomalies, estimated to be equal to 83.6%. So, the AUC of the real data is 1.67 times better than that of a random predictor, confirming that WSC applied to _Swarm_ data is able to extract potential pre-earthquake ionospheric disturbances.