## 3 Results
### Confusion Matrix Analysis  
The qualitative results shown in the previous case studies are only some particular cases to illustrate the technique. A more robust and quantitative way to study if this technique is effective in detecting earthquake-related ionospheric anomalies is to perform a correlation test of the data by means of the confusion matrixes introduced in Section 2.4. Figure 12: Integrated seven day precursory anomaly within a circle of the indicated radius averaged for all the earthquakes taking place in the period studied, with a magnitude larger than the one indicated in each plot. Only earthquakes surrounded by at least 40% of water are computed. As explained, the \(S_{4}\) variations maps from the full period of 180 days are computed pixel by pixel classifying them in the four categories depending on whether they are positive or negative cases for both the \(\Delta S_{4}\) value and the earthquake occurrence. With these results, it is possible to construct the confusion matrix, and its derived statistics for each set of parameters. Table 3 shows a subset of the total number of CMs computed. All of them are calculated for 180 days of data using:  
* _radius_: 100 km
* _eq_mag_: 6 to \(\infty\)
* _d_margin_: +3 d
* _s4_th_: 0, 0.003, \(\cdots\), 0.030  
Remember that the total number of events computed, marked as TOT is the total number of pixels with valid data during the period studied. Positives (P) and Negatives (N) refer to the pixels with earthquakes closer than the radius, and TP, TN, FP, and FN are the four categories of the CM. The results of the statistical parameters described in Section 2.5 are presented here in Table 4 for each case studied, for the same previous cases used in Table 3.  
As introduced in Section 2.4, the Receiver Operating Characteristic (ROC) curves are also a good way to identify better performance of a classification algorithm and to tune the threshold parameter used, in this case _s4_th_. The ROC curve is constructed by plotting the TPR vs. the FPR, so it is able to compare simultaneously the perfo  
\begin{table}
\begin{tabular}{l l l l l l l l l} \hline _s4_th_ & **TPR (\%)** & **FPR (\%)** & **TNR (\%)** & **FNR (\%)** & **ACC** & **F\_1 Score \(\times 10^{-3}\)** & **MCC \(\times 10^{-3}\)** & **DOR** \\ \hline
0.000 & 54.8 & 47.5 & 52.5 & 45.2 & 0.525 & 0.32 & 1.73 & 1.340 \\
0.003 & 41.6 &  
to correctly associate positive changes in the \(S_{4}\) value with earthquakes against the rate of false alarms produced when the algorithm detects an positive change in the scintillation, but there is no earthquake in this region and period. ROC curves for the three magnitude intervals are presented in each of the plots in Figure 13. Four curves per graph are shown with different colors indicating the time interval from the anomaly to the occurrence of the earthquakes, being positive when it is prior to the earthquake so it could be considered as a precursor. Each point in the curve is for a different value of the threshold parameter \(s4\_th\) shown as rows in previous Table 3. Please note that the point closer to \((0,0)\) is the one the with largest \(s4\_th=0.03\), which makes sense because larger threshold are filtering more true detections, and also reducing the false alarm rate. Decreasing the threshold in 0.003 intervals, both parameters increases until the last point (the one closest to \((1,1)\)) for \(s4\_th=0\). ROC curves indicate better performance when they are more separated from the \(y=x\) line, which would be the result of a completely random detection algorithm. ROC-AUC (ROC Area Under the Curve), is a good indicator of this performance, and it will be discussed in the next section. Figures 14 and 15 show the ROC curves for the three magnitude intervals using a radius of 50 km and 100 km, respectively. In these plots, \(s4\_th\) has been swept from 0 to 0.04 in intervals of 0.01. Figure 13: ROC curves for three magnitude intervals using \(radius=100\) km.