# European Geostationary Navigation Overlay Service Performance Prediction +
## 6 User-Level Results  
An assessment was conducted for three example scenarios to evaluate the accuracy of the machine learning application for different ground- and space-segment configurations. The metric used is the protection level, as calculated from the UDREI and GIVEI, for users located on a regular grid across Europe:  
* Scenario 1 (\"nominal case\") runs from the 17 th to the 19 th of September 2016, with a nominal ionospheric environment and no RIMS or GPS satellite outages;
* Scenario 2 (\"GPS outages\") equals Scenario 1 but with GPS satellite outages (loss of G18 on 17/9, loss of G18 and G20 on 18/9, and loss of G18, G20 and G21 on 19/9);
* Scenario 3 (\"RIMS outages\") equals Scenario 1 but with RIMS outages (loss of Golbasi on 17/9, Golbasi and Sofia on 18/9 and Golbasi, Sofia and Athens on 19/9).  
To investigate cases with assumed GPS and RIMS outages, the GIVEI and UDREI values actually broadcasted from EDAS are invalid, and reference GIVEI and UDREI values were therefore produced by the European Space Agency using their EGNOS v2 Performance Engineering Platform (SPEED), which uses the same algorithms as the actual EGNOS ground segment.  
To compare how close the machine learning predictions are to the real EGNOS v2 performances, we first compare predicted and actual the EGNOS LPV200 availability service areas. This is the area where the protection levels are below the Alert Limits (40m horizontally and 35m vertically). This criterion is quite sensitive to UDREI and GIVEI changes. The resulting availability map comparisons are shown in Figure 5. The availability prediction at the centre of the service area is excellent, but towards the borders of the coverage the performance degrades somewhat and the availability is both over- and underestimated in some areas. When the satellite constellation is degraded, the availability  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**UDREI** & **Macro Model** & **ML** \\ \hline
5 & 72 & 70 \\
6 & 59 & 49 \\
7 & 8 & 36 \\
8 & 0 & 31 \\
9 & 9 & 32 \\
10 & 0 & 42 \\
11 & 0 & 68 \\
12–14 & 85 & 89 \\ \hline \hline \end{tabular}
\end{table}
Table 2: UDREI prediction: percentages of correct outcomes for each class.  
Figure 4: Confusion matrices for UDREI prediction of the macro model and our ML model. Darker colors correspond to higher percentages of outcomes falling into the tile.  
at the edges becomes underestimated in more cases and some degradation near the core of the coverage area can be seen. A reduced number of RIMS makes the predictions only slightly worse at the edges of the coverage, with some additional over- and underestimated regions of limited size.  
As a second comparison, some time series of vertical protection levels for users near the east and north borders of the service area (Bulgaria and the north of Finland), as well as the centre of this area (South of France) have been produced, see Figure 6. The protection levels from the core of the service area can be predicted well, as was also found in [2]. This is likely because low values of UDREI and GIVEI are common and these are predicted with high accuracy as evident from Figures 3 and 4. Towards the edges of the coverage area, our algorithm tends to be somewhat conservative and predict larger-than-actual protection levels. The conservatism increases somewhat in the presence of satellites and RIMS outages. In some cases, the actual protection level is unavailable, while the prediction still produces valid protection levels. This has been investigated by ESA and corresponds to rapid  
Figure 5: Machine learning vs. EGNOS v2 (LPV200 99% availability differences). In the green areas, EGNOS performance is overestimated, while in the orange areas it is underestimated.