# An Effective Surrogate Ensemble Modeling Method for Satellite Coverage Traffic Volume Prediction
## 4 Experimental Results
### Accuracy and Robustness  
The accuracy and robustness of different surrogates (obtained with 60 training points) are evaluated by R\({}^{2}\) and NRMSE in each sub-domain as well as globally, as shown in Table 6. The best value in each column is shown in bold for ease of comparison. The results show that in the global domain the overall accuracy of PWTA is better than that of the other surrogates. Moreover, PWTA also outperforms the other surrogates for most sub-domains, except North America, South America, East Asia, and Oceania areas. Although PWTA is not the best for those sub-domains, it performs comparably well (rank second in terms of accuracy) to the best model in these sub-domains (with less than 0.06% relative difference). It suggests that PWTA performs robustly well in this SCTV prediction problem. For the BestGMSE surrogate, it can be seen that its overall performance is poorer than that of PWTA. Especially in South America and Russia areas, the accuracy of the BestGMSE surrogate is the same as that of RBF, i.e., RBF is chosen as the \"optimal model\" based on the criterion of GMSE which only considers training points. However, OK that has better performance on the test set is neglected. This indicates that for the BestGMSE surrogate there is indeed a risk of selecting the suboptimal model which lacks good generalization capability. However, for PWTA, this risk could be effectively avoided by the ensemble modeling. Moreover, it can be also observed that WTA has poorer performance compared to PWTA, which confirms that adding the inaccurate surrogates into the ultimate ensemble would result in loss of accuracy. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Surrogate** & **30 Training** & **40 Training** & **50 Training** & **60 Training** & **70 Training** & **80 Training** \\
**Model** & **Points in Each** & **Points in Each** & **Points in Each** & **Points in Each** & **Points in Each** \\
**Sub-Domain** & **Sub-Domain** & **Sub-Domain** & **Sub-Domain** & **Sub-Domain** & **Sub-Domain** \\ \hline
**PRS** & 0.6333 & 0.6557 & 0.6658 & 0.6760 & 0.6855 & 0.6901 \\
**OK** & **0.9613** & **0.9703** & 0.9694 & 0.9831 & 0.9862 & 0.9880 \\
**RBF** & 0.9428 & 0.9645 & 0.9713 & 0.9830 & 0.9858 & 0.9880 \\
**PWTA** & 0.9596 & 0.9697 & **0.9722** & **0.9845** & **0.9872** & **0.9891** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparisons of R\({}^{2}\) for different surrogates in the global domain under different training sample sizes. \begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Performance** & **Surrogate** & **North** & **Western** & **Caribbean** & **South** & **Russia** & **Middle** & **East** & **Oce-ania** & **Global** \\
**Metric** & **Model** & **America** & **Europe** & **Sea** & **America** & **East** & **Asia** & **Oce-ania** & **Domain** \\ \hline \hline \multirow{2}{*}{**R\({}^{2}\)**} & **PWTASD** & **0.967** & **0.994** & **0.989** & **0.987** & **0.992** & **0.954** & **0.989** & **0.981** & **0.985** \\  & **PWTASD** & 0.960 & 0.992 & 0.986 & 0.978 & 0.985 & 0.946 & 0.987 & 0.970 & 0.980 \\ \hline \multirow{2}{*}{**NRMSE**} & **PWTASD** & **0.082** & **0.037** & **0.050** & **0.081** & **0.053** & **0.077** & **0.059** & **0.082** & **0.069** \\  & **PWTASD** & 0.090 & 0.043 & 0.057 & 0.104 & 0.072 & 0.084 & 0.063 & 0.104 & 0.079 \\ \hline \multirow{2}{*}{**NMAE**} & **PWTASD** & **0.425** & **0.155** & **0.166** & **0.372** & **0.229** & **0.282** & **0.284** & **0.418** & **0.447** \\  & **PWTASD** & 0.426 & 0.168 & 0.213 & 0.377 & 0.306 & 0.284 & 0.286 & 0.513 & 0.549 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparisons of R\({}^{2}\), NRMSE, and NMAE for PWTASD and PWTASD.