# Surface Reconstruction Planning with High-Quality Satellite Stereo Pairs Searching
## 5 Experiments and Evaluation
### Evaluation on Planning Using Single Satellite  
In this part, the planning results are evaluated using a single satellite of World-View3 as shown in Figure 5, i.e., searching from the 150 time windows localized from the World-View3 orbit as the potential imaging opportunities. This means the task planning needs to search from more than 20,000 stereo pairs to achieve optimal objective function values as modeled in Section 4.1. In solving the single-satellite planning model, the DSM quality of an arbitrary image pair is estimated beforehand using the method as proposed in Section 3.2 to quantify each possible combination. Based on this accurate quantification, the planning model is solved by the GWO algorithm whose setting of parameters is as follows: maximum number of iterations \(T=500\); population size \(N=100\); stereo image pair intersection angle range \(\theta_{ij}\in[5^{\circ},25^{\circ}]\). The time complexity of the GWO algorithm in this scenario is \(O(T\cdot N\cdot M^{2})\), where \(M\) is the number of time windows, i.e., possible visible duration calculated according to satellite's orbit. To verify the effectiveness of our planning algorithm in selecting imaging opportunities to generate higher quality stereo surfaces, we compare different prioritized objective functions by controlling the objective weights in Equation (6), i.e., task quantity (\(\omega_{2},\omega_{3}=0\)), stereo image quality (\(\omega_{1},\omega_{3}=0\)), and task timeliness (\(\omega_{1},\omega_{2}=0\)). With these criteria, the combinations of imaging time windows are optimized to achieve the highest objective scores as is possible. In Figure 7, each averaged 3D reconstruction error is calculated over \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & \(\mathbf{R^{2}}\) & **RMSE** & **MAE** \\ \hline SVR & 0.5454 & 0.7032 & 0.4654 \\ SVR-FE & 0.5503 & 0.6994 & 0.4595 \\ XGBoost & 0.7485 & 0.5230 & 0.3612 \\ XGBoost-FE & 0.7719 & 0.4981 & 0.3474 \\ CatBoost-FE & 0.8856 & 0.3527 & 0.2466 \\ CatBoost-LASSO & 0.8606 & 0.3894 & 0.2742 \\ CatBoost-FE-LASSO & 0.8920 & 0.3427 & 0.2462 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Result comparison of different prediction models. Figure 6: Evaluation of quality prediction model. 10 independent planned results for different objectives of quantity only (\(Q_{1}\)), quality (\(Q_{2}\)), timeliness (\(Q_{3}\)), quantity and quality (\(Q_{1,2}\)), quantity and timeliness (\(Q_{1,3}\)), etc., respectively. Results of planning over a whole year (from Nov 2014 to Jan 2016) and two halves of the year 2015 are shown in Figure 7. As shown in this figure, planning with quality (\(Q_{2}\)) achieves the best performance, while planning for task quantity (\(Q_{1}\)) leads to the highest reconstruction error. While \(Q_{3}\) lies between \(Q_{2}\) and \(Q_{1}\), the introduction of quality in planning can still improve the overall performance with higher \(Q_{2,3}\) (\(\omega_{1}=0\)), similar to the improved \(Q_{1,2}\) (\(\omega_{3}=0\)) compared to \(Q_{1}\). This demonstrates the importance of quality in planning for better stereo pairs by the results of planning over both a whole year (blue) and a half year (orange or green) in Figure 7.  
It is interesting to notice that the quality of stereo imaging distributes differently in the first and second halves of the year 2015. This is depicted by the difference of two halves on objectives such as \(Q_{2}\) and \(Q_{3}\). In Figure 7, the second half of 2015 exceeds the first half with lower DSM error (3.57 vs. 3.78) on \(Q_{2}\) while it is outperformed by the first half on \(Q_{3}\) (4.33 vs 4.03). This is because more high-quality stereo pairs are contained in the second half of 2015, but they are distributed in the last quarter of the year. When timeliness is used as the objective, these late stereo pairs can be hardly planned. That is why quality is poor for the second half of the year on \(Q_{3}\) (timeliness).