# Effective Multi-Frame Optical Detection Algorithm for GEO Space Objects
## 4 Experiment
### Comparison Results  
In order to verify the detection performance of the 2D single-frame detector and the proposed CFS technique for space objects in deep space scenery, under the same hardware conditions and environment configuration, several common object detection methods are compared on the SpotGEO dataset, and the detection results are shown in Tables 2 and 3.  
We compare the PP-YOLOv2 method with four state-of-the-art networks including Faster R-CNN, Cascade R-CNN, YOLOv3 and PP-YOLO. Table 2 intuitively denotes the comparison results of different single-frame GEO object detectors. The quantitative comparison results show that the PP-YOLOv2 outperforms the other models on the test dataset with comparable compute speed. The F1 score on the SpotGEO dataset is 84.08%. It achieves a compromise between performance and computational complexity, which can better adapt to dynamic requirements.  
In order to verify the superiority of the proposed CFS technique, we rerun the tests of different models along with the same post-processing procedure. After the CFS step is finished, the final results are shown in Table 3. The F1 score of each model has significantly improved and the corresponding MSE has decreased dramatically. As can be seen from the third column in Table 3, various indicators have increased significantly close to 11% in the F1 score, hence demonstrating the great effectiveness of the proposed CFS technique. In addition, the final F1 score of the proposed algorithm is also 1.47% higher than the accuracy of GEO-FPN [29].  
In the official challenge, the final ranking is based on the 1\(-\)F1 and MSE. For a more intuitive comparison, we have directly listed the F1 scores as presented in Table 4. It is obvious that the proposed algorithm has already exceeded third place. According to the published article of the organizers, the top 1 team adopted the 3-step pipeline including the first step of background removing using L1-spline and estimation of star shifting using a hand-crafted star descriptor and RANSAC. Then, in the second step, they trained an ensemble of 10 U-Net [30] models to predict object locations. Lastly, they also did the procedure of post-processing via line detection and trajectory filling. Whereas, the team POTLAB@BUAA that designs a non-learning-based approach came second. They adopted a pixel-level classification method based on the SNR calculation and extracted candidate targets through adjacent selected pixels. Then, the final detection results are obtained based on the estimation of interframe satellite shifts. Compared to their solution, our pipeline is more intuitive and concise.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **Backbone** & **FPS** & **F1(\%)** & **MSE** \\ \hline GEO-Faster R-CNN & ResNet50\_vd\_dcn & 18.06 & 80.19 & 162,168.66 \\ GEO-Cascade R-CNN & ResNet50\_vd\_dcn & 20.6 & 82.54 & 177,082.09 \\ GEO-YOLOv3 & ResNet50\_vd\_dcn & 61.3 & 81.89 & 141,504.56 \\ GEO-PP-YOLO & ResNet50\_vd\_dcn & 72.9 & 82.21 & 181,461.13 \\ GEO-PP-YOLOv2 & ResNet50\_vd\_dcn & 72.1 & 84.08 & 160,997.98 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison of different 2D single-frame object detectors.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **F1(\%)** & **AF1(\%)\(\uparrow\)** & **MSE** \\ \hline GEO-Faster R-CNN + CFS & 90.02 & 9.83 & 53,001.35 \\ GEO-Cascade R-CNN + CFS & 91.63 & 9.09 & 53,274.17 \\ GEO-YOLOv3 + CFS & 91.54 & 9.65 & 53,888.27 \\ GEO-PP-YOLO + CFS & 92.89 & 10.68 & 49,274.17 \\ GEO-PP-YOLOv2 + CFS & 93.47 & 9.39 & 40,222.44 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of different deep models with CFS.