# Effective Multi-Frame Optical Detection Algorithm for GEO Space Objects
## 4 Experiment
### Evaluation Metric  
We exploit the officially provided evaluation kit [28] to compute the average F1 score and Mean Square Error (MSE) score for all the test images, of which the detailed calculation introduced as the following.  
According to the ground truth of object locations \(\mathcal{Y}\)= \(\{\mathcal{Y}^{f}\}_{f=1}^{5}\) and the predicted object locations \(\chi\)= \(\{\chi^{f}\}_{f=1}^{5}\) in 5 successive frames, each \(\mathcal{Y}^{f}\)= \(\{\mathcal{Y}^{f}_{f}\}_{f=1}^{N}\) and \(\chi^{f}\)= \(\{\mathcal{x}^{f}_{i}\}_{i=1}^{M_{f}}\) contain the true N objects and \(M_{f}\) candidates in frame \(f\), respectively.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Number of Images** & **Number of Objects** & **Average Number of Objects** \\ \hline
**Training** & 5120 & 8964 & 1.751 \\
**Validation** & 1280 & 2241 & 1.750 \\
**Test** & 25,600 & 44,550 & 1.740 \\
**Total** & 32,000 & 55,755 & 1.742 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The SpotGEO Dataset.  
For a given frame, the binary matrix \(\mathrm{H}^{f}\in\{0,1\}^{M_{f}\times N}\) is obtained by a one-to-one matching between \(\chi^{f}\) and \(\mathcal{Y}^{f}\) via solving the minimum weighted unbalanced assignment problem \(\underset{\mathrm{H}^{f}}{\mathrm{argmin}}\sum\limits_{i=1}^{M}\sum\limits_{j= 1}^{N}\mathrm{H}^{f}_{i,j}\delta(x^{f}_{i},y^{f}_{j})\), where function \(\delta\) computes the truncated Euclidean distances as in Equation (2):  
\[\delta(x^{f}_{i},y^{f}_{j})=\left\{\begin{array}{cc}\left\|x^{f}_{i}-y^{f}_{ j}\right\|_{2},&if\left\|x^{f}_{i}-y^{f}_{j}\right\|_{2}\leq\tau,\\ l,&otherwise.\end{array}\right., \tag{2}\]  
where the matching distance threshold \(\tau\) is set to 10 and the \(l\) is a large enough positive number such as 1000. The matching procedure will result in each object being matched with its nearest neighbor prediction. Depending on the distance function \(\delta\), we can obtain the following sets of frame \(f\): true positives \(TP^{f}\), false positives \(FP^{f}\) and false negatives \(FN^{f}\).  
The regression error \(SSE^{f}\), as the sum of squared error is defined by Equation (3):  
\[SSE^{f}=\sum\limits_{(i,j)\in TP^{f}}\pi(x^{f}_{i},y^{f}_{j})+\sum\limits_{j\in FN ^{f}}\tau^{2}+\sum\limits_{i\in TP^{f}}\tau^{2}, \tag{3}\]  
where  
\[\pi(x^{f}_{i},y^{f}_{j})=\left\{\begin{array}{cc}0,&if\left\|x^{f}_{i}-y^{f} _{j}\right\|_{2}\leq\epsilon\\ \left\|x^{f}_{i}-y^{f}_{j}\right\|_{2}^{2},&otherwise.\end{array}\right.. \tag{4}\]  
The smaller tolerance radius \(\epsilon\) is set to 3 and the \(SSE\) for the sequence is thus  
\[SSE=\sum\limits_{f=1}^{5}SSE^{f}. \tag{5}\]  
_MSE_ score is defined by the following Equation (4) based on calculating the SSE for all the K sequence:  
\[MSE=\frac{\sum\limits_{k=1}^{K}SSE_{k}}{\sum\limits_{k=1}^{K}TP_{k}+FN_{k}+FP _{k}}, \tag{6}\]  
where \(TP=\sum\limits_{f=1}^{5}\left|TP^{f}\right|\),\(FN=\sum\limits_{f=1}^{5}\left|FN^{f}\right|\) and \(FP=\sum\limits_{f=1}^{5}\left|FP^{f}\right|\).  
The \(F1\) score is defined as Equation (7):  
\[F_{1}=\ 2\ \times\frac{Precision\times Recall}{Precision+Recall}, \tag{7}\]  
where the overall precision (_Precision_) is calculated according to the \(TP\) and \(FP\) of all the K sequences as in Equation (8):  
\[Precision=\frac{\sum_{k=1}^{K}TP_{k}}{\sum_{k=1}^{K}TP_{k}+FP_{k}}, \tag{8}\]  
and the overall recall (_Recall_) is computed considering false negatives (\(FN\)) as follows:  
\[Recall=\frac{\sum_{k=1}^{K}TP_{k}}{\sum_{k=1}^{K}TP_{k}+FN_{k}}. \tag{9}\]  
Additionally, FPS (frames per second) is the most common metric used to measure detection speed. It represents the number of pictures that can be detected by the model per second.