# Fast Loop Closure Selection Method with Spatiotemporal Consistency for Multi-Robot Map Fusion
## 5 Experiments Evaluation
### Simulations in Synthetic Datasets
Figure 5: Estimated maps for CSAIL dataset. (**a–c**): Optimized graph after our method in the presence of 50, 150, and 250 outliers. Correctly accepted inlier constraints are green, disabled outliers in gray, final optimized map in blue. is not a fixed value but is related to the respective lengths of the two trajectories. In this simulation, the number of inliers was 140, and we generated two versions of the dataset, including 200 outliers and 500 outliers, respectively, based on the number of outliers. Ten random experiments were conducted for each level of outliers to compare the estimated maps. Figure 6 presents the sample plots of the estimated maps. Intuitively, all three methods effectively reject outliers in dense inter-robot measurements. Once the outliers introduced by perceptual confusion are accepted, they corrupt the consistency of the fused map. We compared the results of multiple experiments, as shown in Table 2. In terms of real-time, when the number of outliers is 500, the computation time of both PCM and MEWC methods exceeds 50 s, while our method only takes 3.5 s, which reduces the computation time by 93%. In addition, PCM and MEWC determine consistent subset of inter-robot loop closures at the level of the entire measurement set, and the corresponding TPR both exceed 0.9, which is higher than the TPR estimated by our method. Note that the similarity in MEWC is all set to 1, as the dataset does not provide data similarity of loop closures. Analysis of the above two sets of simulation results, Tables 1 and 2, both prove that different numbers of outliers will have an impact on TPR, FPR, and time:  
(1) Associating TPR and ATE for analysis, the worst result of TPR corresponding to our method is 0.54 in Table 2. However, the average ATE between the global map estimated by our method and the ground truth is only 0.36, which is approximately the same as the ATE corresponding to the PCM and MEWC. The quantitative experimental result is transformed into an intuitive map for comparison. The EVO [38] is used to compare the trajectory estimated by our method and the ground truth of the manhattan dataset and calculate the statistics of the whole trajectory, as shown in Figure 7. The results are suitable for testing the global consistency of the trajectory, and it can be seen that the maximum error in this experiment does not exceed 0.5. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **Outliers** & **Ave Time /s** & **TPR** & **FPR** & **Ave ATE (/m)** & **Max ATE (/m)** \\ \hline \multirow{2}{*}{PCM} & 200 & 19.81 & 0.98 & 0.01 & 0.27 & 0.60 \\  & 500 & 53.73 & 0.92 & 0.008 & 0.37 & 0.62 \\ \multirow{2}{*}{MEWC} & 200 & 19.75 & 0.98 & 0.008 & 0.29 & 0.60 \\  & 500 & 53.90 & 0.92 & 0.008 & 0.37 & 0.62 \\ \multirow{2}{*}{Our} & 200 & 2.61 & 0.59 & 0.005 & 0.30 & 0.60 \\  & 500 & 3.51 & 0.54 & 0.002 & 0.36 & 0.67 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Simulation Results: manhattan. Figure 6: Results for manhattan dataset estimated by our method, PCM, and MEWC, in presence of 500 outliers. Final optimized map in red, correctly accepted inlier constraints are green, disabled outliers in gray. (2) For existing methods, the larger the scale of the measurement set, the higher the corresponding consistency matrix dimension, and the longer the time required to solve the high-dimensional matrix. Our method takes the cluster corresponding to the trajectory overlap region as the basic unit to estimate the consistency subset. It decomposes the high-dimensional matrix into multiple sub-matrix blocks, significantly reducing the computational time required to solve the maximum clique problem. Combined with the experimental results, it can be concluded that the lower TPR means that our method discards a part of the potential true inter-robot loop closures, but the lower ATE demonstrates that the proposed method does not lose any important, meaningful information.