# URSIM: Unique Regions for Sketch Map Interpretation and Matching
## 6 Experiments
### Stability with Respect to the Threshold \(m\)  
The planar matching depends on a parameter \(m\), presented in Section 5.2, representing the threshold under which two vertices are considered similar. When looking at Figure 11, one can see that, for the previous experiment, the best \(m\) values are under 0.25 for the sketch map dataset and under 0.425 for the partial map dataset. We evaluated how stable URSIM's accuracy is with respect to the selection of the parameter \(m\) by calculating the \(F_{1}\) scores with a fixed \(m\) value, for all maps in the datasets. We tested values of \(m\) ranging from \(m=0\) to \(m=0.5\) with a step size of 0.05. While the threshold \(m\) can vary between \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Algorithm** & **Baseline** & \(\mathbf{T_{1}}\) & \(\mathbf{T_{2}}\) & \(\mathbf{T_{3}}\) & \(\mathbf{T_{4}}\) \\ \hline \multicolumn{6}{c}{_Median value_} \\ \hline Hungarian & 0.00 & **0.25** & 0.19 & 0.19 & 0.12 \\ VF2 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 \\ Planar & 0.19 & 0.40 & **0.48** & 0.37 & URSIM: 0.37 \\ DAM & & & 0.94 & & \\ \hline \multicolumn{6}{c}{_Mean \(\pm\) one standard deviation_} \\ \hline Hungarian & \(0.03\pm 0.06\) & **0.25** \(\pm\)**0.19** & 0.21 \(\pm\) 0.20 & 0.17 \(\pm\) 0.16 & 0.22 \(\pm\) 0.21 \\ VF2 & \(0.00\pm 0.00\) & \(0.00\pm 0.00\) & \(0.00\pm 0.00\) & \(0.00\pm 0.00\) \\ Planar & \(0.23\pm 0.21\) & \(0.45\pm 0.19\) & **0.46** \(\pm\)**0.18** & 0.40 \(\pm\) 0.25 & URSIM: 0.39 \(\pm\) 0.22 \\ DAM & & & 0.78 \(\pm\) 0.31 & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Median and mean \(F_{1}\) scores over the partial sensor map dataset. The best results are found when using the planar matching with the anchors. When using the anchors to calculate the LFS, the mean stays stable but the median increases. Hence, overall, while more map matches obtained higher \(F_{1}\) scores, maps with a low \(F_{1}\) score are even lower than the maps with a low score when the anchors are not used. Figure 10: Partial sensor maps used by the two users as URSIM’s _metric_ map to build the ground truth correspondences. Both maps represent the same part of the environment. However, since they were built at different times and different doors were opened, the robot mapped a different set of rooms for each map. (**a**) Partial map used as the _metric_ map for the first user. (**b**) Partial map used as the _metric_ map for the second user. 0 and 1, we found that above 0.5 the \(F_{1}\) scores stayed constant and thus we only present the results between \(m=0\) and \(m=0.5\). The stability is evaluated by looking at the median and mean of the \(F_{1}\) scores. In Figure 12, one can see a box plot of the \(F_{1}\) scores over the sketch map dataset for fixed values of \(m\). While both maximum median and mean are for \(m=1.5\), with the maximum median being 0.67 and the maximum mean 0.57, the mean and median \(F_{1}\) scores using a fixed \(m\) are significantly lower than the median and mean \(F_{1}\) scores when choosing \(m\) on a per-map basis (see Section 6.2). Hence, while for all sketch maps the best \(m\) value is under 0.25, using a fixed \(m\) value over the whole sketch map dataset leads to poorer matching results. In Figure 13, one can see the same box plot as in Figure 12 over the partial map dataset. The region descriptors were built using only the topology and the position of anchors in the graph (see Figure 9), since this strategy returned the highest \(F_{1}\) scores on the partial sensor map dataset, as shown in  
Figure 11: For all maps in the datasets, those charts show for how many maps a given \(m\) value lead to the best set of correspondences. On top are the results for the sketch map dataset and at the bottom the results for the partial map dataset. All best-correspondences \(m\) values are under 0.25 for the sketch map dataset and under 0.425 for the partial map dataset. Figure 12: \(F_{1}\) scores when using URSIM with \(m\) ranging from 0 to 0.5 on the sketch dataset. The highest score was found around \(m=0.15\).