# Accurate Landmark Localization for Medical Images Using Perturbations
## 4 Results
### Quantitative Performance Comparison with Existing Studies  
In this section, we perform a quantitative performance comparison with existing studies by using the best perturbation method for each dataset. We trained two networks, U-net and Attention U-net. In Table 5 using Digital Hand Atlas dataset, the existing study of [PERSON] et al. [33] achieved 2 mm range EDR of 5.01% and mean PE of 0.66. Our hybrid perturbator based on U-net achieved 2 mm range EDR of 3.96% and mean PE of 0.64. The hybrid perturbator based on Attention U-net showed a poorer performance than U-net. It achieved 2 mm range EDR of 4.76% and mean PE of 0.71. In Table 6 using ISBI 2015 dataset (test1), [PERSON] et al. [35] achieved a 2 mm range EDR of 13.33% and median PE of 11.7. Our edge detection perturbator based on Attention U-net achieved 2 mm range EDR of 13.26% and median PE of 11.48. The same perturbator based on U-net achieved 2 mm range EDR of 14.42% and median PE of 11.78. Table 7 shows the results of the ISBI 2015 on-site competition data. The best existing study is [PERSON] et al. [11], which achieved 2 mm range EDR of 24.11% and median PE of 14.45. Our edge detection perturbator based on U-net achieved 2 mm range EDR of 25.58% and median PE of 14.82. The same perturbator based on Attention U-net achieved 2 mm range EDR of 25.21% and median PE of 15.30. The ISBI 2015 test1 data is different from the test2 data. Previous studies [11; 42] have reported that landmarks 3 (Orbitale), 6 (Supramentale), 13 (Upper lip), and 16 (Soft tissue pogonion) have a large performance gap between test1 and test2 data. This factor does not guarantee that experiments in the same environment will produce similar results in the test2 dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**PE\({}_{all}\) (px)**} & \multicolumn{3}{c}{**EDR (\%)**} \\ \cline{2-7}  & **Median** & **Mean \(\pm\) SD** & **\textgreater{}2 mm** & **\textgreater{}2.5 mm** & **\textgreater{}3 mm** & **\textgreater{}4 mm** \\ \hline U-net without & \multirow{2}{*}{9.12} & \multirow{2}{*}{12.03 \(\pm\) 11.39} & \multirow{2}{*}{15.47} & \multirow{2}{*}{8.95} & \multirow{2}{*}{5.58} & \multirow{2}{*}{2.35} \\ perturbation & & & & & & \\ \hline Blackout & 9.03 & 11.91 \(\pm\) 12.00 & 14.88 & 8.88 & **4.98** & 2.42 \\ \hline Whiteout & 9.03 & **11.70**\(\pm\) 10.23 & 14.60 & 8.84 & 5.75 & 2.00 \\ \hline Smoothing & 9.32 & 12.10 \(\pm\) 11.47 & 15.02 & 9.33 & 6.04 & 2.42 \\ \hline Binarization & 9.28 & 11.83 \(\pm\) **10.19** & 15.37 & 9.23 & 5.33 & **1.96** \\ \hline Edge detection & **8.90** & 11.78 \(\pm\) 12.37 & **14.42** & **8.77** & 5.65 & 2.28 \\ \hline Hybrid & 8.94 & 11.72 \(\pm\) 12.01 & 14.70 & 9.33 & 5.12 & 2.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Localization results according to perturbator on the ISBI 2015 dataset (test1). \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**PE\({}_{all}\) (mm)**} & \multicolumn{3}{c}{**EDR (\%)**} \\ \cline{2-5}  & **Median** & **Mean \(\pm\) SD** & **\textgreater{}2 mm** & **\textgreater{}4 mm** & **\textgreater{}10 mm** \\ \hline Payer [33] & 0.91 & 1.13 \(\pm\) 0.98 & 12.4\% & 1.34\% & 0.04\% \\ \hline [PERSON] [26] & 0.51 & 0.80 \(\pm\) 0.91 & 7.80\% & 1.55\% & 0.05\% \\ \hline [PERSON] [34] & 0.51 & 0.80 \(\pm\) 0.93 & 7.81\% & 1.54\% & 0.05\% \\ \hline [PERSON] [10] & **0.43** & 0.66 \(\pm\) 0.74 & 5.01\% & 0.73\% & **0.01**\% \\ \hline Hybrid perturbator (U-net) & 0.45 & **0.64 \(\pm\) 0.64** & **3.96\%** & **0.34**\% & 0.02\% \\ \hline Hybrid perturbator (Attention U-net) & 0.45 & 0.71 \(\pm\) 1.26 & 4.76\% & 0.96\% & 0.27\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Quantitative comparison with existing methods on the Digital Hand Atlas dataset. Figure 4a shows mean radial error (MRE) curves for the Digital Hand Atlas dataset according to different perturbations. As Table 3 illustrates, the hybrid perturbation depicted with black dashed line shows the best performance.