# Realistic Texture Mapping of 3D Medical Models Using RGBD Camera for Mixed Reality Applications
## 3 Results  
This study proposes a method for enhancing a 3D model, obtained from CBCT images, with the real external texture using an RGBD camera. The resulting outcome was then displayed using augmented reality and it was also possible to interact with it through hand gestures. The pointcloud related to the internal tissues of the phantom was obtained by performing a CBCT scan of the same, segmenting its outer edge in each image, and then transforming all the points into a 3D pointcloud using the DICOM standard slice thickness parameter, as reported in Equation (1). The obtained result is shown in Figure 4.  
The pointcloud related to the external texture of the object was obtained using an Intel D435 3D camera, registering several pointclouds using the ChArUco marker shown in Figure 5 for camera pose estimation. The obtained result is shown in Figure 7.  
The first step for registering the pointclouds involved the alignment of their bounding boxes' front faces. The result obtained is shown in Figure 8. Subsequently, the registration process used an iterative algorithm to minimize the distance between the contours of the two models obtained by applying different rotation vectors \(R_{1}\) and projecting the result onto the \(xy\) plane of the 3D camera reference system \(O_{xyz}^{RGBD}\). To better understand how the registration error \(E\) varied with increasing iterations, a graph was constructed and is reported in Figure 13.  
As shown in the graph, the mean registration error \(E\) starts from a value \(E=1.4\), then tends to decrease with increasing iterations. Specifically, the stopping condition of the minimization process (\(E\leq E_{thr}=1\) mm) was reached after three iterations, reaching the value \(E=0.9\). At the end of the registration, the two models were correctly aligned, as shown in Figure 11. To assess the quality of the registration process, the average registration error was calculated based on different sets of randomly chosen points on the RGBD pointcloud. The obtained results are presented in Table 1.  
As shown in the table, the registration error remains nearly stable regardless of the number of randomly chosen points. Starting from \(E_{10,000}\) the average error along the three axes stabilizes at 1.1 \(\pm\) 1.3 mm for the \(x\)-axis, 0.7 \(\pm\) 0.8 mm for the \(y\)-axis, and 0.9 \(\pm\) 1.2 mm for the \(z\)-axis. After evaluating the registration error, a heatmap representing the spatial distribution of the registration error between the two pointclouds was also created (Figure 14). At the end of the alignment process, the colours of the RGBD pointcloud were applied to the DICOM-based one using a criterion based on the minimum euclidean distance. Then, MeshLab software was used to create the real coloured 3D mesh of the object (Figure 12). Finally, a software application was developed to enable the visualization of the coloured mesh in a virtual environment using the Microsoft HoloLens 2 device. The user interaction with the model allowed for translations, rotations, and scaling using hand gestures. The augmented reality visualization of the 3D model and user interactions is illustrated in Figure 15. \begin{table}
\begin{tabular}{c c c c} \hline  & \(E_{x}\) [mm] & \(E_{y}\) [mm] & \(E_{z}\) [mm] \\ \hline \(E_{100}\) & 1.1 \(\pm\) 1.3 & 0.5 \(\pm\) 0.6 & 0.8 \(\pm\) 1 \\ \hline \(E_{1000}\) & 1.1 \(\pm\) 1.2 & 0.6 \(\pm\) 0.8 & 0.9 \(\pm\) 1.2 \\ \hline \(E_{10,000}\) & 1.1 \(\pm\) 1.3 & 0.7 \(\pm\) 0.8 & 0.9 \(\pm\) 1.2 \\ \hline \(E_{20,000}\) & 1.1 \(\pm\) 1.3 & 0.7 \(\pm\) 0.8 & 0.9 \(\pm\) 1.2 \\ \hline \(E_{30,000}\) & 1.1 \(\pm\) 1.3 & 0.7 \(\pm\) 0.8 & 0.9 \(\pm\) 1.2 \\ \hline \end{tabular}
\end{table} 
Table 1: Registration errors on each principal axis for different numbers of random points. \(E_{x}\) represents the error with respect to the \(x\) axis, \(E_{y}\) represents the error with respect to the \(y\) axis, and \(E_{z}\) represents the error with respect to the \(z\) axis. Each value is reported as the mean value and standard deviation.