# A Survey of Visual SLAM Based on RGB-D Images Using Deep Learning and Comparative Study for VOE
## 3 Visual SLAM and Visual Odometry Using Deep Learning: Survey
### Deep Learning-Based Module for Visual SLAM and Visual Odometry
#### 3.1.4 DL Modules Add to the Visual SLAM Algorithm
The GT data include GT trajectory information based on the camera pose GT calculated from the IMU data of the iPhone. To evaluate the DL module for semantic segmentation added to the Visual SLAM system, the studies evaluated the following measurements. The Mean Tracking Rate (\(MTR\)) [115] is the tracking success rate in 50 trials when successful tracking is performed on 80% of the 1000 frames of the sequence, computed based on Formula (5). \[MTR=\frac{1}{m}\sum_{i=1}^{m=50}\left(TrackingRate_{i}\right) \tag{5}\]  
where \(TrackingRate_{i}\) is the \"Tracking Rate\" (%) at time \(i^{th}\), and \(m\) is the number of times the \"Tracking Rate\" is performed. The Mean Trajectory Error (\(MTE\)) [115] is an estimate of the camera's position relative to the defined GT, which is the error distance for each time step and the average value of a sequence as \"Trajectory Error (m)\". Here, it only calculates the \(MTE\) for \"Success Tracking\" i.e., if the \"Tracking Rate\" exceeds 80%, the \(MTE\) is computed based on Formula (6). \[MTE=\frac{1}{m}\sum_{i=1}^{m}(\frac{1}{n_{i}}||X_{t}-Y_{it}||_{2}) \tag{6}\]  
where \(i=1,2,\ldots,50\) is the number of \"Successful Tracking\" trials. \(X_{t}\) is the 3D position of the GT trajectory, and \(Y_{it}\) is the 3D position of the estimated trajectory over the entire time series \((t=1,\ldots,n_{i})\); \(n_{i}\) is the length of the time series for performing the VOE. The \(ATE\) is presented in the Formula (4). The Intersection over Union (\(IOU\)) is a measure to evaluate the results of detecting objects in the scene during the process of building the Visual SLAM system, and the \(IOU\) is calculated according to Formula (7). \[IOU=\frac{BB_{g}\cap BB_{r}}{BB_{g}\cup BB_{r}} \tag{7}\]  
where \(BB_{g}\) is the bounding box GT of the object, and \(BB_{r}\) is the bounding box prediction of the object. The pixel accuracy (\(PA\)) is calculated according to Formula (8). \[PA=\frac{\sum_{i}n_{ii}}{\sum_{i}t_{i}} \tag{8}\]  
The mean precision (\(MP\)) is calculated according to Formula (9). \[MP=\frac{1}{n_{cl}}\frac{\sum_{i}n_{ii}}{\sum_{j}n_{ii}} \tag{9}\]where \(n_{ij}\) is the number of pixels which classified as \(j\), while the true value is \(i\). \(n_{cl}\) is the total classes. \(t_{i}\) is the number of pixels that belong to class \(i\), with \(t_{i}=\sum_{j}n_{ij}\). Another type of measure used is the absolute translation (trans.) error \(RMSE\) (Tabs) [130], which is the distance between the estimated trajectory and the GT trajectory. The calculation of \(RMSE\) (Tabs) was performed in the study of [12]. In [134]'s study, measurements such as \(RMSE\), mean error, and Absolute Pose Error (\(APE\)) were used. These measures were also defined in [140]. The results of Visual SLAM when using the DL semantic segmentation module are presented in Table 6.  
The results have been evaluated on multiple measures (\(MTR\), \(MTE\), \(ATE\), \(IOU\), \(PA\), \(MP\), \(RMSE\) (Tabs), \(RMSE\), mean error, and \(APE\)) with multiple datasets (CARLA [139], TUM RGB-D SLAM dataset, ObjectFusion dataset I, ObjectFusion dataset II, ObjectFusion dataset III, ObjectFusion dataset VI [127], ICL-NUIM dataset [13], and ADVIO dataset [38]). Although they all use DL for semantic segmentation and are added to the Visual SLAM system, each dataset and method uses a different measure, so in Table 6, there are many empty results. c. Pose estimation module DL  
Pose estimation is the process of estimating the camera pose as the subject carrying the camera moves in the environment/scene. In this section, we survey methods and research on using DL for camera pose estimation. [PERSON] et al. [141] proposed an Object-Fusion system to estimate the camera pose of each RGB-D frame and build 3D object surface reconstruction in the scene. To do this, the instance segmentation masks are detected in each frame and used to encode each object instance to a latent vector by a deep implicit object representation; to detect each object instance, the object shape and pose are initialized.