## 3 Hybrid Visual SLAM with Deep Keypoints, Depths, Poses, and Line Features
### Virtual Stereo Visual SLAM
#### 3.3.1 Self-Supervised Learning of Pose and Depth  
The self-supervised visual method uses a dual-network framework consisting of PoseNet for pose estimation and DepthNet for depth estimation. The depth estimates are obtained end-to-end by DepthNet. DepthNet is composed of an encoder and a decoder. Both are convolutional neural networks. The encoder transforms the image into high-level features, while the decoder converts these high-level features into depth estimates. The depth estimate is dense, with each pixel corresponding to the depth value of an object in the scene. The pose estimates are derived end-to-end from PoseNet. The input to PoseNet is a tensor that combines adjacent frame images along the RGB channel. A photometric model is employed to construct a self-supervised loss function, which integrates adjacent images, depth estimates, and pose estimates. Through multiple training epochs, PoseNet and DepthNet learn to accurately estimate both pose and depth. As shown in Equation (8), the relative pose estimate between adjacent frames preserves the luminance consistency of the target image \(\mathbf{I}_{t}\) with respect to the source image \(\mathbf{I}_{s}\). By combining the depth estimate of the target frame \(\mathbf{D}_{t}\) with the camera's intrinsic matrix \(\mathbf{K}\), the source image \(\mathbf{I}_{s}\) is projected onto the target frame's perspective. This projection process generates a synthesized image \(\hat{\mathbf{I}}_{s}\) from the viewpoint of the target frame, maintaining photometric consistency with the source image. Here, \(\mathbf{p}_{s}\) and \(\mathbf{p}_{t}\) represent the pixel points of the source and target frames, respectively.  
\[\mathbf{p}_{s}\sim\mathbf{K}\xi_{t}^{s}\mathbf{D}_{t}(\mathbf{p}_{t})\mathbf{ K}^{-1}\mathbf{p}_{t} \tag{8}\]  
The synthesized image \(\hat{\mathbf{I}}_{s}\) and the target image \(\mathbf{I}_{t}\) not only maintain photometric consistency but also exhibit structural similarity. As shown in Equation (9), we incorporate this quantified similarity as a structural loss into the photometric loss \(\mathcal{L}_{\text{proj}}\) to preserve the structural details of the synthesized image. Here, \(\mathcal{C}_{\text{SSIM}}\) represents the structural similarity loss [38], with \(\mu_{1}\) and \(\mu_{2}\) set to 0.15 and 0.85, respectively.  
\[\mathcal{L}_{\text{proj}}=\mu_{1}\cdot|\mathbf{I}_{t}-\hat{\mathbf{I}}_{s}|+ \mu_{2}\cdot\frac{1-\mathcal{C}_{\text{SSIM}}(\mathbf{I}_{t},\hat{\mathbf{I}} _{s})}{2} \tag{9}\]  
In addition, to maintain depth scale consistency throughout the sequence, we introduce the 3D geometry consistency loss \(\mathcal{L}_{\text{DG}}\). The equation for this loss is shown in 10. \(\hat{\mathbf{D}}\)s refers to the reconstructed depth in the source frame, which should be consistent with the transformed depth obtained through \(\xi^{s}\mathbf{D}_{t}\), and the operation \(\xi_{t}^{s}\mathbf{D}_{t}\) represents the transformation of the target frame's depth estimate \(\mathbf{D}t\) into the source frame's perspective using the relative pose estimate \(\xi_{t}^{s}\).  
\[\mathcal{L}_{\text{DG}}=\frac{|\hat{\mathbf{D}}_{s}-\xi_{t}^{s}\mathbf{D}_{t}| }{\hat{\mathbf{D}}_{s}+\xi_{t}^{s}\mathbf{D}_{t}} \tag{10}\]  
As shown in Equation (11), we combine the photometric loss \(\mathcal{L}_{\text{proj}}\) and the 3D geometric consistency loss \(\mathcal{L}_{\text{DG}}\) to form the self supervised training loss \(\mathcal{L}_{\text{ss}}\), where \(\mu_{3}\) is 0.5.  
\[\mathcal{L}_{\text{ss}}=\mathcal{L}_{\text{proj}}+\mu_{3}\mathcal{L}_{\text{DG}} \tag{11}\]