## 3 Hybrid Visual SLAM with Deep Keypoints, Depths, Poses, and Line Features
### Integrating Line Features and Deep Keypoints in the Front End
#### 3.2.2 Self-Supervised Learning of Deep Keypoints  
Traditional handcrafted keypoint methods, such as ORB [35] and SURF [36], detect corners with significant brightness changes in the image using image gradient operators and then use the local low-level features around the corner as descriptors. However, low-level image features are quite sensitive to lighting changes in environment. If the same scene is captured at different times, the same keypoint may have different descriptors due to varying lighting conditions. This low robustness introduces considerable uncertainty in SLAM feature matching, especially in scenarios with loop closures. In contrast, learning-based keypoint extraction methods transform images into descriptor and keypoint tensors end-to-end. These tensors are derived from the high-level features of the neural network. These high-level features are essentially global features of the image. Compared to the local features obtained by traditional methods, global features are more robust. Learning-based keypoint extraction methods employ supervised training, and the supervised signals come from the self-labeled method [27]. The backbone of our local feature extractor is a convolutional neural network (CNN). Similar to other learned local feature methods, it is divided into an encoder and a decoder. The encoder leverages the convolutional layers of VGG16, while the decoder consists of two parts: a keypoint position decoder and a descriptor decoder. The keypoint position decoder converts high-dimensional visual features from the encoder into a heatmap of the same size as the input image, where each pixel value represents the probability of a keypoint's presence. A threshold is preset, and pixels in the heatmap that exceed this threshold are considered candidate keypoints. The descriptor decoder, on the other hand, transforms the high-dimensional visual features from the encoder into 256-dimensional descriptors. These descriptors are extracted at the candidate keypoint positions to serve as the output descriptors. Our local feature extractor employs a self-supervised approach, using self-labeled training data to supervise the extraction process. Initially, we pretrain corner extractors on a synthetic geometric dataset. Then, we use the pretrained corner extractor to detect keypoints on the COCO dataset [37], treating these detected keypoints as annotations for the corresponding RGB images. We combine the RGB images with the self-annotated keypoints to form the training set for our learned local feature method. The training loss is composed of two components: keypoint localization loss and descriptor loss. The keypoint localization loss measures the discrepancy between the predicted keypoint locations and the self-annotated positions. As shown in Equation (3), we train the keypoint localization loss using both the original image and its homography-warped version. We employ CrossEntropy loss as the metric for keypoint localization. This loss consists of the raw keypoint localization loss \(\mathcal{L}_{k}\) and the homography-warped keypoint localization loss \(\mathcal{L}_{wk}\). Here, \(H_{w}\) and \(W_{w}\) denote the height and width of the heatmap, with \(H_{w}=H/8\) and \(W_{w}=W/8\), while \(i\) represents the keypoint position index, and \(H\) and \(W\) denote the height and width of the input image. The variables \(\mathbf{x}\) and \(\mathbf{\tilde{x}}\) represent the keypoint positions and keypoint label positions, respectively. \[\mathcal{L}_{wk}=\mathcal{L}_{k}=\frac{1}{H_{w}W_{w}}\sum_{i=1}^{H_{w}W_{w}}l _{cross}(\mathbf{x}_{i};\mathbf{\tilde{x}}_{i}) \tag{3}\]  
Unlike SuperPoint, we implement a sparse descriptor loss. This sparse descriptor loss is computed between the original image and its homography-warped version. It consists of two main components: the loss for descriptors matched through homography \(\mathcal{L}_{d1}\) and the loss for those that are unmatched \(\mathcal{L}_{d2}\).