# LiDAR as a Geometric Prior: Enhancing Camera Pose Tracking Through High-Fidelity View Synthesis
## 4 Offline Keypoint-Map Tracking
### Offline Map Generation  
Let the predefined path of the robot be a sequence of \(M\) poses in the world frame \(\mathcal{F}_{W}\), denoted by the trajectory \(\mathcal{T}=\{\mathbf{T}_{1}^{p},\mathbf{T}_{2}^{p},\ldots,\mathbf{T}_{M}^{p}\}\), where \(\mathbf{T}_{k}^{p}=[\mathbf{R}_{k}^{p}|\mathbf{\xi}_{k}^{p}]\).  
1. View Rendering and Keypoint Extraction: For each pose \(\mathbf{T}_{k}^{p}\) along the trajectory, we render a synthetic RGBD image pair \((I_{k}^{s},D_{k}^{s})\) using the projection model defined in Equation (1). From each synthetic image \(I_{k}^{s}\), we detect a set of keypoints \(\mathcal{K}_{k}^{s}=\{(\mathbf{u}_{k,j}^{s},\mathbf{d}_{k,j}^{s})\}\). Critically, since each keypoint \(\mathbf{u}_{k,j}^{s}\) originates from rendering the dense point cloud \(\mathcal{P}\), we know its precise 3D coordinates \(\mathbf{X}_{k,j}\) in \(\mathcal{F}_{W}\).
2. Temporal Keypoint Filtering: To build a robust map, we only retain keypoints that demonstrate stability across consecutive frames. A keypoint with 3D position \(\mathbf{X}\) detected in frame \(k\) is considered stable if it is also visible and detectable in frame \(k+1\). We formalize this by projecting the point \(\mathbf{X}\) into the subsequent view using pose \(\mathbf{T}_{k+1}^{p}\): \[\mathbf{p}_{k\to k+1}=\pi(\mathbf{R}_{k+1}^{p},\mathbf{t}_{k+1}^{p}, \mathbf{X})\] (5) A keypoint is kept if a keypoint detected in frame \(k+1\) exists in the neighborhood of \(\mathbf{p}_{k\to k+1}\) with a similar descriptor. This filtering process yields a consistent set of trackable 3D points.
3. Keyframe-based Map Construction: The final map \(\mathcal{M}_{map}\) is constructed from these filtered 3D points. To enable efficient spatial lookups during online tracking, we structure the map around keyframes. A keyframe \(\mathcal{F}_{k}\) is defined by the tuple: \[\mathcal{F}_{k}=(\mathbf{T}_{k}^{p},\mathcal{M}_{k})\] (6) where \(\mathbf{T}_{k}^{p}\) is the pose of the keyframe and \(\mathcal{M}_{k}\) is the set of stable 3D keypoints \(\{(\mathbf{X}_{j},\mathbf{d}_{j})\}\) visible from that pose. The complete map is the collection of all such keyframes, \(\mathcal{M}_{map}=\{\mathcal{F}_{1},\mathcal{F}_{2},\ldots,\mathcal{F}_{M}\}\).  
The effectiveness of this offline tracking paradigm is critically dependent on the quality and robustness of the pre-generated keypoint map. A key factor in the map's quality is the fidelity of the synthetic views used during its offline creation, as these views must be realistic enough to produce keypoints that reliably match with real-world images. To investigate this relationship, we evaluate two different rendering approaches for the map generation phase.  
The first approach is the point-based rendering technique employed in our online method, which directly projects the 3D point cloud. As a more advanced alternative, we also generate a keypoint map using views synthesized by M2 Mapping [2]. M2 Mapping is a state-of-the-art method designed to produce high-quality, photorealistic novel views from LiDAR scans. Unlike direct point-based projection, M2 Mapping involves a two-stage process. First, it requires an offline training phase where it uses the LiDAR scans and the corresponding video sequence to train a model of the scene's geometry and appearance. Second, in a separate rendering phase, this trained model is used to synthesize novel views from any desired viewpoint. Both stages are computationally intensive and time-consuming, precluding their use in our online method; however, M2 Mapping is an excellent candidate for offline map generation, where rendering quality is crucial. By leveraging its advanced synthesis capabilities, we aim to create a more robust keypoint map that can further improve tracking accuracy, especially in visually complex scenes.