# LiDAR as a Geometric Prior: Enhancing Camera Pose Tracking Through High-Fidelity View Synthesis
## 1 Introduction  
Accurate camera pose estimation is a cornerstone of many computer vision and robotics applications, including autonomous navigation, augmented reality, and 3D scene reconstruction. The ability to precisely localize a camera within a known environment enables systems to interact meaningfully with the world, plan trajectories, overlay virtual content, or maintain spatial awareness. While conventional approaches often rely on sparse feature matching or visual-inertial fusion, recent advances in 3D sensing and mapping technologies offer new opportunities to enhance pose estimation using rich geometric priors. In this paper, we investigate the potential of leveraging high-fidelity LiDAR-based 3D reconstructions to support robust and accurate camera pose tracking. Specifically, we employ FastLIVO2 [1], a state-of-the-art LiDAR (Light Detection and Ranging) odometry algorithm, to generate dense point clouds augmented with color information at each point. This detailed representation not only captures precise geometric structure but also encodes photometric cues, enabling us to synthesize realistic views of the scenefrom arbitrary viewpoints. Figure 1 shows an example of a scene reconstructed using FastLivo2. The dataset was recorded using both an AVIA Livox LiDAR and an RGB camera. The top image, Figure 1a, shows the three-dimensional point cloud reconstruction. Then, Figure 1b,d show images captured with the RGB camera, and Figure 1c,e show rendered images from the exact camera location. As can be seen, the synthetic images are very similar to the real ones. Our method benefits from the realism of the rendered views, which helps bridge the domain gap between synthetic and real images, thereby improving keypoint matching accuracy. Our primary approach uses this reconstruction to render a synthetic image from a known camera pose. Keypoints are then detected in both the rendered and real camera images, and correspondences between them are used to estimate the current pose. This process is iteratively applied to achieve continuous frame-to-frame tracking. To further improve robustness and provide a strong initial estimate for each frame, we integrate inertial data from the IMU (Inertial Measurement Unit) sensor to predict the camera's motion, using this prediction as a prior for our vision-based pose estimation. However, the need to render photorealistic views at every frame introduces significant computational demands, which can limit deployment in resource-constrained platforms, such as embedded systems or low-power mobile robots. To mitigate this challenge, we propose a complementary Offline Keypoint-Map Tracking strategy designed for scenarios in which the camera trajectory is known a priori. Instead of performing rendering at runtime, we precompute a 3D keypoint map tailored to the expected viewpoints along the trajectory. This offline preprocessing shifts the computational load away from runtime, allowing for lightweight pose tracking during deployment by matching incoming frames to the precomputed keypoints. To further evaluate this strategy, we explore two variants of the offline pipeline. The first uses the same point-based rendering technique employed in the real-time approach. The second adopts M2 Mapping [2], an advanced view synthesis method that generates high-quality novel views by learning richer image-based representations. While this method produces highly realistic images, its significant computational requirements, including a lengthy training phase, make it unsuitable for real-time applications, restricting its use to our offline strategy. M2 Mapping enhances the visual fidelity of the rendered images, potentially improving matching robustness in texture-sparse or geometrically complex scenes. Through comprehensive experiments across diverse environments, we demonstrate that both the real-time and offline approaches offer viable solutions under different op  
Figure 1: Retail Street sequence. (**a**) Three-dimensional reconstruction of the Retail Street sequence obtained with FastLivo2.