# DALLoc: Deep-Learning Accelerated Visual Localization Based on Mesh Representation
## 1 Introduction  
Precise visual localization of 6-degree-of-freedom (DoF) camera poses within a known three-dimensional structure is one of the most important abilities in some modern intelligence systems, e.g., self-driving cars [1; 2] and augmented or virtual reality systems [3; 4]. Obtaining a centimeter-accurate 6-DoF pose in real time is essential to guarantee reliable and safe driving and ensure immersive experiences. In terms of accuracy, most state-of-the-art visual localization methods are structure-based [5; 6; 7; 8]. These approaches deploy a coarse-to-fine way to generate 2D-3D correspondences between a feature point from the query image and a database 3D point; each 3D point has its local features in database images.  
However, in these methods, both 2D-2D local feature mismatch of query feature and database image features and 2D-3D mismatch of a database feature and database 3D point will lead to wrong 2D-3D match of a query feature and database 3D point, especially in large scale scene, which contains millions of features and points to match. Therefore, generating robust matching pairs within a limited time is essential for some real-time visual localization algorithms. The general way to filter out mismatches is to apply a modernrandom sample consensus (RANSAC) framework, which utilizes a minimal solver for optimization [9; 10; 11].  
Moreover, the database point cloud is usually generated from the structure-from-motion (Sfm) process and the scene is presented with this point cloud and images [12; 13]. A dense 3D model scene representation [14; 15; 16] is a replacement for using sparse Sfm point clouds and images. This representation uses the mesh generated via multiview stereo, Lidar or depth map to represent the scene which is more flexible at feature point types because it does not need to match local features between database images to triangulate the 3D point cloud. The 3D point can easily be obtained from the depth map rendered from the mesh. More importantly, this rendering process will not provide wrong 2D-3D matches compared with the 3D point cloud from the Sfm model.  
In this paper, we propose to take advantage of mesh-based scene representation to alleviate the complexity of the 2D-3D matching step. We use the pretrained multilayer perceptron (MLP) combined with homotopy continuation to solve the relative poses [17]. We also use the scale consistency of 2D-3D matches to perform efficient RANSAC to find the best 2D inlier set for the subsequential perspective-n-point (PnP) [18] localization step. The input of the MLP in our approach is five-feature correspondences, and the outputs from the MLP are their corresponding depths in the scene; however, the scale is different from the rendered depth map. Using this method to solve relative pose can calculate once quickly. We employ scale consistency to verify whether a solution from the minimal RANSAC sample is a real solution or whether the input of MLP contains outliers to skip the validation step for some wrong solutions, which greatly improves the efficiency of RANSAC implementation.  
The core idea of our approach is summarized as follows. With the mesh-based scene representation, the 2D-3D matches contain no mismatch. We use the pretrained MLP combined with homotopy continuation to calculate the relative pose in a short time. We also use the scale information from rendered 2D-3D correspondences to help the RANSAC to perform more efficiently. In such a way, we can get more robust 2D-2D correspondences in a short time. Thus, the 2D-3D matches established by these 2D features are also more robust. The performance of the algorithm can be greatly improved using such an approach.