# DALLoc: Deep-Learning Accelerated Visual Localization Based on Mesh Representation
## 4 Methods and Material
### Learning-Based Relative Pose Estimation  
Relative pose estimation is not required in the above method. Because regressing the query pose from the database views is inaccurate, as estimating the pose from structure-based methods contains consistency checks within a RANSAC. However, in our method we use the relative pose estimation mechanism as a strong outlier filter, which greatly improves the efficiency and accuracy of the localization algorithm.  
Ref. [17] proposed a deep-learning method to estimate the relative pose of two images, which is also feature-based same as the five-point method [40]. Due to geometry consistency, as illustrated in Figure 3, the distance between every two 3D points, when reconstructed in different cameras, must be the same. Thus, the form of a five-point problem in this method is parametrized by the projection of five 3D points into 2 calibrated cameras. There are ten unknown depths of five 3D points in two images \(\lambda_{k,i}\), \(k=1,...,5,i=1,2\), and the result can be solved by \(10=\binom{5}{2}\) equations given in (1):  
\[\left\|\lambda_{k,i}\
u_{k,i}-\lambda_{m,i}\
u_{m,i}\right\|_{2}=\left\| \lambda_{k,j}\
u_{k,j}-\lambda_{m,j}\
u_{m,j}\right\|_{2} \tag{1}\]  
where \(\lambda_{k,i},\lambda_{k,j},\lambda_{m,i},\lambda_{m,j}\) are two features correspondences, and \(\
u_{k,i},\
u_{k,j},\
u_{m,i},\
u_{k,j}\) are their depths in the scene.  
Figure 2: Mesh-based scene representation. (**a**,**b**) are two different views of the mesh-model.  
We also use deep-learning to solve this problem. We first extract pairs of five local feature correspondences and their ten corresponding depths from the SFM model to be the training data. In this paper, we call this form of pair _anchors_. We choose a certain number of anchors with the best connectivity to be the output of MLP.  
At test time, with arbitrary five local feature correspondences selected from the query image and database image as inputs, the MLP can find which anchor has the nearest geometry to track according to the input local features. It will also track this anchor with homotopy continuation to output the depth of these ten local features. Then, the coordinates of local features and their depths in the scene can be decomposed into the relative pose with a unique solution. Figure 4 shows the entire pipeline of this algorithm.  
There are several reasons why we incorporate this relative pose estimating stage. First, estimating relative pose using MLP combined with homotopy continuation has orders of magnitude advantage of time compared with other methods (calculating time is on the level of us) and time spent on adding additional relative pose estimation is negligible in the entire algorithm. Moreover, the nearest anchors and training data are extracted from the Sfm model. Therefore, we can model the 3D structure of the scene with MLP.  
Second, the solution was tracked from the anchors extracted from the SFM model. Therefore, we can directly obtain the depths of the local features in the 3D structure, where the depths are in a convenient format for the validation step of our localization algorithm.  
Third, this network is lightweight and thus easy to train. It can easily generate training data. Therefore, we can customize this neural network for different scenarios to improve the performance of finding the real geometry. We train our MLP using data extracted from Aachen DayNight v1.1 and RobotCar Seasons datasets. We also compared the performance under the Aachen DayNight v1.1 (Table 1). Here, Aachen/RobotCar Seasons [%] indicates the success ratio of finding the real geometry with MLP trained on two different datasets. We can see that the success ratio of MLP trained on the Aachen DayNight v1.1 dataset is nearly twice as good as that on the RobotCar Seasons dataset.  
Figure 3: Form of five-point problem.