Title: LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D computer vision; Pose estimation; Privacy-preserving localization
Abstract: We propose a new method named LoD-Loc for visual localization in the air. Unlike existing localization algorithms, LoD-Loc does not rely on complex 3D representations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given a coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypotheses to describe pose probability distribution and select a pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises a 6-DoF pose optimization algorithm to refine the previous result with a differentiable Gaussian-Newton method. As no public dataset exists for the studied problem, we collect two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. The code and dataset will be made available upon publication.

LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment

Abstract

We propose a new method named LoD-Loc for visual localization in the air. Unlike existing localization algorithms, LoD-Loc does not rely on complex 3D representations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given a coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypotheses to describe pose probability distribution and select a pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises a 6-DoF pose optimization algorithm to refine the previous result with a differentiable GaussianNewton method. As no public dataset exists for the studied problem, we collect two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. The code and dataset are available at https://victorzoo.github.io/LoD-Loc.github.io/ .

1 Introduction

Aerial visual localization is the process of determining the global position and orientation for a UA V camera relative to a known map. This process benefits many important applications, ranging from cargo transport [69], surveillance [19, 72], to search and rescue [15, 63].

Following localization algorithms on the ground [18, 30, 42, 53, 54, 57, 67, 68, 70, 77], current aerial visual localization approaches [22, 72] typically involve matching pixels in a query image with points in a pre-built high-quality 3D map, which is often derived from 3D texture models [43, 76, 72]. Subsequently, a Perspective-n-Point (PnP) RANSAC [32, 37, 14, 23, 27, 24] technique is commonly used to calculate the camera pose. However, building high-quality 3D maps using photogrammetry [12, 28, 64, 58, 33] is expensive on a global scale and requires frequent updates to account for temporal changes in visual appearance. Besides, these 3D maps are costly to store, which poses significant challenges for terminal deployment on drones. Furthermore, high-resolution 3D maps disclose detailed information about the localization area, raising critical concerns regarding homeland security and privacy preservation.

6-DoF Pose Estimation over LoD Model

To address these challenges, we propose to leverage the Level of Detail (LoD) 3D city maps [38, 16] as the cue for localization, as illustrated in Figure 1. Compared with traditional textured 3D models, LoD 3D models enjoy the following advantages: 1) Ease of acquisition and maintenance : World-scale LoD city models can be generated with the rapid development of remote sensing [17, 34, 48, 52, 31]. Many commercial companies, such as Google Maps [8] and Baidu Maps [7], have already integrated LoD 3D models into their MAP Applications. 2) Light-weight map size : LoD maps are extremely compact, up to 10 4 times smaller in size than textured 3D maps, enabling on-device localization over large areas. 3) Privacy preservation and policy-friendly : As LoD city models only reveal the basic 3D outlines of buildings in a highly abstracted and simplified manner, they raise fewer concerns about the disclosure of privacy and land resource secrets.

To achieve high accuracy and low memory usage, we propose a hierarchical scheme for pose selection that utilizes multiple small pose volumes, to progressively compute poses in a coarse-to-fine manner. Throughout the hierarchy, we adopt an adaptive sampling strategy, where the variance-based uncertainty from the previous stage influences the sampling range of the next stage for constructing pose cost volume. This adaptive process enables reasonable and fine-grained spatial partitioning of poses, resulting in a significant improvement in the final pose output.

To facilitate research in this area, and to train and evaluate our method, we release two datasets with map levels of LoD3.0 and LoD2.0, respectively, as shown in Figure 2. For the LoD3.0 dataset, we employ a semi-automatic method to generate LoD model data from a recent large-scale oblique photography scene [72], covering an area of 2.5 square kilometers. The query images are captured by drones, with sensor data (e.g., GPS, IMU) recorded. For the LOD2.0 dataset, we use LoD model data provided by the Swiss federal authorities, specifically the SwissTOPO [9-11] data near École

LoD-Loc

Polytechnique Fédérale de Lausanne (EPFL), covering an area of 8.2 square kilometers. The query images with ground-truth poses are sourced from the CrossLoc [74] project.

We conduct extensive experiments on these two datasets. The results show that, due to the lack of both color and texture in the LoD 3D model, previous state-of-the-art image retrieval-and-matching methods [44, 53, 54, 29, 41, 25, 66, 46, 47] basically fail. In contrast, our method consistently achieves excellent results, even surpassing current state-of-the-art methods [53, 54, 66, 72] that use textured 3D models for localization.

Contributions.

We propose the use of Level of Detail (LoD) 3D maps for 6-DoF visual localization in the air.

We introduce a novel localization method that utilizes wireframe alignment for pose estimation.

Our method is differentiable, allowing the pipeline to be trained end-to-end with pose supervision.

We release two LoD city datasets, complete with RGB queries and ground-truth pose annotations.

2 Related Works

Localization from SfM or Mesh Map. SfM maps typically consist of reference images and 3D track points with their associated features [58]. For a given query image, an image retrieval method [13, 29] is initially utilized to identify co-visible reference images. Following this, feature matching algorithms [41, 25, 54, 66] are employed to establish accurate 2D-2D correspondences between the query image and the identified reference images, with track information being used to transform these 2D-2D correspondences into 2D-3D relationships. Finally, the pose is resolved using PnP RANSAC [32, 37, 14, 23, 27, 24, 77].

Mesh maps are typically defined by a textured mesh model. Initially, reference images with depth are rendered at appropriate viewpoints surrounding the model [72, 43, 44, 76]. Similar image retrieval [13, 29] and matching [41, 25, 54, 66] processes are employed to identify co-visible images and to establish 2D-2D correspondences. The depth map is utilized to transform 2D-2D correspondences into 2D-3D relationships, and the pose is subsequently determined by a PnP RANSAC [32, 37, 14, 23, 27, 24, 77].

Despite providing high-accuracy localization results, both SfM and mesh models present significant challenges in terms of reconstruction and maintenance. Additionally, their extensive size complicates deployment, necessitating its existence solely in the cloud. Moreover, these maps raise serious concerns about the privacy of personal and land resource leakage.

Localization from other types of Map. To mitigate these issues, researchers have proposed to use alternative types of maps for localization. In addressing the difficulty of map reconstruction and maintenance, some methods opt for overhead imagery such as satellite [59, 60, 73, 56], or leverage OpenStreetMap [55] as a reference. However, these methods are limited to estimating a 3-DoF (planar position and heading) pose at most. To address the issue of large map size, some methods have made attempts to compress the maps [20, 21, 78], reduce model complexity [44], or utilize geometry information without features [49, 79, 39, 75]. In the context of privacy, some methods [36, 61, 65] propose to transform 3D point clouds into 3D line clouds, leverage semantic

.

in-Traj.

out-of-Traj.

out-of-Place

LoD models with details

Query samples

Swiss-EPFL

UAVD4L-LoD

in-Place

information from point clouds, or utilize semantic 3D maps to enhance privacy [50]. Some other methods apply learning-based pose regression or scene point regression models [35, 62, 40] that do not explicitly store the 3D map. However, the effectiveness and generalizability of these methods are often inferior to those that rely on SfM or texture mesh maps. A detailed comparison of attributes from different maps is provided in Table 1.

3 Method

Given a 3D city LoD map M , a query image I , and its coarse sensor pose ξ p , the goal of the proposed method is to compute the absolute 6-DoF pose ξ ∗ . First, a convolutional neural network is used to extract the wireframe probability map for the query image I at multiple levels (Sec. 3.1). Second, at each level, uniform pose sampling and 3D wireframe projection are employed to build a cost volume for various pose hypotheses, describing the pose probability distribution. The pose with the maximum probability is then selected (Sec. 3.2). Finally, a post-processing network refines the wireframe probability map after the last level, and a Gauss-Newton method is applied to refine the pose chosen in the previous stage (Sec. 3.3). Figure 3 provides an overview of the proposed method.

3.1 Multi-Scale Feature Extractor

We use a standard convolutional architecture with U-Net [51] to extract multi-level features from the query image I . Different from previous works that maintain a high-dimensional feature map to encapsulate rich visual information for each level, we abstract and reduce the feature map dimension to a single channel, where each pixel in this map signifies the likelihood of being a wireframe. The resulting feature maps are denoted by F l ∈ R H l × W l × 1 , where l = { 1 , 2 , 3 } is the level index. More details on the architecture of the proposed network can be found in Appendix D.1.

3.2 Pose Selection from Cost Volume

After feature extraction, we construct a cost volume based on various pose hypotheses sampled around the coarse sensor pose, selecting the pose with the highest probability at each level. To ensure

Differentiable Gauss-

Newton method

Pose sampling &

projection

Pose Cost Volume

Localization

1. Multi-Scale Feature Extractor

Coarse-to-fine

Projection

Pose prior

Query image

Feature map

x

y

x

y

x

y

θ

z

Pose & Sampling Range Estimation

C

#3

efficient sampling, we utilize the uncertainty in pose selection at the current level to determine the pose sampling range for the next level.

Pose cost volume reconstruction. This subsection explains how to construct the pose cost volume. For a specific level l , the initial pose is represented by ξ l , and the computed pose is denoted as ̂ ξ l . As part of a progressive process, we keep ξ 1 = ξ p and ξ l +1 = ̂ ξ l . The pose ξ l can be decoupled into six degrees, with ξ l = (x l , y l , z l , θ l , φ l , ψ l ) , where (x l , y l , z l ) represents the translation in 3D space while ( θ l , φ l , ψ l ) refers to the Eular angles (i.e., yaw, pitch, roll). Since the pitch and roll ( φ p , ψ p ) of the gravity direction from the sensor pose data exhibit high accuracy, we fix ( φ l , ψ l ) = ( φ p , ψ p ) and only conduct operations on the remaining 4-DoF (i.e., (x l , y l , z l , θ l ) ) in the following steps.

Specifically, we begin by uniformly sampling 4-DoF poses centered on the initial pose ξ l , with the sampling range and number defined as r l = [r l ( x ) , r l ( y ) , r l ( z ) , r l ( θ )] and [ m l (x) , m l (y) , m l (z) , m l ( θ )] , respectively. The pose hypothesis { ξ hyp l ( d ) } is generated along (x , y , z , θ ) directions separately, where d ∈ (x , y , z , θ ) .

Next, for a given pose hypothesis, denoted as ξ hyp l = ( R hyp l , t hyp l ) and a set of discrete 3D wireframe points denoted as { P i } , we define a line alignment cost:

In this equation, p i = Π( R hyp l · P i + t hyp l ) represents the projection of 3D point P i under pose hypothesis ξ hyp l and [ · ] denotes a lookup with sub-pixel interpolation. The construction of the 3D wireframe points { P i } is provided in the next paragraph. By combining these costs in a grid manner across four distinct dimensions (x , y , z , θ ) , we obtain a pose cost volume C l with dimensions [ m l (x) × m l (y) × m l (z) × m l ( θ )] . Finally, a softmax function is applied to C l to yield a probability distribution volume P l . For pose inference, we select the pose ̂ ξ l with maximum probability by argmax operation upon P l .

Discrete 3D wireframe points generation. For a query image I and its associated sensor pose ξ p , we describe how to sample and identify discrete 3D wireframe points { P i } across the entire LoD map M . Assume the LoD map M is characterized by a number of faces with vertices V j = [ X j , Y j , Z j ] T ∈ R 3 . We derive each line of the LoD model as ℓ jk = [( X j , X k ) , ( Y j , Y k ) , ( Z j , Z k )] by connecting vertices V j and V k . To focus on distinct geometric structures such as building edges, we discard lines whose normals of their neighboring faces exhibit a significant difference, larger than µ = 10 degrees. The line simplification process is facilitated with the assistance of Blender [1].

Level 1

Level 3

Level 2

1.0

0.5

0.0

1.0

0.5

0.0

1.0

0.5

0.0

Predict

GT

Predict

GT

Predict

GT

Level 1

Level 2

Level 3

1.0

0.5

0.0

1.0

0.5

0.0

1.0

0.5

0.0

Predict

GT

Predict

GT

Predict

GT

Subsequently, we set the sampling density δ in meters and uniformly sample points along all simplified lines { ℓ jk } to obtain 3D points as { P u i } , where i represents the index of the 3D points. To obtain visible 3D points for a query image I , we use the pinhole camera projection (with intrinsic matrix K p ) and pose prior ξ p = ( R p , t p ) to identify 3D points, taking factors such as frustum inside and occlusion information into account. In particular, we first project 3D points onto the 2D image plane:

where [ X i , Y i , Z i ] ∈ { P u i } , and d is the projected depth for point [ X i , Y i , Z i ] . We then render a depth map from the LoD map M from pose ξ p , denoted as D . A boolean mask is calculated as:

where H and W denote the size of the image I , D ( u i , v i ) means the interpolating value on depth map D at ( u i , v i ) . The final discrete visible 3D wireframe points { P i } can be obtained using Eq. 5:

Uncertainty sampling range estimation. During the coarse-to-fine process, we leverage the pose selection uncertainty from the previous level to determine the sampling range of the current level. This strategy allows us to progressively subdivide the pose sampling space, thereby enhancing the precision of pose selection. More specifically, for l = 1 , we define the pose sampling range by evaluating the error in the coarse sensor pose. The sampling range for (x , y , z , θ ) is defined as [r 1 ( x ) , r 1 ( y ) , r 1 ( z ) , r 1 ( θ )] = [r p ( x ) , r p ( y ) , r p ( z ) , r p ( θ )] . For l = { 2 , 3 } , we employ the variance of the probability distribution volume P l -1 at l -1 to determine the pose sampling range r l .

In particular, since the pose hypothesis { ξ hyp l } , pose cost volume C l and probability distribution volume P l share the same data structure, we flatten them and index them by t . The variance v l at level l is calculated as:

Here, the symbol ⊖ represents the subtraction operation separately applied to the (x , y , z , θ ) directions. The corresponding standard deviation is computed as σ l = √ v l . We compute the pose sampling range as r l = 2 λ · σ l , where λ is a hyperparameter that adjust the length of the sample range. A visualization of this uncertainty sampling range estimation process can be found in Figure 4.

3.3 Pose Refinement

Based on the selected pose ̂ ξ 3 from the previous stage, we use a refined wireframe probability map F rf , which is further extracted from the feature map F 3 via a post-processing convolutional network, we optimize the pose ξ ∗ = ( R ∗ , t ∗ ) so as to align the 3D wireframe with the 2D predicted wireframe. Specifically, we define the objective function:

where P i is the 3D wireframe point and Π is the projection operation. Minimizing this function aligns the projected 3D wireframe points with the 2D locations that have a higher predicted probability. The pose update formula for ξ ∗ , as derived from the Gauss-Newton method, is given by:

In this formula, ∆ ξ ∈ R 6 represents a six-dimensional transformation vector, where ∆ ξ r ∈ R 3 constitutes the rotational component and ∆ ξ t ∈ R 3 represents the translational component. We transform the rotational component ∆ ξ r into a 3 × 3 rotation matrix by the exponential map of the Lie algebra so (3) . Besides, J i represents the Jacobian matrix of the residual function f i with respect to the pose parameters. A comprehensive explanation and detailed implementation of the Jacobian computation are provided in Appendix D.2.

3.4 Supervision

We employ two separate loss functions to facilitate end-to-end training of the pose selection procedure (Sec. 3.2) and the pose refinement module (Sec. 3.3). For the pose selection module, we minimize the negative log-likelihood loss on the probability distribution volume P l at three levels, where l ∈ { 1 , 2 , 3 } .

For the pose refinement process, the training involves minimizing the reprojection errors between 3D wireframe points transformed by the estimated pose ξ ∗ and the ground truth pose ξ = ( R , t ) :

where ρ represents the Huber robust kernel.

4 Experiment

Extensive experiments are conducted on the UAVD4L-LoD and Swiss-EPFL datasets to demonstrate the effectiveness of our proposed model as described in Sec. 4.2. Additionally, ablation studies are conducted on the UAVD4L-LoD dataset in Sec. 4.3.

Datasets. The released datasets consist of two distinct parts, named UA VD4L-LoD and Swiss-EPFL, providing LoD3.0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans an area of 2.5 square kilometers, is generated through a semi-automatic process which produces a 3D LoD map from the mesh model of the UAVD4L [72] dataset. The Swiss-EPFL dataset, which covers an expansive area of 8.18 square kilometers, derives its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [9-11]. We illustrate the 3D LoD maps and query images of these two datasets in Figure 2. More details can be found in Appendix A, B and C.

Baseline. We compared our approach with two visual localization baselines: UAVD4L [72], predicated on textured mesh models, and CadLoc [44], predicated on LoD models, employing diverse feature extractors and matchers. Both baselines employ a keypoint-based strategy: 1) SIFT [41] descriptor with traditional Nearest Neighbor (NN) matching, 2) learning-based extractor SuperPoint (SPP) [25] with graph-based networks Superglue (SPG) [54], 3) detector-free matcher LoFTR [66] and 4) e-LoFTR [71], 5) dense feature matcher RoMA [26]. Additionally, considering the line structure of the LoD model, we apply three line-based algorithms for the CadLoc: 6) deep neural network SOLD 2 [45] for joint detection and description of line segments, 7) deep line segment detector DeepLSD [46] with line detector in SOLD 2 , 8) DeepLSD with wireframe-based representation and dual-softmax matching method GlueStick [47]. Further details about the implementation of the baseline experiments can be found in Appendix E.

Metrics. We follow the standard localization evaluation procedure [68] and set recall thresholds of (2 m, 2 ◦ ) , (3 m, 3 ◦ ) , and (5 m, 5 ◦ ) .

4.1 Implementation Details

During training, we set a random seed to limit 3D wireframe points { P i } to 2 , 000 points, and the pose sampling number m l (x) , m l (y) , m l (z) , m l ( θ ) for level l = 1 , 2 , 3 is uniformly set to [13 , 7 , 3] due to constraints related to CUDA memory. The image size is (512 , 480) for the UAVD4L dataset and (720 , 480) for the Swiss-EPFL dataset. The pose sampling range at level 1 is set as [10 , 10 , 30 , 7 . 5] which refers to [r p ( x ) , r p ( y ) , r p ( z ) , r p ( θ )] . For the UAVD4L-LoD dataset, we incorporate a subset of synthesized images from UAVD4L [72], which includes buildings, as training data. For Swiss-EPFL, we train the model by combining synthetic images LHS and real query images from the CrossLoc [74] project, following its data split pattern.

During inference, experiments are executed on real query images { I i } derived from two datasets. We make the following changes, the discrete retrieval points from the 3D wireframe are sampled at an interval of 1 meter. The pose sampling number is increased to [ m l (x) , m l (y) , m l (z) , m l ( θ )] = [10 , 10 , 30 , 8] for all levels. λ is set as 0 . 8 . The training and inference of the entire network are executed using 2 NVIDIA RTX 4090 GPUs. Additionally, we employ four variations to validate the effectiveness of our method. Specifically, 1) no NWE means no neural wireframe estimation, which extracts explicit line segments using DeepLSD [46] and constructs a distance field for each segment. It then replaces the cost function in Equations 2 and 7 with the distance function values, and solves for the pose using coarse-to-fine pose selection followed by Gauss-Newton refinement; 2) -no USR means a model without uncertainty sampling range estimation (Sec. 3.2); 3) -no Refine denotes a model without pose refinement (Sec. 3.3); and 4) full model is our proposed LoD-Loc.

4.2 Evaluation Results

Evaluation over UAVD4L-LoD dataset. As described in Table 2, our method shows excellent performance, both in the in-Traj. and out-of-Traj. queries. Apart from the 2 m -2 ◦ and 3 m -3 ◦ metric in the in-Traj queries, which are marginally lower than UA VD4L with RoMA matcher, all other metrics surpass those of contemporary baselines. Note that this comparison is unfair, as baselines reference on a high-precision texture model that is richer in texture and geometry, while we only employ a LoD model. We further compare with CadLoc, which shares the same 3D reference model as ours. However, we observe that regardless of the choice of descriptors (point-based or line-based), these methods perform poorly. We visualize their retrieval and matching failure cases in Appendix E.3. Furthermore, we analyze why our method performs better in the out-of-Traj. scenarios compared to the in-Traj. scenarios in Appendix F.3.

Evaluation over Swiss-EPFL dataset. Table 3 presents the inference results on the Swiss-EPFL dataset. CadLoc continues to exhibit widespread failures due to its poor retrieval and matching results across different modalities. We surpass the state-of-the-art UA VD4L method in the in -Place queries, but fall behind in the out -of -Place queries. Moreover, the overall results obtained on the Swiss-EPFL dataset are not as strong as those on the UAVD4L-LoD dataset. We attribute this discrepancy to the inferior quality of the images in the training database as explained in Appendix F.1. Besides, the LoD2.0 model provides less structured information, making it harder for pose inferences.

Analysis of Methodological Advantages. First, compared to the SOTA texture-based approach, the LoD-Loc employs distinct cues for localization. The texture-based method determines the pose by optimizing the re-projection error of corresponding 2D-3D points. Conversely, the LoD-Loc aligns the 3D wireframe projection to solve the pose. Second, 3D-model-based methods typically employ a two-stage scheme, which involves building 2D-3D matches and then solving the pose with PnP RANSAC. The LoD-Loc method directly solves the pose in an end-to-end manner, potentially leading to better pose accuracy. Third, the LoD-Loc includes several important modules to improve performance, such as coarse-to-fine pose cost volume reconstruction, uncertainty-based sampling range estimation, and differential Gauss-Newton refinement. These factors contribute to the superior performance of our method.

LoD 3.0

Query

Level 1

Level 2

Level 3

LoD 2.0

4.3 Ablation Studies

We perform ablation experiments on the UAVD4L-LoD dataset, focusing on different levels. More ablation studies are provided in Appendix F.2.

Levels. As depicted in Table 4, we present the results of the ablation experiments in terms of recall, translation errors, and rotation errors. The localization accuracy shows a gradual improvement as the number of levels increases. This demonstrates the effectiveness of the progressive coarse-to-fine estimation and final pose refinement. Figure 5 visualizes feature maps for each level, illustrating that wireframe features extracted from deeper levels is clearer.

5 Conclusion

This paper presents LoD-Loc, a novel approach for localizing aerial images using a LoD 3D map. Compared to large and expensive 3D maps that existing methods rely on, the LoD map provides a simple, accessible, and privacy-friendly scene representation. With the coarse sensor pose, the proposed LoD-Loc uses a unified pipeline to estimate the camera pose, including a multi-scale feature extractor, pose selection from cost volume, and pose refinement. Furthermore, we contribute two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries with ground-truth pose annotation. LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. We believe LoD-Loc opens new possibilities for visual localization with simple and scalable 3D maps.

Limitation. LoD-Loc operates under the assumptions of a known gravity direction and a location prior. While these assumptions are reasonable, they restrict the application of LoD-Loc in environments where GPS is denied or unavailable.

Broader impact. This work has implications regarding privacy and surveillance. However, the LoD models represent building structures in a highly abstracted form, which alleviates concerns about the disclosure of personal privacy or land resource information.

Acknowledgments and Disclosure of Funding

The authors would like to acknowledge the support from the National Natural Science Foundation of China (Grant No. 62406331).

References

Blender. URL https://www.blender.org/ .

Dji zenmuse h20 series, . URL https://enterprise.dji.com/cn/ zenmuse-h20-series?site=enterprise&from=nav .

Dji m300 rtk, . URL https://enterprise.dji.com/cn/matrice-300 .

Dp modeler. URL https://www.whulabs.com/DPModeler/index.aspx .

Mavic 3 pro. URL https://www.dji.com/cn/mavic-3-pro .

Dji phantom 4 rtk. URL https://enterprise.dji.com/cn/phantom-4-rtk .

Baidumap. URL https://map.baidu.com .

Googlemap. URL https://www.google.com/maps .

Swissimage 10 cm, . URL https://www.swisstopo.admin.ch/en/geodata/images/ ortho/swissimage10.html .

Swisssurface3d, . URL https://www.swisstopo.admin.ch/en/geodata/height/ surface3d.html .

Swisssurface3d raster, . URL https://www.swisstopo.admin.ch/en/geodata/height/ surface3d-raster.html .

S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building rome in a day. Commun. ACM , 2011.

R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In CVPR , 2016.

D. Barath, J. Matas, and J. Noskova. MAGSAC: marginalizing sample consensus. In CVPR , 2019.

M. B. Bejiga, A. Zeggada, A. Nouffidj, and F. Melgani. A convolutional neural network approach for assisting avalanche search and rescue operations with uav imagery. Remote Sens. , 2017.

F. Biljecki, H. Ledoux, and J. Stoter. An improved lod specification for 3d building models. Comput. Environ. Urban Syst. , 2016.

F. Biljecki, H. Ledoux, and J. Stoter. Generating 3d city models without elevation data. Comput. Environ. Urban Syst. , 2017.

E. Brachmann and C. Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. T-PAMI , 2021.

Y. Cai, Y. Zhou, H. Zhang, Y. Xia, P. Qiao, and J. Zhao. Review of target geo-location algorithms for aerial remote sensing cameras without control points. Appl. Sci. , 2022.

F. Camposeco, A. Cohen, M. Pollefeys, and T. Sattler. Hybrid scene compression for visual localization. In CVPR , 2019.

S. Cao and N. Snavely. Minimal scene descriptions from structure from motion models. In CVPR , 2014.

S. Chen, X. Wu, M. W. Mueller, and K. Sreenath. Real-time geo-localization using satellite imagery and topography for unmanned aerial vehicles. In IROS , 2021.

O. Chum and J. Matas. Optimal randomized RANSAC. T-PAMI , 2008.

O. Chum, J. Matas, and J. Kittler. Locally optimized RANSAC. In DAGM , 2003.

D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPRW , 2018.

J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsberg. RoMa: Robust Dense Feature Matching. 2024.

M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM , 1981.

J.-M. Frahm, P. Fite-Georgel, D. Gallup, T. Johnson, R. Raguram, C. Wu, Y.-H. Jen, E. Dunn, B. Clipp, S. Lazebnik, et al. Building rome on a cloudless day. In ECCV , 2010.

Y. Ge, H. Wang, F. Zhu, R. Zhao, and H. Li. Self-supervising fine-grained region similarities for large-scale image localization. In ECCV , 2020.

H. Germain, G. Bourmaud, and V. Lepetit. S2DNet: Learning accurate correspondences for sparse-to-dense feature matching. In ECCV , 2020.

S. Gui and R. Qin. Automated lod-2 model reconstruction from very-high-resolution satellitederived digital surface model and orthophoto. ISPRS , 2021.

B. M. Haralick, C.-N. Lee, K. Ottenberg, and M. Nölle. Review and analysis of solutions of the three point perspective pose estimation problem. IJCV , 1994.

X. He, J. Sun, Y. Wang, S. Peng, Q. Huang, H. Bao, and X. Zhou. Detector-free structure from motion. arXiv preprint arXiv:2306.15669 , 2023.

D. Jovanovi´ c, S. Milovanov, I. Ruskovski, M. Govedarica, D. Sladi´ c, A. Radulovi´ c, and V . Paji´ c. Building virtual 3d city model for smart cities applications: A case study on campus area of the university of novi sad. ISPRS , 2020.

A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In ICCV , 2015.

J. Kim, J. Jeong, and Y. M. Kim. Fully geometric panoramic localization. In CVPR , 2024.

L. Kneip, D. Scaramuzza, and R. Siegwart. A novel parametrization of the perspective-threepoint problem for a direct computation of absolute camera position and orientation. In CVPR , 2011.

T. H. Kolbe. Representing and exchanging 3d city models with citygml. In 3D Geo-Inf. Sci. 2009.

J. Li and G. H. Lee. Deepi2p: Image-to-point cloud registration via deep classification. In CVPR , 2021.

X. Li, S. Wang, Y. Zhao, J. Verbeek, and J. Kannala. Hierarchical scene coordinate classification and regression for visual localization. In CVPR , 2020.

D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV , 2004.

S. Lynen, B. Zeisl, D. Aiger, M. Bosse, J. Hesch, M. Pollefeys, R. Siegwart, and T. Sattler. Large-scale, real-time visual-inertial localization revisited. IJRR , 2020.

V. Panek, Z. Kukelova, and T. Sattler. Meshloc: Mesh-based visual localization. In ECCV , 2022.

V. Panek, Z. Kukelova, and T. Sattler. Visual localization using imperfect 3d models from the internet. In CVPR , 2023.

R. Pautrat, J.-T. Lin, V. Larsson, M. R. Oswald, and M. Pollefeys. Sold2: Self-supervised occlusion-aware line description and detection. In CVPR , 2021.

R. Pautrat, D. Barath, V. Larsson, M. R. Oswald, and M. Pollefeys. Deeplsd: Line segment detection and refinement with deep image gradients. In CVPR , 2023.

R. Pautrat, I. Suárez, Y . Yu, M. Pollefeys, and V . Larsson. Gluestick: Robust image matching by sticking points and lines together. In ICCV , 2023.

R. Peters, B. Dukai, S. Vitalis, J. van Liempt, and J. Stoter. Automated 3d reconstruction of lod2 and lod1 models for all 10 million buildings of the netherlands. PE&RS , 2022.

N. Piasco, D. Sidibé, V. Gouet-Brunet, and C. Demonceaux. Learning scene geometry for visual localization in challenging conditions. In ICRA , 2019.

M. Pietrantoni, M. Humenberger, T. Sattler, and G. Csurka. Segloc: Learning segmentationbased representations for privacy-preserving visual localization. In CVPR , pages 15380-15391, 2023.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

E. Saretta, P. Bonomo, and F. Frontini. A calculation method for the bipv potential of swiss façades at lod2. 5 in urban areas: A case from ticino region. Sol. Energy , 2020.

P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR , 2019.

P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR , 2020.

P.-E. Sarlin, D. DeTone, T.-Y. Yang, A. Avetisyan, J. Straub, T. Malisiewicz, S. R. Bulò, R. Newcombe, P. Kontschieder, and V. Balntas. Orienternet: Visual localization in 2d public maps with neural matching. In CVPR , 2023.

P.-E. Sarlin, E. Trulls, M. Pollefeys, J. Hosang, and S. Lynen. Snap: Self-supervised neural maps for visual positioning and semantic understanding. NeurIPS , 2024.

T. Sattler, B. Leibe, and L. Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization. T-PAMI , 2016.

J. L. Schönberger and J. Frahm. Structure-from-motion revisited. In CVPR , 2016.

Y. Shi and H. Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In CVPR , 2022.

Y. Shi, X. Yu, D. Campbell, and H. Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In CVPR , 2020.

M. Shibuya, S. Sumikura, and K. Sakurada. Privacy preserving visual slam. In ECCV , 2020.

J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In CVPR , 2013.

M. Silvagni, A. Tonoli, E. Zenerino, and M. Chiaberge. Multipurpose uav for search and rescue operations in mountain avalanche events. Geomatics Nat. Hazards Risk , 2017.

N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: exploring photo collections in 3d. In SIGGRAPH . 2006.

P. Speciale, J. L. Schonberger, S. B. Kang, S. N. Sinha, and M. Pollefeys. Privacy preserving image-based localization. In CVPR , 2019.

J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR , 2021.

H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii. Inloc: Indoor visual localization with dense matching and view synthesis. In CVPR , 2018.

C. Toft, W. Maddern, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, T. Pajdla, et al. Long-term visual localization revisited. T-PAMI , 2020.

D. K. Villa, A. S. Brandao, and M. Sarcinelli-Filho. A survey on load transportation using multirotor uavs. JINT , pages 267-296, 2020.

S. Wang, J. Kannala, and D. Barath. Dgc-gnn: Leveraging geometry and color cues for visual descriptor-free 2d-3d matching. In CVPR , pages 20881-20891, 2024.

Y. Wang, X. He, S. Peng, D. Tan, and X. Zhou. Efficient LoFTR: Semi-dense local feature matching with sparse-like speed. In CVPR , 2024.

R. Wu, X. Cheng, J. Zhu, X. Liu, M. Zhang, and S. Yan. Uavd4l: A large-scale dataset for uav 6-dof localization. arXiv preprint arXiv:2401.05971 , 2024.

Z. Xia, O. Booij, M. Manfredi, and J. F. Kooij. Visual cross-view metric localization with dense uncertainty estimates. In ECCV , 2022.

Q. Yan, J. Zheng, S. Reding, S. Li, and I. Doytchinov. Crossloc: Scalable aerial localization assisted by multimodal synthetic data. In CVPR , 2022.

S. Yan, M. Zhang, Y. Peng, Y. Liu, and H. Tan. Agenti2p: Optimizing image-to-point cloud registration via behaviour cloning and reinforcement learning. Remote Sensing , 2022.

S. Yan, X. Cheng, Y. Liu, J. Zhu, R. Wu, Y. Liu, and M. Zhang. Render-and-compare: Crossview 6-dof localization from noisy prior. In ICME , 2023.

S. Yan, Y. Liu, L. Wang, Z. Shen, Z. Peng, H. Liu, M. Zhang, G. Zhang, and X. Zhou. Long-term visual localization with mobile sensors. In CVPR , 2023.

L. Yang, R. Shrestha, W. Li, S. Liu, G. Zhang, Z. Cui, and P. Tan. Scenesqueezer: Learning to compress scene for camera relocalization. In CVPR , 2022.

Q. Zhou, S. Agostinho, A. Ošep, and L. Leal-Taixé. Is geometry enough for matching in visual localization? In ECCV , 2022.

A Details on Dataset Collection

The released datasets consist of two distinct parts, UA VD4L-LoD and Swiss-EPFL, providing LoD3.0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans an area of 2.5 square kilometers, is derived from a semi-automatic process that generates a 3D LoD map based on the mesh model of the UAVD4L [72] dataset. This dataset includes a diverse array of architectural structures, including skyscrapers, villas, apartment complexes, educational institutions, and rural dwellings. The query images for this dataset were captured using two UAVs equipped with real sensor data: a DJI M300 [3] drone with an H20T [2] camera and a DJI Mavic3 Pro [5] drone. The Swiss-EPFL dataset, which covers an expansive area of 8.18 square kilometers, sources its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [9-11]. This dataset features a variety of architectural styles, such as libraries, residential apartments, and stadiums. The query images for this dataset were acquired through the CrossLoc [74] projects, using a DJI Phantom 4 RTK [6] drone. Figure 2 presents the 3D LoD maps and query images from these two datasets.

A.1 3D LoD Map Collection

The 3D LoD map for the UAVD4L-LoD dataset is generated semi-automatically with the assistance of the DP Modeler tool[4]. The process begins with the automatic generation of building blocks, characterized by their footprints and heights. Manual refinement is then applied to the architectural details of each building, raising them to the LoD3.0 level. The LoD accuracy of the UAVD4L-LoD dataset is consistent with the mesh model derived from UAVD4L.

For the Swiss-EPFL dataset, LoD2.0 models are downloaded from the SwissTOPO website [9-11]. To synchronize the coordinate systems between the map data and the drone-captured data from the CrossLoc dataset (which covers the same area with ground truth pose annotation), we converted the Swiss LoD map data in LV95 and LN02 coordinate systems to the ECEF coordinate system.

A.2 Query Image Collection

The query images of the UAVD4L-LoD dataset are divided into two categories: in-Traj. and outof-Traj. , representing trajectory-based and free-flight scenarios, respectively. The in-Traj. images, totaling 1 , 604 , were captured using a DJI M300 drone equipped with an H20T camera, focusing primarily on residential buildings, villas, and educational institutions. In contrast, the out-Traj. images, totaling 2 , 192 , were captured using a DJI Mavic3 Pro drone, covering a variety of architectural structures such as skyscrapers and rural dwellings. Both the in-Traj. and out-of-Traj. datasets include real sensor priors. Table 5 outlines the specific differences between the in-Traj. and out-Traj. sequences.

The real query images in the Swiss-EPFL dataset come from the CrossLoc [74] dataset. However, because the real-time kinematics (RTK) data from the DJI Phantom4 were used directly as ground truth (GT) poses, some GT poses show significant mislabeling. To resolve this issue, we projected the wireframes of LoD maps onto query images to identify and remove incorrectly labeled poses. The final query dataset comprises 2 , 254 images.

A.3 Query GT Generation

For the UAVD4L-LoD dataset, we employ a semi-automatic annotation approach to generate pseudoGT poses { ξ i } for the query images { I q i } . The process is based on the SfM results and textured mesh model of the UAVD4L [72]. First, we perform SfM separately on the query images { I q i } and the reference images { I r i } from the UAVD4L to obtain SfM results C q and C r . Next, we manually

in-Traj.

out-of-Traj.

select points with distinctive visual features (e.g., building corners) as tie points to align C q with C r . To further enhance the accuracy of the pseudo-GT, we utilize the render-and-compare pipeline [76] to refine the final poses { ξ i } .

Additionally, we analyze the discrepancies between the pose prior and the GT pose, decoupling the poses into 3D translation in WGS84 space and Euler angles in terms of 'yaw-pitch-roll'. It is observed that the translation errors for x and y are within ± 10 , z errors are within ± 30 , yaw errors are within ± 7 . 5 , and pitch and roll errors are approximately 1 degree.

B Details on UAVD4L-LoD Dataset

B.1 Pseudo-GT Generation

In the UAVD4L-LoD dataset, we employed a semi-automatic annotation technique to generate pseudo-GT poses { ξ i } for the query images { I i } . Initially, we performed SfM separately on the query images { I i } and the reference images { I r j } from UAVD4L, yielding corresponding SfM results C q and C r . Subsequently, based on the capture region of the { I i } , we manually identified e distinctive tie points, such as the corner of the building, to align C q with C r , resulting in C f . We then refined the pose accuracy of C f using Bundle Adjustment. The accuracy of the GT poses was evaluated through the median reprojection error, which was 0 . 43 pixels for all connected points and 1 . 19 pixels for the tie points. Finally, we employed a render-and-compare [76] pipeline for the final refinement of the GT poses. In this manner, with the annotation of tens of e = 20 manual tie points, we were able to obtain pseudo-GT poses { ξ i } for a total of 3 , 796 query images { I i } . Figure 6 shows the flight trajectories of the in -Traj . and out -of -Traj .

B.2 Sensor Pose Accuracy

In the UAVD4L-LoD dataset, we conduct a comprehensive data analysis to validate the precision of the sensor pose. This is accomplished by employing absolute error bar charts, as illustrated in Figure 7. Additionally, we assess the accuracy by projecting wireframe points onto the image plane using both sensor and GT poses. Results of these projections are depicted in Figure 8.

C Details on Swiss-EPFL Dataset

C.1 Data Cleaning

In the Swiss-EPFL dataset, the GT poses { ξ i } for the query images { I i } are sourced from the CrossLoc project [74]. This project directly acquires RTK data from the DJI Phantom 4 for GT annotation. Considering that the RTK device may introduce some noise, we identified and excluded query images with incorrect labeling. This was accomplished by projecting the wireframe onto the

in-Traj.

out-of-Traj.

X (mean:2.125)

Y (mean:5.040)

Z(mean:3.855)

250

100

160

200-

140 -

Number of photos

80

120 -

150

60

100 -

80

100

40

60

50

20

40

20

0

-3

2

-1

0

10

8-

-6

4

2

3

4

Errorfm1

Errorfm1

Errorfm1

Yaw (mean:0.766)

Pitch (mean:1.060)

Roll (mean:0.492)

140

200

140

120

175

120

Number of photos

100

150

100

80-

08

125

100 -

60-

60

75

40

40

50 -

20

20

25

0

-1

Error[deg]

0

-3

-2

Error[deg]

-1

0

1

Error[deg]

0

X (mean:1.423)

Y (mean:2.622)

Z (mean:13.03)

250 -

175

250

150

200

200

Number of photos

125

150

100 -

150

100

75

100

50

50

50

25 -

-2

。

2

0

10

20

Error[m]

Error[m]

Yaw (mean:1.000)

Pitch (mean:0.721)

Roll (mean:0.574)

800

400

000

700

350

Number of photos

600

300

250

500

250

200-

400

200

150

0OE

150

100 -

200

100

100

50 -

50

7.55.0

2.5

0.0

2.5

5.0

0

0.5

0.0

0.5

1.0

1.5

2.0

0

0.5

0.0

0.5

1.0

Error[deg]

Error[deg]

Error[deg]

image plane and manually discarding the items exhibiting noticeable misalignment. The process is visualized in Figure 9.

C.2 Sensor Poses Generation

Since the CrossLoc [74] project does not provide GPS or other sensor data, we randomly generate sensor poses ξ p by emulating the pose errors derived from the UAVD4L-LoD dataset. Specifically, X -Y for position range between [ -10 , 10] meters, Z ranges between [ -30 , 30] meters, yaw for rotation ranges in [ -7 . 5 , 7 . 5] degrees, and pitch -roll range between [ -1 , 1] degrees. We present the discrepancy between the generated sensor poses and GT poses in a bar chart, as depicted in Figure 11.

in-Traj.

out-of-Traj.

Priors

GT

Priors

GT

D Details on Method

D.1 Architecture of Multi-scale Feature Extractor

In this section, we provide a detailed description of the architecture of the multi-scale feature extractor in Table 6.

D.2 Jacobian Computation

The objective function for pose refinement is:

We compute the Jacobian matrix of the residual function f i with respect to the pose parameters as followed:

Mislabeled

Selected

where ∂ F rf ∂p i is the gradient of the feature map F rf at the 2D location p i and

Besides, P cam i is the point which transformed to the camera space. To compute the last derivative of Eq. 12, we add a perturbation ∆ ξ to the transformation:

Finally, the derivatives w.r.t the translation component and rotation component are:

where [] × is the skew-symmetric matrix.

E Details on Baseline

E.1 Sensor-guided Image Retrieval

For baselines, a retrieval-and-matching process is used upon the reference images in the dataset. To ensure a fair comparison, we apply the sensor poses to guide the image retrieval process for UAVD4L [72] and Cad-Loc [44]. For each query image I , we narrow the retrieval candidates q I using Eq. 16.

where ∥·∥ denotes the Frobenius Norm between two translation matrices, arccos( · ) calculates the rotation angles between two matrices, γ t and γ o are the threshold for translation and orientation, respectively. To determine the proper values for γ t and γ o for the baseline methods, a series of

RGB

Depth

Mesh-based

LoD-based

RGB

Depth

experiments are conducted on the UAVD4L dataset. In these experiments, we hypothesize that if no reference image could be located within the defined search area, the sensor pose would be utilized as the localization result. Table 7 shows that stricter thresholds result in worse outcomes. Consequently, we set γ t = 150 and γ o = 30 to ensure a sufficient search space. Furthermore, we measure the impact of retrieval number k in Table 8. The results suggest that while a larger k value enhances the performance of the benchmark algorithm, it also leads to an increase in inference time. Following previous work [72], the retrieval number is set at k = 3 . It is worth noting that regardless of the choice of k , our method exhibits a substantial acceleration in speed, outperforming by several-fold, or even an order of magnitude.

E.2 Reference Image Details

We provide a detailed description of the reference images used in the two datasets. These images serve dual purposes: they function as the database images for retrieval and matching in baselines, and they are also utilized as training data for the proposed LoD-Loc method. Specifically, for the UAVD4L-LoD dataset, we use a subset dataset of synthesis images in UAVD4L [72], excluding data that does not contain buildings. For the Swiss-EPFL dataset, synthetic images rendered in Latin Hypercube Sampling (LHS) [74] pattern have been employed as reference images. Notably, the CrossLoc dataset [74] did not include images in proximity to the out -of -Place area. To address this, we adopted the same synthetic scheme from [74] to generate synthetic reference images for this region. Figure 10 shows reference samples of RGB images and Depth images for both the mesh-based model and LoD-based model.

E.3 Failure Cases in Baselines

Although baselines have achieved impressive performance, they suffer from retrieving and matching repetitive texture images and cross-modal images. For example, Figure 13 exhibits deficiencies in retrieving repetitive texture images, and Figure 14 depicts poor matching results between RGB and LoD-rendered images.

15

in-Place

out-of-Place

X (mean:5.047)

Y(mean:5.008)

Z (mean:15.17)

30

35

25

30

30

25

Number of photos

25

20

20

15

15

10

10

10

5

-10

-5

0

10

-10

-5

0

10

-20

0

20

Error[m]

Error[m]

Error[m]

Yaw (mean:3.651)

Pitch (mean:0.493)

Roll (mean:0.492)

35

30

州

30

30

25

25

Number of photos

25

20

20

20

15

15

15 -

10

10-

10

5

5

0

0

-5

0

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

Error[deg]

Error[deg]

Error[deg]

X (mean:5.023)

Y(mean:5.169)

Z (mean:15.54)

20.0

20.0

20.0

17.5 -

17.5

17.5

Number of photos

15.0

15.0

15.0

12.5

12.5

12.5

10.0

10.0

10.0

7.5

7.5

7.5

5.0

5.0

5.0

2.5

2.5

2.5

0.0

0.0-

0.0

10

-5

0

5

10

10

5

0

5

10

20

0

20

Error[m]

Error[m]

Error[m]

Yaw(mean:3.836)

Pitch (mean:0.475)

Roll (mean:0.482)

25

20.0

17.5

20

17.5

15.0

Number of photos

15.0

12.5

12.5 -

10.0

10.0

10

7.5

7.5 -

5.0 -

5.0 -

5

2.5

2.5

0

0.0

0.0

Error[deg]

0

1.0

0.5

Error[deg]

0.0

0.5

1.0

1.0

0.5

Error[deg]

0.0

0.5

1.0

F Details of Experiments

F.1 Visualization of Training Data

We visualize some synthetic training samples of LoD-Loc, as shown in Figure 15. For the SwissEPFL dataset, the reference 3D model is derived from LiDAR point clouds, Terrain Models, and Orthophotos. In contrast, in the UAVD4L-LoD dataset, the reference 3D model is generated from

s

high-resolution aerial imagery through oblique photography reconstruction. As a result, the synthetic images from the former are of a lower quality. This could partly elucidate why our method yields lower results on the Swiss-EPFL dataset compared to UAVD4L-LoD.

F.2 Additional Ablation Studies

We provide more ablation studies in this section, which include the pose sampling number, the sample density δ of 3D wireframes, the sampling range controller lambda λ . Additionally, we explore the convergence and generalization of our method.

Pose sampling number. As illustrated in Table 9, we report the experimental results with varying numbers of pose samples. The findings suggest that a reduction in the number of sampled poses brings about a decrease in accuracy.

3D wireframe points sampling density. We conduct ablation studies for varying sampling densities, which affects the interpolation process on the feature map. As depicted in Table 10, there is no significant fluctuation in localization accuracy with changes in sampling density.

Sampling range controller. The parameter lambda λ adjusts the length of the sampling range. Through ablation studies, we demonstrate that the sensitivity of this parameter during the testing phase is low. The results are shown in Table 11.

Convergence and initial poses. Table 12 reports the localization recall with different initial prior errors on the UAVD4L-LoD dataset. It can be observed that the success rate of localization decreases as the initial prior error increases. Such issues occur when the GPS signal in the air is heavily interfered with. In such cases, we believe using sequence information could be a possible solution.

Cross-scene generalization. Table 13 illustrates the generalization capability of LoD-Loc through training and testing in diverse regions. Figure 16 delineates regional data using distinctive symbols and colors. On the UAVD4L-LoD dataset (A1 and A2), cross-scene testing yields results slightly lower than those obtained from training on the entire scene. For the Swiss-EPFL dataset (B1 and B2), we employ a model trained on the synthetic UAVD4L-LoD dataset, which achieves similar or even better performance compared to a model trained specifically on the Swiss-EPFL dataset. Additionally, the supplementary materials include two demo videos showcasing the model's capacity to localize cross-modal thermal images.

Computational cost comparison. We conducted test experiments on a single batch (Batch Size = 1) of images using the NVIDIA GeForce RTX 4090 device, and recorded the average peak CUDA usage as well as the average inference time. The details are provided in Table 14

F.3 Visualization of Results

We present more visualization results, including examples of corner houses (Figure 12), feature maps (Figure 17) and prediction results (Figure 18) at different levels. We found that the preset zig-zag route in a selected region resulted in some images capturing only the corners of houses, as shown in Figure 18. This led to poorer performance under strict 2m-2° metrics. However, it is important

to note that in the in-Traj. scenario, our method achieves comparable or superior results for coarse metrics. For instance, we achieve 96.95 % on 5m-5° while the closest baseline achieves 92.14 % .

Top 3

(Failure)

UAVD4L

Matching Results

Top 3

(Failure)

UAVD4L

……

……

……

……

Matching Results

Top 3

(Failure)

……

……

……

……

……

……

……

……

……

……

……

……

NM

UAVD4L-LoD

Swiss-EPFL

UAVD4L-LoD

Swiss-EPFL

A1

A2

B1

B2

LoD 3.0

Query

Level 1

Level 2

Level 3

LoD 2.0

UAVD4L-LoD

Swiss-EPFL

Priors

Level 1

Level 2

Level 3

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: It can be found in Abstract and Introduction Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It can be found in the end of the Conclusion Section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: It can be found in Experiment Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and dataset will be made available upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: It can be found in Experiment Section 4

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not included due to the high computational cost of calculating them.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: It can be found in Section Experiments 4

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: It is indicated in the Conclusion Section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data and code used in the paper comply with licensing and usage terms and are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: Once the paper is accepted, the code and data will be open-sourced with detailed usage instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a hierarchical scheme for pose selection to progressively compute high-quality pose in a coarse-to-fine fashion.

STRENGTHS:
The scheme of the proposed method is easily understood and the paper shows good experimental results.

WEAKNESSES:
The innovation of the paper is not clearly described.

QUESTIONS:
This paper give a detailed description of some proposed steps, without providing the target of these steps.  As far as I am concerned, these steps are general steps for localization, what's your innovation?
The problem you want to solve and the shortcomings that current methods exist should be strengthened.
The quality of some figures can be improved.

LIMITATIONS:
As described above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a localization algorithm using LoD maps. Compared to conventional localization pipelines using SfM / SLAM maps, LoD maps are memory efficient and can offer privacy preservation. Nevertheless, due to the lack of texture in LoD maps, it is not straightforward to use pre-existing localization pipelines for LoD map localization. The paper thus presents a localization pipeline that can handle such ambiguities, by presenting a pose search / refinement pipeline based on estimating wireframe probabilities from images. Specifically, the paper suggests training CNNs to predict wireframe probabilities from images, and measures the geometric compatibility of the predicted probabilities against the LoD map. Experiments show that the proposed method can largely outperform existing CAD-based localization methods, while showing competitive performance against methods conventional methods using textured meshes and local feature descriptors.\n\nSTRENGTHS:\n1. The problem of localizing LoD maps seems to be a practical yet quite under-explored problem. I find the problem setup to be very interesting and can foster future works in the direction. While the performance is currently slightly below visual descriptor-based methods, I find the initial results to be promising.\n\n2. The idea of predicting wireframe probabilities is novel and well-motivated. It seems natural to consider such structures to be extracted from 2D images to make valid comparisons against the wireframe map.\n\n3. The writing is very clear, and the supplementary materials section is very thorough which I expect will clarify a large body of details that readers may be interested in.\n\nWEAKNESSES:\nI don't have large concerns for the paper, but below are a few suggestions:\n1. Several related works seem to be missing, including some recent papers on descriptor-free visual localization:\n\na) Wang et al., DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching, arXiv 2024\nb) Kim et al., Fully Geometric Panoramic Localization, CVPR 2024\nc) Pietrantoni et al., SegLoc: Learning Segmentation-based Representations for Privacy-Preserving Visual Localization, CVPR 2023.\n\n2. I wonder how the proposed method could be extended for larger-scale pose search scenarios, for example if GPS / gravity direction is unknown. I feel the search space might be too large for the current grid-based search to work. Do the authors have any rough suggestions for how the work could be extended in such directions?\n\n3. How would be proposed method compare against explicitly extracting line segments from 2D images? To be more specific, suppose one replaces the wireframe probability extraction part explained in Section 3.1 with distance transformed calculated for 2D line extractions using conventional line extractors such as LSD or DeepLSD[1]. The cost functions or features in Equations 2 and 4 could be replaced with  the distance function values, similar to line-based localization methods [2, 3]. I feel this would not fare well since there would be large number of noisy lines and lines from image textures, but still the comparisons would provide insights into why the feature extraction in Section 3.1 is necessary for good performance. \n\n[References]\n1. Pautrat et al., DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients, ICCV 2023\n2. Micusik et al., Descriptor free visual indoor localization with line segments, CVPR 2015\n3. Kim et al., LDL: Line Distance Functions for Panoramic Localization, ICCV 2023\n\nQUESTIONS:\nI listed some minor questions below:\n1. What are the units of delta_x, y, z in Table 11?\n2. Related to Table 11, I feel it will be nice to see more experiments on the impact of the initial pose. For example, what errors would we get if we initialize from place recognition algorithms for UAVs [1]?\n\n\n[References]\n1. Keetha et al., AnyLoc: Towards Universal Visual Place Recognition\n\nLIMITATIONS:\nYes the limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for visual localization of Unmanned Aerial Vehicles (UAVs) using a Level-of-Detail (LoD) 3D map and neural wireframe alignment.  The UAV sensor provides coarse pose estimation. Then, LoD-Loc hierarchically builds a cost volume for uniformly-sampled pose hypotheses to describe pose probability distribution.  A pose with maximum probability is selected as the correct one. Each cost within this volume measures the degree of line alignment  between projected and predicted wireframes. It also refines the previous result with a differentiable Gaussian-Newton method. The paper presents an approach to aerial visual localization with a strong emphasis on practical applicability and privacy considerations.\n\nSTRENGTHS:\n1. This work gives an approach to aerial visual localization that does not require complex 3D representations.\n\n2. The method uses LoD 3D maps, which are easier to acquire and maintain, have a smaller footprint.\n\n3. it contributes new datasets for the research community to use for training and evaluation.\n\nWEAKNESSES:\n1. The importance of the major contribution of \"Level-of-Detail\" is not clearly described. Acctually, Level-of-Detail has been widely investigated. \n\n2. The paper does not explicitly mention the computational efficiency or resource requirements, which could be a concern for real-time applications on UAVs with limited processing power.\n\n3. The reliance on a coarse pose provided by the UAV sensor as an initial input may affect the method's robustness in scenarios where the initial pose is significantly off.\n\nQUESTIONS:\nIs it possible that the method's performance might degrade in environments with poor GPS signal or where the UAV is operating in areas without prior map knowledge?\n\nLIMITATIONS:\n1. The method assumes the availability of a gravity direction and location prior, which might not always be accurate or available, especially in GPS-denied environments.\n\n2. The generalizability of the method to other types of UAVs or different environmental conditions is not fully explored in the paper.\n\n3. The paper does not discuss the potential impact of dynamic elements in the environment (such as moving vehicles or people) on pose estimation accuracy.\n\n4. The robustness of the method against various weather conditions or times of day is not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach to provide an accurate camera pose of an aerial image according to a LoD 3D map, which is a lightweight representation to store the key information of a known map. For each query image, multi-scale wireframe probability maps are extracted by a trained U-net. These maps are then used to select sampled poses from cost volume. Such process is executed in a coarse-to-fine manner. Lastly, Gauss-Newton is used to refine the final pose by minimizing the projection error of 3D wireframe points.\n\nTwo LoD city datasets are also released with the paper.\n\nSTRENGTHS:\nThe proposed method is able to estimate an accurate pose from a LoD model and the experiments show the method can achieve generally comparable results compared to the baseline methods, among which one uses a high-cost texture mesh map for localization.\n\nThe design of coarse-to-fine pose selection is novel in this task and is validated to be useful in the experiments section.\n\nThe release of the LoD dataset could be beneficial to the aerial localization research field.\n\nWEAKNESSES:\nIn this paper, the definition of \"in-Traj\" and \"out-of-Traj\" is not clearly provided. I guess this may mean the data used to train the model and those in the test stage. Please clarify these concepts in the paper.\n\nThe proposed method is significantly worse than UAVD4L in \"out-of-place\" part of Swiss-EPFL dataset. The authors explain that this is due to the relatively low quality of the dataset and LoD2.0 model. However, this shows that the method is much less robust to the data quality and it only works when high-quality data is provided, which can be very costly.\n\nQUESTIONS:\nDo you have any process to deal with the sky pixels in query images?\n\nLIMITATIONS:\nThe authors have mentioned the limitations of the method and the potential privacy impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method named LoD-Loc for visual localization of unmanned aerial vehicles (UAVs) using Level-of-Detail (LoD) 3D maps. Unlike existing algorithms that rely on complex 3D representations, LoD-Loc aligns wireframes derived from LoD model projections with the wireframes predicted by a CNN Unet to estimate the UAV's pose. To solve the problem, given a coarse pose provided by the sensor, this paper utilizes a hierarchical cost volume to pose hypothesis evolution. And eventually, a differentiable Gaussian-Newton algorithm for 6-DoF pose refinement.  This work also contributes two datasets to facilitate further research in this area and provides a comprehensive evaluation of the proposed method's performance. The experimental results demonstrate that LoD-Loc outperforms current state-of-the-art methods using textured 3D models, offering advantages in ease of acquisition, lightweight map size, and privacy preservation.\n\nSTRENGTHS:\nThe strengths of the paper are the following:\n\n**Interesting Methodology:**\n\nThe proposed LoD-Loc method introduces an interesting approach to visual localization using Level-of-Detail (LoD) 3D maps, which is less reliant on complex 3D representations. This makes the method more accessible and scalable compared to current textured 3D model-based approaches.\n\n**Application in practical scenarios:**\n\nLoD maps are easier to acquire and maintain and have a smaller file size. These advantages make LoD-Loc more feasible for real-world applications, particularly in large-scale environments where data storage efficiency is a critical requirement.\n\n**Contribution of Datasets:**\n\nThe paper contributes two largest-scale air localization datasets (LoD3.0 and LoD2.0) with RGB queries and ground-truth pose annotations. These datasets provide interesting resources for further research and benchmarking in aerial visual localization.\n\nWEAKNESSES:\n**Experiment Result**\n\nThe author mentioned CadLoc \"cannot even localize the query image at a coarse threshold at $5m -5^{\\circ}$\". However, what I observed from Table 2 even Table 3 is the result for CadLoc are all zeros. Is that a mistake?\n\nIn Table 2, the authors show their method surpasses the SOTA textured 3D-model-based method in the $2m -5^{\\circ}$ and $3m -5^{\\circ}$ and \"out-of-Traj.\"\nHowever, the author does not explain why their method can perform better than textured 3D-model-based methods in \"out-of-Traj\" scenarios but fails in \"out-of-Traj\" scenarios.\n\nIn the Appendix, the author mentioned that \"in-Traj.\" and \"out-of-Traj.\" represent trajectory-based and free-flight scenarios, respectively.\nBut I didn't catch what might be the key difference between these two scenarios. Does that mean these methods are actually verified in different trajectories in the \"out-of-Traj.\" scenarios? \nIf so, I think it is not a fair comparison. If not, I didn't get the key difference between the \"in-Traj.\" and \"out-of-Traj.\" scenarios. It is suggested to explain why the proposed method performs better than the SOTA textured 3D-model-based method considering they use high-precision texture 3D information.\n\nWhile the author mentions they conducted experiments on many other image retrieval-and-matching and textured 3D models-based approaches, only the results of UAVD4L and CadLoc are presented. It would be better to compare with more methods in each type to show a more comprehensive result.\n\n**Computational Cost**\n\nAlthough not mentioned, the methods involving hierarchical cost volume construction and differentiable optimization algorithms can be computationally intensive. This could pose challenges for real-time applications, especially on resource-constrained UAVs. For more comprehensive research, it would be better to provide a computational cost comparison for that.\n\n**Presentation**\n\nI think the writing and presentation skills in this paper could be further improved. Such as the presentation of Table 1 and the Contribution sections. Besides, I would suggest the author double-check their result in the presented Tables. Besides, I would suggest that the author add more explanations for their experiment result instead of just simply displaying them.\n\nQUESTIONS:\nThe proposed idea in general sounds okay to me. However, my questions are mainly about the result of the experiment result listed in the Weaknesses. I think the result present is questionable, insufficient, and lacks enough explanation to support the proposed method.\nBesides, I think the writing and presentation need to be further improved for this paper to be accepted.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}