LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment
Authors: Juelin Zhu
Shen Yan
Long Wang
zhang shengYue
Yu Liu
Maojun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D computer vision; Pose estimation; Privacy-preserving localization
Primary area: other
TL;DR: A new method for aerial visual localization using LoD 3D map with neural wireframe alignment.

Abstract: We propose a new method named LoD-Loc for visual localization in the air. Unlike existing localization algorithms, LoD-Loc does not rely on complex 3D representations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given a coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypotheses to describe pose probability distribution and select a pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises a 6-DoF pose optimization algorithm to refine the previous result with a differentiable Gaussian-Newton method. As no public dataset exists for the studied problem, we collect two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. The code and dataset will be made available upon publication.

LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment
Juelin Zhu zhujuelin@nudt. in@nudt edu cn
Shen Yan'* yanshen120nudt. edu. cn
Long Wang? wanglongzjjugmail.c com
Shengyue Zhang
Yu Liu' jasonyuliuEnudt. edu. cn
Maojun Zhang"
zhangehengyuu223nuutt edu. cn
mjzhang@nudt. edu, cn
National University of Defense Technology
2 Sense'Time Research
Abstract
We propose method named LoD-Loc for visual localization in the air. Unlike existing localization algorithmss, LoD-Loc does not rely on complex 3D represen- tations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypothe- ses to describe pose probability distribution and select pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises 6-DoF pose optimization algorithm refine the previous result with a differentiable Gaussian- Newton method. As no public dataset exists for the studied problem, collect two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-ar methods that use textured 3D models for localization. The code and datuset are available at httts:: github. -o/Loo-oooo waw.cggithb...... io/
Introduction
Acrial visual localization is the process of determining the global position and orientation for UAV camera relative to a known map. This process benefits many important applications, ranging from cargo transport [69] surveillance
19 72), to search and rescue [5
Following localization algorithms on the ground [18] 30 42
53
54
68
70
77). current
aerial visual localization approaches (22 72 typically involve matching pixels in query image with points in a pre-buill high-quality 3D map, which is often derived from 3D texture models 43
76. 72). Subsequently, Perspective-n-Point (PnP) RANSAC [32 37
14
23
27
24 technique
commonly used to calculate the camera pose. However, building high-quality 3D maps using is expensive global scale and requires frequent updates account for temporal changes in visual appearance. Besides, these 3D maps are costly to store, which poses significant challenges for terminal deployment on drones. Furthermore, high-resolution 3D maps disclose detailed information about the localization area, raising critical concerns regarding homeland security and privacy preservation.
photogrammetry 12 28
64
58
33
"Equal contribution *Correpooning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

6- G-DoF Pose Estimation over LoD Model
Easy Acquiresaiintiin
weight Map Size
Privacy Presevvtion
LoD-Loc
Sensor Prior
Localization Result
Figure 1: In this paper, we propose LoD-Loc to tackle visual localization w.r.t a scene represented by a LoD 3D map, characterized by its case of acquisition, lightweight nature, and built-in privacy- preserving capabilities. Given a query image and its coarse sensor pose, our method utilizes the wireframe alignment of LoD models to recover the camera pose.
To address these challenges, we propose to leverage the Level of Detail (LoD) 3D city maps 38 16
as
the cue for localization, as illustrated Figure [] Compared with traditional textured 3D models, LoD
3D models enjoy the following advantages: 1) Ease of acquisition and maintenance: World-scale
LoD city models can be generated with the rapid development remote sensing [17 34
48 52
Many commercial companies, such Google Maps [8] and Baidu Maps 77. have already integrated
LoD 3D models into their MAP Applications. 2) Light-weight map size: LoD maps are extremely compact, up to 10 times smaller in size than textured 3D maps, enabling on-device localization over large areas. 3) Privacy preservation and policy friiecy-freddyy As LoD city models only reveal the basic 3D outlines of buildings in a highly abstracted and simplified manner, they raise fewer concerns about the disclosure of privacy and land resource secrets.
However, compared with a textured 3D model, using LoD maps for localization is very challenging, primarily due the lack of texture and detail. This deficiency makes nearly impossible to establish local feature- based 2D-to-3D correspondences. Inspired by the idea that, when the pose is correctly solved, the network-predicter building wireframes can align with those projected from the LoD 3D) model, as shown in Figure 1 we introduce LoD-Loc, a novel approach tailored for visual localization in LoD 3D city maps. Our method takes query image and its real sensor data (i.e., GPS, gravity, and compass) input, and estimates the 6-DoF pose of a drone in flight. Specifically, we initially fix the 2- DoF gravity direction and generate pose hypotheses by sampling 4-DoF (comprising position and yaw angle) around the sensor pose, given that the gravity direction provided by the inertial unit exhibits minor error. Following the generation of pose hypotheses, LoD building wireframes are projected onto the query image plane. Each pose hypothesis is then scored based on the alignment between the projected predicted wireframes, thereby forming a 4D pose cost volume. By applying a softmax operation, we derive probability density over the pose, which can be used for pose selection through classification. Moreover, after the pose selection stage, a differentiable Gauss-Newton method, with an optimization objective to maximize the wireframe alignment, is employed to refine the overall 6-DoF pose. The pose selection and optimization processes are fully differentiable w.r.t. the network output, which enables the use of ground-truth poses as supervision for training feature extraction and pose estimation in an end-to-end manner.
To achieve high accuracy and low memory usage, we propose hierarchical scheme for pose selection that utilizes multiple small pose volumes, to progressively compute poses in a coarse-to manner. Throughout the hierarchy, we adopt an adaptive sampling strategy, where the variance-basce based uncertainty from the previous stage influences the sampling range of the next stage for constructing pose cost volume. This adaptive process enables reasonable and fine-grained spatial partitioning of poses, resulting in a significant improvement in the final pose output.
To facilitate research in this area, and to train and evaluate our method, we release two datasets with map levels of LoD3.0 and LoD2.0, respectively, as shown in Figure 2 For the LoD3.0 dataset, we employ a semi-automatic method to generate LoD model data from recent large-scale oblique photography scene [72). covering an area of 2.5 square kilometers. The query images are captured by drones, with sensor data (e.g., GPS, IMU) recorded. For the LOD2.0 dataset, use LoD model data provided by the Swiss federal authorities, specifically the SwissTOPO 9Ill data near École

---

Table 1: Differenet types of maps for visual localization.
SfM SLAM
Mesh model
Satellite images
LoD model (our works)
Map Type
OpenStreetMap
3D points
textured meshes
pixel intensity
polygons, lines, points
wireframes,
What?
+features
faces
Explicit geometry?
3D
3D
x
2D X 3-DoF
3D
Visual appearance?
v
x-DoF pose estimation
6-DoF
6-DoF
3-DoF
6-DoF
Storage per 1km² Size reduction v.s. SfM
42 GB
9.8 GB
75 MB
4.8 MB 8800
2.84 MB
4.28x
550x
15100
Polytechnique Fédérale de Lausanne (EPFL), covering an area of 8.2 square kilometers. The query images with ground-truth poses are sourced from the CrossLoc [74 project.
We conduct extensive experiments on these two datasets. The results show that, due to the lack of both color and texture in the LoD 3D model, previous sfate-of-the-art image retrieval-and-mann matthhhhinngg
methods [44] 53 54 29 41 25 66 (46] [47] basically fail. contrast, our method consistently
achieves excellent results, even surpassing current state-of-the-art methods [33 54 66 72
that use
textured 3D models for localization.
Contributions.
We propose the use of Level of Detail (LoD) 3D maps for 6-DoF visual localization in the air. We introduce a novel localization method that utilizes wireframe alignment for pose estimation. Our method differentiable, allowing the pipeline to be trained end-to-end with pose supervision.
We release two LoD city datasets, complete with RGB queries and ground-truth pose annotations.
2 Related Works
Localization from SfM or Mesh Map. SfM maps typically consist of reference images and 3D track points with their associated features [58] For a given query image, an image retrieval method [ 29 is initially utilized to identify co-visible reference images. Following this, feature matching algorithms [4] 225 54 66 are employed to establish accurate 2D-2D correspondences between the query image and the identified reference images, with track information being used to transform these 2D-2D correspondences into 2D-3D relationships. Finally, the pose is resolved using
PnP RANSAC 32 37
14
23
27 24 (77)
Mesh maps are typically defined by a textured mesh model. Initially, reference images with depth rendered appropriate viewpoints surrounding the model 772
43 44 (6).. Similar image [4 23 54 66) processes are employed to identify co-visible images and to establish 2D-2D correspondences. The depth map is utilized to transform 2D-2D correspondences into 2D-3D relationships, and the pose is subsequently determined by a PnP
retrieval 13 29 and matching
RANSAC 32 37
23 27 24
777.
Despite providing high-accuracy localization results, both StM and mesh models present significant challenges in terms of reconstruction and maintenance. Additionally, their extensive size complicates deployment, necessitating its existence solely in the cloud, Morcover, these maps raise serious concerns about the privacy of personal and land resource Icakage.
Localization frrmm other types of Map. To mitigate these issues, researchers have proposed to use alternative types of maps for localization. In addressing the difficulty of map reconstruction and maintenance, some methods opt for overhead imagery such as satellite [59] 60 73 [66,, or
leverage OpenStreetMap [55] as reference. However, these methods are limited to estimating 3-DoF (planar position and heading) pose at most. To address the issue of large map size, some 78 reduce model complexity [44], 75
methods have made attempts to compress the maps 20
21
or utilize geometry information without features [49
79
In the context of privacy, some
methods (36] 61 65 propose to transform 3D point clouds into 3D line clouds, leverage semantic
3

---

SWWd 1222 1223
in- Place
out-of- Place ofccae--lace
LOL LOVDAL-LO
in- Traj. Query samples
out- oft-oo Traj.
LoD models with details
Figure 2: Overview of datasets. The left side shows the LoD models of the released data. The Lol>2.0 model from Swiss-EPFL includes building height and roof information, while the LoD3.0 model from UAVD4L-LLD contains more detailed structural information such as building height, roof, and side pillars. The right side illustrates samples of query images, which consist of images captured by drones in various scenes.
information from point clouds, utilize semantic 3D maps to enhance privacy [50] Some other
methods apply learning-based pose regression or scene point regression models [35] 62 40 that do not explicitly store the 3D map. However, the effectiveness and generalizability of these methods are often inferior to those that rely on SfM texture mesh maps. detailed comparison of attributes from different maps is provided in Table [
3 Method
Given 3D city LoD map M, query image I, and its coarse sensor pose Ep. the goal of the proposed method is compute the absolute 6-DoF pose e" First, convolutional neural network is used to extract the wireframe probability map for the query image at multiple levels (Sec. 3.T Second, each level, uniform pose sampling and 3D wireframe projection employed build cost volume for various pose hypotheses, describing the pose probability distribution. The pose with the maximum probability is then selected (Sec. 3.2). Finally, post-processing network refines the wireframe probability map after the last level, and Gauss-Newton method is applied to refine the pose chosen in the previous stage (Sec. 3.3 Figure provides an overview of the proposed method.
3.1 Multi-Scale Feature Extractor
We use standard convolutional architecture with U-Net [5]] to extract multi-level features from the query image I. Different from previous works that maintain nigh-dimensional feature map to encapsulate rich visual information for each level, abstract and reduce the feature map dimension to single channel, where each pixel in this map signifies the likelihood of being a wireframe. The resulting feature maps denoted by F, TR " W7x1 where 21,2,3 is the level index. More details on the architecture of the proposed network can be found in Appendix D.I
3.2 Pose Selection from Cost Volume
After feature extraction, we construct cost volume based on various pose hypotheses sampled around the coarse sensor pose, selecting the pose with the highest probability at each level. To ensure

---

Posa prlor
1. Multi Scale Feature Extractor
2. Pose Selection from Cast. V Volume
Pose Refinement
C2
Differentable Gauss Newvoo method
Projection
Pose sampling projection
Pose Volume
Pose Sampling Range Estimation
Query image
Feature map
Coarse-to fine
4.
Localizaiin
Figure 3: Overview LoD-Loc. LoD-Loc employs a CNN extract multi-level features F, for the query image I (Sec. 3.I 2. A cost volume Cj is built for various pose hypotheses sampled around the coarse sensor pose E, to select the pose ε, with the highest probability, based on the projected wireframe of the 3D LoD model (Sec. 3.2). 3. A differentiable Gauss-Newton method is used to refine the final selected pose E,, to obtain more accurate pose { (Sec. 3.3
efficient sampling, we utilize the uncertainty in pose selection at the current level to determine the pose sampling range for the next level.
Pose cost volume reconstruction. This subsection explains how to construct the pose cost volume. For a specific level I, the initial pose is represented by E,, and the computed pose is denoted as &, As part of a progressive process, we keep 1 &, and 61+1 & The pose &, can be decoupled into six degrees, with & = (x, Yt, 24, 0t Pt, tht), where (x1, Yt, 21) represents the translation 3D space while (0:: (Pt, (h)) refers the Eular angles (i.e., yaw, pitch, roll). Since the pitch and roll (ppp: tip) of the gravity direction from the sensor pose data exhibit high accuracy, we fix (sPt, vh)) = (pp, ())) and only conduct operations on the remaining 4-DoF (i.e., (xi, Yı, 24, 01) the following steps.
Specifically, we begin by uniformly sampling 4-DoF poses centered on the initial pose E., with the sampling range and number defined as r (x), my(y), (z), mf)... respectively. The pose hypothesis {E. (d)) generated along (x, y, 2, 0) directions separately, where d E (x,,,2,0).
(z)) (()] and
{E) DE (d)} = (d)/2 d, ... d, ... d + (d)//22) mi(d)
Next, for a given pose hypothesis, denoted as g/ = (R, hyp HVP) ) and set of discrete 3D wireframe points denoted (P:). we define line alignment cost:
t
Ci(E!
Σ Fplp..
In this equation, p; 11(R hMp P. + represents the projection of 3D point P; under pose hypothesis g/ and [-] denotes a lookup with sub-pixel interpolation. The construction of the 3D wireframe points [P,) provided in the next paragraph. By combining these costs grid manner across four distinct dimensions (x, y, 0). we obtain a pose cost volume Cj with dimensions (m((x) myly) mz(z) x (m)]. Finally, softmax function is applied C₁ to yield a probability distribution volume Pt. For pose inference, we select the pose & with maximum probability by argmax operation upon Pt.
Discrete 3D wireframe points generation. For a query image I and its associated sensor pose Sn. we describe how to sample and identify discrete 3D wireframe points (Pi) across the entire LoD map M. Assume the LoD map M is characterized by a number of faces with vertices V = (X,, Y,, 2,) E R.3, We derive each line of the LoD model as Z5x = [(X,, Xx), (Y,, Ys), (Z,, 2x)] by connecting vertices V) and Vg. To focus on distinct geometric structures such as building edges, we discard lines whose normals of their neighboring faces exhibit a significant difference, larger than 10 degrees. The line simplification process is facilitated with the assistance of Blender [I]
5

---

10
Leve
Predict
10
Levd
Predict
0.5
GT
o.s
GT
1.0
Level
Predict
10
Levd 2
Predict
0.5-
10-
Level
Predict
Level 3
Predict
0.5
GT
Figure 4: Toy examples to illustrate the uncertainty sampling range estimation. We show pose distribution (connected blue dots), pose prediction (yellow dash line), the ground truth pose (red dash line), and uncertainty sampling range (gray) in the three levels.
Subsequently, we set the sampling density in meters and uniformly sample points along all simplified lines }lxx} to obtain 3D points {P" J. where i represents the index of the 3D points. To obtain visible 3D points for a query image I, we use the pinhole camera projection (with intrinsic matrix K") and pose prior S (R", t²) identify 3D points, taking factors such frustum inside and occlusion information into account. In particular, we first project 3D points onto the 2D image plane:
de du Us. 1] = K" (R Y,, 2:] + +t²)
(3)
where [X,, Y, Z,] E )..). and is the projected depth for point [X,, Y, Z.). We then render depth map from the LoD map M from pose Ep. denoted as D. A boolean mask is calculated as:
= d, D(u,, e;) & < u, < H & 0 e < W
where H and denote the size of the image I, D(u, 24) means the interpolating value on depth map D at (u,, vz). The final discrete visible 3D wireframe points {P,} can be obtained using Eq.
[P) = [P.]
Uncertainty sampling range estimation. During the coarse-to-fine process, we leverage the pose selection uncertainty from the previous level to determine the sampling range of the current level. This strategy allows us to progressively subdivide the pose sampling space, thereby enhancing the precision of pose selection. More specifically, for / = 1, we define the pose sampling range by evaluating the error in the coarse sensor pose. The sampling range for (x, 2, 0) defined as (x), (v), (2), [z), For / {2, 3). we employ the variance of to determine the pose sampling range ri.
[r (x), ri (w), ri (z), probability distribution volume P
In particular, since the pose hypothesis (E!UP), pose cost volume C₁ and probability distribution volume Pi share the same data structure, we flatten them and index them by t. The variance at level / calculated as:
Vi E'PL1 I e 1
(6)
Here, the symbol O represents the subtraction operation separately applied to the (x, y, z, 9) directions. The corresponding standard deviation is computed o; /vi. We compute the pose sampling range as ri 2 oi, where is a hyperparameter that adjust the length of the sample range. A
visualization of this uncertainty sampling range estimation process can be found in Figure
3.3 Pose Refinement
Based the selected pose 3 from the previous stage, we refined wireframe probability map Fry. which is further extracted from the feature map F3 via post- post-processing convolutional network, we optimize the pose e" = (R*, t*) so as to align the 3D wireframe with the 2D predicted wireframe. Specifically, we define the objective function:
E(E*) = Σ || =
Σ
F [I - t)]l,

---

where P; is the 3D wireframe point and II is the projection operation. Minimizing this function aligns the projected 3D wireframe points with the 2D locations that have higher predicted probability. The method, is given by:
pose update formula for {, as derived from the
A = I ()
(8)
t' R DE, +t R = (..)
In this formula, A E I represents six-dimensional transformation vector, where AG, R constitutes the rotational component and AÇ, E IR3 represents the translational component. We transform the rotational component ΔE, into a 3x3 rotation matrix by the exponential map of the Lie algebra (3). Besides, J; represents the Jacobian matrix of the residual function f; with respect to the pose parameters. A comprehensive explanation and detailed implementation of the Jacobian computation are provided in Appendix D.2
3.4 Supervision
We employ two separate loss functions to facilitate end-to-end training of the pose selection procedure (Sec. (.2) the negative aigliikelhhodd loss on the probability distribution volume Pi at three levels, where I E (1.
(Sec.B.2) and the pose refinement module (Sec. (.3). For the pose selection module, we minimize
-= Σ log PIEE)
For the pose refinement process, the training involves minimizing the reprojection errors between 3D wireframe points transformed by the estimated pose g and the ground truth pose (R, t):
L = EQIITRRR P. t2+ [R P. +))I),
(10)
where p represents the Huber robust kernel.
Experiment
Extensive experiments conducted on the UAVD4L-LoD and Swiss-EPFL datasets to demonstrate the effectiveness of our proposed model as described in Sec. 4.2. Additionally, ablation studies are conducted on the UAVD4L-LoD dataset in Sec. 4.
Datasets. The released datasets consist of two distinct parts, named UAVD4L-LoD and Swiss-EPFL, providing LoD3.0 0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans area of 2.5 square kilometers, is generated through a semi-automatic process which produces a 3D LoD map from the mesh model of the UAVD4L [72] dataset. The Swiss-EPFL dataset, which covers an expansive area of 8.18 square kilometers, derives its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [1] We illustrate the 3D LoD maps and query images of these two datasets in Figure 2] More details can be found Appendix A
B and
Baseline. We compared our approach with two visual localization baselines: UAVD4L 72 pred-
icated on textured mesh models, and CadLoe (44) predicated on LoD models, employing diverse feature extractors and matchers. Both baselines employ keypoint-based strategy: 1) SIFT 41 descriptor with traditional Nearest Neighbor (NN) matching, 2) learning- baased extractor SuperPoint (SPP) (25] with graph- based networks Superglue (SPG) (54). detector-free matcher LoFTR [66] and 4) e-LoFTR 77. 5) dense feature matcher RoMA 26 Additionally, considering the line structure of the LoD model, we apply three line-based algorithms for the CadLoc: 6) deep neural network SOLD² for joint detection and description of line segments, 7) deep line segment detec- tor DeepLSD 46 with line detector in SOLD?. 8) DeepLSD with ceeeframebbsse representation and dual- -oftl-softmxx matching method GlueStick [47] Further details about the implementation of the baseline experiments
be found in Appendix E
Metrics. We follow the standard localization evaluation procedure 68 and set recall thresholds of (2m, 2"), (3m, 3d), and (5m,5%
7

---

Table 2: Quantitative comparison results over the UAVD4L-LoD dataset.
in-Traj.
Out-Of- Outt-ofTTroy Traj.
Method
2m-2"
3m-3
5m-50
2m-2"
3m-3"
5m-5"
Sensor Priors
0
4.3
0.36
SIFT+NN
73.13
78.62
80.42
82.39
85.13
86.36
SPP+SPG
91.71
92.02
92.14
93.43
93.70
93.80
UAVD4L
LoFTR
84.98
88.09
88.90
91.56
92.02
92.11
Mesh model
e-LoFTR
84.47
88.21
88.96
91.06
91.93
92.02
RoMA
93.27
93.70
93.77
95.03
95.53
95.53
SIFT+NN
0
0
0
0
0
SPP+SPG
0
0
LoFTR
0
0
0
0
0
CadLoc
e-LoFTR
0.37
0.87
1.31
0.41
0.78
1.37
LoD model
RoMA
2.18
2.87
3.68
6.93
8.76
10.40
SOLD2
0
0
DecpLSD+SOLD2
0
0
0
0
0
DeupLSDDGlluSttik
0
0
NWE
10.41
16.21
24.19
6.93
12.64
21.62
Ours LoD model
no USR
70.39
85.47
95.32
82.62
94.71
97.63
Refene
51.31
76.06
86.78
74.27
97.95
99.36
Full model
84.41
91.77
96.95
95.94
99.00
99.36
4.1
Implementation Details
During training, we set random seed to limit 3D wireframe points [Pt} to 2, 000 points, and the pose sampling number (x),, my(y,, mz(z), m0) for level 1, 2, uniformly to constraints related to CUDA memory. The image size is (512, 480) for the UAVD4L dataset and (720,480) for the Swiss EPFL dataset. The pose sampling range at level 1 is as [10, 10, 30, 7.5] which refers to (2). pp (v). F. (2), 2, I, (0)] For the UAVD4L-LoD dataset. we incorporate subset of synthesized images from UAVD4L (2).. which includes buildings, as training data, For Swiss-EPFL.
[13, 7, due
we train the model by combining synthetic images LHS and real query images from the CrossLoc 74 project, following its data split pattern.
During inference, experiments are executed on real query images {I:} derived from two datasets.
We make the following changes, the discrete retrieval points from the 3D wireframe are sampled an interval of meter. The pose sampling number's is increased to [mı(x), my) mz(z), mx(0)) [10, 10, 30, 8] for all levels. À is set 0.8. The training and inference of the entire network executed using 2 NVIDIA RTX 4090 GPUs. Additionally, we employ four variations to validate the effectiveness of our method. Specifically, -no NWE means no neural wireframe estimation, which extracts explicit line segments using DeepLSD 46] and constructs a distance field for each segment. then replaces the function Equations 2 and 7 with the distance function values, and solves for the pose using coarse-to- tarseeto-ffie pose selection followed by Gauss Newton refinement; 2) -no USR means model without uncertainty sampling range estimation (Sec. 3.2h; 3) -no Refine denotes model without pose refinement (Sec. 3.3 and 4) full model is our proposed LoD-Loc.
4.2 Evaluation Results
Evaluation over UAVD4L-LoD dataset. As described in Table our method shows excellent performance, both in the in-Traj. and out-of-Traj. queries. Apart from the 2m - 2" and 3m - 3 metric in the in-Traj queries, which are marginally lower than UAVD4L with RoMA matcher, all other metrics surpass those of contemporary baselines. Note that this comparison is unfair, as baselines reference on higg-preeision texture model that is richer in texture and geometry, while we only employ LoD model. We further compare with CadLoc, which shares the same 3D reference model as ours. However, we observe that regardless of the choice of descriptors (point-based or based, these methods perform poorly. We visualize their retrieval and matching failure cases in Appendix E.3 Furthermore, we analyze why our method performs better in the out-of-Traj, scenarios compared to the in- -raj. scenarios in Appendix F3

---

Table Quantitative comparison results over the Swiss-EPEL dataset.
In-Place
out-of- Out-of-Place
Method
2m-2"
3m-3
5m-5
2m-2"
3m-3"
5m-50
Generated Priors
0
0.56
1.06
SIFT+NN
14.47
23.31
36.52
32.98
54.35
71.50
SPP+SPG
34.83
60.39
77.25
77.04
89.71
92.35
UAVD4L
LoFTR
27.67
49.58
66.43
68.87
81.00
84.96
Mesh model
e-LoFTR
37.64
60.96
76.40
81.53
91.03
93.93
RoMA
45.98
66.77
80.73
89.18
98.68
98.94
SIFT+NN
0
0
0
0
0
SPP+SPG
0
0
LoFTR
0
O
0
0
0
0
CadLoc
e-LoFTR
0
0.14
0.14
0.53
LoD model
RoMA
0.98
1.97
2.67
2.37
5.01
6.33
SOLD2
0
0
DecpLSD+SOLD2
0
0
O
0
DeupLSDDGlluSttik
0
0
NWE
11.37
21.35
33.57
18.99
31.39
45.91
Ours LoD model
no USR
42.42
58.29
71.21
31,40
48.81
70.45
Rine
36.10
58.01
76.97
18.21
39.31
66.23
Full model
48.60
65.31
79.78
37.73
57.26
77.57
Table 4: Ablation study on different stages. T.e.R.e. means translation/rotation error.
Recall (%)
Median Error
Category
Level
2m-2
3m-3
5m-5"
T.e(m))
Ree.(")
Level 1
23.88
60.35
83.85
2.58
1.41
Level 2
48.57
75.06
85.10
2.03
1.27
in-Traj.
Level 3
51.31
76.06
86.78
1.97
1.25
Refine
84.41
91.77
96.95
0.97
0.52
Level 1
34.81
78.01
97.67
2.31
1.05
Level 2
65.37
95.35
99.22
1.76
0.97
out-of-Trajj
Level 3
74.27
97.95
99.36
1.63
0.95
Refine
95.94
99.00
99.36
1.06
0.49
Evaluation over Swiss-EPFL dataset. Table presents the inference results on the Swiss-EPFL dataset. CadLoc continues exhibit widespread failures due to its poor retrieval and matching results across different modalities. We surpass the wate-of-the--e UAVD4L method in the in- queries, but fall behind in the fut-of-Place queries. Moreover, the overall results obtained on the Swiss-EPFL dataset are not strong those on the UAVD4L-LoD dataset. We attribute this discrepancy the inferior quality of the images the training database as explained in Appendix F.T Besides, the LoD2.0 model provides less structured information, making it harder for pose inferences.
Analysis of Methodological Advantages. First, compared to the SOTA texture-basee approach, the LoD-Loc employs distinct cues for localization. The texture-bass based method determines the pose by optimizing the re-projection error of corresponding 2D-3D points. Conversely, the LoD-Loc aligns the 3D wireframe projection to solve the pose. Second, 3D- model- methods typically employ two-stage scheme, which involves building 2D-3D matches and then solving the pose with PnP RANSAC. The LoD-Loc method directly solves the pose in an end-to-end manner, potentially leading to better pose accuracy. Third, the LoD-Loc includes several important modules to improve performance, such as coarse-to-fine pose cost volume reconstruction, uncertainty-based sampling range estimation, and differential Gauss-Newton refinement. These factors contribute the superior performance of our method.

---

Query
Level
Level 2
Level 3
Refine
LOD33 3.0
LoD 2.0
Figure 5: Visualization of feature maps from different levels. The feature maps of different levels
reflect different fineness of wireframe extraction.
4.3 Ablation Studies
We perform ablation experiments the UAVD4L-LoD dataset, focusing different levels. More ablation studies are provided Appendix F22
Levels. As depicted in Table 4 present the results of the ablation experiments in terms of recall, translation errors, and rotation errors. The localization accuracy shows a gradual improvement as the number of levels increases. This demonstrates the effectiveness of the progressive coarse-to-fine estimation and final pose refinement. 3 visualizes feature maps for each level, illustrating that wireframe features extracted from deeper levels is clearer.
5
Conclusion
This paper presents LoD-Loc, a novel approach for localizing aerial images using a LoD 3D map. Compared large and expensive 3D maps that existing methods rely on, the LoD map provides simple, accessible, and privacy-friendle scene representation. With the coarse sensor pose, the proposed LoD-Loc uses a unified pipeline to estimate the camera pose, including multi-scale feature extractor, pose selection from cost volume, and pose refinement. Furthermore, we contribute two datasets with map levels of LoD3.0 and LoD2.0. along with real RGB queries with ground-truth pose annotation. LoD-Lot achieves excellent performance, even surpassing current state-of-the methods that use textured 3D models for localization. We believe LoD-Loc opens new possibilities for visual localization with simple and scalable 3D maps.
Linitation. LoD-Loc operates under the assumptions of known gravity direction and location prior. While these assumptions are reasonable, they restrict the application of LoD-Loc environments where GPS is denied unavailable.
Broader impact. This work has implications regarding privacy and surveillance. However, the LoD models represent building structures in highly abstracted form, which alleviates concerns about the disclosure of personal privacy or land resource information.
10

---

Acknowledgments and Disclosure of Funding
The authors would like acknowledge the support from the National Natural Science Foundation of China (Grant No. 62406331).
References
[1] Blender. BRLL:ttp:://www httpp:://www.beedeerr blender org/
[2] Dji zenmuse h20 series,
http::/entterprisee.ji..omm zenmuse- zenmuse-h20-serreee o0
URL
[3] Dji m300 rtk, URL /ttp://ppp//rarprppeeeee.a/////.
[4] Dp modeler. URL [tt.ap/ppp/errr.nnnanoooo//o///r aspx
[5] Mavic 3 pro. pttp////attp/////rrrrrria/cc///a
[6] Dji phantom rtk. /ttpht/pp//aaaappprepee..../oo.a
[7] Baidumap. URL httpps::mma..baaduu.com
[8] Googlemap. URL http://hh////.....ttoo.t...hhh/a
[9] Swissimage 10 em, URL
ortho/swissimagel htmll
[10] Swisssurface3d, surface3d. html
URL
[11] Swisssurface3d raster, URL http://ppp/..t...t..o........hhe
surface3d- html
[12] S. Agarwal, Y. Furukawa, N. Snavely, I. Simon, B. Curless, S. M. Seitz, and R. Szeliski. Building rome in a day. Commun. ACM, 2011.
[13] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. CVPR, 2016.
[14] D. Barath, J. Matas, and J. Noskova. MAGSAC: marginalizing sample consensus. In CVPR, 2019.
[15] M. B. Bejiga, A. Zeggada, A. Nouffidj, and F. Melgani. A convolutional neural network approach for assisting avalanche search and rescue operations with uav imagery. Remote Sens., 2017.
[16] F. Biljecki,, H. Ledoux, and J. Stoter. An improved lod specification for 3d building models. Comput. Environ. Urban Syst., 2016.
[17] F. Biljecki, H. Ledoux, and J. Stoter. Generating 3d city models without elevation data. Comput. Environ. Urban Syst., 2017.
[18] E. Brachmann and C. Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. T-PAMI, 2021.
[19] Y. Cai, Y. Zhou, H. Zhang, Y. Xia, P. Qiao, and Zhao. Review of target geo-location algorithms for acrial remote sensing cameras without control points. Appl. Sci., 2022.
[20] Camposeco, A. Cohen, M. Pollefeys, and T. Sattler. Hybrid scene compression for visual localization. In CVPR, 2019.
[21] S. Cao and Snavely. Minimal scene descriptions from structure from motion models. In CVPR, 2014.

---

[22] S. Chen, X. Wu, M. W. Mueller, and K. Sreenath. Real-time localizziintion using satellite imagery and topography for unmanned aerial vehicles. In IROS, 2021.
[23] O. Chum and Matas. Optimal randomized RANSAC. T-PAMI, 2008.
[24] O. Chum, J. Matas, and Kittler. Locally optimized RANSAC. In DAGM, 2003.
[25] D. DeTone, T. Malisiewicz, and Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPRW, 2018.
[26] J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsberg. RoMa: Robust Dense Feature Matching. 2024.
[27] M. A. Fischler and C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 1981.
[28] J.-M. Frahm, P. Fite-Georgel, D. Gallup, T. Johnson, R. Raguram, C. Wu, Y.-H. Jen, E. Dunn, B. Clipp, S. Lazebnik, et Building rome on cloudless day. ECCV, 2010.
[29] Y. Ge, H. Wang, Zhu, R. Zhao, and H. Li. Self-supervising fine-grained region similarities for large-scale image localization. In ECCV, 2020.
[30] H. Germain, G. Bourmaud, and V. Lepetit. S2DNet: Learning accurate correspondences for sparse-to-dense feature matching. In ECCV, 2020.
[31] S. Gui and R. Qin. Automated lod-2 model reconstruction from eeryhhigh-esoluttin satellite- derived digital surface model and orthophoto. ISPRS, 2021.
[32] B. M. Haralick, C.-N. Lee, K. Ottenberg, and M. Nölle. Review and analysis of solutions of the three point perspective pose estimation problem. IJCV, 1994.
[33] X. He, J. Sun, Y. Wang, S. Peng. Q. Huang, H. Bao, and X. Zhou, Detector-free structure from motion. arXiv preprint arXiv:2306. 15669, 2023.
[34] D. Jovanović, S. Milovanov, I. Ruskovski, M. Govedarica, D. Sladić, A. Radulović, and V. Pajić. Building virtual 3d city model for smart cities applications: A case study on campus area the university of novi sad. ISPRS, 2020.
[35] A. Kendall, M. Grimes, and R. Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In ICCV, 2015.
[36] J. Kim, J. Jeong, and Y. M. Kim. Fully geometric panoramic localization. In CVPR, 2024.
[37] L. Kneip, D. Scaramuzza, and R. Siegwart. A novel parametrization of the perspective-three point problem for direct computation of absolute camera position and orientation. In CVPR. 2011.
[38] T. H. Kolbe. Representing and exchanging 3d city models with citygml. In 3D Geo-Inf. Sci. 2009.
[39] Li and H. Lee. Deepi2p: Image-to-point cloud registration via deep classification. In CVPR, 2021.
[40] X. Li, S. Wang, Y. Zhao, J. Verbeek, and J. Kannala. Hierarchical scene coordinate classification and regression for visual localization. In CVPR, 2020.
[41] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 2004.
[42] S. Lynen, Zeisl, D. Aiger, M. Bosse, J. Hesch, M. Pollefeys, Siegwart, and Sattler. Large-scale, real-time visual-inertial localization revisited. IJRR, 2020.
[43] V. Panek, Z. Kukelova, and T. Sattler. Meshloc: Mesh-based visual localization. In ECCV, 2022.
[44] V. Panek, Z. Kukelova, and T. Sattler. Visual localization using imperfect 3d models from the internet. In CVPR, 2023.
12

---

[45] R. Pautrat, J.-T. Lin, V. Larsson, M. R. Oswald, and M. Pollefeys. Sold2: Self-supervised occlusion-awarc line description and detection. CVPR, 2021.
[46] R. Pautrat, D. Barath, V. Larsson, M. R. Oswald, and M. Pollefeys. Deeplsd: Line segment detection and refinement with deep image gradients. In CVPR, 2023.
[47] R. Pautrat, I. Suárez, Yu, M. Pollefeys, and V. Larsson. Gluestick: Robust image matching by sticking points and lines together. In ICCV, 2023.
[48] R. Peters, B. Dukai, S. Vitalis, J. van Liempt, and J. Stoter. Automated 3d reconstruction of lod2 and lodl models for all 10 million buildings of the netherlands. PE&RS, 2022.
[49] N. Piasco, D. Sidibé, V. Gooet-Brunet, and C. Demonceaux. Learning scene geometry for visual localization challenging conditions. In ICRA, 2019.
[50] M. Pietrantoni, M. Humenberger, T. Sattler, and G. Csurka. Segloc: Learning segmentation- based representations for 2023.
visual localization. In CVPR. pages 15380-15391,
[51] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisn -sssstte--sssteed Intervention-MIC 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part 111 18, pages 234-241. Springer, 2015.
[52] E. Saretta, P. Bonomo, and F. Frontini. A calculation method for the bipv potential of swiss façades at lod2. 5 in urban areas: A case from ticino region. Sol. Energy, 2020.
[53] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymezyk. From coarse to fine: Robust hierarchical localization large scale. In CVPR, 2019.
[54] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
[55] P.-E. Sarlin, D. DeTone, T.-Y. Yang, A. Avetisyan, Straub, T. Malisiewicz, S. R. Buld, R. Newcombe, P. Kontschieder, and V. Balntas. Orienternet: Visual localization in 2d public maps with neural matching. In CVPR, 2023.
[56] P.-E. Sarlin, E. Trulls, M. Pollefeys, J. Hosang, and S. Lynen. Snap: Self-supervised neural maps for visual positioning and semantic understanding. NeurlPS, 2024.
[57] T. Sattler, B. Leibe, and Kobbelt. Efficient effective prioritized matching for large-scale image-based localization. 7-PAMI, 2016.
[58] J. L. Schönberger and J. Frahm. Structure-fromt frote-ftfoooooooooonooooo revisited. In CVPR, 2016.
[59] Y. Shi and H. Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In CVPR, 2022.
[60] Y. Shi, X. Yu, D. Campbell, and H. Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In CVPR, 2020.
[61] M. Shibuya, S. Sumikura, and K. Sakurada. Privacy preserving visual slam. In ECCV, 2020.
[62] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In CVPR, 2013.
[63] M. Silvagni, A. Tonoli, E. Zenerino, and M. Chiaberge. Multipurpose uav for search and rescue operations in mountain avalanche events. Geomatics Nat. Hazards Risk, 2017.
[64] N. Snavely, S. M. Seitz, and R. Szeliski. Photo tourism: exploring photo collections in 3d. In SIGGRAPH. 2006.
[65] Speciale, J. Schonberger, S. B. Kang, S. Sinha, and M. Pollefeys. Privacy preserving image-based localization. In CVPR, 2019.
13

---

[66] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, 2021.
[67] H. Taira, M. Okutomi, Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii. Inloc: Indoor visual localization with dense matching and view synthesis. In CVPR, 2018.
[68] C. Toft, W. Maddem, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Polle- feys, J. Sivic, T. Pajdla, et al. Long-term visual localization revisited. T-PAMI, 2020.
[69] D. K. Villa, A. S. Brandao, and M. Sarcinelli-Filho. survey on load transportation using multirotor uavs. JINT, pages 267-296, 2020.
[70] S. Wang. J. Kannala, and D. Barath. Dge-gnn: Leveraging geometry and color cues for visual descriptor-free free 2d-3d matching. In CVPR, pages 20881-20891, 2024.
[71] Y. Wang, X. He, S. Peng, D. Tan, and X. Zhou. Efficient LoFTR: Semi-dense local feature matching with sparse-like speed. In CVPR, 2024.
[72] R. Wu, X. Cheng, J. Zhu, X. Liu, M. Zhang, and S. Yan. Uavd4l: A large-scale dataset for uay 6-dof localization. arXiv preprint arXiv:2401.05977 2024.
[73] Z. Xia, O. Booij, M. Manfredi, and J. F. Kooij. Visual cross-view metric localization with dense uncertainty estimates. In ECCV, 2022.
[74] Q. Yan, J. Zheng, S. Reding, S. Li, and I. Doytchinov. Crossloc: Scalable aerial localization assisted by multimodal synthetic data. In CVPR, 2022.
[75] S. Yan, M. Zhang, Y. Peng, Y. Liu, and H. Tan. Agenti2p: Optimizing image-to-pon poont cloud registration via behaviour cloning and reinforcement learning. Remote Sensing, 2022.
[76] S. Yan, X. Cheng, Y. Liu, J. Zhu, R. Wu, Y. Liu, and M. Zhang. Render-and-compare: Cross- view 6-dof localization from noisy prior. In ICME, 2023.
[77] S. Yan, Y. Liu, L. Wang, Z. Shen, Z. Peng. H. Liu, M. Zhang, G. Zhang, and X. Zhou. Long-term visual localization with mobile sensors. In CVPR, 2023.
[78] L. Yang, Shrestha, W. Li, Liu, G. Zhang, Z. Cui, and P. Tan. Scenesqueezer: Learning to compress scene for camera relocalization. In CVPR, 2022.
[79] Q. Zhou, S. Agostinho, A. OSep, and L. Lexl-Taixé. Is geometry enough for matching in visual localization? ECCV, 2022.
14

---

A Details on Dataset Collection
The released datasets consist of two distinct parts. UAVD4L-LoD and Swiss-EPFL, providing LoD3.0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans an area of 2.5 square kilometers, is derived from a semi-automatie process that generates 3D LoD map based on the mesh model of the UAVD4L (72) dataset. This dataset includes a diverse array of architectural structures, including skyserapers, villas, apartment complexes, educational institutions, and rural dwellings. The query images for this dataset were captured using two UAVs equipped with real sensor data: a DJI M300 [ drone with an H20T [2] camera and DJI Mavic3 Pro [5] drone. The Swiss-EPFL, dataset, which covers an expansive area of 8.18 square kilometers, sources its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [9H This dataset features variety of architectural styles, such as libraries, residential apartments, and stadiums. The query images for this dataset were acquired through the CrossLoc [74] projects, using DJI Phantom 4 RTK drone. Figure presents the 3D LoD maps and query images from these two datasets.
A.1 3D LoD Map Collection
The 3D LoD map for the UAVD4L-LoD dataset is generated semi-automatically with the assistance of the DP Modeler to)l[4] The process begins with the automatic generation of building blocks, characterized by their footprints and heights. Manual refinement then applied to the architectural details of each building, raising them to the LoD3.0 0 level. The LoD accuracy of the UAVD4L-LoD dataset is consistent with the mesh model derived from UAVD4L.
For the Swiss-EPFL dataset, LoD2.0 models are downloaded from the SwissTOPO website [H] To synchronize the coordinate systems between the map data and the drone-captured data from the CrossLoc dataset (which covers the same area with ground truth pose annotation), we converted the Swiss LoD map data in LV95 and LN02 coordinate systems the ECEF coordinate system.
A.2 Query Image Collection
The query images of the UAVD4L-LoD dataset are divided into two categories: in-Traj. and out- Traj,, representing trajectory- bassed and free-flight scenarios, respectively. The in-Traj. images, totaling 1, 604, were captured using a DJI M300 drone equipped with an H20T camera, focusing primarily residential buildings, villas, and educational institutions. In contrast, the out-Traj. images, totaling 2, 192, were captured using a DJI Mavic3 Pro drone, covering variety of architectural structures such skyscrapers and rural dwellings. Both the in-Traj. and out-of-Traj. datasets include real sensor priors. Table SaalesJootliigg [Juulinnss the specific differences between the in-Traj. and out-Traj. sequences.
Table 5: Key distinctions between the in-Traj. and out-of-Traj. sequences.
Name
Capture device
Capture pitch angle
Capture height
Capture route
in-Traj.
DJI M300+H20t
or 45"
120m
Zig-zag flight on se-
lected region
out-of-Traaj
DJI Mavic3 Pro
30 ~ 60
90m ~ 150m
Manually controlled
flight on the map
The real query images in the Swiss-EPFL dataset come from the CrossLoc [74] dataset. However, because the real-time kinematics (RTK) data from the DJI Phantom4 were used directly as ground truth (GT) poses, some GT poses show significant mislabeling. To resolve this issue, we projected the wireframes of LoD maps onto query images to identify and remove incorrectly labeled poses. The final query dataset comprises 2, 254 images.
A.3 Query GT Generation
For the UAVD4L-LoD dataset, we employ a semi-automatic annotation approach to generate pseudo- GT poses } for the query images ). The process is based on the SfM results and textured mesh model of the UAVD4L (72). First, we perform SfM separately on the query images (I?) and the reference images {Iā} from the UAVD4L to obtain SfM results C2 and C,.. Next, manually
15

---

in- Traj.
out-of-Tray
Figure 6: Flight trajectories of query images the UAVD4L-LoD dataset. We present the flight trajectories of the registered in-Traj. and out-of-Traj. query images. The in-Traj. images follow a predetermined flight path, primarily covering the left half of the map. In contrast, the oul-of-Trajj. images navigate arbitrarily without a fixed route, randomly covering the entire map.
select points with distinctive visual features (e.g., building corners) as tie points to align Cg with C,. To further enhance the accuracy of the pseudo-GT, we utilize the sender-and-compnt pipeline (76 refine the final poses {
Additionally, we analyze the discrepancies between the pose prior and the GT pose, decoupling the poses into 3D translation in WGS84 space and Euler angles in terms of "yaw-pitch-roll,, It is observed that the translation errors for x and are within 10, z errors are within 4-30, yaw errors within +7.5, and pitch and roll erross approximately 1 degree.
B Details on UAVD4L-LoD Dataset
B.1 Pseudo-GT Generation
In the UAVD4L-LoD dataset, we employed semi-automatic annotation technique to generate pseudo-GT poses {5.) for the query images (I,) Initially, performed SfM separately the query images {I;} and the reference images {Ij} from UAVD4L, yielding corresponding SIM results Cg and C,. Subsequently, based the capture region of the (L, , manually identified distinctive tie points, such as the corner of the building, to align C, with C,, resulting in C,. We then refined the pose accuracy of C using Bundle Adjustment. The accuracy of the GT poses the median reprojection error, which was 0.43 pixels for all connected points and 1.19 pixels for the tie points. Finally, employed render-and-compare (76) pipeline for the final refinement of the GT poses. this manner, with the annotation of tens of e = 20 manual tie points, we were able to obtain pseudo-GT poses {E,} trajectories of the in-Traj. and out-of-Traj..
evaluated through
} for a total of 3, 796 query images (I,). Figure shows the flight
B.2 Sensor Pose Accuracy
In the UAVD4L-LoD dataset, we conduct a comprehensive data analysis to validate the precision of the sensor pose. This is accomplished by employing absolute error bar charts, illustrated in Figure Additionally, we assess the accuracy by projecting wireframe points onto the image plane using both sensor and GT poses. Results of these projections are depicted in Figure g
C Details on Swiss-EPFL Dataset
C.1 Data Cleaning
In the Swiss-EPFL dataset, the GT poses } for the query images (I.) are sourced from CrossLoc project (74. This project directly acquires RTK data from the DJI Phantom 4 for GT annotation. Considering that the RTK device may introduce some noise, we identified and excluded query images with incorrect labeling. This was accomplished by projecting the wireframe onto the
16

---

Imman 2.1252
Y 5.060n.5.060
Inien: Tmncnr3.8555 3.8031
100
243
223
200
03
200
63
in- Traj
Grortel
Fredin
Frarinl
10W (msan:0.766
Pitch (mean:1.260)
Roi Imean:0.9911
Numberd phonee
120
20-
75
54
20-
Emeeddee)
knordee)
X Imeen 1.423)
Y Imeeni2 622)
(meen-133003
250
272
392
150
30C
150
100
out-of- Traj.
(mean:1.000)
0.721)
0.574)
400
352
700
202
350
50
enoodee1
Figure The errors between priors and GT poses. We visualize the absolute pose errors between the sensor and GT pose in 6-DoF. The errors in the X-Y-Z-yaw dimensions show indicate substantial discrepancies. Specifically, the errors in the X-Y range from -10 to 10 meters, the ranges from -30 to 30 meters, and the yow fluctuates within the range of -7.5 to 7.5 degrees.
image plane and manually discarding the items exhibiting noticeable misalignment. The process is visualized Figure
C.2 Sensor Poses Generation
Since the CrossLoc 74] project does not provide GPS or other sensor data, we randomly generate sensor poses &, by emulating the pose errors derived from the UAVD4L-LoD dataset. Specifically, X-Y for position range between [-10, 10] meters, Z ranges between [-, 30) meters, yaw for rotation ranges [-7.5, 7.5) degrees, and pitch-roll range between [-1, degrees. We present discrepancy between the generated sensor poses and GT poses in bar chart, depicted in Figure []]
17

---

Priors
GT
Priors
GT
200
Traj.
out- out-of-Tray Traj.
Figure 3D wireframe projection over UAVD4L-Lod dataset. We visualize the projected wireframe on query images based on sensor and GT poses to demonstrate their accuracy.
D Details on Method
D.1 Architecture of Multi-scale Feature Extractor
In this section, we provide a detailed description of the architecture of the multi-scale feature extractor in Table 6.
D.2 Jacobian Computation
The objective function for pose refinement
E(E") - Σ =
=
-
PP;+
(11)
We compute the Jacobian matrix of the residual function with respect to the pose parameters as followed:
0f:
OF
r
Op, P
J 0
(12)
Op;
Pcam
0E
18

---

Mislabeled X
Selected
Figure 9: Samples of mislabeled and selected query images over Swiss-EPFL dataset. We eliminate mislabeled query images by manually identifying the alignment between the projected 2D wireframe and the corresponding RGB image.
where
0F.f the gradient of the feature map F,f at the 2D location p; and Sp.
Xeam
Zeams fx + Cr.
P: (Pamm
hbam
Zeam f + Cyy
(13)
Öp:
1 Z.aa
Ja
(Z y
X.am
Jx
Doam (Zaamzz fy.
Ppe
7eam
Besides, Peam is the point which transformed to the camera space. To compute the last derivative of we add a perturbation A the transformation:
Eq. 12
P.am R R (ARP: + At) +t",
(14)
Finally, the derivatives w.r.t the translation component and rotation component are:
àP;
aP:
R
AAt
(15)
OP:
OP:
[P,x
050 ØAR
where Ox is the skew-symmetric matrix.
E Details on Baseline
E.1 Sensor-guided Image Retrieval
For baselines, retrieval-and-macchin process is used upon the reference images in the dataset. To ensure a fair comparison, we apply the sensor poses to guide the image retrieval process for narrow the retrieval candidates "I
UAVD4L [72 and Cad-Loc 44 For each query image
using Eq. 16
(16) where ||+|| denotes the Frobenius Norm between two translation matrices, arccos(-) calculates the rotation angles between two matrices, Ttt and 7o are the threshold for translation and orientation, respectively. To determine the proper values for y and % for the bascline methods, series of
II I
||C
|| S Tts arccos (R, < <o)
19

---

Mesh-basse
LoD-based
RGB
Depth
RGB
Depth
Figure 10: Visualization reference RGB and depth maps. RGB and depth maps are rendered using a textured mesh model 3D LoD map.
experiments are conducted on the UAVD4L dataset. these experiments, we hypothesize that no reference image could be located within the defined search area, the sensor pose would be utilized as the localization result. Table 7 that stricter thresholds result worse outcomes. Consequently,
we set % = 150 and % 30 to ensure sufficient search space. Furthermore, we measure the impact of retrieval number & in Table 8 The results suggest that while a larger value enhances the performance of the benchmark algorithm, it also leads an increase in inference time. Following previous work [72], the retrieval number is set at k 3. is worth noting that regardless of the choice of k our method exhibits a substantial acceleration in speed, outperforming by several-fold, or even an order magnitude.
E.2 Reference Image Details
We provide detailed description of the reference images used in the two datasets. These images serve dual purposes: they function as the database images for retrieval and matching in baselines, and they also utilized training data for the proposed LoD-Loe method. Specifically, for the UAVDAL--LoD dataset, we use a subset dataset of synthesis images in UAVD4L (72), excluding data that does contain buildings. For the Swiss-EPFL dataset, synthetic images rendered in Latin Hypercube Sampling (LHS) [74] pattern have been employed as reference images. Notably, the CrossLoc dataset [74] did not include images in proximity to the out-of-Place area. To address this, we adopted the same synthetic scheme from 774 to generate synthetic reference images for this region. Figure 10 shows reference samples of RGB images and Depth images for both the mesh-based model and LoD-based model.
E.3
Failure Cases in Baselines
Although baselines have achieved impressive performance, they suffer from retrieving and matching repetitive texture images and cross-modal images. For example, Figure [3 exhibits exhhibits deficiencies in retrieving repetitive texture images, and Figure [4 depicts poor matching results between RGB and LoD-rendered images.
20

---

Imean:: D47?
(mmean=5 008)
7mman:115 17)
in-Place
Emortee
25-
20-
Nambe
30-
Enerideg)
37.0
Sumber Sumeraaltan dl
15.0
10.0
255
75
51
out= out-of-Plaae Place
x
25
-10
-10
10
-20
Emaimn
(mean:3.8B)
17.5
hunoerel photee
13.0
13.0
10.0
10.0
15
2.5
0.6
85
16
-1.0
-0.5 0.6
Emoridegl
Emwridgg
Emwrldeg)
Figure 11: The discrepancy between our generated poses and GT poses the Swiss-EPFL dataset. We use the generated poses to simulate the pose of the sensor.
F Details of Experiments
F1 Visualization of Training Data
We visualize some synthetic training samples of LoD-Loc, as shown in Figure 15 For the Swiss- EPFL dataset, the reference 3D model is derived from LiDAR point clouds, Terrain Models, and Orthophotos. In contrast, in the UAVD4L-LoD dataset, the reference 3D model is generated from
21

---

Table The architecture our multi-scale feature extractor. We discuss the details of each con- volutional unit. conu represents a unit consisting of a 2D convolutional layer, batch normalization layer and a ReLU layer. While f'ne fine_come come denotes general convolutional layer. decone means deconvolutional unit. The colored cells are the outputs for each level / with single channel.
Layer
Stride
Kernel
Channel
Input
convO 0
1x1
3x3
3-+8
rgb
convo 1
1x1
3x3
8-38
convÒ 0
conv 0
2x2
5x5
8-+16
conv0
conv1
1x
3x3
16->16
conv 0
conv 2
Ix
3x3
16-+16
conv
conv2 0
2x2
5x5
16-+32
conv 2
conv2
1x1
3x3
32 >32
conv2 0
conv2_2 2
1x1
3x3
32-+32
conv
conv_out
1xl
1x1
32-11
conv
deconv1
2x2
3x2
32->16
conv2_2
concat
deconvI_00 conv 1_2
conv3_0
1x1
3x3
32 92-$16
concat
conv out2
1x1
1x1
16-11
conv3 0
deconv2_0
2x2
3x3
16-+8
conv3_
concat2
deconv2_00 conv0_
conv4 0
1xL
3x3
16-+8
concat2
conv_oie
1x1
1x1
8->1
conv4_0
conv4 0, conv_out3,
concat3
rgb
fine_com line_conv(
1x1
5x5
12-+24
concat3
fine_conv
1x
5x5
24-112
fine_conv(
conv_oute
1x1
1x1
12-11
fine_conv
Table 7: Ablation study on different threshold and % for baselines.
Threshold
in- in-Traj.
out-of out-of-Traj.
Method
(7t 7a)
2m-2-
3m-30
5m-5
2m-2-
3m-33
5m-50
(30, 7.5)
0.62
0.69
4.99
25.87
26.82
27.4
SIFT+NN
(50, 15)
27.00
28.30
32.29
55.66
57.44
58.26
(150, 30)
73.13
78.62
80.42
82.39
85.13
86.36
7.3)
0.94
0.94
5.24
30.11
30.20
30.29
UAVD4L
SPP+SPG
(50, 15)
33.92
33.92
37.28
60.99
61.13
61.18
(150,
91.71
92.02
92.14
93.43
93.70
93.80
(30, 7.5)
0.94
0.94
5.20
29.79
30.02
30.16
LoFTR
(50, 15)
33.29
33.35
36.72
60.90
60.90
60.99
(150, 30)
84.98
88.09
88.90
91.56
92.02
92.11
high-resolution aerial imagery through oblique photography reconstruction. As result, the synthetic images from the former are of lower quality. This could partly elucidate why our method yields lower results the Swisx-EPFL EPFL dataset compared to UAVD4L-LoD.
F.2
Additional Ablation Studies
We provide more ablation studies in this section, which include the pose sampling number, the sample density 5 of 3D wireframes, the sampling range controller lambda X. Additionally, we explore the convergence and generalization of our method.
Pose sampling number. As illustrated in Table we report the experimental results with varying numbers of pose samples. The findings suggest that a reduction the number of sampled poses brings about a decrease in accuracy.
22

---

Table 8: Ablation study on different Top-k for baselines.
in-Traj.
out-of- -f-Traj.
Time
Method
Top-
(s)
2m-2
3m-3"
5m-5
2m-22
3m-3"
5m-5"
73.13
78.62
80.42
82.39
85.13
86.36
1.85
SIFT+NN
10
85.97
89.65
90.52
90.28
92,43
93.66
1.96
20
88.09
91.33
92.64
92.75
94.71
95.99
2.13
3
91.71
92.02
92.14
93.43
93.70
93.80
1.79
UAVD4L
SPP+SPG
99.25
99.31
99.31
98.45
98.49
98.49
3.31
20
99.75
99.81
99.81
99.91
99.95
99.95
5,44
84.98
88.09
88.90
91.56
92.02
92.11
1.70
LoFTR
10
90.21
91.65
92.08
94.75
94.89
94.89
3.78
20
85.97
87.53
87.91
90.37
90.83
91.29
6.26
Ours
84.41
91.77
96.95
95.94
99.00
99.36
0.34
Table 9: Ablation study on different pose sampling numbers for LoD-Loc.
Numbers on
Recall (%)
Median Error
Category
[0, x. y, z]
2m-2"
3m-3*
5m-5
T.e. (m)
R.e. (*)
[2 2,,3,3,0
18.83
24.94
36.03
7.67
4.37
in-Traj.
[4, 5,5,
77.68
84.98
90.15
1.07
0.59
[8, 10, 10, 30]
84.41
91.77
96.95
0.97
0.52
[2,3.3.8]
12.36
16.93
23.81
11.49
5,51
out-of-Traj.
[4, 5, 15]
87.27
93.25
94.25
1.15
0.54
18, 10, 10, 30]
95.94
99.00
99.36
1.06
0.49
3D wireframe points sampling density. We conduct ablation studies for varying sampling densities, which affects the interpolation process on the feature map. As depicted in Table [0] there is no significant fluctuation localization accuracy with changes sampling density.
Sampling range controller. The parameter lambda À adjusts the length of the sampling range. Through ablation studies, demonstrate that the sensitivity of this parameter during the testing phase is low. The results are shown in Table
Convergence and initial poses. Table 12 reports the localization recall with different initial prior errors on the UAVD4L-LoD dataset. can be observed that the success rate localization decreases the initial prior eтror increases. Such issues occur when the GPS signal in the air is heavily interfered with. In such cases, we believe using sequence information could be a possible solution.
Cross-scene generalization. Table 13 illustrates the generalization capability of LoD-LLoc through regional data using distinctive symbols and colors. On the UAVD4L-LoD dataset (A1 and A2), cross-scene testing yields results slightly lower than those obtained from training on the entire scene. For the Swiss-EPFL. dataset (B1 and B2), employ model trained on the synthetic UAVD4L-LoD dataset, which achieves similar better performance compared a model trained specifically on the Swiss-EPFL dataset. Additionally, the supplementary materials include two demo videos showcasing the model's capacity to localize cross cross-modal modal thermal images.
training and testing diverse regions. Figure [6]deinnettss /elee
Computational cost comparison. We conducted test experiments single batch (Batch Size of images using the NVIDIA GeForce RTX 4090 device, and recorded the average peak CUDA usage as well as the average inference time. The details are provided in Table 14
F.3 Visualization of Results
present more visualization results, including examples of corner houses (Figure 12). feature maps (Figure 17) and prediction results (Figure T8) at different levels. We found that the preset zig-zag route in selected region resulted in some images capturing only the corners of houses, as shown in Figure 18 This led to poorer performance under strict 2m-22 metrics. However, it is important
23

---

Table 10: Ablation study on different wireframe sampling density. x-m means sampling per-x meter on each wireframes.
Recall (%)
Median Error
Category
Density
2m-2
3m-3"
5m-5
T.e. (m)
R.e. (*)
4-m
85.10
92.39
96.51
0.95
0.52
in-Traj.
2-m
84.16
91.08
96.95
0.97
0.52
1-m
84.41
91.77
96.95
0.97
0.52
LoD-Loc
4-m
95.21
98.68
99.18
1.00
0.45
out-of- Traj.
2-m
95,44
98.91
99.32
1.06
0.48
1-m
95.94
99.00
99.36
1.06
0.49
Table 11: Ablation study different Lambda X.
Recall (%)
Median Error
Category
Lambda
X
2m2
3m3
5m5
T.e. (m)
R.e. (9)
1.5
83,42
91.02
96.57
1.00
0.49
84.41
91.77
97.01
0.95
0.53
in- -naTraj.
0.8
84.41
91.77
96.95
0.97
0.52
LoD-Loc
0.5
84.04
91.58
96.45
0.97
0.52
1.5
91.97
97.54
98.45
1.11
0.53
95.71
99.04
99.36
1.07
0,50
out-of-Traj.
0.8
95.94
99.00
99.36
1.06
0.49
0.5
95.71
98.86
99.32
1.06
0.49
note that in the in-Traj. scenario, our method achieves comparable superior results for coarse metrics. For instance, we achieve 96.95% on 5m-5% while the closest baseline achieves 92, 14%.
24

---

Table 12: Impact of the initial pose for LoD-Loc. The parameters Ar and Ay denote the error range in the horizontal plane, while Az represents the error range in the vertical dimension. For instance, Az 10 implies that the imitial error in the r value lies within the interval [-10, 10]. The rotation error remains consistent with the real sensor data. All error ranges are measured in meters.
Recall (%)
Category
Prior Error Range
Az, Ay, Ae]
2m-2
3m-3
5m-5
[10, 10, 30]
84.41
91.77
96.95
[20, 20, 30]
87.28
90.77
91.65
in-Traj.
[30, 30, 30]
78.93
82.98
83.85
[50, 50, 30]
43.08
48.82
50.69
[100, 100, 30]
5.67
7.36
8.79
LoD-Loc
[10, 10, 30]
95.94
99.00
99.36
ofl-of-Traj.
[20, 20, 30]
82.07
88.05
89.55
[30, 30, 30]
74.27
80.66
81.79
[50, 50, 30]
46.53
53.60
55.98
[100, 100, 30]
6.93
9.95
11.99
Table 13: Cross-scene generalization. We assess the generalization ability of method by training and testing on different regions. The regional divisions are illustrated in Figure identified by specific color and letter.
Recall (%)
Train region Synthesis
Test region Real
2m-2º
3m-3"
5m-50
A2
A1
83.39
91.50
96.81
A1, A2
A1
89.51
95.01
97.98
A1
A2
82.54
91.01
91.52
A1, A2
A2
95.56
98.66
99.38
LoD-Loc
AI, A2
B1
55.41
71.77
84.17
B1, B2
B1
37.73
57.26
77.57
AI, A2
B2
50.00
59.27
65,45
B1, B2
B2
48.60
65.31
79.78
Method
Memory (Mb)
Time (s)
SPP
610
1.79
SIFT
1.85
UAVD4L
LoFTR
2631
1.70
RoMA
5488
4.68
eLoFTR
1650
1.06
ours
4810
0.34
Table 14: Computational cost comparison.
Figure 12: Example of corner houses.
25

---

Query Image
Retrieval Results Top 3 (Failure)
UAVD4L
Cac-Loc
Figure 13: Failure retrieval cases of baselines. Even with narrowed searching scopes, the retrieval phase still suffers from issues such as repetitive textures and cross-modal challenges.
26

---

Query Image
Matching Results Top 3 (Failure)
Query Image
Matching Results Top (Failure)
UAVIAA
Cad+Loo
Figure 14: Failure matching cases of baselines. The differences in viewpoint and modality influence the results for image matching.
27

---

U UAVDALA LoD
Swise EPFL
Figure 15: Samples of the training dataset. The UAVD4L-LoD dataset offers high-quality training set, while the Swiss-EPFL. dataset suffers from lower quality, as evidenced by issues such as blurriness and voids on the sides of buildings.
B2
A2
A1
B1
UAVDAL-LOD
Swiss- EPFL
Figure 16: Region of training and testing. We use boxes with different colors and symbols to delineate different regions.
28

---

Query
Level 1
Level 2
Level 3
Refine
LoD 3.0
LOD2A0 2.0
Figure 17: Visualization of feature maps at different levels. The feature maps at different levels reflect varying degrees of fineness in wireframe extraction.
29

---

Priors
Level 1
Level 2
Level 3
Refine
UAVIAL-LOD
Swiss EPFL
Figure 18: Visualization of predictions at different levels. Based on the predicted poses at each stage, we can obtain 2D projected wireframe and overlay them on the query image check the accuracy of the poses. It can be observed that as the levels progress, the projected wireframes gradually align with the edges the buildings. Please zoom in see the details of the alignment.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: can be found in Abstract and Introduction Section |
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: can be found in the end the Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: can be found in Experiment Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer:
[Yes]
Justification:
The code and dataset will be made available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee //nips.com htte:/ubbl///liiiee////eeeuuussl ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: can be found in Experiment Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not included due to the high computational cost of calculating them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: can be found in Section Experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
It is indicated in the Conclusion Section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The data and code used the paper comply with licensing and usage terms and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Once the paper is accepted, the code and data will be open-sourced with detailed usage instructions.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "T2dbFB9AFu", "reviewer": "Reviewer_RdXx", "summary": "The paper proposes a method named LoD-Loc for visual localization of unmanned aerial vehicles (UAVs) using Level-of-Detail (LoD) 3D maps. Unlike existing algorithms that rely on complex 3D representations, LoD-Loc aligns wireframes derived from LoD model projections with the wireframes predicted by a CNN Unet to estimate the UAV's pose. To solve the problem, given a coarse pose provided by the sensor, this paper utilizes a hierarchical cost volume to pose hypothesis evolution. And eventually, a differentiable Gaussian-Newton algorithm for 6-DoF pose refinement.  This work also contributes two datasets to facilitate further research in this area and provides a comprehensive evaluation of the proposed method's performance. The experimental results demonstrate that LoD-Loc outperforms current state-of-the-art methods using textured 3D models, offering advantages in ease of acquisition, lightweight map size, and privacy preservation.", "questions": "The proposed idea in general sounds okay to me. However, my questions are mainly about the result of the experiment result listed in the Weaknesses. I think the result present is questionable, insufficient, and lacks enough explanation to support the proposed method.\nBesides, I think the writing and presentation need to be further improved for this paper to be accepted.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are the following:\n\n**Interesting Methodology:**\n\nThe proposed LoD-Loc method introduces an interesting approach to visual localization using Level-of-Detail (LoD) 3D maps, which is less reliant on complex 3D representations. This makes the method more accessible and scalable compared to current textured 3D model-based approaches.\n\n**Application in practical scenarios:**\n\nLoD maps are easier to acquire and maintain and have a smaller file size. These advantages make LoD-Loc more feasible for real-world applications, particularly in large-scale environments where data storage efficiency is a critical requirement.\n\n**Contribution of Datasets:**\n\nThe paper contributes two largest-scale air localization datasets (LoD3.0 and LoD2.0) with RGB queries and ground-truth pose annotations. These datasets provide interesting resources for further research and benchmarking in aerial visual localization.", "weaknesses": "**Experiment Result**\n\nThe author mentioned CadLoc \"cannot even localize the query image at a coarse threshold at $5m -5^{\\circ}$\". However, what I observed from Table 2 even Table 3 is the result for CadLoc are all zeros. Is that a mistake?\n\nIn Table 2, the authors show their method surpasses the SOTA textured 3D-model-based method in the $2m -5^{\\circ}$ and $3m -5^{\\circ}$ and \"out-of-Traj.\"\nHowever, the author does not explain why their method can perform better than textured 3D-model-based methods in \"out-of-Traj\" scenarios but fails in \"out-of-Traj\" scenarios.\n\nIn the Appendix, the author mentioned that \"in-Traj.\" and \"out-of-Traj.\" represent trajectory-based and free-flight scenarios, respectively.\nBut I didn't catch what might be the key difference between these two scenarios. Does that mean these methods are actually verified in different trajectories in the \"out-of-Traj.\" scenarios? \nIf so, I think it is not a fair comparison. If not, I didn't get the key difference between the \"in-Traj.\" and \"out-of-Traj.\" scenarios. It is suggested to explain why the proposed method performs better than the SOTA textured 3D-model-based method considering they use high-precision texture 3D information.\n\nWhile the author mentions they conducted experiments on many other image retrieval-and-matching and textured 3D models-based approaches, only the results of UAVD4L and CadLoc are presented. It would be better to compare with more methods in each type to show a more comprehensive result.\n\n**Computational Cost**\n\nAlthough not mentioned, the methods involving hierarchical cost volume construction and differentiable optimization algorithms can be computationally intensive. This could pose challenges for real-time applications, especially on resource-constrained UAVs. For more comprehensive research, it would be better to provide a computational cost comparison for that.\n\n**Presentation**\n\nI think the writing and presentation skills in this paper could be further improved. Such as the presentation of Table 1 and the Contribution sections. Besides, I would suggest the author double-check their result in the presented Tables. Besides, I would suggest that the author add more explanations for their experiment result instead of just simply displaying them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1: \"What I observed from Table 2 even Table 3 is the result for CadLoc are all zeros. Is that a mistake?\"**\n\n**A1.1:** Upon thorough verification, we confirm that the results for CadLoc in both Table 2 and Table 3 are indeed zeros. The primary reason may be that advanced RGB-based retrieval and matching algorithms are not effective for cross-modality tasks involving RGB-to-LoD images. For further insights, please refer to \"Figure 12: Failure retrieval cases of baselines” and “Figure 13: Failure matching cases of baselines” in the **Appendix**, which illustrate the limitations of these methods.\n\n**Q1.2: \"The author does not explain why their method can perform better than textured 3D-model-based methods in \"out-of-Traj\" scenarios but fails in \"out-of-Traj\" scenarios.\"**\n\n**A1.2:** We acknowledge that reviewer would like to figure out why our method performs better than textured 3D-model-based methods in \"out-of-Traj.\" but lags behind in \"in-Traj.\" scenarios. We apologize for the lack of detailed explanations and will incorporate additional clarifications in the revised paper.\n\nFirstly, we provide a comprehensive comparison of 'in-Traj.' and 'out-of-Traj.' scenarios. Specifically, 'in-Traj' and 'out-of-Traj' scenarios refer to two distinct query image collections as introduced in the **Appendix A2** and depicted in \"Figure 6: Flight trajectories of query images in the UAVD4L-LoD dataset.\" The main differences are summarized in the following table:\n\n| **Name** | **Capture device**  | **Capture pitch angle**   |  **Capture height**   |   **Capture route**  | \n|:--------:|:--------:|:----------:|:----------:|----------|\n| in-Traj.   | DJI M300+H20t    |  0&deg; or 45&deg;  |  120m   |  Zig-zag flight on a selected region | \n| out-of-Traj.   | DJI Mavic3 Pro  | 30&deg; ~ 60&deg;     | 90m ~ 150m   |  Manually controlled flight on the map   |\n\nSecondly, upon careful examination of the failure cases in the 'in-Traj.' scenario, we found that the preset zig-zag route on a selected region resulted in some images capturing only the corners of houses, as shown in Figure 1 in the **PDF** of the global response. This led to poorer performance in strict 2m-2° metrics. However, it is important to note that for coarse metrics in the ‘in-Traj’ scenario, our method achieves comparable or superior results. For instance, we achieve 96.95% on 5m-5° while the closest baseline achieves 92.14%.\n\n**Q1.3: \"I didn't catch what might be the key difference between these two scenarios ( 'in-Traj.' and 'out-of-Traj.') ...\"**\n\n**A1.3:** Please refer to **A1.2** for the key difference between 'in-Traj.' and 'out-of-Traj.' scenarios.\n\n**Q1.4: \"It is suggested to explain why the proposed method performs better than the SOTA textured 3D-model-based method considering they use high-precision texture 3D information.\"**\n\n**A1.4:** First, compared to the SOTA textured-based approach, our method employs distinct cues for localization. The textured-based method determines pose by optimizing the re-projection error of corresponding 2d-3d points. Conversely, our novel approach aligns the 3D wireframe projection to solve the pose. Given that building structures are primarily composed of wireframes, we posit that leveraging such cues for localization is a well-motivated solution.\n\nSecond, 3D-model-based methods typically employ a two-stage scheme, which involves building 2d-3d matches and then solving the pose with PnP RANSAC. Our method directly solves for the pose in an end-to-end manner, which may result in better pose accuracy.\n\nThird, we introduce several important modules to improve performance, such as coarse-to-fine pose cost volume reconstruction, uncertainty sampling range estimation, and differential Gauss-Newton refinement. These factors contribute to the superior performance of our method.\n\nWe will provide additional explanations for the experimental results in the revised paper.\n\n**Q1.5: \"It would be better to compare with more methods in each type (UAVD4L and CadLoc) to show a more comprehensive result.\"**\n\n**A1.5:** The UAVD4L and CadLoc frameworks, which are the SOTA and the most suitable for this task, are designed for texture models and CAD models, respectively.  To provide more comparison, we include SOTA matching methods (RoMA[1] and e-Loftr[2]) within the UAVD4L and CadLoc frameworks. Please refer to Table 1 and Table 2 in the **PDF** of the global response for detailed comparisons.\n\n**Q2: \"For more comprehensive research, it would be better to provide a computational cost comparison for that.\"**\n\n**A2:** Please refer to Table 3 in the **PDF** of the global response. We conduct test experiments on a single batch (Batch Size = 1) of images using the NVIDIA GeForce RTX 4090 device, and list the average peak CUDA usage as well as the average inference time.\n\n**Q3.1: \"I think the writing and presentation skills in this paper could be further improved. Such as the presentation of Table 1 and the Contribution sections.\"**\n\n**A3.1:** Thanks a lot for your suggestion. We have updated our manuscript to enhance the presentation of Table 1 and the Contribution sections, providing more detailed and clear information.\n\n**Q3.2: \"I would suggest the author double-check their result in the presented Tables.\"**\n\n**A3.2:** Thanks a lot for your suggestion. Please refer to **A1.1** for the verification of the results presented in the Tables.\n\n**Q3.3: \"I would suggest that the author add more explanations for their experiment result instead of just simply displaying them.\"** \n\n**A3.3:** Thanks a lot for your suggestion. We have added more explanations, including why our method surpasses 3D-model-based methods and why our method performs better in “out-of-Traj.” scenarios compared to “in-Traj.” scenarios.\n\n**Reference**\n\n[1] Edstedt et al., RoMa: Robust Dense Feature Matching, CVPR 2024\n\n[2] Wang et al., Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x40TfIv1Aq", "reviewer": "Reviewer_UAbv", "summary": "The paper presents an approach to provide an accurate camera pose of an aerial image according to a LoD 3D map, which is a lightweight representation to store the key information of a known map. For each query image, multi-scale wireframe probability maps are extracted by a trained U-net. These maps are then used to select sampled poses from cost volume. Such process is executed in a coarse-to-fine manner. Lastly, Gauss-Newton is used to refine the final pose by minimizing the projection error of 3D wireframe points.\n\nTwo LoD city datasets are also released with the paper.", "questions": "Do you have any process to deal with the sky pixels in query images?", "limitations": "The authors have mentioned the limitations of the method and the potential privacy impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is able to estimate an accurate pose from a LoD model and the experiments show the method can achieve generally comparable results compared to the baseline methods, among which one uses a high-cost texture mesh map for localization.\n\nThe design of coarse-to-fine pose selection is novel in this task and is validated to be useful in the experiments section.\n\nThe release of the LoD dataset could be beneficial to the aerial localization research field.", "weaknesses": "In this paper, the definition of \"in-Traj\" and \"out-of-Traj\" is not clearly provided. I guess this may mean the data used to train the model and those in the test stage. Please clarify these concepts in the paper.\n\nThe proposed method is significantly worse than UAVD4L in \"out-of-place\" part of Swiss-EPFL dataset. The authors explain that this is due to the relatively low quality of the dataset and LoD2.0 model. However, this shows that the method is much less robust to the data quality and it only works when high-quality data is provided, which can be very costly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “The definition of \"in-Traj\" and \"out-of-Traj\" is not clearly provided.”**\n\n**A1:** Sorry for the confusion. The 'in-Traj' and 'out-of-Traj' scenarios refer to two distinct query image collections as introduced in  **A.2 Query Image Collection** in the **Appendix** and depicted in \"Figure 6: Flight trajectories of query images in the UAVD4L-LoD dataset.\" The main differences are summarized in the following table:\n\n| **Name** | **Capture device**  | **Capture pitch angle**   |  **Capture height**   |   **Capture route**  | \n|:--------:|:--------:|:----------:|:----------:|----------|\n| in-Traj.   | DJI M300+H20t    |  0&deg; or 45&deg;  |  120m   |  Zig-zag flight on a selected region | \n| out-of-Traj.   | DJI Mavic3 Pro  | 30&deg; ~ 60&deg;     | 90m ~ 150m   |  Manually controlled flight on the map   |\n\nIn the revised version of the paper, we will provide clear definitions of “in-Traj.” and “out-of-Traj.,” along with a detailed explanation of the key distinctions between these scenarios.\n\n**Q2: “The proposed method is significantly worse than UAVD4L in \"out-of-place\" part of Swiss-EPFL dataset […]”**\n\n**A2:** Our work aims to explore the use of LoD maps for 6-DoF localization in the air.  We have designed a method specifically tailored for this task and collected two datasets featuring LoD3.0 and LoD2.0 models to validate our algorithm and benefit future research. The experimental results demonstrate that our method exhibits promising performance compared to visual descriptor-based methods.\n\nRegarding the poorer performance in the ‘out-of-place’ part of the Swiss-EPFL dataset, we identify several contributing factors:\n\nFirst, the image quality used to train our network in the Swiss-EPFL dataset is relatively low, as shown in “Figure 14: Samples of training dataset” in the **Appendix**. We observe that model trained on UAVD4L dataset (even with LoD3.0) using high-quality image, performs better than model trained directly on Swiss-EPFL dataset, as indicated in the last two rows of \"Table 12: Cross-scene generalization\" in the **Appendix**.\n\nSecond, we find that using LoD3.0 maps results in better localization accuracy compared to LoD2.0. This is intuitive because our method relies on wireframe representations, and the richer lines in LoD3.0 maps lead to improved results. These results give us insight that for LoD2.0 or lower LoD models, regions could be used as cues for localization in future work.\n\nThird, it is noteworthy that our results based on the LoD2.0 model are still quite good. In fact, in the \"in-place\" part, our method even outperforms SOTA textured-based methods.\n\n**Q3: “Do you have any process to deal with the sky pixels in query images?”**\n\n**A3:** Our method employs pose as supervision and trains the network to extract image features in an end-to-end manner. We do not specifically process the sky pixels in the query images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hVPHCOag7", "reviewer": "Reviewer_nkE1", "summary": "This paper introduces a method for visual localization of Unmanned Aerial Vehicles (UAVs) using a Level-of-Detail (LoD) 3D map and neural wireframe alignment.  The UAV sensor provides coarse pose estimation. Then, LoD-Loc hierarchically builds a cost volume for uniformly-sampled pose hypotheses to describe pose probability distribution.  A pose with maximum probability is selected as the correct one. Each cost within this volume measures the degree of line alignment  between projected and predicted wireframes. It also refines the previous result with a differentiable Gaussian-Newton method. The paper presents an approach to aerial visual localization with a strong emphasis on practical applicability and privacy considerations.", "questions": "Is it possible that the method's performance might degrade in environments with poor GPS signal or where the UAV is operating in areas without prior map knowledge?", "limitations": "1. The method assumes the availability of a gravity direction and location prior, which might not always be accurate or available, especially in GPS-denied environments.\n\n2. The generalizability of the method to other types of UAVs or different environmental conditions is not fully explored in the paper.\n\n3. The paper does not discuss the potential impact of dynamic elements in the environment (such as moving vehicles or people) on pose estimation accuracy.\n\n4. The robustness of the method against various weather conditions or times of day is not addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work gives an approach to aerial visual localization that does not require complex 3D representations.\n\n2. The method uses LoD 3D maps, which are easier to acquire and maintain, have a smaller footprint.\n\n3. it contributes new datasets for the research community to use for training and evaluation.", "weaknesses": "1. The importance of the major contribution of \"Level-of-Detail\" is not clearly described. Acctually, Level-of-Detail has been widely investigated. \n\n2. The paper does not explicitly mention the computational efficiency or resource requirements, which could be a concern for real-time applications on UAVs with limited processing power.\n\n3. The reliance on a coarse pose provided by the UAV sensor as an initial input may affect the method's robustness in scenarios where the initial pose is significantly off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “The importance of the major contribution of \"Level-of-Detail\" is not clearly described. Actually, Level-of-Detail has been widely investigated.”**\n\n**A1:** The LoD model is a well-established concept used in building reconstruction and design to manage the complexity of 3D models. Compared to Structure-from-Motion (SfM) and textured mesh models, LoD models offer several advantages, including ease of acquisition and maintenance, lightweight map size, and enhanced privacy preservation and policy compliance. While we acknowledge that LoD has been extensively investigated in these fields, previous research has not leveraged this model to guide 6-DoF visual localization. One of the key contributions of our paper is the first use of LoD 3D maps for 6-DoF aerial visual localization.\n\n**Q2: “The paper does not explicitly mention the computational efficiency or resource requirements.”**\n\n**A2:** Please refer to Table 3 in the **PDF** of the global response.\n\n**Q3: “The reliance on a coarse pose provided by the UAV sensor […] Is it possible that the method's performance might degrade in environments with poor GPS signal or where the UAV is operating in areas without prior map knowledge?”**\n\n**A3:**  As indicated in “Table 11: Impact of the initial pose for LoD-Loc,” our method’s performance indeed deteriorates with declining GPS quality. This phenomenon is also discussed in the Limitations section of our paper.\n\nHowever, it is noteworthy that we utilize **real** built-in sensors from the UAV as priors. We believe that employing such priors to guide localization is reasonable in practical scenarios. Additionally, many previous studies have also used GPS and gravity direction as priors, such as [1,2,3,4,5] for GPS-aided localization and [1,2,3,5,6,7,8,9,10,11] for gravity direction-aided localization.\n\nIn cases where GPS is denied, we propose that employing an RGB-to-LoD cross-modal retrieval or utilizing the UAV’s sequence motion information to find the initial position could effectively address this issue.\n\nSimilar to [12,13,14], our method relies on 3D maps for localization. Without a map, it is impossible to determine the UAV’s absolute pose; only the relative motion of the UAV sequence can be known.\n\n**Q4: “The method assumes the availability of a gravity direction and location prior, which might not always be accurate or available, especially in GPS-denied environments.”**\n\n**A4:** Please refer to **A3**.\n\n**Q5: “The generalizability of the method to other types of UAVs or different environmental conditions is not fully explored in the paper.”**\n\n**A5:** In fact, we conducted localization using various types of UAVs across different environments. In the UAVD4L-LoD dataset, we captured scenarios labeled as “in-traj.” and “out-of-traj.” as detailed in the table below:\n\n| **Name** | **Capture device**  | **Capture pitch angle**   |  **Capture height**   |   **Capture route**  | \n|:--------:|:--------:|:----------:|:----------:|----------|\n| in-Traj.   | DJI M300+H20t    |  0&deg; or 45&deg;  |  120m   |  Zig-zag flight on a selected region | \n| out-of-Traj.   | DJI Mavic3 Pro  | 30&deg; ~ 60&deg;     | 90m ~ 150m   |  Manually controlled flight on the map   |\n\nIn the EPFL-Swiss dataset, the DJI Phantom 4 RTK is used to collect data from both “in-Place” and “out-of-Place” regions. For more details, please refer to [15].\n\n**Q6: “The paper does not discuss the potential impact of dynamic elements in the environment (such as moving vehicles or people) on pose estimation accuracy.”**\n\n**A6:** Theoretically, our method uses the building's wireframe as a cue for localization, thus dynamic elements in the environment have little impact on our algorithm. We will incorporate a discussion on this aspect in the revised paper.\n\n**Q7: “The robustness of the method against various weather conditions or times of day is not addressed.”**\n\n**A7:** As demonstrated in the **video demo** in the **Supplementary Material**, our method is capable of performing localization at night using thermal cameras. Besides, we are in the process of collecting more data to validate the robustness of our method, leveraging the scalable solution we provide for capturing query data and ground truth annotations. More data under various weather conditions or times of day will be incorporated in our project to further present the robustness of our method.\n\n**Reference**\n\n[1] yan et al., Long-term Visual Localization with Mobile Sensors, CVPR 2023\n\n[2] Sarlin et al., OrienterNet: Visual Localization in 2D Public Maps with Neural Matching, CVPR 2023\n\n[3] Sarlin et al., SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding, NeurIPS 2023\n\n[4] Zeisl et al., Camera pose voting for large-scale image-based localization, ICCV 2015\n\n[5] Lynen et al., Large-scale, real-time visual-inertial localization revisited, IJRR 2020\n\n[6] Albl et al., Rolling shutter absolute pose problem with known vertical direction, CVPR  2016\n\n[7] Kukelova et al., Closed- form solutions to minimal absolute pose problems with known vertical direction, ACCV 2010\n\n[8] Svärm et al., City-scale localization for cameras with known vertical direction, T-PAMI 2016\n\n[9] Sweeney et al., Efficient computation of absolute pose for gravity-aware augmented reality , ISMAR  2015\n\n[10] Fragoso et al., gdls*: Gen- eralized pose-and-scale estimation given scale and gravity priors, CVPR 2020\n\n[11] Sweeney et al., gdls: A scalable solution to the generalized pose and scale problem, ECCV  2014\n\n[12] Sattler et al., Benchmarking 6dof outdoor visual localization in changing conditions, CVPR 2018\n\n[13] Sarlin et al., From Coarse to Fine: Robust Hierarchical Localization at Large Scale, CVPR 2019\n\n[14] Miao et al., A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation, IEEE Transactions on Intelligent Vehicles, 2024 \n\n[15]Yan et al., Crossloc: Scalable aerial localization assisted by multimodal synthetic data, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4b8hRatnbk", "reviewer": "Reviewer_H1Dr", "summary": "The paper presents a localization algorithm using LoD maps. Compared to conventional localization pipelines using SfM / SLAM maps, LoD maps are memory efficient and can offer privacy preservation. Nevertheless, due to the lack of texture in LoD maps, it is not straightforward to use pre-existing localization pipelines for LoD map localization. The paper thus presents a localization pipeline that can handle such ambiguities, by presenting a pose search / refinement pipeline based on estimating wireframe probabilities from images. Specifically, the paper suggests training CNNs to predict wireframe probabilities from images, and measures the geometric compatibility of the predicted probabilities against the LoD map. Experiments show that the proposed method can largely outperform existing CAD-based localization methods, while showing competitive performance against methods conventional methods using textured meshes and local feature descriptors.", "questions": "I listed some minor questions below:\n1. What are the units of delta_x, y, z in Table 11?\n2. Related to Table 11, I feel it will be nice to see more experiments on the impact of the initial pose. For example, what errors would we get if we initialize from place recognition algorithms for UAVs [1]?\n\n\n[References]\n1. Keetha et al., AnyLoc: Towards Universal Visual Place Recognition", "limitations": "Yes the limitations are discussed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of localizing LoD maps seems to be a practical yet quite under-explored problem. I find the problem setup to be very interesting and can foster future works in the direction. While the performance is currently slightly below visual descriptor-based methods, I find the initial results to be promising.\n\n2. The idea of predicting wireframe probabilities is novel and well-motivated. It seems natural to consider such structures to be extracted from 2D images to make valid comparisons against the wireframe map.\n\n3. The writing is very clear, and the supplementary materials section is very thorough which I expect will clarify a large body of details that readers may be interested in.", "weaknesses": "I don't have large concerns for the paper, but below are a few suggestions:\n1. Several related works seem to be missing, including some recent papers on descriptor-free visual localization:\n\na) Wang et al., DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching, arXiv 2024\nb) Kim et al., Fully Geometric Panoramic Localization, CVPR 2024\nc) Pietrantoni et al., SegLoc: Learning Segmentation-based Representations for Privacy-Preserving Visual Localization, CVPR 2023.\n\n2. I wonder how the proposed method could be extended for larger-scale pose search scenarios, for example if GPS / gravity direction is unknown. I feel the search space might be too large for the current grid-based search to work. Do the authors have any rough suggestions for how the work could be extended in such directions?\n\n3. How would be proposed method compare against explicitly extracting line segments from 2D images? To be more specific, suppose one replaces the wireframe probability extraction part explained in Section 3.1 with distance transformed calculated for 2D line extractions using conventional line extractors such as LSD or DeepLSD[1]. The cost functions or features in Equations 2 and 4 could be replaced with  the distance function values, similar to line-based localization methods [2, 3]. I feel this would not fare well since there would be large number of noisy lines and lines from image textures, but still the comparisons would provide insights into why the feature extraction in Section 3.1 is necessary for good performance. \n\n[References]\n1. Pautrat et al., DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients, ICCV 2023\n2. Micusik et al., Descriptor free visual indoor localization with line segments, CVPR 2015\n3. Kim et al., LDL: Line Distance Functions for Panoramic Localization, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “Several related works seem to be missing, including some recent papers on descriptor-free visual localization. […]”**\n\n**A1:** Thank you for your suggestion. We will incorporate the related works in the revised paper.\n\n**Q2: ”I wonder how the proposed method could be extended for larger-scale pose search scenarios, for example if GPS / gravity direction is unknown. […]”**\n\n**A2:** Thank you for your question. Possible solutions could include utilizing an RGB-to-LoD cross-modal retrieval network to determine the initial pose or leveraging the UAV’s sequence motion information as a prior. These strategies may effectively address the issue.\n\nAdditionally, it is noteworthy that we utilize **real** built-in sensors from the UAV as priors. We believe that employing such priors to guide localization is reasonable in practical scenarios. As we know, many previous studies have used GPS and gravity direction as priors, such as [1,2,3,4,5] for GPS-based localization and [1,2,3,5,6,7,8,9,10,11] for gravity direction-based localization.\n\n**Q3: “How would be proposed method compare against explicitly extracting line segments from 2D images? […]”**\n\n**A3:** Thank you for your suggestion. We believe this is a highly valuable experiment to demonstrate the effectiveness of our method. As you suggested, we have added additional experiments named **Explicit Line Alignment**, as shown in Tables 1 and 2 of the **PDF** in the global response and the Table below.\n\n| Method                  | UAVD4L-LoD            |                  | Swiss-EPFL              |                   |\n| -----------------------| ----------------- | ---------------- | ----------------- | ----------------- |\n|                         | in-Traj            | out-of-Traj          | in-Place           | out-of-Place          |\n| Explicit Line Alignment | 10.41/16.21/24.19 | 6.93/12.64/21.62 | 11.37/21.35/33.57 | 18.99/31.39/45.91 |\n| Ours | 84.41/91.77/96.95  | 95.94/99.00/99.36 |  48.60/65.31/79.78  | 37.73/57.26/77.57   |\n\nSpecifically, we extract explicit line segments using DeepLSD[12] and construct a distance field for each segment. We then replace the cost function in Equations 2 and 4 with the distance function values and solve for the pose using coarse-to-fine pose selection and Gauss-Newton refinement. The results indicate that our method surpasses **Explicit Line Alignment** by a significant margin, proving that the feature extraction in Section 3.1 is necessary for good performance. This experiment will be included in the revised paper.\n\n**Q4: “What are the units of delta_x, y, z in Table 11?”**\n\n**A4:** The units for delta_x, y, and z are **meters**. We will also include this note in caption of Table 11 in the revised paper.\n\n\n**Q5: \"Related to Table 11, I feel it will be nice to see more experiments on the impact of the initial pose. For example, what errors would we get if we initialize from place recognition algorithms for UAVs [13]?\"**\n\n**A5:** Thank you for your suggestion. We agree that initialization from place recognition algorithms for UAVs could be a reasonable solution for GPS-denied environments, and it warrants further experimentation. However, our method aims to localize the UAV solely using the LoD model, whereas conventional place recognition algorithms such as [13] rely on rich-information maps (i.e., satellite images), which are not available in our dataset. In future work, we aim to train a global network with the capability of RGB-to-LoD retrieval and test our algorithm with this initial pose.\n\n**Reference**\n\n[1] yan et al., Long-term Visual Localization with Mobile Sensors, CVPR 2023\n\n[2] Sarlin et al., OrienterNet: Visual Localization in 2D Public Maps with Neural Matching, CVPR 2023\n\n[3] Sarlin et al., SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding, NeurIPS2023\n\n[4] Zeisl et al., Camera pose voting for large-scale image-based localization, ICCV 2015\n\n[5] Lynen et al., Large-scale, real-time visual-inertial localization revisited, IJRR 2020\n\n[6] Albl et al., Rolling shutter absolute pose problem with known vertical direction, CVPR  2016\n\n[7] Kukelova et al., Closed- form solutions to minimal absolute pose problems with known vertical direction, ACCV 2010\n\n[8] Svärm et al., City-scale localization for cameras with known vertical direction, T-PAMI 2016\n\n[9] Sweeney et al., Efficient computation of absolute pose for gravity-aware augmented reality , ISMAR  2015\n\n[10] Fragoso et al., gdls*: Gen- eralized pose-and-scale estimation given scale and gravity priors, CVPR 2020\n\n[11] Sweeney et al., gdls: A scalable solution to the generalized pose and scale problem, ECCV  2014\n\n[12] Pautrat et al., DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients, ICCV 2023\n\n[13] Keetha et al., AnyLoc: Towards Universal Visual Place Recognition, IEEE Robotics and Automation Letters 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OF5nqS3SB6", "reviewer": "Reviewer_CDoh", "summary": "This paper proposes a hierarchical scheme for pose selection to progressively compute high-quality pose in a coarse-to-fine fashion.", "questions": "This paper give a detailed description of some proposed steps, without providing the target of these steps.  As far as I am concerned, these steps are general steps for localization, what's your innovation?\nThe problem you want to solve and the shortcomings that current methods exist should be strengthened.\nThe quality of some figures can be improved.", "limitations": "As described above", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The scheme of the proposed method is easily understood and the paper shows good experimental results.", "weaknesses": "The innovation of the paper is not clearly described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: “As far as I am concerned, these steps are general steps for localization, what's your innovation?”**\n\n**A1:**\nWe believe our innovation could be summarized in the following parts: \n\n**Task:** We establish a novel task that utilizes Level-of-Detail (LoD) 3D maps for 6-DoF aerial visual localization, which offers advantages such as ease acquisition and maintenance, lightweight map size, and enhanced privacy preservation and policy compliance. The task has been recognized as \"**practical**\" by reviewers RdXx, UAbv, nkE1 and H1Dr. \n\n**Method:** We propose a novel method tailored for this task, which utilizes wireframe alignment for pose estimation. This approach is distinct from previous research that typically solves for the pose via descriptor-based matching and PnP RANSAC solver. Additionally, our method integrates several modules, such as coarse-to-fine pose cost volume reconstruction, uncertainty sampling range estimation, and differential Gauss-Newton refinement into a whole network, which can be trained in an end-to-end manner with only pose supervision. Our method has been admired by other reviewers, noted as \"**novel**\" from UAbv and H1Dr, \"**interesting**\" from RdXx and H1Dr, and \"**privacy-protecting**\" from H1Dr and nkE1.\n\n**Dataset:** We have release two LoD city datasets featuring LoD2.0 and LoD3.0 models, complete with RGB queries and ground-truth pose annotations. Our dataset has been recognized as \"**useful**\" by reviewers  RdXx, UAbv and nkE1, and is noted to “**provide interesting resources for further research and benchmarking in aerial visual localization**” by RdXx.\n\n**Result:** We have achieved \"**promising**\" results as noted by reviewers UAbv and H1Dr, demonstrating comparable or better performance with state-of-the-art texture-based approaches, while only using simplified LoD models to guide the localization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our sincere thanks to all reviewers for their insightful feedback. We are honored that our work has been recognized as "**practical**" (RdXx, UAbv, nkE1, H1Dr), our dataset as "**useful**" (RdXx, UAbv, nkE1), our method as "**novel**" (UAbv, H1Dr), "**interesting**" (RdXx, H1Dr), and "**privacy-protecting**" (H1Dr, nkE1). Additionally, our results have been described as "**promising**" (UAbv, H1Dr, CDoh), and our writing as "**clear**" (H1Dr, CDoh). We hope that our work indeed "**provides interesting resources for further research in aerial visual localization**" (RdXx) and "**can foster future works in this direction**" (H1Dr).

We express our gratitude to the reviewers for acknowledging our work, which is the first effort to utilize LoD 3D maps for  aerial 6-DoF localization. Inspired by your helpful comments, we incorporated the following changes into the next/final revision of our paper:

- **Additional Baseline Experiments:** We conducted further baseline experiments using RoMA[1] and e-Loftr[2] within the CadLoc and UAVD4L frameworks. Explicit line segment localization experiments have also been included for comparative analysis (refer to Table 1 and 2 in the attached PDF).
- **Computational Cost Comparison:** We added a comprehensive comparison of computational costs and resource requirements to offer more comprehensive research (refer to Table 3 in the attached PDF).
- **Clarification of Terms:** We provided the definition of "in-Traj." and "out-of-Traj." along with a detailed explanation of the key differences between them (see Table 4 of the attaches PDF).
- **Enhanced Explanations:** We updated our manuscript to include more explanations for their experiment results, moving beyond mere presentation.
- **Manuscript Corrections:** We updated our manuscript to fix minor mistakes and reduce the risk of misunderstandings.

Reference：

[1] Edstedt et al., RoMa: Robust Dense Feature Matching, CVPR 2024

[2] Wang et al., Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed, CVPR 2024

Meta-review: This paper received a mixed recommendation (54568), where a majority of the reviewers voted for acceptance.
The major concern from the only negative reviewers is related to experimental settings, the authors provided a rebuttal and justifications, while this reviewer has no response. The AC has checked the comments and paper carefully and found that the comments of the positive reviewers are more convincing, thus recommending an acceptance.
Decision: Accept (poster)