LoD-Loc: Aerial Visual Localization using LoD 3D Map with Neural Wireframe Alignment

Authors: juelin  zhu
shen  yan
long  wang
yu  liu
maojun  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 04:41:38

Abstract: We propose a new method named LoD-Loc for visual localization in the air. Unlike existing localization algorithms, LoD-Loc does not rely on complex 3D representations and can estimate the pose of an Unmanned Aerial Vehicle (UAV) using a Level-of-Detail (LoD) 3D map. LoD-Loc mainly achieves this goal by aligning the wireframe derived from the LoD projected model with that predicted by the neural network. Specifically, given a coarse pose provided by the UAV sensor, LoD-Loc hierarchically builds a cost volume for uniformly sampled pose hypotheses to describe pose probability distribution and select a pose with maximum probability. Each cost within this volume measures the degree of line alignment between projected and predicted wireframes. LoD-Loc also devises a 6-DoF pose optimization algorithm to refine the previous result with a differentiable Gaussian-Newton method. As no public dataset exists for the studied problem, we collect two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries and ground-truth pose annotations. We benchmark our method and demonstrate that LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. The code and dataset will be made available upon publication.

Main paper: 
 introduction: 
 Aerial visual localization is the process of determining the global position and orientation for a UAV camera relative to a known map. This process benefits many important applications, ranging from cargo transport [69] , surveillance [19, 72] , to search and rescue [15, 63] .Following localization algorithms on the ground [18, 30, 42, 53, 54, 57, 67, 68, 70, 77] , current aerial visual localization approaches [22, 72] typically involve matching pixels in a query image with points in a pre-built high-quality 3D map, which is often derived from 3D texture models [43, 76, 72] . Subsequently, a Perspective-n-Point (PnP) RANSAC [32, 37, 14, 23, 27, 24] technique is commonly used to calculate the camera pose. However, building high-quality 3D maps using photogrammetry [12, 28, 64, 58, 33] is expensive on a global scale and requires frequent updates to account for temporal changes in visual appearance. Besides, these 3D maps are costly to store, which poses significant challenges for terminal deployment on drones. Furthermore, high-resolution 3D maps disclose detailed information about the localization area, raising critical concerns regarding homeland security and privacy preservation. by a LoD 3D map, characterized by its ease of acquisition, lightweight nature, and built-in privacypreserving capabilities. Given a query image and its coarse sensor pose, our method utilizes the wireframe alignment of LoD models to recover the camera pose.To address these challenges, we propose to leverage the Level of Detail (LoD) 3D city maps [38, 16] as the cue for localization, as illustrated in Figure 1 . Compared with traditional textured 3D models, LoD 3D models enjoy the following advantages: 1) Ease of acquisition and maintenance: World-scale LoD city models can be generated with the rapid development of remote sensing [17, 34, 48, 52, 31] . Many commercial companies, such as Google Maps [8] and Baidu Maps [7] , have already integrated LoD 3D models into their MAP Applications. 2) Light-weight map size: LoD maps are extremely compact, up to 10 4 times smaller in size than textured 3D maps, enabling on-device localization over large areas. 3) Privacy preservation and policy-friendly: As LoD city models only reveal the basic 3D outlines of buildings in a highly abstracted and simplified manner, they raise fewer concerns about the disclosure of privacy and land resource secrets.However, compared with a textured 3D model, using LoD maps for localization is very challenging, primarily due to the lack of texture and detail. This deficiency makes it nearly impossible to establish local feature-based 2D-to-3D correspondences. Inspired by the idea that, when the pose is correctly solved, the network-predicted building wireframes can align with those projected from the LoD 3D model, as shown in Figure 1 , we introduce LoD-Loc, a novel approach tailored for visual localization in LoD 3D city maps. Our method takes a query image and its real sensor data (i.e., GPS, gravity, and compass) as input, and estimates the 6-DoF pose of a drone in flight. Specifically, we initially fix the 2-DoF gravity direction and generate pose hypotheses by sampling 4-DoF (comprising position and yaw angle) around the sensor pose, given that the gravity direction provided by the inertial unit exhibits minor error. Following the generation of pose hypotheses, LoD building wireframes are projected onto the query image plane. Each pose hypothesis is then scored based on the alignment between the projected and predicted wireframes, thereby forming a 4D pose cost volume. By applying a softmax operation, we derive a probability density over the pose, which can be used for pose selection through classification. Moreover, after the pose selection stage, a differentiable Gauss-Newton method, with an optimization objective to maximize the wireframe alignment, is employed to refine the overall 6-DoF pose. The pose selection and optimization processes are fully differentiable w.r.t. the network output, which enables the use of ground-truth poses as supervision for training feature extraction and pose estimation in an end-to-end manner.To achieve high accuracy and low memory usage, we propose a hierarchical scheme for pose selection that utilizes multiple small pose volumes, to progressively compute poses in a coarse-to-fine manner. Throughout the hierarchy, we adopt an adaptive sampling strategy, where the variance-based uncertainty from the previous stage influences the sampling range of the next stage for constructing pose cost volume. This adaptive process enables reasonable and fine-grained spatial partitioning of poses, resulting in a significant improvement in the final pose output.To facilitate research in this area, and to train and evaluate our method, we release two datasets with map levels of LoD3.0 and LoD2.0, respectively, as shown in Figure 2 . For the LoD3.0 dataset, we employ a semi-automatic method to generate LoD model data from a recent large-scale oblique photography scene [72] , covering an area of 2.5 square kilometers. The query images are captured by drones, with sensor data (e.g., GPS, IMU) recorded. For the LOD2.0 dataset, we use LoD model data provided by the Swiss federal authorities, specifically the SwissTOPO [9] [10] [11] data near École We conduct extensive experiments on these two datasets. The results show that, due to the lack of both color and texture in the LoD 3D model, previous state-of-the-art image retrieval-and-matching methods [44, 53, 54, 29, 41, 25, 66, 46, 47] basically fail. In contrast, our method consistently achieves excellent results, even surpassing current state-of-the-art methods [53, 54, 66, 72] that use textured 3D models for localization.• We propose the use of Level of Detail (LoD) 3D maps for 6-DoF visual localization in the air.• We introduce a novel localization method that utilizes wireframe alignment for pose estimation.• Our method is differentiable, allowing the pipeline to be trained end-to-end with pose supervision.• We release two LoD city datasets, complete with RGB queries and ground-truth pose annotations. 
related works: 
 Localization from SfM or Mesh Map. SfM maps typically consist of reference images and 3D track points with their associated features [58] . For a given query image, an image retrieval method [13, 29] is initially utilized to identify co-visible reference images. Following this, feature matching algorithms [41, 25, 54, 66] are employed to establish accurate 2D-2D correspondences between the query image and the identified reference images, with track information being used to transform these 2D-2D correspondences into 2D-3D relationships. Finally, the pose is resolved using PnP RANSAC [32, 37, 14, 23, 27, 24, 77] .Mesh maps are typically defined by a textured mesh model. Initially, reference images with depth are rendered at appropriate viewpoints surrounding the model [72, 43, 44, 76] . Similar image retrieval [13, 29] and matching [41, 25, 54, 66] processes are employed to identify co-visible images and to establish 2D-2D correspondences. The depth map is utilized to transform 2D-2D correspondences into 2D-3D relationships, and the pose is subsequently determined by a PnP RANSAC [32, 37, 14, 23, 27, 24, 77] .Despite providing high-accuracy localization results, both SfM and mesh models present significant challenges in terms of reconstruction and maintenance. Additionally, their extensive size complicates deployment, necessitating its existence solely in the cloud. Moreover, these maps raise serious concerns about the privacy of personal and land resource leakage.Localization from other types of Map. To mitigate these issues, researchers have proposed to use alternative types of maps for localization. In addressing the difficulty of map reconstruction and maintenance, some methods opt for overhead imagery such as satellite [59, 60, 73, 56] , or leverage OpenStreetMap [55] as a reference. However, these methods are limited to estimating a 3-DoF (planar position and heading) pose at most. To address the issue of large map size, some methods have made attempts to compress the maps [20, 21, 78] , reduce model complexity [44] , or utilize geometry information without features [49, 79, 39, 75] . In the context of privacy, some methods [36, 61, 65] propose to transform 3D point clouds into 3D line clouds, leverage semantic information from point clouds, or utilize semantic 3D maps to enhance privacy [50] . Some other methods apply learning-based pose regression or scene point regression models [35, 62, 40] that do not explicitly store the 3D map. However, the effectiveness and generalizability of these methods are often inferior to those that rely on SfM or texture mesh maps. A detailed comparison of attributes from different maps is provided in Table 1 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: It can be found in the end of the Conclusion Section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Given a 3D city LoD map M, a query image I, and its coarse sensor pose ξ p , the goal of the proposed method is to compute the absolute 6-DoF pose ξ * . First, a convolutional neural network is used to extract the wireframe probability map for the query image I at multiple levels (Sec. 3.1). Second, at each level, uniform pose sampling and 3D wireframe projection are employed to build a cost volume for various pose hypotheses, describing the pose probability distribution. The pose with the maximum probability is then selected (Sec. 3.2). Finally, a post-processing network refines the wireframe probability map after the last level, and a Gauss-Newton method is applied to refine the pose chosen in the previous stage (Sec. 3.3). Figure 3 provides an overview of the proposed method.We use a standard convolutional architecture with U-Net [51] to extract multi-level features from the query image I. Different from previous works that maintain a high-dimensional feature map to encapsulate rich visual information for each level, we abstract and reduce the feature map dimension to a single channel, where each pixel in this map signifies the likelihood of being a wireframe. The resulting feature maps are denoted by F l ∈ R H l ×W l ×1 , where l = {1, 2, 3} is the level index. More details on the architecture of the proposed network can be found in Appendix D.1.After feature extraction, we construct a cost volume based on various pose hypotheses sampled around the coarse sensor pose, selecting the pose with the highest probability at each level. To ensure efficient sampling, we utilize the uncertainty in pose selection at the current level to determine the pose sampling range for the next level.Pose cost volume reconstruction. This subsection explains how to construct the pose cost volume.For a specific level l, the initial pose is represented by ξ l , and the computed pose is denoted as ξ l . As part of a progressive process, we keep ξ 1 = ξ p and ξ l+1 = ξ l . The pose ξ l can be decoupled into six degrees, with ξ l = (x l , y l , z l , θ l , φ l , ψ l ), where (x l , y l , z l ) represents the translation in 3D space while (θ l , φ l , ψ l ) refers to the Eular angles (i.e., yaw, pitch, roll). Since the pitch and roll (φ p , ψ p ) of the gravity direction from the sensor pose data exhibit high accuracy, we fix (φ l , ψ l ) = (φ p , ψ p ) and only conduct operations on the remaining 4-DoF (i.e., (x l , y l , z l , θ l ) ) in the following steps.Specifically, we begin by uniformly sampling 4-DoF poses centered on the initial pose ξ l , with the sampling range and number defined as r l = [r l (x), r l (y), r l (z), r l (θ)] and [m l (x), m l (y), m l (z), m l (θ)], respectively. The pose hypothesis {ξ hyp l (d)} is generated along (x, y, z, θ) directions separately, where d ∈ (x, y, z, θ).EQUATIONNext, for a given pose hypothesis, denoted asξ hyp l = (R hyp l , t hyp l) and a set of discrete 3D wireframe points denoted as {P i }, we define a line alignment cost:EQUATIONIn this equation,p i = Π(R hyp l • P i + t hyp l) represents the projection of 3D point P i under pose hypothesis ξ hyp l and [•] denotes a lookup with sub-pixel interpolation. The construction of the 3D wireframe points {P i } is provided in the next paragraph. By combining these costs in a grid manner across four distinct dimensions (x, y, z, θ), we obtain a pose cost volume C l with dimensions [m l (x) × m l (y) × m l (z) × m l (θ)]. Finally, a softmax function is applied to C l to yield a probability distribution volume P l . For pose inference, we select the pose ξ l with maximum probability by argmax operation upon P l .Discrete 3D wireframe points generation. For a query image I and its associated sensor pose ξ p , we describe how to sample and identify discrete 3D wireframe points {P i } across the entire LoD map M. Assume the LoD map M is characterized by a number of faces with vertices V j = [X j , Y j , Z j ] T ∈ R 3 . We derive each line of the LoD model as ℓ jk = [(X j , X k ), (Y j , Y k ), (Z j , Z k )] by connecting vertices V j and V k . To focus on distinct geometric structures such as building edges, we discard lines whose normals of their neighboring faces exhibit a significant difference, larger than µ = 10 degrees. The line simplification process is facilitated with the assistance of Blender [1] . Subsequently, we set the sampling density δ in meters and uniformly sample points along all simplified lines {ℓ jk } to obtain 3D points as {P u i }, where i represents the index of the 3D points. To obtain visible 3D points for a query image I, we use the pinhole camera projection (with intrinsic matrix K p ) and pose prior ξ p = (R p , t p ) to identify 3D points, taking factors such as frustum inside and occlusion information into account. In particular, we first project 3D points onto the 2D image plane:EQUATIONwhere[X i , Y i , Z i ] ∈ {P u i }, and d is the projected depth for point [X i , Y i , Z i ].We then render a depth map from the LoD map M from pose ξ p , denoted as D. A boolean mask is calculated as:EQUATIONwhere H and W denote the size of the image I, D(u i , v i ) means the interpolating value on depth map D at (u i , v i ). The final discrete visible 3D wireframe points {P i } can be obtained using Eq. 5:EQUATIONUncertainty sampling range estimation. During the coarse-to-fine process, we leverage the pose selection uncertainty from the previous level to determine the sampling range of the current level. This strategy allows us to progressively subdivide the pose sampling space, thereby enhancing the precision of pose selection. More specifically, for l = 1, we define the pose sampling range by evaluating the error in the coarse sensor pose. The sampling range for (x, y, z, θ) is defined as [r 1 (x), r 1 (y), r 1 (z), r 1 (θ)] = [r p (x), r p (y), r p (z), r p (θ)]. For l = {2, 3}, we employ the variance of the probability distribution volume P l-1 at l -1 to determine the pose sampling range r l .In particular, since the pose hypothesis {ξ hyp l }, pose cost volume C l and probability distribution volume P l share the same data structure, we flatten them and index them by t. The variance v l at level l is calculated as:EQUATIONHere, the symbol ⊖ represents the subtraction operation separately applied to the (x, y, z, θ) directions.The corresponding standard deviation is computed as σ l = √ v l . We compute the pose sampling range as r l = 2λ • σ l , where λ is a hyperparameter that adjust the length of the sample range. A visualization of this uncertainty sampling range estimation process can be found in Figure 4 .Based on the selected pose ξ 3 from the previous stage, we use a refined wireframe probability map F rf , which is further extracted from the feature map F 3 via a post-processing convolutional network, we optimize the pose ξ * = (R * , t * ) so as to align the 3D wireframe with the 2D predicted wireframe. Specifically, we define the objective function:EQUATIONwhere P i is the 3D wireframe point and Π is the projection operation. Minimizing this function aligns the projected 3D wireframe points with the 2D locations that have a higher predicted probability. The pose update formula for ξ * , as derived from the Gauss-Newton method, is given by:EQUATIONIn this formula, ∆ξ ∈ R 6 represents a six-dimensional transformation vector, where ∆ξ r ∈ R 3 constitutes the rotational component and ∆ξ t ∈ R 3 represents the translational component. We transform the rotational component ∆ξ r into a 3×3 rotation matrix by the exponential map of the Lie algebra so(3). Besides, J i represents the Jacobian matrix of the residual function f i with respect to the pose parameters. A comprehensive explanation and detailed implementation of the Jacobian computation are provided in Appendix D.2.We employ two separate loss functions to facilitate end-to-end training of the pose selection procedure (Sec. 3.2) and the pose refinement module (Sec. 3.3). For the pose selection module, we minimize the negative log-likelihood loss on the probability distribution volume P l at three levels, where l ∈ {1, 2, 3}.EQUATIONFor the pose refinement process, the training involves minimizing the reprojection errors between 3D wireframe points transformed by the estimated pose ξ * and the ground truth pose ξ = (R, t):EQUATIONwhere ρ represents the Huber robust kernel.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: The code and dataset will be made available upon publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiment: 
 Extensive experiments are conducted on the UAVD4L-LoD and Swiss-EPFL datasets to demonstrate the effectiveness of our proposed model as described in Sec. 4.2. Additionally, ablation studies are conducted on the UAVD4L-LoD dataset in Sec. 4.3.Datasets. The released datasets consist of two distinct parts, named UAVD4L-LoD and Swiss-EPFL, providing LoD3.0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans an area of 2.5 square kilometers, is generated through a semi-automatic process which produces a 3D LoD map from the mesh model of the UAVD4L [72] dataset. The Swiss-EPFL dataset, which covers an expansive area of 8.18 square kilometers, derives its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [9] [10] [11] . We illustrate the 3D LoD maps and query images of these two datasets in Figure 2 . More details can be found in Appendix A, B and C.Baseline. We compared our approach with two visual localization baselines: UAVD4L [72] , predicated on textured mesh models, and CadLoc [44] , predicated on LoD models, employing diverse feature extractors and matchers. Both baselines employ a keypoint-based strategy: 1) SIFT [41] descriptor with traditional Nearest Neighbor (NN) matching, 2) learning-based extractor SuperPoint (SPP) [25] with graph-based networks Superglue (SPG) [54] , 3) detector-free matcher LoFTR [66] and 4) e-LoFTR [71] , 5) dense feature matcher RoMA [26] . Additionally, considering the line structure of the LoD model, we apply three line-based algorithms for the CadLoc: 6) deep neural network SOLD 2 [45] for joint detection and description of line segments, 7) deep line segment detector DeepLSD [46] with line detector in SOLD 2 , 8) DeepLSD with wireframe-based representation and dual-softmax matching method GlueStick [47] . Further details about the implementation of the baseline experiments can be found in Appendix E.Metrics. We follow the standard localization evaluation procedure [68] and set recall thresholds of (2m, 2 • ), (3m, 3 • ), and (5m, 5 • ). During training, we set a random seed to limit 3D wireframe points {P i } to 2, 000 points, and the pose sampling number m l (x), m l (y), m l (z), m l (θ) for level l = 1, 2, 3 is uniformly set to [13, 7, 3] due to constraints related to CUDA memory. The image size is (512, 480) for the UAVD4L dataset and (720, 480) for the Swiss-EPFL dataset. The pose sampling range at level 1 is set as [10, 10, 30, 7.5] which refers to [r p (x), r p (y), r p (z), r p (θ)]. For the UAVD4L-LoD dataset, we incorporate a subset of synthesized images from UAVD4L [72] , which includes buildings, as training data. For Swiss-EPFL, we train the model by combining synthetic images LHS and real query images from the CrossLoc [74] project, following its data split pattern.During inference, experiments are executed on real query images {I i } derived from two datasets. We make the following changes, the discrete retrieval points from the 3D wireframe are sampled at an interval of 1 meter. The pose sampling number is increased to [m l (x), m l (y), m l (z), m l (θ)] = [10, 10, 30, 8] for all levels. λ is set as 0.8. The training and inference of the entire network are executed using 2 NVIDIA RTX 4090 GPUs. Additionally, we employ four variations to validate the effectiveness of our method. Specifically, 1) -no NWE means no neural wireframe estimation, which extracts explicit line segments using DeepLSD [46] and constructs a distance field for each segment. It then replaces the cost function in Equations 2 and 7 with the distance function values, and solves for the pose using coarse-to-fine pose selection followed by Gauss-Newton refinement; 2) -no USR means a model without uncertainty sampling range estimation (Sec. 3.2); 3) -no Refine denotes a model without pose refinement (Sec. 3.3); and 4) full model is our proposed LoD-Loc.Evaluation over UAVD4L-LoD dataset. As described in Table 2 , our method shows excellent performance, both in the in-Traj. and out-of-Traj. queries. Apart from the 2m -2 • and 3m -3 • metric in the in-Traj queries, which are marginally lower than UAVD4L with RoMA matcher, all other metrics surpass those of contemporary baselines. Note that this comparison is unfair, as baselines reference on a high-precision texture model that is richer in texture and geometry, while we only employ a LoD model. We further compare with CadLoc, which shares the same 3D reference model as ours. However, we observe that regardless of the choice of descriptors (point-based or line-based), these methods perform poorly. We visualize their retrieval and matching failure cases in Appendix E.3. Furthermore, we analyze why our method performs better in the out-of-Traj. scenarios compared to the in-Traj. scenarios in Appendix F.3. Evaluation over Swiss-EPFL dataset. Table 3 presents the inference results on the Swiss-EPFL dataset. CadLoc continues to exhibit widespread failures due to its poor retrieval and matching results across different modalities. We surpass the state-of-the-art UAVD4L method in the in-P lace queries, but fall behind in the out-of -P lace queries. Moreover, the overall results obtained on the Swiss-EPFL dataset are not as strong as those on the UAVD4L-LoD dataset. We attribute this discrepancy to the inferior quality of the images in the training database as explained in Appendix F.1.Besides, the LoD2.0 model provides less structured information, making it harder for pose inferences.Analysis of Methodological Advantages. First, compared to the SOTA texture-based approach, the LoD-Loc employs distinct cues for localization. The texture-based method determines the pose by optimizing the re-projection error of corresponding 2D-3D points. Conversely, the LoD-Loc aligns the 3D wireframe projection to solve the pose. Second, 3D-model-based methods typically employ a two-stage scheme, which involves building 2D-3D matches and then solving the pose with PnP RANSAC. The LoD-Loc method directly solves the pose in an end-to-end manner, potentially leading to better pose accuracy. Third, the LoD-Loc includes several important modules to improve performance, such as coarse-to-fine pose cost volume reconstruction, uncertainty-based sampling range estimation, and differential Gauss-Newton refinement. These factors contribute to the superior performance of our method. We perform ablation experiments on the UAVD4L-LoD dataset, focusing on different levels. More ablation studies are provided in Appendix F.2.Levels. As depicted in Table 4 , we present the results of the ablation experiments in terms of recall, translation errors, and rotation errors. The localization accuracy shows a gradual improvement as the number of levels increases. This demonstrates the effectiveness of the progressive coarse-to-fine estimation and final pose refinement. Figure 5 visualizes feature maps for each level, illustrating that wireframe features extracted from deeper levels is clearer. 
conclusion: 
 This paper presents LoD-Loc, a novel approach for localizing aerial images using a LoD 3D map.Compared to large and expensive 3D maps that existing methods rely on, the LoD map provides a simple, accessible, and privacy-friendly scene representation. With the coarse sensor pose, the proposed LoD-Loc uses a unified pipeline to estimate the camera pose, including a multi-scale feature extractor, pose selection from cost volume, and pose refinement. Furthermore, we contribute two datasets with map levels of LoD3.0 and LoD2.0, along with real RGB queries with ground-truth pose annotation. LoD-Loc achieves excellent performance, even surpassing current state-of-the-art methods that use textured 3D models for localization. We believe LoD-Loc opens new possibilities for visual localization with simple and scalable 3D maps.Limitation. LoD-Loc operates under the assumptions of a known gravity direction and a location prior. While these assumptions are reasonable, they restrict the application of LoD-Loc in environments where GPS is denied or unavailable.Broader impact. This work has implications regarding privacy and surveillance. However, the LoD models represent building structures in a highly abstracted form, which alleviates concerns about the disclosure of personal privacy or land resource information. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: It can be found in Experiment Section 4Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not included due to the high computational cost of calculating them.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: It can be found in Section Experiments 4Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: It is indicated in the Conclusion Section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The data and code used in the paper comply with licensing and usage terms and are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The released datasets consist of two distinct parts, UAVD4L-LoD and Swiss-EPFL, providing LoD3.0 and LoD2.0 models, respectively. The UAVD4L-LoD dataset, which spans an area of 2.5 square kilometers, is derived from a semi-automatic process that generates a 3D LoD map based on the mesh model of the UAVD4L [72] dataset. This dataset includes a diverse array of architectural structures, including skyscrapers, villas, apartment complexes, educational institutions, and rural dwellings. The query images for this dataset were captured using two UAVs equipped with real sensor data: a DJI M300 [3] drone with an H20T [2] camera and a DJI Mavic3 Pro [5] drone. The Swiss-EPFL dataset, which covers an expansive area of 8.18 square kilometers, sources its LoD2.0 models from data made publicly accessible by the Swiss federal authorities [9] [10] [11] . This dataset features a variety of architectural styles, such as libraries, residential apartments, and stadiums. The query images for this dataset were acquired through the CrossLoc [74] projects, using a DJI Phantom 4 RTK [6] drone. Figure 2 presents the 3D LoD maps and query images from these two datasets.The 3D LoD map for the UAVD4L-LoD dataset is generated semi-automatically with the assistance of the DP Modeler tool [4] . The process begins with the automatic generation of building blocks, characterized by their footprints and heights. Manual refinement is then applied to the architectural details of each building, raising them to the LoD3.0 level. The LoD accuracy of the UAVD4L-LoD dataset is consistent with the mesh model derived from UAVD4L.For the Swiss-EPFL dataset, LoD2.0 models are downloaded from the SwissTOPO website [9] [10] [11] .To synchronize the coordinate systems between the map data and the drone-captured data from the CrossLoc dataset (which covers the same area with ground truth pose annotation), we converted the Swiss LoD map data in LV95 and LN02 coordinate systems to the ECEF coordinate system.The query images of the UAVD4L-LoD dataset are divided into two categories: in-Traj. and outof-Traj., representing trajectory-based and free-flight scenarios, respectively. The in-Traj. images, totaling 1, 604, were captured using a DJI M300 drone equipped with an H20T camera, focusing primarily on residential buildings, villas, and educational institutions. In contrast, the out-Traj. images, totaling 2, 192, were captured using a DJI Mavic3 Pro drone, covering a variety of architectural structures such as skyscrapers and rural dwellings. Both the in-Traj. and out-of-Traj. datasets include real sensor priors. Table 5 outlines the specific differences between the in-Traj. and out-Traj. sequences. The real query images in the Swiss-EPFL dataset come from the CrossLoc [74] dataset. However, because the real-time kinematics (RTK) data from the DJI Phantom4 were used directly as ground truth (GT) poses, some GT poses show significant mislabeling. To resolve this issue, we projected the wireframes of LoD maps onto query images to identify and remove incorrectly labeled poses. The final query dataset comprises 2, 254 images.For the UAVD4L-LoD dataset, we employ a semi-automatic annotation approach to generate pseudo-GT poses ξ i for the query images {I q i }. The process is based on the SfM results and textured mesh model of the UAVD4L [72] . First, we perform SfM separately on the query images {I q i } and the reference images {I r i } from the UAVD4L to obtain SfM results C q and C r . Next, we manually in-Traj.out-of-Traj. select points with distinctive visual features (e.g., building corners) as tie points to align C q with C r .To further enhance the accuracy of the pseudo-GT, we utilize the render-and-compare pipeline [76] to refine the final poses ξ i .Additionally, we analyze the discrepancies between the pose prior and the GT pose, decoupling the poses into 3D translation in WGS84 space and Euler angles in terms of 'yaw-pitch-roll'. It is observed that the translation errors for x and y are within ±10, z errors are within ±30, yaw errors are within ±7.5, and pitch and roll errors are approximately 1 degree.In the UAVD4L-LoD dataset, we employed a semi-automatic annotation technique to generate pseudo-GT poses ξ i for the query images {I i }. Initially, we performed SfM separately on the query images {I i } and the reference images {I r j } from UAVD4L, yielding corresponding SfM results C q and C r . Subsequently, based on the capture region of the {I i }, we manually identified e distinctive tie points, such as the corner of the building, to align C q with C r , resulting in C f . We then refined the pose accuracy of C f using Bundle Adjustment. The accuracy of the GT poses was evaluated through the median reprojection error, which was 0.43 pixels for all connected points and 1.19 pixels for the tie points. Finally, we employed a render-and-compare [76] pipeline for the final refinement of the GT poses. In this manner, with the annotation of tens of e = 20 manual tie points, we were able to obtain pseudo-GT poses ξ i for a total of 3, 796 query images {I i }. Figure 6 shows the flight trajectories of the in-T raj. and out-of -T raj.In the UAVD4L-LoD dataset, we conduct a comprehensive data analysis to validate the precision of the sensor pose. This is accomplished by employing absolute error bar charts, as illustrated in Figure 7 . Additionally, we assess the accuracy by projecting wireframe points onto the image plane using both sensor and GT poses. Results of these projections are depicted in Figure 8 .In the Swiss-EPFL dataset, the GT poses ξ i for the query images {I i } are sourced from the CrossLoc project [74] . This project directly acquires RTK data from the DJI Phantom 4 for GT annotation. Considering that the RTK device may introduce some noise, we identified and excluded query images with incorrect labeling. This was accomplished by projecting the wireframe onto the in-Traj. out-of-Traj. image plane and manually discarding the items exhibiting noticeable misalignment. The process is visualized in Figure 9 .Since the CrossLoc [74] project does not provide GPS or other sensor data, we randomly generate sensor poses ξ p by emulating the pose errors derived from the UAVD4L-LoD dataset. Specifically, X-Y for position range between [-10, 10] meters, Z ranges between [-30, 30] meters, yaw for rotation ranges in [-7.5, 7.5] degrees, and pitch-roll range between [-1, 1] degrees. We present the discrepancy between the generated sensor poses and GT poses in a bar chart, as depicted in Figure 11 . In this section, we provide a detailed description of the architecture of the multi-scale feature extractor in Table 6 .The objective function for pose refinement is:EQUATION)We compute the Jacobian matrix of the residual function f i with respect to the pose parameters as followed:EQUATION)Figure 9 : Samples of mislabeled and selected query images over Swiss-EPFL dataset. We eliminate mislabeled query images by manually identifying the alignment between the projected 2D wireframe and the corresponding RGB image.where∂F rf∂pi is the gradient of the feature map F rf at the 2D location p i andEQUATION)Besides, P cam i is the point which transformed to the camera space. To compute the last derivative of Eq. 12, we add a perturbation ∆ξ to the transformation:EQUATIONFinally, the derivatives w.r.t the translation component and rotation component are:EQUATIONwhere [] × is the skew-symmetric matrix.For baselines, a retrieval-and-matching process is used upon the reference images in the dataset.To ensure a fair comparison, we apply the sensor poses to guide the image retrieval process for UAVD4L [72] and Cad-Loc [44] . For each query image I, we narrow the retrieval candidates q I using Eq. 16. experiments are conducted on the UAVD4L dataset. In these experiments, we hypothesize that if no reference image could be located within the defined search area, the sensor pose would be utilized as the localization result. Table 7 shows that stricter thresholds result in worse outcomes. Consequently, we set γ t = 150 and γ o = 30 to ensure a sufficient search space. Furthermore, we measure the impact of retrieval number k in Table 8 . The results suggest that while a larger k value enhances the performance of the benchmark algorithm, it also leads to an increase in inference time. Following previous work [72] , the retrieval number is set at k = 3. It is worth noting that regardless of the choice of k, our method exhibits a substantial acceleration in speed, outperforming by several-fold, or even an order of magnitude.EQUATIONWe provide a detailed description of the reference images used in the two datasets. These images serve dual purposes: they function as the database images for retrieval and matching in baselines, and they are also utilized as training data for the proposed LoD-Loc method. Specifically, for the UAVD4L-LoD dataset, we use a subset dataset of synthesis images in UAVD4L [72] , excluding data that does not contain buildings. For the Swiss-EPFL dataset, synthetic images rendered in Latin Hypercube Sampling (LHS) [74] pattern have been employed as reference images. Notably, the CrossLoc dataset [74] did not include images in proximity to the out-of -P lace area. To address this, we adopted the same synthetic scheme from [74] to generate synthetic reference images for this region. Figure 10 shows reference samples of RGB images and Depth images for both the mesh-based model and LoD-based model.Although baselines have achieved impressive performance, they suffer from retrieving and matching repetitive texture images and cross-modal images. For example, Figure 13 exhibits deficiencies in retrieving repetitive texture images, and Figure 14 depicts poor matching results between RGB and LoD-rendered images.Table 6 : The architecture of our multi-scale feature extractor. We discuss the details of each convolutional unit. conv represents a unit consisting of a 2D convolutional layer, a batch normalization layer and a ReLU layer. While f ine_conv denotes a general convolutional layer. deconv means a deconvolutional unit. The colored cells are the outputs for each level l with a single channel. high-resolution aerial imagery through oblique photography reconstruction. As a result, the synthetic images from the former are of a lower quality. This could partly elucidate why our method yields lower results on the Swiss-EPFL dataset compared to UAVD4L-LoD.Layer Stride Kernel Channel Input conv0_0 1×1 3×3 3→8 rgb conv0_1 1×1 3×3 8→8 conv0_0 conv1_0 2×2 5×5 8→16 conv0_1 conv1_1 1×1 3×3 16→16 conv1_0 conv1_2 1×1 3×3 16→16 conv1_1 conv2_0 2×2 5×5 16→32 conv1_2 conv2_1 1×1 3×3 32→32 conv2_0 conv2_2 1×1 3×3 32→32 conv2_1 conv_out1 1×1 1×1 32→1 conv2_2 deconv1_0 2×2 3×2 32→16 conv2_2 concat1 - - - deconv1_0, conv1_2 conv3_0 1×1 3×3 32→16 concat1 conv_out2 1×1 1×1 16→1 conv3_0 deconv2_0 2×2 3×3 16→8 conv3_0 concat2 - - - deconv2_0, conv0_1 conv4_0 1×1 3×3 16→8 concat2 conv_out3 1×1 1×1 8→1 conv4_0 concat3 - - - conv4_0, conv_out3, rgb fine_conv0 1×1 5×5 12→24 concat3 fine_conv1 1×1 5×5 24→12 fine_conv0 conv_out4 1×1 1×1 12→1 fine_conv1We provide more ablation studies in this section, which include the pose sampling number, the sample density δ of 3D wireframes, the sampling range controller lambda λ. Additionally, we explore the convergence and generalization of our method.Pose sampling number. As illustrated in Table 9 , we report the experimental results with varying numbers of pose samples. The findings suggest that a reduction in the number of sampled poses brings about a decrease in accuracy. 3D wireframe points sampling density. We conduct ablation studies for varying sampling densities, which affects the interpolation process on the feature map. As depicted in Table 10 , there is no significant fluctuation in localization accuracy with changes in sampling density.Sampling range controller. The parameter lambda λ adjusts the length of the sampling range.Through ablation studies, we demonstrate that the sensitivity of this parameter during the testing phase is low. The results are shown in Table 11 .Convergence and initial poses. Table 12 reports the localization recall with different initial prior errors on the UAVD4L-LoD dataset. It can be observed that the success rate of localization decreases as the initial prior error increases. Such issues occur when the GPS signal in the air is heavily interfered with. In such cases, we believe using sequence information could be a possible solution.Cross-scene generalization. Table 13 illustrates the generalization capability of LoD-Loc through training and testing in diverse regions. Figure 16 delineates regional data using distinctive symbols and colors. On the UAVD4L-LoD dataset (A1 and A2), cross-scene testing yields results slightly lower than those obtained from training on the entire scene. For the Swiss-EPFL dataset (B1 and B2), we employ a model trained on the synthetic UAVD4L-LoD dataset, which achieves similar or even better performance compared to a model trained specifically on the Swiss-EPFL dataset. Additionally, the supplementary materials include two demo videos showcasing the model's capacity to localize cross-modal thermal images.Computational cost comparison. We conducted test experiments on a single batch (Batch Size = 1) of images using the NVIDIA GeForce RTX 4090 device, and recorded the average peak CUDA usage as well as the average inference time. The details are provided in Table 14We present more visualization results, including examples of corner houses (Figure 12 ), feature maps (Figure 17 ) and prediction results (Figure 18 ) at different levels. We found that the preset zig-zag route in a selected region resulted in some images capturing only the corners of houses, as shown in Figure 18 . This led to poorer performance under strict 2m-2°metrics. However, it is important to note that in the in-Traj. scenario, our method achieves comparable or superior results for coarse metrics. For instance, we achieve 96.95% on 5m-5°while the closest baseline achieves 92.14%.Table 13 : Cross-scene generalization. We assess the generalization ability of our method by training and testing on different regions. The regional divisions are illustrated in Figure 16 , identified by a specific color and letter.Recall (%) Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received a mixed recommendation (54568), where a majority of the reviewers voted for acceptance.
The major concern from the only negative reviewers is related to experimental settings, the authors provided a rebuttal and justifications, while this reviewer has no response. The AC has checked the comments and paper carefully and found that the comments of the positive reviewers are more convincing, thus recommending an acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes a method named LoD-Loc for visual localization of unmanned aerial vehicles (UAVs) using Level-of-Detail (LoD) 3D maps. Unlike existing algorithms that rely on complex 3D representations, LoD-Loc aligns wireframes derived from LoD model projections with the wireframes predicted by a CNN Unet to estimate the UAV's pose. To solve the problem, given a coarse pose provided by the sensor, this paper utilizes a hierarchical cost volume to pose hypothesis evolution. And eventually, a differentiable Gaussian-Newton algorithm for 6-DoF pose refinement.  This work also contributes two datasets to facilitate further research in this area and provides a comprehensive evaluation of the proposed method's performance. The experimental results demonstrate that LoD-Loc outperforms current state-of-the-art methods using textured 3D models, offering advantages in ease of acquisition, lightweight map size, and privacy preservation.
strength_weakness: strength_weakness: **Experiment Result**

The author mentioned CadLoc "cannot even localize the query image at a coarse threshold at $5m -5^{\circ}$". However, what I observed from Table 2 even Table 3 is the result for CadLoc are all zeros. Is that a mistake?

In Table 2, the authors show their method surpasses the SOTA textured 3D-model-based method in the $2m -5^{\circ}$ and $3m -5^{\circ}$ and "out-of-Traj."
However, the author does not explain why their method can perform better than textured 3D-model-based methods in "out-of-Traj" scenarios but fails in "out-of-Traj" scenarios.

In the Appendix, the author mentioned that "in-Traj." and "out-of-Traj." represent trajectory-based and free-flight scenarios, respectively.
But I didn't catch what might be the key difference between these two scenarios. Does that mean these methods are actually verified in different trajectories in the "out-of-Traj." scenarios? 
If so, I think it is not a fair comparison. If not, I didn't get the key difference between the "in-Traj." and "out-of-Traj." scenarios. It is suggested to explain why the proposed method performs better than the SOTA textured 3D-model-based method considering they use high-precision texture 3D information.

While the author mentions they conducted experiments on many other image retrieval-and-matching and textured 3D models-based approaches, only the results of UAVD4L and CadLoc are presented. It would be better to compare with more methods in each type to show a more comprehensive result.

**Computational Cost**

Although not mentioned, the methods involving hierarchical cost volume construction and differentiable optimization algorithms can be computationally intensive. This could pose challenges for real-time applications, especially on resource-constrained UAVs. For more comprehensive research, it would be better to provide a computational cost comparison for that.

**Presentation**

I think the writing and presentation skills in this paper could be further improved. Such as the presentation of Table 1 and the Contribution sections. Besides, I would suggest the author double-check their result in the presented Tables. Besides, I would suggest that the author add more explanations for their experiment result instead of just simply displaying them.
questions: questions: The proposed idea in general sounds okay to me. However, my questions are mainly about the result of the experiment result listed in the Weaknesses. I think the result present is questionable, insufficient, and lacks enough explanation to support the proposed method.
Besides, I think the writing and presentation need to be further improved for this paper to be accepted.
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents an approach to provide an accurate camera pose of an aerial image according to a LoD 3D map, which is a lightweight representation to store the key information of a known map. For each query image, multi-scale wireframe probability maps are extracted by a trained U-net. These maps are then used to select sampled poses from cost volume. Such process is executed in a coarse-to-fine manner. Lastly, Gauss-Newton is used to refine the final pose by minimizing the projection error of 3D wireframe points.

Two LoD city datasets are also released with the paper.
strength_weakness: strength_weakness: In this paper, the definition of "in-Traj" and "out-of-Traj" is not clearly provided. I guess this may mean the data used to train the model and those in the test stage. Please clarify these concepts in the paper.

The proposed method is significantly worse than UAVD4L in "out-of-place" part of Swiss-EPFL dataset. The authors explain that this is due to the relatively low quality of the dataset and LoD2.0 model. However, this shows that the method is much less robust to the data quality and it only works when high-quality data is provided, which can be very costly.
questions: questions: Do you have any process to deal with the sky pixels in query images?
limitations: limitations: The authors have mentioned the limitations of the method and the potential privacy impacts.
paper_summary: paper_summary: This paper introduces a method for visual localization of Unmanned Aerial Vehicles (UAVs) using a Level-of-Detail (LoD) 3D map and neural wireframe alignment.  The UAV sensor provides coarse pose estimation. Then, LoD-Loc hierarchically builds a cost volume for uniformly-sampled pose hypotheses to describe pose probability distribution.  A pose with maximum probability is selected as the correct one. Each cost within this volume measures the degree of line alignment  between projected and predicted wireframes. It also refines the previous result with a differentiable Gaussian-Newton method. The paper presents an approach to aerial visual localization with a strong emphasis on practical applicability and privacy considerations.
strength_weakness: strength_weakness: 1. The importance of the major contribution of "Level-of-Detail" is not clearly described. Acctually, Level-of-Detail has been widely investigated. 

2. The paper does not explicitly mention the computational efficiency or resource requirements, which could be a concern for real-time applications on UAVs with limited processing power.

3. The reliance on a coarse pose provided by the UAV sensor as an initial input may affect the method's robustness in scenarios where the initial pose is significantly off.
questions: questions: Is it possible that the method's performance might degrade in environments with poor GPS signal or where the UAV is operating in areas without prior map knowledge?
limitations: limitations: 1. The method assumes the availability of a gravity direction and location prior, which might not always be accurate or available, especially in GPS-denied environments.

2. The generalizability of the method to other types of UAVs or different environmental conditions is not fully explored in the paper.

3. The paper does not discuss the potential impact of dynamic elements in the environment (such as moving vehicles or people) on pose estimation accuracy.

4. The robustness of the method against various weather conditions or times of day is not addressed.
paper_summary: paper_summary: The paper presents a localization algorithm using LoD maps. Compared to conventional localization pipelines using SfM / SLAM maps, LoD maps are memory efficient and can offer privacy preservation. Nevertheless, due to the lack of texture in LoD maps, it is not straightforward to use pre-existing localization pipelines for LoD map localization. The paper thus presents a localization pipeline that can handle such ambiguities, by presenting a pose search / refinement pipeline based on estimating wireframe probabilities from images. Specifically, the paper suggests training CNNs to predict wireframe probabilities from images, and measures the geometric compatibility of the predicted probabilities against the LoD map. Experiments show that the proposed method can largely outperform existing CAD-based localization methods, while showing competitive performance against methods conventional methods using textured meshes and local feature descriptors.
strength_weakness: strength_weakness: I don't have large concerns for the paper, but below are a few suggestions:
1. Several related works seem to be missing, including some recent papers on descriptor-free visual localization:

a) Wang et al., DGC-GNN: Leveraging Geometry and Color Cues for Visual Descriptor-Free 2D-3D Matching, arXiv 2024
b) Kim et al., Fully Geometric Panoramic Localization, CVPR 2024
c) Pietrantoni et al., SegLoc: Learning Segmentation-based Representations for Privacy-Preserving Visual Localization, CVPR 2023.

2. I wonder how the proposed method could be extended for larger-scale pose search scenarios, for example if GPS / gravity direction is unknown. I feel the search space might be too large for the current grid-based search to work. Do the authors have any rough suggestions for how the work could be extended in such directions?

3. How would be proposed method compare against explicitly extracting line segments from 2D images? To be more specific, suppose one replaces the wireframe probability extraction part explained in Section 3.1 with distance transformed calculated for 2D line extractions using conventional line extractors such as LSD or DeepLSD[1]. The cost functions or features in Equations 2 and 4 could be replaced with  the distance function values, similar to line-based localization methods [2, 3]. I feel this would not fare well since there would be large number of noisy lines and lines from image textures, but still the comparisons would provide insights into why the feature extraction in Section 3.1 is necessary for good performance. 

[References]
1. Pautrat et al., DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients, ICCV 2023
2. Micusik et al., Descriptor free visual indoor localization with line segments, CVPR 2015
3. Kim et al., LDL: Line Distance Functions for Panoramic Localization, ICCV 2023
questions: questions: I listed some minor questions below:
1. What are the units of delta_x, y, z in Table 11?
2. Related to Table 11, I feel it will be nice to see more experiments on the impact of the initial pose. For example, what errors would we get if we initialize from place recognition algorithms for UAVs [1]?


[References]
1. Keetha et al., AnyLoc: Towards Universal Visual Place Recognition
limitations: limitations: Yes the limitations are discussed.
paper_summary: paper_summary: This paper proposes a hierarchical scheme for pose selection to progressively compute high-quality pose in a coarse-to-fine fashion.
strength_weakness: strength_weakness: The innovation of the paper is not clearly described.
questions: questions: This paper give a detailed description of some proposed steps, without providing the target of these steps.  As far as I am concerned, these steps are general steps for localization, what's your innovation?
The problem you want to solve and the shortcomings that current methods exist should be strengthened.
The quality of some figures can be improved.
limitations: limitations: As described above

Comments: title: To all reviewers
comment: Updated
Dear Reviewer UAbv,

We hope this message finds you well.

We appreciate your responses and are pleased to have addressed your concerns. We noticed that the system not yet updated your rating. We are unsure if this is due to a display issue with the system or if you have any other concerns regarding this work. If there are any additional clarifications needed, please do not hesitate to reach out.


Best regards,

Authors
Dear Reviewer nkE1,

Thank you for your review. May I politely ask if you have any other questions regarding this work? If any further clarification is needed, please do not hesitate to reach out. Your comments is important to this work.

Best regards,

Authors
Thanks for your time. May I humbly inquire about any specific actions you would recommend that could potentially increase the score as per your initial review? Because, we notice that there has been no change on the rating score in the system. If there are any additional clarifications needed, please do not hesitate to reach out.
Thank you for your time. May I humbly inquire about any specific actions you would recommend that could potentially increase the score as per your initial review? Because, we notice that there has been no change on the rating score in the system. If there are any additional clarifications needed, please do not hesitate to reach out.
Thanks for your response.
I appreciate the authors' response. It addresses my questions. I will raise my rating.
Thanks for the detailed response from the author. I think it addresses my concern and provides a better presentation. I will raise my rating.
Thank you for the detailed response: it has answered most of my questions. Therefore I will raise my score.
Dear reviewers,

This paper received mixed ratings, the author(s) have provided a rebuttal. It would be appreciated if reviewers could carefully read the rebuttal and let the authors and the AC know if it has changed your mind (and detailed justification). Your opinion will play a significant role in the final decision of this paper. Thank you so much.

Best Regards,

AC