Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator
Authors: Xiaolong Wang
Runsen Xu
Zhuofan Cui
Zeyu Wan
Yu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Fine-Grained Cross-View Geo-Localization
Homography Estimation
TL;DR: We present HC-Net, an end-to-end network for fine-grained cross-view geo-localization, which outperforms the previous state-of-the-art by a large margin.

Abstract: In this paper, we introduce a novel approach to fine-grained cross-view geo-localization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to an image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by 21.3\% and 32.4\% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4\% on the KITTI benchmark in same-area evaluation.

Fine-Grained Pross-View Geo-Localization Using a Correlation-Aware Homography Estimator
Xiaolong Wang"
Runsen Xu², Zuofan Cui', Zeyu Wan', Yu Zhang
College of Control Science and Engineering, Zhejiang University
Key Laboratory of Collaborative sensing and autonomous unmanned systems of Zhejiang Province The Chinese University of Hong Kong
Abstract
In this paper, we introduce a novel approach to fine-grained cross-view geo- localization. Our method aligns warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric prin- ciples, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using point above the central axis. Operating at a speed of 30 FPS, our method outperforms state- of-the-art techniques, reducing the mean metric localization error by 21.3% and 32.4% in same-area area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4% on the KITTI benchmark in same-area evaluation.
Introduction
Accurate localization of ground cameras is essential for various applications such as autonomous driving, robot navigation, and geospatial data analysis. In crowded urban areas, cross-view localiza- tion using satellite images has proven to have great potential for correcting noisy GPS signals 38
and improving navigation accuracy 42 13 [9]. In this paper, consider the task of fine-grained cross-vine view gco-localization, which estimates the 3-Dof pose ground camera, i.e., GPS location and orientation (yaw), from given ground-level query image and a geo-referenced aerial image.
Previous works on coarse cross-view localization [10 16
37 4T have achieved high recall rates by formulating it as an image retrieval problem. However, the localization accuracy obtained using this method limited by the segmentation size of satellite image patches, resulting in localization errors of tens of meters.
Recently, there has been a growing interest in fine-grained cross-view geo-localization. which assumes the availability of the ground image and a corresponding GPS-labeled satellite image patch covering the same area. Existing methods can be divided into categories: those based on repeated sampling
27 24
26 and those employing descriptor candidates splitting from satellite image features
Contact:
The Code is available http:///gg..tttttoooooo//aareeee
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Spharical Transform
(a) Ground Image
(b) BEV Image
(c) Satellite Image (d) Homography Maped
Figure I: Visualization of the cross- view localization process in method. The satellite image (c) accompanied by a correlation map that represents the dense probability distribution for localization. Aligning the BEV with the satellite image using the homography matrix is shown in (d).
Sampling-based methods transform satellite or ground images (or feature maps) to compare with another view, involving the multiple applications of geometric transformations based on potential ground camera poses. However, these methods suffer from low localization accuracy
36
12 35
[27 22 26 and prolonged time consumption 27 26). [36 22 35 split the features of satellite images into numerous candidates to obtain more effective and refined descriptors for comparison with the ground image descriptor. Their limited pixel-Ievel localization resolution and high memory usage present challenges for deployment in real-world end-terminals. Hence, the pursuit of a methodology that encompasses swift computation, low memory utilization, and elevated levels of both localization accuracy and resolution remains paramount area of research.
We observe that projecting ground images onto bird's-eyy view perspective can make satellite-base localization tasks more intuitive, similar to the way humans use maps for navigation, eliminating the need for multiple sampling further splitting of satellite patches. In this study, we develop spherical transform module that leverages the imaging model ground cameras to project panoramic images onto aerial perspective, effectively bridging the gap between distinct viewpoints. As depicted in Figure]] (b), our approach does not necessitate prior pose information, and it facilitates the wrrighhfoowwrd determination of the ground image's 's position within the satellite image (c). By transforming the complex cross-view localization problem into a 2D image alignment problem, our method enables the acquisition precise ground GPS location and orientation.
The core problem in 2D image alignment lies in obtaining the homography transformation between two images. Given that feature- based methods like [22 28 may not yield effective results in obstructed or unclear scenes, we employ a correlation-awe aware homography estimation approach. We utilize recurrent convolutional neural network block to maximize the correlation between similar regions in the feature maps, unlike previous iterative optimization methods [24], which minimize overall feature differences. Our method ensures low correlation in unobservable areas, minimizing their impact on homography estimation, resulting in an end-to-end network that aligns the most similar parts of the transformed ground image with the satellite image, directly outputting its GPS location. Moreover, our approach has successfully overcome the challenge of lacking compact supervision information in homography estimation, i.e., the absence of at least four matching point pairs. While utilizing the VIGOR dataset [42, we also address the inherent errors in the original ground truth labels to facilitate further research.
The main contributions this paper include:
A novel and effective method for fine-grained cross-view geo-localization, which strictly aligns ground and acrial image domains using geometry-constrained spherical transform, reducing the problem to 2D image alignment.
A correlation-awee aware homography estimation module that eliminates repeated sampling and disregards unobservable content the images, resulting in excellent generalization performance and rapid inference speed at 30 FPS.
Extensive experiments demonstrating that our method outperforms the state-of-the -te-hat on two fine-graing -raingrrindd cross-view localization benchmarks. Specifically, on the same-area and cross-area splits of the VIGOR benchmark [42), our method reduces the mean localization error by 21.3% and 32.4%, respectively. On the same-area split of the KITTI benchmark [8), method reduces the mean localization error by 34.4%.

---

2 Related Work
Cross-view image retrieval methods for geo-localization, which use ground images as queries and all patches in a satellite image database as references, have been studied for years and rely on global
image descriptors for successful retrieval. Early works [5] 34 33
39] suffer from low retrieval
accuracy due to significant appearance gaps and poor metric learning techniques. SAFA [25], Shi et
al. (27] L2LTR (37), and TransGeo[4] [41] have improved localization accuracy by employing polar
transform algorithms, considering orientation, and using transformers 32
Fine-grained cross-view localization beyond one-to-one retrieval first proposed in CVR 42 and they propose corresponding benchmark VIGOR. [27] and [24) project the satellite view using the candidate poses or the iterative optimization method and select the one that is most similar to the ground as the localization result. 7 and [26] employ transformers to acquire BEV representations of ground images, and then transform BEV maps by all possible ground camera poses to compare with the feature representations of satellite images. However, there remains untapped potential for refinement, and their inference entails prolonged time. (36) splits the features of satellite patches into several sub-volumes. [12] considers orientation and geometric information to create effective satellite descriptors for a set of candidate poses. (35] incorporates orientation and employs a coarse-to- fine manner to enhance accuracy. Despite these improvements, all three methods face resolution and high memory usage challenges stemming from the splitting or candidate selection process.
Bird's-Eye View Perspective has been thought of as the preferred representation for many tasks (e.g. navigation or localization). Recent methods [1 7 26 utilize transformers to project feature maps of ground images onto BEV. However, there has been no utilization of explicit BEV images for fine-grat fine-grained cross-view geo-localization. A challenge could stem from the requirement of known camera calibration for traditional Inverse Projective Mapping methods, as [3] used. In an attempt to enhance BEV representations, Boosting (26) also explores geometric-based methodologies, but due to the necessity for depth information, transformers remain a crucial tool to handle ambiguity.
Homography estimation is the process of estimating a mapping between two images of a planar surface from different perspectives. DeTone et al. [6] is the first to propose deep learning-base based approach homography estimation. Nie al. [20] proposed the use of global correlation layer to address the problem of small overlapping regions between the two images. Zhao et al. [40] incorporate the Kanade (LK) algorithm [18] as a trainnaie iterator and combine it with CNNs. They also investigate GPS-denied navigation using Google static maps and satellite datasets based on this method. Inspired by RAFT (30]. IHN 4 designed a completely iterative trainable deep homography estimation network, achieving high accuracy but requiring strong supervision. HomoGAN 9 designed an unsupervised homography estimation network based on transformer and GAN methods, achieving promising results while consuming more computational resources.
3 Method
This paper presents a novel approach to address the task of fine-grained cross-view localization, as illustrated in Figure 2 Our Homography estimation-based Cross-view geo-localization Network, named HC-Net, takes corresponding spherical-transte image as input and outputs the homography matrix between them, as well as the precise GPS location and orientation of the ground camera. The following sections introduce the spherical transform for projecting panoramas
ground image and GPS- GPS-tagged satellite
bird's s-ye view (Section 3.Ip, the homography estimator for precise alignment of ground BEV images with satellite images (Section 3.2). and the supervision method of our network (Section 3.3
3.1 Spherical Transform
Panoramic imaging model Panoramas capture full 360-degree view on a spherical projection plane and use equirectangular projections for display, as shown in Figure (a). We represent points in the ground camera coordinate system as P = (I1, U1, 21) and projected points in the normalized equirectangular coordinate system of the panorama as P (22,8/2) The spherical coordinates (ré, 0) are acquired from (I1, (1) using inverse trigonometric functions, with north latitude and east longitude being positive. And the projection between the spherical coordinates (1b, 0) and the
normalised equirectangular coordinates (I2,, y2) is expressed in Figure) 2 (a), (b).
3

---

Spherical Transform
1.Feature Extraction
3. Correlation-Aware Supervision
Fistare
Hyree
Waide
1/32
30 201n
CP8
Sre
(0) Normalieed Equrectanguter
(i) Spterical coordinates
2.
Hoooggrrett Estimation
Recurrent Part
Vher
corricttion coondrickcs
Ober
Paaking
Sampliing
Propection
Project
(4]
4D Conokation Volumes
Comclation Pytorid
giSA
AM
Figure 2: An overview of the proposed method for fine-grained cross-view localization. (0) illustrates the geometric principles behind our spherical transformation, while (1), (2), and (3) demonstrate network architecture and supervision methods.
Spherical transform To obtain the corresponding bird's-eye view of the panorama, we place tangent plane at the south pole of the spherical imaging plane as a new imaging plane, as shown in Figure 2 (c). Formally, let Hp Wp be the size of the panorama and H' W, be the target size of the bird's s-ed' view after the spherical transform. We connect the camera's 's optical center with each pixel point the BEV imaging plane, determining the corresponding pixel position in the panorama through the intersection of the connection line with the spherical imaging plane. We demonstrate the pixel coordinates
the bird's-eye view imaging plane as (us, 16). The camera
coordinates (I1, V1 21) corresponding to the pixel coordinates (us, 0b) can be directly obtained. We set the parameter f'ov 85 to determine the field of view of the bird's-ey s-eye view. The focal length of the BEV in the imaging process is ʃ = 0.5W, (on()ov). Therefore, each corresponding panoramic image coordinates (Up, Up) is established as:
up = [1 arctan2 (Wotn 2 - 22/2
Up [0.5 a
u.)² (Hr/2 vy) 2/)2)/ //////H//p...
After the spherical transform, the ground image is projected into the aerial image's perspective, as seen in Figure[[dd) This approach enables high-resolution localization without multiple sampling or splitting of satellite patches. Unlike the satellite-based image projection discussed in 22 24), this type of projection does not require the selection of a projection point. We adopt similar approach for the KITTI dataset [8] by setting up an overlooking imaging plane. This allows to project front-view bird's-eye s-eye view without any known camera calibration, shown in Figure 4 (a), see details Supplementary Material. Despite being applied as a preprocessing step in our method, note that the transform is differentiable. This property opens up possibilities for future research on the 5-DoF pose of the ground camera (details in Supplementary Material).
images into a bird's
3.2 Homography Estimator
To achieve precise localization by aligning the transformed ground image I, with the satellite image Ig. we propose correlation-basee deep homography estimator inspired by IHN (4. We resize the two images Ig and I3 to the same size and feed them into a "seudo-Siamese CNN to extract features.
These features denoted as F, E xx and F, E D As shown in Figure 2 after feature extraction, correlation computation, and recurrent homography estimation, the homography matrix H between I, and I, be obtained.
Correlation Computation Given the feature maps Fg and F., the correlation volume C e X is formed by taking the dot product between all pairs of feature vectors as:
C(F,, E
H HXW Cigıı (i.) F.(k,)).
(2)

---

To enlarge the perception range within a feature scale, we conduct average pooling on C at the last 2 dimensions with stride 2 to form another correlation volume C² E x
Recurrent Homography Estimation We refine the estimation of the homography through loop iterations. The coordinates set on Fg are denoted as X E R2xHxW and the coordinates set on F,, which is projected from X using the homography matrix H, is denoted as X' E RR For each denote
coordinate position,
(u, v) in X and x' (u', e²) in X'. In each iteration, Equation 3 to calculate x and use it to sample the last two dimensions C, C with a local square grid of fixed search radius r, resulting in correlation slices S and S of size H x W xrxr.
the
telk
H
H
H
2
13
ofk
~
H H
H
H 23
V
(3)
H
Then we utilize a convolutional neural network module for residual homography estimation. As in 4 we parameterize the homography matrix using the displacement vectors of the 4 corner points of an image, namely the displacement cube D. In iteration k we feed the concatenated correlation slice s S the coordinates set X, and the currently projected coordinates set X into the CNN module. The module is mainly composed of multiple convolutional units until the spatial resolution of the feature map reaches 2 x 2, where each unit downsamples the input by a scale 2. Then a 1 x 1 convolutional unit projects the feature map into a x x cube AD which is the estimated residual displacement vector of the 4 corner points. In iteration k, we update D$ by adding the estimated residual displacement vector AD to D Using D* we can obtain the homography matrix H* through the direct linear transform
2 The updated H is then used to project in the next iteration.
3.3 Network Supervison
Label Correction We use the VIGOR dataset and evaluation. However, the original labels in 42 approximate and consistent meter-to-pixel resolutions of the aerial images. Although [12] attempts to correct the labels, they require city- city-based computations or selections for resolution, which brings significant inconvenience. In our approach, we propose using the Web Mercator projection [I used by virtually all major online map providers to accurately convert GPS coordinates to pixel coordinates on satellite patches, thereby enhancing generality and convenience. The main equation is
42
for fine-grained cross-view localization training contain errors up to meters due the use of
256
2 200m (lon + T) pixels,
2T
256
lat π -T In tan (
))} pixels,
2 200mm
+
2T
2
where zoom indicates the zoom level for satellite images (zoom 20 in VIGOR and zoom 19 in KITTI) and 256 is the resolution of each satellite tile. We have applied the same method to create Supplementary Material for details.
training labels for KITTI dataset g
Loss Function Using the homography matrix H obtained Section 3.2 we project the center point of BEV onto the satellite image to obtain the localization pixel (u,, v.). Using Equation 4 we calculated the pixel coordinates of the ground truth GPS corresponding to the satellite image as (u,, v Furthermore, we use Equation 4 to project another point the BEV centerline onto the satellite image. Connecting this point with (u,, V.) allows us to determine the orientation of the ground camera as 8.
We use hybrid loss function, defined as ai Cais + ar2Cori + CaaCino to guide our training. Here, Laie = (u,- - u²)² - (v. 2 represents the L2 norm loss between the predicted result and the ground truth. Cor: = 10 0*) represents the LI norm loss between the predicted 0 and the ground truth 0 To generate a probability distribution that can be further utilized for localization, we propose the use of an infoNCE loss [2T] to reinforce the correlation between the BEV point used for localization and the satellite image. Linfo is defined as:
exp (Clie.je,k* kt,))
Linfo
log
(5)
En. exp (C (Ke, k, (T))
5

---

where (e)) represents the center coordinates of Fg. and (k*,1*) ,1)) represents the downsampled position of (ut, eg) in the satellite feature map
F.. The hyper-parameter is introduced to adjust the sharpness of the resulting probability distribution.
Finally, through the inverse process of Equation ing to (u,, Ve). thereby obtaining highly accurate localization outputs.
4 can determine the GPS coordinates correspond-
Experiments
In this section, we first introduce two used datasets, evaluation metrics, and implement details of our network. We then compare the performance our HC-Net to state-of-the-art and examine its ability to generalize to new measurements within the same areas, across different areas, and across datasets. Finally, we present ablation studies and computational efficiency analysis.
4.1 Datasets and Evaluation Metries
VIGOR dataset [42 contains geo-tagged ground-level panoramas and aerial images collected in four cities in the US. Each aerial patch corresponds ground area of approximately 70m x 70m. From Figure T it can be observed that the effective field of view of the ground image is slightly smaller than this range. A patch is considered positive if its center 1/4 region contains the ground camera's 's location, otherwise, it is semi mosiiooiiieee In our experiments, we use positive acrial images for training and testing all models. The panoramas are shifted based on orientation information to align North in the middle, indicating that the orientation prior is known. During training, we introduce +:45 noise the orientation prior to generating orientation labels. The dataset provides 105, 214 panoramas for the cc--localization experiments. We adopt the Same-Ares and Cross-Area splits from 44]. For validation and hyperparameter tuning, we randomly select 20% the data from the training set, as done in [36] 12 35 Compared to [[2), we have more accurately corrected the ground truth
labels in [22] as mentioned in Section 3.3
KITTI dataset g contains ground-level images captured by moving vehicle with a forward- facing viewpoint, which a restricted viewpoint. (24] augments the dataset with aerial images. Each aerial patch corresponds to ground area of approximately 100m x 100m. The Training and Test sets consist of different measurements from the same region, while the Test2 set has been captured in different region. As assumed in [2], ground images are located within 40 x 40m area in the center of the corresponding aerial patches, and there is an orientation prior with noise between 100
Evaluation metries in our experiments are the mean and median error between the predicted and ground truth over all samples for both localization and orientation separately in meters and in degrees. Following (35], for the KITTI dataset, we additionally include the recall under a certain threshold for longitudinal (driving direction) and lateral localization error, and orientation estimation error. Our thresholds are set to Im and 5m for localization and to 1 and 5" for orientation estimation.
4.2 Implementation Details
Our network uses EfficientNet-BOO [29 with pretrained weights Imagenet [5] as both the ground and aerial feature extractors, with non-shared weights. The satellite image and bird s-eee-view (BEV) transformed from the ground image both have size of 512 x 512 both the VIGOR 42 and KITTI 8 datasets. PyTorch is used for network implementation, and training is done using the AdamW [7 optimizer with maximum learning rate of 3.5 x 1 The network is trained with batch size of 16 and a training iteration of 180000. We set the search radius of the correlation updater 4 and set 01 0.1, 02 10, 03 1.0, 4 in the loss function. The total iteration K = 6/110 and the feature map size D x H x W = 320 x 16 x 16/320 x 16 x 16 on VIGOR/KITTI.
4.3 Comparison with State-of-the-Art Methods
VIGOR dataset On the VIGOR dataset, we compare our method against several state-of-the. art methods: CVR [42], SliceMatch [2], Boosting (26 and CCVPE [35]. For the performance
evaluations of CVR [2], SliceMatch 12.. and Boosting [26] with known orientation priors, we
directly utilize the results provided by CCVPE or the corresponding paper. Regarding most state-
of-the-art CCVPE (55. we executed its official code to obtain results not presented in the original

---

Table I: Location and orientation estimation error on VIGOR 42 dataset. Best in bold. Different levels of noise are added to the orientation prior. "ø" indicates methods using our corrected labels.
Same-Area
Cross-Area
Noise
Method
[Localization(m) _Orientation(") median
[Localization(m) median
[Orientation() mean
mean
mean median
mean
median
CVR
42
8.82
7.68
9.45
8.33
SliceMatch
12
5.18
2.58
5.53
2.55
Boosting
26
4.12
1.34
5.16
1,40
0F
CCVPE [35
3.60
1.36
10.59
5.43
4.97
1.68
27.78
14.11
CCVPE* [35
3.37
1.33
9.47
5.23
4.96
1.69
26,30
14.21
HC-Net(ours)
2.65
1.17
1.92
1.04
3.35
1.59
2.58
1.35
CCVPE* 335
9.80
5.49
5.16
26.36
+:20
HC-Net(ours)
2.65
1.17
1.93
1.02
3.36
1.59
2.62
1.34
CCVPE* 35
3,50
1.39
10.56
5.95
5.16
1.78
26.77
15.29
445
HC-Net(ours)
2.70
1.18
2.12
1.04
3,46
1.60
3.00
1.35
(a) example 1
(b) example 2
(c) example 3
(d) example 4
Figure 3: Visualization of our method the VIGOR 2 dataset with localization confidence probability indicated in the legend. (d) demonstrates a failure case of our method when dealing with scenes that radially monotonous and pnpetitiveeunifformale
paper and used the published results where applicable. We also re-train and evaluate CCVPE with our corrected labels to ensure fairer comparison. We conduct comprehensive evaluation with different levels of noise in the orientation prior to ground images, including 09, +20% =
As shown in Table T our method outperforms all previous methods in both same-area and cross-area settings terms the mean localization error metric, which suggests that our method can cope with more challenging scenarios, as shown in Figure The mean localization error is reduced by 21.3% and 32.4% respectively. The previously berf-performing method (35] shows a notable performance gap between same-area and cross-area settings, while our method significantly narrows this gap. This suggests that our method possesses superior generalization capabilities. We further demonstrate our superior generalization ability across datasets in Section 4.4
When prior information exhibits varying degrees of noise, our method still outperforms CCVPE 335 and provides a more accurate estimation of the orientation. Moreover, our network generates confidence probabilities for localization results, allowing us filter out potential misestimations. As shown in Figure (c), when the scene contains symmetric layout like zebra crossings and noise exceeds 90" the homography estimation may incorrectly align the ground image with the satellite map. However, by rotating the BEV image four times and selecting the prediction with the highest confidence probability, we can obtain the correct result as in Figure 3 Despite the mislocalized point having similar appearance to the ground observation, its probability output is significantly smaller than the localization confidence probability from the correct location. This property is erucial for safety-crin cafety-critical applications like autonomous driving.

---

Table 2: Location and orientation estimation error KITTI 8 dataset. Best in bold. Long. and Orien. are abbreviations for Longitudinal and Orientation, respectively.
[Location(m) (m)
[atteral R@
[ongg.(S)
JOrien (*)
1Oricn.(8)
Method
Area
Mean
Median
R 5m
R Im
R
Mean
Median
R R@1
R@5
LM 24
Same
12.08
11.42
35.54
80.36
5.22
26.13
3.72
2.83
19.64
71.72
SliceMatch
Same
7.96
4.39
49.09
98.52
15.19
57.35
4.12
3.65
13.41
64.17
Boosting 20
Same
76.44
98.89
23.54
62.18
99.10
100.00
CCVPE 35
Same
1.22
0.62
97.35
99.71
77.13
97.16
0.67
0.54
77.39
99.95
HC-Net(
Same
0.80
0.50
99.01
99.73
92.20
99.25
0.45
0.33
91.35
99.84
LM 224
Cross
12.58
12.11
27.82
72.89
5.75
26.48
3,95
3.03
18.42
71.00
SliceMatch 12
Cross
13.50
9.77
32.43
76.44
8,30
35.57
4.20
6.61
46.82
46.82
Boosting 20
Cross
57.72
91.16
14.15
45.00
98.98
100.00
CCVPE
35
Cross
9.16
44.06
90.23
23.08
64.31
1.55
0.84
57.72
96.19
HC.
HC-Net(OOrs)
Cross
8.47
4.57
75.00
97.76
58.93
76.46
3.22
1.63
33.58
83.78
(a) Projection of KITTI
(b) Localization Results
Figure 4: (a) depicts the projection of the frontal perspective into bird's-eye view in the KITTI dataset 8 accompanied by the corresponding satellite imagery. (b) visualizes our method's localiza- tion results on KITTI [8: the top row shows aligned ground and satellite images using our estimated homography matrix; the bottom row presents original satellite images for reference.
KITTI dataset On the KITTI dataset, compare method with the state-of-the-ar LM (24), SliceMatch [12], Boosting (66 and CCVPE Similar to the evaluation on VIGOR, we directly the results provided by CCVPE 35 the corresponding paper. We utilize increased number
of iteration steps in response to the limited coverage of ground images from KITTI g within satellite images. The results are shown in Table 2 When +10 orientation prior is considered in both Training and Testl, our method has a 34.4% lower mean error for localization than CCVPE [35]. Figure 4 presents the visualization results the KITTI dataset. Boosting (26) achieves exceptional accuracy in estimating orientation. This may be attributed to their utilization of a two-stage approach, where the first stage is dedicated orientation estimation.
4.4 Cross-Dataset Generalization
The CVUSA dataset 39
commonly used for cross-view localization, but it only provides retrieval labels and cannot be used for training precise localization models. Despite the panoramas in CVUSA being cropped and having non-standard aspect ratio, our spherical transform method successfully projects them to a bird's aad''-eye view by completing it to the correct proportion, shown in Figure S We use images from CVUSA in different cities to test our model's generalization ability, which is trained on VIGOR. The alignment results in Figure 5 demonstrate that our model has strong potential and can align significantly different BEV images with their corresponding satellite images for precise localization, even in unfamiliar cities without retraining.

---

Table 3: Ablation Study on Secudo-Siamess Backbone and Homography Estimation Module.
Table 4: Comparison of computational efficiency between our method and CCVPE 35
Experiment
Mean Error
Median Error
CCVPE 35
(CC-Ne((our))
Non-shared weights
2.65 m
1.17 m
Parameters
57.40 M
11.21
Shared weights
3.36 m
1.36 m
Memory Usage
4730 MiB
1900 MiB
Inference Time
33 ms
31 ms
SuperGlue 22
13.06 m
3.76 m
Mean Error
3.37
2.65
LoFTR 28
28.85 m
4.89 m
spwbtd banke
Figure 5: Three examples in CVUSA[39]. The top of each example shows the completed panorama. The bottom-left shows the bird's-eye s-eye view obtained using the spherical transform. The bottom-right shows the alignment result of our method between the BEV and the satellite image.
4.5 Ablation Study
Pseudo-Siamese Backbone We employ a Pseudo-Sianen Siamese backbone with non-shared weights for processing images from two different sources. To demonstrate the effectiveness this structure, we train two models with shared and non-shared weights. The results in Table 3 show that applying the Rxeudo-Siamest backbone leads to a mean localization error reduction 0.71m.
Homography Estimation Module We also conduct ablation experiments to demonstrate the effectiveness of our homography estimation module. We replace the homography estimation module with feature-based methods SuperGlue (22) and LoFTR (28), utilizing RANSAC-based functions from OpenCV to compute homography matrix from matched feature points. The experimental results presented in Table E] The fact that the feature-based methods have low median localization errors demonstrates the effectiveness of transforming localization into 2D alignment. However, the significantly lower mean errors achieved by our proposed deep homography estimator indicate its superiority in achieving more accurate localization.
4.6 Computational Efficiency Analysis
We assess our method's computational efficiency against the state-of-the-art CCVPE [35]. Table compares model parameters, inference memory, per-frame inference time, and mean localization errr on the VIGOR dataset using 12th Gen Intel(R) Core(TM) 12-92F processor, 16GB memory, and an NVIDIA RTX 3050 GPU. Our method achieves higher accuracy, faster speed, and lower memory usage, demonstrating its computational efficiency.
Conclusion
In this study, we introduce HC-Net, an end-to-end network designed for fine- ane-grained cross-view geo-localization. The network processes ppherical-transforee ground images and GPS- GPS-tagged satellite images as inputs and generates the homography matrix between them, along with the precise GPS location and orientation of the ground camera. Compared to the previous state-of-the-an HC-Net demonstrates a reduction in mean localization error by 21.3% and 32.4% on the same-area and cross-ares area splits of the VIGOR datasct. respectively.

---

Acknowledgements
This work supported by NSFC 62088101 Autonomous Intelligent Unmanned Systems, in part by TI 2030-Maajo Projects 20212D0201403.
References
[1]
[2] Yousset Abdel-Aziz, Hauck Michael Karara, and Michael Hauck. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogrammetrric engineering & remote sensing, 81(2):103-100 2015.
[3] Eli Brosh, Matan Friedmann, Ilan Kadar, Lev Yitzhak Lavy, Elad Levi, Shmuel Rippa, Yair Lempert, Bruno Fernandez-Ruiz, Roei Herzig, and Trevor Darrell. Accurate visual localization for automotive applications. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition Workshops, pages 0-0, 2019.
[4] Yuan Cao, Jianxin Hu, Zehua Sheng, and Hui-Liang Shen. Iterative deep homography estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1879-1888, 2022.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and paltern recognition, pages 248-255. Icce, 2009.
[6] Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Deep image homography estimation. arXiv preprint arXiv: 100.0039606 2016.
[7] Florian Fervers, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelhagen. Uncertainty-aware vision-based metric cross-view geolocalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21621-21631, 2023.
[8] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11): 2013.
[9] Mingbo Hong, Yuhang Lu, Niajjin Ye, Chunyu Lin, Qijun Zhao, and Shuaicheng Liu. Unsuper- vised homography estimation with coplanarity- gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17663-17672, 2022.
[10] Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7258-7267, 2018
[11] Sixing Hu and Gim Hee Lee. based geo-localization using satellite imagery, International Journal of Computer Vision, 128(5); 2020.
[12] Ted Lentsch, Zimin Xia, Holger Caesar, and Julian FP Kooij. Slicematch: Geometry-guided aggregation for cross-view pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[13] Ang Li, Huiyi Hu, Piotr Mirowski, and Mehrdad Farajtabar. Cross-view policy learning for street navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8100-8109, 8109, 2019.
[14] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning aed's-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[15] Tsung- Yi Lin, Serge Belongie, and James Hays. Cross-view image geolocalization. In Pro- ceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 891-898. 2013.
10

---

[16] Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geo- localization. In Proceedings of the IEEE/CVE conference on computer vision and pattern 2019.
recognition, pages
[17] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[18] Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In : 7th international joint conference on Artificial intelligence, volume 2, pages 674-679, 1981.
[19] Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. Advances in neural information processing systems, 31, 2018.
[20] Lang Nie, Chunyu Lin, Kang Liao, Meiqin Liu, and Yao Zhao. A view-free image stitching net- work based on global homography. Journal of Visual Communication and Image Representation, 73:102950, 2020.
[21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 180i:: 007748, 2018.
[22] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4947,2020,, 2020.
[23] Paul-Edouard Sarlin, Daniel DeTone, Tsun-Y Yi Yang. Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulò, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. Orienterne:: Visual localization in 2d public maps with neural matching. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21632-21641 2023.
[24] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 17010-17020, 2022.
[25] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial feature aggregation for image based cross-view geo-localization. Advances in Neural Information Processing Systems, 32, 2019.
[26] Yujiao Shi, Fei Wu, Ankit Vora, Akhil Perincherry, and Hongdong Li. Boosting 3-dof ground- to-satellite camera localization accuracy via geometry-guided cross-view transformer. arXiv preprint wrXiv:2307.08015. 2023.
[27] Yujiao Shi, Xin Yu, Liu Liu, Dylan Campbell, Piotr Koniusz, and Hongdong Li. Accurate 3-dof camera geo-localization via sannd-t--satellite image matching. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[28] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8922-8931, 2021.
[29] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on muchine learning, pages 6105-61 14. PMLR, 2019.
[30] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer VESi 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[31] Yicong Tian, Chen Chen, and Mubarak Shah, Cross-view image matching for geo- coollocclizotion urban environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2017.

---

[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[33] Nam N Vo and James Hays. Localizing and orienting street views using overhead imagery. In Computer Vision-EcCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 494-509. Springer, 2016.
[34] Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision, pages 3961-3969, 2015.
[35] Zimin Xia, Olaf Booij, and Julian F. P. Kooij. Convolutional cross-view pose estimation, ArXiv, abs/2303.05915, 2023.
[36] Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 90-106. Springer, 2022.
[37] Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems, 11-29009-290200 2021.
[38] Amir Roshan Zamir and Mubarak Shah. Accurate image localization based on google maps street view. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pages 255-268. Springer, 2010.
[39] Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs. Predicting ground-level scene layout from aerial imagery. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 867-875, 2017.
[40] Yiming Zhao, Xinming Huang, and Ziming Zhang. Deep lucas-kanade homography for multimodal image alignment. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15950-15959, 2021.
[41] Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer all you need for cross-view image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1162-1171, 2022.
[42] Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3640-3649 3649, June 2021.
12

---

Supplementary Material for Fine-Grained Cross- View Geo-Localization Using a Correlation-Aware Homography Estimator
In this supplementary material, we provide detailed information on projecting ground images to bird's-eg s-eye view for VIGOR 42 and KITTI [8] datasets in Section A as well as the computation details for converting GPS labels to pixel coordinates used during training in Section B We also discuss the potential for future research based on our method in Section C and broader societal and industry impact of our approach in Section D
A
Projection Details
A.1 Details for Spherical Transform
In the main paper, provide overview of the derivation process of the Spherical Transform and the final result Equation [ Here we will provide detailed information on the process for the projection. We will use the expressions defined in Section 3.I To clarify the presentation, we also provide the schematic diagram of the projection process, as shown in Figure 6
In the camera coordinates, the conversion formula between the Cartesian coordinates P = (æ1 1/, 21) and the spherical coordinates (ø. 8) is given by:
{
Φ arctan2 (1) 0 arctan2
E {--T, m]. E [-n/2,
21, of vî
The equirectangular projection is used project spherical coordinates onto plane, which is the display format for panoramic images. The conversion formula between the spherical coordinates (ø, 0) and the Normalised Equirectangular coordinates p' (I2,, 92) is given by:
32
11-1,1]
π
= n/2
E [-11,
The negative sign in the I2 expression is due to the fact that panoramic images display the scene as viewed from the camera optical center to the outside. Therefore, when is positive, it corresponds to the negative half-plane of the Equirectangular plane in terms of 22. The mapping between pixel coordinates (up, Up) and Normalised Equirectangular coordinates (æ2, y2) on the panorama established as:
Up = (#2 + 1)W,
E [0. Wp), Up -
(8)
E [0, H,)-
To obtain the corresponding bird's-eye view of the panorama, place tangent plane the south pole of the spherical imaging plane as a new imaging plane, as shown in Figure (c). The focal length of the BEV in the imaging process is ff = 0.5W tan(fov). The camera coordinate system coordinates (x) y1 2 corresponding to a pixel (un, 21)) on the bird's hird's-eye view (BEV) imaging plane determined as:
Øt 3/1 = Up + W//2
Ho/2
(9)
21 f-
By substituting Equation into Equation 6 and then substituting the result into Equation 7 we can
obtain the Normalised Equirectangular coordinates (2) for pixel on the bird's Bird's-eye view (BEV) image plane. Finally. by substituting the Normalised Equirectangular coordinates into Equation g
we can obtain the pixel coordinates (up: Up) on the panoramic image. This leads to the Spherical Transform mapping formula in Equation!
13

---

P()
y
U. upano
latitude 0
P2Xx.\22) 1
pano
N: = 20/m
longtitude
@ -- X; 0 ys n/2
X
(a) Normalised Equirectangular coordinates
(b) Spherical coordinates
21
beu
Y1
bev
bev
bee
(c) BEV Project
(d) BEV coordinates
Figure 6: Illustration of panorama imaging model and the spherical transform mechanism for projecting panoramas a bird's-ey. s-ede view.
We test the effectiveness of the bird's-eye view projection with different field of view (fov) parameters, as shown in Figure 7 As seen in (d), when fou 85 the field of view of the bird's s-eey view image is similar to that of the corresponding satellite image, and there are few invalid parts in the image. Therefore, we use this parameter for all of our experiments.
The spherical transform projects the ground image to a bird's eye view, while the polar transform (25 or projective transform (24] projects the satellite image to the perspective of the ground image (aligning with a panoramic view). There a growing recognition and research on the role of Bird's Eye View (BEV) representation localization, navigation, and related tasks.
The advantage of projecting to a bird's eye view, as compared to projecting the satellite image to perspective of the ground image, lies in its more intuitive nature for the localization task and the need for only one projection at the start of the localization process. On the other hand, the latter approach requires selecting projection points (initial pose or candidate poses) for projecting the satellite image, and if these points are not accurately chosen, the resulting projection may not align well with the ground image. Additionally, for more accurate localization, the latter approach requires multiple applications of the projection method from different positions, which could potentially make it less real-time compared our method.
A.2 Details for Projecting Front View Images in KITTI
In the KITTI dataset { the ground is front view image, so the process of projecting it onto bird's-eye view is completely different from that of the panorama in VIGOR (42. We illustrate the mechanism of this projection process Figure where AB represents the height the frontal view image, and AC' represents the desired height of the bird's-eye view image.
We define the size of the front view image H W,. and the target size of the bird's view image as H₂ x W.. Using a method similar to that described in Section we imagine a bird's s-ye projection plane parallel the ground and adjacent to the front view image. Then connect the camera optical center with each pixel on the bird's s-eye view image to obtain the corresponding pixel coordinates the front view image.
In Figure (b), for represents the field of view of the front view image (LADC), f represents the focal length DE, h represents the distance from the camera optical center D to the
14

---

(6) Ground Image
Satellite Image
(c) fov 75"
(d) fov 85"
(e) fov 88"
Figure 7: The examples of bird's s-ye view images obtained through the Spherical Transform different field of view (fov) parameters.
Uront
front
D
Vbew
Ubev
6
0
02
(a) Front-View and BEV Image
(b) Geometric Mechanism
Figure 8: Illustration of the projection mechanism for KITTI
projection plane, and 0 represents the angle between the camera imaging plane AB and the vertical direction. To facilitate the subsequent derivation, also define å to represent LFAD), yp to represent LBAD, 82 to represent LAGD, 0g to represent LAHG, f to represent FA, and lo to represent AD. According to 88, for 17.5% in the KITTI dataset. These variables are calculated as follows:
f Hj tan(foe)
2
6 foo
8
(p
(10)
H 2
10
f +
h = 1o sin
f'
6.
We denote the pixel coordinates the bird's seyy' view imaging plane as (u,, v6). The values of 0g and Og can be calculated using the arctangent formula and the properties exterior angle as follows:
02 arctan (h/ (J' + H -b))
(11)
03
+ 0 00-02- 02.
Then, based on the sine and similarity triangle theorems, we can calculate the corresponding pixel coordinates (uy, vt) on the front view image for each pixel on the bird's-eye view image. The calculation is as follows:
H, of
sin 02
Hy - Ub
sin 0g
(12)
W uf
S' + (H, U1) sin
Wo/2 ub
I + Hb Vb
15

---

(a) Front- View Image
(h) Satellite Image
0=)
0 0.8
(e) 1.6'
8 0.8 and Rotate
Figure 9: The examples of bird's s-eye view images obtained from front view image different parameters.
sin 02
Uf H
(Hs Ub) sin 0z
(13)
W I + ( Hb Ub
Uz) sin W, 2
u 2
We introduce the angle & because we found that in practical testing, the imaging plane of the ground camera may not be perfectly vertical, as shown in Figure Based on experimental results, we use 0 = 0.8 for all experiments. During the actual projection process, we set the target size of the bird's-ey. s-eye view image be H, W; approximately 40m x 40m. After the first projection calculation, we directly applied the homography matrix H obtained from the above projection process to obtain the BEV, Then, we used scaling and rotation homography matrices Hi and H2 obtain the final BEV with the specified size.
= Wy) )) x (6* W,). which corresponds to a field of view of
scale 0
cos - sin sin
uo(1 - cos 7) + de sin vo - cos 7) u. sin
H1
scale
0
H2
cos
(14)
0
Here, scale represents the ratio of the current H, x W, to the desired input size of the network (512 x 512). " represents the yaw angle of the ground camera with noise, and (Ney Uo) represent the pixel coordinates of the center the final BEV image.
B
Label Correction
B.1 Label Correction in VIGOR Dataset
During research, found that the ground truth for pixel coordinates of ground images satellite patches in the VIGOR dataset 42] is inaccurate. The VIGOR dataset uses uniform meter-to-pixe resolution for converting the latitude and longitude of ground images to their location in aerial images across all four cities. SliceMatch [12 also discovered this issue and proposed a correction, but their improvement is limited to using different average meter-to- to-pixel resolutions for each city, which may still result in some inaccuracies in different regions of the same city.
We propose the use of Mercator projection [ to directly compute the pixel coordinates of ground images on specified satellite images using the GPS information provided the dataset. Equation allows us calculate the pixel coordinates of a GPS coordinate global scale. We compute the
16

---

(a) Street View
(b) center
(c) Satellte center
(d) Certain Point
Origial Correction
(e) Pixel Label
ni Ground GPS
(g) Satelite GPS
(h) Corresponding GPS
Figure 10: Illustration label problem in VIGOR dataset 42.. In (c), we show the label provided by the VIGOR [2). the label corrected by SliceMatch [22, and the pixel coordinates computed using our method. The space distance between label provided by VIGOR and ours reaches 1.93 m. The pair of (d) and (h) represent the the pixel coordinates obtained by method and their corresponding GPS location on Google Maps, demonstrating the accuracy of our method.
Table 5: Absolute error statistics for labels in VIGOR 42 and SliceMatch 12 in four cities. The absolute error is defined the distance between the original and the corrected locations.
VIGOR (m)
SliceMatch (m)
City
Min.
Mcan
Median
Max.
Min.
Mcan
Median
Max.
Chicago
0.00
0.44
0.45
0.80
0.00
0.10
0.13
0.31
New York
0.00
0.20
0.21
0.39
0.00
0.16
0.16
0.42
San Francisco
0.00
0.42
0.45
0.86
0.00
0.09
0.13
0.27
Seattle
0.00
1.73
1.80
3.13
0.00
0.12
0.13
0.33
pixel coordinates of the satellite patch's GPS label and the pixel coordinates of the ground image's 's GPS label. We then add the difference between them to the center coordinates of the current satellite patch, resulting in accurate pixel coordinates of the ground image on the satellite patch. The result of the correction shown Figure I0, which shows that the label provided by VIGOR has a significant deviation from the true position. The label corrected by MliceMatch [12] reduces this deviation, but there still some error between the corrected label and the true position. The statistical information on the absolute error, measured in meters, between the labels provided in VIGOR [2] and SliceMatch [12 and the corrected labels using our method is presented Table 5
Note that in the model training process, including our method and all other methods, we do not directly use the GPS coordinates for training. This is because if GPS is used directly calculate the training loss, the data truncation problem will occur when using Float32 tensor format since the valid data five decimal places and trigonometric functions are used in the calculation.
B.2 Label Creation in KITTI Dataset
Supervised information required for training network includes the pixel location the bird's ''dd''ss s-eye view image, as well as its corresponding true GPS value. However, obtaining this information directly from the KITTI dataset (8] is not feasible, and need calculate it ourselves.
To obtain the required supervised information, we use the calibration files provided by KITTI and the point cloud data in the dataset. Given a 3D point in Velodyne coordinates, project it to point
17

---

(a) Example
(b) Example 2
(c) Example 3
Figure 11: Three examples of labels created for the KITTI dataset [8 The red dots indicate the location of the same point in the front view, bird s-ree view, and satellite view, respectively.
in the i-th camera image follows:
(1) =
Team X,
(15)
feet roof
where i = 2, (r)ct()), Rrect( and Team are calibration parameters provided in the KITTI dataset.
Additionally, we calculate the distance deviation in terms of latitude and longitude for the 3D point with respect to the origin of the IMU/GPS coordinate system in the world coordinate system. This is represented as Xops:
XGPS
Tw imu T
imu volo X.
(16)
Using the above process, we compute the latitude and longitude deviations for particular 3D point relative to the GPS sensor, and determine its corresponding GPS value using the meter-to-GPS calculation method. Finally, utilize the homography transformation H HyH, H obtained from Section A.2 to map the pixel coordinates to their corresponding pixel coordinates on the BEV image. During model inference, image. Using the inverse process of the above method, we can obtain the latitude and longitude deviations of the corresponding 3D point relative to the GPS sensor, and estimate the GPS value corresponding to the camera. Some examples is shown in Figure
obtain the GPS coordinates of particular pixel on the BEV
C Future Research
In our projection of the panoramic images into a bird's-eye view, we place a tangent plane at the south pole of the spherical imaging plane as the new imaging plane. However, this method assumes that the ground camera oriented vertically, which may not always be the case due roll and pitch deviations. Assuming (a, B, 7) represent the angles of (roll, pitch, yaw). the rotation matrix can be obtained as follows:
Cos y cos B sin cos Ø
cos y sin 3 sin a - sin y cos a sin sin B sin cos sin a
cos y sin 3 cos a + sin cos sin y sin B cos cos y sin cos 3 cos a
y sin a'
R
(17)
sin 3
By multiplying the rotation matrix with the camera coordinates obtained from Equation 9 obtain the new coordinates (xf, ví, 21) in the camera coordinate system. We then proceed with the subsequent calculations obtain the bird's-an s-eye view image under the assumption of the specified (roll, pitch, yaw) angles. Different angle configurations yield different bird's-eye s-ede view images, as illustrated in Figure can be observed that the resulting BEV appearance varies significantly when different (roll, pitch) angles are given. Therefore, we suggest that in future research, (roll, pitch) can be treated as additional predicted outputs to obtain more degrees of freedom in estimating the ground camera pose. Our differentiable spherical transform provides possibility for this idea.
18

---

panorama image
0', B 0-, y 0
0" -176 0
CHEE
panorama image
== = 0.53', B =
Figure 12: Bird's-eye view images obtained under different roll, pitch, and yaw angles using our spherical transform method.
D Broader Impact
This paper introduces innovative approach to achieving high-precision localization of ground cameras, a capability of immense value in applications such as autonomous driving, augmented reality, and mapping services. Accurate localization is crucial for these industries operate safely and effectively. However, it is imperative to acknowledge that the position of a vehicle or user's ser's camera often sensitive and private information. The misuse or abuse of this technology could lead to privacy violations, potentially endangering individuals' security and personal information. We advocate safeguards and regulations to ensure that the privacy and security of individuals are not compromised in the process.
19


Reviews:
[{"review_id": "hsW3e5ksh8", "reviewer": "Reviewer_NnZj", "summary": "The paper proposes a homography estimation-based method for cross-view geo-localization. Contrary to existing methods that tackle the problem as a retrieval problem, the paper proposes to reformulate the problem as homography estimation of aligning the birds-eye-view against the satellite image. To this end, the authors propose a recurrent homography estimation module which learns to estimate the homography matrix given the feature maps of birds-eye-view and satellite images. Experiments show that the method can demonstrate effective performance against the tested baselines in most metrics.", "questions": "Please refer to the 'weaknesses' section.", "limitations": "The authors have not explicitly discussed limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using bird's eye view for matching against satellite images is intuitive, and the proposed method well leverages the formulation by casting the geo-localization problem as homography estimation.\n\n2. Experiments make valid comparisons against both existing geo-localization methods and also on a few image matching-based methods, where the proposed method shows performance enhancements. Notably, the paper shows large amounts of improvements in translation estimation, which I find to be highly practical for geo-localization tasks.\n\n3. Writing is very clear. It was straightforward to follow the paper's motivation and how the proposed method functions as a whole.\n", "weaknesses": "My concerns are two-fold:\n1. While I favor the paper's formulation of aligning bird's eye view against satellite images, I feel the paper has only included a limited number of homography-based baselines. Currently the paper has only made comparisons against homography estimation from local feature maches in Table 3. However there are many more homography estimation methods that could additionally be tested, for example 'Deep Image Homography Estimation' (which is also introduced in the related works section). Adding more homography-based methods for comparisons will better validate why the proposed recurrent homography estimator is needed. Related, how are the homographies estimated from SuperGlue, LoFTR matches? Have the authors used a RANSAC-style estimator to handle outliers? It seems that the paper is currently missing details on this part, which is crucial for local feature matching-based methods to work effectively.\n\n2. I have not fully grasped the reason why the method performs poorly for rotation estimation in KITTI. Is this due to not using the orientation data labels during training? A better clarification of this phenomenon will be helpful for readers to better understant the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1: Comparison with other homography-based methods.**\n\n**A1.1:** Thank you for your constructive suggestion. In the introduction of our paper, we highlight that two significant challenges arise during homography estimation in the cross-view localization task. The first challenge pertains to the substantial presence of occlusions and ambiguous content in the scene (as indicated in L52-L53), while the second challenge is the lack of compact supervision information for homography estimation, i.e., a minimum of four matching point pairs (as stated in L59-L60).\n\nThese challenges are why we did not employ other homography estimation methods. For instance, methods like \"Deep Image Homography Estimation\" require complete supervision. We also considered the use of unsupervised methods, where such techniques generally compute the similarity between two images to obtain a loss. However, our early experiments demonstrated that L1 loss (photometric similarity loss) and SSIM loss (Structural Similarity loss) were not very effective.\n\nFurthermore, our envisioned application scenarios, such as autonomous driving and robot navigation, demand high real-time performance from the network. Hence, methods like those mentioned in the \"Related Work\" section, which are based on GANs and transformers and involve substantial computational overhead, are not suitable. \n\nOne of the most pivotal attributes of our proposed homography estimator lies in its correlation-aware mechanism. Our choice of the Correlation-Aware Homography Estimator is inherently well-suited to our task's demands, encompassing aspects such as weakly supervised training, adeptness in handling occluded scenes, and real-time capabilities.\n\nHowever, we will explore testing other advanced unsupervised Homography Estimation networks in the future.\n\n**Q1.2: How are the homographies estimated from SuperGlue, LoFTR matches?**\n\n**A1.2:** We employed the RANSAC method, specifically applying the findHomography function from the OpenCV library on the matches obtained from SuperGlue and LoFTR, utilizing the cv2.RANSAC parameter. We will add relevant information to the paper\n\n**Q2: Performance of rotation estimation in the KITTI dataset**\n\n**A2:** Thank you for the thoughtful question. Indeed, our method does not utilize orientation labels, unlike the other methods we compared against in Table 2 of our paper. L204-L206 in the main paper detail how we estimate orientations.\n\nOur experimental results highlight that our model can estimate orientation without orientation labels during training, particularly showing a marked improvement in estimation accuracy on the VIGOR dataset compared to previous methods. The diminished performance in orientation estimation on the KITTI dataset may be due to the presence of only front-view images, in contrast to VIGOR's panoramic images. This limited field of view in KITTI restricts the available information, leading to suboptimal performance. For instance, the projected BEV image covers less than one-fourth of the corresponding satellite image, as depicted in Fig 4(d), compared to over two-thirds in the VIGOR dataset, as shown in Fig 1(d).\n\nWe have also tried using orientation labels to guide our model training, employing L1 loss of orientation error, which has led to substantial improvement in orientation estimation on the KITTI dataset. The table below illustrates the enhanced orientation performance compared to training without orientation labels. Even without meticulous hyper-parameter tuning, the orientation estimation has improved significantly. Specifically, under the \"Same\" and \"Cross\" settings, the mean orientation estimation error has decreased by 3.13 degrees and 3.92 degrees, respectively. We have also updated Table 2 to include orientation supervision; please refer to Table 1 in the PDF of the global response.\n\n|Area|Ori. Label| ↓Mean| ↓Median | ↑R@1$^\\circ$  | ↑R@5$^\\circ$ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Same| w/o. | 3.93 | 3.34 | 15.45 | 70.05 |\n| Same| w.  | 0.80(↓ 3.13) | 0.61(↓ 2.73) | 71.87(↑ 56.42) | 99.62(↑ 29.57) |\n| Cross| w/o. | 7.10 | 4.60 | 11.78 | 53.23 |\n| Cross| w.  | 3.18(↓ 3.92) | 1.94(↓ 2.66) | 27.83(↑ 16.05) | 82.21(↑ 28.98) |"}, {"review_id": "JOSrrsghmh", "reviewer": "Reviewer_ZVTo", "summary": "The paper addresses fine-grained cross-view geo-localization task that that matches the camera ground images with a satellite image patch covering the same area to determine the geo-pose of camera. The proposed approach projects ground images onto a bird’s-eye view perspective and formulate the task as a 2D image alignment problem. A correlation-based homography estimation module is proposed to achieve precise localization. Experiments are presented on VIGOR and KITTI datasets.", "questions": "Please see the weaknesses.\n\nI have read the author’s rebuttal and other reviews. I am still not convinced and keep the score the same.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Proposed correlation-aware homography estimation module is interesting\n+ Experiments show promising results in VIGOR and KITTI datasets.", "weaknesses": "The paper has interesting ideas, but I think the paper needs more work to be convincing. The weaknesses are mentioned below:\n\n-- The use of BEV representation has been explored for  Fine-Grained Cross-View Geo-Localization task [Ref1]. This paper should have been discussed and compared in the experiments.\n\n[Ref1] F. Fervers, et. al., Uncertainty-aware Vision-based Metric Cross-view Geolocalization, CVPR 2023\n\n\n-- The novelty behind the spherical transform module to project ground images to a birds-eye-view (BEV) perspective is not clear. Inverse Perspective Mapping (IPM) is commonly used for long time for transforming camera images to BEV [Ref2, Ref3]. Also, looking at the generated BEV images and from experience of using [Ref3] for generation, the BEV images do not seem any better than the quality achieved by applying the IPM technique.\n\n[Ref2] H. A. Mallot, et. al., “Inverse perspective mapping simplifies optical flow computation and obstacle detection,” Biological Cybernetics, 1991\n\n[Ref3] L. Reiher, et. al., \"A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras\nto a Semantically Segmented Image in Bird’s Eye View\", IEEE International Conference on Intelligent Transportation Systems, 2020\n\n-- There are also many works that use neural networks to generate BEV representation from ground camera images. Some of these papers should be discussed. (e.g., [Ref4, Ref5])\n\n[Ref4] Z. Li, et. al., Translating Images into Maps, ICRA 2022\n[Ref5] A. Saha, et. al., BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, ECCV 2022\n\n-- The utilization of infoNCE loss for cross-view image geo-localization is also not new [Ref6, Ref7]\n\nY. Zhu, et. al., Simple, Effective and General: A New Backbone for Cross-view Image Geo-localization, arXiv 2023\nF. Deuser, et. al., Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation, arXiv 2023\n\n\n-- Experiments do not show consistent improvements, as evident from Table 2. There are several cases where other baselines perform better.\n\n-- Not sure of the reproducibility of this work and no code is provided in the supplementary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: \"The use of BEV representation has been explored ...\"**\n\n**A1:** Sorry for the confusion. We'd like to clarify a point. The **BEV (Bird's Eye View) images** used in our paper are explicitly derived by exploiting the geometry of the scene. In contrast, the references [Ref1] and others utilize networks to learn how to project other-view image inputs to BEV representations at the **feature level**. It's important to note that none of these references employ explicit BEV images resembling Fig. 1(b) presented in our paper.\n\nWhile some research focuses on learning BEV representations within the BEV space, we explore the use of BEV images for Fine-Grained Cross-View Geo-Localization. In the Related Work of [Ref 1], it is explicitly mentioned that \"PV2BEV (the perspective view to bird's eye view transformation) methods can be categorized based on whether they explicitly exploit the geometry of the scene to bridge the gap between PV and BEV or learn the mapping in a data-centric manner.\" They follow the second approach, using a spatiotemporal transformer encoder for multi-camera/timestamp to BEV mapping.\n\nOur method uses the first approach. Our proposed pipeline specifically requires BEV images for direct alignment with Aerial imagery, rather than employing a high-level BEV representation (which exists in the feature space).\n\nAs evident from Figure 1 we provided, our BEV method intuitively enhances information alignment with aerial imagery, reducing network complexity.\n\nWe plan to expand the discussion on various BEV presentation methods in the related work section. However, direct comparisons may not be feasible due to fundamental differences in our BEV approach compared to [Ref 1]. Furthermore, [Ref 1] uses datasets(KITTI-360) with multiple vehicle cameras for joint localization involving temporal data. In contrast, our method directly localizes using single panoramic(VIGOR) or frontal-view images(KITTI) aligned with satellite maps.\n\n**Q2: The novelty and advantage of our spherical transform compared with IPM.**\n\n**A2**: Sorry for the confusion. Our spherical transform achieves high-quality BEV images from ground images comparable to IPM without the need for the camera's intrinsic and extrinsic parameters, detailed in the 'A Projection Details' section of Supplementary Materials.\n\nAs mentioned in [Ref1]'s related work, \"IPM transforms PV features to BEV via a homography based on the camera’s intrinsic and extrinsic parameters.\" The IPM method requires knowledge of the camera's intrinsic and extrinsic parameters, which are not available for the VIGOR dataset. Our proposed spherical transform module can project ground images to a BEV perspective without the need for any camera’s intrinsic and extrinsic parameters. \n\nIn conventional IPM, distinct parameters are required for calculating BEV images in distinct cities, tailored to each camera setup. In contrast, our spherical transform enables a unified approach and interface for projecting BEV images.\n\nMoreover, our spherical transform operates efficiently. The pixel correspondences between ground-level images and BEV can be precomputed during initialization. Obtaining the BEV image takes under 1 ms for each processing instance. Moreover, pixel projections are matrix-based and GPU-accelerated in PyTorch, completing in <10ms.\n\n**Q3: \"There are also many works that use neural networks to generate BEV representation ...\"**\n\n**A3**: Please refer to Q1 for the distinction between BEV representation and the BEV image derived from our method. We will add discussions of the differences between BEV representation and our method in our revised paper.\n\nAlso, there is a significant distinction between our explicit method for obtaining BEV images and the BEV representation approaches[Ref1, Ref4, Ref5]. The significance of our method lies in its ability to operate without the need for camera intrinsic and extrinsic parameters, coupled with its rapid processing speed. This allows for seamless integration into applications employing BEV representation methods.\n\n**Q4: \"The utilization of infoNCE loss for cross-view image geo-localization is also not new.\"**\n\n**A4:** The utilization of infoNCE loss is not the main contribution of our work. Despite InfoNCE loss having been previously explored ([Ref6, Ref7]), we share different motivations.\n\nWe highlight its role in homography estimation given limited point supervision. It maximizes label utilization during training. As stated in our introduction, homography estimation typically requires a minimum of four corresponding point pairs for accurate computation. However, the majority of datasets we work with only provide a single pair of points as supervision (i.e., the localization position).\n\nFurthermore, in Section 4.5, we have also indicated that our model training can be accomplished even without this loss, yet still yield competitively robust outcomes (Table 3).\n\n**Q5: \"Experiments do not show consistent improvements, as evident from Table 2.\"**\n\n**A5**: The primary discrepancy in performance in the original paper lies in the orientation estimation. Experimental results demonstrate our model's capability to estimate orientation even without the requirement for orientation labels during training. The introduction of orientation supervision can lead to a significant improvement in the performance of orientation estimation.\n\nWe include an updated Table 2 after incorporating orientation supervision. Please refer to the global response and Table 1 in the uploaded PDF for more details.\n\n**Q6: Reproducibility of our work.**\n\n**A6**: We will release our code, model checkpoints, and training scripts."}, {"review_id": "7Bq8MC3BIK", "reviewer": "Reviewer_NjYP", "summary": "This paper addresses the problem of ground camera pose refinement by ground-to-satellite image matching. For this purpose, this paper proposes to project a ground image to the overhead view image plane by using a Homography, and then iteratively update the residual Homography between the projected overhead view image and the reference satellite image, from which the relative pose between the ground camera and the satellite image can be estimated. Experiments are conducted on two cross-view datasets, VIGOR and KITTI, and the results demonstrate the effectiveness of the proposed method. ", "questions": "Plz refer to my comments on Weaknesses", "limitations": "Using homography estimation between the ground and satellite images ignores the correspondences of objects not in the tomography plane, potentially limiting the performance. \n\n", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ A new pipeline for ground-to-satellite image localization using Homography update is proposed; \n+ The proposed method achieves state-of-the-art performance. ", "weaknesses": "(1) This paper estimates the residual Homography transformation between the reference satellite image and an overhead view image projected from a ground view image. The ground camera's pose is estimated from the Homography. However, no illustrations are about how to estimate the camera pose (rotation and translation) from the Homography. \n\nOne sentence in L205-206 is for orientation estimation. However, it seems that we should select different points from the ground view image and project them to the overhead view by using the estimated Homography, and the orientation is estimated by connecting the two points. If this is the way to estimate orientation, my first question is: \"how to estimate the translation\"? This is not described. My second question is, how to select the two points, randomly? Will this be reliable? The severe occlusion between the ground and satellite images makes that not all ground image pixels are visible in the overhead view and vice versa. How to select/determine points that are co-visible by the two views for this orientation estimation?\n\n(2) If the pose is derived from the estimated Homography, it should be a deterministic solution. How the probability map in Fig.3 is estimated?\n\n(3) The previous work by Shi and Li [22] proposes an iterative optimization network that directly optimizes pose parameters. From a high-level idea, the difference between this paper and Shi and Li is that this paper optimizes Homography parameters. However, the Homography contains 5 degrees of freedom which are larger than that of the pose parameters (3 DoF). What is the superiority of this? What is the performance of the proposed method if we simply modify the Homography parameter output to pose parameter output? and similarly, since this paper uses optical flow for supervision, why not directly estimate the optical flow between the two images and then compute the relative pose from the flow map?\n\n(4) This paper claims that the GPS labels of the VIGOR and KITTI datasets are inaccurate and thus proposes to re-calibrate the pose labels. However, if the GPS labels themselves are inaccurate, the different GPS-to-UTM conversion methods would not correct the error. \nMoreover, any investigations of what kind of GPS conversion methods are used in previous works when introducing the cross-view dataset, don't they also use the Mercator projection or an equivalent projection, or why the Mercator projection used in this paper should provide more accurate UTM coordinates than the methods used in their original papers? \n\nFurthermore, if the ground truth has been modified, it is unfair to compare previous works' results reported in their original papers with the inaccurate poses for training and evaluation. \n\n(5) The network is supervised using ground truth (GT) pixel correspondences. How are the GT correspondences derived, according to the GT relative RT? However, as I mentioned before, due to the severe occlusions between ground and satellite images, not every pixel in the ground view is visible in the overhead view and vice versa. Thus, there must be incorrectly derived GT correspondences if all the pixels in the overhead/satellite images are used. \n\n(6) For the ground-to-satellite projection, this paper assumes the satellite image has a pin-hole camera projection with an FoV of 85-degree. However, Shi et al. [26] approximate a parallel projection for the satellite images. What is the superiority of this pin-hole camera projection over the parallel projection, and why use an FoV of 85 degrees instead of other numbers? \n\n(7) The writing needs to be thoroughly improved and should be more accurate. For example:\n    (i) Plz do not simply use a number [*] for references as a component in a sentence without mentioning the author's names.\n    (ii) Sentence in L33 is not strictly correct. Slicematch is also a kind of \"repeat (pose) sampling\".\n    (iii) Sentence in L144 \"without any intrinsic parameters\" is very ambiguous, this paper does use focal length and FoV for satellite images (L136-137), and the ground-to-satellite tomography-based projection should also use the ground camera's intrinsic when it is a pin-hole camera as in the KITTI dataset. \n    (iv) The symbols in Sec 3.2 are very confusing. The authors mixed symbols for three images: the original ground image, the projected overhead view image from the ground-view image, the satellite image. This section needs to be thoroughly revised. \n    (v) ...\n\n(8) Missing important references:\n     (i) Fervers, Florian, et al. \"Uncertainty-aware Vision-based Metric Cross-view Geolocalization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. (available on arxiv in 2022, before SliceMatch [11])\n     (ii) Shi, Yujiao, et al. \"Where am i looking at? joint location and orientation estimation by cross-view matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n     (iii) Shi, Yujiao, et al. \"CVLNet: Cross-view Semantic Correspondence Learning for Video-Based Camera Localization.\" Asian Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n     (iv) Vyas, Shruti, Chen Chen, and Mubarak Shah. \"Gama: Cross-view video geo-localization.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n     (v) Zhang, Xiaohan, Waqas Sultani, and Safwan Wshah. \"Cross-View Image Sequence Geo-localization.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed reviews. Before addressing each individual question, we want to briefly introduce the pipeline of our proposed method to better clarify the technical details of our method.\n\n1. We use a spherical transform to project ground images onto a bird's-eye view (BEV). Subsequent operations are on the BEV image and the satellite image only. The projection effect is shown in Figure 1(a)(b).\n\n2. The Homography Estimator yields a matrix. With this matrix, we align the BEV and satellite images(Figure 1(b)(c)), creating overlap (Figure 1(d)).\n\n3. **(3.1) Translation:** After aligning the two images, we can project the center point of the BEV image onto the satellite image (L189), derive its pixel coordinates, and obtain GPS via the inverse of Equation 4. The location of the BEV center point corresponds to the camera position. **(3.2) Orientation:** We project both the center point of the BEV image and another point along its vertical centerline onto the satellite image. Connecting these projected points establishes the camera's orientation. This process is analogous to determining a direction on a map by connecting two points: your current location and a point directly ahead on your path.\n\n4. Training labels: Our method, like others, avoids direct GPS training. Instead, relies on pixel coordinate labels. Prior methods used meter-to-pixel resolutions, posing city-specific challenges. In contrast, our Mercator-based approach is universally applicable and accurate across cities.\n\n**Q1.1: How we estimate the camera pose (T, R) through Homography.**\n\n**A1.1:** See 3.1, 3.2 of the above pipeline.\n\n**Q1.2: \"how to select the two points ... reliable? ... How ... co-visible by the two views ...\"**\n\n**A1.2:** See 3.2 above for selecting points. Correlation-aware mechanism aligns correlated portions of BEV and satellite images, reducing non-visible area impact(L53-57).\nCo-visibility between points isn't strictly required for translation/orientation estimation via homography. So, there should not be a concern about the reliability of selected points. In Fig 1(d), the vehicle in BEV isn't visible in the corresponding satellite image. Using homography from visible parts alignment, we can align the vehicle's position on the satellite image and get its GPS.\n\n**Q2: How the probability map in Fig.3 is estimated?**\n\n**A2**: The Homography-derived pose is deterministically derived.\nThe probability map doesn't affect pose estimation. Map gauges localization confidence, crucial for navigation tasks. It's a correlation map between the BEV center and satellite points, an intermediate in our network. Part 2 of Figure 2 illustrates map generation.\n\n**Q3: Why deriving Homography instead of direct pose? \"Why not directly estimate the optical flow.\"**\n\n**A3**: Our network does not use optical flow supervision. We use a pair of pixel coordinates obtained from the localization point.\n\nHomography aligns images with geometry, uniquely solving pose. There is no need to learn Homography-to-pose transition. Our method can simplify networks and yield analytical solutions.\n\nShi and Li's [22] needs camera parameters (Equation 3) for projection and invokes the Geometry Projection model iteratively. Our method doesn't need parameters and repetitive projection.\n\nOur data lacks optical flow supervision and optical flow estimation is inconsequential as we only use two points for translation and orientation.\n\n**Q4: \"Claims that the GPS labels ... inaccurate\" and \"... provide more accurate UTM coordinate ...\"**\n\n**A4:** We **do not** consider GPS labels problematic. Direct use of GPS info in training is challenging, necessitating pixel labels. VIGOR used uniform meter-to-pixel conversion across cities to calculate pixel-level labels from GPS, leading to significant errors(Sup. L73-75). \n\nSliceMatch highlights this, improving by calculating city-specific resolutions but still lacking precision. They mention in Sup.: \"VIGOR dataset have used a ground resolution equal to 0.114m/pixel for all 4 cities\" \"We have calculated a new ground resolution for each city by averaging the ground resolutions.\"\n\nVIGOR and KITTI satellite images are from Google Maps, which uses Mercator. So Mercator is more accurate.\n\nSliceMatch fixed VIGOR labels, CCVPE uses them. Table 5 in our Sup. shows minor differences, averaging <0.2m vs. ours. Ours is universal and convenient.\n\n**Q5: \"The network ... pixel correspondences. ... are used.\"**\n\n**A5**: For network supervision, we use one-pixel correspondence, not all image pixels. BEV center corresponds to camera GPS; Mercator gives pixel label from GPS.\n\n**Q6: \"Assumes the satellite image has a pin-hole camera projection with an FoV of 85-degree.\"**\n\n**A6:** Our paper **do not** assume satellite imaging. We use the ground camera's imaging model for generating BEV. Our projection doesn't need repeated sampling for pose estimation like Shi et al.'s [26]. \n\nThe \"FoV of 85 degrees\" parameter affects BEV image FoV, as shown in Fig 7. We chose 85 degrees to achieve the optimal alignment with satellite map FoV (Sup. L34-37).\n\n**Q7: Writing.**\n\n**A7:** Thank you for your suggestion, but there are some misunderstandings as listed below.\n\n(i) Thank you. We will revise our paper.\n\n(ii) Our 'repeat sampling' means involving full process per pose, SliceMatch includes all candidate poses initially. So it is not \"repeat sampling.\"\n\n(iii) No calibrated parameters are needed. See A6 and Sup. \"A Projection Details.\"\n\n(iv) Sec 3.2 only alligns BEV and satellite images(subscripts 'g' and 's'). Mixed symbols are misunderstood. We didn't use the original ground image in sec 3.2.\n\n**Q8: References.**\n\n**A8:** We will add these papers in the revised paper. \n\n**Limit.1: \"Using homography  ...\"**\n\n**A9:** Our method handles discernible contours, even those not co-planar. Co-visible regions yield good outcomes. Experimental results show lower errors, confirming robustness and efficacy."}, {"review_id": "0kXoMsJP57", "reviewer": "Reviewer_Ak83", "summary": "This paper propose to wrap the ground-view image to align with the corresponding aerial-view image using homography estimation. Firstly, a differentiable spherical transform is adopted to align the perspective between the ground-view and aerial-view images. Then a correlation-based homography estimator is proposed to align the similar parts of the transformed ground image and aerial-view image. It achieves state-of-the-art performance on VIGOR and KITTI datasets with a speed of 28 FPS.", "questions": "See limitations and weaknesses. I am open to raise my rating after rebuttal.", "limitations": "- The spherical transform is applied on query imaged on-the-fly, which might be a disadvantage compared to other transforms on aerial-view images, but the method is still fast. Is there a speed comparison with other methods? I think this could a limitation that needs to be discussed. \n- The authors could add more discussion on positive/negative societal impact in supplementary materials. I think this method should have some potential impact on industry.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The idea is very interesting and the writing is good.\n+ The presentation of figures and tables are also good.\n+ The evaluation is comprehensive and the performance is promising.\n", "weaknesses": "- The figure looks good, but I am not able to get the detailed flow of data. The part 2 and part 3 of Fig 2 are connected only on the correlation map. Would the code be released? It would be better to add this claim to help reproduce the result.\n- The authors could add some insight about why the spherical transform is better than polar transform/ projective transform in the other papers. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The detailed flow of data between part 2 and part 3 of Fig 2 and code.**\n\n**A1:** Thank you for the thoughtful question. The mentioned \"part 2\" refers to the computation of the homography matrix between the BEV image and its corresponding satellite image, denoted as $H^k$ in the diagram. The resulting matrix is subsequently passed to \"part 3\" for the computation of the loss. \nIn \"part 3,\" the final homography matrix obtained from \"part 2\" is denoted as $H^{itr}$.\n\nWe appreciate your interest. We will release our codes. \n\n**Q2: Why the spherical transform is better than the polar transform/ projective transform.**\n\n**A2**: Thank you for your constructive suggestion. Firstly, our spherical transform projects the ground image to a bird's eye view, while polar transform or projective transform projects the satellite image to the perspective of the ground image (such as aligning with a panoramic view). There is a growing recognition and research on the role of Bird's Eye View (BEV) representation in localization, navigation, and related tasks.\n\nThe advantage of projecting to a bird's eye view, as compared to projecting the satellite image to the perspective of the ground image, lies in its more intuitive nature for the localization task and the need for only one projection at the start of the localization process. On the other hand, the latter approach requires selecting projection points (initial pose or candidate poses) for projecting the satellite image, and if these points are not accurately chosen, the resulting projection may not align well with the ground image. Additionally, for more accurate localization, the latter approach requires multiple applications of the projection method from different positions, which could potentially make it less real-time compared to our method.\n\nFurthermore, in contrast to the polar transform methods, our spherical transform is derived from the imaging model of the ground camera, adhering closely to geometric principles, while the polar transform is considered a rough geometric alignment.\n\nIn comparison to traditional Inverse Perspective Mapping (IPM) methods, our approach excels by not requiring any camera intrinsic or extrinsic parameters, thereby offering strong generality and versatility. It is precisely this advantage that allows us to project the panoramic images from the VIGOR dataset, which lacks camera calibration parameters, to a bird's-eye view perspective.\n\n**Limit.1: The speed comparison between spherical transform and other methods.**\n\n**A3**: We have evaluated the runtime of our spherical transform module.\nOur spherical transform module operates efficiently. Whether applied to the panoramic images in VIGOR or the frontal images in KITTI, the pixel correspondences between ground-level images and the bird's-eye-view can be precomputed during initialization. Subsequently, by employing `torch.nn.functional.grid_sample`, obtaining the corresponding BEV image takes under 1 ms for each processing instance. \nFurthermore, pixel projection calculations are matrix-based and GPU-accelerated using PyTorch, requiring less than 10ms to complete.\n\nAfter the acceptance of our paper, we plan to release our code as open source. Additionally, we will provide a Colab link for those who are interested in exploring and evaluating our spherical transform module. \n\n**Limit.2: Potential impact on industry.**\n\n**A4**: Thank you for your constructive suggestion. We will add potential impact on the industry in supplementary materials. Here we briefly discuss the positive impact of our method.\n\nWe believe that our approach holds significant promise for a range of sectors, particularly in domains closely tied to autonomous driving, navigation, and geospatial technology. By introducing a novel pipeline, our method has the potential to redefine workflows within these industries, opening up exciting avenues for exploration and innovation. "}]

Author rebuttal:
We extend our sincere thanks to all reviewers for their insightful feedback. Our method has been recognized as "interesting" (Reviewer Ak83, ZVTo), "new" (Reviewer NjYP), and "intuitive" (Reviewer NnZj). We are gratified that the "large amounts of improvements" (Reviewer NnZj), "state-of-the-art performance" (Reviewer NjYP), and "promising results" (Reviewer ZVTo, Ak83) of our method, along with the "comprehensive evaluation" (Reviewer Ak83), have been collectively acknowledged. The clarity of writing, as well as the presentation of figures and tables, have also been commended (Reviewer NnZj, Ak83).

We have addressed individual comments in the following sections and hope our responses satisfy any concerns.

---
**For Reviewer NnZj and Reviewer ZVTo**

In Table 1 of the newly uploaded PDF, we have included an updated version of Table 2 from our main paper, utilizing orientation labels (without meticulous hyper-parameter tuning). This update demonstrates significant improvement in orientation estimation. We will include relevant details in our revised paper.

We would like to draw your attention to the SliceMatch metrics, which were directly borrowed from the original paper. However, the two red-marked values are likely erroneous, as they are identical. The values for R@$1^{\circ}$ and R@$5^\circ$ should not be equal, and based on the comparison with our method on other metrics, it is likely that the value for R@$1^{\circ}$ is incorrect. We suspect this value may have been copied inaccurately.

Additionally, it's worth noting that CCVPE is our concurrent work and has not yet undergone peer review. Compared to SliceMatch (CVPR 2023), our model consistently exhibits substantial improvements across all metrics.

---
Thank you once again for all your constructive insights!

Meta-review: Reviewers are split on the paper: weak accept, borderline accept, and 2 rejects. Authors provided extensive rebuttals but all the reviewers kept the original ratings even after the rebuttals and the discussions. The area chair looked at the paper and the reviews carefully. In the end, the area chair recommends acceptance for the following reasons: 1) The negative reviews are sometimes due to misunderstanding of the paper; 2) Many concerns of the reviewers appear nit-picking, where authors' rebuttal sounded more convincing; and 3) The other two positive reviewers are excited with the paper. Please read all the reviews carefully and prepare the camera ready by addressing them.
Decision: Accept (poster)