Title: Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator
Venue: NeurIPS 2023
Accepted: yes
Keywords: Fine-Grained Cross-View Geo-Localization

Homography Estimation
Abstract: In this paper, we introduce a novel approach to fine-grained cross-view geo-localization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to an image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by 21.3\% and 32.4\% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4\% on the KITTI benchmark in same-area evaluation.

Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator

Abstract

In this paper, we introduce a novel approach to fine-grained cross-view geolocalization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to an image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms stateof-the-art techniques, reducing the mean metric localization error by 21.3% and 32.4% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4% on the KITTI benchmark in same-area evaluation.

1 Introduction

Accurate localization of ground cameras is essential for various applications such as autonomous driving, robot navigation, and geospatial data analysis. In crowded urban areas, cross-view localization using satellite images has proven to have great potential for correcting noisy GPS signals [3, 38] and improving navigation accuracy [42, 13, 19]. In this paper, we consider the task of fine-grained cross-view geo-localization, which estimates the 3-Dof pose of a ground camera, i.e. , GPS location and orientation (yaw), from a given ground-level query image and a geo-referenced aerial image.

Previous works on coarse cross-view localization [10, 16, 25, 11, 42, 37, 41] have achieved high recall rates by formulating it as an image retrieval problem. However, the localization accuracy obtained using this method is limited by the segmentation size of satellite image patches, resulting in localization errors of tens of meters.

Recently, there has been a growing interest in fine-grained cross-view geo-localization, which assumes the availability of the ground image and a corresponding GPS-labeled satellite image patch covering the same area. Existing methods can be divided into two categories: those based on repeated sampling [27, 24, 7, 26] and those employing descriptor candidates splitting from satellite image features

(b) BEV Image

Spherical

Transform

GT

★

Ours

x

CCVPE

[36, 12, 35]. Sampling-based methods transform satellite or ground images (or feature maps) to compare with another view, involving the multiple applications of geometric transformations based on potential ground camera poses. However, these methods suffer from low localization accuracy [27, 24, 26] and prolonged time consumption[27, 24, 7, 26]. [36, 12, 35] split the features of satellite images into numerous candidates to obtain more effective and refined descriptors for comparison with the ground image descriptor. Their limited pixel-level localization resolution and high memory usage present challenges for deployment in real-world end-terminals. Hence, the pursuit of a methodology that encompasses swift computation, low memory utilization, and elevated levels of both localization accuracy and resolution remains a paramount area of research.

We observe that projecting ground images onto a bird's-eye view perspective can make satellite-based localization tasks more intuitive, similar to the way humans use maps for navigation, eliminating the need for multiple sampling or further splitting of satellite patches. In this study, we develop a spherical transform module that leverages the imaging model of ground cameras to project panoramic images onto an aerial perspective, effectively bridging the gap between distinct viewpoints. As depicted in Figure 1 (b), our approach does not necessitate prior pose information, and it facilitates the straightforward determination of the ground image's position within the satellite image (c). By transforming the complex cross-view localization problem into a 2D image alignment problem, our method enables the acquisition of precise ground GPS location and orientation.

The core problem in 2D image alignment lies in obtaining the homography transformation between two images. Given that feature-based methods like [22, 28] may not yield effective results in obstructed or unclear scenes, we employ a correlation-aware homography estimation approach. We utilize a recurrent convolutional neural network block to maximize the correlation between similar regions in the feature maps, unlike previous iterative optimization methods [24], which minimize overall feature differences. Our method ensures low correlation in unobservable areas, minimizing their impact on homography estimation, resulting in an end-to-end network that aligns the most similar parts of the transformed ground image with the satellite image, directly outputting its GPS location. Moreover, our approach has successfully overcome the challenge of lacking compact supervision information in homography estimation, i.e. , the absence of at least four matching point pairs. While utilizing the VIGOR dataset [42], we also address the inherent errors in the original ground truth labels to facilitate further research.

The main contributions of this paper include:

A novel and effective method for fine-grained cross-view geo-localization, which strictly aligns ground and aerial image domains using a geometry-constrained spherical transform, reducing the problem to 2D image alignment.

A correlation-aware homography estimation module that eliminates repeated sampling and disregards unobservable content in the images, resulting in excellent generalization performance and rapid inference speed at 30 FPS.

Extensive experiments demonstrating that our method outperforms the state-of-the-art on two fine-grained cross-view localization benchmarks. Specifically, on the same-area and cross-area splits of the VIGOR benchmark [42], our method reduces the mean localization error by 21.3% and 32.4%, respectively. On the same-area split of the KITTI benchmark [8], our method reduces the mean localization error by 34.4%.

2 Related Work

Cross-view image retrieval methods for geo-localization, which use ground images as queries and all patches in a satellite image database as references, have been studied for years and rely on global image descriptors for successful retrieval. Early works [15, 34, 33, 31, 39] suffer from low retrieval accuracy due to significant appearance gaps and poor metric learning techniques. SAFA [25], Shi et al . [27], L2LTR[37], and TransGeo[41] have improved localization accuracy by employing polar transform algorithms, considering orientation, and using transformers [32].

Fine-grained cross-view localization beyond one-to-one retrieval is first proposed in CVR[42] and they propose a corresponding benchmark VIGOR. [27] and [24] project the satellite view using the candidate poses or the iterative optimization method and select the one that is most similar to the ground as the localization result. [7] and [26] employ transformers to acquire BEV representations of ground images, and then transform BEV maps by all possible ground camera poses to compare with the feature representations of satellite images. However, there remains untapped potential for refinement, and their inference entails prolonged time. [36] splits the features of satellite patches into several sub-volumes. [12] considers orientation and geometric information to create effective satellite descriptors for a set of candidate poses. [35] incorporates orientation and employs a coarse-to-fine manner to enhance accuracy. Despite these improvements, all three methods face resolution and high memory usage challenges stemming from the splitting or candidate selection process.

Bird's-Eye View Perspective has been thought of as the preferred representation for many tasks ( e.g. navigation or localization). Recent methods [14, 7, 26] utilize transformers to project feature maps of ground images onto BEV. However, there has been no utilization of explicit BEV images for fine-grained cross-view geo-localization. A challenge could stem from the requirement of known camera calibration for traditional Inverse Projective Mapping methods, as [23] used. In an attempt to enhance BEV representations, Boosting [26] also explores geometric-based methodologies, but due to the necessity for depth information, transformers remain a crucial tool to handle ambiguity.

Homography estimation is the process of estimating a mapping between two images of a planar surface from different perspectives. DeTone et al . [6] is the first to propose a deep learning-based approach to homography estimation. Nie et al . [20] proposed the use of a global correlation layer to address the problem of small overlapping regions between the two images. Zhao et al . [40] incorporate the Lucas-Kanade (LK) algorithm [18] as a non-trainable iterator and combine it with CNNs. They also investigate GPS-denied navigation using Google static maps and satellite datasets based on this method. Inspired by RAFT [30], IHN [4] designed a completely iterative trainable deep homography estimation network, achieving high accuracy but requiring strong supervision. HomoGAN [9] designed an unsupervised homography estimation network based on transformer and GAN methods, achieving promising results while consuming more computational resources.

3 Method

This paper presents a novel approach to address the task of fine-grained cross-view localization, as illustrated in Figure 2. Our H omography estimation-based C ross-view geo-localization Net work, named HC-Net, takes corresponding spherical-transformed ground image and GPS-tagged satellite image as input and outputs the homography matrix between them, as well as the precise GPS location and orientation of the ground camera. The following sections introduce the spherical transform for projecting panoramas to a bird's-eye view (Section 3.1), the homography estimator for precise alignment of ground BEV images with satellite images (Section 3.2), and the supervision method of our network (Section 3.3).

3.1 Spherical Transform

Panoramic imaging model Panoramas capture a full 360-degree view on a spherical projection plane and use equirectangular projections for display, as shown in Figure 2 (a). We represent points in the ground camera coordinate system as P = ( x 1 , y 1 , z 1 ) and projected points in the normalized equirectangular coordinate system of the panorama as P ′ = ( x 2 , y 2 ) . The spherical coordinates ( ϕ, θ ) are acquired from ( x 1 , y 1 , z 1 ) using inverse trigonometric functions, with north latitude and east longitude being positive. And the projection between the spherical coordinates ( ϕ, θ ) and the normalised equirectangular coordinates ( x 2 , y 2 ) is expressed in Figure 2 (a), (b).

x2 =

-

φ

/

π

y2 = 2

�

/

π

φ

=

-

x2

π

�

= y2

π

/2

(d) BEV coordinates

�풃��

�풃��

0. Spherical Transform

u' k

2

1

0

2

1

0

2

1

0

2

２ ２

1

１ １

0

０ ０

1/32

1.Feature Extraction

��

,

��

��

,

��

∙

,

∙

2. Recurrent Homography Estimation

4D Correlation Volumes

�

correlation map

H

W

W/2

H/2

u

2

1

0

2

1

0

2

1

0

2

２ ２

1

１ １

0

０ ０

v

coordinates set

X

P

� �-

1

v' k

S

r

r

C

�� �

� �

,

�

1/

2,�

╋

� �-

1

� �

Calculate

Homography

� �

backbone

퐺�����

Correlation Pyramid

�

,

�

1/

2

3. Correlation-Aware Supervision

2

H W

r

r

Correlation

Pooling

Sampling

S

Projection

P

Concatenation

C

Add

╋

Recurrent Part

coordinates set

�

'

�

GPS

P

� ���

�� ��

퐺�����

��

∗

��

∗

Distance

Loss

Loss

�� ��

╋

����

�����

�표��

latitude

θ

longtitude

φ

z

1

x

1

y

1

P(x

1

,y

1

,z

1

)

(b) Spherical coordinates

0

(c) BEV Project

z

1

x

1

y

1

�풃��

��풂��

fov

f

(x

1

,y

1

,z

1

)

�풃��

�풃��

0

(a) Normalised Equirectangular

coordinates

x

2

-1

1

1

-1

��풂��

��풂��

y

2

P'(x

2

,y

2

)

0

Orientation

Loss

����

Spherical transform To obtain the corresponding bird's-eye view of the panorama, we place a tangent plane at the south pole of the spherical imaging plane as a new imaging plane, as shown in Figure 2 (c). Formally, let H p × W p be the size of the panorama and H b × W b be the target size of the bird's-eye view after the spherical transform. We connect the camera's optical center with each pixel point on the BEV imaging plane, determining the corresponding pixel position in the panorama through the intersection of the connection line with the spherical imaging plane. We demonstrate the pixel coordinates on the bird's-eye view imaging plane as ( u b , v b ) . The camera coordinates ( x 1 , y 1 , z 1 ) corresponding to the pixel coordinates ( u b , v b ) can be directly obtained. We set the parameter fov = 85 ◦ to determine the field of view of the bird's-eye view. The focal length of the BEV in the imaging process is f = 0 . 5 W b / tan( fov ) . Therefore, each corresponding panoramic image coordinates ( u p , v p ) is established as:

After the spherical transform, the ground image is projected into the aerial image's perspective, as seen in Figure 2(d). This approach enables high-resolution localization without multiple sampling or splitting of satellite patches. Unlike the satellite-based image projection discussed in [27, 24], this type of projection does not require the selection of a projection point. We adopt a similar approach for the KITTI dataset [8] by setting up an overlooking imaging plane. This allows us to project front-view images into a bird's-eye view without any known camera calibration, as shown in Figure 4 (a), see details in Supplementary Material. Despite being applied as a preprocessing step in our method, note that the transform is differentiable. This property opens up possibilities for future research on the 5-DoF pose of the ground camera (details in Supplementary Material).

3.2 Homography Estimator

To achieve precise localization by aligning the transformed ground image I g with the satellite image I s , we propose a correlation-based deep homography estimator inspired by IHN [4]. We resize the two images I g and I s to the same size and feed them into a Pseudo-Siamese CNN to extract features. These features are denoted as ˜ F g ∈ R D × H × W and ˜ F s ∈ R D × H × W . As shown in Figure 2, after feature extraction, correlation computation, and recurrent homography estimation, the homography matrix H between I g and I s can be obtained.

Correlation Computation Given the feature maps ˜ F g and ˜ F s , the correlation volume C ∈ R H × W × H × W is formed by taking the dot product between all pairs of feature vectors as:

To enlarge the perception range within a feature scale, we conduct average pooling on C at the last 2 dimensions with stride 2 to form another correlation volume C 1 2 ∈ R H × W × H/ 2 × W/ 2 .

Recurrent Homography Estimation We refine the estimation of the homography through loop iterations. The coordinates set on ˜ F g are denoted as X ∈ R 2 × H × W , and the coordinates set on ˜ F s , which is projected from X using the homography matrix H , is denoted as X ′ ∈ R 2 × H × W . For each coordinate position, we denote x = ( u, v ) in X and x ′ = ( u ′ , v ′ ) in X ′ . In each iteration, we use the Equation 3 to calculate X ′ k and use it to sample the last two dimensions of C , C 1 2 with a local square grid of fixed search radius r , resulting in correlation slices S k and S 1 2 ,k of size H × W × r × r .

Then we utilize a convolutional neural network module for residual homography estimation. As in [4], we parameterize the homography matrix using the displacement vectors of the 4 corner points of an image, namely the displacement cube D . In iteration k , we feed the concatenated correlation slice S k , S 1 2 ,k , the coordinates set X , and the currently projected coordinates set X ′ k into the CNN module. The module is mainly composed of multiple convolutional units until the spatial resolution of the feature map reaches 2 × 2 , where each unit downsamples the input by a scale of 2. Then a 1 × 1 convolutional unit projects the feature map into a 2 × 2 × 2 cube ∆ D k , which is the estimated residual displacement vector of the 4 corner points. In iteration k, we update D k by adding the estimated residual displacement vector ∆ D k to D k -1 . Using D k , we can obtain the homography matrix H k through the direct linear transform [2]. The updated H k is then used to project X in the next iteration.

3.3 Network Supervison

Label Correction We use the VIGOR dataset [42] for fine-grained cross-view localization training and evaluation. However, the original labels in [42] contain errors up to 3 meters due to the use of approximate and consistent meter-to-pixel resolutions of the aerial images. Although [12] attempts to correct the labels, they require city-based computations or selections for resolution, which brings significant inconvenience. In our approach, we propose using the Web Mercator projection [1] used by virtually all major online map providers to accurately convert GPS coordinates to pixel coordinates on satellite patches, thereby enhancing generality and convenience. The main equation is as:

where zoom indicates the zoom level for satellite images ( zoom = 20 in VIGOR and zoom = 19 in KITTI) and 256 is the resolution of each satellite tile. We have applied the same method to create training labels for KITTI dataset [8], see Supplementary Material for details.

Loss Function Using the homography matrix H obtained in Section 3.2, we project the center point of BEV onto the satellite image to obtain the localization pixel ( u s , v s ) . Using Equation 4, we calculated the pixel coordinates of the ground truth GPS corresponding to the satellite image as ( u ∗ s , v ∗ s ) . Furthermore, we use Equation 4 to project another point on the BEV centerline onto the satellite image. Connecting this point with ( u s , v s ) allows us to determine the orientation of the ground camera as θ .

We use a hybrid loss function, defined as L = α 1 L dis + α 2 L ori + α 3 L info , to guide our training. Here, L dis = ( u s -u ∗ s ) 2 + ( v s -v ∗ s ) 2 represents the L2 norm loss between the predicted result and the ground truth. L ori = | θ -θ ∗ | represents the L1 norm loss between the predicted θ and the ground truth θ ∗ . To generate a probability distribution that can be further utilized for localization, we propose the use of an infoNCE loss [21] to reinforce the correlation between the BEV point used for localization and the satellite image. L info is defined as:

where ( i c , j c ) represents the center coordinates of ˜ F g , and ( k + , l + ) represents the downsampled position of ( u ∗ s , v ∗ s ) in the satellite feature map ˜ F s . The hyper-parameter τ is introduced to adjust the sharpness of the resulting probability distribution.

Finally, through the inverse process of Equation 4, we can determine the GPS coordinates corresponding to ( u s , v s ) , thereby obtaining highly accurate localization outputs.

4 Experiments

In this section, we first introduce two used datasets, evaluation metrics, and implement details of our network. We then compare the performance of our HC-Net to state-of-the-art and examine its ability to generalize to new measurements within the same areas, across different areas, and across datasets. Finally, we present ablation studies and computational efficiency analysis.

4.1 Datasets and Evaluation Metrics

VIGOR dataset [42] contains geo-tagged ground-level panoramas and aerial images collected in four cities in the US. Each aerial patch corresponds to a ground area of approximately 70 m × 70 m . From Figure 1, it can be observed that the effective field of view of the ground image is slightly smaller than this range. A patch is considered positive if its center 1 / 4 region contains the ground camera's location, otherwise, it is semi-positive. In our experiments, we use positive aerial images for training and testing all models. The panoramas are shifted based on orientation information to align North in the middle, indicating that the orientation prior is known. During training, we introduce ± 45 ◦ noise to the orientation prior to generating orientation labels. The dataset provides 105, 214 panoramas for the geo-localization experiments. We adopt the Same-Area and Cross-Area splits from [42]. For validation and hyperparameter tuning, we randomly select 20% of the data from the training set, as done in[36, 12, 35]. Compared to [12], we have more accurately corrected the ground truth labels in [42], as mentioned in Section 3.3.

KITTI dataset [8] contains ground-level images captured by a moving vehicle with a forward-facing viewpoint, which is a restricted viewpoint. [24] augments the dataset with aerial images. Each aerial patch corresponds to a ground area of approximately 100 m × 100 m . The Training and Test1 sets consist of different measurements from the same region, while the Test2 set has been captured in a different region. As assumed in [24], ground images are located within a 40 × 40 m area in the center of the corresponding aerial patches, and there is an orientation prior with noise between ± 10 ◦ .

Evaluation metrics in our experiments are the mean and median error between the predicted and ground truth over all samples for both localization and orientation separately in meters and in degrees. Following [35], for the KITTI dataset, we additionally include the recall under a certain threshold for longitudinal (driving direction) and lateral localization error, and orientation estimation error. Our thresholds are set to 1m and 5m for localization and to 1 ◦ and 5 ◦ for orientation estimation.

4.2 Implementation Details

Our network uses EfficientNet-B0 [29] with pretrained weights on Imagenet [5] as both the ground and aerial feature extractors, with non-shared weights. The satellite image and bird's-eye-view (BEV) transformed from the ground image both have a size of 512 × 512 on both the VIGOR [42] and KITTI [8] datasets. PyTorch is used for network implementation, and training is done using the AdamW [17] optimizer with a maximum learning rate of 3 . 5 × 10 -4 . The network is trained with a batch size of 16 and a training iteration of 180000 . We set the search radius of the correlation updater r = 4 and set α 1 = 0 . 1 , α 2 = 10 , α 3 = 1 . 0 , τ = 4 in the loss function. The total iteration K = 6 / 10 and the feature map size D × H × W = 320 × 16 × 16 / 320 × 16 × 16 on VIGOR/KITTI.

4.3 Comparison with State-of-the-Art Methods

VIGOR dataset On the VIGOR dataset, we compare our method against several state-of-theart methods: CVR [42], SliceMatch [12], Boosting [26] and CCVPE [35]. For the performance evaluations of CVR [42], SliceMatch [12], and Boosting [26] with known orientation priors, we directly utilize the results provided by CCVPE or the corresponding paper. Regarding most stateof-the-art CCVPE [35], we executed its official code to obtain results not presented in the original

(a) example 1

(b) example 2

(c) example 3

(d) example 4

1D

Ours(5.19e-2)

Ours (5.9e-2)

Ours(1.5e-2)

Ours(4.47e-2)

×

CCVPE(1.16e-4)

CCVPE(3.1e-3)

CCVPE(3.1e-3)

CCVPE(1.60e-5)

paper and used the published results where applicable. We also re-train and evaluate CCVPE with our corrected labels to ensure a fairer comparison. We conduct a comprehensive evaluation with different levels of noise in the orientation prior to ground images, including 0 ◦ , ± 20 ◦ , ± 45 ◦ .

As shown in Table 1, our method outperforms all previous methods in both same-area and cross-area settings in terms of the mean localization error metric, which suggests that our method can cope with more challenging scenarios, as shown in Figure 3(a). The mean localization error is reduced by 21.3% and 32.4% respectively. The previously best-performing method [35] shows a notable performance gap between same-area and cross-area settings, while our method significantly narrows this gap. This suggests that our method possesses superior generalization capabilities. We further demonstrate our superior generalization ability across datasets in Section 4.4.

When prior information exhibits varying degrees of noise, our method still outperforms CCVPE [35] and provides a more accurate estimation of the orientation. Moreover, our network generates confidence probabilities for localization results, allowing us to filter out potential misestimations. As shown in Figure 3 (c), when the scene contains a symmetric layout like zebra crossings and noise exceeds 90 ◦ , the homography estimation may incorrectly align the ground image with the satellite map. However, by rotating the BEV image four times and selecting the prediction with the highest confidence probability, we can obtain the correct result as in Figure 3 (b). Despite the mislocalized point having a similar appearance to the ground observation, its probability output is significantly smaller than the localization confidence probability from the correct location. This property is crucial for safety-critical applications like autonomous driving.

(a) Projection of KITTI

(b) Localization Results

Leereceaproeecag

KITTI dataset On the KITTI dataset, we compare our method with the state-of-the-art LM [24], SliceMatch [12], Boosting [26] and CCVPE[35]. Similar to the evaluation on VIGOR, we directly use the results provided by CCVPE [35] or the corresponding paper. We utilize an increased number of iteration steps in response to the limited coverage of ground images from KITTI [8] within satellite images. The results are shown in Table 2. When a ± 10 ◦ orientation prior is considered in both Training and Test1, our method has a 34.4% lower mean error for localization than CCVPE [35]. Figure 4 presents the visualization results on the KITTI dataset. Boosting [26] achieves exceptional accuracy in estimating orientation. This may be attributed to their utilization of a two-stage approach, where the first stage is dedicated to orientation estimation.

4.4 Cross-Dataset Generalization

The CVUSA dataset [39] is commonly used for cross-view localization, but it only provides retrieval labels and cannot be used for training precise localization models. Despite the panoramas in CVUSA being cropped and having a non-standard aspect ratio, our spherical transform method successfully projects them to a bird's-eye view by completing it to the correct proportion, as shown in Figure 5. We use images from CVUSA in different cities to test our model's generalization ability, which is trained on VIGOR. The alignment results in Figure 5 demonstrate that our model has strong potential and can align significantly different BEV images with their corresponding satellite images for precise localization, even in unfamiliar cities without retraining.

spherical transform

spherical transform

spherical transform

?

4.5 Ablation Study

Pseudo-Siamese Backbone We employ a Pseudo-Siamese backbone with non-shared weights for processing images from two different sources. To demonstrate the effectiveness of this structure, we train two models with shared and non-shared weights. The results in Table 3 show that applying the Pseudo-Siamese backbone leads to a mean localization error reduction of 0.71m.

Homography Estimation Module We also conduct ablation experiments to demonstrate the effectiveness of our homography estimation module. We replace the homography estimation module with feature-based methods SuperGlue [22] and LoFTR [28], utilizing RANSAC-based functions from OpenCV to compute homography matrix from matched feature points. The experimental results are presented in Table 3. The fact that the feature-based methods have low median localization errors demonstrates the effectiveness of transforming localization into 2D alignment. However, the significantly lower mean errors achieved by our proposed deep homography estimator indicate its superiority in achieving more accurate localization.

4.6 Computational Efficiency Analysis

We assess our method's computational efficiency against the state-of-the-art CCVPE [35]. Table 4 compares model parameters, inference memory, per-frame inference time, and mean localization error on the VIGOR dataset using a 12th Gen Intel(R) Core(TM) i5-12490F processor, 16GB memory, and an NVIDIA RTX 3050 GPU. Our method achieves higher accuracy, faster speed, and lower memory usage, demonstrating its computational efficiency.

5 Conclusion

In this study, we introduce HC-Net, an end-to-end network designed for fine-grained cross-view geo-localization. The network processes spherical-transformed ground images and GPS-tagged satellite images as inputs and generates the homography matrix between them, along with the precise GPS location and orientation of the ground camera. Compared to the previous state-of-the-art, HC-Net demonstrates a reduction in mean localization error by 21.3% and 32.4% on the same-area and cross-area splits of the VIGOR dataset, respectively.

References

https://en.wikipedia.org/wiki/Web_Mercator_projection#References .

Yousset I Abdel-Aziz, Hauck Michael Karara, and Michael Hauck. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogrammetric engineering & remote sensing , 81(2):103-107, 2015.

Eli Brosh, Matan Friedmann, Ilan Kadar, Lev Yitzhak Lavy, Elad Levi, Shmuel Rippa, Yair Lempert, Bruno Fernandez-Ruiz, Roei Herzig, and Trevor Darrell. Accurate visual localization for automotive applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 0-0, 2019.

Si-Yuan Cao, Jianxin Hu, Zehua Sheng, and Hui-Liang Shen. Iterative deep homography estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1879-1888, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Deep image homography estimation. arXiv preprint arXiv:1606.03798 , 2016.

Florian Fervers, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelhagen. Uncertainty-aware vision-based metric cross-view geolocalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21621-21631, 2023.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Mingbo Hong, Yuhang Lu, Nianjin Ye, Chunyu Lin, Qijun Zhao, and Shuaicheng Liu. Unsupervised homography estimation with coplanarity-aware gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17663-17672, 2022.

Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7258-7267, 2018.

Sixing Hu and Gim Hee Lee. Image-based geo-localization using satellite imagery. International Journal of Computer Vision , 128(5):1205-1219, 2020.

Ted Lentsch, Zimin Xia, Holger Caesar, and Julian FP Kooij. Slicematch: Geometry-guided aggregation for cross-view pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Ang Li, Huiyi Hu, Piotr Mirowski, and Mehrdad Farajtabar. Cross-view policy learning for street navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8100-8109, 2019.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision , pages 1-18. Springer, 2022.

Tsung-Yi Lin, Serge Belongie, and James Hays. Cross-view image geolocalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 891-898, 2013.

Liu Liu and Hongdong Li. Lending orientation to neural networks for cross-view geolocalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5624-5633, 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In IJCAI'81: 7th international joint conference on Artificial intelligence , volume 2, pages 674-679, 1981.

Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. Advances in neural information processing systems , 31, 2018.

Lang Nie, Chunyu Lin, Kang Liao, Meiqin Liu, and Yao Zhao. A view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation , 73:102950, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4938-4947, 2020.

Paul-Edouard Sarlin, Daniel DeTone, Tsun-Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulò, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. Orienternet: Visual localization in 2d public maps with neural matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21632-21642, 2023.

Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17010-17020, 2022.

Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for image based cross-view geo-localization. Advances in Neural Information Processing Systems , 32, 2019.

Yujiao Shi, Fei Wu, Ankit Vora, Akhil Perincherry, and Hongdong Li. Boosting 3-dof groundto-satellite camera localization accuracy via geometry-guided cross-view transformer. arXiv preprint arXiv:2307.08015 , 2023.

Yujiao Shi, Xin Yu, Liu Liu, Dylan Campbell, Piotr Koniusz, and Hongdong Li. Accurate 3-dof camera geo-localization via ground-to-satellite image matching. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8922-8931, 2021.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020.

Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3608-3616, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Nam N Vo and James Hays. Localizing and orienting street views using overhead imagery. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 494-509. Springer, 2016.

Scott Workman, Richard Souvenir, and Nathan Jacobs. Wide-area image geolocalization with aerial reference imagery. In Proceedings of the IEEE International Conference on Computer Vision , pages 3961-3969, 2015.

Zimin Xia, Olaf Booij, and Julian F. P. Kooij. Convolutional cross-view pose estimation. ArXiv , abs/2303.05915, 2023.

Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , pages 90-106. Springer, 2022.

Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems , 34:29009-29020, 2021.

Amir Roshan Zamir and Mubarak Shah. Accurate image localization based on google maps street view. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11 , pages 255-268. Springer, 2010.

Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs. Predicting ground-level scene layout from aerial imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 867-875, 2017.

Yiming Zhao, Xinming Huang, and Ziming Zhang. Deep lucas-kanade homography for multimodal image alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15950-15959, 2021.

Sijie Zhu, Mubarak Shah, and Chen Chen. Transgeo: Transformer is all you need for cross-view image geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1162-1171, 2022.

Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3640-3649, June 2021.

Supplementary Material for Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator

In this supplementary material , we provide detailed information on projecting ground images to bird's-eye view for VIGOR [42] and KITTI [8] datasets in Section A, as well as the computation details for converting GPS labels to pixel coordinates used during training in Section B. We also discuss the potential for future research based on our method in Section C and broader societal and industry impact of our approach in Section D.

A Projection Details

A.1 Details for Spherical Transform

In the main paper, we provide an overview of the derivation process of the Spherical Transform and the final result in Equation 1. Here we will provide detailed information on the process for the projection. We will use the expressions defined in Section 3.1. To clarify the presentation, we also provide the schematic diagram of the projection process, as shown in Figure 6.

In the camera coordinates, the conversion formula between the Cartesian coordinates P = ( x 1 , y 1 , z 1 ) and the spherical coordinates ( ϕ, θ ) is given by:

The equirectangular projection is used to project spherical coordinates onto a plane, which is the display format for panoramic images. The conversion formula between the spherical coordinates ( ϕ, θ ) and the Normalised Equirectangular coordinates P ′ = ( x 2 , y 2 ) is given by:

The negative sign in the x 2 expression is due to the fact that panoramic images display the scene as viewed from the camera optical center to the outside. Therefore, when ϕ is positive, it corresponds to the negative half-plane of the Equirectangular plane in terms of x 2 . The mapping between pixel coordinates ( u p , v p ) and Normalised Equirectangular coordinates ( x 2 , y 2 ) on the panorama is established as:

To obtain the corresponding bird's-eye view of the panorama, we place a tangent plane at the south pole of the spherical imaging plane as a new imaging plane, as shown in Figure 6 (c). The focal length of the BEV in the imaging process is f = 0 . 5 W b / tan( fov ) . The camera coordinate system coordinates ( x 1 , y 1 , z 1 ) corresponding to a pixel ( u b , v b ) on the bird's-eye view (BEV) imaging plane are determined as:

By substituting Equation 9 into Equation 6, and then substituting the result into Equation 7, we can obtain the Normalised Equirectangular coordinates ( x 2 , y 2 ) for a pixel on the bird's-eye view (BEV) image plane. Finally, by substituting the Normalised Equirectangular coordinates into Equation 8, we can obtain the pixel coordinates ( u p , v p ) on the panoramic image. This leads to the Spherical Transform mapping formula in Equation 1.

x2 =

-�

/

π

y2 = 2

�

/

π

�

=

-

x2

π

�

= y2

π

/2

(d) BEV coordinates

�풃��

�풃��

latitude

θ

longtitude

φ

z

1

x

1

y

1

P(x

1

,y

1

,z

1

)

(b) Spherical coordinates

0

(c) BEV Project

z

1

x

1

y

1

�풃��

��풂��

fov

f

(x

1

,y

1

,z

1

)

�풃��

�풃��

0

(a) Normalised Equirectangular

coordinates

x

2

-1

1

1

-1

��풂��

��풂��

y

2

P'(x

2

,y

2

)

0

i

Wetest the effectiveness of the bird's-eye view projection with different field of view ( fov ) parameters, as shown in Figure 7. As seen in (d), when fov = 85 ◦ , the field of view of the bird's-eye view image is similar to that of the corresponding satellite image, and there are few invalid parts in the image. Therefore, we use this parameter for all of our experiments.

The spherical transform projects the ground image to a bird's eye view, while the polar transform [25] or projective transform [24] projects the satellite image to the perspective of the ground image (aligning with a panoramic view). There is a growing recognition and research on the role of Bird's Eye View (BEV) representation in localization, navigation, and related tasks.

The advantage of projecting to a bird's eye view, as compared to projecting the satellite image to the perspective of the ground image, lies in its more intuitive nature for the localization task and the need for only one projection at the start of the localization process. On the other hand, the latter approach requires selecting projection points (initial pose or candidate poses) for projecting the satellite image, and if these points are not accurately chosen, the resulting projection may not align well with the ground image. Additionally, for more accurate localization, the latter approach requires multiple applications of the projection method from different positions, which could potentially make it less real-time compared to our method.

A.2 Details for Projecting Front View Images in KITTI

In the KITTI dataset [8], the ground is a front view image, so the process of projecting it onto a bird's-eye view is completely different from that of the panorama in VIGOR [42]. We illustrate the mechanism of this projection process in Figure 8, where AB represents the height of the frontal view image, and AC represents the desired height of the bird's-eye view image.

We define the size of the front view image as H f × W f , and the target size of the bird's-eye view image as H b × W b . Using a method similar to that described in Section 3.1, we imagine a bird's-eye projection plane parallel to the ground and adjacent to the front view image. Then we connect the camera optical center with each pixel on the bird's-eye view image to obtain the corresponding pixel coordinates on the front view image.

In Figure 8 (b), fov represents the field of view of the front view image ( ∠ ADC ), f represents the focal length DE , h represents the distance from the camera optical center D to the bird's-eye

(b) Satellite Image

(c)

fov = 75 °

(d)

fov = 85 °

(e)

fov = 88 °

�

�

ℎ

�

'

�

�

3

�

2

(a) Front-View and BEV Image

(b) Geometric Mechanism

A

B

C

D

E

F

G

H

�

���

I

�풃��

�풃��

�풇풓���

�풇풓���

projection plane, and θ represents the angle between the camera imaging plane AB and the vertical direction. To facilitate the subsequent derivation, we also define δ to represent ∠ FAD , φ to represent ∠ BAD , θ 2 to represent ∠ AGD , θ 3 to represent ∠ AHG , f ′ to represent FA , and l 0 to represent AD . According to [8], fov = 17 . 5 ◦ in the KITTI dataset. These variables are calculated as follows:

We denote the pixel coordinates on the bird's-eye view imaging plane as ( u b , v b ) . The values of θ 2 and θ 3 can be calculated using the arctangent formula and the properties of exterior angle as follows:

Then, based on the sine and similarity triangle theorems, we can calculate the corresponding pixel coordinates ( u f , v f ) on the front view image for each pixel on the bird's-eye view image. The calculation is as follows:

(d) θ = 0.8 °

(c) θ = 0 °

(e) θ = 1.6 °

(f) θ = 0.8 ° and Rotate

We introduce the angle θ because we found that in practical testing, the imaging plane of the ground camera may not be perfectly vertical, as shown in Figure 9. Based on experimental results, we use θ = 0 . 8 ◦ for all experiments. During the actual projection process, we set the target size of the bird's-eye view image to be H b × W b = (6 ∗ W f ) × (6 ∗ W f ) , which corresponds to a field of view of approximately 40 m × 40 m . After the first projection calculation, we directly applied the homography matrix H obtained from the above projection process to obtain the BEV. Then, we used scaling and rotation homography matrices H 1 and H 2 to obtain the final BEV with the specified size.

Here, scale represents the ratio of the current H b × W b to the desired input size of the network ( 512 × 512 ). γ represents the yaw angle of the ground camera with noise, and ( u c , v c ) represent the pixel coordinates of the center of the final BEV image.

B Label Correction

B.1 Label Correction in VIGOR Dataset

During research, we found that the ground truth for pixel coordinates of ground images on satellite patches in the VIGOR dataset [42] is inaccurate. The VIGOR dataset uses a uniform meter-to-pixel resolution for converting the latitude and longitude of ground images to their location in aerial images across all four cities. SliceMatch [12] also discovered this issue and proposed a correction, but their improvement is limited to using different average meter-to-pixel resolutions for each city, which may still result in some inaccuracies in different regions of the same city.

We propose the use of Mercator projection [1] to directly compute the pixel coordinates of ground images on specified satellite images using the GPS information provided in the dataset. Equation 4 allows us to calculate the pixel coordinates of a GPS coordinate on a global scale. We compute the

(f) Ground_GPS

(g) Satellite_GPS

(h) Corresponding GPS

Center

Center

Certain Point

Original

SliceMatch

Correction

pixel coordinates of the satellite patch's GPS label and the pixel coordinates of the ground image's GPS label. We then add the difference between them to the center coordinates of the current satellite patch, resulting in accurate pixel coordinates of the ground image on the satellite patch. The result of the correction is shown in Figure 10, which shows that the label provided by VIGOR has a significant deviation from the true position. The label corrected by SliceMatch [12] reduces this deviation, but there is still some error between the corrected label and the true position. The statistical information on the absolute error, measured in meters, between the labels provided in VIGOR [42] and SliceMatch [12] and the corrected labels using our method is presented in Table 5.

Note that in the model training process, including our method and all other methods, we do not directly use the GPS coordinates for training. This is because if GPS is used directly to calculate the training loss, the data truncation problem will occur when using Float32 tensor format since the valid data is five decimal places and trigonometric functions are used in the calculation.

B.2 Label Creation in KITTI Dataset

Supervised information required for training our network includes the pixel location on the bird's-eye view image, as well as its corresponding true GPS value. However, obtaining this information directly from the KITTI dataset [8] is not feasible, and we need to calculate it ourselves.

To obtain the required supervised information, we use the calibration files provided by KITTI and the point cloud data in the dataset. Given a 3D point x in Velodyne coordinates, we project it to a point y

9077

3579

(a) Example 1

(b) Example 2

(c) Example 3

in the i-th camera image as follows:

where i = 2 , P rect ( i ) , R rect (0) , and T cam velo are calibration parameters provided in the KITTI dataset.

Additionally, we calculate the distance deviation in terms of latitude and longitude for the 3D point with respect to the origin of the IMU/GPS coordinate system in the world coordinate system. This is represented as x GPS:

Using the above process, we compute the latitude and longitude deviations for a particular 3D point relative to the GPS sensor, and determine its corresponding GPS value using the meter-to-GPS calculation method. Finally, we utilize the homography transformation H final = H 2 H 1 H obtained from Section A.2 to map the pixel coordinates y to their corresponding pixel coordinates on the BEV image. During model inference, we can obtain the GPS coordinates of a particular pixel on the BEV image. Using the inverse process of the above method, we can obtain the latitude and longitude deviations of the corresponding 3D point relative to the GPS sensor, and estimate the GPS value corresponding to the camera. Some examples is shown in Figure 11.

C Future Research

In our projection of the panoramic images into a bird's-eye view, we place a tangent plane at the south pole of the spherical imaging plane as the new imaging plane. However, this method assumes that the ground camera is oriented vertically, which may not always be the case due to roll and pitch deviations. Assuming ( α, β, γ ) represent the angles of ( roll, pitch, yaw ) , the rotation matrix can be obtained as follows:

By multiplying the rotation matrix R with the camera coordinates obtained from Equation 9, we obtain the new coordinates ( x ′ 1 , y ′ 1 , z ′ 1 ) in the camera coordinate system. We then proceed with the subsequent calculations to obtain the bird's-eye view image under the assumption of the specified ( roll, pitch, yaw ) angles. Different angle configurations yield different bird's-eye view images, as illustrated in Figure 12. It can be observed that the resulting BEV appearance varies significantly when different ( roll, pitch ) angles are given. Therefore, we suggest that in future research, ( roll, pitch ) can be treated as additional predicted outputs to obtain more degrees of freedom in estimating the ground camera pose. Our differentiable spherical transform provides a possibility for this idea.

�

= 0 °

,

�

= 0 °

,

�

= 0 °

�

= 0.53 ° ,

�

= 1.4 ° ,

�

= 0 °

D Broader Impact

This paper introduces an innovative approach to achieving high-precision localization of ground cameras, a capability of immense value in applications such as autonomous driving, augmented reality, and mapping services. Accurate localization is crucial for these industries to operate safely and effectively. However, it is imperative to acknowledge that the position of a vehicle or a user's camera is often sensitive and private information. The misuse or abuse of this technology could lead to privacy violations, potentially endangering individuals' security and personal information. We advocate safeguards and regulations to ensure that the privacy and security of individuals are not compromised in the process.

Peer reviews:
comments: SUMMARY:
This paper propose to wrap the ground-view image to align with the corresponding aerial-view image using homography estimation. Firstly, a differentiable spherical transform is adopted to align the perspective between the ground-view and aerial-view images. Then a correlation-based homography estimator is proposed to align the similar parts of the transformed ground image and aerial-view image. It achieves state-of-the-art performance on VIGOR and KITTI datasets with a speed of 28 FPS.

STRENGTHS:
+ The idea is very interesting and the writing is good.
+ The presentation of figures and tables are also good.
+ The evaluation is comprehensive and the performance is promising.

WEAKNESSES:
- The figure looks good, but I am not able to get the detailed flow of data. The part 2 and part 3 of Fig 2 are connected only on the correlation map. Would the code be released? It would be better to add this claim to help reproduce the result.
- The authors could add some insight about why the spherical transform is better than polar transform/ projective transform in the other papers.

QUESTIONS:
See limitations and weaknesses. I am open to raise my rating after rebuttal.

LIMITATIONS:
- The spherical transform is applied on query imaged on-the-fly, which might be a disadvantage compared to other transforms on aerial-view images, but the method is still fast. Is there a speed comparison with other methods? I think this could a limitation that needs to be discussed. 
- The authors could add more discussion on positive/negative societal impact in supplementary materials. I think this method should have some potential impact on industry.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of ground camera pose refinement by ground-to-satellite image matching. For this purpose, this paper proposes to project a ground image to the overhead view image plane by using a Homography, and then iteratively update the residual Homography between the projected overhead view image and the reference satellite image, from which the relative pose between the ground camera and the satellite image can be estimated. Experiments are conducted on two cross-view datasets, VIGOR and KITTI, and the results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n+ A new pipeline for ground-to-satellite image localization using Homography update is proposed; \n+ The proposed method achieves state-of-the-art performance.\n\nWEAKNESSES:\n(1) This paper estimates the residual Homography transformation between the reference satellite image and an overhead view image projected from a ground view image. The ground camera's pose is estimated from the Homography. However, no illustrations are about how to estimate the camera pose (rotation and translation) from the Homography. \n\nOne sentence in L205-206 is for orientation estimation. However, it seems that we should select different points from the ground view image and project them to the overhead view by using the estimated Homography, and the orientation is estimated by connecting the two points. If this is the way to estimate orientation, my first question is: \"how to estimate the translation\"? This is not described. My second question is, how to select the two points, randomly? Will this be reliable? The severe occlusion between the ground and satellite images makes that not all ground image pixels are visible in the overhead view and vice versa. How to select/determine points that are co-visible by the two views for this orientation estimation?\n\n(2) If the pose is derived from the estimated Homography, it should be a deterministic solution. How the probability map in Fig.3 is estimated?\n\n(3) The previous work by Shi and Li [22] proposes an iterative optimization network that directly optimizes pose parameters. From a high-level idea, the difference between this paper and Shi and Li is that this paper optimizes Homography parameters. However, the Homography contains 5 degrees of freedom which are larger than that of the pose parameters (3 DoF). What is the superiority of this? What is the performance of the proposed method if we simply modify the Homography parameter output to pose parameter output? and similarly, since this paper uses optical flow for supervision, why not directly estimate the optical flow between the two images and then compute the relative pose from the flow map?\n\n(4) This paper claims that the GPS labels of the VIGOR and KITTI datasets are inaccurate and thus proposes to re-calibrate the pose labels. However, if the GPS labels themselves are inaccurate, the different GPS-to-UTM conversion methods would not correct the error. \nMoreover, any investigations of what kind of GPS conversion methods are used in previous works when introducing the cross-view dataset, don't they also use the Mercator projection or an equivalent projection, or why the Mercator projection used in this paper should provide more accurate UTM coordinates than the methods used in their original papers? \n\nFurthermore, if the ground truth has been modified, it is unfair to compare previous works' results reported in their original papers with the inaccurate poses for training and evaluation. \n\n(5) The network is supervised using ground truth (GT) pixel correspondences. How are the GT correspondences derived, according to the GT relative RT? However, as I mentioned before, due to the severe occlusions between ground and satellite images, not every pixel in the ground view is visible in the overhead view and vice versa. Thus, there must be incorrectly derived GT correspondences if all the pixels in the overhead/satellite images are used. \n\n(6) For the ground-to-satellite projection, this paper assumes the satellite image has a pin-hole camera projection with an FoV of 85-degree. However, Shi et al. [26] approximate a parallel projection for the satellite images. What is the superiority of this pin-hole camera projection over the parallel projection, and why use an FoV of 85 degrees instead of other numbers? \n\n(7) The writing needs to be thoroughly improved and should be more accurate. For example:\n    (i) Plz do not simply use a number [*] for references as a component in a sentence without mentioning the author's names.\n    (ii) Sentence in L33 is not strictly correct. Slicematch is also a kind of \"repeat (pose) sampling\".\n    (iii) Sentence in L144 \"without any intrinsic parameters\" is very ambiguous, this paper does use focal length and FoV for satellite images (L136-137), and the ground-to-satellite tomography-based projection should also use the ground camera's intrinsic when it is a pin-hole camera as in the KITTI dataset. \n    (iv) The symbols in Sec 3.2 are very confusing. The authors mixed symbols for three images: the original ground image, the projected overhead view image from the ground-view image, the satellite image. This section needs to be thoroughly revised. \n    (v) ...\n\n(8) Missing important references:\n     (i) Fervers, Florian, et al. \"Uncertainty-aware Vision-based Metric Cross-view Geolocalization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. (available on arxiv in 2022, before SliceMatch [11])\n     (ii) Shi, Yujiao, et al. \"Where am i looking at? joint location and orientation estimation by cross-view matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n     (iii) Shi, Yujiao, et al. \"CVLNet: Cross-view Semantic Correspondence Learning for Video-Based Camera Localization.\" Asian Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n     (iv) Vyas, Shruti, Chen Chen, and Mubarak Shah. \"Gama: Cross-view video geo-localization.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n     (v) Zhang, Xiaohan, Waqas Sultani, and Safwan Wshah. \"Cross-View Image Sequence Geo-localization.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\nQUESTIONS:\nPlz refer to my comments on Weaknesses\n\nLIMITATIONS:\nUsing homography estimation between the ground and satellite images ignores the correspondences of objects not in the tomography plane, potentially limiting the performance.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses fine-grained cross-view geo-localization task that that matches the camera ground images with a satellite image patch covering the same area to determine the geo-pose of camera. The proposed approach projects ground images onto a bird’s-eye view perspective and formulate the task as a 2D image alignment problem. A correlation-based homography estimation module is proposed to achieve precise localization. Experiments are presented on VIGOR and KITTI datasets.\n\nSTRENGTHS:\n+ Proposed correlation-aware homography estimation module is interesting\n+ Experiments show promising results in VIGOR and KITTI datasets.\n\nWEAKNESSES:\nThe paper has interesting ideas, but I think the paper needs more work to be convincing. The weaknesses are mentioned below:\n\n-- The use of BEV representation has been explored for  Fine-Grained Cross-View Geo-Localization task [Ref1]. This paper should have been discussed and compared in the experiments.\n\n[Ref1] F. Fervers, et. al., Uncertainty-aware Vision-based Metric Cross-view Geolocalization, CVPR 2023\n\n\n-- The novelty behind the spherical transform module to project ground images to a birds-eye-view (BEV) perspective is not clear. Inverse Perspective Mapping (IPM) is commonly used for long time for transforming camera images to BEV [Ref2, Ref3]. Also, looking at the generated BEV images and from experience of using [Ref3] for generation, the BEV images do not seem any better than the quality achieved by applying the IPM technique.\n\n[Ref2] H. A. Mallot, et. al., “Inverse perspective mapping simplifies optical flow computation and obstacle detection,” Biological Cybernetics, 1991\n\n[Ref3] L. Reiher, et. al., \"A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras\nto a Semantically Segmented Image in Bird’s Eye View\", IEEE International Conference on Intelligent Transportation Systems, 2020\n\n-- There are also many works that use neural networks to generate BEV representation from ground camera images. Some of these papers should be discussed. (e.g., [Ref4, Ref5])\n\n[Ref4] Z. Li, et. al., Translating Images into Maps, ICRA 2022\n[Ref5] A. Saha, et. al., BEVFormer: Learning Bird’s-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers, ECCV 2022\n\n-- The utilization of infoNCE loss for cross-view image geo-localization is also not new [Ref6, Ref7]\n\nY. Zhu, et. al., Simple, Effective and General: A New Backbone for Cross-view Image Geo-localization, arXiv 2023\nF. Deuser, et. al., Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation, arXiv 2023\n\n\n-- Experiments do not show consistent improvements, as evident from Table 2. There are several cases where other baselines perform better.\n\n-- Not sure of the reproducibility of this work and no code is provided in the supplementary.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nI have read the author’s rebuttal and other reviews. I am still not convinced and keep the score the same.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a homography estimation-based method for cross-view geo-localization. Contrary to existing methods that tackle the problem as a retrieval problem, the paper proposes to reformulate the problem as homography estimation of aligning the birds-eye-view against the satellite image. To this end, the authors propose a recurrent homography estimation module which learns to estimate the homography matrix given the feature maps of birds-eye-view and satellite images. Experiments show that the method can demonstrate effective performance against the tested baselines in most metrics.\n\nSTRENGTHS:\n1. The idea of using bird's eye view for matching against satellite images is intuitive, and the proposed method well leverages the formulation by casting the geo-localization problem as homography estimation.\n\n2. Experiments make valid comparisons against both existing geo-localization methods and also on a few image matching-based methods, where the proposed method shows performance enhancements. Notably, the paper shows large amounts of improvements in translation estimation, which I find to be highly practical for geo-localization tasks.\n\n3. Writing is very clear. It was straightforward to follow the paper's motivation and how the proposed method functions as a whole.\n\nWEAKNESSES:\nMy concerns are two-fold:\n1. While I favor the paper's formulation of aligning bird's eye view against satellite images, I feel the paper has only included a limited number of homography-based baselines. Currently the paper has only made comparisons against homography estimation from local feature maches in Table 3. However there are many more homography estimation methods that could additionally be tested, for example 'Deep Image Homography Estimation' (which is also introduced in the related works section). Adding more homography-based methods for comparisons will better validate why the proposed recurrent homography estimator is needed. Related, how are the homographies estimated from SuperGlue, LoFTR matches? Have the authors used a RANSAC-style estimator to handle outliers? It seems that the paper is currently missing details on this part, which is crucial for local feature matching-based methods to work effectively.\n\n2. I have not fully grasped the reason why the method performs poorly for rotation estimation in KITTI. Is this due to not using the orientation data labels during training? A better clarification of this phenomenon will be helpful for readers to better understant the paper.\n\nQUESTIONS:\nPlease refer to the 'weaknesses' section.\n\nLIMITATIONS:\nThe authors have not explicitly discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}