SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, our algorithm demonstrates good performance in angle prediction, with average and median angle errors lower than those of SliceMatch by  $67\%$  and  $69\%$ , respectively.
Figure 3: The localization error under varying noise conditions at different angles on the VIGOR dataset. Orange indicates cross-area, and blue indicates same-area.
4.5 Cross-Area Generalization
Generalizing new ground images across different regions is a more challenging task than within the same region, as the test area may appear vastly different from the training area. Despite this increased difficulty, our approach still demonstrates strong performance. As shown in Table 1 Cross-Area, Our average localization error is  $40\%$  lower than SliceMatch, and our median localization error is  $64\%$  lower than SliceMatch. Compared to descriptor-based matching methods, our approach exhibits greater robustness.
In the cross area of VIGOR, our algorithm performs better than previous methods in both known and unknown directions. Additionally, as shown in Figure 3, we tested the performance of our algorithm across various angles using a model trained in unknown directions. Our algorithm demonstrated strong stability against noise for different angle variations in both the same area and cross area.
(A) KITTI dataset
(C) VIGOR dataset
(D) Oxford RobotCar dataset
Figure 4: For each scene, the up left is the ground image, the bottom left denotes matching inliers and the right shows the satellite image and localization results.
Table 5: Ablation Study on the KITTI dataset[3, 13]
Model	Same-Area	Cross-Area
Location(m) mean median	Azimuth(°) mean median	Location(m) mean median	Azimuth(°) mean median
Ours w/o Ground Feature Projection	3.52	2.66	2.09	1.48	9.83	7.11	3.13	2.22
Ours w/o BEV Feature RefineBlock	2.02	0.78	0.82	0.53	8.72	4.55	2.56	1.47
Ours w/o Matching Loss	9.06	8.10	1.93	1.12	10.60	9.97	2.75	1.82
Ours w/o Position Loss	0.99	0.53	0.51	0.36	9.63	4.79	2.87	1.65
Ours w/o Confidence	1.65	0.65	0.98	0.75	7.76	3.57	2.31	1.50
Ours	1.48	0.47	0.49	0.30	7.97	3.52	2.17	1.21
4.6 Vary-Environment Generalization
In addition, to validate the performance of our algorithm under different environmental conditions, we tested it on the Ford multi-AV dataset and the Oxford RobotCar dataset. The results of the Ford multi-AV dataset are presented in Table 2, which is primarily focused on suburban scenes with fewer surrounding buildings and difficulty in obtaining semantic information, resulting in generally poor localization accuracy. Moreover, in the limited HFOV scenario, the longitudinal position is more challenging than the lateral position in the driving direction. Consequently, the recall rate for the longitudinal position is significantly lower than that of the lateral position, and this trend applies to all the compared methods. Nonetheless, our algorithm still outperforms previous algorithms on this dataset, and we found that our lateral recall rate is significantly higher, indicating our algorithm's ability to capture road surface information under semantically unclear conditions. Our algorithm demonstrated superior performance across the three distinct test sets of the Oxford RobotCar dataset, as shown in Table 4. These results suggest that our approach exhibits strong generalization capabilities across diverse environmental conditions, seasonal variations, and roadway scenarios.
4.7 Ablation Study
Table 5 presents the results of our ablation experiments conducted on the KITTI dataset to evaluate the efficacy of the view unification mechanism comprised of Ground Feature Projection and BEV Feature RefineBlock, the loss function composed of Matching Loss and Position Loss, and the Confidence mechanism. We employ Ground Feature Projection and BEV Feature RefineBlock to address significant viewpoint differences between cross-view images and incorporate BEV and satellite image features within the same scope. Our experiments demonstrate that utilizing Ground Feature Projection and BEV Feature RefineBlock significantly improves matching accuracy, reducing average localization error by  $57\%$  and  $27\%$ , respectively. Furthermore, by incorporating matching points during the training process and employing a more robust supervised training approach, we
Ground image
Sat. image
Confidence map
BEV feature map Sat. feature map
(a)
Ground image
Confidence map
Figure 5: Matching confidence illustration. The top row of each subfigure shows the ground and satellite images, highlighting corresponding points. Circles indicate confident correspondences (score  $>0.5$ ), while stars represent potentially incorrect correspondences (score  $<0.5$ ). Matching points share the same color and shape in both images. In the bottom row, the left image shows confidence scores on the BEV feature map (blue for  $>0.5$ , green for  $<0.5$ ). The middle and right images depict the BEV and satellite feature maps, respectively, with connected lines indicating matching pixels. The circles and stars in these images align with the corresponding points in the top row.
BEV feature map Sat. feature map
(b)
observe a  $57\%$  decrease in average localization error. Finally, the addition of a confidence mechanism further enhances network precision, resulting in a  $10\%$  reduction in average error. Through conducting ablation experiments, we have demonstrated the efficacy of the view unification, loss, and confidence mechanisms proposed by us.
In Figure 5, we conducted a qualitative analysis of the confidence mechanism on the KITTI dataset. We utilized a threshold of 0.5 to distinguish between high-confidence matches and low-confidence matches. In these visualizations, circles represent matching points with high confidence, while stars indicate matching points with low confidence. Additionally, for clarity, we displayed estimated corresponding points on both the original ground and satellite images. From the visualizations, it is evident that pixels in the bird's-eye-view image, which are mutually visible in both ground and satellite images (e.g., road areas), tend to have higher confidence. On the other hand, pixels that are visible only in one view (such as building rooftops, tree canopies, and occluded regions) generally exhibit lower confidence. As a result, our confidence map effectively selects reliable matching points for accurate pose estimation. This holds significant implications for improving precision.
This paper presents a novel approach based on dense matching to address the problem of cross-view localization. Our approach utilizes a view unification module to bridge the visual gap between satellite and ground images. Additionally, we introduce a matching loss to provide stronger supervision for the model during training. To evaluate the effectiveness of our algorithm, we conduct experiments on four datasets that vary in geographical regions, environmental conditions, and road scenarios. Our results demonstrate the impressive generalization capability of our approach. One possible future direction is to explore the integration of spatial and temporal information, to further improve the accuracy and robustness of our approach for cross-view localization.
This work was supported in part by the National Natural Science Foundation of China (No. 62302220), in part by the China Postdoctoral Foundation (No. 2023M731691), and in part by the Jiangsu Funding Program for Excellent Postdoctoral Talent (No.2022ZB268).
[20] Shruti Vyas, Chen Chen, and Mubarak Shah. Gama: Cross-view video geo-localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 440-456. Springer, 2022.
[21] Shan Wang, Yanhao Zhang, and Hongdong Li. Satellite image based cross-view localization for autonomous vehicle. arXiv preprint arXiv:2207.13506, 2022.
[22] Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 90-106. Springer, 2022.
[23] Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems, 34:29009-29020, 2021.
[24] Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3640-3649, 2021.
A Theoretical Proof
We provide a detailed derivation and proof of the least-squares pose regression process in the text. The camera position parameters  $\theta$  and  $t$  can be obtained by solving the following loss function.
$$
\xi = \underset {\theta , t} {\arg \min } \sum_ {i = 1} ^ {n} S _ {i} \left(\boldsymbol {R} \boldsymbol {p} _ {i} ^ {\prime} + \boldsymbol {t} - \hat {\boldsymbol {p}} _ {i}\right) ^ {2}, \boldsymbol {R} = \left[ \begin{array}{l l} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{array} \right] \tag {8}
$$
The relationship between the corresponding points  $\pmb{p}_i'$  and  $\hat{\pmb{p}}_i$  in point sets  $\pmb{p}'$  and  $\hat{\pmb{p}}$  can be expressed as  $\pmb{p}_i' = \pmb{R} \hat{\pmb{p}}_i + \pmb{t}$ . To simplify the problem, we calculate the centroids  $\pmb{g}'$  and  $\hat{\pmb{g}}$  of point sets  $\pmb{p}'$  and  $\hat{\pmb{p}}$ , respectively. We then subtract the centroids from each point in their respective sets, resulting in  $\pmb{q}_i' = \pmb{p}_i' - \pmb{g}'$  and  $\hat{\pmb{q}}_i = \hat{\pmb{p}}_i - \hat{\pmb{g}}$ . This simplifies the problem to:
$$
\begin{array}{l} \xi = \arg \min _ {\theta} \sum_ {i = 1} ^ {n} S _ {i} \left(\boldsymbol {R} \boldsymbol {q} _ {i} ^ {\prime} - \hat {\boldsymbol {q}} _ {i}\right) ^ {2} \\ = \arg \min  _ {\theta} \sum_ {i = 1} ^ {n} S _ {i} \left(\boldsymbol {R} \boldsymbol {q} _ {i} ^ {\prime} - \hat {\boldsymbol {q}} _ {i}\right) ^ {t} \left(\boldsymbol {R} \boldsymbol {q} _ {i} ^ {\prime} - \hat {\boldsymbol {q}} _ {i}\right) \tag {9} \\ = \arg \min  _ {\theta} \sum_ {i = 1} ^ {n} S _ {i} \left(\boldsymbol {q} _ {i} ^ {\prime t} \boldsymbol {q} _ {i} ^ {\prime}\right) + \hat {\boldsymbol {q}} _ {i} ^ {t} \hat {\boldsymbol {q}} _ {i} - 2 \hat {\boldsymbol {q}} _ {i} ^ {t} \boldsymbol {R} \boldsymbol {q} _ {i} ^ {\prime} \\ \end{array}
$$
Therefore, minimizing  $\xi$  is equivalent to maximizing  $\sum_{i=1}^{n} S_i (\hat{\pmb{q}}_i^t \pmb{R} \pmb{q}_i^{\prime})$ . Assuming that  $\pmb{H} = \sum_{i=1}^{n} S_i (\pmb{q}_i^{\prime} \hat{\pmb{q}}_i^t)$ , then
$$
\sum_ {i = 1} ^ {n} S _ {i} \left(\hat {\boldsymbol {q}} _ {i} ^ {t} \boldsymbol {R} \boldsymbol {q} _ {i} ^ {\prime}\right) = t r \left(\sum_ {i = 1} ^ {n} \boldsymbol {R} S _ {i} \boldsymbol {q} _ {i} ^ {\prime} \hat {\boldsymbol {q}} _ {i} ^ {t}\right) = t r (\boldsymbol {R} \boldsymbol {H}) \tag {10}
$$
Upon performing an SVD decomposition on  $H$ , the result is  $H = U\Lambda V^t$ , where  $U$  and  $V$  are orthogonal matrices of dimensions  $3 \times 3$ , and  $\Lambda$  is  $3 \times 3$  diagonal matrix with non-negative elements.
Then we introduce an orthogonal matrix  $\mathbf{X} = \mathbf{U}\mathbf{V}^t$ , and observe that  $\mathbf{X}\mathbf{H} = \mathbf{V}\mathbf{\Lambda}\mathbf{V}^t$  is a symmetric positive-definite matrix. According to the properties of positive definite matrices, it holds that  $tr(\mathbf{X}\mathbf{H}) \geq tr(\mathbf{B}\mathbf{X}\mathbf{H})$  for any  $3 \times 3$  orthogonal matrix  $\mathbf{B}$ . Therefore, we obtain the maximum value of  $\sum_{i=1}^{n} S_i (\hat{\boldsymbol{q}}_i^t\mathbf{R}\boldsymbol{q}_i')$  when  $\mathbf{R} = \mathbf{X}$ , which implies that
$$
\boldsymbol {R} = \boldsymbol {U} \boldsymbol {V} ^ {t} \tag {11}
$$
The dense matching relationship between the graphs gives rise to the uniqueness of solution for R. Subsequently, the relative displacement can be determined as
$$
\boldsymbol {t} = \boldsymbol {g} ^ {\prime} - \boldsymbol {R} \hat {\boldsymbol {g}} \tag {12}
$$
By utilizing equations 11 and 12, we can obtain the camera location and azimuth through a straightforward differentiable process, using the least-squares regression method.
KITTI. The ground images in the KITTI dataset were obtained from the original data. [13] collected satellite images of the KITTI dataset, with each ground image accompanied by a corresponding satellite image whose center corresponds to the position of the ground camera. The dataset is divided into Training, Test1, and Test2 subsets. The images in Test1 are from the same area as the images in the training set, while the images in Test2 are from different areas. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images is approximately  $100 \times 100$  square meters.
Ford Multi AV. For the Ford Multi AV dataset, we only utilized the data of the V2 vehicles from the original dataset. Shi and Li first plotted the trajectories of the ground-view images and collected satellite images along these trajectories. Ground images captured on one date were used for training, while images captured on another date were used for testing. The satellite images remained unchanged during both training and testing. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images is approximately  $100 \times 100$  square meters.
VIGOR. The satellite images of the VIGOR dataset were collected by [24]. The ground images are north-aligned panoramicas, and they are from four cities: New York, San Francisco, Chicago, and Seattle. There is a positive satellite image in the database for each ground image, meaning that the ground image is within the center 1/4 quarter of the satellite image. All the satellite images are north-aligned. The dataset includes two evaluation splits: same-area and cross-area, based on whether the images in training and testing sets are from the same region. Research is conducted to estimate the relative pose between the ground image and its positive satellite image center. In this dataset, the ground resolution of satellite images for New York, San Francisco, Chicago, and Seattle is 0.113, 0.118, 0.111, and 0.101, respectively. The size of the aerial views for these four cities is  $640 \times 640$ .
Oxford RobotCar. The satellite images of the Oxford RobotCar dataset were collected by [8]. They provided a very large satellite map covering the whole region where the ground images were collected. The ground images were collected through multiple traversals on a single route in Oxford, UK, encompassing various time periods, seasons, and weather conditions. During implementation, for each ground image, a random relative translation is generated, and a small satellite patch corresponding to the randomly generated pose is extracted from the large satellite map. Research is conducted to estimate the relative pose. The ground sample distance of the satellite images in this dataset is  $0.0924\mathrm{m}$  per pixel, and the coverage of the satellite images used is around  $55 \times 55$  square meters.
C Comparison with LM[13]
To make a fair comparison with the LM[13], we trained both the LM and our method for an equal number of epochs and found that our method still outperforms the LM method.
Figure 6: Performance comparison between LM[13] and our method with longer training epochs on the KITTI dataset.
The LM[13] aims to optimize the relative pose by minimizing the feature differences between corresponding pixels in two-view images. However, the pixel correspondences are computed based on planar homography (and relative camera pose), which is incorrect for objects above the ground plane. Minimizing feature differences between non-corresponding features leads to suboptimal relative pose estimation. In contrast, our objective in this paper is to first estimate the precise correspondences between the two views and then compute the relative pose. The estimation of correspondences is independent of the relative pose. Once the correspondences are sufficiently accurate and robust, the relative pose can be directly computed by solving a least squares problem instead of training additional networks. Therefore, the main capability of our method lies in accurate correspondence learning rather than relative pose estimation. As a result, our method achieves better performance.