Title: Learning Dense Flow Field for Highly-accurate Cross-view Camera Localization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optical flow

correspondence learning

cross-view

camera localization
Abstract: This paper addresses the problem of estimating the 3-DoF camera pose for a ground-level image with respect to a satellite image that encompasses the local surroundings. We propose a novel end-to-end approach that leverages the learning of dense pixel-wise flow fields in pairs of ground and satellite images to calculate the camera pose. Our approach differs from existing methods by constructing the feature metric at the pixel level, enabling full-image supervision for learning distinctive geometric configurations and visual appearances across views. Specifically, our method employs two distinct convolution networks for ground and satellite feature extraction. Then, we project the ground feature map to the bird's eye view (BEV) using a fixed camera height assumption to achieve preliminary geometric alignment. To further establish the content association between the BEV and satellite features, we introduce a residual convolution block to refine the projected BEV feature. Optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT. After obtaining dense flow correspondences, we apply the least square method to filter matching inliers and regress the ground camera pose. Extensive experiments demonstrate significant improvements compared to state-of-the-art methods. Notably, our approach reduces the median localization error by 89\%, 19\%, 80\%, and 35\% on the KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar datasets, respectively.

Learning Dense Flow Field for Highly-accurate Cross-view Camera Localization

Abstract

This paper addresses the problem of estimating the 3-DoF camera pose for a ground-level image with respect to a satellite image that encompasses the local surroundings. We propose a novel end-to-end approach that leverages the learning of dense pixel-wise flow fields in pairs of ground and satellite images to calculate the camera pose. Our approach differs from existing methods by constructing the feature metric at the pixel level, enabling full-image supervision for learning distinctive geometric configurations and visual appearances across views. Specifically, our method employs two distinct convolution networks for ground and satellite feature extraction. Then, we project the ground feature map to the bird's eye view (BEV) using a fixed camera height assumption to achieve preliminary geometric alignment. To further establish the content association between the BEV and satellite features, we introduce a residual convolution block to refine the projected BEV feature. Optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT. After obtaining dense flow correspondences, we apply the least square method to filter matching inliers and regress the ground camera pose. Extensive experiments demonstrate significant improvements compared to state-of-the-art methods. Notably, our approach reduces the median localization error by 89%, 19%, 80%, and 35% on the KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar datasets, respectively.

1 Introduction

Cross-view camera localization, also known as ground-to-aerial/satellite image matching, has evolved from the study of coarse localization through image retrieval[4, 12, 16, 20] to a highly accurate 3-DoF pose estimation task[5, 13, 22, 24]. This task involves accurately determining the precise position and orientation of a ground camera in relation to a satellite patch that encompasses the surrounding environment.

The key challenge in cross-view camera localization is to bridge the gap between the visual observations captured by the ground camera and the corresponding top-view camera, enabling robust and accurate regression of the camera's pose in real-world spatial coordinates. This capability holds immense significance across various applications, including augmented reality, autonomous navigation, virtual reality, and robotics, where precise camera localization is crucial for enabling advanced functionalities and interactions with the environment.

This paper focuses on investigating the problem of single-frame ground image localization, which aims to estimate the 3 DoF pose of a ground camera relative to a single satellite image. This problem

(A) Ground image

(B) Ground feature

(C) Satellite image

(E) Dense matching

(D) Satellite feature

BEV

Projection

(F) Pose estimation

Ours

GT

5

assumes that the satellite image covers the same geographic region as the ground image. The objective of localization is to determine which pixel of the satellite image corresponds to the ground camera's position and its current shooting direction[24]. In practical applications, as each pixel of the satellite image is associated with GPS labels, it becomes feasible to obtain the geographical location information and spatial orientation information of the ground camera.

Most current methods address this problem by implicitly learning cross-view correspondences to regress camera localization [5, 13, 22, 24]. These approaches establish aggregated descriptorbased representations for both ground and aerial images, resulting in relatively short inference time. However, these methods suffer from two main issues. Firstly, they lack an explicit establishment of spatial unification between views and heavily rely on brute-force training to simultaneously solve spatial and semantic alignment. Secondly, in theory, precise relative pose estimation requires obtaining local cross-view matching relationships, such as point-to-point correspondences. Implicit cross-view feature correlation and pose regression may lead to a loss in localization accuracy. In contrast, methods[13] based on point-to-point matches have been proposed, which utilize LM optimization for feature map alignment between the ground and satellite views. These methods often leverage end-to-end ground truth pose as supervision to learn local correspondences between cross-views. Such supervision may be weak for point-wise deep feature metric learning. Based on the above observations, we propose a method that explicitly represents point-wise correspondences between cross-views, thus facilitating accurate pose estimation. Our key idea is to learn dense flow fields between cross-views and directly compute accurate ground camera poses from such flows in an end-to-end manner. To achieve this, we develop two distinct convolutional neural networks for feature extraction from the aerial and ground images, respectively. But ground and aerial feature maps exhibit significant domain differences in terms of spatial configuration and visual appearance, posing a critical challenge for the neural network to learn effective matching on such feature maps. Furthermore, due to the absence of depth information in the ground images, direct cross-view matching leads to a one-to-many correspondence issue. For instance, a building facade in the ground image may correspond to multiple edges in the aerial image. Existing methods often assume a fixed ground height to enable the projection from the ground to the satellite image[13]. In line with this approach, our method employs the same strategy to geometrically project the ground features, obtaining a bird's eye view (BEV) feature map that corresponds to the ground image. This projection facilitates spatial alignment between the cross-view feature maps. As mentioned above, for the uncertainty in the ground height assumption, we introduce a RefineBlock to further refine the BEV features and achieve content alignment. Subsequently, optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT[17]. Specifically, we calculate multi-scale correlation volumes using the BEV and satellite features and employ GRU modules iteratively for dense flow estimation. Furthermore, we assign scores to each established matching relationship based on their similarity, effectively reducing the weight of erroneous matches. A notable advantage of computing flows on the BEV and satellite features is that the matching relationship allows us to directly utilize the least square method to solve the 3 DoF pose, i.e. , the

2-DoF translation, and 1-DoF rotation. In order to learn robust matching results, we use the dense optical flow as well as the ground truth pose as supervision to train the network progressively.

Our main contributions can be summarized as follows:

This paper introduces a novel approach that learns dense pixel-wise flow fields between ground and satellite images. This enables accurate cross-view camera localization by directly calculating precise ground camera poses from the learned flow.

We propose spatial alignment and content refinement modules for ground image feature enhancement. By leveraging the BEV representation of ground feature maps, the proposed method achieves accurate optical flow estimation according to both spatial and semantical alignments.

We conduct extensive experiments on various datasets, including KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar, which demonstrates that our method achieves superior performance and exhibits strong generalization ability across all datasets.

2 Related Works

Localization based on retrieval. Image retrieval methods are widely utilized for same-view localization tasks[2, 10-12]. This approach assumes the availability of a reference dataset containing a vast number of geotagged ground images. Query images are then matched with ground images in the reference dataset to determine their corresponding geographic locations. However, this method requires a significant number of ground markers, which can be expensive. As satellite maps become more widely available, cross-view localization has gained attention as a potential alternative. In cross-view retrieval tasks, a common approach is to create separate global descriptors for the ground image and the satellite image, and then compare their similarity[4, 14, 18, 19, 23]. Alternatively, recent studies have explored fusing consecutive multi-frame ground images to generate a descriptor, which is then matched with the descriptor of the satellite image[16, 20]. In this cross-view retrieval methods, the camera is assumed to be located at the center of the satellite map. However, this assumption limits the generalization of these methods.

Cross-view camera pose estimation. This work differs from the retrieval task in that it relies on an initial estimate of a given camera pose and then proceeds to refine it to obtain a pose that is close to the true value. Previous works have used various approaches to tackle this problem. For example, VIGOR[24] first uses an end-to-end framework to roughly locate the query through retrieval and then refines the location by predicting offsets through regression. Xia et al. [22] formulates the localization problem as a multi-class classification problem and uses a dense multimodal space for localization. [13] uses a geometric projection module to align the cross-view features, and then optimize the camera pose end-to-end using the Levenberg-Marquardt algorithm[6]. Wang et al. [21] uses 3D points to bridge the geometric gap between ground and overhead views, iteratively fusing ground images with 3D points iteratively for pose estimation. SliceMatch[5] is given a set of candidate photo poses and then compares a single ground descriptor with a set of pose-dependent satellite map descriptors to generate the final camera pose. It is worth noting that previous works have primarily relied on global descriptors [22, 24] or local descriptors[5, 6, 13, 21] for cross-view positional estimation. However, in our work, we propose an alternative approach that relies on dense point matching, which we believe to be a reliable basis for cross-view localization.

3 Methods

As shown in Figure 2, the proposed method is built upon the RAFT architecture. The main differences are the heterogeneous ground and aerial feature extractors and feature transformation modules, i.e. the ground feature projection module, and the BEV feature refinement module. In addition, we propose a differentiable least squares pose regression module to filter matching inliers and calculate the camera pose through dense flow fields. Mathematically, given a ground image I g and a corresponding satellite image I s , the task is to determine the position t = (∆ u, ∆ v ) and orientation θ of the camera with respect to the satellite image. To achieve this, we propose a robust method that leverages the similarity between the points in the two images to estimate the corresponding satellite image ˆ p coordinates of each point p in I g . By finding the position relation between points, we can derive a geometric relation

Satellite Image

Feature Extractor

Feature Extractor

Ground Feature

Projection

BEV feature

RefineBlock

Correlation

GRU

Context Encoder

GRU

· · ·

Dense Flow

Estimation

Least Square

𝑥1

′

, 𝑦

1

′

↔

ො

𝑥1, ො

𝑦1

(𝑥 2

′

, 𝑦

2

′

)

↔

(ො

𝑥2, ො

𝑦2)

···

(𝑥𝑛

′

, 𝑦 𝑛

′

)

↔

(ො

𝑥𝑛, ො

𝑦n)

⟹ 𝜽,𝒕

function M that maps points between the two images such that ˆ p = M ( p ) . Next, we introduce the three key modules of our approach: (i) Feature extraction and view unification, (ii) Dense flow estimation, (iii) Least squares pose regression.

3.1 Feature Extraction and View Unification

In the presence of a large field-of-view error, it is a great challenge to match points directly to I g and I s . Our work addresses the significant differences in field-of-view between ground and satellite images by introducing a feature extraction module that incorporates projection transformation information.

The input I g and I s are first mapped to the feature map, F g = CNN g ( I g ) and F s = CNN s ( I s ) , where CNN g and CNN s are feature extraction modules. Merely relying on the feature extractor is insufficient to address the significant difference between the two fields of view. As a solution, we introduce the camera's intrinsics K , extrinsics T to perform perspective transformation. Specifically, for any world point x W , the perspective transformation maps it to its corresponding image coordinate x I , as follows:

Here, ≃ denotes equality up to a scale factor. However, since there is no exact ground height, the correspondence between ground and satellite images is ambiguous. To address this issue, we exploit the fact that the overlap between ground and satellite images mainly occurs on the ground plane[13]. Therefore, within our ground feature projection module, we utilize Eq. 1 to map ground pixels x I = [ p , 1] to the bird's-eye view (BEV) perspective via p ′ = G ( p ) by incorporating ground height h , where the world point is denoted by x W = [ p ′ , h, 1] . Subsequently, we employ bilinear interpolation to sample the ground image and project it onto a top-down view using this correspondence.

To further enhance the quality of our BEV feature map, we further propose a RefineBlock that addresses the following:

The RefineBlock is a key component in our network architecture, aiming to address the limitations of ground-to-overhead projection based on ground-plane homography, including incorrect correspondences for objects above the ground plane and distortions in the projections. To overcome these issues, the network should have a larger receptive field based on ground-plane homography projection of the input overhead view feature map for better and higher-level understanding of the feature map. Therefore, in the RefineBlock, a 7 × 7 convolutional layer is first employed to achieve a larger receptive field. Then, a 3 × 3 residual block is applied to refine the details in local regions. Finally, a 1 × 1 convolutional layer is utilized to facilitate feature interaction and integration, introducing enhanced non-linear transformation capabilities in the feature space. Ultimately, it generates a self-centered

BEV image that closely resembles the mental image humans construct during their own localization process.

3.2 Dense Flow Estimation

Similar to RAFT, the satellite features and the refined BEV features are subjected to a dot product operation, resulting in a four-dimensional correlation matrix that represents the similarity between every point in the two feature maps. This four-dimensional matrix is then subjected to average pooling with a 2 × 2 kernel on the last two dimensions, yielding features of varying resolutions. The features obtained from the different resolutions are then combined to form correlation volumes.

Our architecture consists of two separate feature extractors, namely the feature encoder and the context encoder. Here, we have utilized a feature encoder equipped with a view unification module to deal with the significant differences in viewing angles between the I g and I s . The context encoder has the same architecture as the feature encoder but is designed to extract only semantic and contextual information from the reference image I g .

Finally, we create an index based on the initialized match positions to query the correlation volume. We use the GRU structure to iteratively update the match positions, which generates the final match results p ′ ↔ ˆ p as well as the match score S for each point, which is dependent on the similarity scores obtained from the matching process.

3.3 Least Squares Pose Regression

In the case of a BEV image F g 2 s centered around itself, alignment with an uncalibrated satellite image can be achieved through rotation and translation. By doing so, it becomes apparent that the points p ′ i contained in F g 2 s and their corresponding points ˆ p i in F s are related through Euclidean transformations. So our objective is to find the optimal values of the positional parameters θ and t of the ground camera by minimizing the following loss function:

Here, the confidence score of the i -th match is represented by S i , which is comprised of two components. The first component indicates whether the i -th match is visible in the BEV image, with a value of 1 for visible matches and 0 for invisible matches. The second component is the score of the i -th match, which is calculated by the network based on the similarity of the matches.

By using the least square methods, we can calculate the relative translation and rotation of the camera in the satellite image through matching pairs of points. Detailed computation procedure is presented in the appendix materials. It is noteworthy that this process is differentiable, which implies that we can compute gradients and optimize the model's parameters through backpropagation.

3.4 Training Objective

Our loss function L consists of three components: matching loss L m , confidence loss L c , and position loss L p .

Matching Loss. The process of mapping matching points between F g 2 s and F s is supervised by L m . To evaluate the accuracy of the matching points, we calculate the distance between the matching points and the real matching points.

where for the i -th point, f gt i denotes its ground truth flow and f pred i denotes the predicted flow. The true matching points are generated by the camera poses, which provide a way to calculate the corresponding positions of each point in both F g 2 s and F s .

Confidence Loss. The loss of confidence generated by the network is measured using L c . For each matching point, the network generates a confidence score, and we aim for high confidence when the network predicts a small distance between the matching point and the true matching point, and low confidence when it predicts a large distance.

To balance the two terms of the formula, we first transform d into a standard normal distribution by using the formula ˜ d i = ( d i -mean ( d )) /std ( d ) . We then use κ to make the results more discretely distributed. In the beginning of the experiment, the variance of d is large, so we set κ to 200. After fifteen rounds of training, dis converges and the distribution becomes more concentrated. At this point, we set κ to 20.

Position Loss. We also utilized end-to-end supervised prediction, using L p , to estimate the difference between the predicted location and the true location. Although L m already incorporates location information, we observed that incorporating end-to-end supervision further improved the accuracy of the network's predictions.

where θ gt and t gt is the GT camera pose.

Total Loss. The overall loss is the sum of the matching loss L l m and confidence loss L l c generated by each iteration, as well as the final resulting position loss L p .

Here, the subscript l denotes the l -th iteration of the GRU. During training, we set α = 100 and initialize β = 1 . After 15 rounds of training, we increase the momentum parameter to β = 10 . This adjustment is made to prevent the model from getting trapped in local optima during the initial stage of training.

4 Experiments

In this section, we will begin by introducing the dataset and evaluation metrics employed. After that, we conduct a comparison between our proposed methodology and state-of-the-art approaches. Finally, we will present a comprehensive ablative study.

4.1 Datasets

We conducted tests on the KITTI[3, 13], Ford multi-AV[1, 13], VIGOR[5, 24], and Oxford RobotCar[8, 9, 22] datasets. The query images in KITTI, Ford multi-AV, and Oxford RobotCar datasets have a limited horizontal field of view (HFoV), while the query images in the VIGOR dataset are panoramic. Given that KITTI and VIGOR were captured on different road segments, we evaluated the model's performance on both same-area and cross-area scenarios using these two datasets. Ford multi-AV was mainly captured in suburban areas with fewer reference points, and Oxford RobotCar was captured on the same road segment at different times. We tested the model's generalization ability to different environments, including suburban scenes, varying lighting conditions, and road situations, on these two datasets. The datasets used in this paper are obtained under academic licenses and are not original datasets specifically created for this work. A more detailed description of each dataset will be provided in the appendix.

4.2 Evaluation Metrics

We followed the settings of [5, 13, 22]. Specifically, we calculate the mean and median errors in meters between predicted and ground truth positions, as well as the mean and median angular errors

in degrees between predicted and ground truth bearings. For the KITTI and Ford multi-AV datasets, we also included the recall values for longitudinal and lateral localization errors, as well as orientation estimation errors, under certain thresholds. Our localization thresholds were set at 1 meter and 5 meters, while our orientation estimation thresholds were set at 1 degree and 5 degrees.

4.3 Implementation Details

We employed ResNet18 to extract features from both ground and satellite images, respectively, ensuring that the resolution was maintained at 1/8th of the original with 256 channels. Subsequently, in the dense flow estimation stage, we performed 12 iterations to obtain the final dense matching relationship. Finally, we used the least squares method to estimate relative position and relative angles based on the estimated matching relationships. During the training period, we utilized the Adam optimizer[7] for end-to-end training, with a learning rate of 2 × 10 -5 , β 1 = 0 . 9 , and β 2 = 0 . 999 . The entire training schedule consisted of 25 epochs. At the 15th epoch, we adjusted the parameter κ in the loss function from 200 to 20 and β from 1 to 10. We conducted 25 rounds of training on two TITAN V GPUs with a batch size of 6. Each epoch required approximately two hours of training time. During the evaluation process, estimating the pose of a ground image took an average of 0.25 seconds and approximately occupied 6GB of GPU memory.

4.4 Same-Area Evaluations

We conducted performance evaluations of the model in the same region on both the KITTI and Vigor datasets. The results of the KITTI dataset tests are shown in Table 1. Our method significantly outperformed previous algorithms in camera pose estimation, with an average error reduction of 81% and a median error reduction of 89% compared to SliceMatch. Notably, our algorithm's predictive performance for vertical position in the same region was significantly higher than that of other algorithms, indicating that despite the limited field of view, our approach learned the semantic relationship between roadside buildings in this area and their corresponding features in satellite images.

In Table 3, the testing results on the VIGOR dataset show that our method achieves significantly lower average and median localization errors than SliceMatch by 42% and 62%, respectively, when the direction is known. When the direction is unknown, our algorithm still performs well, with both the average and median localization errors lower than those of SliceMatch.

Furthermore, our algorithm demonstrates good performance in angle prediction, with average and median angle errors lower than those of SliceMatch by 67% and 69%, respectively.

4.5 Cross-Area Generalization

Generalizing new ground images across different regions is a more challenging task than within the same region, as the test area may appear vastly different from the training area. Despite this increased difficulty, our approach still demonstrates strong performance. As shown in Table 1 Cross-Area, Our average localization error is 40% lower than SliceMatch, and our median localization error is 64% lower than SliceMatch. Compared to descriptor-based matching methods, our approach exhibits greater robustness.

In the cross area of VIGOR, our algorithm performs better than previous methods in both known and unknown directions. Additionally, as shown in Figure 3, we tested the performance of our algorithm across various angles using a model trained in unknown directions. Our algorithm demonstrated strong stability against noise for different angle variations in both the same area and cross area.

Location (mean)

8

7.6

7.6

7.6

7.6

17.5

18.1

17.9

17.7

17.6

17.6

7

15.0

6

5

5.0

5.1

5.2

5.0

12.5

10.0

11.4

12.0

12.7

11.2

4

3.8

8.3

3

7.5

2

5.0

1

2.5

0

180

0.0

0

180

Ours

MCC

GT

Ours

MCC

GT

Ours

LM

GT

(A) KITTI dataset

Ours

LM

GT

(B) Ford multi-AV dataset

Ours

MCC

GT

Ours

LM

GT

(C) VIGOR dataset

(D) Oxford RobotCar dataset

4.6 Vary-Environment Generalization

In addition, to validate the performance of our algorithm under different environmental conditions, we tested it on the Ford multi-A V dataset and the Oxford RobotCar dataset. The results of the Ford multi-AV dataset are presented in Table 2, which is primarily focused on suburban scenes with fewer surrounding buildings and difficulty in obtaining semantic information, resulting in generally poor localization accuracy. Moreover, in the limited HFOV scenario, the longitudinal position is more challenging than the lateral position in the driving direction. Consequently, the recall rate for the longitudinal position is significantly lower than that of the lateral position, and this trend applies to all the compared methods. Nonetheless, our algorithm still outperforms previous algorithms on this dataset, and we found that our lateral recall rate is significantly higher, indicating our algorithm's ability to capture road surface information under semantically unclear conditions. Our algorithm demonstrated superior performance across the three distinct test sets of the Oxford RobotCar dataset, as shown in Table 4. These results suggest that our approach exhibits strong generalization capabilities across diverse environmental conditions, seasonal variations, and roadway scenarios.

4.7 Ablation Study

Table 5 presents the results of our ablation experiments conducted on the KITTI dataset to evaluate the efficacy of the view unification mechanism comprised of Ground Feature Projection and BEV Feature RefineBlock, the loss function composed of Matching Loss and Position Loss, and the Confidence mechanism. We employ Ground Feature Projection and BEV Feature RefineBlock to address significant viewpoint differences between cross-view images and incorporate BEV and satellite image features within the same scope. Our experiments demonstrate that utilizing Ground Feature Projection and BEV Feature RefineBlock significantly improves matching accuracy, reducing average localization error by 57% and 27%, respectively. Furthermore, by incorporating matching points during the training process and employing a more robust supervised training approach, we

Ground image

Sat. image

Confidence map

BEV feature map Sat. feature map

Ground image

Sat. image

Confidence map BEV feature map Sat. feature map

(a)

(b)

observe a 57% decrease in average localization error. Finally, the addition of a confidence mechanism further enhances network precision, resulting in a 10% reduction in average error. Through conducting ablation experiments, we have demonstrated the efficacy of the view unification, loss, and confidence mechanisms proposed by us.

In Figure 5, we conducted a qualitative analysis of the confidence mechanism on the KITTI dataset. We utilized a threshold of 0.5 to distinguish between high-confidence matches and low-confidence matches. In these visualizations, circles represent matching points with high confidence, while stars indicate matching points with low confidence. Additionally, for clarity, we displayed estimated corresponding points on both the original ground and satellite images. From the visualizations, it is evident that pixels in the bird's-eye-view image, which are mutually visible in both ground and satellite images (e.g., road areas), tend to have higher confidence. On the other hand, pixels that are visible only in one view (such as building rooftops, tree canopies, and occluded regions) generally exhibit lower confidence. As a result, our confidence map effectively selects reliable matching points for accurate pose estimation. This holds significant implications for improving precision.

5 Conclusion

This paper presents a novel approach based on dense matching to address the problem of cross-view localization. Our approach utilizes a view unification module to bridge the visual gap between satellite and ground images. Additionally, we introduce a matching loss to provide stronger supervision for the model during training. To evaluate the effectiveness of our algorithm, we conduct experiments on four datasets that vary in geographical regions, environmental conditions, and road scenarios. Our results demonstrate the impressive generalization capability of our approach. One possible future direction is to explore the integration of spatial and temporal information, to further improve the accuracy and robustness of our approach for cross-view localization.

References

Siddharth Agarwal, Ankit Vora, Gaurav Pandey, Wayne Williams, Helen Kourous, and James McBride. Ford multi-av seasonal dataset. The International Journal of Robotics Research , 39(12):1367-1376, Sep 2020.

Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5297-5307, 2016.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7258-7267, 2018.

Ted de Vries Lentsch, Zimin Xia, Holger Caesar, and Julian FP Kooij. Slicematch: Geometry-guided aggregation for cross-view pose estimation. arXiv preprint arXiv:2211.14651 , 2022.

Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics , 2(2):164-168, 1944.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Will Maddern, Geoffrey Pascoe, Matthew Gadd, Dan Barnes, Brian Yeomans, and Paul Newman. Real-time kinematic ground truth for the oxford robotcar dataset. arXiv preprint arXiv:2002.10152 , 2020.

Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research , 36(1):3-15, 2017.

Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3384-3391. IEEE, 2010.

Filip Radenovi´ c, Giorgos Tolias, and Ondˇ rej Chum. Fine-tuning cnn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence , 41(7):1655-1668, 2018.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4938-4947, 2020.

Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17010-17020, 2022.

Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for image based cross-view geo-localization. Advances in Neural Information Processing Systems , 32, 2019.

Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am i looking at? joint location and orientation estimation by cross-view matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4064-4072, 2020.

Yujiao Shi, Xin Yu, Shan Wang, and Hongdong Li. Cvlnet: Cross-view semantic correspondence learning for video-based camera localization. In Computer Vision-ACCV 2022: 16th Asian Conference on Computer Vision, Macao, China, December 4-8, 2022, Proceedings, Part I , pages 123-141. Springer, 2023.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020.

Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixé. Coming down to earth: Satelliteto-street view synthesis for geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6488-6497, 2021.

Nam N Vo and James Hays. Localizing and orienting street views using overhead imagery. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 494-509. Springer, 2016.

Shruti Vyas, Chen Chen, and Mubarak Shah. Gama: Cross-view video geo-localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII , pages 440-456. Springer, 2022.

Shan Wang, Yanhao Zhang, and Hongdong Li. Satellite image based cross-view localization for autonomous vehicle. arXiv preprint arXiv:2207.13506 , 2022.

Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , pages 90-106. Springer, 2022.

Hongji Yang, Xiufan Lu, and Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer. Advances in Neural Information Processing Systems , 34:29009-29020, 2021.

Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-one retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3640-3649, 2021.

Appendix

A Theoretical Proof

We provide a detailed derivation and proof of the least-squares pose regression process in the text. The camera position parameters θ and t can be obtained by solving the following loss function.

The relationship between the corresponding points p ′ i and ˆ p i in point sets p ′ and ˆ p can be expressed as p ′ i = R ˆ p i + t . To simplify the problem, we calculate the centroids g ′ and ˆ g of point sets p ′ and ˆ p , respectively. We then subtract the centroids from each point in their respective sets, resulting in q ′ i = p ′ i -g ′ and ˆ q i = ˆ p i -ˆ g . This simplifies the problem to:

Therefore, minimizing ξ is equivalent to maximizing ∑ n i =1 S i ( ˆ q i t Rq ′ i ) . Assuming that H = ∑ n i =1 S i ( q ′ i ˆ q i t ) . then

Upon performing an SVD decomposition on H , the result is H = U Λ V t , where U and V are orthogonal matrices of dimensions 3 × 3 , and Λ is 3 × 3 diagonal matrix with non-negative elements.

Then we introduce an orthogonal matrix X = UV t , and observe that XH = V Λ V t is a symmetric positive-definite matrix. According to the properties of positive definite matrices, it holds that tr ( XH ) ≥ tr ( BXH ) for any 3 × 3 orthogonal matrix B . Therefore, we obtain the maximum value of ∑ n i =1 S i ( ˆ q i t Rq ′ i ) when R = X , which implies that

The dense matching relationship between the graphs gives rise to the uniqueness of solution for R. Subsequently, the relative displacement can be determined as

By utilizing equations 11 and 12, we can obtain the camera location and azimuth through a straightforward differentiable process, using the least-squares regression method.

B Datasets

KITTI. The ground images in the KITTI dataset were obtained from the original data. [13] collected satellite images of the KITTI dataset, with each ground image accompanied by a corresponding satellite image whose center corresponds to the position of the ground camera. The dataset is divided into Training, Test1, and Test2 subsets. The images in Test1 are from the same area as the images in the training set, while the images in Test2 are from different areas. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images is approximately 100 × 100 square meters.

Ford Multi AV. For the Ford Multi AV dataset, we only utilized the data of the V2 vehicles from the original dataset. Shi and Li first plotted the trajectories of the ground-view images and collected satellite images along these trajectories. Ground images captured on one date were used for training, while images captured on another date were used for testing. The satellite images remained unchanged during both training and testing. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images is approximately 100 × 100 square meters.

VIGOR. The satellite images of the VIGOR dataset were collected by [24]. The ground images are north-aligned panoramas, and they are from four cities: New York, San Francisco, Chicago, and Seattle. There is a positive satellite image in the database for each ground image, meaning that the ground image is within the center 1/4 quarter of the satellite image. All the satellite images are north-aligned. The dataset includes two evaluation splits: same-area and cross-area, based on whether the images in training and testing sets are from the same region. Research is conducted to estimate the relative pose between the ground image and its positive satellite image center. In this dataset, the ground resolution of satellite images for New York, San Francisco, Chicago, and Seattle is 0.113, 0.118, 0.111, and 0.101, respectively. The size of the aerial views for these four cities is 640 × 640 .

Oxford RobotCar. The satellite images of the Oxford RobotCar dataset were collected by [8]. They provided a very large satellite map covering the whole region where the ground images were collected. The ground images were collected through multiple traversals on a single route in Oxford, UK, encompassing various time periods, seasons, and weather conditions. During implementation, for each ground image, a random relative translation is generated, and a small satellite patch corresponding to the randomly generated pose is extracted from the large satellite map. Research is conducted to estimate the relative pose. The ground sample distance of the satellite images in this dataset is 0.0924m per pixel, and the coverage of the satellite images used is around 55 × 55 square meters.

C Comparison with LM[13]

To make a fair comparison with the LM[13], we trained both the LM and our method for an equal number of epochs and found that our method still outperforms the LM method.

Test 1

Test 2

12

Ours

LM

13

LM

Ours

10

12

Distance Average(m)

Distance Average(m)

8

11

6

10

4

0

5

10

15

20

25

0

5

10

15

20

25

Epoch

Epoch

The LM[13] aims to optimize the relative pose by minimizing the feature differences between corresponding pixels in two-view images. However, the pixel correspondences are computed based on planar homography (and relative camera pose), which is incorrect for objects above the ground plane. Minimizing feature differences between non-corresponding features leads to suboptimal relative pose estimation. In contrast, our objective in this paper is to first estimate the precise correspondences between the two views and then compute the relative pose. The estimation of correspondences is independent of the relative pose. Once the correspondences are sufficiently accurate and robust, the relative pose can be directly computed by solving a least squares problem instead of training additional networks. Therefore, the main capability of our method lies in accurate correspondence learning rather than relative pose estimation. As a result, our method achieves better performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of cross-view camera localization, i.e. of estimating a 3D camera pose for a ground-level image with respect to a satellite image of its local surroundings (assuming a coarse localization is already given). For this, it proposes a deep learning pipeline that aims at estimating a dense flow field between the jointly computed feature maps of the satellite image and a bird's-eye-view (BEV) rendering of the ground-level feature maps. Based on an initial correlation volume between those two feature maps, it then employs several GRU stages (similar to RAFT) in order to estimate the dense flow field and finally performs least-squares pose regression. The approach is quantitatively evaluated on the KITTI, Ford multi-AV, VIGOR and Oxford RobotCar datasets. Compared to previous approaches, the proposed method achieves a considerably improved localization accuracy.\n\nSTRENGTHS:\n+ The proposed approach is well-engineered and achieves considerably improved localization accuracies on all four datasets, compared with the previous SOTA methods. \n\n+ Experimental evaluations of 4 datasets and comparisons against several published baselines support the claimed contributions.\n\n+ The effects of the different elements of the proposed approach are investigated in an ablation study.\n\nWEAKNESSES:\n- The paper makes a relatively minor technical contribution. It presents a novel network architecture for a very specialized problem setting. The architecture is well suited for the estimation problem at hand and achieves very good localization accuracy, but the impact will still be limited, since the problem setting is interesting only for a relatively small subcommunity. Specifically, there is no strong machine learning content that would make this a particularly good fit for NeurIPS. \n\n- As the proposed approach relies on a coarse localization being already given, its success in practice will heavily rely on how robust the estimation is with respect to a coarse localization error. In other words, how close will the location of the satellite image already have to be to the camera location and over what search range will the approach reliably find a correct localization. Unfortunately, this is not discussed at all, nor is it reflected in the experimental design. The quantitative evaluation follows \"the settings of [14, 29, 43]\", which presumably means that the ground truth location lies within the center quarter of the satellite image tile?\n\nQUESTIONS:\nThis paper leaves a mixed impression. On the one hand, the proposed approach is well-engineered and achieves very good results. On the other hand, I am also not terribly excited about the paper, since it addresses a rather niche topic without a strong learning-focused contribution. I would be more excited about it in a more application focused conference, but I do not think that it has something great to offer particularly for the NeurIPS audience.\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with  the 3-DoF camera pose estimation method for a ground-level image with respect to a satellite image. The proposed end-to-end approach that leverages the learning of dense pixel-wise flow fields between the projected ground-level image and the satellite image is built upon RAFT architecture. \nEffectiveness of the proposed method is evaluated on competitive benchmarks and outperforms other SoTA methods in both location and azimuth measures.\n\nSTRENGTHS:\nAlthough the method is simple and the DNN structure is also easily understandable,  the proposed approach has achieved state-of-the-art (SoTA) performance by an adequate margin on various standard benchmarks.\nThe paper is written clearly and adequate figures support to understand the proposed approach.\n\nWEAKNESSES:\nThe main difference compared to [29], as mentioned in L. 75-77, is the part where camera poses are estimated from the explicit matching. However, the idea itself is well-known, for example, in [*], so I think the novelty of the approach is limited. While the representation of the loss function may differ, the basic idea has been previously addressed.\n\n[*]A.R. Zamir et al. \"Generic 3D Representation via Pose Estimation and Matching\", ECCV16.\n\nWhen dense correspondences are estimated, it is common to use RANSAC-based methods. It would be beneficial to compare the proposed method with RANSAC-based approaches to evaluate their performance.\n\nQUESTIONS:\n- Are the colors in Figure 1 meant to represent the same feature by using the same color? Also, the road appears red in the original image (Figure 1(B)), but it appears green in the BEV (Figure 1(E)). Additionally, why isn't the vertical road in the satellite image shown in red?\n- Is Equation 5 correct? When rearranged, it becomes $\\frac{1}{1+exp(-\\tilde d_i/ \\kappa)}$, and $S_i$ disappears.\n- Why is the error of Localization/Azimuth for \"Ours\" in Table 1 (4.97/11.2) different from the values shown in Figure 3 (3.8/8.3)?\n-  How is the ground truth for the dense estimation flow in Equation 4 actually obtained? It is mentioned that it is determined by the camera pose in L.188, but in reality, the 3D position (depth) will be required.\n\nLIMITATIONS:\nWhen it comes to practical use, I believe it is important to verify the robustness and accuracy of the initial value between ground-level images and satellite images.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors try to address the cross-view geo-localization by learning dense pixel-wise flow fields between ground and satellite images. To achieve this, they utilize a ground feature projection module to align the spatial feature and propose a refinement block to enhance the context feature for accurate optical flow estimation. However, some details about refinement modules, data collection, and experiment setup are unclear.\n\nSTRENGTHS:\nThis paper tries to address the cross-view geo-localization by learning dense pixel-wise flow estimation. And the authors conduct an account of experiments on KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar.\n\nWEAKNESSES:\nHowever, there are some concerns about this paper:\n1. More detail about the satellite image collection. In Sec 4.1, the authors only present the detail about the ground-view dataset. Is there any detail about the satellite image, including collection, preprocessing, ground sample distance, data distribution, and so on?\n2. As stated in Line 146, RefineBlock is a crucial module of the proposed method. However, the authors provide no detail about RefineBlock. Please provide the detail about RefineBlock. And more analysis and experiments are appreciated to demonstrate the importance of RefineBlock.\n3. LM [1] are trained with 2 epochs. However, this method is needed to train with 25 epochs in Line 237, which is 10x longer than LM [1]. Are the results of LM [1] in Table 1 and Table 2 trained with 25 epochs? Could you provide the results of LM [1] trained with 25 epochs?\n4. The training schedule is unclear. In Line 235, the feature extractors are optimized with 12 epochs in the Dense Flow Estimation stage. However, in Line 237, the pipeline is conducted with 25 epochs in an end-to-end schedule. And in Line 207, the momentum parameter is changed to 10 after 15 epochs. Hence, I am confused about the training schedule. Does it mean the whole pipeline consists of two stages and is trained with 37 epochs in total?\n5. The most similar state-of-the-art work is proposed by Shi and Li [1]. In Table 4 of LM [1], the results show that LM achieves better translation performance but inferior rotation performance than the network-based optimizer, including GRU. Hence, the relationship and main difference with the prior work [1] are appreciated to be discussed in the related work. It is better to analyze the advantages and disadvantages of the proposed approach compared with LM [1]. And more experiments and analyses are appreciated.\n\nReference\n\n[1] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. CVPR 2022.\n\nQUESTIONS:\nSee the Weaknesses.\n\nLIMITATIONS:\nThe limitations and potential negative societal impact are not explicitly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method that estimates a 3-DoF pose that aligns correspondences between ground-level and satellite/aerial images. The proposed solution uses a CNN-based model to extract image features, map image features from the ground-level camera to a BEV so that it can be matched with features from the satellite image. Lastly, a least-square regression us run to estimate the final 3-DoF pose (rotation and translation). The paper presents results on the KITTI and Ford multi-AV datasets where the proposed method consistently shows performance improvements over a few baselines.\n\nSTRENGTHS:\nIn sum, I think the proposed solution is simple and straightforward. Thus, I think the proposed approach is easily reproducible.\nThe particular strengths I see are the following:\n\nS1. Simplicity and straightforward of solution. The solution essentially mimics a classical dense matching of features coming from two different images (i.e., aerial/satellite view and ground-level view). Because of this simplicity, I think the proposed solution can be easily adopted and reproduced.\n\nS2. End-to-End approach. I do like that the proposed solution includes the feature matching pipeline but also optimizing for the 3-DoF pose which ultimately is the final solution. This is great as it directly learns features that are useful to obtain the desired pose.\n\nS3. Use of a confidence value as part of the solution is also very useful. I think this should be able to handle outliers and allow a robust computation of the estimated pose.\n\nWEAKNESSES:\nWhile I value the simplicity of the solution, I question its novelty. To me, the novelty is very important as this is the way we break common perspectives and open new avenues of research. In detail here are my concerns about novelty:\n\nW1. The proposed solution uses CNN features in the era of Transformers. While this is a feasible solution (as shown by the experiments), it has been shown in the computer vision community that attention mechanisms can help in feature matching and also deal with outliers; a good example of this is LoFTR presented at CVPR 2021. Unfortunately, I don't see any justification in the paper about the use of CNNs when Transformers inherently provides mechanisms to do feature matching.\n\nW2. The proposed solution mainly is composed of well known components, and the proposed method is quite straightforward as it follows classical pipelines of feature matching (e.g., PatchMatch-Based Neighborhood Consensus for Semantic Correspondence CVPR 2021) combined with well known optimization methods to estimate a pose.\n\nW3. The paper uses terms in a wrong way. For example, in Eq. 3 the $\\mathbf{t}$ is introduced as position in line 119. However, the Eq. 3 shows this is a translation and not a position. This is very confusing as it conflicts the math and the stated definition, which decreases clarity. Second, the $G(\\cdot)$ introduced in line 141 is not defined. What is it? Lastly, in Eq. 3, the loss seems to be using L2 norm and not using a squared error on scalars as suggested by the parenthesis in the equation. Thus, the \"Position Loss\" described in 199 is not optimizing directly the position but rather the pose since it penalizes deviations of the rotation angles and translations.\n\n--------\nPost Rebuttal.\n\nI am satisfied with the answers the authors have provided. Although I think the proposed is simple and straightforward, I think it produces good results according to the experiments. I think the community can get intrigued by the results and the simplicity of the methods, and thus I expect this paper to cause some interest in the CV community. For these reasons, I will upgrade my rating to borderline accept.\n\nQUESTIONS:\nPlease see Weaknesses section.\n\nLIMITATIONS:\nThe limitations of the paper are not properly discussed. I think the paper may suffer from images with repetitive features (e.g., an image of a large field of grass, forests, etc.). It is unclear if the proposed method of estimating a confidence can handle this cases correctly. Lastly, since this technology matches aerial views with ground-level views, this can be used for surveillance. I think this an ethics review for this case.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a method for estimating 3DOF camera pose of satellite images given a ground-level image and an initial pose estimate as input. The ground-level image is mapped to a satellite view using the camera parameters, confidence-weighted 2D matches are estimated between the warped image and the satellite image, from which the 3DOF parameters are solved. The authors show that this approach works well in the wild across many standard datasets, and outperforms prior work by a large margin.\n\nSTRENGTHS:\nThe method follows a procedure that has worked very well for other 3D vision problems (SLAM, SfM): 1) Align features 2) 2D-2D matching 3) Solve for camera pose which explains the 2D observations. This sounds more principled than the prior work.\n\nThe method is interpretable and learns outlier rejection automatically.\n\nThe paper is very well written and easy to understand.\n\nThe authors provide an analytical solution to their solver layer, as opposed to prior work which uses the LM algorithm to align features.\n\nStrong empirical results across many datasets compared to prior work.\n\nPaper includes ablation experiments on several of the components\n\nWEAKNESSES:\nThe related work section doesn’t explain the main difference between the proposed method and the referenced methods. For example, what is the main difference between *Wang et al.[42]* and the proposed method? At face-value they sounds similar.\n\nPaper is missing resource requirements (time or memory) compared to prior work\n\nQUESTIONS:\nWhy do the confidence estimates need their own loss? Can they not be learned automatically just by supervising on the solver output (e.g. DROID-SLAM)? \n\nIn Eq. 1, how is T obtained? Is it provided as input or is there a canonical ground-camera pose assumed (e.g. upright, pointing north, 0 translation)?\n\nLIMITATIONS:\nNo limitations section, so this is not clear.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}