Learning Dense Flow Field for Highly-accurate Cross-view Camera Localization
Authors: Zhenbo Song
Xianghui Ze
Jianfeng Lu
Yujiao Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Optical flow
correspondence learning
cross-view
camera localization
TL;DR: Learning Dense Flow Field for Highly-accurate Cross-view Camera Localization

Abstract: This paper addresses the problem of estimating the 3-DoF camera pose for a ground-level image with respect to a satellite image that encompasses the local surroundings. We propose a novel end-to-end approach that leverages the learning of dense pixel-wise flow fields in pairs of ground and satellite images to calculate the camera pose. Our approach differs from existing methods by constructing the feature metric at the pixel level, enabling full-image supervision for learning distinctive geometric configurations and visual appearances across views. Specifically, our method employs two distinct convolution networks for ground and satellite feature extraction. Then, we project the ground feature map to the bird's eye view (BEV) using a fixed camera height assumption to achieve preliminary geometric alignment. To further establish the content association between the BEV and satellite features, we introduce a residual convolution block to refine the projected BEV feature. Optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT. After obtaining dense flow correspondences, we apply the least square method to filter matching inliers and regress the ground camera pose. Extensive experiments demonstrate significant improvements compared to state-of-the-art methods. Notably, our approach reduces the median localization error by 89\%, 19\%, 80\%, and 35\% on the KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar datasets, respectively.

Learning Dense Flow Field for Highly-aceurate Cross-view Camera Localization
Zhenbo Song'; Xianghui Ze': Jianfeng Lu', Yujiao ShiPt
'Nanjing University of Science and Technology. "ShanghaiTech University {songzb, zexh, lujf] @njust. edu. en, shixy22shhaghaitechh edu. cn
Abstract
This paper addresses the problem of estimating the 3-DoF camera pose for a ground-level image with respect satellite image that encompasses the local surroundings. We propose a novel end-to-end approach that leverages the learning of dense pixel-wise flow fields in pairs of ground and satellite images calculate the camera pose. Our approach differs from existing methods by constructing the feature metric at the pixel level, enabling full-image supervision for learning distinctive geometric configurations and visual appearances across views. Specif- ically, our method employs two distinet convolution networks for ground and satellite feature extraction. Then, we project the ground feature map to the bird's s eye view (BEV) using a fixed camera height assumption achieve preliminary geometric alignment. To further establish the content association between the BEV and satellite features, we introduce a residual convolution block to refine the projected BEV feature. Optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based RAFT. After obtaining dense flow correspondences, we apply the least square method to filter matching inliers and regress the ground camera pose. Extensive experiments demonstrate significant improvements compared to state-of-the-art methods. Notably, our approach reduces the median localization error by 89%, 19%, 80%, and 35% the KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar datasets, respectively.
Introduction
Cross-view camera localization, also known as ground-to- touudd-t--rrrillssteliiice image matching, has evolved from the study of coarse localization through image retrieval 4 12
16 20 to highly accurate 3-DoF pose estimation task B 22 24). This task involves accurately determining the precise position and orientation of a ground camera in relation to satellite patch that encompasses the surrounding environment.
The key challenge in cross-view camera localization is to bridge the gap between the visual observa- tions captured by the ground camera and the corresponding top-view camera, enabling robust and accurate regression of the camera's 's pose in real-world spatial coordinates. This capability holds immense significance across various applications, including augmented reality, autonomous naviga- tion, virtual reality, and robotics, where precise camera localization is crucial for enabling advanced functionalities and interactions with the environment.
This paper focuses on investigating the problem of single-frame ground image localization, which aims estimate the 3 DoF pose of ground camera relative single satellite image. This problem
"Equal contribution "Coreepooning author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A) Ground image
BEV Projection
(B) Ground feature
Ours
(C) Satellite imago (D) Satellite feature
(E) Dense matching
(F) Pose estimation
Figure I: Visualizations of the ground and aerial feature maps, and the dense matching correspon-
dences cross views for localization. For visual simplicity, dense matching displays only partial inliers.
assumes that the satellite image covers the same geographic region as the ground image. The objective of localization is to determine which pixel of the satellite image corresponds to the ground camera's 's position and its current shooting direction [24] practical applications. as each pixel of the satellite image is associated with GPS labels, it becomes feasible to obtain the geographical location information and spatial orientation information of the ground camera.
Most current methods address this problem by implicitly learning cross-view correspondences K [2 24]] These approaches establish aggregated descriptor- based representations for both ground and aerial images, resulting in relatively short inference time. However, these methods suffer from two main issues. Firstly, they lack an explicit establishment of spatial unification between views and heavily rely on brute-force training to simultaneously solve spatial and semantic alignment. Secondly, in theory, precise relative pose estimation requires obtaining local cross- cross-view matching relationships, such as point-to- toptont correspondences. Implicit cross-view feature correlation and pose regression may lead to loss in localization accuracy. In contrast, methods
regress camera localization (5]
[3 based on point-to-point matches have been proposed, which utilize LM optimization for feature map alignment between the ground and satellite views. These methods often leverage end-to-end ground truth pose as supervision to learn local correspondences between cross-views. Such supervision may be weak for point-wise deep feature metric learning. Based on the above observations, we propose a method that explicitly represents point-wise correspondences between cross-views, thus facilitating accurate pose estimation. Our key idea is to learn dense flow fields between cross-views and directly compute accurate ground camera poses from such flows in an end-to-end manner. To achieve this, we develop two distinct convolutional neural networks for feature extraction from the acrial and ground images, respectively. But ground and aerial feature maps exhibit significant domain differences in terms of spatial configuration and visual appearance, posing a critical challenge for the neural network to learn effective matching such feature maps. Furthermore, due to the absence of depth information in the ground images, direct cross matching leads one-to-many correspondence issue. For instance, building facade in the ground image may correspond to multiple edges in the aerial image. Existing methods often assume a fixed ground height enable the projection from the ground to the satellite image In line with this approach, our method employs the same strategy to geometrically project the ground features, obtaining a bird's eye view (BEV) feature map that corresponds the ground image. This projection facilitates spatial alignment between the cross-view feature maps. As mentioned above, for the uncertainty in the ground height assumption, we introduce RefineBlock to further refine the BEV features and achieve content alignment. Subsequently, optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT[I7). Specifically, we calculate multi-scale correlation volumes using the BEV and satellite features and employ GRU modules iteratively for dense flow estimation. Furthermore, we assign scores to each established matching relationship based on their similarity, effectively reducing the weight of erroneous matches. notable advantage of computing flows on the BEV and satellite features is that the matching relationship allows us to directly utilize the least square method to solve the DoF pose, i.e., the

---

2-DoF translation, and I-DoF rotation. In order to learn robust matching results, we use the dense optical flow as well the ground truth pose as supervision train the network progressively.
Our main contributions can be summarized as follows:
This paper introduces novel approach that learns dense pixel-wise flow fields between ground and satellite images. This enables accurate cross-view camera localization by directly caleulating precise ground camera poses from the learned flow.
We propose spatial alignment and content refinement modules for ground image feature enhancement. By leveraging the BEV representation of ground feature maps, the proposed method achieves accurate optical flow estimation according to both spatial and semantical alignments.
We conduct extensive experiments on various datasets, including KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar, which demonstrates that our method achieves superior performance and exhibits strong generalization ability across all datasets.
2 Related Works
Localization based on retrieval. Image retrieval methods are widely utilized for same-view localization tasks 10. This approach assumes the availability of reference dataset containing a vast number of geotagged ground images. Query images are then matched with ground images in the reference dataset to determine their corresponding geographic locations. However, this method requires a significant number of ground markers, which can be expensive. As satellite maps become more widely available, cross-view localization has gained attention as potential alternative. In cross-view retrieval tasks, a common approach is to create separate global descriptors for the ground image and the satellite image, and then compare their similarity [4] 14 18
19 23 Alternatively,
recent studies have explored fusing consecutive multi-frame ground images to generate a descriptor, which is then matched with the descriptor the satellite image [66 20]. this cross-view retrieval methods, the camera is assumed to be located at the center of the satellite map. However, this assumption limits the generalization of these methods.
Cross-view camera pose estimation. This work differs from the retrieval task in that it relies on an initial estimate of given camera pose and then proceeds to refine it to obtain pose that close to the true value. Previous works have used various approaches to tackle this problem. For example, (24 first uses an end-to-end framework to roughly locate the query through retrieval and then refines the location by predicting offsets through regression. Xia al. [22) formulates the localization problem as a multi-class classification problem and uses a dense multimodal space for localization.
13 uses geometric projection module to align the eross- view features, and then optimize the camera pose end-to-end using the wweebbeg-Marquuar
algorithm[6] 6 Wang et al. [21 uses 3D points to bridge the geometric gap between ground and overhead views, iteratively fusing ground images with 3D points iteratively for pose estimation. SliceMatch 5 is given a set of candidate photo poses and then compares single ground descriptor with of pose-dependent satellite map descriptors to generate the final camera pose. It is worth noting that previous works have primarily relied on global descriptors (22 24 or local descriptors 6 13 21] for cross-view positional estimation. However, in our work, we propose an alternative approach that relies on dense point matching, which we believe to be reliable basis for cross-view localization.
3 Methods
As shown in Figure the proposed method is built upon the RAFT architecture. The main differences are the heterogeneous ground and aerial feature extractors and feature transformation modules, i.e. the ground feature projection module, and the BEV feature refinement module. In addition, we propose differentiable least squares pose regression module to filter matching inliers and calculate the camera pose through dense flow fields. Mathematically, given ground image I, and corresponding satellite image I,, the task is to determine the position t = (Au, Av) and orientation 0 of the camera with respect to the satellite image. To achieve this, we propose a robust method that leverages the similarity between the points in the two images to estimate the corresponding satellite image Đ coordinates of each point p in Lg. By finding the position relation between points, we can derive geometric relation

---

Correlation Volumes
Satellite Image Feature Extractor
GRU
GRU
Context Encoder
Dense Flow Estimation
(4 (4) (x2.9)) + (R2.92)
(0.t)
Ground Image
BEV feature RefineBlock Figure Illustration of the pipeline our proposed method.
Ground Feature Projection
Least Square Pose Regression
Feature Extractor
function M that maps points between the two images such that ý M Next, we introduce the three key modules of approach: (i) Feature extraction and view unification, (ii) Dense flow estimation, (ii) Least squares pose regression.
3.1 Feature Extraction and View Unification
the presence of a large field-of-view error, it great challenge to match points directly to I, and I, Our work addresses the significant differences in field-of-vier between ground and satellite images by introducing a feature extraction module that incorporates projection transformation information.
The input Lg and I, are first mapped to the feature map. F = (g and F = CNN. (1.). where CNN, and CNN, are feature extraction modules. Merely relying on the feature extractor insufficient to address the significant difference between the two fields of view. As solution, we introduce the camera's intrinsics K, extrinsics to perform perspective transformation. Specifically, for any world point x the perspective transformation maps it to its corresponding image coordinate a. as follows:
(1) Here, < denotes equality up to scale factor. However, since there is exact ground height, the correspondence between ground and satellite images is ambiguous. To address this issue, we exploit the fact that the overlap between ground and satellite images mainly occurs on the ground plane 13 Therefore, within our ground feature projection module, we utilize Eq. Io map ground pixels 2x [p, 1] to the bird' s-eye view (BEV) perspective via p' = G(p) by incorporating ground height h, where the world point denoted by x W = [p',
KTx W
, h, 1]. Subsequently, we employ bilinear interpolation sample the ground image and project it onto top-down view using this correspondence.
To further enhance the quality of our BEV feature map, we further propose a Refine that addresses the following:
g2s = Re Block
The Re Block is key component in our network architecture, aiming to address the limitations of ground-to-oon to-overhead projection based on ground-plane homography, including incorrect correspon- dences for objects above the ground plane and distortions in the projections. To overcome these issues, the network should have larger receptive field based on ground plane homography projection of the input overhead view feature map for better and higher-level understanding of the feature map. Therefore, the RefineBlock, 7 convolutional layer is first employed to achieve larger recep- tive field. Then, a 3 x 3 residual block is applied to refine the details in local regions. Finally, a x convolutional layer utilized to facilitate feature interaction and integration, introducing enhanced non-linear transformation capabilities in the feature space. Ultimately, it generates self-centered

---

BEV image that closely resembles the mental image humans construct during their own localization process.
3.2 Dense Flow Estimation
Similar to RAFT, the satellite features and the refined BEV features are subjected to a dot product operation, resulting in four-dimensional correlation matrix that represents the similarity between every point in the two feature maps. This four-dimensional matrix is then subjected to average pooling with a 2 kernel on the last two dimensions, yielding features of varying resolutions. The features obtained from the different resolutions are then combined to form correlation volumes.
Our architecture consists of two separate feature extractors, namely the feature encoder and the context encoder. Here, we have utilized a feature encoder equipped with a view unification module to deal with the significant differences in viewing angles between the 1 and g.. The context encoder has the same architecture as the feature encoder but is designed to extract only semantic and contextual information from the reference image Iy.
Finally, we create an index based on the initialized match positions to query the correlation volume. We the GRU structure to iteratively update the match positions, which generates the final match results p' + P as well as the match score S for each point, which is dependent on the similarity scores obtained from the matching process.
3.3 Least Squares Pose Regression
In the case of a BEV image Eg2, centered around itself, alignment with an uncalibrated satellite image be achieved through rotation and translation. By doing so, it becomes apparent that the points P. contained in F Fg2s and their corresponding points P. in F, are related through Fuclidean transformations. So objective is to find the optimal values of the positional parameters and t of the ground camera by minimizing the following loss function:
cos 0
sin Ø
{ = arg min 0,t
Σ S.Rpp' +t +
R R=
sin 8
cos
Here, the confidence score of the i-th match represented by S,, which is comprised of two components. The first component indicates whether the i-th match is visible in the BEV image, with a value of 1 for visible matches and for invisible matches. The second component is the score of the i-th match, which is calculated by the network based the similarity of the matches.
By using the least square methods, we can calculate the relative translation and rotation of the camera in the satellite image through matching pairs of points. Detailed computation procedure is presented in the appendix materials. It is noteworthy that this process is differentiable, which implies that we can compute gradients and optimize the model's parameters through backpropagation.
3.4 Training Objective
Our loss function L consists of three components: matching loss C.. confidence loss Le, and position loss Cp.
Matching Loss. The process of mapping matching points between F 92s and F, is supervised by Lm- To evaluate the accuracy of the matching points, we calculate the distance between the matching points and the real matching points.
Cm
fgd ffreed
(4)
where for the i-th point, J denotes its ground truth flow and frred denotes the predicted flow. The true matching points are generated by the camera poses, which provide a way to calculate the corresponding positions of each point in both F o2s and F..
5

---

The loss of confidence generated by the network is measured using Le. For each matching point, the network generates a confidence score, and we aim for high confidence when the network predicts small distance between the matching point and the true matching point, and low confidence when it predicts large distance.
Confidence Loss.
1- S L. = Σ 1+ exp(-d./ + exp(di/k)
S
112
(5)
To balance the two terms of the formula, first transform d into standard normal distribution by using the formula d, = (d: mean(d))/sld(()) We then use to make the results more discretely distributed. the beginning the experiment, the variance is large, so we set K to 200. After fifteen rounds of training, dis converges and the distribution becomes more concentrated. At this point, we set A to 20.
Position Loss. We also utilized end-to-end supervised prediction, using Cp. estimate the differ- ence between the predicted location and the true location. Although L. already incorporates location information, observed that incorporating end-to-end supervision further improved the accuracy of the network's ' predictions.
C g0g 0 + gl t||2
(6)
where ggt and tg: is the GT camera pose.
Total Loss. The overall loss is the sum of the matching loss L and confidence loss C generated by each iteration, well the final resulting position loss Lp.
+ E + ac.) L 36
1=1
Here, the subscript I denotes the I-th iteration of the GRU. During training, we set a 100 and
initialize 3 = 1. After 15 rounds of training, we increase the momentum parameter to B 10. This adjustment made to prevent the model from getting trapped in local optima during the initial stage of training.
4 Experiments
In this section, we will begin by introducing the dataset and evaluation metrics employed. After that, conduct comparison between proposed methodology and state-of-the-e approaches. Finally, we will present a comprehensive ablative study.
4.1 Datasets
We conducted tests on the KITTIß 13
Ford multi-Av[I 13), VIGOR 24 and Oxford RobotCar g 22 datasets. The query images in KITTI, Ford multi-AV, and Oxford RobotCar datasets have limited horizontal field of view (HFoV), while the query images in the VIGOR dataset are panoramic. Given that KITTI and VIGOR were captured on different road segments, we evaluated the model's performance on both same-area and cross-area scenarios using these two datasets. Ford multi-AV was mainly captured in suburban areas with fewer reference points, and Oxford RobotCar captured the same road segment at different times. We tested the model's generalization ability to different environments, including suburban scenes, varying lighting conditions, and road situations, these two datasets. The datasets used in this paper obtained under academic licenses and are not original datasets specifically created for this work. A more detailed description of each dataset will be provided in the appendix.
4.2 Evaluation Metrics
We followed the settings of 13 22 Specifically, we caleulate the mean and median errors in meters between predicted and ground truth positions, well as the mean and median angular errors

---

Table Location and orientation estimation error and recall on KITTI dataset [3) Angular noise was constrained within range of =
[3]
(ocation(m)
(%)
Longitudinal(9))
Azimuth(*) mean median
Azimuth(%) r@
Area
Model
mean median
r@ Im r@5m
r@ Im
r@ rm
r@5
DSM 15
10.12
48.24
4.08
20.14
3,58
24.44
LM 13
12.08
11.42
35.54
80.36
5.22
26.13
3.72
2.83
19.64
71.72
Same
SliceMatch S
7.96
4.39
49.09
98.52
15.19
57.35
4.12
3.65
13.41
64.17
Ours
1.48
0.47
95.47
99.79
87.89
94.78
0.49
0.30
89.40
99.31
DSM 15
10.77
48.24
3.87
19.50
3.53
23.95
LM 3
12.58
12.11
27.82
72.89
5.75
26.48
3.95
3.03
18.42
71.00
Cross
SliceMatch S
13.50
9.77
32.43
86.44
8.30
35.57
4.20
6.61
46.82
46.82
Ours
7.97
3.52
54.19
91.74
23.10
61.75
2.17
1.21
43.44 89.31
Table 2: Location and orientation estimation error and recall on Ford multi-AV dataset 13
angular
noise was constrained within a range of +00 [3
Location(m) mean median
Lateral %)
Longitudinal %))
Azimuth(" )
Azimuth(%) (%)
Area
Model
r@ Im
r@ 5m
r@ Im
r@ 5m
mean median
r@1
r@5
[SM$
12.00
53.67
4.33
21.43
3.52
13.33
Log1
LM
12.54
12.63
48.57
71.57
5,90
26.33
3.13
1.29
42.90
79.62
Ours
10.55
10.19
68.67
95.48
5.48
25.86
1.43
0.47
75.76
94.29
DSM*[5]
8.45
37.64
3,94
21.41
2.23
13.42
Log2
LM
12.01
11.49
29.94
77.78
4.94
26.00
4.36
3.75
15.35
63.00
Ours
10.50
9.40
33.43
97.59
15.62
40.94
1.65
1.03
48.99
95.57
in degrees between predicted and ground truth bearings. For the KITTI and Ford multi-AV datasets, we also included the recall values for longitudinal and lateral localization errors, as well as orientation estimation errors, under certain thresholds. Our localization thresholds were set at I meter and meters, while our orientation estimation thresholds were set at I degree and 5 degrees.
4.3 Implementation Details
We employed ResNet18 to extract features from both ground and satellite images, respectively, ensuring that the resolution was maintained at 1/8th the original with 256 channels. Subsequently, in the dense flow estimation stage, we performed 12 iterations to obtain the final dense matching relationship. Finally, we used the least squares method to estimate relative position and relative angles based the estimated matching relationships. During the training period, utilized the Adam oe
optimizer[Z] [ for end-to-end training, with a learning rate of x 10-3, 31 = 0.9, and 32 = 0.999. The entire training schedule consisted of 25 epochs. At the 15th epoch, adjusted the parameter in the loss function from 200 to 20 and /3 from to 10. We conducted 25 rounds training on two TITAN V GPUs with a batch size of 6. Each epoch required approximately two hours of training time. During the evaluation process, estimating the pose of a ground image took an average of 0.25 seconds and approximately occupied 6GB of GPU memory.
4.4 Same-Area Evaluations
We conducted performance evaluations of the model in the same region on both the KITTI and Vigor
datasets. The results of the KITTI dataset tests shown in Table [T Our method significantly outperformed previous algorithms camera pose estimation, with an average error reduction of 81 % and median error reduction of 89% compared to SliceMatch. Notably, our algorithm's 's predictive performance for vertical position in the same region was significantly higher than that of other algorithms, indicating that despite the limited field of view, our approach learned the semantic re relationship between roadside buildings in this area and their corresponding features in satellite images.
7

---

Table 3: Location and orientation estimation error and recall on VIGOR dataset 24 Aligned Images means the ground image orientation is known.
Same-Area
Cross-Ares
Model
Aligned
(ocation(m) median
Azimuth(*) mean
(ocation(m) median
Azimuth(") median
mean
median
CVR
8.99
7.81
8.89
7.73
MCC 22
6.94
3.64
9.05
5.14
SliceMatch
5.18
2.58
5.53
2.55
Ours
3.03
0.97
5.01
2,42
SliceMatch X 8.41
5.07
28.43
5.15
8.48
5.64
26.20
5.18
Ours X
4.97
1.90
11.20
1.59
7.67
3.67
17.63
2.94
Table 4: Location eтror on Oxford RobotCar dataset
g 9
22
The orientation of the ground image is
known 27
Test Location(m)
Test2 Location(m)
Test3 Location(m)
Model
mcan
median
mean
median
mean
median
CVR 24 MCC 22
1.88
1.47
2.64
1.99
2.35
1.71
1.42
1.10
1.95
1.33
1.94
1.29
Ours
1.17
0.72
1.76
0.97
1.79
0.92
In Table 3 the testing results the VIGOR dataset show that our method achieves sig- nificantly lower average and median localiza- tion errors than SliceMatch by 42% and 62%, respectively, when the direction is known. When the direction is unknown, our algorithm still performs well, with both the average and median localization errors lower than those of SliceMatch.
17.
172
12.5
11.
Location (trass))
Azimuth (mesn)
5.0
12.5
30.0
5.0
35
Azimuuh noise
32
180
Azimuth noise
56
135
180
Furthermore, our algorithm demonstrates good performance in angle prediction, with average and median angle errors lower than those of SliceMatch by 67% and 69%, respec- tively.
Figure The localization error under varying noise conditions at different angles on the VIGOR dataset. Orange indicates cross-area, and blue indicates same- area.
4.5 Cross-Area Generalization
Generalizing new ground images across different regions is more challenging task than within the same region, the test area may appear vastly different from the training area. Despite this increased difficulty, our approach still demonstrates strong performance. As shown in Table CrosssCroeeee Our average localization error is 40% lower than SliceMatch, and our median localization error is 64% lower than SliceMatch. Compared to descriptor-bart based matching methods, our approach exhibits greater robustness.
In the cross area of VIGOR, our algorithm performs better than previous methods in both known and unknown directions. Additionally, as shown in Figure we tested the performance of our algorithm across various angles using model trained unknown directions. Our algorithm demonstrated strong stability against noise for different angle variations in both the same area and cross area.

---

Dan
07
(A] TI dataset
mulb-AV dataset
VIGOR dataset
(D) Oxtunt RobetCar dataset
Figure 4: For each scene, the up left is the ground image, the bottom left denotes matching inliers and the right shows the satellite image and localization results.
Table 5: Ablation Study on the KITTI dataset 3
Same-Area Location(m) mean median
Cross-Area
Model
Azimuth(") mean median
Location(m) mean median mean median
Azimuth(")
Ours w/o Ground Feature Projection
3.52
2.66
2.09
1.48
9.83
7.11
3.13
2.22
Ours w/o BEV Feature RefineBlock
2.02
0.78
0.82
0.53
8.72
4.55
2.56
1.47
Ours w/o Matching Loss
9.06
8.10
1.93
1.12
10.60
9.97
2.75
1.82
Ours w/o Position Loss
0.99
0.53
0.51
0.36
9.63
4.79
2.87
1.65
Ours w/o Confidence
1.65
0.65
0.98
0.75
7.76
3.57
2.31
1,50
Ours
1.48
0.47
0.49
0.30
7.97
3.52
2.17
1.21
4.6 Vary-Environment Generalization
In addition, to validate the performance of our algorithm under different environmental conditions, we tested it the Ford multi-AV dataset and the Oxford RobotCar dataset. The results the Ford multi-AV dataset are presented in Table which is primarily focused on suburban scenes with fewer surrounding buildings and difficulty in obtaining semantic information, resulting generally poor localization accuracy. Moreover, in the limited HFOV scenario. the longitudinal position is more challenging than the lateral position in the driving direction. Consequently, the recall rate for the longitudinal position is significantly lower than that of the lateral position, and this trend applies to all the compared methods. Nonetheless, our algorithm still outperforms previous algorithms on this algorithm's 's
dataset, and we found that our lateral recall rate is significantly higher, indicating ability to capture road surface information under semantically unclear conditions. Our algorithm demonstrated superior performance across the three distinct test sets of the Oxford RobotCar dataset, as shown in Table These results suggest that our approach exhibits strong generalization capabilities across diverse environmental conditions, seasonal variations, and roadway scenarios.
4.7 Ablation Study
Table 5 presents the results of our ablation experiments conducted on the KITTI dataset to evaluate the efficacy of the view unification mechanism comprised of Ground Feature Projection and BEV Feature RefineBlock, the loss function composed of Matching Loss and Position Loss, and the Confidence mechanism. We employ Ground Feature Projection and BEV Feature RefineBlock to address significant viewpoint differences between cross-view images and incorporate BEV and satellite image features within the same scope. Our experiments demonstrate that utilizing Ground Feature Projection and BEV Feature RefineBlock significantly improves matching accuracy, reducing average localization error by 57% and 27%, respectively. Furthermore, by incorporating matching points during the training process and employing more robust supervised training approach,

---

Ground image
Sat. image
Ground image
Sat. image
Confidence map BEV feature map Sat. feature map
Confidence map BEV feature map Sat. feature map
(a)
(b)
Figure 5: Matching confidence illustration. The top row of each subfigure shows the ground and satellite images, highlighting corresponding points. Circles indicate confident correspondences (score > 0.5), while stars represent potentially incorrect correspondences (score < 0.5). Matching points share the same color and shape in both images. In the bottom row, the left image shows confidence scores on the BEV feature map (blue for > 0.5, green for 0.5). The middle and right images depict the BEV and satellite feature maps, respectively, with connected lines indicating matching pixels. The circles and stars in these images align with the corresponding points in the top row.
observe 57% decrease in average localization error. Finally, the addition of confidence mechanism further enhances network precision, resulting in 10% reduction in average error. Through conducting ablation experiments, we have demonstrated the efficacy of the view unification, loss, and confidence mechanisms proposed by us.
conducted qualitative analysis of the confidence mechanism the KITTI dataset. We utilized a threshold of 0.5 to distinguish between high-confidence matches and low-confidence matches. In these visualizations, circles represent matching points with high confidence, while stars indicate matching points with low confidence. Additionally, for clarity, we displayed estimated corresponding points on both the original ground and satellite images. From the visualizations, evident that pixels in the bird serre'-eye-view image, which are mutually visible in both ground and satellite images (e.g., road areas), tend to have higher confidence. On the other hand, pixels that visible only in one view (such as building rooftops, tree canopies, and occluded regions) generally exhibit lower confidence. As result, our confidence map effectively selects reliable matching points for accurate pose estimation. This holds significant implications for improving precision.
In Figure 5
5 Conclusion
This paper presents novel approach based dense matching address the problem of cross-view localization. Our approach utilizes view unification module to bridge the visual gap between satellite and ground images. Additionally, we introduce a matching loss provide stronger supervision for the model during training. To evaluate the effectiveness of our algorithm, we conduct experiments on four datasets that vary geographical regions, environmental conditions, and road scenarios. Our results demonstrate the impressive generalization capability of our approach. One possible future direction is explore the integration of spatial and temporal information, further improve the accuracy and robustness of our approach for cross-view localization.
6
Acknowledgement
This work was supported part by the National Natural Science Foundation of China (No. 62302220), in part by the China Postdoctoral Foundation (No. 2023M731691 and in part by the Jiangsu Funding Program for Excellent Postdoctoral Talent (No.2022ZB268).
10

---

References
[1] Siddharth Agarwal, Ankit Vora, Gaurav Pandey, Wayne Williams, Helen Kourous, and James McBride. Ford multi-av scasonal datasct. The International Journal of Robotics Research, 39(12):1367-1376, Sep 2020.
[2] Relja Arandjelovic. Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Can architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297-5307, 2016.
[3] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 (3(11))121112 1237, 2013.
[4] Sixing Hu, Mengdan Feng, Rang MH Nguyen, and Gim Hee Lee. Cvm-net: Cross-view matching network for image-based ground-to-acria gco-localization, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7258-7267, 2018.
[5] Ted de Vries Lentsch, Zimin Xia, Holger Caesar, and Julian FP Kooij. Slicematch: Geometry-guider aggregation for cross-view pose estimation. arXiv preprint arXiv:2211.14633 2022.
[6] Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 128::164116, 1944.
[7] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.22 2017.
[8] Will Maddern, Geoffrey Pascoe, Matthew Gadd, Dan Barnss, Brian Yeomans, and Paul Newman, Real-time kinematic ground truth for the oxford robotear dataset. arXiv preprint 2rXiv:2002.10111 2020.
[9] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36,1)33-5,,001, 2017.
[10] Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In 2010 IEEE computer society conference on computer vision and pattern recognition. pages 33844-391. IEEE, 2010.
[11] Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning cnn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655 1668, 2018.
[12] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz. and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVE conference on computer vision and puttern recognition, pages 4938-4947 2020.
[13] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17010- 17020, 2022.
[14] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li. Spatial-aware feature aggregation for image based tross-view gco-localization. Advances in Neural Information Processing Systems, 32, 2019.
[15] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am i looking at? joint location and orientation estimation by view matching, In Proceedings of the IEEE/CVF CEnf/eCeCConfreencee on Computer Vision and Patiern Recognition, pages 4064-4072, 2020.
[16] Yujiao Shi, Xin Yu, Shan Wang, and Hongdong Li. Cvlnet: Cross-view semantic correspondence learning for rideo-based camera localization. In Computer Vision, Macao, China, December 4 8, 2022, Proceedings. Part I, pages 123 2311 Springer, 2023.
2022: 16th Asian Conference on Computer
[17] Zachary Teed and Jia Deng. Raft: Recurrent all- field transforms for optical flow. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part 16. pages 402-419. 419. Springer, 2020.
[18] Aysim Toker, Qunjie Zhou, Maxim Maximov, and Laura Leal-Taixé. Coming down earth: Satellite- to-street view synthesis for geo-localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6488-6497, 2021.
[19] Nam N Vo and James Hays. Localizing and orienting street views using overhead imagery. In Computer Vision- ECCV 2016: 14th European Conference. Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 999-509. Springer, 2016.

---

[20] Shruti Vaas, Chen Chen, and Mubarak Shah. Gama: Cross-view video geo-localization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 440-456, Springer, 2022.
[21] Shan Wang. Yanhao Zhang. and Hongdong Li. Satellite image based cross-view localization for autonomous vehicle. arXiv preprint arXiv:2207. 135066 2022.
[22] Zimin Xia, Olaf Booij. Marco Manfredi, and Julian FP Kooij. Visual cross-view metric localization with dense uncertainty estimates. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings. Part XXXIX, pages 90 106. Springer, 2022.
[23] Hongji Yang. Xiufan Advancey in Neural Information Processing Systems, 34:29009-29002 29020, 2021.
Yingying Zhu. Cross-view geo-localization with layer-to-layer transformer.
[24] Sijie Zhu, Taojiannan Yang, and Chen Chen. Vigor: Cross-view image geo-localization beyond one-to-on retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3640-3649, 2021.
12

---

Appendix
A Theoretical Proof
We provide a detailed derivation and proof of the least-squares pose regression process in the text. The camera position parameters 0 and t can be obtained by solving the following loss function.
cos 0 -sin 0
く arg min
Σ (.pp +t D.) R 0,8
(8)
sin 0 cos
The relationship between the corresponding points p/ and p, in point sets p' and 5 can be expressed pl Rp; + t. To simplify the problem, we calculate the centroids g and of point sets p' and f.. respectively. We then subtract the centroids from each point in their respective sets, resulting in d': pp g' and q,- = P.- ğ. This simplifies the problem to:
ς arg min
Si q
arg min
SR - -Rq' q,)
(9)
0
arg min
(1 g))) + - - -24," Raq
Therefore, minimizing is equivalent to maximizing Σ. Si(g.,,") ") then
Σi=1 S:(q; RQ())
Assuming that H
Σ Rqy) tr Σ RS -
(10)
Upon performing an SVD decomposition H. the result is H = UAV' where U and V are orthogonal matrices of dimensions 3 x 3, and A is 3 x 3 diagonal matrix with non-negative elements.
Then we introduce an orthogonal matrix X = UV* and observe that XH = VAV' is symmetric positive-defiite matrix. According to the properties of positive definite matrices, it holds that tr ( H tr (H for any 3 x 3 orthogonal matrix B. Therefore, we obtain the maximum value of Zim1 (q, Rqy) when R = X, which implies that
R = UV
(11)
The dense matching relationship between the graphs gives rise to the uniqueness of solution for R. Subsequently, the relative displacement can be determined as
t t= g Rg
(12)
we can obtain the camera location and azimuth through straight- forward differentiable process, using the least-squares regression method.
By utilizing equations 11 and 12
B Datasets
KITTI. The ground images the KITTI dataset were obtained from the original data. 13 collected satellite images of the KITTI dataset, with each ground image accompanied by corresponding satellite image whose center corresponds to the position of the ground camera. The dataset is divided into Training, Testl, and Test2 subsets. The images in Testl are from the same area as the images in the training set, while the images in Test2 are from different areas. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images is approximately 100 x 100 square meters.
Ford Multi AV. For the Ford Multi AV dataset, we only utilized the data of the V2 vehicles from the original dataset. Shi and Li first plotted the trajectories of the ground-view images and collected satellite images along these trajectories. Ground images captured on one date were used for training, while images captured on another date were used for testing. The satellite images remained unchanged during both training and testing. The ground resolution of the satellite images in this dataset is approximately 0.2 meters per pixel, and the coverage area of the satellite images approximately 100 x 100 square meters.
13

---

VIGOR.
The satellite images of the VIGOR dataset were collected by [2]] The ground images are north-aligned panoramas, and they are from four cities: New York, San Francisco, Chicago, and Seattle. There is positive satellite image in the database for each ground image, meaning that the ground image is within the center 1/4 quarter of the satellite image. All the satellite images are north-aligned. The dataset includes two evaluation splits: same-area and cross-area, based on whether the images in training and testing sets are from the same region. Research is conducted to estimate the relative pose between the ground image and its positive satellite image center. In this dataset, the ground resolution of satellite images for New York, San Francisco, Chicago, and Seattle 0, 13, 0.118, 0.111, and 0. 0.101, respectively. The size of the aerial views for these four cities is 640 x 640.
Oxford RobotCar. The satellite images of the Oxford RobotCar dataset were collected by [8]. They provided very large satellite map covering the whole region where the ground images were collected. The ground images were collected through multiple traversals on a single route in Oxford, UK, encompassing various time periods, seasons, and weather conditions. During implementation, for each ground image, a random relative translation is generated, and small satellite patch corresponding to the randomly generated pose is extracted from the large satellite map. Research is conducted to estimate the relative pose. The ground sample distance of the satellite images in this dataset is 0.0924m per pixel, and the coverage of the satellite images used around 55 x 55 square meters.
Comparison with LM 13
To make fair comparison with the [M] trained both the LM and our method for an equal number of epochs and found that our method still outperforms the LM method.
12-
UM
Durs
LM
10.
12-
Averagemm)
Disance Avengglmm
11-
6-
10-
7-
9-
Epoch
Epoch
Figure 6: Performance comparison between LM and our method with longer training epochs on the KITTI dataset.
The LM aims to optimize the relative pose by minimizing the feature differences between corresponding pixels in two-view images. However, the pixel correspondences are computed based on planar homography (and relative camera pose), which is incorrect for objects above the ground plane. Minimizing feature differences between correpprespnn features leads suboptimal relative pose estimation. In contrast, our objective in this paper is to first estimate the precise correspondences between the two views and then compute the relative pose. The estimation of correspondences is independent of the relative pose. Once the correspondences are sufficiently accurate and robust, the relative pose can directly computed by solving least squares problem instead of training additional networks. Therefore, the main capability of our method lies in accurate correspondence learning rather than relative pose estimation. As a result, our method achieves better performance.
14


Reviews:
[{"review_id": "Rc1Bqjr4dH", "reviewer": "Reviewer_pNph", "summary": "The paper introduces a method for estimating 3DOF camera pose of satellite images given a ground-level image and an initial pose estimate as input. The ground-level image is mapped to a satellite view using the camera parameters, confidence-weighted 2D matches are estimated between the warped image and the satellite image, from which the 3DOF parameters are solved. The authors show that this approach works well in the wild across many standard datasets, and outperforms prior work by a large margin.", "questions": "Why do the confidence estimates need their own loss? Can they not be learned automatically just by supervising on the solver output (e.g. DROID-SLAM)? \n\nIn Eq. 1, how is T obtained? Is it provided as input or is there a canonical ground-camera pose assumed (e.g. upright, pointing north, 0 translation)?", "limitations": "No limitations section, so this is not clear.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method follows a procedure that has worked very well for other 3D vision problems (SLAM, SfM): 1) Align features 2) 2D-2D matching 3) Solve for camera pose which explains the 2D observations. This sounds more principled than the prior work.\n\nThe method is interpretable and learns outlier rejection automatically.\n\nThe paper is very well written and easy to understand.\n\nThe authors provide an analytical solution to their solver layer, as opposed to prior work which uses the LM algorithm to align features.\n\nStrong empirical results across many datasets compared to prior work.\n\nPaper includes ablation experiments on several of the components", "weaknesses": "The related work section doesn’t explain the main difference between the proposed method and the referenced methods. For example, what is the main difference between *Wang et al.[42]* and the proposed method? At face-value they sounds similar.\n\nPaper is missing resource requirements (time or memory) compared to prior work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer-pNph for the valuable comments. Please find our response below. \n\n**1:Missing resource requirements.**\n\nOur method is trained and evaluated with an NVIDIA TITAN V GPU with 12GB memory. The average evaluation time for each ground image is 0.25s. Using the same GPU device, the evaluation time of the LM method [29] (CVPR 2022) is 2s, which is around eight times longer than ours. The code and trained models of SliceMatch have not been released yet. Thus, we cannot compare with it now, but we would like to add it when their code is publically available.\n\n\n**2:Difference with Wang et al.[42].**\n\nWang et al. [42] require 3D LiDAR points for localizing a ground image, while our method does not rely on any 3D points. Thus, our method tackles a more challenging scenario. \n\nCompared to other works on ground-to-satellite image localization, this paper is the first to demonstrate the possibility of estimating the optical flow between the ground-and-satellite image pair with extreme viewpoint change for relative pose estimation without the ground truth optical flow map as supervision. \n\n\n\n\n**3:Necessity of the confidence loss.**\n\nThe confidence map selects \"good correspondences\" between the synthesized BEV map and the reference satellite image that contributes an accurate relative pose. Only applying supervision on the solver output is able to provide implicit supervision signals on the confidence map. The results of this ablation variant are shown in the second last row of Tab. 5. However, this supervision might be weak. Using the additional independent supervision proposed by this paper, we can further strengthen the supervision signal and enhance the pose estimation accuracy, as demonstrated by the last row of Tab. 5. \n\n\n\n**4:Meaning of variable $\\mathbf{T}$ in Eq. 1.**\n\nHere, we put the origin of the world coordinate system ($x^W$) on the ground, with the directions of the three axes parallel to the three axes of the camera coordinate system. Thus, there is no rotation difference but a translation difference between the world and camera coordinate systems. The translation difference is along the vertical direction, because the origin of the camera coordinate system is above the ground plane. We use $\\mathbf{T}$ to denote this relative translation and omit the rotation matrix when writing Eq. 1. We will clarify this in the paper. \n\n"}, {"review_id": "utJAi32Uhv", "reviewer": "Reviewer_Bgje", "summary": "The paper presents a method that estimates a 3-DoF pose that aligns correspondences between ground-level and satellite/aerial images. The proposed solution uses a CNN-based model to extract image features, map image features from the ground-level camera to a BEV so that it can be matched with features from the satellite image. Lastly, a least-square regression us run to estimate the final 3-DoF pose (rotation and translation). The paper presents results on the KITTI and Ford multi-AV datasets where the proposed method consistently shows performance improvements over a few baselines.", "questions": "Please see Weaknesses section.", "limitations": "The limitations of the paper are not properly discussed. I think the paper may suffer from images with repetitive features (e.g., an image of a large field of grass, forests, etc.). It is unclear if the proposed method of estimating a confidence can handle this cases correctly. Lastly, since this technology matches aerial views with ground-level views, this can be used for surveillance. I think this an ethics review for this case.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In sum, I think the proposed solution is simple and straightforward. Thus, I think the proposed approach is easily reproducible.\nThe particular strengths I see are the following:\n\nS1. Simplicity and straightforward of solution. The solution essentially mimics a classical dense matching of features coming from two different images (i.e., aerial/satellite view and ground-level view). Because of this simplicity, I think the proposed solution can be easily adopted and reproduced.\n\nS2. End-to-End approach. I do like that the proposed solution includes the feature matching pipeline but also optimizing for the 3-DoF pose which ultimately is the final solution. This is great as it directly learns features that are useful to obtain the desired pose.\n\nS3. Use of a confidence value as part of the solution is also very useful. I think this should be able to handle outliers and allow a robust computation of the estimated pose.", "weaknesses": "While I value the simplicity of the solution, I question its novelty. To me, the novelty is very important as this is the way we break common perspectives and open new avenues of research. In detail here are my concerns about novelty:\n\nW1. The proposed solution uses CNN features in the era of Transformers. While this is a feasible solution (as shown by the experiments), it has been shown in the computer vision community that attention mechanisms can help in feature matching and also deal with outliers; a good example of this is LoFTR presented at CVPR 2021. Unfortunately, I don't see any justification in the paper about the use of CNNs when Transformers inherently provides mechanisms to do feature matching.\n\nW2. The proposed solution mainly is composed of well known components, and the proposed method is quite straightforward as it follows classical pipelines of feature matching (e.g., PatchMatch-Based Neighborhood Consensus for Semantic Correspondence CVPR 2021) combined with well known optimization methods to estimate a pose.\n\nW3. The paper uses terms in a wrong way. For example, in Eq. 3 the $\\mathbf{t}$ is introduced as position in line 119. However, the Eq. 3 shows this is a translation and not a position. This is very confusing as it conflicts the math and the stated definition, which decreases clarity. Second, the $G(\\cdot)$ introduced in line 141 is not defined. What is it? Lastly, in Eq. 3, the loss seems to be using L2 norm and not using a squared error on scalars as suggested by the parenthesis in the equation. Thus, the \"Position Loss\" described in 199 is not optimizing directly the position but rather the pose since it penalizes deviations of the rotation angles and translations.\n\n--------\nPost Rebuttal.\n\nI am satisfied with the answers the authors have provided. Although I think the proposed is simple and straightforward, I think it produces good results according to the experiments. I think the community can get intrigued by the results and the simplicity of the methods, and thus I expect this paper to cause some interest in the CV community. For these reasons, I will upgrade my rating to borderline accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank Reviewer-Bgje for the valuable comments. Please find our response below. \n\n**1:Reasons for using CNN.**\n\nThere are currently two lines of work for estimating correspondences between two images (or two point clouds, etc.) The first line of work first constructs a noisy cost volume between the two images using the features extracted by a simple network (usually CNNs), then refine the noisy cost volume to a clean one by an additional network (3D CNN, RNN, etc.). \nThe other line of work aims to learn strong and discriminative representations by self and cross attentions (transformers) for each pixel in the image and then compute the cost volume (or similarity matrix), from which the correspondence is directly computed by an argmin (or argmax) operation.\nOur method of using RAFT aligns with the first category, and the LoFTR belongs to the second category. \n\nDetermining which method is generally better is hard, especially when ground truth correspondence supervision is available. While in our task where such ground truth correspondence supervision is not available, we empirically find that a simpler network architecture is easier to learn. \n\nSpecifically, we aim to estimate the correspondences between the ground and satellite images to estimate their relative pose. However, no ground truth correspondence is available for these two view images to supervise a network. Thus, we propose to first convert the ground view image to a Bird's-eye view (BEV) and then estimate the correspondences between the BEV image and the reference satellite image. In this case, the GT correspondences between the BEV and the satellite image can be computed from their GT relative pose. However, due to the severe occlusions between the ground and overhead view images, it is impossible to synthesize a perfect BEV image corresponding to the ground image. Thus, the computed GT correspondences are not really applicable to every pixel in the BEV image. A learnable confidence map might solve this issue, by filtering out unreliable correspondences. However, the confidence map does not have GT supervision either. \nWe further propose a new mechanism to supervise this confidence map. One is from the estimated flow, indicated in Eq. 5. The other is applying the discrepancy between the computed pose from the estimated flow map and the GT pose to supervise the confidence map and estimated optical flow map implicitly. \n\n\nIn summary, we have explored various signals to supervise the optical flow (correspondence) estimation network, but all these signals are implicit and not ground truth. Given such weak supervision, it is hard to train a complex network built by transformers, and we indeed observe the non-convergence phenomenon when using LoFTR for correspondence estimation in our task.  \n\n\nFor comparison in terms of time complexity and memory consumption, employing the LoFTR pipeline for inference requires approximately 400ms and consumes 10GB of storage, whereas utilizing GRU for inference only necessitates 200ms and occupies 6GB of storage. The experimental results of this comparison are presented in Tab.A of the rebuttal PDF. \n\n\n**2:Novelty.** \n\nWe acknowledge that first conducting feature matching to establish correspondences and then computing the relative pose from the correspondences is a common structure, and our method aligns with this pipeline. However, we want to highlight that our contribution is not a simple adaption of this common pipeline to a new task, but how to address the unique challenges in the new task faced by this existing technique.\n\nSpecifically, the GT correspondences between the ground and satellite image in our task, neither pixel-wise nor keypoint-wise, is unknown, given the absence of the 3D data. Furthermore, due to the extreme viewpoint change between the ground and satellite images and their different capture time, it is hard to directly use a network pre-trained on existing data with GT correspondences (usually both ground views and captured at a close time) to solve our task. Thus, our main contribution is how to estimate correspondences between the extreme stereo pair without GT supervision and then use the estimated correspondences for relative pose estimation.\n\nThis paper discovers that the supervision is computable when converting the ground view to the overhead view by a BEV-synthesis, and introduces additional implicit supervisions either to strengthen the supervision (Eq. 6) or to filter out unreliable correspondences (Eq. 5.). Please refer to our response to the first question for more details. The experiments in Tab. 5 also demonstrate that naively applying existing techniques for solving the task achieves undesirable performance. Only when incorporating our proposed solutions for addressing the unique challenges can the method achieve the best performance. Our improvement over the state-of-the-art is also significant.\n\nWe believe this paper should also benefit some similar tasks. For example, the self-supervision in Eq. 5 for the confidence map also applies to other self-supervised optical flow estimation tasks.\n\n**3:Error in variable $\\boldsymbol{t}$ in Eq. 3.** \n\nThe variable $\\mathbf{t}$ in L119 denotes the relative position of the ground camera with respect to the reference satellite image center. Thus it is also the relative translation between the ground camera coordinate system to the satellite camera coordinate system in Eq. 3. \n\n\n**4:Meaning of variable $G(.)$ in line 141.** \n\n$G$ represents the process of projecting point $\\boldsymbol{p}$ onto the Bird's Eye View (BEV) to obtain point $\\boldsymbol{p}'$. We will make this clearer in the paper. \n\n**5:Ambiguity in the position loss at line 199.** \n\nThank you for pointing out this issue. The \"Position Loss\" is actually the \"Pose Loss\" as the reviewer mentioned. We will correct this issue in the revision if accepted.\n\n"}, {"review_id": "q7leGjdsiv", "reviewer": "Reviewer_1HCJ", "summary": "In this paper, the authors try to address the cross-view geo-localization by learning dense pixel-wise flow fields between ground and satellite images. To achieve this, they utilize a ground feature projection module to align the spatial feature and propose a refinement block to enhance the context feature for accurate optical flow estimation. However, some details about refinement modules, data collection, and experiment setup are unclear.", "questions": "See the Weaknesses.", "limitations": "The limitations and potential negative societal impact are not explicitly discussed.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper tries to address the cross-view geo-localization by learning dense pixel-wise flow estimation. And the authors conduct an account of experiments on KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar.", "weaknesses": "However, there are some concerns about this paper:\n1. More detail about the satellite image collection. In Sec 4.1, the authors only present the detail about the ground-view dataset. Is there any detail about the satellite image, including collection, preprocessing, ground sample distance, data distribution, and so on?\n2. As stated in Line 146, RefineBlock is a crucial module of the proposed method. However, the authors provide no detail about RefineBlock. Please provide the detail about RefineBlock. And more analysis and experiments are appreciated to demonstrate the importance of RefineBlock.\n3. LM [1] are trained with 2 epochs. However, this method is needed to train with 25 epochs in Line 237, which is 10x longer than LM [1]. Are the results of LM [1] in Table 1 and Table 2 trained with 25 epochs? Could you provide the results of LM [1] trained with 25 epochs?\n4. The training schedule is unclear. In Line 235, the feature extractors are optimized with 12 epochs in the Dense Flow Estimation stage. However, in Line 237, the pipeline is conducted with 25 epochs in an end-to-end schedule. And in Line 207, the momentum parameter is changed to 10 after 15 epochs. Hence, I am confused about the training schedule. Does it mean the whole pipeline consists of two stages and is trained with 37 epochs in total?\n5. The most similar state-of-the-art work is proposed by Shi and Li [1]. In Table 4 of LM [1], the results show that LM achieves better translation performance but inferior rotation performance than the network-based optimizer, including GRU. Hence, the relationship and main difference with the prior work [1] are appreciated to be discussed in the related work. It is better to analyze the advantages and disadvantages of the proposed approach compared with LM [1]. And more experiments and analyses are appreciated.\n\nReference\n\n[1] Yujiao Shi and Hongdong Li. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image. CVPR 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank Reviewer-1HCJ for the valuable comments. Please find our response below. \n\n**1: Details of satellite images in the datasets, including collection, preprocessing, ground sample distance, data distribution, and so on.**\n\nPlease refer to our response to all the reviewers in the overall response. \n\n\n**2:Explanation of the RefineBlock.**\n\nThe RefineBlock is proposed to address the limitations of the ground-plane homography-based ground-to-overhead projection, including incorrect correspondences for objects above the ground plane, distortions in the projection, etc. In order to eliminate these issues, the network should have a larger receptive field of the input overhead-view feature map projected based on the ground plane homography to achieve a better and high-level understanding of the feature map. Thus, a 7x7 convolutional layer is first employed in the RefineBlock. Then, a 3×3 residual block is applied to refine the details in local regions. Finally, a 1×1 convolutional layer is adopted to facilitate feature interaction and integration, introducing enhanced non-linear transformation capabilities in the feature space. Our source code containing all implementation details will be released. \n\n\n**3：The results of longer training epochs of LM [29].**\n\nThank you for the suggestion. We indeed trained the LM method for longer epochs when preparing this submission, but we did not observe too much performance increase with longer training epochs. Please refer to Fig. A in the rebuttal PDF for the results. Thus, we choose to use the released performance in the original paper for comparison. We would like to modify the numbers to our reproduced results with a longer training epoch (25) for fair comparison. \n\nWe should also note that our method has a significantly faster training and evaluation speed than the LM method. For example, on the same NVIDIA TITAN V GPU with 12GB memory, our method takes 2 hours for training one epoch, while LM needs 6 hours; our method achieves a 0.25s evaluation time on average for each ground image, while LM needs 2s for one ground image's evaluation. \n\n\n\n**4:Unclear training schedule.** \n\nThanks for pointing out this confusing illustration. In line 235, the optimization of the Dense Flow Estimation stage refers to iterating the GRU structure 12 times, independent of the training process. The entire training schedule involves 25 epochs. At the 15th (out of 25) epoch, we change the value of $\\beta$ to 10. \n\n\n**5:Differences from LM.**\n\nThe differences between LM and our method include different network architectures, different projection directions for bridging the domain gap (satellite-to-ground Vs. ground-to-satellite), different optimizers (LM Vs. GRU), different optimization parameters (3-DoF pose Vs. dense flow map) and different training objective functions. \n\n\nFrom a high-level perspective, both the LM paper and our method estimate the relative pose from the two-view correspondences. \nThe LM paper aims to optimize the relative pose by minimizing the feature differences of corresponding pixels in the two-view images.\nHowever, the correspondences of pixels are computed based on ground plane homography (and the relative camera pose), which is incorrect for objects above the ground plane. \nMinimizing the feature differences between non-corresponding features would lead to sub-optimal relative poses. \n\nThus, in this paper, we aim to estimate accurate correspondences between the two views first and compute the relative pose second. \nThe correspondences are estimated in a way that is agnostic to relative poses. \nOnce the correspondences are accurate and robust enough, the relative pose can be directly computed by solving a least squares problem instead of training an additional network. \nThus, the main capacity of our method is spent on accurate correspondence learning instead of the relative pose, while the LM paper spent its main effort on pose estimation by a fixed correspondence relationship according to the pose. \n\n\nTo achieve this goal, our paper addresses a set of unique challenges encountered, such as how to train the correspondence estimation network without accurate ground truth supervision, how to select the accurate and robust correspondence pairs from the dense flow map, etc. As a result, our method achieves significantly better performance and a faster evaluation speed than the LM method. \n\n"}, {"review_id": "nsNtJOtsE1", "reviewer": "Reviewer_63Cb", "summary": "This paper deals with  the 3-DoF camera pose estimation method for a ground-level image with respect to a satellite image. The proposed end-to-end approach that leverages the learning of dense pixel-wise flow fields between the projected ground-level image and the satellite image is built upon RAFT architecture. \nEffectiveness of the proposed method is evaluated on competitive benchmarks and outperforms other SoTA methods in both location and azimuth measures.", "questions": "- Are the colors in Figure 1 meant to represent the same feature by using the same color? Also, the road appears red in the original image (Figure 1(B)), but it appears green in the BEV (Figure 1(E)). Additionally, why isn't the vertical road in the satellite image shown in red?\n- Is Equation 5 correct? When rearranged, it becomes $\\frac{1}{1+exp(-\\tilde d_i/ \\kappa)}$, and $S_i$ disappears.\n- Why is the error of Localization/Azimuth for \"Ours\" in Table 1 (4.97/11.2) different from the values shown in Figure 3 (3.8/8.3)?\n-  How is the ground truth for the dense estimation flow in Equation 4 actually obtained? It is mentioned that it is determined by the camera pose in L.188, but in reality, the 3D position (depth) will be required.", "limitations": "When it comes to practical use, I believe it is important to verify the robustness and accuracy of the initial value between ground-level images and satellite images.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "Although the method is simple and the DNN structure is also easily understandable,  the proposed approach has achieved state-of-the-art (SoTA) performance by an adequate margin on various standard benchmarks.\nThe paper is written clearly and adequate figures support to understand the proposed approach.", "weaknesses": "The main difference compared to [29], as mentioned in L. 75-77, is the part where camera poses are estimated from the explicit matching. However, the idea itself is well-known, for example, in [*], so I think the novelty of the approach is limited. While the representation of the loss function may differ, the basic idea has been previously addressed.\n\n[*]A.R. Zamir et al. \"Generic 3D Representation via Pose Estimation and Matching\", ECCV16.\n\nWhen dense correspondences are estimated, it is common to use RANSAC-based methods. It would be beneficial to compare the proposed method with RANSAC-based approaches to evaluate their performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer-63Cb for the valuable comments. Please find our response below. \n\n**1:Differences compared to Shi and Li[29] and Zamir, Amir R., et al. [\\*] are minor.**\n\nFrom a high-level perspective, Shi and Li [29], Amir R., et al. [\\*], and our method all estimate the relative pose from the two-view correspondences. However, our contribution is not a framework that estimates relative camera poses from correspondences. Rather, our contribution is how to estimate correspondences from an extreme stereo pair (ground and satellite images) without ground truth supervision (no 3D data).\n\nShi and Li [29] assume a fixed correspondence between the ground and satellite view images based on the ground plane homography. However, correspondences computed this way are not accurate for objects above the ground plane. (Please refer to our response to the fifth question of Reviewer-1HCJ for more details.) Amir R., et al. [\\*] have ground-truth correspondences between input images for their network supervision, while this is not available in our task.\n\n\nThis paper addresses a set of unique challenges in the ground-to-satellite localization task, such as how to train the correspondence estimation network without accurate ground truth supervision, how to select the accurate and robust correspondence pairs from the dense flow map, etc. As a result, our method achieves significantly better performance than the most recent state-of-the-art. \n\n\n\n**2:Comparison with RANSAC.**\n\nTab.B in the rebuttal PDF presents the comparison between RANSAC and our method. The maximum iteration of the RANSAC is set to 50, and the inlier ratio threshold is 0.5. These parameters are carefully engineered to achieve the best performance. The results in Tab.B show that the performance between RANSAC and our method is similar. Regarding computational speed, the RANSAC takes around 0.3s on average for each image, while our method needs 0.25s.\n\n**3:Visualized colors.**\n\nWe used PCA to decrease the high-dimensional feature channels to three channels for RGB visualization. We did not use a fixed PCA for all feature maps. Thus, the same color across different feature map visualizations might be different, and the same object might be visualized with different colors. We will make it consistent in our revision. The vertical road in the satellite map does exist in the feature map, but not obvious. This is because the PCA method only highlights the main features using which the correspondence is correctly estimated. The vertical road is not salient on the ground image; thus, its corresponding features are not highlighted as much as the horizontal road. \n\n\n**4:Mistake in Eq. 5.**\n\nThank you for pointing out this problem. The correct version should be:\n\\begin{equation}\n  \\mathcal{L}_c = \\sum_{i=1}^n \\Vert \\frac{S_i}{1+exp(-\\tilde{d_i}/\\kappa)} + \\frac{1 - S_i}{1+exp(\\tilde{d_i}/\\kappa)} \\Vert_1 .\n\\end{equation}\nIt aims to penalize high confidence for poor matching performance and low confidence for excellent matching performance. The original simple $C_i$ should be changed to $S_i$ to make it consistent with Eq. 3. \n\n\n**5:Inconsistent number between Fig.3 and Tab.3.**\n\nThe error of Location/Azimuth for \"Ours\" in Tab.3 (4.97/11.2) corresponds to the last blue bars (5.0/11.2) in the two subfigures in Fig. 3. The numbers are the performance of our method when trained and evaluated under completely unknown orientation ($\\pm 180^\\circ$) scenario on the VIGOR dataset.  \n\nThe numbers (3.8/8.3) in Fig. 3 show the performance of our method when *evaluated* on known orientation ($0^\\circ$) but *trained* on completely unknown orientation ($\\pm 180^\\circ$). This is different from the numbers in fourth row of Tab. 3, because the fourth row of Tab. 3 presents the performance of our method when trained and evaluated on the known orientation ($0^\\circ$) scenario. \n\nWe will make this clearer in the paper. \n\n\n**6:How to obtain ground truth for dense estimation flow.**\n\nThe ground truth optical flow between the ground and satellite image is hard to be derived, given the absence of 3D data. Thus, we propose to convert the ground image to the Bird's-eye view (BEV) and estimate the correspondences between the BEV image (/feature map) and the satellite image (/feature map). In this case, there is a inplane rotation and translation difference between the BEV image and the satellite image. Thus, their pixel correspondences can be computed by a homography derived from this relative rotation and translation. This relative rotation and translation is also the ground truth relative pose between the ground image and the satellite image.  \n\n\n**7:Robustness to different initial values.**\n\nThe location and orientation search range used in the experiments in the paper follow the official settings in their original papers [6, 14, 23]. \n\nThe coarse locations of ground cameras on the VIGOR dataset are in the center quarter of the satellite image. The search range is approximately $36\\times 36$ m$^2$. The location search range for the KITTI, Ford multi-AV, and Oxford RobotCar datasets is about $40 \\times 40$ m$^2$. The error of the coarse pose in the experiment settings is similar to a consumer-level noisy GPS. \n\nWe further present the performance of our method across a broader location search range in Tab. C of the rebuttal PDF, specifically, $60 \\times 60$ m$^2$. As the location search range increases, so does the location ambiguity, leading to a slight decline in our method's performance. However, we should note that even with the larger location search range of $60 \\times 60$ m$^2$, our approach outperforms the recent state-of-the-art SliceMatch [6] when it is tested within a smaller location search range of $40 \\times 40$ m$^2$.\n\n\nThe experiments on varying orientation ambiguity are presented in Fig. 3 of the paper. \n\n"}, {"review_id": "bWSR80EAWg", "reviewer": "Reviewer_NkYA", "summary": "The paper addresses the problem of cross-view camera localization, i.e. of estimating a 3D camera pose for a ground-level image with respect to a satellite image of its local surroundings (assuming a coarse localization is already given). For this, it proposes a deep learning pipeline that aims at estimating a dense flow field between the jointly computed feature maps of the satellite image and a bird's-eye-view (BEV) rendering of the ground-level feature maps. Based on an initial correlation volume between those two feature maps, it then employs several GRU stages (similar to RAFT) in order to estimate the dense flow field and finally performs least-squares pose regression. The approach is quantitatively evaluated on the KITTI, Ford multi-AV, VIGOR and Oxford RobotCar datasets. Compared to previous approaches, the proposed method achieves a considerably improved localization accuracy.\n", "questions": "This paper leaves a mixed impression. On the one hand, the proposed approach is well-engineered and achieves very good results. On the other hand, I am also not terribly excited about the paper, since it addresses a rather niche topic without a strong learning-focused contribution. I would be more excited about it in a more application focused conference, but I do not think that it has something great to offer particularly for the NeurIPS audience.", "limitations": ".", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed approach is well-engineered and achieves considerably improved localization accuracies on all four datasets, compared with the previous SOTA methods. \n\n+ Experimental evaluations of 4 datasets and comparisons against several published baselines support the claimed contributions.\n\n+ The effects of the different elements of the proposed approach are investigated in an ablation study.\n", "weaknesses": "- The paper makes a relatively minor technical contribution. It presents a novel network architecture for a very specialized problem setting. The architecture is well suited for the estimation problem at hand and achieves very good localization accuracy, but the impact will still be limited, since the problem setting is interesting only for a relatively small subcommunity. Specifically, there is no strong machine learning content that would make this a particularly good fit for NeurIPS. \n\n- As the proposed approach relies on a coarse localization being already given, its success in practice will heavily rely on how robust the estimation is with respect to a coarse localization error. In other words, how close will the location of the satellite image already have to be to the camera location and over what search range will the approach reliably find a correct localization. Unfortunately, this is not discussed at all, nor is it reflected in the experimental design. The quantitative evaluation follows \"the settings of [14, 29, 43]\", which presumably means that the ground truth location lies within the center quarter of the satellite image tile?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer-NkYA for the valuable comments. Please find our response below. \n\n\n**1:Minor technical contribution, limited impact.**\n\nInstead of a trivial ensemble of existing techniques, our contribution is mainly on how to solve unique challenges in a new scenario faced by existing techniques. \n\nIt is well-known that RAFT is a powerful network to estimate optical flow between two images with GT supervision, but it is unknown how to supervise RAFT when such GT supervision is unavailable, and the application scenario (ground-view Vs. overhead-view) significantly differs from the publicly available training dataset (ground-view Vs. ground-view). Furthermore, the extreme stereo case (ground Vs. overhead) makes the image reconstruction loss for self-supervision of the optical flow impossible due to severe occlusions and different capture times (different years, illumination, color, etc.) \n\n\nThis paper discovers that the supervision is computable when converting the ground view to the overhead view by a BEV-synthesis, and introduces additional implicit supervisions either to strengthen the supervision (i.e., pose supervision in Eq. 6) or to filter out unreliable correspondences (i.e., confidence map with implicit supervision from pose error in Eq. 6 and flow error in Eq. 5.). Please refer to our response to the first question of Reviewer-Bgje for more details. The experiments in Tab. 5 also demonstrate that naively applying existing techniques for solving the task achieves undesirable performance. Only when incorporating our proposed solutions for addressing the unique challenges can the method achieve the best performance. Our improvement over the state-of-the-art is also significant. \n\nWe believe this paper should also benefit some similar tasks. For example, the self-supervision in Eq. 5 for the confidence map is also applicable to other self-supervised optical flow estimation tasks.  \n\n\n\n**2: Error range of the coarse locations.**\n\nThe coarse locations of ground cameras on the VIGOR dataset are in the center quarter of the satellite image. The search range is approximately $36\\times 36$ m$^2$. The location search range for the KITTI, Ford multi-AV, and Oxford RobotCar datasets is about $40 \\times 40$ m$^2$. The error of the coarse pose in the experiment settings is similar to a consumer-level noisy GPS. \n\nWe present the performance of our method across a broader location search range in Tab. C of the rebuttal PDF, specifically, $60 \\times 60$ m$^2$. As the location search range increases, so does the location ambiguity, leading to a slight decline in our method's performance. However, we should note that even with the larger location search range of $60 \\times 60$ m$^2$, our approach outperforms the recent state-of-the-art SliceMatch [6] when it is tested within a smaller location search range of $40 \\times 40$ m$^2$.\n\n"}]

Author rebuttal:
We appreciate all reviewers' valuable feedback and constructive suggestions. We are glad that two ethics reviewers confirmed that the datasets used in this paper are all licensed for academic use and not original to this work. We are committed to adhering to the guidelines set forth by the original data publishers. Our approach will remain adaptable and responsive to any modifications in these guidelines.

Both ethics reviewers asked for details about the datasets. Reviewer-1HCJ also inquired about the details of satellite images in the datasets, including the collection, preprocessing, ground sample distance, data distribution, etc. Our response is as follows. Point-to-point responses for other questions are provided for each reviewer, respectively. 

**1:More detailed information about the datasets.**

(1) The satellite images of the **KITTI** and **Ford Multi-AV** datasets were collected by Shi and Li [29]. 
The ground images are from the raw data for the cross-view KITTI dataset. Each ground image is accompanied by a satellite image whose center corresponds to the location of the ground camera. This dataset is divided into Training, Test1, and Test2. Images in Test1 come from the same region as those in the training set, while images from Test2 are from other regions. For the Ford Multi-AV dataset, we exclusively utilize data from V2 vehicles. The authors first drew a trajectory of the ground view images and then collected satellite images along the trajectory. Ground images captured on one date are used for training, and those captured on another date are used for testing. Satellite images are kept the same during training and testing. For implementation in both datasets, the corresponding satellite image for each ground image is randomly rotated and translated within an error range to mimic the misalignment between the two view images, and research is conducted to estimate the relative pose between the two images. The ground resolution of the satellite images in these two datasets is around 0.2m per pixel, and the coverage of the satellite images used is around 100m x 100m. 

(2) The satellite images of the **VIGOR** dataset were collected by Zhu et al. [14]. The ground images are north-aligned panoramas, and they are from four cities:  New York, San Francisco, Chicago, and Seattle. There is a positive satellite image in the database for each ground image, meaning that the ground image is within the center 1/4 quarter of the satellite image. All the satellite images are north-aligned. The dataset includes two evaluation splits: same-area and cross-area, based on whether the images in training and testing sets are from the same region. Research is conducted to estimate the relative pose between the ground image and its positive satellite image center. In this dataset, the ground resolution of satellite images for New York, San Francisco, Chicago, and Seattle is 0.113, 0.118, 0.111, and 0.101, respectively. The size of the aerial views for these four cities is 640×640.

(3) The satellite images of the **Oxford RobotCar** dataset were collected by Xia et al. [23]. They provided a very large satellite map covering the whole region where the ground images were collected. The ground images were collected through multiple traversals on a single route in Oxford, UK, encompassing various time periods, seasons, and weather conditions. During implementation, for each ground image, a random relative translation is generated, and a small satellite patch corresponding to the randomly generated pose is extracted from the large satellite map. Research is conducted to estimate the relative pose. The ground sample distance of the satellite images in this dataset is 0.0924m per pixel, and the coverage of the satellite images used is around 55m x 55m. 

**2:Is the dataset being used in its entirety?**

The KITTI, VIGOR, Ford multi-AV, and Oxford RobotCar datasets we used are approximately 409GB, 183GB, 361GB, and 499GB, respectively. After downloading these datasets, we store them on a NAS hard drive.

When using the KITTI [9] and Ford multi-AV datasets [2,29], we follow the setup described by Shi and Li [29]. For the KITTI dataset, daytime images captured from different trajectories at various times are used. Regarding the Ford multi-AV dataset, images captured by the front left camera of the V2 vehicle are used. For the VIGOR dataset, we follow SliceMatch [6] in using the ground and its positive satellite images. The semi-positive satellite images are not used. 

Xia et al. [43] selected a portion of the whole Oxford RobotCar dataset and supplied it with a large satellite map. Specifically, they selected several traversals recorded on different days and at various times of the day to capture different weather conditions. The weather conditions were labeled as "sunny," "cloudy," or "overcast." Additionally, the traversals were subsampled to ensure a minimum distance of 5 meters between consecutive frames. In the end, the dataset consisted of 17,067 images for the training set, 1,698 images for the validation set, and 5,089 images for the test set, all of which are ground-level frontal views. Our usage of this dataset follows this setup. 


**3:Computational cost.**

We utilize an NVIDIA TITAN V GPU with 12GB memory for training and testing. Our method takes two hours for training one epoch, 0.25s on average for estimating one ground image's pose during evaluation. During the evaluation, our method consumes 6GB GPU memory for one ground image.

**4:Can the code be shared?**

The source code is provided in the supplementary material. We will make it publically available to facilitate reproducible research. 

Meta-review: Most of the reviewers have given it a positive rating, with one reviewer championing for its acceptance, but not all the positive reviewers were enthusiastic with the draft. 

Meta-reviewer do agree with 1HCJ that authors should clarify and detail all the points regarding architecture, training, experiments etc...  At the same time  Meta-reviewer agree with reviewer pNph that "fact that the overall approach is simple but works very well is a virtue, not a detriment". This however, should be the duty of authors to make such case by improving their presentation and writing of the paper. After a discussion with SAC it was decided that it should be accepted to NeurIPS. We hope that authors will update the paper to include information about RefineBlock, clarify difference between their work and existing state-of-the-art  (especially Wang et al. [42] and [29]) in the final submission, and add other clarifications requested by the reviewers.  
 Overall draft itself need careful proofreading to improve clarity.
Decision: Accept (poster)