SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenbo Song $^{1*}$ , Xianghui Ze $^{1*}$ , Jianfeng Lu $^{1}$ , Yujiao Shi $^{2\dagger}$ $^{1}$ Nanjing University of Science and Technology,  $^{2}$ ShanghaiTech University  
{songzb, zexh, lujf}@njust.edu.cn, shiyj2@shanghaitech.edu.cn
This paper addresses the problem of estimating the 3-DoF camera pose for a ground-level image with respect to a satellite image that encompasses the local surroundings. We propose a novel end-to-end approach that leverages the learning of dense pixel-wise flow fields in pairs of ground and satellite images to calculate the camera pose. Our approach differs from existing methods by constructing the feature metric at the pixel level, enabling full-image supervision for learning distinctive geometric configurations and visual appearances across views. Specifically, our method employs two distinct convolution networks for ground and satellite feature extraction. Then, we project the ground feature map to the bird's eye view (BEV) using a fixed camera height assumption to achieve preliminary geometric alignment. To further establish the content association between the BEV and satellite features, we introduce a residual convolution block to refine the projected BEV feature. Optical flow estimation is performed on the refined BEV feature map and the satellite feature map using flow decoder networks based on RAFT. After obtaining dense flow correspondences, we apply the least square method to filter matching inliers and regress the ground camera pose. Extensive experiments demonstrate significant improvements compared to state-of-the-art methods. Notably, our approach reduces the median localization error by  $89\%$ ,  $19\%$ ,  $80\%$ , and  $35\%$  on the KITTI, Ford multi-AV, VIGOR, and Oxford RobotCar datasets, respectively.
Cross-view camera localization, also known as ground-to-aerial/satellite image matching, has evolved from the study of coarse localization through image retrieval[4, 12, 16, 20] to a highly accurate 3-DoF pose estimation task[5, 13, 22, 24]. This task involves accurately determining the precise position and orientation of a ground camera in relation to a satellite patch that encompasses the surrounding environment.
The key challenge in cross-view camera localization is to bridge the gap between the visual observations captured by the ground camera and the corresponding top-view camera, enabling robust and accurate regression of the camera's pose in real-world spatial coordinates. This capability holds immense significance across various applications, including augmented reality, autonomous navigation, virtual reality, and robotics, where precise camera localization is crucial for enabling advanced functionalities and interactions with the environment.
This paper focuses on investigating the problem of single-frame ground image localization, which aims to estimate the 3 DoF pose of a ground camera relative to a single satellite image. This problem
(A) Ground image
(B) Ground feature
(C) Satellite image
(D) Satellite feature
BEV rejection
(E) Dense matching
Figure 1: Visualizations of the ground and aerial feature maps, and the dense matching correspondences cross views for localization. For visual simplicity, dense matching displays only partial inliers.
(F) Pose estimation
assumes that the satellite image covers the same geographic region as the ground image. The objective of localization is to determine which pixel of the satellite image corresponds to the ground camera's position and its current shooting direction[24]. In practical applications, as each pixel of the satellite image is associated with GPS labels, it becomes feasible to obtain the geographical location information and spatial orientation information of the ground camera.
2-DoF translation, and 1-DoF rotation. In order to learn robust matching results, we use the dense optical flow as well as the ground truth pose as supervision to train the network progressively.
- This paper introduces a novel approach that learns dense pixel-wise flow fields between ground and satellite images. This enables accurate cross-view camera localization by directly calculating precise ground camera poses from the learned flow.
- We propose spatial alignment and content refinement modules for ground image feature enhancement. By leveraging the BEV representation of ground feature maps, the proposed method achieves accurate optical flow estimation according to both spatial and semantical alignments.
- We conduct extensive experiments on various datasets, including KITTI, Ford multi-AV VIGOR, and Oxford RobotCar, which demonstrates that our method achieves superior performance and exhibits strong generalization ability across all datasets.
Localization based on retrieval. Image retrieval methods are widely utilized for same-view localization tasks[2, 10-12]. This approach assumes the availability of a reference dataset containing a vast number of geotagged ground images. Query images are then matched with ground images in the reference dataset to determine their corresponding geographic locations. However, this method requires a significant number of ground markers, which can be expensive. As satellite maps become more widely available, cross-view localization has gained attention as a potential alternative. In cross-view retrieval tasks, a common approach is to create separate global descriptors for the ground image and the satellite image, and then compare their similarity[4, 14, 18, 19, 23]. Alternatively, recent studies have explored fusing consecutive multi-frame ground images to generate a descriptor, which is then matched with the descriptor of the satellite image[16, 20]. In this cross-view retrieval methods, the camera is assumed to be located at the center of the satellite map. However, this assumption limits the generalization of these methods.
Cross-view camera pose estimation. This work differs from the retrieval task in that it relies on an initial estimate of a given camera pose and then proceeds to refine it to obtain a pose that is close to the true value. Previous works have used various approaches to tackle this problem. For example, VIGOR[24] first uses an end-to-end framework to roughly locate the query through retrieval and then refines the location by predicting offsets through regression. Xia et al.[22] formulates the localization problem as a multi-class classification problem and uses a dense multimodal space for localization. [13] uses a geometric projection module to align the cross-view features, and then optimize the camera pose end-to-end using the Levenberg-Marquardt algorithm[6]. Wang et al.[21] uses 3D points to bridge the geometric gap between ground and overhead views, iteratively fusing ground images with 3D points iteratively for pose estimation. SliceMatch[5] is given a set of candidate photo poses and then compares a single ground descriptor with a set of pose-dependent satellite map descriptors to generate the final camera pose. It is worth noting that previous works have primarily relied on global descriptors [22, 24] or local descriptors[5, 6, 13, 21] for cross-view positional estimation. However, in our work, we propose an alternative approach that relies on dense point matching, which we believe to be a reliable basis for cross-view localization.
As shown in Figure 2, the proposed method is built upon the RAFT architecture. The main differences are the heterogeneous ground and aerial feature extractors and feature transformation modules, i.e. the ground feature projection module, and the BEV feature refinement module. In addition, we propose a differentiable least squares pose regression module to filter matching inliers and calculate the camera pose through dense flow fields. Mathematically, given a ground image  $\mathbf{I}_g$  and a corresponding satellite image  $\mathbf{I}_s$ , the task is to determine the position  $t = (\Delta u, \Delta v)$  and orientation  $\theta$  of the camera with respect to the satellite image. To achieve this, we propose a robust method that leverages the similarity between the points in the two images to estimate the corresponding satellite image  $\hat{\pmb{p}}$  coordinates of each point  $\pmb{p}$  in  $\mathbf{I}_g$ . By finding the position relation between points, we can derive a geometric relation
Figure 2: Illustration of the pipeline of our proposed method.
function  $M$  that maps points between the two images such that  $\hat{\pmb{p}} = M(\pmb{p})$ . Next, we introduce the three key modules of our approach: (i) Feature extraction and view unification, (ii) Dense flow estimation, (iii) Least squares pose regression.
3.1 Feature Extraction and View Unification
In the presence of a large field-of-view error, it is a great challenge to match points directly to  $\mathbf{I}_g$  and  $\mathbf{I}_s$ . Our work addresses the significant differences in field-of-view between ground and satellite images by introducing a feature extraction module that incorporates projection transformation information.
The input  $\mathbf{I}_g$  and  $\mathbf{I}_s$  are first mapped to the feature map,  $\mathbf{F}_g = C N N_g(I_g)$  and  $\mathbf{F}_s = C N N_s(I_s)$ , where  $C N N_g$  and  $C N N_s$  are feature extraction modules. Merely relying on the feature extractor is insufficient to address the significant difference between the two fields of view. As a solution, we introduce the camera's intrinsics  $\pmb{K}$ , extrinsics  $\pmb{T}$  to perform perspective transformation. Specifically, for any world point  $\pmb{x}^W$ , the perspective transformation maps it to its corresponding image coordinate  $\pmb{x}^I$ , as follows:
$$
\boldsymbol {x} ^ {I} \simeq \boldsymbol {K T} \boldsymbol {x} ^ {W} \tag {1}
$$
Here,  $\simeq$  denotes equality up to a scale factor. However, since there is no exact ground height, the correspondence between ground and satellite images is ambiguous. To address this issue, we exploit the fact that the overlap between ground and satellite images mainly occurs on the ground plane[13]. Therefore, within our ground feature projection module, we utilize Eq. 1 to map ground pixels  $x^{I} = [p,1]$  to the bird's-eye view (BEV) perspective via  $p^{\prime} = G(p)$  by incorporating ground height  $h$ , where the world point is denoted by  $x^{W} = [p^{\prime},h,1]$ . Subsequently, we employ bilinear interpolation to sample the ground image and project it onto a top-down view using this correspondence.
To further enhance the quality of our BEV feature map, we further propose a RefineBlock that addresses the following:
$$
\mathbf {F} _ {g 2 s} = \operatorname {R e f i n e B l o c k} \left(\mathbf {F} _ {g}, \boldsymbol {p} ^ {\prime}\right) \tag {2}
$$
The RefineBlock is a key component in our network architecture, aiming to address the limitations of ground-to-overhead projection based on ground-plane homography, including incorrect correspondences for objects above the ground plane and distortions in the projections. To overcome these issues, the network should have a larger receptive field based on ground-plane homography projection of the input overhead view feature map for better and higher-level understanding of the feature map. Therefore, in the RefineBlock, a  $7 \times 7$  convolutional layer is first employed to achieve a larger receptive field. Then, a  $3 \times 3$  residual block is applied to refine the details in local regions. Finally, a  $1 \times 1$  convolutional layer is utilized to facilitate feature interaction and integration, introducing enhanced non-linear transformation capabilities in the feature space. Ultimately, it generates a self-centered
BEV image that closely resembles the mental image humans construct during their own localization process.
3.2 Dense Flow Estimation
Similar to RAFT, the satellite features and the refined BEV features are subjected to a dot product operation, resulting in a four-dimensional correlation matrix that represents the similarity between every point in the two feature maps. This four-dimensional matrix is then subjected to average pooling with a  $2 \times 2$  kernel on the last two dimensions, yielding features of varying resolutions. The features obtained from the different resolutions are then combined to form correlation volumes.
Our architecture consists of two separate feature extractors, namely the feature encoder and the context encoder. Here, we have utilized a feature encoder equipped with a view unification module to deal with the significant differences in viewing angles between the  $\mathbf{I}_g$  and  $\mathbf{I}_s$ . The context encoder has the same architecture as the feature encoder but is designed to extract only semantic and contextual information from the reference image  $\mathbf{I}_g$ .
Finally, we create an index based on the initialized match positions to query the correlation volume. We use the GRU structure to iteratively update the match positions, which generates the final match results  $p' \leftrightarrow \hat{p}$  as well as the match score  $S$  for each point, which is dependent on the similarity scores obtained from the matching process.
3.3 Least Squares Pose Regression
In the case of a BEV image  $\mathbf{F}_{g2s}$  centered around itself, alignment with an uncalibrated satellite image can be achieved through rotation and translation. By doing so, it becomes apparent that the points  $\pmb{p}_i^{\prime}$  contained in  $\mathbf{F}_{g2s}$  and their corresponding points  $\hat{\pmb{p}}_i$  in  $\mathbf{F}_s$  are related through Euclidean transformations. So our objective is to find the optimal values of the positional parameters  $\theta$  and  $\pmb{t}$  of the ground camera by minimizing the following loss function:
$$
\xi = \arg \min  _ {\theta , t} \sum_ {i = 1} ^ {n} S _ {i} \left(\boldsymbol {R} \boldsymbol {p} _ {i} ^ {\prime} + \boldsymbol {t} - \hat {\boldsymbol {p}} _ {i}\right) ^ {2}, \boldsymbol {R} = \left[ \begin{array}{l l} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{array} \right] \tag {3}
$$
Here, the confidence score of the  $i$ -th match is represented by  $S_{i}$ , which is comprised of two components. The first component indicates whether the  $i$ -th match is visible in the BEV image, with a value of 1 for visible matches and 0 for invisible matches. The second component is the score of the  $i$ -th match, which is calculated by the network based on the similarity of the matches.
By using the least square methods, we can calculate the relative translation and rotation of the camera in the satellite image through matching pairs of points. Detailed computation procedure is presented in the appendix materials. It is noteworthy that this process is differentiable, which implies that we can compute gradients and optimize the model's parameters through backpropagation.
3.4 Training Objective
Our loss function  $\mathcal{L}$  consists of three components: matching loss  $\mathcal{L}_m$ , confidence loss  $\mathcal{L}_c$ , and position loss  $\mathcal{L}_p$ .
Matching Loss. The process of mapping matching points between  $\mathbf{F}_{g2s}$  and  $\mathbf{F}_s$  is supervised by  $\mathcal{L}_m$ . To evaluate the accuracy of the matching points, we calculate the distance between the matching points and the real matching points.
$$
\mathcal {L} _ {m} = \sum_ {i = 1} ^ {n} \| f _ {i} ^ {g t} - f _ {i} ^ {p r e d} \| _ {1} \tag {4}
$$
where for the  $i$ -th point,  $f_{i}^{gt}$  denotes its ground truth flow and  $f_{i}^{pred}$  denotes the predicted flow. The true matching points are generated by the camera poses, which provide a way to calculate the corresponding positions of each point in both  $\mathbf{F}_{g2s}$  and  $\mathbf{F}_s$ .
**Confidence Loss.** The loss of confidence generated by the network is measured using  $\mathcal{L}_c$ . For each matching point, the network generates a confidence score, and we aim for high confidence when the network predicts a small distance between the matching point and the true matching point, and low confidence when it predicts a large distance.
$$
\mathcal {L} _ {c} = \sum_ {i = 1} ^ {n} \| \frac {S _ {i}}{1 + \exp \left(- \tilde {d} _ {i} / \kappa\right)} + \frac {1 - S _ {i}}{1 + \exp \left(\tilde {d} _ {i} / \kappa\right)} \| _ {1} \tag {5}
$$
To balance the two terms of the formula, we first transform  $d$  into a standard normal distribution by using the formula  $\tilde{d}_i = (d_i - mean(d)) / std(d)$ . We then use  $\kappa$  to make the results more discretely distributed. In the beginning of the experiment, the variance of  $d$  is large, so we set  $\kappa$  to 200. After fifteen rounds of training,  $dis$  converges and the distribution becomes more concentrated. At this point, we set  $\kappa$  to 20.
Position Loss. We also utilized end-to-end supervised prediction, using  $\mathcal{L}_p$ , to estimate the difference between the predicted location and the true location. Although  $\mathcal{L}_m$  already incorporates location information, we observed that incorporating end-to-end supervision further improved the accuracy of the network's predictions.
$$
\mathcal {L} _ {p} = \left\| \theta_ {g t} - \theta \right\| _ {1} + \left\| \boldsymbol {t} _ {g t} - \boldsymbol {t} \right\| _ {1} \tag {6}
$$
where  $\theta_{gt}$  and  $t_{gt}$  is the GT camera pose.
Total Loss. The overall loss is the sum of the matching loss  $\mathcal{L}_m^l$  and confidence loss  $\mathcal{L}_c^l$  generated by each iteration, as well as the final resulting position loss  $\mathcal{L}_p$ .
$$
\mathcal {L} = \beta \mathcal {L} _ {p} + \sum_ {i = 1} ^ {l} \left(\mathcal {L} _ {m} ^ {l} + \alpha \mathcal {L} _ {c} ^ {l}\right) \tag {7}
$$
Here, the subscript  $l$  denotes the  $l$ -th iteration of the GRU. During training, we set  $\alpha = 100$  and initialize  $\beta = 1$ . After 15 rounds of training, we increase the momentum parameter to  $\beta = 10$ . This adjustment is made to prevent the model from getting trapped in local optima during the initial stage of training.
In this section, we will begin by introducing the dataset and evaluation metrics employed. After that, we conduct a comparison between our proposed methodology and state-of-the-art approaches. Finally, we will present a comprehensive ablative study.
We conducted tests on the KITTI[3, 13], Ford multi-AV[1, 13], VIGOR[5, 24], and Oxford RobotCar[8, 9, 22] datasets. The query images in KITTI, Ford multi-AV, and Oxford RobotCar datasets have a limited horizontal field of view (HFOV), while the query images in the VIGOR dataset are panoramic. Given that KITTI and VIGOR were captured on different road segments, we evaluated the model's performance on both same-area and cross-area scenarios using these two datasets. Ford multi-AV was mainly captured in suburban areas with fewer reference points, and Oxford RobotCar was captured on the same road segment at different times. We tested the model's generalization ability to different environments, including suburban scenes, varying lighting conditions, and road situations, on these two datasets. The datasets used in this paper are obtained under academic licenses and are not original datasets specifically created for this work. A more detailed description of each dataset will be provided in the appendix.
We followed the settings of [5, 13, 22]. Specifically, we calculate the mean and median errors in meters between predicted and ground truth positions, as well as the mean and median angular errors
Table 1: Location and orientation estimation error and recall on KITTI dataset[3, 13]. Angular noise was constrained within a range of  $\pm 10^{\circ}$  [13].
Area	Model	Location(m)	Lateral(%)	Longitudinal(%)	Azimuth(°)	Azimuth(%)
mean	median	r@1m	r@5m	r@1m	r@5m	mean	median	r@1°	r@5°
Same	DSM*[15]	-	-	10.12	48.24	4.08	20.14	-	-	3.58	24.44
LM[13]	12.08	11.42	35.54	80.36	5.22	26.13	3.72	2.83	19.64	71.72
SliceMatch[5]	7.96	4.39	49.09	98.52	15.19	57.35	4.12	3.65	13.41	64.17
Ours	1.48	0.47	95.47	99.79	87.89	94.78	0.49	0.30	89.40	99.31
Cross	DSM*[15]	-	-	10.77	48.24	3.87	19.50	-	-	3.53	23.95
LM[13]	12.58	12.11	27.82	72.89	5.75	26.48	3.95	3.03	18.42	71.00
SliceMatch[5]	13.50	9.77	32.43	86.44	8.30	35.57	4.20	6.61	46.82	46.82
Ours	7.97	3.52	54.19	91.74	23.10	61.75	2.17	1.21	43.44	89.31
Table 2: Location and orientation estimation error and recall on Ford multi-AV dataset[1, 13]. angular noise was constrained within a range of  $\pm 10^{\circ}$  [13].
Area	Model	Location(m)	Lateral(%)	Longitudinal(%)	Azimuth(°)	Azimuth(%)
mean	median	r@1m	r@5m	r@1m	r@5m	mean	median	r@1°	r@5°
Log1	DSM*[15]	-	-	12.00	53.67	4.33	21.43	-	-	3.52	13.33
LM[13]	12.54	12.63	48.57	71.57	5.90	26.33	3.13	1.29	42.90	79.62
Ours	10.55	10.19	68.67	95.48	5.48	25.86	1.43	0.47	75.76	94.29
Log2	DSM*[15]	-	-	8.45	37.64	3.94	21.41	-	-	2.23	13.42
LM[13]	12.01	11.49	29.94	77.78	4.94	26.00	4.36	3.75	15.35	63.00
Ours	10.50	9.40	33.43	97.59	15.62	40.94	1.65	1.03	48.99	95.57
in degrees between predicted and ground truth bearings. For the KITTI and Ford multi-AV datasets, we also included the recall values for longitudinal and lateral localization errors, as well as orientation estimation errors, under certain thresholds. Our localization thresholds were set at 1 meter and 5 meters, while our orientation estimation thresholds were set at 1 degree and 5 degrees.
We employed ResNet18 to extract features from both ground and satellite images, respectively, ensuring that the resolution was maintained at 1/8th of the original with 256 channels. Subsequently, in the dense flow estimation stage, we performed 12 iterations to obtain the final dense matching relationship. Finally, we used the least squares method to estimate relative position and relative angles based on the estimated matching relationships. During the training period, we utilized the Adam optimizer[7] for end-to-end training, with a learning rate of  $2 \times 10^{-5}$ ,  $\beta_{1} = 0.9$ , and  $\beta_{2} = 0.999$ . The entire training schedule consisted of 25 epochs. At the 15th epoch, we adjusted the parameter  $\kappa$  in the loss function from 200 to 20 and  $\beta$  from 1 to 10. We conducted 25 rounds of training on two TITAN V GPUs with a batch size of 6. Each epoch required approximately two hours of training time. During the evaluation process, estimating the pose of a ground image took an average of 0.25 seconds and approximately occupied 6GB of GPU memory.
4.4 Same-Area Evaluations
We conducted performance evaluations of the model in the same region on both the KITTI and Vigor datasets. The results of the KITTI dataset tests are shown in Table 1. Our method significantly outperformed previous algorithms in camera pose estimation, with an average error reduction of  $81\%$  and a median error reduction of  $89\%$  compared to SliceMatch. Notably, our algorithm's predictive performance for vertical position in the same region was significantly higher than that of other algorithms, indicating that despite the limited field of view, our approach learned the semantic relationship between roadside buildings in this area and their corresponding features in satellite images.
Table 3: Location and orientation estimation error and recall on VIGOR dataset[5, 24]. Aligned Images means the ground image orientation is known.
Model	Aligned	Same-Area	Cross-Area
Location(m)	Azimuth(°)	Location(m)	Azimuth(°)
mean	median	mean	median	mean	median	mean	median
CVR[24]	✓	8.99	7.81	-	-	8.89	7.73	-	-
MCC[22]	✓	6.94	3.64	-	-	9.05	5.14	-	-
SliceMatch[5]	✓	5.18	2.58	-	-	5.53	2.55	-	-
Ours	✓	3.03	0.97	-	-	5.01	2.42	-	-
SliceMatch[5]	×	8.41	5.07	28.43	5.15	8.48	5.64	26.20	5.18
Ours	×	4.97	1.90	11.20	1.59	7.67	3.67	17.63	2.94
Table 4: Location error on Oxford RobotCar dataset[8, 9, 22]. The orientation of the ground image is known[22].
Model	Test1	Test2	Test3
Location(m)	Location(m)	Location(m)
mean	median	mean	median	mean	median
CVR[24]	1.88	1.47	2.64	1.99	2.35	1.71
MCC[22]	1.42	1.10	1.95	1.33	1.94	1.29
Ours	1.17	0.72	1.76	0.97	1.79	0.92
In Table 3, the testing results on the VIGOR dataset show that our method achieves significantly lower average and median localization errors than SliceMatch by  $42\%$  and  $62\%$ , respectively, when the direction is known. When the direction is unknown, our algorithm still performs well, with both the average and median localization errors lower than those of SliceMatch.