SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaolong Wang\*,1, Runsen Xu $^{3}$ , Zuofan Cui $^{1}$ , Zeyu Wan $^{1}$ , Yu Zhang ${}^{†,1,2}$
$^{1}$  College of Control Science and Engineering, Zhejiang University  
 $^{2}$  Key Laboratory of Collaborative sensing and autonomous unmanned systems of Zhejiang Province  
 $^{3}$  The Chinese University of Hong Kong
In this paper, we introduce a novel approach to fine-grained cross-view geolocation. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. This transformation effectively places ground and aerial images in the same view and on the same plane, reducing the task to an image alignment problem. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by  $21.3\%$  and  $32.4\%$  in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by  $34.4\%$  on the KITTI benchmark in same-area evaluation.
Accurate localization of ground cameras is essential for various applications such as autonomous driving, robot navigation, and geospatial data analysis. In crowded urban areas, cross-view localization using satellite images has proven to have great potential for correcting noisy GPS signals [3, 38] and improving navigation accuracy [42, 13, 19]. In this paper, we consider the task of fine-grained cross-view geo-localization, which estimates the 3-Dof pose of a ground camera, i.e., GPS location and orientation (yaw), from a given ground-level query image and a geo-referenced aerial image.
Previous works on coarse cross-view localization [10, 16, 25, 11, 42, 37, 41] have achieved high recall rates by formulating it as an image retrieval problem. However, the localization accuracy obtained using this method is limited by the segmentation size of satellite image patches, resulting in localization errors of tens of meters.
Recently, there has been a growing interest in fine-grained cross-view geo-localization, which assumes the availability of the ground image and a corresponding GPS-labeled satellite image patch covering the same area. Existing methods can be divided into two categories: those based on repeated sampling [27, 24, 7, 26] and those employing descriptor candidates splitting from satellite image features
* Contact: xlking@zju.edu.cn. The Code is available at https://github.com/xlwangDev/HC-Net.
Figure 1: Visualization of the cross-view localization process in our method. The satellite image (c) is accompanied by a correlation map that represents the dense probability distribution for localization. Aligning the BEV with the satellite image using the homography matrix is shown in (d).
[36, 12, 35]. Sampling-based methods transform satellite or ground images (or feature maps) to compare with another view, involving the multiple applications of geometric transformations based on potential ground camera poses. However, these methods suffer from low localization accuracy [27, 24, 26] and prolonged time consumption[27, 24, 7, 26]. [36, 12, 35] split the features of satellite images into numerous candidates to obtain more effective and refined descriptors for comparison with the ground image descriptor. Their limited pixel-level localization resolution and high memory usage present challenges for deployment in real-world end-terminals. Hence, the pursuit of a methodology that encompasses swift computation, low memory utilization, and elevated levels of both localization accuracy and resolution remains a paramount area of research.
We observe that projecting ground images onto a bird's-eye view perspective can make satellite-based localization tasks more intuitive, similar to the way humans use maps for navigation, eliminating the need for multiple sampling or further splitting of satellite patches. In this study, we develop a spherical transform module that leverages the imaging model of ground cameras to project panoramic images onto an aerial perspective, effectively bridging the gap between distinct viewpoints. As depicted in Figure 1 (b), our approach does not necessitate prior pose information, and it facilitates the straightforward determination of the ground image's position within the satellite image (c). By transforming the complex cross-view localization problem into a 2D image alignment problem, our method enables the acquisition of precise ground GPS location and orientation.
The core problem in 2D image alignment lies in obtaining the homography transformation between two images. Given that feature-based methods like [22, 28] may not yield effective results in obstructed or unclear scenes, we employ a correlation-aware homography estimation approach. We utilize a recurrent convolutional neural network block to maximize the correlation between similar regions in the feature maps, unlike previous iterative optimization methods [24], which minimize overall feature differences. Our method ensures low correlation in unobservable areas, minimizing their impact on homography estimation, resulting in an end-to-end network that aligns the most similar parts of the transformed ground image with the satellite image, directly outputting its GPS location. Moreover, our approach has successfully overcome the challenge of lacking compact supervision information in homography estimation, i.e., the absence of at least four matching point pairs. While utilizing the VIGOR dataset [42], we also address the inherent errors in the original ground truth labels to facilitate further research.
The main contributions of this paper include:
- A novel and effective method for fine-grained cross-view geo-localization, which strictly aligns ground and aerial image domains using a geometry-constrained spherical transform, reducing the problem to 2D image alignment.
- A correlation-aware homography estimation module that eliminates repeated sampling and disregards unobservable content in the images, resulting in excellent generalization performance and rapid inference speed at 30 FPS.
- Extensive experiments demonstrating that our method outperforms the state-of-the-art on two fine-grained cross-view localization benchmarks. Specifically, on the same-area and cross-area splits of the VIGOR benchmark [42], our method reduces the mean localization error by  $21.3\%$  and  $32.4\%$ , respectively. On the same-area split of the KITTI benchmark [8], our method reduces the mean localization error by  $34.4\%$ .
Cross-view image retrieval methods for geo-localization, which use ground images as queries and all patches in a satellite image database as references, have been studied for years and rely on global image descriptors for successful retrieval. Early works [15, 34, 33, 31, 39] suffer from low retrieval accuracy due to significant appearance gaps and poor metric learning techniques. SAFA [25], Shi et al. [27], L2LTR[37], and TransGeo[41] have improved localization accuracy by employing polar transform algorithms, considering orientation, and using transformers [32].
Fine-grained cross-view localization beyond one-to-one retrieval is first proposed in CVR[42] and they propose a corresponding benchmark VIGOR. [27] and [24] project the satellite view using the candidate poses or the iterative optimization method and select the one that is most similar to the ground as the localization result. [7] and [26] employ transformers to acquire BEV representations of ground images, and then transform BEV maps by all possible ground camera poses to compare with the feature representations of satellite images. However, there remains untapped potential for refinement, and their inference entails prolonged time. [36] splits the features of satellite patches into several sub-volumes. [12] considers orientation and geometric information to create effective satellite descriptors for a set of candidate poses. [35] incorporates orientation and employs a coarse-to-fine manner to enhance accuracy. Despite these improvements, all three methods face resolution and high memory usage challenges stemming from the splitting or candidate selection process.
Bird's-Eye View Perspective has been thought of as the preferred representation for many tasks (e.g. navigation or localization). Recent methods [14, 7, 26] utilize transformers to project feature maps of ground images onto BEV. However, there has been no utilization of explicit BEV images for fine-grained cross-view geo-localization. A challenge could stem from the requirement of known camera calibration for traditional Inverse Projective Mapping methods, as [23] used. In an attempt to enhance BEV representations, Boosting [26] also explores geometric-based methodologies, but due to the necessity for depth information, transformers remain a crucial tool to handle ambiguity.
Homography estimation is the process of estimating a mapping between two images of a planar surface from different perspectives. DeTone et al. [6] is the first to propose a deep learning-based approach to homography estimation. Nie et al. [20] proposed the use of a global correlation layer to address the problem of small overlapping regions between the two images. Zhao et al. [40] incorporate the Lucas-Kanade (LK) algorithm [18] as a non-trainable iterator and combine it with CNNs. They also investigate GPS-denied navigation using Google static maps and satellite datasets based on this method. Inspired by RAFT [30], IHN [4] designed a completely iterative trainable deep homography estimation network, achieving high accuracy but requiring strong supervision. HomoGAN [9] designed an unsupervised homography estimation network based on transformer and GAN methods, achieving promising results while consuming more computational resources.
This paper presents a novel approach to address the task of fine-grained cross-view localization, as illustrated in Figure 2. Our Homography estimation-based Cross-view geo-localization Network, named HC-Net, takes corresponding spherical-transformed ground image and GPS-tagged satellite image as input and outputs the homography matrix between them, as well as the precise GPS location and orientation of the ground camera. The following sections introduce the spherical transform for projecting panoramas to a bird's-eye view (Section 3.1), the homography estimator for precise alignment of ground BEV images with satellite images (Section 3.2), and the supervision method of our network (Section 3.3).
3.1 Spherical Transform
Panoramic imaging model Panoramas capture a full 360-degree view on a spherical projection plane and use equirectangular projections for display, as shown in Figure 2 (a). We represent points in the ground camera coordinate system as  $P = (x_{1},y_{1},z_{1})$  and projected points in the normalized equirectangular coordinate system of the panorama as  $P^{\prime} = (x_{2},y_{2})$ . The spherical coordinates  $(\phi ,\theta)$  are acquired from  $(x_{1},y_{1},z_{1})$  using inverse trigonometric functions, with north latitude and east longitude being positive. And the projection between the spherical coordinates  $(\phi ,\theta)$  and the normalised equirectangular coordinates  $(x_{2},y_{2})$  is expressed in Figure 2 (a), (b).
Figure 2: An overview of the proposed method for fine-grained cross-view localization. (0) illustrates the geometric principles behind our spherical transformation, while (1), (2), and (3) demonstrate our network architecture and supervision methods.
Spherical transform To obtain the corresponding bird's-eye view of the panorama, we place a tangent plane at the south pole of the spherical imaging plane as a new imaging plane, as shown in Figure 2 (c). Formally, let  $H_{p} \times W_{p}$  be the size of the panorama and  $H_{b} \times W_{b}$  be the target size of the bird's-eye view after the spherical transform. We connect the camera's optical center with each pixel point on the BEV imaging plane, determining the corresponding pixel position in the panorama through the intersection of the connection line with the spherical imaging plane. We demonstrate the pixel coordinates on the bird's-eye view imaging plane as  $(u_{b}, v_{b})$ . The camera coordinates  $(x_{1}, y_{1}, z_{1})$  corresponding to the pixel coordinates  $(u_{b}, v_{b})$  can be directly obtained. We set the parameter  $fov = 85^{\circ}$  to determine the field of view of the bird's-eye view. The focal length of the BEV in the imaging process is  $f = 0.5W_{b} / \tan(fov)$ . Therefore, each corresponding panoramic image coordinates  $(u_{p}, v_{p})$  is established as:
$$
\left\{ \begin{array}{l} u _ {p} = \left[ 1 - \arctan 2 \left(W _ {b} / 2 - u _ {b}, H _ {b} / 2 - v _ {b}\right) / \pi \right] W _ {p} / 2, \\ v _ {p} = \left[ 0. 5 - \arctan 2 \left(- f, \sqrt {\left(W _ {b} / 2 - u _ {b}\right) ^ {2} + \left(H _ {b} / 2 - v _ {b}\right) ^ {2}}\right) / \pi \right] H _ {p}. \end{array} \right. \tag {1}
$$
After the spherical transform, the ground image is projected into the aerial image's perspective, as seen in Figure 2(d). This approach enables high-resolution localization without multiple sampling or splitting of satellite patches. Unlike the satellite-based image projection discussed in [27, 24], this type of projection does not require the selection of a projection point. We adopt a similar approach for the KITTI dataset [8] by setting up an overlooking imaging plane. This allows us to project front-view images into a bird's-eye view without any known camera calibration, as shown in Figure 4 (a), see details in Supplementary Material. Despite being applied as a preprocessing step in our method, note that the transform is differentiable. This property opens up possibilities for future research on the 5-DoF pose of the ground camera (details in Supplementary Material).
3.2 Homography Estimator
To achieve precise localization by aligning the transformed ground image  $I_{g}$  with the satellite image  $I_{s}$ , we propose a correlation-based deep homography estimator inspired by IHN [4]. We resize the two images  $I_{g}$  and  $I_{s}$  to the same size and feed them into a Pseudo-Siamese CNN to extract features. These features are denoted as  $\widetilde{F}_{g} \in \mathbb{R}^{D \times H \times W}$  and  $\widetilde{F}_{s} \in \mathbb{R}^{D \times H \times W}$ . As shown in Figure 2, after feature extraction, correlation computation, and recurrent homography estimation, the homography matrix  $\mathbf{H}$  between  $I_{g}$  and  $I_{s}$  can be obtained.
Correlation Computation Given the feature maps  $\widetilde{F}_g$  and  $\widetilde{F}_s$ , the correlation volume  $\mathbf{C} \in \mathbb{R}^{H \times W \times H \times W}$  is formed by taking the dot product between all pairs of feature vectors as:
$$
\mathbf {C} \left(\widetilde {F} _ {g}, \widetilde {F} _ {s}\right) \in \mathbb {R} ^ {H \times W \times H \times W}, \quad C _ {i j k l} = \operatorname {R e L U} \left(\widetilde {F} _ {g} (i, j) ^ {\mathrm {T}} \widetilde {F} _ {s} (k, l)\right). \tag {2}
$$
To enlarge the perception range within a feature scale, we conduct average pooling on  $\mathbf{C}$  at the last 2 dimensions with stride 2 to form another correlation volume  $\mathbf{C}^{\frac{1}{2}} \in \mathbb{R}^{H \times W \times H/2 \times W/2}$ .
Recurrent Homography Estimation We refine the estimation of the homography through loop iterations. The coordinates set on  $\widetilde{F}_g$  are denoted as  $\mathbf{X} \in \mathbb{R}^{2 \times H \times W}$ , and the coordinates set on  $\widetilde{F}_s$ , which is projected from  $\mathbf{X}$  using the homography matrix  $\mathbf{H}$ , is denoted as  $\mathbf{X}' \in \mathbb{R}^{2 \times H \times W}$ . For each coordinate position, we denote  $x = (u, v)$  in  $\mathbf{X}$  and  $x' = (u', v')$  in  $\mathbf{X}'$ . In each iteration, we use the Equation 3 to calculate  $\mathbf{X}'^k$  and use it to sample the last two dimensions of  $\mathbf{C}$ ,  $\mathbf{C}^{\frac{1}{2}}$  with a local square grid of fixed search radius  $r$ , resulting in correlation slices  $\mathbf{S}^k$  and  $\mathbf{S}^{\frac{1}{2}, k}$  of size  $H \times W \times r \times r$ .
$$
\left[ \begin{array}{l} u ^ {\prime k} \\ v ^ {\prime k} \\ 1 \end{array} \right] \sim \left[ \begin{array}{c c c} \mathbf {H} _ {1 1} ^ {k} & \mathbf {H} _ {1 2} ^ {k} & \mathbf {H} _ {1 3} ^ {k} \\ \mathbf {H} _ {2 1} ^ {k} & \mathbf {H} _ {2 2} ^ {k} & \mathbf {H} _ {2 3} ^ {k} \\ \mathbf {H} _ {3 1} ^ {k} & \mathbf {H} _ {3 2} ^ {k} & 1 \end{array} \right] \left[ \begin{array}{l} u \\ v \\ 1 \end{array} \right] \tag {3}
$$
Then we utilize a convolutional neural network module for residual homography estimation. As in [4], we parameterize the homography matrix using the displacement vectors of the 4 corner points of an image, namely the displacement cube  $\mathbf{D}$ . In iteration  $k$ , we feed the concatenated correlation slice  $\mathbf{S}^k$ ,  $\mathbf{S}^{\frac{1}{2}, k}$ , the coordinates set  $\mathbf{X}$ , and the currently projected coordinates set  $\mathbf{X}'^k$  into the CNN module. The module is mainly composed of multiple convolutional units until the spatial resolution of the feature map reaches  $2 \times 2$ , where each unit downsamples the input by a scale of 2. Then a  $1 \times 1$  convolutional unit projects the feature map into a  $2 \times 2 \times 2$  cube  $\Delta \mathbf{D}^k$ , which is the estimated residual displacement vector of the 4 corner points. In iteration  $k$ , we update  $\mathbf{D}^k$  by adding the estimated residual displacement vector  $\Delta \mathbf{D}^k$  to  $\mathbf{D}^{k-1}$ . Using  $\mathbf{D}^k$ , we can obtain the homography matrix  $\mathbf{H}^k$  through the direct linear transform [2]. The updated  $\mathbf{H}^k$  is then used to project  $\mathbf{X}$  in the next iteration.
3.3 Network Supervision
Label Correction We use the VIGOR dataset [42] for fine-grained cross-view localization training and evaluation. However, the original labels in [42] contain errors up to 3 meters due to the use of approximate and consistent meter-to-pixel resolutions of the aerial images. Although [12] attempts to correct the labels, they require city-based computations or selections for resolution, which brings significant inconvenience. In our approach, we propose using the Web Mercator projection [1] used by virtually all major online map providers to accurately convert GPS coordinates to pixel coordinates on satellite patches, thereby enhancing generality and convenience. The main equation is as:
$$
\left\{ \begin{array}{l} x = \left[ \frac {2 5 6}{2 \pi} 2 ^ {\text {z o o m}} (\operatorname {l o n} + \pi) \right] \text {p i x e l s}, \\ y = \left[ \frac {2 5 6}{2 \pi} 2 ^ {\text {z o o m}} \left(\pi - \ln \left[ \tan \left(\frac {\pi}{4} + \frac {\operatorname {l a t}}{2}\right) \right]\right) \right] \text {p i x e l s}, \end{array} \right. \tag {4}
$$
where zoom indicates the zoom level for satellite images (zoom = 20 in VIGOR and zoom = 19 in KITTI) and 256 is the resolution of each satellite tile. We have applied the same method to create training labels for KITTI dataset [8], see Supplementary Material for details.
Loss Function Using the homography matrix  $\mathbf{H}$  obtained in Section 3.2, we project the center point of BEV onto the satellite image to obtain the localization pixel  $(u_{s}, v_{s})$ . Using Equation 4, we calculated the pixel coordinates of the ground truth GPS corresponding to the satellite image as  $(u_{s}^{*}, v_{s}^{*})$ . Furthermore, we use Equation 4 to project another point on the BEV centerline onto the satellite image. Connecting this point with  $(u_{s}, v_{s})$  allows us to determine the orientation of the ground camera as  $\theta$ .
We use a hybrid loss function, defined as  $\mathcal{L} = \alpha_{1}\mathcal{L}_{dis} + \alpha_{2}\mathcal{L}_{ori} + \alpha_{3}\mathcal{L}_{info}$ , to guide our training. Here,  $\mathcal{L}_{dis} = (u_s - u_s^*)^2 + (v_s - v_s^*)^2$  represents the L2 norm loss between the predicted result and the ground truth.  $\mathcal{L}_{ori} = |\theta - \theta^*|$  represents the L1 norm loss between the predicted  $\theta$  and the ground truth  $\theta^*$ . To generate a probability distribution that can be further utilized for localization, we propose the use of an infoNCE loss [21] to reinforce the correlation between the BEV point used for localization and the satellite image.  $\mathcal{L}_{info}$  is defined as:
$$
\mathcal {L} _ {i n f o} = - \log \frac {\exp \left(C \left(i _ {c} , j _ {c} , k ^ {+} , l ^ {+}\right) / \tau\right)}{\sum_ {k , l} \exp \left(C \left(i _ {c} , j _ {c} , k , l\right) / \tau\right)}, \tag {5}
$$
where  $(i_c, j_c)$  represents the center coordinates of  $\widetilde{F}_g$ , and  $(k^+, l^+)$  represents the downsampled position of  $(u_s^*, v_s^*)$  in the satellite feature map  $\widetilde{F}_s$ . The hyper-parameter  $\tau$  is introduced to adjust the sharpness of the resulting probability distribution.
Finally, through the inverse process of Equation 4, we can determine the GPS coordinates corresponding to  $(u_{s}, v_{s})$ , thereby obtaining highly accurate localization outputs.
In this section, we first introduce two used datasets, evaluation metrics, and implement details of our network. We then compare the performance of our HC-Net to state-of-the-art and examine its ability to generalize to new measurements within the same areas, across different areas, and across datasets. Finally, we present ablation studies and computational efficiency analysis.
4.1 Datasets and Evaluation Metrics
VIGOR dataset [42] contains geo-tagged ground-level panoramas and aerial images collected in four cities in the US. Each aerial patch corresponds to a ground area of approximately  $70m\times 70m$ . From Figure 1, it can be observed that the effective field of view of the ground image is slightly smaller than this range. A patch is considered positive if its center  $1/4$  region contains the ground camera's location, otherwise, it is semi-positive. In our experiments, we use positive aerial images for training and testing all models. The panoramas are shifted based on orientation information to align North in the middle, indicating that the orientation prior is known. During training, we introduce  $\pm 45^{\circ}$  noise to the orientation prior to generating orientation labels. The dataset provides 105, 214 panoramas for the geo-localization experiments. We adopt the Same-Area and Cross-Area splits from [42]. For validation and hyperparameter tuning, we randomly select  $20\%$  of the data from the training set, as done in[36, 12, 35]. Compared to [12], we have more accurately corrected the ground truth labels in [42], as mentioned in Section 3.3.
KITTI dataset [8] contains ground-level images captured by a moving vehicle with a forward-facing viewpoint, which is a restricted viewpoint. [24] augments the dataset with aerial images. Each aerial patch corresponds to a ground area of approximately  $100m \times 100m$ . The Training and Test1 sets consist of different measurements from the same region, while the Test2 set has been captured in a different region. As assumed in [24], ground images are located within a  $40 \times 40m$  area in the center of the corresponding aerial patches, and there is an orientation prior with noise between  $\pm 10^{\circ}$ .