SlowGuess/NeurIPS_2023_OCR huggingface dataset:
During research, we found that the ground truth for pixel coordinates of ground images on satellite patches in the VIGOR dataset [42] is inaccurate. The VIGOR dataset uses a uniform meter-to-pixel resolution for converting the latitude and longitude of ground images to their location in aerial images across all four cities. SliceMatch [12] also discovered this issue and proposed a correction, but their improvement is limited to using different average meter-to-pixel resolutions for each city, which may still result in some inaccuracies in different regions of the same city.
We propose the use of Mercator projection [1] to directly compute the pixel coordinates of ground images on specified satellite images using the GPS information provided in the dataset. Equation 4 allows us to calculate the pixel coordinates of a GPS coordinate on a global scale. We compute the
(a) Street View
(b) BEV_center
(c) Satellite_center
(d) Certain Point
(e) Pixel Label
Figure 10: Illustration of label problem in VIGOR dataset [42]. In (e), we show the label provided by the VIGOR [42], the label corrected by SliceMatch [12], and the pixel coordinates computed using our method. The space distance between label provided by VIGOR and ours reaches  $1.93\mathrm{m}$ . The pair of (d) and (h) represent the the pixel coordinates obtained by our method and their corresponding GPS location on Google Maps, demonstrating the accuracy of our method.
(f) Ground_GPS
(g)Satellite_GPS
(h) Corresponding GPS
Table 5: Absolute error statistics for labels in VIGOR [42] and SliceMatch [12] in four cities. The absolute error is defined as the distance between the original and the corrected locations.
City	VIGOR (m)	SliceMatch (m)
Min.	Mean	Median	Max.	Min.	Mean	Median	Max.
Chicago	0.00	0.44	0.45	0.80	0.00	0.10	0.13	0.31
New York	0.00	0.20	0.21	0.39	0.00	0.16	0.16	0.42
San Francisco	0.00	0.42	0.45	0.86	0.00	0.09	0.13	0.27
Seattle	0.00	1.73	1.80	3.13	0.00	0.12	0.13	0.33
pixel coordinates of the satellite patch's GPS label and the pixel coordinates of the ground image's GPS label. We then add the difference between them to the center coordinates of the current satellite patch, resulting in accurate pixel coordinates of the ground image on the satellite patch. The result of the correction is shown in Figure 10, which shows that the label provided by VIGOR has a significant deviation from the true position. The label corrected by SliceMatch [12] reduces this deviation, but there is still some error between the corrected label and the true position. The statistical information on the absolute error, measured in meters, between the labels provided in VIGOR [42] and SliceMatch [12] and the corrected labels using our method is presented in Table 5.
Note that in the model training process, including our method and all other methods, we do not directly use the GPS coordinates for training. This is because if GPS is used directly to calculate the training loss, the data truncation problem will occur when using Float32 tensor format since the valid data is five decimal places and trigonometric functions are used in the calculation.
B.2 Label Creation in KITTI Dataset
Supervised information required for training our network includes the pixel location on the bird's-eye view image, as well as its corresponding true GPS value. However, obtaining this information directly from the KITTI dataset [8] is not feasible, and we need to calculate it ourselves.
To obtain the required supervised information, we use the calibration files provided by KITTI and the point cloud data in the dataset. Given a 3D point  $\mathbf{x}$  in Velodyne coordinates, we project it to a point  $\mathbf{y}$
(a) Example 1
Figure 11: Three examples of labels created for the KITTI dataset [8]. The red dots indicate the location of the same point in the front view, bird's-eye view, and satellite view, respectively.
(b) Example 2
(c) Example 3
in the i-th camera image as follows:
$$
\mathbf {y} = \mathbf {P} _ {\text {r e c t}} ^ {(i)} \mathbf {R} _ {\text {r e c t}} ^ {(0)} \mathbf {T} _ {\text {v e l o}} ^ {\text {c a m}} \mathbf {x}, \tag {15}
$$
where  $i = 2$ ,  $\mathbf{P}\mathrm{rect}^{(i)}$ ,  $\mathbf{R}\mathrm{rect}^{(0)}$ , and  $\mathbf{T}_{\mathrm{velo}}^{\mathrm{cam}}$  are calibration parameters provided in the KITTI dataset. Additionally, we calculate the distance deviation in terms of latitude and longitude for the 3D point with respect to the origin of the IMU/GPS coordinate system in the world coordinate system. This is represented as  $\mathbf{x}_{\mathrm{GPS}}$ :
$$
\mathbf {x} _ {\mathbf {G P S}} = \mathbf {T} _ {\mathrm {i m u}} ^ {\mathrm {w}} \mathbf {T} _ {\mathrm {v e l o}} ^ {\mathrm {i m u}} \mathbf {x}. \tag {16}
$$
Using the above process, we compute the latitude and longitude deviations for a particular 3D point relative to the GPS sensor, and determine its corresponding GPS value using the meter-to-GPS calculation method. Finally, we utilize the homography transformation  $H_{\mathrm{final}} = H_2H_1H$  obtained from Section A.2 to map the pixel coordinates  $\mathbf{y}$  to their corresponding pixel coordinates on the BEV image. During model inference, we can obtain the GPS coordinates of a particular pixel on the BEV image. Using the inverse process of the above method, we can obtain the latitude and longitude deviations of the corresponding 3D point relative to the GPS sensor, and estimate the GPS value corresponding to the camera. Some examples is shown in Figure 11.
C Future Research
In our projection of the panoramic images into a bird's-eye view, we place a tangent plane at the south pole of the spherical imaging plane as the new imaging plane. However, this method assumes that the ground camera is oriented vertically, which may not always be the case due to roll and pitch deviations. Assuming  $(\alpha, \beta, \gamma)$  represent the angles of (roll, pitch, yaw), the rotation matrix can be obtained as follows:
$$
\mathbf {R} = \left[ \begin{array}{c c c} \cos \gamma \cos \beta & \cos \gamma \sin \beta \sin \alpha - \sin \gamma \cos \alpha & \cos \gamma \sin \beta \cos \alpha + \sin \gamma \sin \alpha \\ \sin \gamma \cos \beta & \sin \gamma \sin \beta \sin \alpha + \cos \gamma \cos \alpha & \sin \gamma \sin \beta \cos \alpha - \cos \gamma \sin \alpha \\ - \sin \beta & \cos \beta \sin \alpha & \cos \beta \cos \alpha \end{array} \right]. \tag {17}
$$
By multiplying the rotation matrix  $\mathbf{R}$  with the camera coordinates obtained from Equation 9, we obtain the new coordinates  $(x_1', y_1', z_1')$  in the camera coordinate system. We then proceed with the subsequent calculations to obtain the bird's-eye view image under the assumption of the specified (roll, pitch, yaw) angles. Different angle configurations yield different bird's-eye view images, as illustrated in Figure 12. It can be observed that the resulting BEV appearance varies significantly when different (roll, pitch) angles are given. Therefore, we suggest that in future research, (roll, pitch) can be treated as additional predicted outputs to obtain more degrees of freedom in estimating the ground camera pose. Our differentiable spherical transform provides a possibility for this idea.
Figure 12: Bird's-eye view images obtained under different roll, pitch, and yaw angles using our spherical transform method.
This paper introduces an innovative approach to achieving high-precision localization of ground cameras, a capability of immense value in applications such as autonomous driving, augmented reality, and mapping services. Accurate localization is crucial for these industries to operate safely and effectively. However, it is imperative to acknowledge that the position of a vehicle or a user's camera is often sensitive and private information. The misuse or abuse of this technology could lead to privacy violations, potentially endangering individuals' security and personal information. We advocate safeguards and regulations to ensure that the privacy and security of individuals are not compromised in the process.