SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paul-Edouard Sarlin<sup>1*</sup>  
psarlin.com
Eduard Trulls<sup>2</sup>  
trulls@google.com
Marc Pollefeys<sup>1</sup>  
marc.pollefeys@ethz.ch
Jan Hosang $^{2}$   
hosang@google.com
Simon Lynen  $^{2}$  slynen@google.com
$^{1}$ ETH Zurich
Figure 1: We learn neural  $2D$  maps from multi-modal imagery using camera poses. SNAP outperforms the state of the art in visual positioning, and by solving localization as a proxy task learns easily interpretable, high-level semantics through self-supervision alone, without any semantic cues.
Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain, especially in an automated fashion. Can we use raw imagery to automatically create better maps that can be easily interpreted by both humans and machines? We introduce SNAP, a deep network that learns rich neural 2D maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps. $^{\dagger}$
Semantic 2D maps such as Google Maps are ubiquitous in our daily lives, used by billions of people. They offer compact, yet easily interpretable representations of the world from a bird's-eye view, allowing us to effectively navigate large outdoor environments by foot or vehicle. By contrast,
*Work done during an internship at Google. †Code available at github.com/google-research/snap 37th Conference on Neural Information Processing Systems (NeurIPS 2023).
Figure 2: Training architecture. We feed overhead and ground-level imagery to per-stream encoders (Sec. 2.1) to produce 2D bird's-eye-view neural maps, fused via cell-wise max-pooling (Sec. 2.2). We also extract a 'query' neural map from a single ground-level image with the same ground-level encoder. Given known poses, we train SNAP by simply registering 'query' and 'scene' maps (Sec. 3).
machines position themselves in the real world through computer vision, which remains dominated by structure-based approaches [79, 83, 56, 74, 39, 65] relying on basic hand-crafted [54, 4] or learned [58, 111, 23, 69, 101, 75, 53] primitives, such as points or lines. These approaches build 3D maps with Structure-from-Motion (SfM) and then localize query images via 2D-3D registration. Their complexity and many components (feature extraction and matching, bundle adjustment, pose refinement, etc.) make it difficult to tune [42] or update [25] them, and to learn high-level priors end-to-end [10, 8, 76]. They are also costly to store and generally not reusable for other applications.
Recent works such as OrienterNet [78] instead learn planar, neural representations from the same 2D semantic maps that humans use. These maps encode scene geometry and semantics and can be used for visual positioning with sub-meter accuracy. This approach is however limited to a few semantic classes, and the maps it is based on can be inaccurate, costly to obtain, and difficult to maintain.
We argue that maps are most useful for figuring out where we are when they are abstract enough to be robust to temporal changes, yet preserve enough geometric and semantic information to yield high-quality correspondences with the physical world. Our work, SNAP, shows that, by learning 2D neural maps for localization, meaningful semantics emerge without explicitly supervising them. These semantics improve positioning accuracy and also make our maps usable for other tasks (Fig. 1).
SNAP leverages the complementary strengths of different input modalities, like ground-level and overhead imagery, by fusing them into a single 2D neural map (Fig. 2). It can flexibly and efficiently integrate arbitrary combinations of data captured at different points in time, which is key to continuously update maps in a changing world. We train it end-to-end to estimate the pose of a query image relative to the mapping images, by simply aligning their neural maps. This kind of contrastive learning requires only sensor poses, which can be easily obtained with photogrammetry [45, 37]. We train and evaluate SNAP on a dataset with 50M StreetView images from 5 continents, orders-of-magnitude larger and more diverse than comparable academic benchmarks.
Despite training only for a positioning objective, we observe that our neural maps learn easily interpretable, high-level semantics without the need for explicit semantic cues (Fig. 3), and demonstrate that they provide an effective pretraining for semantic understanding tasks by fine-tuning them on little labeled data. This can potentially unlock cost-efficient creation of more detailed and richer maps, readable by humans and machines alike, while providing state-of-the-art visual positioning.
Our main contributions are as follows. (i) We introduce a simple and lightweight encoder to estimate bird's-eye view maps from ground-level imagery, combining principles from multi-view geometry with strong monocular cues. (ii) We fuse different imaging modalities to integrate and benefit from complementary cues. (iii) We show how to train our model by aligning neural maps in a contrastive learning framework, using RANSAC to mine hard negatives. (iv) We outperform the state of the art on visual positioning and register image queries beyond the reach of traditional methods (Fig. 3). (v) We demonstrate that high-level semantics emerge by learning to align neural maps, without any explicit supervision, and fine-tune them on semantic understanding with few labels (Fig. 6).
${}^{ \ddagger  }$  Analytical use of StreetView imagery was done with special permission from Google.
query  $\Delta t = 0.3\mathrm{m}$ $\Delta R = 0.3^{\circ}$
a) A facade with repeated structure and large viewpoint differences between query and map images (medium).
query  $\Delta t = 0.7\mathrm{m}$ $\Delta R = 3.6^{\circ}$
b) Opposite views, localized using the ground and poles, observable in the neural map (hard).
query  $\Delta t = 0.6\mathrm{m}\Delta R = 1.1^{\circ}$
c) Opposite views on a road scene with linear structure, localized using a pole and shrubbery (hard).
query  $\Delta t = 0.5\mathrm{m}$  AR=1.4°
d) Opposite views, localized using a close-up of a pole and the markings on the ground (hard).
Figure 3: Single-image localization. We show the 3-DoF poses of map images (red), the GT query pose (black), and the pose predicted by SNAP (green) with its error  $(\Delta t, \Delta R)$ . SNAP can estimate accurate poses even for extreme opposite-views. We visualize neural maps by projecting them to RGB using PCA. Notice how objects like trees, poles, curbs or road markings are clearly recognizable.
2 Mapping the world with neural maps
We now formalize neural maps, and describe a neural network architecture to infer them from raw sensor data. Our goal is to infer a more generic neural representation that can encode both the geometry, semantics, and appearance of a given point in the 2D world.
Problem formulation: For a 3D scene, such as a large outdoor environment, we consider a local, 3D Cartesian coordinate system such that the  $z$  axis points upwards along the gravity direction. A neural map  $\mathbf{M}$  is defined over a regular grid that partitions the  $xy$  plane into  $I\times J$  square cells of size  $\Delta$ . Each cell  $(i,j)$  is associated with a  $D$ -dimensional feature  $\mathbf{M}_{ij}\in \mathbb{R}^D$ . To infer such neural map, we leverage large quantities of raw imagery captured by diverse cameras.
Input modalities: Ground-level images are captured by cameras mounted on StreetView cars or backpacks [18]. They are often part of a sequence of multi-camera frames. As such, they are very unevenly distributed throughout space. Each image offers a high resolution view of a small area, mainly limited by the occlusion of static or dynamic objects like buildings or vehicles. On the other hand, overhead images are captured by cameras mounted on planes or satellites. These images benefit from high spatial coverage at a uniform but low resolution. Their visibility is mostly affected by vertical occluders like trees. Ground-level and overhead images capture different aspects of the environment and are thus complementary.
Assumptions: All images of either modality are calibrated and registered with respect to the map coordinate system. Each image  $n$  follows a projection function  $\Pi_n : \mathbb{R}^3 \to \mathbb{R}^2$  that maps a 3D point in the world to a 2D point on the image plane.  $\Pi_n$  combines the camera pose  $w\mathbf{T}_n \in \mathrm{SE}(3)$  and the camera calibration, including lens distortions. Overhead images are ortho-rectified, such that world points along the  $z$  axis project onto the same pixel coordinate. As this process relies on a coarse digital surface model [27], fine details like poles are not rectified and may result in artifacts, which SNAP can however learn to account for.
2.1 Fusing multi-modal representations
Each location in the world is observed by an arbitrary number of images for each modality, captured at arbitrary points in time. We thus follow a late-fusion strategy that first encodes each modality separately and only finally fuses them (Fig. 2). This can flexibly adapt to the available inputs and efficiently handle arbitrary spatial distributions of data.
Encoding: We design two encoders that each combine a subset of observations  $n$  into a single-modality neural map  $\mathbf{M}^n$  defined over the same grid as  $\mathbf{M}$ .  $\Phi_{\mathrm{OV}}$  encodes a single tile of overhead orthoimagery, while  $\Phi_{\mathrm{SV}}$  encodes a single image or multiple covisible ground-level StreetView images, e.g., a multi-view sequence. To best resolve the 3D information from perspective shots at arbitrary viewpoints,  $\Phi_{\mathrm{SV}}$  leverages both multi-view observations and monocular cues. We describe its architecture in detail in Sec. 2.2.  $\Phi_{\mathrm{OV}}$ , on the other hand, is a simple U-Net-style CNN [71] that computes a feature for each pixel of the overhead orthoimage, which is then resampled into the grid.
Fusion: We obtain the final neural map by fusing the set of encoded maps  $\{\mathbf{M}^n\}$  using a cell-wise max-pooling operation, i.e.,  $\mathbf{M}_{ij} = \max_n\mathbf{M}_{ij}^n\forall (i,j)\in I\times J$ . This can combine maps with different spatial extents, which is essential to scale to large areas. The max aggregation picks the best estimate among all inputs for each feature channel and thus handles partial observations, such as when the road surface cannot be resolved in overhead images because it is occluded by trees.
2.2 Ground-level image encoder
We design a single module,  $\Phi_{\mathrm{SV}}$ , that can arbitrarily encode one or multiple images, ordered or not.  $\Phi_{\mathrm{SV}}$  first fuses the image data into 3D space and later projects it vertically into the map plane (Fig. 4). This design can handle arbitrary ground geometries and accurately resolve the 2D location of overhanging 3D structures, like street lights. The 3D fusion leverages both multi-view geometry and strong monocular cues learned end-to-end.  $\Phi_{\mathrm{SV}}$  can thus resolve objects that are observed by a single image, while maximizing accuracy when multiple observations are available.
Monocular inference: We consider an unordered set of  $N$  images  $\{\mathbf{I}^n\}$ ,  $N \geq 1$ . Each image  $n$  is encoded independently by a CNN  $\Phi_{\mathbf{I}}$  into a  $C$ -dimensional feature image  $\mathbf{F}^n \in \mathbb{R}^{H \times W \times C}$ .  $\Phi_{\mathbf{I}}$  also estimates a pixel-wise depth  $\mathbf{S}^n \in \mathbb{R}^{H \times W \times D}$  as a score over  $D$  depth planes along the ray of each
Figure 4: Ground-level encoder: combining multi-view geometry and monocular priors. We use a CNN to predict pixel-wise features and a monocular occupancy volume, separately for each view. We then interpolate them over a column of 3D points (at predefined heights), for each 2D cell. Finally, a simple MLP combines them into features  $\mathbf{X}_k$  that are pooled along the column, into a neural cell.
pixel.  $\mathbf{S}^n$  is similar to a frustum-aligned occupancy volume [78, 67] but contains unnormalized logits of a depth distribution. Instead of regressing a single value, this encodes the full depth uncertainty along the ray and thus allows  $\Phi_{\mathbf{I}}$  to provide meaningful multi-modal estimates. We distribute the depth planes uniformly in log space to correlate with the uncertainty of monocular depth estimation [3, 78].
Multi-view fusion: To fuse information in 3D, we define  $K$  horizontal planes at heights  $\{z_k\}$ , which are uniformly distributed within a range of interest defined with respect to the height of the camera [85, 50], e.g., from  $4\mathrm{m}$  below to  $8\mathrm{m}$  above. For a 2D map cell  $(i,j)\in I\times J$ , we consider its center point  $(x,y)$  and a column of 3D points  $\{\mathbf{P}_k = (x,y,z_k)\}$ . For each 3D point  $k$ , we define the subset of views that best observe it as  $\mathcal{N}_k\subseteq \{1\dots N\}$ , e.g., those that are closest spatially. We project the point to each of these views, obtain a 2D observation  $\mathbf{p}_k^n = \Pi_n(\mathbf{P}_k)$ , and sample the corresponding feature image with bi-linear interpolation:  $\mathbf{F}_k^n = \mathbf{F}^n[\mathbf{p}_k^n]$ . Given the depth  $d_k^n$  of  $\mathbf{P}_k$  in the corresponding view, we also tri-linearly interpolate a score from the depth prior:  $\mathbf{S}_k^n = \mathbf{S}^n[\mathbf{p}_k^n,d_k^n]$ . Intuitively,  $\mathbf{S}_k^n$  is low if the 3D point is in free space or is occluded in view  $n$ . Following common practice in learned multi-view stereo [110, 106], we then compute feature consistency statistics, as mean and variance  $(\boldsymbol {\mu}_k,\boldsymbol {\sigma}_k)\in \mathbb{R}^C$ , weighted by the depth priors:
$$
\boldsymbol {\mu} _ {k} = \sum_ {n \in \mathcal {N} _ {k}} w _ {k} ^ {n} \mathbf {F} _ {k} ^ {n} \quad \text {a n d} \quad \boldsymbol {\sigma} _ {k} = \sum_ {n \in \mathcal {N} _ {k}} w _ {k} ^ {n} \left(\mathbf {F} _ {k} ^ {n} - \boldsymbol {\mu} _ {k}\right) ^ {2} \quad \text {w i t h} \quad w _ {k} ^ {n} = \operatorname * {s o f t m a x} _ {n \in \mathcal {N} _ {k}} \mathbf {S} _ {k} ^ {n}. \tag {1}
$$
A Multi-Layer Perceptron (MLP) fuses this information into a feature  $\mathbf{X}_k$ , which is finally pooled across all points in the column, resulting in a neural map cell  $\mathbf{M}_{ij}$ :
$$
\mathbf {M} _ {i j} = \max  _ {k} \mathbf {X} _ {k} \quad \text {w i t h} \quad \mathbf {X} _ {k} = \operatorname {M L P} \left(\left[ \boldsymbol {\mu} _ {k}, \boldsymbol {\sigma} _ {k}, \max  _ {n \in \mathcal {N} _ {k}} \mathbf {S} _ {k} ^ {n} \right]\right). \tag {2}
$$
Adding the maximum depth score differentiates free and occupied space when the point is observed by a single image. This makes it possible to use the same model for single images and sequences.
By tightly combining 3D geometry and representation learning, our approach leverages both monocular priors and multi-view information, while past research on 2D mapping or 3D reconstruction typically relies on only one of the two. Compared to expensive Transformers [102] or 3D CNNs [33], we show that a simpler, lightweight MLP is effective at fusing multi-view information, inspired by [82]. Compared to top-down 2D CNNs that squash the vertical dimension [34, 70], this MLP is more expressive and makes our neural maps equivariant to 2D translations and rotations and invariant to translations along the vertical axis. Overall, this simple design enables scaling to very large scenes, which is critical to provide hard negatives for contrastive learning and ultimately learn rich semantics.
3 Learning from pose supervision
Alignment as contrastive learning: We want neural maps to encode high-level semantic information about the environment. Given recent advances in self-supervised learning [14, 64], we hypothesize that this can emerge from learning distinctive features that distinguish one location from another and that are invariant to viewpoint and temporal appearance changes. Intuitively, good maps help us identify where we are. More generally, good maps are such that we can unambiguously align them
when inferred from partial inputs. Consider neural maps  $\mathbf{M}^Q$  and  $\mathbf{M}^R$  obtained from two disjoint subsets of inputs, the query  $Q$  and the reference  $R$ . In camera pose estimation,  $Q$  corresponds to a single ground-level image and  $R$  to a sequence of images with an aerial tile. Because our encoder is flexible, we can use the same shared model to encode  $Q$  and  $R$  (Fig. 2).  $\mathbf{M}^Q$  is defined over a grid  $\mathbf{G}^Q \in \mathbb{R}^{I \times J \times 2}$  in a local coordinate frame, e.g., aligned with the query camera, where  $\mathbf{G}_{ij}^Q$  is the center point of cell  $(i,j)$ , while  $\mathbf{M}^R$  is defined in the world frame.
We define a score function  $E(\mathbf{T}; \mathbf{M}^Q, \mathbf{M}^R) : \mathrm{SE}(2) \to \mathbb{R}$  that evaluates the consistency between  $\mathbf{M}^Q$  and  $\mathbf{M}^R$  given an estimate of their 3-DoF relative pose  $_R\mathbf{T}_Q \in \mathrm{SE}(2)$ . To distinguish the ground-truth pose  $_R\mathbf{T}_Q^*$  from  $K$  other, incorrect poses  $\{_{R}\mathbf{T}_{Q}^{k}\}$ , we want to increase  $E(_{R}\mathbf{T}_{Q}^{*})$  and decrease  $E(_{R}\mathbf{T}_{Q}^{k})$  (omitting  $\mathbf{M}^Q$  and  $\mathbf{M}^R$  for brevity). This corresponds to a contrastive learning problem, for which we minimize the InfoNCE loss [63]
$$
\operatorname {L o s s} \left(\mathbf {M} ^ {Q}, \mathbf {M} ^ {R}\right) = - \log \frac {\exp \left(E \left(_ {R} \mathbf {T} _ {Q} ^ {*}\right) / \tau\right)}{\sum_ {k \in \{*, 1 \dots K \}} \exp \left(E \left(_ {R} \mathbf {T} _ {Q} ^ {k}\right) / \tau\right)}, \tag {3}
$$
where  $\tau$  is a learnable temperature parameter. Neural maps are trained end-to-end and require only relative poses  ${}_{R}\mathbf{T}_{Q}^{*}$ , which can be easily obtained at a large scale using photogrammetry [45, 37].
Feature metric pose scoring: A linear layer projects each neural map  $\mathbf{M}$  to a lower-dimensional, L2-normalized map  $\bar{\mathbf{M}}$ . This creates an information bottleneck that encourages compact features. The score  $E$  evaluates the consistency of two neural maps as the similarity of each cell after warping:
$$
E \left(_ {R} \mathbf {T} _ {Q}\right) = \frac {1}{I J} \sum_ {(i, j) \in I \times J} \max  \left(\bar {\mathbf {M}} _ {i j} ^ {Q \top} \bar {\mathbf {M}} ^ {R} \left[ _ {R} \mathbf {T} _ {Q} \cdot \mathbf {G} _ {i j} ^ {Q} \right], 0\right), \tag {4}
$$
where  ${}_{R}\mathbf{T}_{Q}$  transforms a grid point from coordinate frames  $Q$  to  $R$  and  $[\cdot]$  interpolates the map at this location. max clips negative scores to zero to reduce the impact of outliers, as in robust optimization.
Negative sampling: A critical and well-studied aspect of contrastive learning is the selection of negative samples [35, 98, 108]. Hard negatives should be high-likelihood but incorrect predictions, so as to push the probability mass to the ground truth. Random poses can be easily distinguished and exhaustive voting in the 3-DoF pose space is computationally infeasible at high resolution [78, 6, 28]. Instead, we use RANSAC [29] to sample poses that are consistent with the predicted features. We sample pairs of 2D-2D correspondences between all cells of both neural maps and solve for the relative pose using the Kabsch algorithm [43]. Inspired by PROSAC [19], we sample a correspondence between cells  $(i,j)$  and  $(k,l)$  based on its feature similarity with probability  $P_{ijkl} = \text{softmax}_{ijkl}\left(\bar{\mathbf{M}}_{ij}^{Q\top}\bar{\mathbf{M}}_{kl}^{R} / \tau\right)$ . Unlike NG-RANSAC [10], gradients are propagated through the scoring rather than the sampling and are thus much smoother. Because the sampling and scoring mirror similar feature metric errors, negative samples become harder as the learning proceeds.
Inference-time alignment: SNAP can estimate the unknown 3-DoF relative pose between any two neural maps. We estimate each map in the sensor coordinate frame, establish tentative correspondences by matching their cells, sample pose hypotheses, and select the pose with the highest score. This includes single-image positioning, where the query map  $\mathbf{M}^Q$  covers the camera frustum. The vertical pooling requires that the gravity direction is known, which is a reasonable assumption for applications like Augmented Reality (AR) and robotics [56, 113, 78]. Our framework also applies more generally to aligning any pair of inputs, including sequence-to-sequence and aerial-to-ground registration, which is required in the first place to pose mapping data in a common reference frame.
Visual positioning is most commonly tackled with geometric approaches [80, 40, 74] that rely on point correspondences across images and sparse 3D point clouds built with SfM [83, 2]. They then estimate the 6-DoF query pose with a robust solver [29, 20, 21, 19, 15, 5] from correspondences with the reference model or images. Such correspondences are most often estimated by sparse local features [54, 4]. This process is complex and end-to-end back-propagation is impractical [8]. Past works have thus focused on learning specific components like feature extraction [111, 58, 23, 24,
99, 69, 26, 101, 105, 55], matching [112, 115, 75, 91, 41, 117, 107, 53], and pose [104, 76] or point cloud refinement [52]. Coarse GPS location and gravity direction are commonly assumed to be known [113, 56, 93]. In AR and robotics, the height of the camera can be estimated as the distance to the ground in a local SLAM reconstruction [78]. These assumptions reduce the problem to 3-DoF estimation and make it more amenable to end-to-end learning. MapNet [38] also learns end-to-end 3-DoF visual mapping and localization but requires sequences of depth inputs. Recent works leverage overhead instead of ground-level images [87, 86, 109, 28]. They easily scale to large scenes but only in open-sky areas. Their accuracy is also limited by the low resolution of aerial imagery. Our work combines the strengths of both ground-level and overhead imagery by learning end-to-end how to best fuse them for 3-DoF positioning. Our differentiable pose estimation, based on RANSAC, is more efficient [38, 28, 78], robust [87], and stable [10, 8] than previous approaches.
Semantic representations can largely benefit loop closure [84] and pose estimation [100]. Ori-enterNet [78] learns 3-DoF positioning end-to-end from public 2D semantic maps that are more compact yet detailed enough for localization. Its accuracy is however limited because these maps have low spatial accuracy and are infrequently updated. It is also also restricted to few, explicit semantic classes that are often not discriminative. Differently, [49] learns finer-grained semantic classes for temporal and viewpoint consistency. Our work instead learns implicit semantics from posed imagery by combining end-to-end self-supervised learning with large amounts of data. This boosts the positioning accuracy and is an effective pre-training for semantic tasks.