SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Variant	Easy (25%)	Med. (50%)	Hard (25%)	All (100%)
SNAP (StreetView-only)	39.8 / 51.8	36.5 / 47.2	22.3 / 32.3	34.0 / 44.9
Fixed-height plane (a)	34.0 / 45.4	29.6 / 39.8	18.6 / 28.0	28.2 / 38.6
No monocular occupancy (b)	29.8 / 41.7	26.1 / 37.3	16.1 / 25.7	24.7 / 35.8
No multi-view variance (c)	28.8 / 39.1	23.5 / 32.6	14.0 / 21.7	22.7 / 31.8
Ray conditioning (d)	11.9 / 22.5	8.7 / 17.4	4.5 / 10.1	8.6 / 17.1
Table 2: Ablation study of the ground-level encoder. We report the single-image positioning AUC up to thresholds  $(2.5\mathrm{m} / 5^{\circ})$  and  $(5\mathrm{m} / 10^{\circ})$ . Variant (a) aggregates the information on an horizontal plane at a fixed height below the camera [1, 86]. This is insufficient for overhanging objects, when ground footprints are occluded, or when the scene is not planar. Variant (b) performs multi-view fusion without monocular priors [106, 110]. This makes it impossible to resolve the depth of objects in the single-image query. Variant (c) further drops the variance term and simply averages feature volumes [34]. This makes it harder to resolve surfaces. Variant (d) replaces the occupancy volume by conditioning each observation feature on the ray and distance using an MLP [85]. This is much more expensive and thus constrains both batch size and scene size, which in turn lowers performance.
Component	Operator	Easy (25%)	Med. (50%)	Hard (25%)	All (100%)
Vertical pooling
(StreetView-only)	max	39.8 / 51.8	36.5 / 47.2	22.3 / 32.3	34.0 / 44.9
average	32.4 / 43.6	29.1 / 39.3	17.8 / 27.2	27.3 / 37.6
Multi-modal fusion
(StreetView+aerial)	max	45.9 / 58.6	41.5 / 53.0	27.3 / 37.9	39.3 / 51.0
average	45.5 / 58.4	40.9 / 52.6	27.8 / 38.8	39.0 / 50.9
Table 3: Ablation study on pooling operators. Top: In the ground-level encoder, pooling the features vertically with the max operator performs much better than averaging, as measured by single-image positioning AUC. Bottom: Fusing StreetView and aerial neural maps with either max or average pooling performs comparably.
et al. [34] flatten the vertical elements with a space-to-depth (or pixel-shuffling) operation, followed by an MLP. This makes the model sensitive to a translation along the vertical axis, which rarely occurs in small driving datasets based on a few cars with identical specifications, but matters for heterogeneous data captured by backpacks and cars with widely different setups. We thus found that this approach yields a lower performance than a simpler pooling. This also makes it impossible to adjust the number of height planes at inference time.
Multi-modal fusion: Tab. 3-bottom shows that the choice of pooling operator makes little difference when fusing neural maps inferred from StreetView and aerial inputs.
B Qualitative examples
Visual positioning: Due to space constraints, we showcase only a handful of visual positioning examples in the main paper. We show additional visualizations of successfully localized queries in Figs. 8 to 10 and failure cases in Figs. 17 and 18. We also show some examples with non-linear mapping trajectories in Fig. 19, which our method is robust to. As in Fig. 3, we show the 3-DoF poses of map images (red), the GT query pose (black), and the pose predicted by SNAP (green) with its error  $(\Delta t,\Delta R)$ . We visualize neural maps by projecting them to RGB using PCA.
Sequence to sequence alignment In the paper we focus on the alignment of single-image queries to maps built from multiple views, but SNAP can arbitrarily align any pair of neural maps. We show two examples of sequence to sequence alignment in Figs. 11 and 12. These maps are built from five views. For ease of understanding, we lift our neural maps into 3D using LiDAR, color-coding each LiDAR point with the neural map values of the cell it belongs to – note that this is strictly for visualization purposes, and our algorithm does not rely on LiDAR.
Semantic mapping: We show additional visualizations of the semantic maps predicted by our model after fine-tuning it in only 2k scenes in Figs. 14 and 15.
a) The query is localized within  $0.6\mathrm{m}$  from a different road (medium).
query  $\Delta t = 0.7\mathrm{m}$ $\Delta R = 1.3^{\circ}$
b) The query is localized within  $0.7\mathrm{m}$  despite significant occlusions in the reference views (medium).
query  $\Delta t = 0.2\mathrm{m}\Delta R = 2.1^{\circ}$
c) The tree is clearly visible in both the query and the reference neural maps (hard).
query  $\Delta t = 0.3\mathrm{m}$ $\Delta R = 1,2^{\circ}$
d) The query is correctly localized despite drastic perspective changes and reflections in the window (medium).
Figure 8: Single-image localization (1/3). We show successful examples. See legend in Fig. 3.
a) The query is localized across opposite views (hard).
query  $\Delta t = 0.5\mathrm{m}\Delta R = 2.5^{\circ}$
b) The query is on the other side of the street, far away from the reference views (medium).
query  $\Delta t = 0.1\mathrm{m}\Delta R = 0.1^{\circ}$
query  $\Delta t = 0.2\mathrm{m}\Delta R = 1.1^{\circ}$
c) Road markings are the only elements in common across opposite views (medium).
query  $\Delta t = 0.2\mathrm{m}\Delta R = 0.5^{\circ}$
d) The query is correctly localized across a different street and despite drastic perspective changes (hard).
Figure 9: Single-image localization (2/3). We show successful examples. See legend in Fig. 3.
a) Notice the salient spot corresponding to the utility pole in the neural maps (medium).
b) Notice that the building facade closest to the query is not visible from the reference views (hard).
c) The query is localized within  $1.6\mathrm{m}$  from a side street (hard).
d) The query is localized from a difficult angle (medium).
Figure 10: Single-image localization (3/3). We show successful examples. See legend in Fig. 3.
Figure 11: Sequence-to-sequence alignment (1/2). We align a car sequence (red) to a backpack sequence (blue) across opposite views. Notice how the neural maps (top right) are clearly similar despite the large viewpoint difference, and can be easily aligned (bottom right). In the bottom left we lift the neural map for the first sequence into 3D by coloring a LiDAR point cloud with the RGB value of the grid cell it belongs to – note that this is strictly for visualization purposes, and our method does not rely on LiDAR. Notice how the semantics learned by our model correspond to real scene features, such as the pole (in cyan, enclosed in a black box).
Large-scale mapping: We can easily build large tiles by 'stitching' smaller neural maps together via cell-wise max-pooling, similarly to how we fuse aerial and ground-level neural maps. Starting from multiple sequences posed in a common reference frame, Fig. 13 shows how neural maps are inferred for each of them and finally combined together.
C Detailed results
Fig. 21 shows the single-image positioning recall performance for each for the 6 test cities. The performance of the different approaches is fairly consistent across cities, with the exception of Osaka. We observe that Osaka has a large number of small, narrow streets, and StreetView sequences often feature close-ups of building facades: see Fig. 18-(c-d). This is somewhat out of distribution with the training set and could be solved by training on more data. It also explains why structured-based matching approaches are either very accurate (when there is sufficient visual overlap between query and reference views) or fail to register the image (when there isn't).
Fig. 22 shows the recall for queries taken by cameras mounted on either backpacks or cars. Backpack queries have a lower recall because they are typically taken from a sidewalk and thus have larger viewpoint differences (Fig. 16).
Figure 12: Sequence-to-sequence alignment (2/2). We align two car scenes, the second of which is heavily occluded due to a passing bus. Notice how the neural maps (top right) are visually similar and can be easily aligned. The neural map for the first sequence clearly shows a row of poles, which are occluded in the second sequence (black box). Our method is robust and can align the sequences using road markings and building boundaries. On the top right we plot an alignment heatmap by running exhaustive alignment in 3-DoF between both neural maps (we max-pool over rotations, for ease of understanding): notice the clear maximum (circled), with smaller maxima along the road. In the bottom left we lift the neural map for the first sequence into 3D by coloring a LiDAR point cloud with the RGB value of the grid cell it belongs to – note that this is strictly for visualization purposes, and our method does not rely on LiDAR.
Figure 13: Building neural tiles. We combine an aerial view and car sequences captured over multiple years into a single neural map that spans a large area. An arbitrary number of inputs can be combined in such way.
a) The curb is mostly occluded by the parked cars and has very sparse ground truth labels. Notice how our neural maps encode the location of street lights that are hanging above the street.
b) Because of the bike lane and confusing markings at the intersection, the model incorrectly segments the crosswalk.
Figure 14: Semantic mapping (1/2). 2D segmentation predicted from pre-trained neural maps by a small CNN trained in a supervised fashion with GT semantic labels of 2k scenes. We also show 3D lidar point clouds colored by the predicted segmentation (lidar is not used as input).
a) The StreetView car turns around the corner, so the right side of the map is behind the cameras and thus very sparsely observed. Areas with good visual coverage are much better segmented.
b) The model roughly segments road, sidewalk and buildings, and correctly resolves small objects such as the poles or the overhanging traffic light. It gets confused around the sidewalk, which has multiple striped patterns that are also difficult to interpret by the human eye.
Figure 15: Semantic mapping (2/2). 2D segmentation predicted from pre-trained neural maps by a small CNN trained in a supervised fashion with GT semantic labels of 2k scenes. We also show 3D lidar point clouds colored by the predicted segmentation (lidar is not used as input).
Figure 16: Large viewpoints and temporal differences. Left: The easy split contains more car-mounted queries, and the hard split contains more backpack queries. This makes sense, as examples captured from backpacks are typically captured from sidewalks instead of the road, which results in difficult localization scenarios across opposite views. Right: The time difference between query and mapping images spans from a few days to a few years. This yields challenging localization scenarios with large appearance and even structural changes, e.g., due to construction work.
We provide more details about the data used for the experiments conducted in the paper.
Platform type: Ground-level mapping sequences are selected from sequences captured by cars only. Query images are sampled from sequences captured by either cars or backpacks. For each map tile, we sample as many car queries as backpack queries, when possible. As cities are only partially covered by backpack sequences, the final ratio of backpack queries varies per city, from  $20\%$  to  $40\%$ . Fig. 16-left shows the ratio of both types for the different difficulty splits of the test set.
Map grid: We found that maps of size  $64 \times 16\mathrm{m}$  provide a good trade-off between visual diversity and memory consumption. Since multi-view frames are provided every  $\sim 5\mathrm{m}$ , segments of 14 frames generally cover a length of  $64\mathrm{m}$ . We therefore sample segments of 14 frames and define the map grid to be aligned with the average pose of the  $7^{\mathrm{th}}$  and  $8^{\mathrm{th}}$  side cameras. Given that most car segments are rectilinear, the map is most often aligned with the mapping views. In turns, the coverage of the map varies and can make the localization more challenging, which is generally a good property for both training and evaluation. Fig. 19 shows examples of maps with partial coverage.
Spatial distribution: For each of 11 training cities, we define disjoint training and validation regions. We show these regions in Fig. 20. We sample at most 5M training examples from each training region. The validation region corresponds to 5 (10 for New York) randomly-sampled level-14 S2 cells $^{\S}$ , from which we sample 1024 examples. 4k test queries are sampled from 20 random S2 cells in each of the 6 test cities, as shown in Fig. 23. This corresponds to an average area of  $5.6 \, \text{km}^2$  per city.
Temporal distribution: Sequences used for mapping or localization are randomly drawn from the pool of available StreetView data. Fig. 16-right shows the distribution of temporal differences. This yields challenging queries with large appearance changes, e.g., due to the seasons, or structural changes, e.g., due to construction work. Aerial images are selected as captured as close as possible to the ground-level mapping images. The average time difference varies from 6 months to 3 years.
Ground truth semantic rasters: In the semantic mapping experiments, we used a small labeled dataset to train and evaluate our fine-tuning. We start with 1.5k segments of data captured in 84 cities across the world by StreetView cars equipped with LiDARs. We aggregate the measurements into point clouds that we label into 10 classes spanning objects encountered in urban environments. These classes include surfaces with a direct ground footprint (road, sidewalk, crosswalk, building, tree) and small overhanging objects (fence, pole, street light, traffic sign, traffic light). As each segment is about  $80\mathrm{m}$  long, we derived two maps of size  $64\times 16\mathrm{m}$  from each, facing either side of the street. We derived semantic rasters by binning the points into each map cell and inferring two types of labels: a) Class surfaces are mutually-exclusive in 2D so we compute multi-class surface labels corresponding to the class most represented in each map cell. b) Differently, objects might appear on top of each other so they are not mutually-exclusive. We thus compute binary labels for each object class using a threshold on the number of points in each map cell. We obtain 3k semantic rasters which are split between training and test sets, each including data from all 84 cities.
<sup>8</sup>As defined by Google's S2 geometry library: https://s2geometry.io
Semantic rasters for OrienterNet: To obtain labels at the scale required to train OrienterNet [78], we apply this rasterization process to sparse Structure-from-Motion points that have similar semantic labels. Those labels are obtained fully automatically by fusing 2D image segmentations, without any human work [30]. While the resulting rasters are noisier than those derived from LiDAR, they cover our entire training set rather than only 3k maps. They also more consistently cover small objects (like trees and street lights) than OpenStreetMap.
Training: We develop our models with JAX [11] and Scenic [22], and format our dataset with TFDS [97]. All models are trained on 16 A100 GPUs over 3-4 days with a total batch size of 32 (2 examples per GPU). We use the ADAM [44] optimizer over 400k iterations for the small model and 200k iterations for the larger one. This corresponds to  $24\%$  and  $12\%$  of the training data, respectively, but is sufficient for convergence. We use a constant learning rate of  $10^{-4}$  for the first half and then decay it to zero with a cosine decay. Streetview images are resized to  $684 \times 456$ . We train our models with mixed precision and use gradient checkpointing in order to reduce GPU memory usage as much as possible. The temperature  $\tau$  is parameterized as  $\log \tau$ , initialized to 0, and optimized alongside other model parameters. All hyperparameters are tuned on the validation set.
In the multi-view fusion, we fuse observations from the 4 closest views that observe a point. This saves a significant amount of memory compared to fusing observations from all views. Using fewer views (20 instead of 36 at inference time) also makes the problem artificially harder.
Pose estimation: We use 10k RANSAC samples during training and 20k during inference. Since only 3 constraints are required to solve for a 3-DoF pose, 22D correspondences over-constrain the problem. Randomly sampling each correspondence independently thus yields a large number of invalid poses. To increase the ratio of valid poses, we sample  $8 \times$  the number of desired correspondences and pick the ones with the ratio of distances closest to 1. This improves the pose quality, at both training and test time, and is well amenable to parallelized GPU processing, without requiring an expensive sorting. When computing the feature metric score, query points that project outside the map are given the value of the map boundary. We found this to be more robust than ignoring these points. At inference time, we refine the pose by exhaustive search in a grid centered at the maximum-score pose, with size  $4\mathrm{m} \times 4\mathrm{m} \times 5^{\circ}$  and resolution  $20\mathrm{cm} \times 0.5^{\circ}$ .
Architecture: The decoder of the image encoder is an FPN with bilinear upsampling. We progressively upsample features from stages 4, 3, and 2 to the resolution of stage-1 ResNet features, which corresponds to an output stride of 4. For the aerial encoder, we remove the root block ( $7 \times 7$  convolution and max pooling) such that the output stride is 1.
Semantic fine-tuning: We train a small 2-layer CNN with two heads: one for the multi-class classification of ground surfaces, using a softmax with cross-entropy loss, and the other for the multi-label classification of objects, using sigmoid and binary cross-entropy losses. We train for 5k iterations and retain the checkpoint with the lowest validation loss.
F Social impact
CO2 emissions: Experiments were conducted using Google Cloud in region us-central1, which has a carbon efficiency of  $0.57\mathrm{kgCO_2eq / kWh}$ . A cumulative of 88.8k hours of computation was performed on NVidia A100 GPUs. Total emissions are estimated to be  $12654\mathrm{kgCO_2eq}$ , of which 100 percent were directly offset by Google. Had this model been run in region europe-west6, the emissions would have been of  $355.31\mathrm{kgCO_2eq}$ . Estimations were conducted using the ML Impact calculator [47].
a) The query is too far from the buildings and fails to localize using features along the road (medium).
b) SNAP match the right visual content from the wrong side, due to scene symmetries (medium).
c) Reasonable but incorrect guess along a row of parking spots with very little structure otherwise (medium).
d) Similar to (c), but across opposite views, and thus more challenging. Note that the rotation error is  $< 1^{\circ}$  (hard).
Figure 17: Localization failures (1/2). See legend in Fig. 3.
Figure 18: Localization failures (2/2). See legend in Fig. 3.
a) The tree is smeared yet clearly visible on the left side of the query BEV as a pink dash in a green circle (hard).
query  $\Delta t = 1.3\mathrm{m}\Delta R = 3.6^{\circ}$
b) The query is accurately localized (10 cm) from the other side of the square (hard).
query  $\Delta t = 0.1\mathrm{m}\Delta R = 1.1^{\circ}$
query  $\Delta t = 0.1\mathrm{m}\Delta R = 1.2^{\circ}$
c) The reference views are irregularly distributed due to the car's motion, to which our method is robust (hard).
query  $\Delta t = 0.8\mathrm{m}\Delta R = 0.1^{\circ}$
d) The query is localized within  $0.8\mathrm{m}$  from a very difficult angle with little visual overlap (hard).
Figure 19: Single-image localization. Examples of scenes with non-rectilinear mapping sequences, which results in partial coverage. Please refer to Appendix D for details.
Tokyo
Figure 20: Spatial distribution of the training data. In each city, training examples are sampled from the red areas while validation examples are sampled from the blue areas.
Figure 21: Single-image positioning per city. We plot the position and orientation recall for each of the 6 test cities.
Figure 22: Single-image positioning per platform. We plot the position and orientation recall for queries taken by cameras mounted on either backpacks or cars.
New York City
Figure 23: Spatial distribution of the test data. For each city, test queries are sampled from 20 level-14 S2 cells and are here shown as heatmaps.