SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Neural scene representation is an active topic of research. MLPs [57, 96] and tokens [73] are compact but lack geometric inductive bias. 3D voxel grids are more expressive and thus popular for reconstruction [66, 60, 92, 9, 119], rendering [59, 95], and semantic perception [17, 102, 13, 7] but are expensive to store and thus often restricted to small scenes. 2D grids, or Bird's-Eye Views (BEV), are more compact and thus scale to larger outdoor scenes by compressing the information along the vertical axis. Neural BEVs can be learned from images for supervised semantic tasks [72, 50, 34, 67, 70], 3D reconstruction [66], self-supervised view synthesis [85], and 3-DoF positioning [78, 28, 38]. These approaches assume planar scenes or rely on monocular priors only, even if multiple views are available. Instead, we combine these priors with multi-view fusion [110, 82, 106] to leverage information from image sequences and better resolve objects in large scenes.
Self-supervised learning leverages unlabeled datasets to learn representations useful for down-stream tasks. Many works focus on image- or pixel-level contrastive learning for semantic tasks [63, 35, 14, 62, 36]. View synthesis from few images typically learns lower-level representations [57, 85]. Some works [49, 89] learn features for image matching across appearance changes. CoCoNets [48] learns representations for 3D scenes but requires perfect, synthetic depth maps. We learn high-level contrastive scene representations from posed images and show that it translates to semantic mapping.
Data: StreetView images are captured by rigs of 6 rolling-shutter cameras mounted on cars or on backpacks worn by pedestrians [18], which results in a wide diversity of viewpoints in street-level scenes. Multi-view 'frames' are captured synchronously every  $\sim 5\mathrm{m}$ . Sequences are captured between 2017 and 2022. We build mapping segments only from car sequences by partitioning each sequence into groups of 36 images that face either the left or right side of the road. We define each map grid as a  $64\times 16\mathrm{m}$  tile aligned with the segment mid-frame, in which we render an aerial orthophoto with  $20\mathrm{cm}$  ground sample distance. Query images are sampled from different sequences, captured from cars or backpacks, based on their frustum overlap, and are often taken years apart. We train with 2.5M segments and  $\sim 50\mathrm{M}$  queries from 11 cities across the world: Barcelona, London, Paris (Europe), New York, San Francisco (North America), Rio de Janeiro (South America), Manila, Singapore, Taipei, Tokyo (Asia), and Sydney (Oceania), reserving some areas in each city for validation. We test on 6 different cities (Amsterdam, Melbourne, Mexico City, Osaka, São Paulo, and Seattle), with 4k queries per city. This covers 5 continents, while academic localization benchmarks focus on tourism landmarks [42] or single cities in Europe or the US [81, 116, 77] – see details in Appendix D.
Training and implementation: In the ground-level encoder,  $\Phi_{\mathbf{I}}$  is a U-Net [71] with a BiT ResNet backbone [46], pre-trained as in [114], and an FPN decoder [51], initialized randomly. We consider two models with different backbones: a 'large' R152x2 (353M parameters) and a 'small' R50x1 (84M parameters).  $\Phi_{\mathrm{OV}}$  is a similarly-defined R50x1+FPN. In multi-view fusion (Sec. 2.2) we use
Figure 5: Single-image positioning with different maps. Localizing with our neural maps yields a higher recall than established approaches based on feature matching  $(SfM + X)$ , especially for hard queries with low visual overlap. Neural maps are also more suitable for positioning than semantic maps because they encode richer and thus more discriminative information.
Algorithm	Inputs	Easy (25%)	Med. (50%)	Hard (25%)	All (100%)
SfM + SIFT [54]	StreetView	47.0 / 54.4	24.9 / 29.9	7.6 / 9.7	27.1 / 32.1
+ SuperGlue [75]	StreetView	63.0 / 71.1	38.0 / 44.4	13.1 / 16.1	39.2 / 45.2
OrienterNet [78]	semantic	35.6 / 47.2	29.3 / 39.5	24.8 / 34.8	30.0 / 40.6
SNAP-large ResNet-152x2	multi-modal	48.9 / 62.3	46.9/ 59.5	34.5 / 47.6	44.4 / 57.4
StreetView	45.8 / 58.4	43.9 / 56.0	29.5 / 41.7	41.0 / 53.2
aerial	27.4 / 40.6	25.3 / 37.5	20.8 / 32.3	24.8 / 37.1
SNAP-small ResNet-50	multi-modal	45.2 / 59.0	41.9 / 54.8	29.6 / 42.0	39.9 / 52.9
StreetView	42.2 / 54.9	38.1 / 50.1	24.5 / 36.4	36.0 / 48.2
aerial	23.9 / 35.6	21.9 / 32.8	17.9 / 27.5	21.5 / 32.3
Table 1: Single-image positioning. We report the area under the recall curve (AUC) up to thresholds  $(2.5\mathrm{m} / 5^{\circ})$  and  $(5\mathrm{m} / 10^{\circ})$ . Our large and small multi-modal models are more accurate than classical  $SfM + X$  approaches for medium and hard queries, which matter most in practical applications. Fusing both StreetView and aerial imagery is more accurate than using only one of them.
$D = 32$  depth planes and  $K = 60$  height planes  $\{z_k\}$  uniformly distributed within  $12\mathrm{m}$ . Neural maps  $\mathbf{M}$  and matching maps  $\bar{\mathbf{M}}$  have dimensions 128 and 32, respectively, and are defined over  $64\times 16\mathrm{m}$  grids with  $20\mathrm{cm}$  ground sample distance. Query BEVs have a maximum depth of  $16\mathrm{m}$ . At training time, neural maps are built from one aerial tile and one SV segment, with each of the two randomly dropped, similarly to dropout [90]. We use a subset of  $N = 20$  views, some of them at a  $\pm 60^{\circ}$  angle, which we empirically found provides a good coverage/memory trade-off. See details in Appendix E.
Visual positioning: We build a map for each segment using all 36 views and evaluate the 3-DoF query pose in terms of position and orientation errors. While many academic benchmarks use much larger mapping areas, we argue that GPS and motion priors often make this unnecessary for practical applications [56]. We slice the results by difficulty in terms of query-scene overlap based on the distance between the query and its closest map view, in position  $\Delta t$ , and orientation  $\Delta \theta$ . We split in the data into 3 groups: 'easy' ( $\Delta t < 10\mathrm{m}$  and  $\Delta \theta < 45^{\circ}$ ,  $\sim 25\%$  of the data), 'hard' ( $\Delta t > 10\mathrm{m}$  and  $\Delta \theta > 60^{\circ}$ ,  $\sim 25\%$ ), and 'medium' (the remaining  $\sim 50\%$ ). We compare our approach to hloc [74], a state-of-the-art [81, 77] structure-based 6-DoF localization system based on COLMAP [83], a popular Structure-from-Motion framework, with correspondences estimated by either RootSIFT [54, 4] or SuperPoint+SuperGlue [23, 75], a learned feature and matcher. Note that these approaches can only leverage ground-level imagery. We match the query to all map images, without using hloc's
Figure 6: 2D Semantic mapping. Left: t-SNE visualization of the neural map features learned by SNAP, colored by their ground-truth semantic class. SNAP discovers different categories of objects common in outdoor urban scenes, which yields clearly distinguishable clusters. Middle: Given a small labeled dataset, training a tiny CNN classifier to predict such classes from pre-trained features is more effective than training the entire SNAP model from scratch, especially for small and infrequent objects. Right: Test example with ground truth raster (middle) and prediction of the CNN (bottom).
retrieval component, and estimate the query's pose using RANSAC and a P2P solver with gravity constraint [94]. We evaluate the 6-DoF pose projected to 3-DoF. We also evaluate OrienterNet [78], which matches a query BEV with a semantic map. We re-implement and train it on overhead semantic rasters derived from SfM points with semantic labels obtained by fusing 2D image segmentations. Note that OrienterNet was originally trained on OpenStreetMap, which has limited coverage of small objects. While our rasters are noisy, they provide a consistent, global coverage of fine-grained classes like tree, streetlight, poles, etc. We checked our implementation with the authors. We also evaluate versions of SNAP trained with only either ground-level or aerial imagery – the latter is an extreme case of cross-view localization, similar to [28].
Fig. 5 and Tab. 1 show that SNAP outperforms the state of the art, COLMAP with SuperPoint+SuperGlue, by a large margin:  $25\%$  relative. Structure-based approaches are more accurate for easy queries but significantly worse for hard ones (Fig. 3). Using ground-level imagery is crucial in most localization scenarios and performs  $\sim 46\%$  relative better than using aerial imagery, whereas our multi-modal model performs  $\sim 8\%$  relative better than the StreetView-only variant. We justify our design decisions using an ablation study in Appendix A. We report detailed results per city and per sequence type in Appendix C. Our framework is also efficient, as mapping takes  $223\mathrm{ms}$  per segment and  $6\mathrm{ms}$  per aerial tile, estimating a query BEV takes  $14\mathrm{ms}$  and localizing it takes  $86\mathrm{ms}$ , on an A100 GPU. In comparison, matching with SuperGlue takes  $100\mathrm{ms}$  per pair for 36 pairs per query, and is thus 36 times slower. Each tile of our matching maps has size  $1.6\mathrm{MB}$  in fp16, while storing SuperPoint descriptors requires  $5.3\mathrm{MB}$  on average.
Semantic mapping: We show that SNAP's neural maps are an effective pre-training for 2D semantic mapping. Existing approaches rely on ground truth 2D semantic rasters derived from the segmentation of LiDAR 3D point clouds. These are manually labeled, which is too expensive to generate enough data to train from scratch models that generalize across countries, sensors, seasons, and times of the day. Existing datasets [12, 7] thus rarely span more than a few cities and overfit supervised models to the local appearance. Instead, our self-supervised pre-training learns better features from a much larger dataset of posed imagery, which is much cheaper to acquire at scale. The information bottleneck forces SNAP to learn unified representations for objects, like street crossings or lights, that look very different across countries, and would require larger amounts of labeled data. Fig. 6-left shows a 2D t-SNE [103] visualization of SNAP's neural maps at points sampled on a few types of objects common in street scenes, according to their ground-truth semantic label. Points of the same class are clustered together. This clearly shows that neural maps learn to distinguish these objects without any semantic supervision, even if they are geometrically similar, e.g., tree vs pole.
To evaluate the pre-training, we train a tiny CNN to predict semantic rasters from pre-trained neural maps, keeping SNAP frozen. We compare this to training the entire model from scratch (with the same backbones initialization [46, 114]). We derive  $3\mathrm{k}64\times 16\mathrm{m}$  ground truth rasters from LiDAR point clouds captured by StreetView cars in 84 cities across the world. We train with 2k examples and report the recall of both approaches on 1k test examples in Fig. 6-center. Pre-training consistently
Figure 7: Monocular depth priors learned by the ground-level encoder. For each query image (a), we show: (b) the expected log-depth across all depth planes, from blue (close) to red (distant); (c) the total score along each ray  $\log \sum_{i\in \{1\dots D\}}\exp \mathbf{S}[:, :, d_i]$ , which reflects how useful or confident the prediction is; (d) the resulting bird's-eye view. The predictions are sensible for areas close to the ground and for lower parts of objects and buildings. Predictions in the sky and upper facades are not reliable because these areas are never covered by the height planes  $\{z_k\}$  of the point columns.
yields better results for every class, with larger gains on more difficult/infrequent classes. While training from scratch massively overfits to such small dataset, our neural maps encode enough information to reach recalls over  $70\%$ . We show qualitative examples in Fig. 6-right and Appendix B.
Monocular priors: We visualize in Fig. 7 the occupancy predicted by SNAP as depth and confidence maps. SNAP learns sensible priors over the geometry of street scenes from only pose supervision.
**Limitations:** Our approach is not as accurate as structure-based methods given easy queries closer to map images (Fig. 5). We hypothesize this is partly due to operating at lower image resolutions. It also assumes gravity direction and a location prior, which are reasonable assumptions but restrict its use.
We present SNAP, a novel approach to build semantic, 2D neural maps from multi-modal inputs and train it by simply learning to align two neural maps in a contrastive framework. This simple objective yields a model that can localize queries beyond the reach of the state of the art in structure-based matching by discovering high-level semantics from self-supervision. Our neural maps are easily interpretable and provide an effective pre-training towards unlocking semantic understanding at scale.
Broader impact: This work has implications to privacy and surveillance. However, our 2D maps are too compact to preserve personal identifiable information, and likely more difficult to invert than point clouds [68], which despite ongoing efforts [88, 31, 32, 61, 118] remain susceptible to attacks [16].
Acknowledgements: We thank Bernhard Zeisl, Songyou Peng, Rémi Pautrat, and Michal Tyszzkiewicz for their valuable feedback, Manuel Cabral and Johann Volz for providing useful code reviews, Tianqi Fan for helping processing the data, Lucas Beyer for providing the pre-trained ResNet backbones, Thomas Funkhouser and Kyle Genova for providing the ground truth semantic labels, and Arjun Karpur for helping run SuperGlue.
[1] Ammar Abbas and Andrew Zisserman. A Geometric Approach to Obtain a Bird's Eye View From an Image. ICCV workshop, 2019. 17, 18
[2] Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle Adjustment in the Large. ECCV, 2010. 6
[3] Manuel López Antequera, Pau Gargallo, Markus Hofinger, Samuel Rota Bulò, Yubin Kuang, and Peter Kontschieder. Mapillary Planet-Scale Depth Dataset. ECCV, 2020. 5
[4] Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. CVPR, 2012. 2, 6, 8
[5] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiří Matas. MAGSAC++, a Fast, Reliable and Accurate Robust Estimator. CVPR, 2020. 6
[6] Ioan Andrei Barsan, Shenlong Wang, Andrei Pokrovsky, and Raquel Urtasun. Learning to Localize Using a LiDAR Intensity Map. CoRL, 2018. 6
[7] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyril Stachniss, and Jurgen Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. ICCV, 2019. 7, 9
[8] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task. CVPR, 2020. 2, 6, 7
[9] Aljaž Božić, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. TransformerFusion: Monocular RGB Scene Reconstruction using Transformers. NeurIPS, 2021. 7
[10] Eric Brachmann and Carsten Rother. Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses. ICCV, 2019. 2, 6, 7
[11] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. http://github.com/google/jax, 2018.27
[12] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. CVPR, 2020. 9
[13] Anh-Quan Cao and Raoul de Charette. MonoScene: Monocular 3D Semantic Scene Completion. CVPR, 2022. 7
[14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV, 2021. 5, 7
[15] Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, and Marc Pollefeys. AdaLAM: Revisiting Handcrafted Outlier Detection. ECCV, 2020. 6
[16] Kunal Chelani, Fredrik Kahl, and Torsten Sattler. How Privacy-Preserving are Line Clouds? Recovering Scene Details from 3D Lines. CVPR, 2021. 10
[17] Ricson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry-Aware Recurrent Neural Networks for Active Visual Recognition. NeurIPS, 2018. 7
[18] Danny Cheung. Mapping stories with a new Street View Trekker. https://blog.google/products/maps/mapping-stories-new-street-view-trekker/, 2018.4.7
[19] Ondrej Chum and Jiří Matas. Matching with PROSAC – Progressive Sample Consensus. CVPR, 2005. 6
[103] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 2008. 9
[104] Lukas Von Stumberg, Patrick Wenzel, Qadeer Khan, and Daniel Cremers. GN-Net: The Gauss-Newton loss for multi-weather relocalization. RA-L, 2020. 7
[105] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning Feature Descriptors using Camera Pose Supervision. ECCV, 2020. 7
[106] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. IBRNet: Learning Multi-View Image-Based Rendering. CVPR, 2021. 5, 7, 17, 18
[107] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. MatchFormer: Interleaving Attention in Transformers for Feature Matching. ACCV, 2022. 7
[108] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling Matters in Deep Embedding Learning. ICCV, 2017. 6
[109] Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual Cross-View Metric Localization with Dense Uncertainty Estimates. ECCV, 2022. 7
[110] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo. ECCV, 2018. 5, 7, 17, 18
[111] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned Invariant Feature Transform. ECCV, 2016. 2, 6
[112] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to Find Good Correspondences. CVPR, 2018. 7
[113] Bernhard Zeisl, Torsten Sattler, and Marc Pollefeys. Camera Pose Voting for Large-Scale Image-Based Localization. ICCV, 2015. 6, 7
[114] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. CVPR, 2022. 7, 9
[115] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning Two-View Correspondences and Geometry Using Order-Aware Network. ICCV, 2019. 7
[116] Zichao Zhang, Torsten Sattler, and Davide Scaramuzza. Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis. IJCV, 2021. 7
[117] Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive Correspondence Pruning by Consensus Learning. ICCV, 2021. 7
[118] Qunjie Zhou, Sérgio Agostinho, Aljosa Osep, and Laura Leal-Taixe. Is Geometry Enough for Matching in Visual Localization? ECCV, 2022. 10
[119] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. CVPR, 2022. 7
A Design decisions
In this section, we explain our design decisions and support them with an ablation study.
Constraints: Learning features that are discriminative requires sufficiently challenging negative pose samples. These arise from viewpoints that are visually similar to the ground truth viewpoint, for example due to repeated patterns, lack of distinctive features, or occlusions. In order to find sufficiently difficult negative samples in a training example, the map should be as large as possible. The size of the map and the number of mapping images are however limited by the amount of GPU memory available. We therefore found it critical to find the right balance between the size of the map, its spatial resolution, the number of views, and the batch size, as also reported in [34].
We found that a simple and lightweight model that saves memory is beneficial over a complex model that requires reducing the size of the map, the number of views, or the batch size. We thus favor explicit constraints by geometry (camera projection, 3D occupancy) rather than flexible but heavy mechanisms like attention or 3D convolutions. The effectiveness of our design shows that such complexity is not required. This enables the training of high-capacity models with maps of  $64 \times 16\mathrm{m}$ , at a resolution of  $20\mathrm{cm}$  ( $\sim 25\mathrm{k}$  cells per map), from 20 images, and within 20 GB of GPU memory per example. Despite extensive memory optimizations, including the use of gradient checkpointing and mixed-precision training, using larger maps for training remains challenging.
Ground-level encoder: We consider four alternative designs for the ground-level encoder  $\Phi_{\mathrm{SV}}$
(a) Fixed-height plane. Instead of lifting the image information to 3D and pooling it vertically, the geometry of the scene can be approximated from a ground plane [1]. Driving applications commonly assume that this plane is horizontal and at a fixed height below the camera [86]. We train a variant that aggregates the multi-view information on a plane  $2.5\mathrm{m}$  below the camera. This approach can hardly resolve the spatial location of overhanging structures, like street lights. It furthermore introduces distortions in the BEV if the image is not gravity aligned, which is the case for many capture platforms (such as StreetView backpacks or consumer phones), or the ground is not planar, as in many real environments.
(b) No monocular occupancy. Multi-view stereo [106, 110] typically does not explicitly leverage monocular geometry priors. We thus train a variant that omits the weighting by occupancy scores. Image features are thus painted identically along all depth planes for each ray and their mean and variance across all views is fed to the fusion MLP. This model can only leverage the bearing angles of point features (like poles) to disambiguate a pose but cannot resolve the location of line features.
(c) No multi-view variance. SimpleBEV [34] simply averages feature volumes obtained by painting image features along each ray. This corresponds to removing both variance and monocular priors from SNAP, making it even harder to resolve surfaces.
(d) Ray conditioning. Close to our approach, Sharma et al. [85] augment multi-view fusion with monocular cues, but do so by conditioning each ray feature by its 3D location, using an MLP that encodes the ray direction and camera-space coordinate. This requires evaluating the MLP for each observation of each point. Our approach, based on an occupancy volume, requires only performing a tri-linear interpolation for each observation, which is significantly cheaper. We train a variant based on this MLP conditioning, replacing the weighting by occupancy score. It increases the memory requirement by 4, since we use 4 observing views per point  $(|\mathcal{N}_k| = 4)$ . We thus need to halve the batch size and the number of height planes.
We train all variants, including our model, for an identical number of steps. To save compute resources, we train for fewer steps than in the main paper (200k vs 400k) and we use the smaller ResNet-50 architecture with only ground-level inputs. Tab. 2 shows that each of these variants yields a lower positioning accuracy than our model.
Vertical pooling: In our design, the vertical pooling is performed with max pooling. Tab. 3-top shows that average pooling is significantly less effective. We hypothesize that averaging makes it harder to ignore features of points located in empty space. We found that pooling with an attention mechanism [85] performs similarly as max pooling despite the increase in computation. Harley