SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding
Authors: Paul-Edouard Sarlin
Eduard Trulls
Marc Pollefeys
Jan Hosang
Simon Lynen
Venue: NeurIPS 2023 poster (2023)
Keywords: neural maps
visual positioning
semantic mapping
TL;DR: Learn semantic neural maps from self-supervision with state-of-the-art visual positioning

Abstract: Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain, especially in an automated fashion. Can we use  _raw imagery_ to automatically create _better maps_ that can be easily interpreted by both humans and machines? We introduce SNAP, a deep network that learns rich 2D _neural_ maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps.

SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding
Paul-Edouard Sarlin psarlin. com
Eduard Trulls
Mare Pollefeys
trulls@google. com
marc.pollef www.oeeeeeeeeehhha..o. ch
Jan Hosang? hosangggoogle. com
Simon Lynen
s slynen@google.cm com
ETH Zurich
2 Google Research
sensor data
self-supervised learning
downstream tasks
visual positioning
satellite
acrial
SNAP multi-view fusion
semantic mapping
street view
multi-modal neural map
learn neural 2D maps from multi-modal imagery using camera poses. SNAP outper- forms the state of the art in visual positioning, and by solving localization as proxy task learns easily interpretable, high-level semantics through self-supervision alone, without any semantic cues.
Figure I:
Abstract
Semantic 2D maps commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult create and maintain, especially in an automated fashion. Can we use raw imagery to automatically create better maps that can be easily interpreted by both humans and machines? We introduce SNAP, deep network that learns rich neural 2D maps from ground-level and overhead images. We train our model align neural maps estimated from dif- ferent inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps."
Introduction
Semantic 2D maps such as Google Maps are ubiquitous in our daily lives, used by billions of people. They offer compact, yet easily interpretable representations of the world from a bird's-ey s-eye view, allowing to effectively navigate large outdoor environments by foot or vehicle. By contrast,
"Work done during internship Google. *Code available github.com wtwc/ooo/////eeee comm/
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

posed inputs
multi-modal mapping
overhead encoder
partial map final map
aerial image
fuse
sequence
ground-level
encoder
query image
query encoder
query BEV
neural map alignment
Figure 2: Training architecture. We feed overhead and ground-level imagery to per-stream encoders
(Sec. 2.1) to produce 2D hird'seeye-view neural maps, fused via cell- aillwwise max-pooling (Sec. 2.2). We also extract a "query' neural map from a single ground-level image with the same ground-level encoder. Given known poses, we train SNAP by simply registering 'query" and 'scene' maps (Sec. 3).
machines position themselves in the real world through computer vision, which remains dominated by structure-base approaches [79, 83, 56, 74, 39, 65] relying on basic hand-crafted [54, 4] or learned [58, 111, 23, 69, 101, 75, 53] primitives, such as points or lines. These approaches build 3D maps with Structure-from Motion (SfM) and then localize query images via 2D-3D registration. Their complexity and many components (feature extraction and matching. bundle adjustment, pose refinement, etc.) make it difficult to tune [42] or update [25] them, and to learn high-level priors end-to-end [10, 8, 76.. They are also costly to store and generally not reusable for other applications.
Recent works such as OrienterNet [78] instead learn planar, neural representations from the same 2D semantic maps that humans use. These maps encode scene geometry and semantics and can be used for visual positioning with sub-meter accuracy. This approach is however limited to a few semantic classes, and the maps it is based on can be inaccurate, costly to obtain, and difficult to maintain.
We argue that maps are most useful for figuring out where we are when they are abstraci enough to be robust to temporal changes. yet preserve enough geometric and semantic information to yield high-quality correspondences with the physical world. Our work, SNAP. shows that, by learning 2D neural maps for localization, meaningful semantics emerge without explicitly supervising them. These semantics improve positioning accuracy and also make our maps usable for other tasks (Fig. 1).
SNAP leverages the complementary strengths of different input modalities, like ground-level and overhead imagery, by fusing them into a single 2D neural map (Fig. 2). It can flexibly and efficiently integrate arbitrary combinations of data captured at different points in time, which is key to continu- ously update maps in a changing world. We train it end-to-end to estimate the pose of a query image relative to the mapping images, by simply aligning their neural maps. This kind of contrastive learning requires only sensor poses, which can be casily obtained with photogrammetry [45, 37J. We train and evaluate SNAP on a dataset with 50M Street images from 5 continents, onders-of-mageen larger and more diverse than comparable academic benchmarks.
Despite training only for a positioning objective, we observe that our neural maps learn easily- interpretable, high-level semantics without the need for explicit semantic cues (Fig. 3), and demon- strate that they provide an effective pretraining for semantic understanding tasks by fine-tuning them on little labeled data. This can potentially unlock cost-efficient creation of more detailed and richer maps, readable by humans and machines alike, while providing state-of-the-art visual positioning.
Our main contributions are as follows. (i) We introduce simple and lightweight encoder to estimate bird's seyye-yye view maps from ground-level imagery, combining principles from multi-view geometry with strong monocular cues. (ii) We fuse different imaging modalities to integrate and benefit from complementary cues. (iii) We show how to train our model by aligning neural maps in a contrastive learning framework, using RANSAC to mine hard negatives. (iv) We outperform the state of the art on visual positioning and register image queries beyond the reach of traditional methods (Fig. 3). (v) We demonstrate that high-level semantics emerge by learning align neural maps, without any explicit supervision, and fine-tune them on semantic understanding with few labels (Fig. 6).
Analytical use of StreetView imagery was done with special permission from Google.

---

map images
query 4t-0.3m 0.30.3
a) facade with repeated structure and large viewpoint differences between query and map images (medium).
images
query At=0.7m 3R=3.6
b) Opposite views, localized using the ground and poles, observable in the neural map (hard).
map images
query
0t=0.6m AR-1.1
Opposite views road scene with linear structure, localized using polc and shrubbery (hard).
map images
guery
5t=0,5m 1R=1.4
d) Opposite views, localized using closc-up of pole and the markings the ground (hard).
Single-image localization. We show the 3-DoF poses of map images (red), the GT query pose (black), and the pose predicted by SNAP (green) with its error (At,AR), SNAP can estimate accurate poses even for extreme opposite-views. We visualize neural maps by projecting them to RGB using PCA. Notice how objects like trees, poles, curbs or road markings are clearly recognizable.
Figure 3:

---

2 Mapping the world with neural maps
We now formalize neural maps, and describe neural network architecture to infer them from raw sensor data. Our goal is to infer a more generic neural representation that can encode both the geometry, semantics, and appearance a given point in the 2D world.
For 3D scene, such as a large outdoor environment, we consider local, 3D Cartesian coordinate system such that the axis points upwards along the gravity direction. neural map M is defined over regular grid that partitions the zy plane into [xJ square cells of size A. Each cell (i, is associated with a D-dimensional feature M,j E TR. To infer such neural map, we leverage large quantities of raw imagery captured by diverse cameras.
Problem formulation:
Ground-level images captured by cameras mounted StreetView cars backpacks [18]. They are often part of a sequence of multi-camera frames. As such, they are very unevenly distributed throughout space. Each image offers high resolution view of small area, mainly limited by the occlusion of static or dynamic objects like buildings or vehicles. On the other hand, overhead images are captured by cameras mounted on planes or satellites. These images benefit from high spatial coverage at a uniform but low resolution. Their visibility is mostly affected by vertical occluders like trees. Ground-level and overhead images capture different aspects of the environment and are thus complementary.
Input modalities:
Assumptions: All images of either modality are calibrated and registered with respect to the map coordinate system. Each image n follows a projection function IT, R - R2 that maps a 3D point in the world to 2D point on the image plane. II, combines the camera pose w Tr E SE(3) and the camera calibration, including lens distortions. Overhead images are ortho-rectified, such that world points along the axis project onto the same pixel coordinate. As this process relies on coarse digital surface model [27], fine details like poles are not rectified and may result in artifacts, which SNAP can however learn to account for.
2.1 Fusing multi-modal representations
Each location in the world is observed by arbitrary number of images for each modality, captured at arbitrary points in time. We thus follow late-fusion strategy that first encodes each modality separately and only finally fuses them (Fig. 2). This can flexibly adapt to the available inputs and efficiently handle arbitrary spatial distributions of data.
Encoding: We design two encoders that each combine a subset of observations n into a single- modality neural map M defined over the same grid as M. 4ov encodes a single tile of overhead orthoimagery, while Øsv encodes single image or multiple covisible ground-level StreetView images, e.g., multi-view sequence. To best resolve the 3D information from perspective shots at arbitrary viewpoints, Pgy leverages both multi view observations and monocular cues. We describe its architecture detail in Sec. 2.2, &ov, on the other hand, is a simple U-Net-style CNN [71] that computes feature for each pixel of the overhead orthoimage, which is then resampled into the grid.
Fusion: We obtain the final neural map by fusing the set of encoded maps [M") using a cell-wise max-pooling operation, i.e., Mg max, M.; V (i.j) E IxJ. This can combine maps with different spatial extents, which is essential to scale large areas. The max aggregation picks the best
estimate among all inputs for each feature channel and thus handles partial observations, such as when the road surface cannot be resolved overhead images because it is occluded by trees.
2.2 Ground-leve image encoder
We design a single module, lsv. that can arbitrarily encode one multiple images, ordered or not. Øsv first fuses the image data into 3D space and later projects it vertically into the map plane (Fig. 4). This design can handle arbitrary ground geometries and accurately resolve the 2D location of overhanging 3D structures, like street lights. The 3D fusion leverages both multi-view geometry and strong monocular cues learned end-to-end. dsyv can thus resolve objects that observed by a single image, while maximizing accuracy when multiple observations are available.
Monocular inference: We consider an unordered set of N images (I"), N21. Each image is encoded independently by CNN Φ into C-dimensional feature image F" E Hx XC, & also estimates pixel-wise depth S e B X
score over D depth planes along the ray of each

---

3D
vertical pooling
occupancy volume
view #1
fusion MLP
view #2
CNN
image features
OCC. score
X k
point
map cell
column
S
K
F
neural cell Mij
Figure 4: Ground-level encoder: combining multi-view geometry and monocular priors. We use CNN to predict pixel-wise features and a monocular occupancy volume, separately for each view. We then interpolate them over column of points (at predefined heights), for each 2D cell. Finally, simple MLP combines them into features X that are pooled along the column, into neural cell.
pixel. S is similar frustum-aligned occupancy volume [78, 67] but contains unnormalized logits of a depth distribution. Instead of regressing a single value, this encodes the full depth uncertainty along the ray and thus allows @ to provide meaningful multi-modal estimates. We distribute the depth planes uniformly in log space to correlate with the uncertainty of monocular depth estimation [3, 78].
Multi-view fusion: To fuse information in 3D, define K horizontal planes at heights {za} J. which are uniformly distributed within a range of interest defined with respect to the height of the camera [85, 50], e.g., from 4 m below m above. For 2D map cell (i,j) Ix we consider its center point (a, y) and a column 3D points {Px k = (I. y, 2x. For each 3D point k. we define the subset of views that best observe it as Nr. {1. {1....N N.. e.g.. those that are closest spatially.
We project the point to each of these views, obtain a 2D observation p = IT, (Px), and sample the corresponding feature image with bi-linear interpolation: F F [PE]. Given the depth de of P in the corresponding view, we also tri-linearily interpolate a score from the depth prior: St S [p. dī]. Intuitively, St is low if the 3D point is in free space or occluded in view n. Following common practice in learned multi-view stereo [110, 106], we then compute feature consistency statistics, as mean and variance (Hy, σx) E F.,, weighted by the depth priors:
Σ n
W F
and OF Σ W (F 23) with wx softmax S nENL
neA
Multi-Layer Perceptron (MLP) fuses this information into a feature X which is finally pooled across all points in the column, resulting in neural map cell Myj:
Mj = max X with X MLP
ohe max S nEN:
(2)
Adding the maximum depth score differentiates free and occupied space when the point is observed by a single image. This makes possible use the same model for single images and sequences.
By tightly combining 3D geometry and representation learning, our approach leverages both monoc- ular priors and multi-view information, while past research on 2D mapping 3D reconstruction typically relies on only one of the two. Compared to expensive Transformers [102] or 3D CNNs [33], show that simpler, lightweight MLP is effective fusing multi-view information, inspired by [82]. Compared to top-down 2D CNNs that squash the vertical dimension [34, 70], this MLP is more expressive and makes our neural maps equivariant to 2D translations and rotations and invariant to translations along the vertical axis. Overall, this simple design enables scaling to very large scenes, which is critical to provide hard negatives for contrastive learning and ultimately learn rich semanties.
3 Learning from pose supervision
Alignment as contrastive learning: We want neural maps to encode high-level semantic information about the environment. Given recent advances self-supervised learning [14, 64], we hypothesize that this can emerge from learning distinctive features that distinguish one location from another and that are invariant to viewpoint and temporal appearance changes. Intuitively, good maps help us identify where are. More generally, good maps such that unambiguously align them
5

---

when inferred from partial inputs. Consider neural maps M and M obtained from two disjoint subsets of inputs, the query Q and the reference R. In camera pose estimation. Q corresponds to a single ground-level image and R to sequence of images with an aerial tile. Because our encoder is flexible, we can use the same shared model to encode Q and R (Fig. 2). M is defined over grid G E RIxxx2 in local coordinate frame, e.g,, aligned with the query camera, where G. is the center point of cell (i, while M is defined in the world frame.
We define a score function F M M SF(() + that evaluates the consistency between M and M given an estimate of their 3-DoF relative pose RTQ SE(2). To distinguish the ground-truth pose R RTQ from K other, incorrect poses {A we want to increase F and decrease E (omitting M and M for brevity). This corresponds to a contrastive learning problem, for which we minimize the InfoNCE loss [63]
exp (E (ATT) /T) F (nTT² /T)
Loss (M², .M)
loog
(3)
Σ Eke exp
where r is a learnable temperature parameter. Neural maps are trained end-to-end and require only relative poses n TT which can be easily obtained a large scale using photogrammetry [45, 37].
Featuremetric pose scoring: A linear layer projects each neural map M to a lover-dimensional, L2-normalized map M. This creates an information bottleneck that encourages compact features. The score E evaluates the consistency of two neural maps as the similarity of each cell after warping:
E(RTQ)
1 IJ
E (5JETX
max
M
QT M
R RTQ
Go .0)
where R TQ transforms grid point from coordinate frames Q to R and interpolates the map at this location. max clips negative scores to zero to reduce the impact of outliers, as in robust optimization.
Negative sampling: A critical and well-studied aspect of contrastive learning is the selection of negative samples [35, 98, 108]. Hard negatives should be high-likelihon but incorrect predictions,
so as to push the probability mass to the ground truth. Random poses can be easily distinguished and exhaustive voting in the 3-DoF pose space is computationally infeasible at high resolution [78,
6. 28]. Instead, we use RANSAC [29] to sample poses that are consistent with the predicted features. We sample pairs of 2D-2D correspondences between all cells of both neural maps and solve for the relative pose using the Kabsch algorithm [43]. Inspired by PROSAC [19], we sample a correspondence between cells (i,j) and (k based on its feature similarity with probability QT R. (M.) M /T). Unlike NG-RANSAC [10], gradients are propagated through
Poyki softmax ijki
the scoring rather than the sampling and are thus much smoother. Because the sampling and scoring mirror similar featuremetric errors, negative samples become harder as the learning proceeds.
Inference-time alignment: SNAP can estimate the unknown 3-DoF relative pose between any two neural maps. We estimate each map in the sensor coordinate frame, establish tentative correspon- dences by matching their cells, sample pose hypotheses, and select the pose with the highest score. This includes iingle-image positioning, where the query map 2 covers the camera frustum. The vertical pooling requires that the gravity direction is known, which is reasonable assumption for applications like Augmented Reality (AR) and robotics [56, 113, 78]. Our framework also applies more generally to aligning any pair of inputs, including couuence-tt-sequeee registration, which is required in the first place to pose mapping data in a common reference frame.
and perial-to-groume
4 Related work
Visual positioning is most commonly tackled with geometric approaches [80, 40, 74] that rely on
point correspondences across images and sparse 3D point clouds built with SfM [83, 2]. They then estimate the 6-DoF query pose with a robust solver [29, 20, 21, 19, 15, 5] from correspondences with the reference model or images. Such correspondences are most often estimated by sparse local features [54, 4]. This process complex and end-to-end propaaaaaat is impractical [8]. Past works have thus focused on learning specific components like feature extraction [111, 58, 23, 24,

---

99, 69. 26, 101, 105, 55], matching [112, 115, 75, 91, 41, 117, 53], and pose [104, 76] or point cloud refinement [52]. Coarse GPS location and gravity direction are commonly assumed to be known [1 13, 56, 93]. In AR and robotics, the height of the camera can be estimated as the distance to the ground in local SL.AM reconstruction [78]. These assumptions reduce the problem to 3-DoF estimation and make it more amenable to end-to-end learning. MapNet [38] also learns end-to-end
3-DoF visual mapping and localization but requires sequences of depth inputs. Recent works leverage overhead instead of ground-level images [87, 86, 109, 28]. They easily scale to large scenes but only in open-sky areas. Their accuracy is also limited by the low resolution of aerial imagery. Our work combines the strengths of both ground-level and overhead imagery by learning end-to-end how to best fuse them for 3-DoF positioning. Our differentiable pose estimation, based on RANSAC, is more efficient [38, 28, 78]. robust [87], and stable [10, 8] than previous approaches.
Semantic representations can largely benefit loop closure [84] and pose estimation [100]. Ori- enterNet [78] learns 3-DoF positioning end-to-end from public 2D semantic maps that are more compact yet detailed enough for localization. Its accuracy however limited because these maps have low spatial accuracy and are infrequently updated. It is also also restricted to few, explicit semantic classes that are often not discriminative. Differently, [49] learns finer-grained semantic classes for temporal and viewpoint consistency. Our work instead learns implicit semantics from posed imagery by combining end-to-end self-suue learning with large amounts of data. This boosts the positioning accuracy and is effective pre- mre-trainng for semantic tasks.
Neural scene representation is an active topic of research. MLPs [57, 96] and tokens [73] are compact but lack geometric inductive bias. 3D voxel grids are more expressive and thus popular for reconstruction [66, 60, 92, 9, 119], rendering [59, 95], and semantic perception [17, 102, 13, 7] but are expensive to store and thus often restricted to small scenes. 2D grids, or Sird's-Eye Views (BEV), are more compact and thus scale to larger outdoor scenes by compressing the information along the vertical axis. Neural BEVs can be learned from images for supervised semantic tasks [72, 50, 34, 67, 70J, 3D reconstruction [66], self-supervised view synthesis [85], and 3-DoF positioning [78, 28, 38). These approaches assume planar scenes or rely monocular priors only, even if multiple views are available. Instead, we combine these priors with multi-view fusion [110, 82, 106] to leverage information from image sequences and better resolve objects in large scenes.
Self-supervised learning leverages unlabeled datasets to learn representations useful for down-stream tasks. Many works focus on image- or pixel-level contrastive learning for semantic tasks [63, 35, 14, 62, 36.. View synthesis from few images typically learns lower-level representations [57, 85]. Some works [49, 89] learn features for image matching across appearance changes. CoCoNets [48] learns representations for 3D scenes but requires perfect, synthetic depth maps. We learn high-level contrastive scene representations from posed images and show that it translates to semantic mapping.
5
Experiments
Data: Street images are captured by rigs of 6 rolling-shuttec cameras mounted on cars or on backpacks worn by pedestrians [18], which results in a wide diversity of viewpoints in strect-level scenes. Multi-view 'frames' are captured synchronously every ~5m. Sequences are captured between 2017 and 2022. We build mapping segments only from car sequences by partitioning each sequence into groups of 36 images that face either the left or right side of the road. We define each map grid as 64x 16 m tile aligned with the segment mid-frame, in which we render an aerial orthophoto with 20 ground sample distance. Query images are sampled from different sequences, captured from cars or backpacks, based on their frustum overlap, and are often taken years apart. We train with 2.5M segments and ~50M queries from 11 citics across the world: Barcelona, London, Paris (Europe), New York. San Francisco (North America), Rio de Janeiro (South America), Manila, Singapore, Taipei. Tokyo (Asia), and Sydney (Oceania), reserving some areas in each city for validation. We test on 6 different cities (Amsterdam, Melbourne, Mexico City, Osaka, São Paulo, and Seattle), with 4k queries per city. This covers 5 continents, while academic localization benchmarks focus on tourism landmarks [42] single cities in Europe or the US [81, 1 116, 77] see details in Appendix D.
Training and implementation: In the ground-level encoder, <b is a U-Net [71] with a BiT ResNet backbone [46], pre-trained as in [114], and an FPN decoder [51], initialized randomly. We consider two models with different backbones: 'large' R152x2 (353M parameters) and a 'small' R50x1 (84M parameters). Φoy is similarly-defined 150xI+FPN, In multi-view fusion (Sec. 2.2) use
7

---

easy (25%)
medium (50%)
(25%)
75
Recail [%)
50
0
2
3
3
Position [meter]
75
Recall [66
50
0
2
10
8
0
Orientation error [degree]
SIM + SIFT
SFM SuperGlue
Semantic maps (OrienterNet)
Ours
Figure 5: Single-image positioning with different maps. Localizing with our neural maps yields higher recall than established approaches based on feature matching (SfM X), especially for hard queries with low visual overlap. Neural maps are also more suitable for positioning than semantic maps because they encode richer and thus more discriminative information.
Algorithm
Inputs
Easy (25%)
Med. (50%)
Hard (25%)
All (100%)
SfM + SIFT [54]
StreetView
47.0/ 54.4
24.9/ 29.9
7.6/ 9.7
27.1 / 32.1
+ SuperGlue [75]
StreetView
63.0 / 71.1
38.0 / 44.4
13.1 16.1
39.2 / 45.2
OrienterNet [78]
semantic
35.6/47.2
29.3 / 39.5
24.8/ / 34.8
30.0/ / 40.6
multi-modaa
/2.3
46.9/ 59.5
34.5 47.6
44.4 57.4
SNAP-large ResNet-152x2
StreetView
45.8/58.4
43.9 / 56.0
29.5
41.0/ 53.2
aerial
27.4/40.6
25.3 15.3/37.5
00.8/32.3
24.8 / 37.1
multi-modal
45.2/
41.9/ 54.8
29.6 42.0
39.9 52.9
SNAP-small
StreetView
54.9
38.1/50.1 50.1
24.5 36.4
36.0 / 48.2
ResNet-50
aerial
33.9/35.6
22.8
17.9 / 27.5
21.5 32.3
Table I: Single-image positioning, We report the area under the recall curve (AUC) up to thresholds (2.5 m/5") and (5 m/10"). Our large and small multi-modal models are accurate than classical SjM + X approaches for medium and hard queries, which matter most in practical applications. Fusing both View and aerial imagery is more accurate than using only one of them.
D=32 depth planes and K -60 height planes } uniformly distributed within 12 m. Neural maps M and matching maps M have dimensions 128 and 32, respectively, and are defined over 64x m grids with 20 cm ground sample distance. Query BEVs have maximum depth of 16m. At training time, neural maps are built from one aerial tile and one SV segment, with each of the two randomly dropped, similarly to dropout [90]. We use subset of 2 views, some of them at +-0 angle, which we empirically found provides good coverage/memory trade-off. See details in Appendix
Visual positioning: We build a map for each segment using all 36 views and evaluate the 3-DoF query pose terms of position and orientation eπrors. While many academic benchmarks use much larger mapping areas, we argue that GPS and motion priors often make this unnecessary for practical applications [56]. We slice the results by difficulty in terms of query-scene overlap based on the distance between the query and its closest map view, in position At, and orientation A6. We split in the data into groups: 'easy' 10t<10mm and A8<459 -25% of the data), 'hard' (At>
and
A0>600 22),) and 'medium' (the remaining ~50%). We compare our approach to hloc [74], state-of-the-arr [81, 77] structure-based 6-DoF localization system based on COLMAP [83], popular coctture-fro--Mooieee framework, with correspondences estimated by either RootSIFT [54, 4] or SuperPoint+-Sperriii [23, 75], a learned feature and matcher. Note that these approaches can only leverage ground-leve level imagery. We match the query to all map images, without using hloc's

---

huidiig
100
pre-trained
scretch
strect light
pole
tree
Recall [96]
50-
Distribution
10
road' sidewatk
building
crosswalk
tree
fence
traffic sign
pole
street light
traffic light
Figure 6: 2D Semantic mapping. Left: t-SNE visualization of the neural map features learned by SNAP, colored by their ground-truth semantic class. SNAP discovers different categories of objects common in outdoor urban scenes, which yields clearly distinguishable clusters. Middle: Given a small labeled dataset, training tiny CNN classifier to predict such classes from pre-trained features is more effective than training the entire SNAP model from scratch, especially for small and infrequent objects. Right: Test example with ground truth raster (middle) and prediction of the CNN (bottom).
retrieval component, and estimate the query's 's pose using RANSAC and a P2P solver with gravity constraint [94]. We evaluate the D-DoF pose projected to 3-DoF. We also evaluate OrienterNet [78], which matches query BEV with a semantic map. We re-implement and train on overhead semantic rasters derived from SfM points with semantic labels obtained by fusing 2D image segmentations. Note that OrienterNet was originally trained on OpenStreetMap, which has limited coverage of small objects. While our rasters are noisy, they provide a consistent, global coverage of fine- fine-grained classes like tree, streetlight, poles, etc. We checked our implementation with the authors. We also evaluate versions of SNAP trained with only either ground-level or aerial imagery - the latter is an extreme case of cross-view view localization, similar to [28].
Fig. 5 and Tab. show that SNAP outperforms the state of the art, COLMAP with Super- PointtSuperGlue, by a large margin: 25% relative. Structure-based approaches are more accurate for easy queries but significantly worse for hard ones (Fig. 3). Using ground-level imagery is crucial in most localization scenarios and performs ~46% relative better than using aerial imagery, whereas our multi-modal model performs ~8% relative better than the Street View--Vye variant. We justify our design decisions using an ablation study in Appendix A. We report detailed results per city and per sequence type in Appendix C. Our framework is also efficient, as mapping takes 223 ms per segment and 6 ms per aerial tile, estimating a query BEV takes 14 ms and localizing it takes 86 ms, on an A100 GPU. In comparison, matching with SuperGlue takes 1 100ms per pair for 36 pairs per query, and is thus 36 times slower. Each tile of our matching maps has size 1.6 MB in fp16, while storing SuperPoint descriptors requires 5.3 MB average.
Semantic mapping: We show that SNAP's neural maps are an effective pre-training for 2D semantic mapping. Existing approaches rely on ground truth 2D semantic rasters derived from the segmentation of LiDAR 3D point clouds. These are manually labeled, which too expensive to generate enough data to train from scratch models that generalize across countries, sensors, seasons, and times of the day. Existing datasets [12, 7] thus rarely span more than few cities and overfit supervised models to the local appearance. Instead, our self-supervised pre-training learns better features from a much larger dataset of posed imagery, which is much cheaper to acquire at scale. The information bottleneck forces SNAP to learn unified representations for objects, like street crossings or lights, that look very different across countries, and would require larger amounts of labeled data. Fig. 6-left shows a 2D t-SNE [103] visualization of SNAP's neural maps at points sampled on a few types of objects common in street scenes, according to their ground-truth semantic label. Points of the same class are clustered together. This clearly shows that neural maps learn to distinguish these objects without any semantic supervision, even if they are geometrically similar, e.g., tree vs pole.
To evaluate the pre-training, we train a tiny CNN to predict semantic rasters from pre-trained neural maps, keeping SNAP frozen. We compare this to training the entire model from scratch (with the same backbones initialization [46, 114]). We derive 3k 64x 16 m ground truth rasters from LiDAR point clouds captured by StreetView cars in 84 cities across the world. We train with 2k examples and report the recall of both approaches on Ik test examples in Fig. 6-center. Pre-training consistently

---

London
Paris
Rio de Janeiro
Manhattan
Taipei
Tokyo
(a) image
(b) expected log. -eepphhhhh
(c) confidence
(d) BEV
Figure 7: Monocular depth priors learned by the ground-level encoder. For each query image (a), we show: (b) the expected log-depth across all depth planes, from blue (close) to red (distant); (c) the total score along each ray log Cie(I...) exp S [;, di]. which reflects how useful or confident the prediction is; (d) the resulting bird s-eye view. The predictions ground and for lower parts objects and buildings. Predictions in the sky and upper facades are not reliable because these areas never covered by the height planes {2 of the point columns.
sensible for areas close to the
yields better results for every class, with larger gains on more difficult/infrequent classes. While training from scratch massively overfits to such small dataset, our neural maps encode enough information to reach recalls over 70%. We show qualitative examples in Fig. 6-right and Appendix B.
Monocular priors: We visualize in Fig. 7 the occupancy predicted by SNAP depth and confidence maps. SNAP learns sensible priors over the geometry of street scenes from only pose supervision.
Limitations: Our approach is not accurate as structure-based based methods given easy queries closer to map images (Fig. 5). We hypothesize this partly due to operating at lower image resolutions. also assumes gravity direction and location prior, which reasonable assumptions but restrict its use.
6 Conclusion
We present SNAP, a novel approach to build semantic, 2D neural maps from multi-modal inputs and train it by simply learning to align two neural maps in contrastive framework. This simple objective yields a model that can localize queries beyond the reach of the state the art in structure- based matching by discovering high-level semantics from self-supervision. Our neural maps easily interpretable and provide an effective pre-training towards unlocking semantic understanding at scale.
This work has implications to privacy and surveillance. However, our 2D maps too compact to preserve personal identifiable information, and likely more difficult to invert than point clouds [68], which despite ongoing efforts [88, 31, 32, 61, 118] remain susceptible to attacks [16].
Broader impact:
10

---

Acknowledgements: We thank Bernhard Zeisl, Songyou Peng, Rémi Pautrat, and Michal Tyszkiewicz for their valuable feedback, Manuel Cabral and Johann Volz for providing useful code reviews, Tianqi Fan for helping processing the data, Lucas Beyer for providing the pre-trained ResNet backbones, Thomas Funkhouser and Kyle Genova for providing the ground truth semantic labels, and Arjun Karpur for helping run SuperGiue.
References
[1] Ammar Abbas and Andrew Zisserman. A Geometric Approach Obtain Bird's Eye View From an Image. ICCV workshop, 2019. 17, 18
[2] Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle Adjustment in the Large. ECCV, 2010.
[3] Manuel López Antequera, Pau Gargallo, Markus Hofinger, Samuel Rota Bulò, Yubin Kuang, and Peter Kontschieder. Mapillary Planet-Scale Depth Dataset. ECCV, 2020. 5
[4] Relja Arandjelović and Andrew Zisserman. Three things everyone should know to improve object retrieval. CVPR, 2012. 6,
[5] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiff Matas. MAGSAC++, a Fast, Reliable and Accurate Robust Estimator. CVPR, 2020. 6
[6] Ioan Andrei Barsan, Shenlong Wang. Andrei Pokrovsky, and Raquel Urtasun. Learning to Localize Using a LiDAR Intensity Map. CoRL,, 2018.
[7] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. ICCV, 2019. 7.9
[8] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task. CVPR, 2020. 2,
6,7 7
[9] Aljaž Božič, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. TransformerFu- sion: Monocular RGB Scene Reconstruction using Transformers. NeurIPS, 2021. 7
[10] Eric Brachmann and Carsten Rother. Neural-Guider RANSAC: Learning Where to Sample
Model Hypotheses. ICCV, 2019. 2, 6,
[11] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. http://g /ttp:///gthub.. thub. com/ hum//oogle//aan 2018. 27
[12] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. CVPR, 2020. 9
[13] Anh-Quan Cao and Raoul de Charette. MonoScene: Monocular 3D Semantic Scene Comple- tion. CVPR, 2022. 7
[14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV, 2021. 7.
[15] Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, and Mare Pollefeys. AdaLAM: Revisiting Handcrafted Outlier Detection. ECCV, 2020. 6
[16] Kunal Chelani, Fredrik Kahl, and Torsten Sattler. How Privacy-Preserving are Line Clouds?
Recovering Scene Details from 3D Lines. CVPR, 2021. 10
[17] Rieson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry-Awan Recurrent Neural Networks for Active Visual Recognition. NeurIPS, 2018. 7
[18] Danny Cheung. Mapping stories with new Street View Trekker. https://blog /ttp:://blooggooolee/ hatpticts//mmm///mpppingg--eee
view- trekker/.. 2018. 4,
[19] Ondfej Chum and Jifí Matas. Matching with PROSAC Progressive Sample Consensus.
CVPR, 2005. 6

---

[20] Ondiej Chum, Jifí Matas, and Josef Kittler. Locally Optimized RANSAC. GCPR DAGM Symposium, 2003.
[21] Ondřej Chum, Tomas Werner, and Jifí Matas. Two-view geometry estimation unaffected by dominant plane. CVPR, 2005. 6
[22] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenie:
A JAX Library for Computer Vision Research and Beyond. CVPR, 2022. 27
[23] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Supe-Suue
Interest Point Detection and Description. CVPR workshop, 2018. 2, 6,
[24] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Mare Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-Net: A Trainable CNN for Joint Detection and Description of Local Features. CVPR, 2019. 6
[25] Mihai Dusmanu, Ondrej Miksik, Johannes L Schönberger, and Mare Pollefeys. Cross- Descriptor Visual Localization and Mapping. ICCV, 2021.
[26] Patrick Ebel, Anastasiia Mishchuk, Kwang Moo Yi, Pascal Fua, and Eduard Trulls. Beyond Cartesian Representations for Local Descriptors. 2019, 7
[27] Esri. Introduction to Ortho Mapping AreGIS Pro documentatiion https://pro aregis. com/en/pzz htm, 2023. 4
[28] Florian Fervers, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelhagen. Uncertainty-aware Vision-based Metric Cross-view Geolocalization. CVPR, 2023, 6, 7.9
[29] Martin A Fischler and Robert C Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM, 1981. 6
[30]
Kyle Genova, Xiaoqi Yin, Abhijit Kundu, Caroline Pantofaru, Forrester Cole, Avneesh Sud, Brian Brewington, Brian Shucker, and Thomas Funkhouser. Learning 3D Semantic Segmenta- tion with only 2D Image Supervision. 3DV, 2021. 27
[31] Marcel Geppert, Viktor Larsson, Pablo Speciale, Johannes I. Schönberger, and Mare Pollefeys.
Privacy Preserving Structure-from-Mootiee ECCV, 2020. 10
[32] Marcel Geppert, Viktor Larsson, Johannes Schönberger, and Marc Pollefeys. Privacy Preserving Partial Localization. CVPR, 2022. 10
[33] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR, 2018.
[34] Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. Simple- BEV: What Really Matters for Multi-Sensor BEV Perception? ICRA, 2023. 5, 7, 17, 18
[35] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. CVPR, 2020. 6, 7
[36] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR, 2022. 7
[37] Jared Heinly, Johannes L. Schonberger, Enrique Dunn, and Jan-Michael Frahm. Reconstructing the World* in Six Days * *(As Captured by the Yahoo 100 Million Image Dataset). CVPR, 2015.
2, 6
[38] Joao F Henriques and Andrea Vedaldi. MapNet: An Allocentric Spatial Memory for Mapping Environments. CVPR, 2018.
[39] Martin Humenberger, Yohann Cabon, Nicolas Guerin, Julien Morat, Vincent Leroy, Jérôme Revaud, Philippe Rerole, Noé Pion, Cesar de Souza, and Gabriela Csurka. Robust Image Retrieval-based Visual Localization using Kapture. arXiv preprint, 2020. 2
[40] Arnold Irschara, Christopher Zach, Jan-Michael Frahm, and Horst Bischof. From Structure- from-Motion Point Clouds to Fast Location Recognition. CVPR, 2009.
12

---

[41] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: Correspondence Transformer for Matching Across Images. ICCV, 2021. 7
[42] Yuhe Jin, Dmytro Mishkin, Anastasiaa Mishchuk, Jifí Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. IJCV, 2021.
2, 7
[43] Wolfgang Kabsch. solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 1976. 6
[44] Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint, 2014. 27
[45] Bryan Klingner, David Martin, and James Rosehorough. Street View Sision-from-Stuucuuu from-Motion. ICCV, 2013. 2, 6
[46] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. ECCV, 2020. 7,9
[47] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Carbon Emissions of Machine Learning. NeurlPS workshop on Climate Change AI, 2019.
27
[48] Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. CoCoNets: Continuous Contrastive 3D Scene Representations. CVPR, 2021. 7
[49] Mâns Larsson, Erik Stenborg, Carl Toft, Lars Hammarstrand, Torsten Sattler, and Fredrik Kahl. Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization. ICCV, 2019.
[50] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning s-Fye-Viers Representation From Multi-Camera Images Via Spatiotemporal Transformers. ECCV, 2022. 5,7
[51] Ysung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Be- longie. Feature Pyramid Networks for Object Detection. CVPR, 2017. 7
[52] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-Perfect Structure-from-MMotion with Featuremetric Refinement. ICCV, 2021.
[53] Philipp Lindenberger, Paul-Edouard Sarlin, and Mare Pollefeys. LightGlue: Local Feature Matching at Light Speed. ICCV, 2023. 2,7
[54] David G Lowe. Distinctive Image Features from Scale-Invarian Keypoints. IJCV, 2004. 2, 6,
[55] Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. ASLFeat: Learning Local Features of Accurate Shape and Localization. CVPR, 2020.
[56] Simon Lynen, Bernhard Zeisl, Dror Aiger, Michael Bosse, Joel Hesch, Mare Pollefeys, Roland Siegwart, and Torsten Sattler. Large-scale, real-time visual-inertial localization revisited. IJRR, 2020. 2, 7,8
[57] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020. 7
[58] Anastasia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiří Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. NeurIPS, 2017. 2, 6
[59] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph., 2022.
[60] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-End 3D Scene Reconstruction from Posed Images. ECCV, 2020. 7
[61] Tony Ng, Hyo Jin Kim, Vincent Lee, Daniel DeTone, Tsun-Yi Yi Yang, Tianwei Shen, Eddy Ilg. Vassileios Balntas, Krystian Mikolajczyk, and Chris Sweeney, NinjaDesc: Content-Concealing
13

---

Visual Descriptors via Adversarial Learning, CVPR, 2022. 10
[62] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C
Courville. Unsupervised Learning of Dense Visual Representations. NewrIPS, 2020. 7
[63] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, arXiv preprint, 2018. 6,
[64] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. DINOv2:
Learning Robust Visual Features without Supervision. arXiv preprint, 2023. 5
[65] Shuxue Peng, Zihang Hc, Haotian Zhang, Ran Yan, Chuting Wang, Qingtian Zhu, and Xiao Liu. MegLocc Robust and Accurate Visual Localization Pipeline. arXiv preprint, 2021.
[66] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger.
Convolutional Occupancy Networks. (CCV,20200 7
[67] Jonah Philion and Sanja Fidler. Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV, 2020. 5,7
[68] Francesco Pittaluga, Sanjeev Koppal, Sing Bing Kang, and Sudipta N Sinha. Revealing Scenes by Inverting Structure from Motion Reconstructions. CVPR, 2019. 10
[69] Jerome Revaud, Philippe Weinzaepfel, César Roberto de Souza, and Martin Humenberger.
R2D2: Repeatable and Reliable Detector and Descriptor. NeurlPS, 2019. 2,7
[70] Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic Feature Transform for Monocular 3D Object Detection. BMVC, 2019. 7,
[71] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. MICCAI, 2015. 4,7
[72] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden. Translating Images into
Maps. ICRA, 2022. 7
[73] Mehdi S. M. Sajjadi. Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucie, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Let-n Scene Representations. CVPR, 2022. 7
[74] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From Coarse to Fine: Robust Hierarchical Localization at Large Scale. CVPR, 2019. 2, 6,
[75] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Su- perGlue: Learning Feature Matching with Graph Neural Networks. CVPR, 2020. 2, 7, 8
[76] Paul-Edouard Sarlin, Ajaykumar Unagar, Mâns Larsson, Hugo Germain, Carl Toft, Victor Larsson, Mare Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose. CVPR, 2021. 2.7
[77] Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L. Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Mare Pollefeys. LaMAR: Benchmarking Localization and Mapping for Augmented Reality. ECCV, 2022. 7,8
[78] Paul-Edouard Sarlin, Daniel DeTone, Tsun-Yi Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulo, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. OrienterNet: Visual Localization in 2D Public Maps with Neural Matching. CVPR, 2023. 2, 5, 6. 7, 8, 27
[79] Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Fast Based Localization using Direct 2D-to-3D Matching. ICCV, 201
[80] Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Improving Image-Based Localization by Active Correspondence Search. ECCV, 2012. 6
[81] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, Fredrik Kahl, and Tomas Pajdla. Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions. CVPR,
14

---

2018. 7,8
[82] Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. SimpleRecon: 3D reconstruction without 3D convolutions. ECCV, 2022. 5.7
[83] Johannes Schönberger and Lun-Michael Frahm. Biructurefrom--Molie Revisited. CVPR.
2016. 2, 6,
[84] Johannes L Schönberger, Mare Pollefeys, Andreas Geiger, and Torsten Sattler. Semantic Visual Localization. CVPR, 2018.
[85] Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand. Joshua B Tenenbaum, and Vincent Sitzmann. Sceing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement. NeurIPS,
2022. 5, 7, 17, 18
[86] Yujiao Shi and Hongdong Li. Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image. CVPR, 2022. 7, 17, 18
[87] Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am looking at? Joint Location and Orientation Estimation by Cross-View Matching. CVPR, 2020. 7
[88] Pablo Speciale, Johannes L Schonberger, Sing Bing Kang, Sudipta N Sinha, and Marc Pollefeys. Privacy Preserving Image-Based Localization. CVPR, 2019. 10
[89] Jaime Spencer, Richard Bowden, and Simon Hadfield. Same Features, Different Day: Weakly Supervised Feature Learning for Seasonal Invariance. CVPR, 2020. 7
[90] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way Prevent Neural Networks from Overfitting. JMLR. 2014. 8
[91]
Jiaming Sun, Zehong Shen, Yuang Wang. Hujum Bao, and Xiaowei Zhou. LoFTR: Detector-
Free Local Feature Matching with Transformers. CVPR, 2021. 7
[92]
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujum Bao. NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video. CVPR, 2021.
[93] Linus Svärm, Olof Enqvist, Fredrik Kahl, and Magnus Oskarsson. City-Scale Localization for Cameras with Known Vertical Direction. TPAMI, 2017. 7
[94] Chris Sweeney, John Flynn, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. Efficient Computation of Absolute Pose for Gravity-Awar Augmented Reality. ISMAR, 2015.
[95] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes. CVPR, 2021. 7
[96] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-NeRF: Scalable Large Scene
Neural View Synthesis. CVPR, 2022. 7
[97] TensorFlow Datasets. TensorFlow Datasets, collection of ready-to-uss datasets. https: h/www. www..tensorflo... wwwpsooowoooo.og//ttaaeeeeeeeeee dataetts,e 2019. 27
[98] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.
What Makes for Good Views for Contrastive Learning? NeurIPS, 2020. 6
[99] Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Baintas. SOSNet: Second Order Similarity Regularization for Local Descriptor Learning. CVPR, 2019.
[100] Carl Toft, Erik Stenborg, Lars Hammarstrand, Lucas Brynte, Marc Pollefeys, Torsten Sattler, and Fredrik Kahl. Semantic Match Consistency for Long-Term Visual Localization. ECCV, 2018. 7
[101] Michat Tyszkiewicz, Paseal Fua, and Eduard Trulls. DISK: Learning local features with policy gradient. NeurlPS, 2020. 2,
[102] Michat J Tysskiewicz, Kevis-Koking Maninis, Stefan Popov, and Vittorio Ferrari. RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. ECCV, 2022. 5,
15

---

[103] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 2008.
[104]
Lukas Von Stumberg, Patrick Wenzel, Qadeer Khan, and Daniel Cremers. GN-Net: The Gauss-Newton loss for
relocalization. RA-L, 2020. 7
[105] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning Feature
Descriptors using Camera Pose Supervision. ECCV, 2020.
[106] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla. Noah Snavely, and Thomas Funkhouser. IBRNet: Learning
Multi-View Image- Based Rendering. CVPR, 2021. 5, 7, 17, 18
[107] Qing Wang. Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. MatchFormer: Interleaving Attention Transformers for Feature Matching. ACCV, 2022.
[108] Chao- Yuan Wu, R Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling Matters in Deep Embedding Learning. ICCV, 2017. 6
[109] Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual Cross- View Metric Localization with Dense Uncertainty Estimates. ECCV, 2022.
[110] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo. ECCV, 2018. 5, 7, 17, 18
[111] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned Invariant Feature Transform. ECCV, 2016.
[112] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to Find Good Correspondences. CVPR, 2018.
[113] Bernhard Zeisl, Torsten Sattler, and Mare Pollefeys. Camera Pose Voting for Large-Scale Image-Based Localization. ICCV, 2015. 6,
[114] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Trans- formers. CVPR, 2022. 7.9
[115] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning Two-View Correspondences and Geometry Using Order-Aware Network. ICCV, 2019.
[116] Zichao Zhang, Torsten Sattler, and Davide Scaramuzza. Reference Pose Generation for Long-term Visual Localization Learned Features and View Synthesis. IJV, 2021. 7
[117] Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progres- sive Correspondence Pruning by Consensus Learning. ICCV, 2021. 7
[118] Qunjie Zhou, Sérgio Agostinho, Aljoša OSep, and Laura Leal-Taixe Is Geometry Enough for Matching Visual Localization? ECCV, 2022. 10
[119] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Mare Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM.
CVPR, 2022.1
16

---

Appendix
A Design decisions
In this section, we explain our design decisions and support them with an ablation study.
Learning features that are discriminative requires sufficiently challenging negative pose samples. These arise from viewpoints that are visually similar to the ground truth viewpoint, for example due repeated patterns, lack of distinctive features, occlusions. In order to find sufficiently difficult negative samples in a training example, the map should be as large as possible. The size of the map and the number of mapping images are however limited by the amount of GPU memory available. We therefore found it critical to find the right balance between the size of the map, its spatial resolution, the number of views, and the batch size, as also reported in [34].
Constraints:
We found that a simple and lightweight model that saves memory is beneficial over a complex model that requires reducing the size of the map, the number of views, the batch size. We thus favor explicit constraints by geometry (camera projection, 3D occupancy) rather than flexible but heavy mechanisms like attention or 3D convolutions. The effectiveness of our design shows that such complexity is not required. This enables the training of high-capacity models with maps of 64 16 m, resolution of 20 cm (~25k cells per map), from 20 images, and within 20 GB of GPU memory per example. Despite extensive memory optimizations, including the use of gradient checkpointing and preccisoe training, using larger maps for training remains challenging.
Ground-level encoder: We consider four alternative designs for the ground-level encoder Øsv.
(a) Fixed-height plane. Instead of lifting the image information 3D and pooling it vertically, the geometry of the scene can be approximated from a ground plane [1]. Driving applications commonly assume that this plane is horizontal and fixed height below the camera [86]. We train a variant that aggregates the multi-view information on plane 2.5 m below the camera. This approach can hardly resolve the spatial location of overhanging structures, like street lights. It furthermore introduces distortions in the BEV if the image is not gravity aligned, which is the case for many capture platforms (such as Stre backpacks or consumer phones), or the ground is not planar, as in many real environments.
(b) No monocular occupancy. Multi-view stereo [106, 110] typically does not explicitly leverage monocular geometry priors. We thus train a variant that omits the weighting by occupancy scores. Image features are thus painted identically along all depth planes for each ray and their mean and variance across all views is fed to the fusion MLP. This model can only leverage the bearing angles of point features (like poles) to disambiguate pose but cannot resolve the location of line features.
(c) No multi-view variance. SimpleBEV [34] simply averages feature volumes obtained by painting image features along each ray. This corresponds to removing both variance and monocular priors from SNAP, making even harder to resolve surfaces.
(d) Ray conditioning. Close to our approach, Sharma al. [85] augment multi-view fusion with monocular cues, but do so by conditioning each ray feature by its 3D location, using an MLP that encodes the ray direction and camera-space coordinate. This requires evaluating the MLP for each observation of each point. Our approach, based on an occupancy volume, requires only performing a tri- tri-lincar interpolation for each observation, which is significantly cheaper. We train a variant based on this MIP conditioning, replacing the weighting by occupancy score. It increases the memory requirement by 4, since we use observing views per point ([N4/=4). We thus need to halve the batch size and the number of height planes.
We train all variants, including our model, for an identical number of steps. To save compute resources, we train for fewer steps than in the main paper (200k vs 400k) and we use the smaller ResNet-50 architecture with only ground-level inputs. Tab. 2 shows that each of these variants yields lower positioning accuracy than our model.
Vertical pooling: our design, the vertical pooling is performed with max pooling. Tab. 3-top shows that average pooling is significantly less effective. We hypothesize that averaging makes it harder to ignore features points located in empty space. We found that pooling with an attention mechanism [85] performs similarly as max pooling despite the increase in computation. Harley
17

---

Variant
Easy (25%)
Med. (50%)
Hard (25%)
All (100%)
SNAP SStreetViewoonly
39.8 51.8
36.5 47.2
22.3 32.3
34.0/ / 44.9
Fixed-heigh plane (a)
34.0 / 45,4
29.6 39.8
28.0
28.2/38.6
No monocular occupancy (b)
29.8 / 41.7
26.1/37.3
/6.1/25.7
24.7 35.8
No multi-view variance (c)
28.8 39.1
23.5 32.6
14.0/21.7
22.7 /3.8
Ray conditioning (d)
11.9 22.5
8.7 / 17.4
4.5 10.1
8.6 17.1
Table Ablation study of the ground-level encoder. We report the single-image positioning AUC up to thresholds (2.5 m/5%) and (5 m/10%. Variant (a) aggregates the information on an horizontal plane fixed height below the camera [1, 86]. This is insufficient for overhanging objects, when ground footprints are occluded, or when the scene is not planar. Variant (b) performs multi-view fusion without monocular priors [106, 110]. This makes it impossible resolve the depth of objects in the single-image query. Variant (c) further drops the variance term and simply averages feature volumes [34]. This makes it harder to resolve surfaces. Variant (d) replaces the occupancy volume by conditioning each observation feature on the ray and distance using an MLP [85]. This is much more expensive and thus constrains both batch size and scene size, which in turn lowers performance.
Component
Operator
Easy (25%)
Med. (50%)
Hard (25%)
All (100%)
Vertical pooling SStreetView-ong
max
39.8 51.8
36.5 47.2
22.3 32.3
34.0/ 44.9
average
32.4 /3.6
29.1 / 39.3
17.8 / 27.2
27.3 / 37.6
Multi-modal fusion
max
55.6
41.5 53.0
27.3/30 37.9
39.3 51.0
average
45.5
40.9 / 52.6
27.8 / 38.8
39.0 / 50.9
Table 3: Ablation study pooling operators. Top: In the ground-level encoder, pooling the features vertically with the max operator performs much better than averaging, as measured by single-image positioning AUC. Bottom: Fusing StreetView and aerial neural maps with either max or average pooling performs comparably.
et al. [34] flatten the vertical elements with a space-to-depth (or pixel-shuffling) operation, followed by MLP. This makes the model sensitive to a translation along the vertical axis, which rarely occurs in small driving datasets based on a few cars with identical specifications, but matters for heterogeneous data captured by backpacks and cars with widely different setups. We thus found that this approach yields a lower performance than a simpler pooling. This also makes it impossible to adjust the number of height planes at inference time.
Multi-modal fusion: Tab. 3-bottom shows that the choice of pooling operator makes little difference when fusing neural maps inferred from Street View and acrial inputs.
B Qualitative examples
Visual positioning: Due to space constraints, we showcase only a handful of visual positioning examples in the main paper. We show additional visualizations of successfully localized queries in Figs. 8 to 10 and failure cases in Figs. 17 and 18. We also show some examples with non-linear mapping trajectories in Fig. 19, which our method is robust to. As in Fig. 3, show the 3-DoF poses of map images (red), the GT query pose (black), and the pose predicted by SNAP (green) with its error (At, AR). We visualize neural maps by projecting them RGB using PCA.
Sequence to sequence alignment In the paper we focus on the alignment of single-image queries to maps built from multiple views, but SNAP can arbitrarily align any pair of neural maps. We show two examples of sequence to sequence alignment in Figs. 11 and 12. These maps are built from five views. For ease of understanding, we lift our neural maps into 3D using LiDAR, color-coding each LiDAR point with the neural map values of the cell it belongs to - note that this is stricly for visualization purposes, and our algorithm does rely LiDAR.
Semantic mapping: We show additional visualizations of the semantic maps predicted by our model after fine-tuning it in only 2k scenes in Figs. 14 and 15.
18

---

images
query 4t= AR=0.1
NOC
a) The query is localized within 0.6 from different road (medium).
images
query At=0.7m AR=1.3*
b) The query is localized within 0.7 despite significant occlusions in the reference views (medium).
images
query At-0.2m 4R- 2.1
c) The tree clearly visible both the query and reference neural maps (hard).
map images
query 4t-0.3m 4R-1.2
d) The query is correctly localized despite drastic perspective changes and reflections in the window (medium).
Figure Single-image localization (1/3). We show successful examples. See legend in Fig.
19

---

map images
query At=0.5m 0.5m 2R=2.5
The query is localized across opposite views (hard).
images
query At= 1m 0R-
b) The query is on the other side of the street, far away from the reference views (medium).
map Images
query 2t-0.2m AR-1.1
c) Road markings are the only elements in common across opposite views (medium).
lmages
query At=0 .2 0R=
d) The query is correctly localized across a different street and despite drastic perspective changes (hard).
Figure 9: Single-image localization (2/3). We show successful examples. See legend in Fig. 3.
20

---

map images
query At=0.2m AR=0.2*
90
a) Notice the salient spot corresponding to the utility pole in the neural maps (medium).
map images
query At=0.6m AR=1188
b) Notice that the building facade closest to the query not visible from the reference views (hard).
map images
query 1t-1.6m 2
c) The query is localized within 1.6 from side street (hard).
map images
query 0..0.3m 1R-1.4
13
d) The query is localized from a difficult angle (medium).
Figure 10: Single-image localization (3/3). We show successful examples. See legend in Fig. 3.
21

---

Alignment error: A/=0.5m, 2R=0.33
Figure 11: Sequence-to-sequeecce alignment (1/2). We align a car sequence (red) to backpack sequence (blue) across opposite views. Notice how the neural maps (top right) clearly similar despite the large viewpoint difference, and can be easily aligned (bottom right). In the bottom left lift the neural map for the first sequence into 3D by coloring LiDAR point cloud with the RGB value of the grid cell it belongs to - note that this is strictly for visualization purposes, and our method does rely LiDAR. Notice how the semantics learned by model correspond real scene features, such as the pole (in cyan, enclosed in black box).
Large-scale mapping: We can easily build large tiles by 'stitching" smaller neural maps together via cell-wise max-pooling, similarly to how we fuse aerial and ground-level neural maps. Starting from multiple sequences posed in a common reference frame, Fig. 13 shows how neural maps are inferened for each of them and finally combined together.
C Detailed results
Fig. 21 shows the single-image positioning recall performance for each for the test cities. The performance of the different approaches is fairly consistent across cities, with the exception of Osaka. We observe that Osaka has large number of small, narrow streets, and StreetView sequences often feature close-ups of building facades: see Fig. 18-(c-d), This is somewhat out of distribution with the training set and could be solved by training on more data. also explains why structured-bast matching approaches either very accurate (when there is sufficient visual overlap between query and reference views) or fail to register the image (when there isn't t)).
Fig. 22 shows the recall for queries taken by cameras mounted on either backpacks or cars. Backpack queries have lower recall because they are typically taken from sidewalk and thus have larger viewpoint differences (Fig. 16).
22

---

125
5.0
0.0
-10
Alignment error: At=0.3m, AR=0.05
Figure 12: Sequence-to-ssequene alignment (2/2). We align two car scenes, the second of which is heavily occluded due to passing bus. Notice how the neural maps (top right) are visually similar and can easily aligned. The neural map for the first sequence clearly shows row of poles, which are occluded in the second sequence (black box). Our method is robust and can align the sequences using road markings and building boundaries. On the top right we plot an alignment heatmap by running exhaustive aligment in 3-DoF between both neural maps (we max-pool over rotations, for case of understanding): notice the clear maximum (circled), with smaller maxima along the road. In the bottom left lift the neural map for the first sequence into 3D by coloring LiDAR point cloud with the RGB value of the grid cell it belongs to note that this strictly for visualization purposes, and our method does not rely LiDAR.
inputs
acrial
2021
neural map tile
fuse
2019
2022
Figure 13:
Building neural tiles. We combine aerial view and car sequences captured over multiple years into a single neural map that spans a large area. An arbitrary number inputs can be combined in such way.
23

---

Greand truhh
a) The curb is mostly occluded by the parked cars and has very sparse ground truth labels. Notice how neural maps encode the location of street lights that hanging above the street.
Grardtruhh
tema
b) Because of the bike lane and confusing markings at the intersection, the model incorrectly segments the crosswalk.
Semantic mapping (1/2). 2D segmentation predicted from pre-trained neural maps by a small CNN trained in supervised fashion with GT semantic labels of 2k scenes. We also show 3D lidar point clouds colored by the predicted segmentation (lidar not used as input).
Figure 14:

---

Grouddonn bfth
a) The Vire car turns around the corner, so the right side of the map is behind the cameras and thus very sparsely observed. Areas with good visual coverage are much better segmented.
Prediction
b) The model roughly segments road, sidewalk and buildings, and correctly resolves small objects such as the poles or the overhanging traffic light. gets confused around the sidewalk, which has multiple striped patterns that are also difficult interpret by the human eye.
Figure 15: Semantic mapping (2/2). 2D segmentation predicted from pre-trained neural maps by small CNN trained in supervised fashion with GT semantic labels of 2k scenes. We also show 3D lidar point clouds colored by the predicted segmentation (lidar not used as input).
25

---

query platform the splits (%)
Time difference between query and mapping data
backpack
Number of test examples
10a
40
20
10a
easy
medium
hard
all
1 day
1 month
1 year
years
Figure 16: Large viewpoints and temporal differences. Left: The easy split contains more car- mounted queries, and the hard split contains more backpack queries. This make sense, as examples captured from backpacks are typically captured from sidewalks instead of the road, which results in difficult localization scenarios across opposite views. Right: The time difference between query and mapping images spans from few days to few years. This yields challenging localization scenarios with large appearance and even structural changes, egg,, due to construction work.
D Dataset
We provide more details about the data used for the experiments conducted in the paper.
Platform type: Ground-level mapping sequences are selected from sequences captured by cars only. Query images sampled from sequences captured by either or backpacks. For each map tile, we sample as many car queries as backpack queries, when possible. As cities are only partially covered by backpack sequences, the final ratio of backpack queries varies per city, from 20% 40%. Fig. 16-left shows the ratio of both types for the different difficulty splits of the test set.
Map grid: We found that maps of size 64x 16 provide good trade-off between visual diversity and memory consumption. Since multi-view frames are provided every ~5 m, segments of 14 frames generally cover length of 64 m. We therefore sample segments of 14 frames and define the map grid to be aligned with the average pose of the 7th and 8th side cameras. Given that most car segments rectilinear, the map is most often aligned with the mapping views. In turns, the coverage of the map varies and can make the localization more challenging, which is generally a good property for both training and evaluation. Fig. 19 shows examples of maps with partial coverage.
For each of 11 training cities, we define disjoint training and validation regions. We show these regions in Fig. 20. We sample at most 5M training examples from each training region. The validation region corresponds
Spatial distribution:
level-14 4 S2 cells from which sample 1024 examples. 4k test queries are sampled from 20 random S2 cells in each of the test cities, as shown in Fig. 23. This corresponds to an average area of 5.6 km2 per city.
(10 for New York) randomly-sample
Sequences used for mapping or localization are randomly drawn from the pool of available StreetView data. Fig. 16-right shows the distribution of temporal differences. This yields challenging queries with large appearance changes, e.g., due to the seasons, or structural changes, e.g., due to construction work. Aerial images are selected as captured as close as possible to the ground-level mapping images. The average time difference varies from 6 months to 3 years.
Temporal distribution:
Ground truth semantic rasters: In the semantic mapping experiments, we used small labeled dataset to train and evaluate our fine-tuning. We start with 1.5k segments of data captured in 84 cities across the world by StreetVVew cars equipped with LiDARs. We aggregate the measurements into point clouds that we label into 10 classes spanning objects encountered in urban environments. These classes include surfaces with direct ground footprint (road, sidewalk, crosswalk, building, tree) and small overhanging objects (fence, pole, street light, traffic sign, traffic light). As each segment is about 80 m long, we derived two maps of size 64x 16 m from each, facing either side of the street. We derived semantic rasters by bimning the points into each map cell and inferring two types of labels: a) Class surfaces are mutually-excluss labels corresponding to the class most represented in each map cell. b) Differently, objects might appear on top of each other so they are not mutually-exclusee each object class using threshold on the number of points in each map cell. We obtain 3k semantic rasters which are split between training and test sets, each including data from all cities.
in 2D so we compute multi-class surface
We thus compute binary labels for
As defined by Google's S2 geometry library: https: htttpp:///22gemetzyyy.o
26

---

Semantic rasters for OrienterNet: To obtain labels at the scale required to train OrienterNet [78], we apply this rasterization process to sparse
points that have similar semantic labels. Those labels are obtained fully automatically by fusing 2D image segmentations, without any human work [30]. While the resulting rasters are noisier than those derived from LiDAR, they cover our entire training set rather than only 3k maps. They also more consistently cover small objects (like trees and street lights) than OpenStreetMap.
E Implementation details
Training: We develop our models with JAX [11] and Scenic [22], and format dataset with TEDS [97]. All models are trained on 16 A100 GPUs over 3-4 days with a total batch size of 32 (2 examples per GPU). We use the ADAM [44] optimizer over 400k iterations for the small model and 200k iterations for the larger one. This corresponds to 24% and 12% of the training data, respectively, but is sufficient for convergence. We use a constant learning rate of 10 for the first half and then decay it to zero with a cosine decay. Streetview images are resized to 684x456. We train our models with mixed precision and use gradient checkpointing in order to reduce GPU memory usage as much as possible. The temperature T is parameterized as log initialized to O, and optimized alongside other model parameters. All hyperparameters are tuned on the validation set.
In the multi-view fusion, we fuse observations from the 4 closest views that observe point. This saves significant amount of memory compared fusing observations from all views. Using fewer views (20 instead of 36 at inference time) also makes the problem artificially harder.
Pose estimation: We use 10k RANSAC samples during training and 20k during inference. Since only 3 constraints are required to solve for 3-DoF pose, 2D correspondences over-constrain the problem. Randomly sampling each correspondence independently thus yields a large number of invalid poses. To increase the ratio of valid poses, we sample 8x the number of desired correspondences and pick the ones with the ratio of distances closest to 1. This improves the pose quality, at both training and test time, and is well amenable to parallelized GPU processing, without requiring an expensive sorting. When computing the featuremetric score, query points that project outside the map given the value of the map boundary. We found this to be more robust than ignoring these points. At inference time, we refine the pose by exhaustive search in grid centered the maximum-score pose, with size mx4 mx and resolution cm..0.5
The decoder of the image encoder is an FPN with bilinear upsampling. We pro- gressively upsample features from stages 4, 3, and 2 to the resolution of stage-1 ResNet features, which corresponds to output stride of 4. For the aerial encoder, remove the root block (7x7 convolution and max pooling) such that the output stride is 1.
Architecture:
We train a small 2-layer CNN with two heads: one for the multi-class classification of ground surfaces, using softmax with cross-entropy loss, and the other for the multi-label classification of objects, using sigmoids and binary cross-entropy losses. We train for 5k iterations and retain the checkpoint with the lowest validation loss.
Semantic fine-tuning:
F
Social impact
CO2 emissions: Experiments were conducted using Google Cloud in region us-centrall, which has carbon efficiency of 0.57 kgCOzeq/kWh, A cumulative of 88.8k hours of computation was performed on NVidia A100 GPUs. Total emissions are estimated to be 12654 kgCOzeq, of which 100 percent were directly offset by Google. Had this model been run in region curope-west6, the emissions would have been of 355.31 kgCOzeq. Estimations were conducted using the MI. Impact calculator [47].
27

---

map images
query At=32.0m AR=3.1
The query is too far from the buildings and fails to localize using features along the road (medium).
images
query 2t- 22.6m AR 179.6*
0
SNAP match the right visual content from the wrong side, due scene symmetrics (medium).
map images
query 2t=24,8m 4R=4.3
Reasonable but incorrect guess along a row of parking spots with very little structure otherwise (medium).
map images
query Mt=7.9m 0R=
Similar (c), but across opposite views, and thus more challenging. Note that the rotation error <19 (hard).
Figure 17: Localization failures (1/2). See legend in Fig. 3.
28

---

images
query At- d.-0.1
A difficult with shadows and occluders (medium).
images
query 2t-27. AR-1.1
b) Correct orientation (19) at the wrong location along the road, opposite querylneference views (medium).
images
query
tt= 15.2m 4R=80.0"
[Osaka] Query and reference views very close to the buildings, which is rarely seen during training (easy).
images
query 4t= 20.2m AR 179.9*
d) [Osaka] Query is on side street with little visual overlap with the reference views (easy).
Figure 18: Localization failures (2/2). See legend in Fig. 3.
29

---

map images
query At=1.3m 3R=
a) The tree smeared yet clearly visible on the left side of the query BEV as a pink dash in a green circle (hard).
images
query At=0.1m AR=1.11
b) The query accurately localized (10 from other side the square (hard).
map Images
query
At-0.1m AR-1.2
The reference views irregularly distributed due the car's motion, which method robust (hard).
map images
query 0t-0.8m 0R-
8
d) The query localized within 0.8 from very difficult angle with little visual overlap (hard). Figure 19: Single-image localization. Examples scenes with non-rectiline non-rectilinear mapping sequences, which results in partial coverage. Please refer to Appendix D for details.
30

---

Barcelona
Mempttate
Barcelona
San Francisco
Manila
Sydpey
Paris
New York
Sydney
London
Taipei
Rio de Janeiro
Singapore
Taipei
Tokyo
Figure 20: Spatial distribution of the training data. In each city, training examples are sampled from red areas while validation examples are sampled from the blue areas.
31

---

Amsterdam
Melbourne
Mexico City
Recail [86]
60
40
0
Paulo
Seattle
Recall (96
60
20
0
1
2
4
2
Position error [meter]
Amsterdam
Melbourne
Mexico City
80
Recall 16]
60
40
20
Osaka
Sao Paulo
Seattle
80
Recall 1%)
40
20
10
Orientation error [degree]
SfM SIFT
SfM SuperGlue
Semantic maps (OrienterNet)
Ours
Figure 21: Single-image positioning per city. We plot the position and orientation recall for each of the 6 test cities.
BACKPACK
80
Recall (%)
60
40
20
CAR
80
Recall (%6
60
40
20
0
Position
[meter]
Orientation [degree]
Figure 22: Single-image positioning per platform. We plot the position and orientation recall for
queries taken by cameras mounted on either backpacks or cars.
32

---

Fleiington
Noord
Meticure Moseum
Adootalord Conmett
LAS BRURNS
MAVDOLLNN
Kew
Wrme Foar
Melbourne
Amssterdam
Me D City
WHSSTERO
Pestr Mater
Amoreca
Amsterdam
Melbourne
Mexico City
BRANCS
Paulo
Osak
Mizah
SHO Paulo
Weatee
K
JARGING
riame Bos
Seattie
PIN
BROOKUM
BROOBLM
New York City
Sao Paulo
Seattle
Figure 23: Spatial distribution of the test data. For each city, test queries are sampled from 20
level-14 S2 cells and are here shown as heatmaps.
33


Reviews:
[{"review_id": "NXj6Z14xps", "reviewer": "Reviewer_sYmT", "summary": "This paper proposes SNAP, a novel architecture to obtain 2D neural maps from street-view and top-down images. The model works by encoding street-view and top-down images separately using corresponding encoders, projecting features to the top-down 2D space, and fusing them via max-pooling to obtain the final features. The paper also proposes to train the model using a contrastive map alignment objective, where a query map is aligned with a reference map at the ground-truth pose, and differentiated from negative poses. The learned representations are evaluated on single-image positioning and semantic mapping applications. The results show strong improvement over prior approaches.", "questions": "Please address the weaknesses listed.", "limitations": "Yes, this is discussed well.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem of learning semantic map representations in a self-supervised way from raw images and camera poses is interesting and well-motivated. While the approach shares some technical similarities with prior work in terms of the mapping functions, the large-scale nature of training, self-supervised learning, and use of top-down + street-view images is exciting. \n* The proposed approach is sensible, the training methodology is simple yet effective, and is clearly described in the paper.\n* The experiments are well designed and show strong improvements over baselines, and demonstrate the semantic quality of the learned representations.\n\n", "weaknesses": "* L41 - 42 - Will the \"50M street view images\" dataset be shared publicly for reproducibility?\n* L73 - 77 - How reasonable are the assumptions made? E.g., Is the projection function accurate and noise-free in practice? How easy is it to get ortho-rectified overhead images? How are they obtained?\n* Clarifications needed\n    * L92 - \"picks the best estimate ... handles partial observability\" -- how exactly is this achieved?\n    * L163 - 165 - How is inference performed to generate pose hypotheses? How scalable is this as map sizes grow larger?\n    * Figure 6 - The classification setup for the center plot is not clearly described. Is there quantitative evaluation for the segmentation shown in the right?\n* Similarity to prior work not discussed. Please compare and contrast.\n    * The mapping approach described in  L99 - 101 is very similar to [R1, R2]. \n    * The objective of learning features by localization is demonstrated in R1, albeit in a classification approach instead of the proposed contrastive learning.\n\n\n[R1] Henriques, Joao F., and Andrea Vedaldi. \"Mapnet: An allocentric spatial memory for mapping environments.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[R2] Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. \"Geometry-aware recurrent neural networks for active visual recognition.\" Advances in Neural Information Processing Systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Will the \"50M street view images\" dataset be shared publicly for reproducibility?**\n\nAs stated in the global response, we would really like to, but we cannot, because we do not own the data. Additionally, please note that a large-scale dataset of ground-level images must comply with regulations such as GDPR, including takedown requests, which require non-trivial infrastructure and maintenance and are impossible to enforce after the data has been released. Existing projects that distribute datasets crawled from StreetView do so illegally, without permission from the data owners, and do not comply with such regulations.\n\n**Is the projection function accurate and noise-free in practice?**\n\nYes, and estimating it for different devices/cameras is easy in practice thanks to Structure-from-Motion.\n\n**How easy is it to get ortho-rectified overhead images? How are they obtained?**\n\nAerial images are captured by planes and are ortho-rectified using a coarse 3D model derived from the same imagery. We use this data because it is readily available and aligned to StreetView images (note that similarly to StreetView images, we use this data with special permission from the owners). There is no conceptual difference between ortho-rectified aerial images and satellite imagery, other than image resolution, which is typically a bit lower for the latter.\n\n**L92 - \"picks the best estimate ... handles partial observability\" -- how exactly is this achieved?**\n\nThe max operator picks the most distinctive features among all observations. If a view doesn't have good information about a point (e.g. because of occlusion), it can be given a low value that can be ignored by the max.\n\n**L163 - 165 - How is inference performed to generate pose hypotheses?**\n\nLike at training time, pose candidates are computed and scored from high-similarity correspondences obtained by exhaustive matching or even (approximate) k nearest-neighbor search. At inference time we additionally refine the pose via exhaustive matching in a grid centered at the highest-score pose: please refer to appendix E for details.\n\n**How scalable is this as map sizes grow larger?**\n\nThis would not scale very well if maps became orders of magnitude larger, such as for instance if GPS priors of any quality are not available. This is also the case for any other approach to fine-grained visual positioning, including structure-based relocalization with maps from SfM. The standard in academia and industry is to use a retrieval index to find coarsely-localized candidates (see e.g. [A]) and refine them. This can also be applied to our approach.\n\n[A] \"From Coarse to Fine: Robust Hierarchical Localization at Large Scale\", Sarlin et al, CVPR 2019.\n\n**Figure 6 - The classification setup for the center plot is not clearly described. Is there quantitative evaluation for the segmentation shown in the right?**\n\nWe evaluate the segmentation with classification metrics. We have 3k non-overlapping, labeled scenes, and use 2k for training and the remaining 1k for testing. The scenes come from 84 different cities across the world, and each of them is represented in the training and testing splits. We will add this information to the revised supplementary material.\n\nAll results shown in Fig. 6 in the paper are from the test set. In the middle plot, we show the quantitative results aggregated over the test set. The plot shown on the right is a single qualitative example from the test split. Please note that we added a few more examples in Figs. 6 and 7 in the supplementary material.\n\n**The mapping approach described in L99 - 101 is very similar to [MapNet, Geometry-aware recurrent neural networks for active visual recognition].**\n\nPlease note that we cite MapNet in the related work section (L186): its localization step relies on exhaustive voting, similarly to [21, 64], which is much more expensive than our inference scheme based on RANSAC (complexity $R N \\log N$ vs $N$, for a map with $N$ cells and voting with $R$ rotations). Its mapping step relies on input depth maps, while SNAP learns monocular priors end-to-end. We thank the reviewer for mentioning the other work -- it is indeed relevant and we will cite and discuss it in the final revision of the paper (NeurIPS does not allow paper revisions during the rebuttal phase)."}, {"review_id": "95mwd4Ohjk", "reviewer": "Reviewer_Lqgd", "summary": "This work develops neural 2D maps in an self-supervised manner by matching multi-view ground level imagery with aerial images.\nTo construct a neural map (2D map of features), they encode an image into a feature frustum (similar to lift-splat-shoot), then pool vertically - for multi-view images, they perform a similar fusion method and interpolate between image features at multiple defined heights to construct the map.  \nFor supervision, they use known 3-dof poses to align two neural maps. They use a contrastive loss and mine for negative samples using RANSAC. \nThey train on a large scale streetview dataset with aerial imagery and evaluate performance on a visual positioning task - given 36 views, predict the 3-dof pose of where a query image was taken. Results show improvements over traditional SFM and more recent learning based methods.  ", "questions": "* Are the neural maps biased towards features that are greater in height? During training with multi-modal inputs, the aerial tiles are biased towards features that are maximum in height (all other stuff is occluded) and so in theory there is no need for the feature encoder to learn anything below the highest cell.\n* Line 105: I didn't understand this. I thought lift-splat-shoot normalized the distribution.  \n* Is the data used public?", "limitations": "Limitations are provided (accuracy + method assumptions).  \nCO2 usage is specified in supplementary.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Innovative and novel idea with real world applications.  \n* Technical section fairly clear to follow.\n* Overall the work is well written and descriptive.\n* The multi-view fusion could be applied to other fields (3D object detection)", "weaknesses": "* There are many simplifying assumptions (which the authors have provided discussion for) - relative poses which they \"claim can easily be obtained\", gravity direction, pose in only 3-Dof.\n* Quantitative evaluation is not as fair as I would like. Especially when comparing streetview inputs only, the performance improvement is not particularly strong.  \n* All ablations are in the supplementary. The system is consists of many design choices and it would be useful to present which choices were critical and which did not matter much.  \n* All baselines are fairly strong in the \"Easy\" group. I imagine this is actually the most important category for real world applications since the pose estimation would be applied in the loop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Quantitative evaluation is not as fair as I would like. Especially when comparing streetview inputs only, the performance improvement is not particularly strong.**\n\nWe note that with StreetView inputs only, our model still outperforms SfM+SuperGlue (a strong and popular baseline) on the medium (43.9 / 56.0 vs 38.0 / 44.4) and hard (29.5 / 41.7 vs 13.1 / 16.1) splits, and also on average (41.0 / 53.2 vs 39.2 / 45.2): see Table 1.\n\n**All ablations are in the supplementary. The system consists of many design choices and it would be useful to present which choices were critical and which did not matter much.**\n\nWe believe that the ablation study already substantiates our architecture choices: all the choices evaluated in Table 1 of the supplementary material have a large impact on the localization accuracy. Choices which we observed had very little effect were not considered here.\n\nWe put the ablation studies section in the supplementary material due to space constraints, but we plan to move it to the main paper in the camera-ready version if the paper is accepted, as NeurIPS grants an extra page then.\n\n**All baselines are fairly strong in the \"Easy\" group. I imagine this is actually the most important category for real world applications since the pose estimation would be applied in the loop.**\n\nBaselines based on SfM work well in the presence of easily discriminated structure, such as building facades, but often fail if they are occluded or if the patterns are ambiguous or repeated. Our approach can actually _leverage_ structure such as trees or road markings as semantic cues and localize queries beyond the reach of traditional approaches. Figures 9-14 in the supplementary material show examples that the baselines would fail to resolve: these are all from real settings in some of the most populated cities in the world. We thus think that solving examples in the \"medium\" and \"hard\" categories is actually _more_ important for many applications in urban environments.\n\n**Are the neural maps biased towards features that are greater in height? During training with multi-modal inputs, the aerial tiles are biased towards features that are maximum in height (all other stuff is occluded) and so in theory there is no need for the feature encoder to learn anything below the highest cell.**\n\nLooking at the RGB visualizations of the neural maps, tree trunks are clearly distinguishable (in pink), but they are usually not visible in the aerial images, because they are occluded by foliage. This shows that neural maps can encode distinctive features at any height. Also, quantitatively, we observe that using only ground-level imagery performs ~46% better than using only aerial imagery due to the loss of information. Combining both input modalities improves the positioning accuracy over ground-level only by ~8%.\n\n**Line 105: I didn't understand this. I thought lift-splat-shoot normalized the distribution.**\n\nLift-splat-shoot indeed infers a normalized distribution over depths along each ray. Instead, SNAP infers an absolute score $S$ that is normalized across multiple observations of the same point ($w\\_k^n$ in Eq. 1). This makes it possible 1) to compare monocular priors between multiple views and 2) to ignore some features, e.g. those that belong to dynamic objects.\n\n**Is the data used public?**\n\nWe cannot release the training data, as we do not own it: please see our comment at the end of the global response. We will open-source and release the code upon acceptance."}, {"review_id": "Hjuk8gRjDM", "reviewer": "Reviewer_qSVT", "summary": "This paper presents a method for self-supervised learning of BEV features, by using a contrastive loss that links features computed from perspective-view images to features computed from aerial images, where BEV correspondences are given by poses. Interesting method details include depth-weighted splatting in the style of Lift-Splat-Shoot, local decoding/fusion with an MLP, and hard negative sampling via RANSAC. The method provides good localization especially in \"hard\" cases where the camera moves a lot, and the features are useful pre-training for downstream semantic segmentation tasks. \n", "questions": "\"Overhead images are ortho-rectified, such that all world points along the z axis project onto the same pixel coordinate.\"\n\nThis sounds odd to me, so I want to double check. I am imagining the aerial imagery is captured from very high up, and the images cover a wide FOV, and only a few pixels will be well-aligned with the z axis (defined as the gravity direction). Fixing this seems like it should require depth information (e.g., especially for tall buildings).\n\n\n\"we define K horizontal planes at heights {zk}, which are uniformly distributed around the camera heights and points of interest\"\n\nWhat exactly are the \"points of interest\"?\n\n\nIt's interesting that the architecture does not have any BEV CNN involved. To my understanding, it is often this BEV computation which resolves much of the 3D structure of the scene, since depth information is so difficult to estimate from RGB alone (without supervision). Is it really just an MLP applied on each cell independently? The citation here (SimpleRecon) seems not as simple as what's presented here -- it uses a cost volume, various geometric \"metadata\", and another 2D CNN. Can you comment a bit more here, to help me get a sense of what's going on in the architecture? \n\n\n\"Our approach tightly combines 3D geometry and representation learning instead of relying on expensive Transformer [60] or 3D CNNs [25].\"\n\nOK but I think the more reasonable baseline is a 2D CNN applied in BEV (e.g., [26]). This is lightweight also. \n\n\n\"We build mapping segments only from car sequences\"\n\nWhy? And why discuss the data collected from pedestrians/backpacks, if it is never used?\n\n\n", "limitations": "Looks good", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is very well written. The problem and the method are very clearly articulated, the figures and sectioning and style are all helpful to getting the key ideas across efficiently. \n\nSelf-supervised feature learning from the combination of car-mounted and aerial data is a new and exciting area, and the approach here is well motivated. The method makes sense. \n\nThe approach achieves fairly good results, and outperforms SfM methods in some cases. \n\n\nUsing featuremetric similarity to guide the sampling is a very nice detail.\n", "weaknesses": "The pre-training experiment would be more convincing if it were benchmarked agaginst some existing approach. I am particularly interested to see if the pre-trained model outperforms some SOTA on some BEV segmentation task (e.g., on nuScenes). The danger is something like: as you add more supervised data, the impact of the pre-training goes to zero.\n\nThe interpolation operator used in eq 4 is confusing for me. Can you write it or explain it in another way? (One additional question here, beyond the main confusion of that equation, is: are you really doing a plain sum here? Typically a mean makes more sense here, so that the scale of the term does not depend on a hyperparameter like resolution.)\n\nThe overall map seems like it may be huge. I need more implementation details on this aspect to really understand what is going on, especially with respect to generating pairs (and non-pairs) for the contrastive loss. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**I am particularly interested to see if the pre-trained model outperforms some SOTA on some BEV segmentation task (e.g., on nuScenes). The danger is something like: as you add more supervised data, the impact of the pre-training goes to zero.**\n\nObtaining training labels for 3D semantic tasks is very expensive. Such tasks are also more prone to domain shifts than geometric tasks (like registration) and thus require new labels when deploying models on new environments. Getting enough training data for global generalization (across countries, sensors, times of the day) is unfeasible. Our pre-training leverages camera poses, which are much cheaper to acquire at scale.\n\nThe information bottleneck forces SNAP to learn unified representations for objects, like street crossings or lights, that look very different across countries, and would require larger amounts of labels. Showing this on public datasets is difficult since they rarely span more than a few cities (nuScenes has data from Boston and Singapore) and overfit supervised models to local appearance.\n\n**The interpolation operator used in eq 4 is confusing for me.**\n\nWe consider a point $ (i,j) $ in the query map, with corresponding query feature $ \\bar{\\mathbf{M}}\\_{ij}^Q\\in\\mathbb{R}^D $.\n\nWe write $ \\mathbf{C}\\_{ij} $ the $ I\\times J $ similarity map between this query feature and each reference feature $ \\bar{\\mathbf{M}}^R\\in\\mathbb{R}^{I\\times J\\times D} $.\n\nThis point has a 2D coordinate $ \\mathbf{G}\\_{ij}^Q $ in the coordinate system of the query grid.\n\nA relative pose candidate $ {}\\_R\\mathbf{T}\\_Q $ transforms this point into the reference coordinate system as $ \\mathbf{p} = {}\\_R\\mathbf{T}\\_Q\\cdot\\mathbf{G}\\_{ij}^Q $.\n\nTo obtain the similarity of this point given this pose, we bi-linearly interpolate the similarity map at $ \\mathbf{p} $: $ \\mathbf{C}\\_{ij}\\left[\\mathbf{p}\\right] $.\n\n(Line breaks for readability.) We apologize for the confusion and will improve this explanation in the revised paper.\n\n**are you really doing a plain sum here? Typically a mean makes more sense here**\n\nWe indeed normalize by the number of query points $I\\cdot J$, which is missing in Eq 4. We will fix it in the final version.\n\n**The overall map seems like it may be huge.**\n\nOur neural tiles are 64x16m with 20cm resolution, and 128 dimensions. Our tiles are smaller than SuperPoint (L266): ~1.6 MB vs ~5.3 MB. As stated in the supplementary material (L112), we train with a batch size of 32 over 16 GPUs (=two examples per GPU). We found that reducing the grid to increase the batch size hurt performance, as large areas are necessary to find meaningful hard negatives for the contrastive loss (see appendix, Sec. A).\n\nWe will release our code, for reference.\n\n**[On orthorectification of aerial images:] I am imagining the aerial imagery is captured from very high up, and the images cover a wide FOV, and only a few pixels will be well-aligned with the z axis (defined as the gravity direction). Fixing this seems like it should require depth information**\n\nAerial images are captured by planes and are ortho-rectified using a coarse 3D model derived from the same imagery. We use this data because it is available and aligned to StreetView images (we use it with special permission from the data owners). There is no conceptual difference between ortho-rectified aerial images and satellite imagery (resolution is typically a bit lower for the latter).\n\n**\"we define K horizontal planes at heights ${z\\_k}$, uniformly distributed around the camera heights and points of interest\". What exactly are the \"points of interest\"?**\n\nWe meant uniformly within a range \"of interest\". We use 32 height planes distributed every 20cm within 12m (L230), starting 4m below the camera. We will clarify this.\n\n**It's interesting that the architecture does not have any BEV CNN involved. [...] it is often this BEV computation which resolves much of the 3D structure of the scene [...] Is it really just an MLP applied on each cell independently? [SimpleRecon] seems not as simple [as this]**\n\nAn MLP is indeed applied to each 3D point individually (Eq 2). Its output $\\mathbf{X}\\_k$ is pooled vertically to obtain cell feature $\\mathbf{M}\\_{ij}$. We found that the supervision signal and amount of data are sufficient to learn very good monocular depth (appendix, Fig. 3). This, combined with multi-view fusion, makes it unnecessary to apply a top-down BEV CNN or an elaborate vertical pooling mechanism.\n\nUsing a simple MLP has multiple benefits: 1) we can infer a map feature for any 2D point and cells, so they don't need to be distributed on a regular grid; 2) neural maps are fully equivariant to any translation/rotation of the coordinate system of the camera poses; 3) avoids boundary artifacts introduced by padding in a BEV CNN.\n\nThis idea is similar to SimpleRecon, which replaces an expensive 3D CNN with a simpler per-cell MLP, which can be very effective if fed the right information (here multi-view features and monocular scores). We do agree that the comparison is limited.\n\n**\"Our approach tightly combines 3D geometry and representation learning instead of relying on expensive Transformer or 3D CNNs.\"\nI think the more reasonable baseline is a 2D CNN applied in BEV. This is lightweight also.**\n\nAs mentioned above, a BEV CNN easily introduces boundary artifacts due to padding and breaks the rotation equivariance of the mapping process. This CNN would increase the complexity of the model and it is unclear what problem it would solve. Our experiments show that the MLP-only design already performs well.\n\n**Why discuss the data collected from pedestrians/backpacks if it is never used?**\n\nWe use both (L215-219): we build neural maps from car images, but use car and backpack images for queries. This mirrors the typical use-case where one would map from car-mounted rigs (StreetView cars) and query with single images (e.g. pedestrians using cell phones). Fig. 2 in the appendix breaks the results down.\n"}, {"review_id": "dufY5OTbma", "reviewer": "Reviewer_dKqM", "summary": "This paper proposes a self-supervised learning approach to create a neural representation of the 3D environment from aerial-view and ground-view images. It utilizes contrastive learning to align neural maps based on the same pose and distinguish maps with inconsistent poses. Large-scale pretraining enhances the model's capabilities for visual positioning and semantic mapping. Experimental results on two datasets validate the effectiveness of the approach.", "questions": "Overall, the paper presents a novel and well-executed approach to learning neural scene representations from multimodal images. Its technical implementation, combined with the potential for practical applications, makes it a valuable contribution to the field. However, there are some concerns listed in the weakness section. This reviewer will consider raising the score if the concerns are well addressed.\n", "limitations": "There seems no checklist.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The overall idea of learning a neural scene representation from multimodal images via pose contrast is both elegant and effective. This approach allows for a comprehensive understanding of scenes by leveraging different modalities, resulting in a robust representation.\n\n2. The implementation of the overall framework is detailed and non-trivial. The monocular inference, multiview fusion, multimodal fusion, and differentiable pose estimation are well executed and contribute together to the final performance.\n\n3. The large-scale pretraining enhances the neural map with high-level semantics. This pretrained model can be further fine-tuned for tasks such as visual positioning and semantic mapping. This versatility makes it a strong candidate for serving as the foundation model in areas like robotic localization and mapping.\n\n4. The writing in the paper is clear and easy to follow. Additionally, the visualization provided is informative.\n", "weaknesses": "1. **Pose supervision**: The authors should discuss the influence of pose accuracy on SNAP's performance and address how reliable pose information can be obtained to improve SNAP's learning capabilities. Exploring the impact of pose accuracy on the model's performance would provide valuable insights.\n\n\n2. **Overall illustration**: It would be beneficial for the authors to include a pseudo-algorithm to better illustrate the notations used and clarify the training and inference procedures of SNAP. This addition would enhance the understanding of the model's implementation.\n\n\n3. **Training and inference time**: Given the large scale of the training set, it would be helpful for the authors to discuss the training time of SNAP and justify the necessity of such a large-scale dataset (will reducing the training data significantly reduce the performance of SNAP?). Additionally, considering the practicality for mobile robotics, it would be appropriate to provide information on the inference time on a more affordable GPU. This discussion would provide insights into the computational requirements of the model.\n\n\n4. **Depth estimation**: The authors should clarify how the depth model handles the large distortions present in the input images. Providing details on how these distortions are addressed and mitigated would enhance the understanding of the model's performance.\n\n\n5. **Ablation studies**: The reviewer suggests conducting ablation studies to investigate the influence of Equation (1) and the max-pooling used for multimodal fusion. Including these studies would demonstrate the effectiveness of different modules and contribute to a deeper understanding of the model's performance.\n\n\n6. **Details of semantic mapping**: More information is needed regarding the input, scale, and output of the semantic mapping process. Additionally, since the semantic classes appear to be limited, the authors are encouraged to test the model on publicly available datasets such as SemanticKITTI SSCBenchmark to provide a comprehensive evaluation.\n\n\n7. **Dataset and code availability**: It is essential for the authors to make the datasets used for pretraining and semantic mapping publicly available. Furthermore, open-sourcing the code and data would greatly benefit the field, allowing researchers to replicate and build upon the presented work.\n\n\n**Minor**\n1. The authors are suggested to show easy, medium, and hard examples’ visualizations respectively.\n\n2. In the literature review section (line 196-line 204), it would be beneficial for the authors to include references to works related to semantic occupancy mapping, specifically addressing voxel representations for large-scale scene understanding (not only just object-centric applications). Adding these related works will provide a broader context. Some relevant works to consider include MonoScene [1], VoxFormer [2], TPVFormer [3], and SemanticKITTI [4]. \n\n[1] Cao, A.Q. and de Charette, R., 2022. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3991-4001).\n\n[2] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C. and Anandkumar, A., 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9087-9098).\n\n[3] Huang, Y., Zheng, W., Zhang, Y., Zhou, J. and Lu, J., 2023. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9223-9232).\n\n[4] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C. and Gall, J., 2019. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9297-9307).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Exploring the impact of pose accuracy [of the training data] on the model's performance would provide valuable insights.**\n\nWe did not try polluting the poses while training. While we can try this, it was not possible to regenerate the data and retrain the model within the rebuttal window.\n\n**It would be beneficial to include a pseudo-algorithm to better illustrate the notations and clarify training/inference procedures**\n\nWe will include a glossary and pseudo-algorithm in the appendix for the final version of the paper. \n\n**Training and inference time: [...] it would be helpful to discuss the training time and justify the necessity of such a large-scale dataset (will reducing the training data significantly reduce the performance?)**\n\nThe supplementary material mentions that the small and large models were trained for 400k and 200k iterations respectively. This corresponds to 3-4 days of training for each. We anneal the learning rate at half of the total iterations but observe that the loss is still decreasing. Training for a longer time would likely improve the results further but with diminishing returns and prohibitive costs (each model is trained with 16 A100 GPUS). This limitation is similar to other foundation models, like large language models.\n\nThe diversity of the training data (i.e. covering multiple neighborhoods of multiple cities) is critical to the generalization of SNAP to different locations. Scaling an earlier version of the model showed that adding more cities (up to 11, as eventually used) directly improves the performance on a held-out test set. We are not able to reproduce such experiments with the final model within the rebuttal period, but will try to add this study to the final version.\n\n**Considering the practicality for mobile robotics, it would be appropriate to provide information on the inference time on a more affordable GPU.**\n\nCurrently we only have access to GPUs mounted on remote cluster machines (e.g. A100/P100/V100). We will try to run this on embedded devices and answer this question in the final version, but it is not feasible within the rebuttal window. We will also release the code.\n\n**Depth estimation: The authors should clarify how the depth model handles the large distortions present in the input images.**\n\nThe projection model accounts for radial distortion and rolling shutter. Note that the rolling shutter prevents us from undistorting the images, so we feed them as-is to the ResNet backbone. The model learns to handle such distortions given the large amount of data.\n\n**[Ablation] The reviewer suggests conducting ablation studies to investigate the influence of Eq. (1) and the max-pooling used for multimodal fusion.**\n\n**On Eq. 1:** Table 1 in the supplementary material compares SNAP to different variants (of Eq. 1) that either remove the monocular prior (uniform instead of weighted average and variance) or replace it with a more expensive ray conditioning [74]. It is unclear what other aspects of Eq. 1 should be studied: any clarification is very much welcome.\n\n**On max-pooling:** The _max_ operator has a number of theoretical benefits over _average_ or _attention_ pooling:\n- It is transitive (when fusing 3 maps recursively, one can perform pairwise fusion in any order with no impact on the final result)\n- It is robust to noise, as it can ignore inputs with low information (e.g. because of occlusion).\n\nWhile we validated this experimentally with an earlier version of the model (and thus chose the max), we are not able to reproduce such results with the final model within the rebuttal period, but will try to add this ablation to the final version.\n\n**Details of semantic mapping: More information is needed regarding the input, scale, and output of the semantic mapping process. Additionally [...] the authors are encouraged to test the model on publicly available datasets**\n\nWe have 3k non-overlapping, labeled scenes, and use 2k for training and the remaining 1k for testing. The scenes come from 84 different cities across the world, and each of them is represented in the training and testing splits. We will add this information to the revised supplementary material.\n\nAs stated in the global response, we will open-source and release the code. While we plan to train and evaluate our model on open benchmarks, we can not do so within the scope of the rebuttal.\n\n**Dataset and code availability: It is essential for the authors to make the datasets used for pretraining and semantic mapping publicly available. Furthermore, open-sourcing the code and data would greatly benefit the field**\n\nAs stated in the global response, we will open-source the code. We cannot make the data available because we do not own it. Please note that a large-scale dataset of ground-level images must comply with regulations such as GDPR, including takedown requests, which require non-trivial infrastructure/maintenance and are impossible to enforce after the data has been released. Existing projects that distribute datasets crawled from StreetView do so without permission from the data owners and ignoring these regulations.\n\n**The authors are suggested to show easy, medium, and hard examples’ visualizations respectively.**\n\nWe will label each qualitative example accordingly.\n\n**In the literature review section, it would be beneficial for the authors to include references to works related to semantic occupancy mapping, specifically addressing voxel representations for large-scale scene understanding (not only just object-centric applications). Adding these related works will provide a broader context. Some relevant works to consider include MonoScene [1], VoxFormer [2], TPVFormer [3], and SemanticKITTI [4].**\n\nWe thank the reviewer for this suggestion. We agree that these works are relevant and will add them to the related work section.\n\n**Limitations: There seems no checklist.**\n\nWe note some limitations in L280-282."}, {"review_id": "isF8TaspBh", "reviewer": "Reviewer_PKpW", "summary": " Currently，2D semantic maps have limitations:  they lack detail, often contain inaccuracies, and are difficult to create and maintain， in order to solve these problem,  the author proposes fusing features from different input modalities to obtain a 2D neural map that is robust over time and retains sufficient geometric and semantic information. The effectiveness of the method was demonstrated by using the 2D neural map generated by SNAP for other tasks.", "questions": "1.It is necessary to supplement experimental evidence for the effectiveness of fuse and contrastive learning supervision in the generation process of neural maps.\n2.Can other satellite-based localization methods generate neural maps? The authors need to conduct experiments to demonstrate the advantages of their method compared to other satellite-based localization methods in generating neural maps.\n3.Does the method also perform well on other general datasets such as KITTI?\n", "limitations": "The author discussed the limitations of the method on accuracy and the impact of the method on privacy.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "  This paper proposes to fuse various input modalities and use camera pose estimation to generate a 2D neural map, which is the originality. Moreover, the paper delves into this methodology, related work, and conducts a plethora of experiments, culminating in high-quality output. However, the clarity of the paper requires further refinement, and its significance remains somewhat limited.", "weaknesses": "The author demonstrated the effectiveness of using self-supervised pre-training in the article, but there is no relevant experimental evidence to prove whether the fusion of multi-mode features and the proposed supervision method are reasonable. In addition, it is unfair to only compare with OrienterNet in Experiment 1, as the generation of neural graphs involves aerial images, which are more similar to [1][2].\n[1] Shi Y, Li H. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 17010-17020.\n[2] Lentsch T, Xia Z, Caesar H, et al. SliceMatch: Geometry-guided Aggregation for Cross-View Pose Estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17225-17234.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[...] there is no relevant experimental evidence to prove whether the fusion of multi-mode features and the proposed supervision method are reasonable.\nIt is necessary to supplement experimental evidence for the effectiveness of fuse and contrastive learning supervision in the generation process of neural maps.**\n\nWe show the effectiveness of multi-modal fusion _precisely_ with experimental evidence. Table 1 shows that fusing ground-level and aerial cues improves performance by ~8% relative to using only ground-level cues. Likewise, we show that our supervision method is reasonable through our experiments: we outperform traditional SfM-based methods in very challenging situations (and overall) by training only with posed imagery within the contrastive learning framework we present. It is not clear to us how one would alternatively learn neural maps given only image poses.\n\nThe ablation study in Section A of the supplementary material demonstrates the effectiveness of our design decisions. We invite the reviewer to clarify what they mean with these statements and we will try to address them during the discussion phase.\n\n**In addition, it is unfair to only compare with OrienterNet in Experiment 1, as the generation of neural graphs involves aerial images.**\n\nWe believe our comparison with OrienterNet is fair. Please note that OrienterNet takes as input 2D, overhead semantic maps (the kind used for navigation), _not_ aerial images. We train our version of OrienterNet with semantic rasters similar to the semantic maps from OpenStreetMap used in their paper. While our rasters are noisy they provide a consistent, global coverage of fine-grained classes relevant for localization (e.g. tree, streetlight, poles). As stated in L253, we checked our implementation with the authors.\n\n**Can other satellite-based localization methods generate neural maps? The authors need to conduct experiments to demonstrate the advantages of their method compared to other satellite-based localization methods in generating neural maps.**\n* **[1] Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image, Shi et al, CVPR 2022**\n* **[2] SliceMatch: Geometry-guided Aggregation for Cross-View Pose Estimation, Lentsch et al, CVPR 2023**\n\nTable 1 reports the performance of a variant of SNAP trained and evaluated using only aerial images. This model is conceptually similar to the approach of Fervers et al [21] (aerial/satellite map, BEV-to-map matching, with known gravity), which itself yields significantly more accurate localization than the approaches of both Shi et al. and Lentsch et al on identical or similar datasets (FordAV, KITTI 360). We thus believe that evaluating these works is unnecessary. Conceptually, both approaches have a limited performance: Shi et al make restrictive assumptions that do not hold in practice (strong location and orientation prior) while SliceMatch lacks geometric inductive bias to produce fine-grained predictions (no explicit BEV matching).\n\nMoreover, our experiments show that using only ground-level imagery performs ~46% (relative) better than using only aerial imagery, and that combining them improves the performance over ground-level only by a more modest ~8% (relative). This is due to the loss of information from occluders (particularly vegetation) in overhead images, which are inevitable in cluttered urban environments. This is a large gap and, we believe, a difficult one to bridge with better architectures that still rely _only_ on aerial/satellite imagery.\n\nFinally, please note that SliceMatch [2] is a CVPR 2023 paper. Code was not available until June 2023, whereas the NeurIPS submission deadline was in May 2023 (please refer to their GitHub repository).\n\n**Does the method also perform well on other general datasets such as KITTI?**\n\nWhile we plan to tackle this in the future, the domain gap is significant and this is out of the scope of what we can achieve within the rebuttal window.\n"}]

Author rebuttal:
We thank the reviewers for their time and well-thought comments. Reviewers noted very clearly that our method is **novel, well-motivated, and exciting**:
* _"the large-scale nature of training, self-supervised learning, and use of top-down + street-view images is exciting"_ -- R.sYmT
* _"the approach here is well motivated. The method makes sense"_ -- R.qSVT
* _"Self-supervised feature learning from the combination of car-mounted and aerial data is a new and exciting area"_ -- R.qSVT.
* _"The overall idea of learning a neural scene representation from multimodal images via pose contrast is both elegant and effective"_ -- R.dKqM
* _"strong candidate for serving as the foundation model in areas like robotic localization and mapping"_ -- R.dKqM
* _"Innovative and novel idea with real world applications."_ -- R.Lqgd

They also praised our **formulation and presentation**:
* _"the training methodology is simple yet effective, and is clearly described in the paper"_ -- R.sYmT
* _"well written and descriptive"_ -- R.Lqgd
* _"very clearly articulated, the figures and sectioning and style are all helpful to getting the key ideas across efficiently"_ -- R.qSVT
* _"The implementation of the overall framework is detailed and non-trivial. The monocular inference, multiview fusion, multimodal fusion, and differentiable pose estimation are well executed and contribute together to the final performance."_ -- R.dKqM

And mostly agree that it **performs strongly,** against both classical and learned approaches:
* _"The results show strong improvement over prior approaches."_ -- R.sYmT
* _"Results show improvements over traditional SFM and more recent learning based methods."_ -- R.Lqgd
* _"The approach achieves fairly good results, and outperforms SfM methods in some cases."_ -- R.qSVT
* _"Experimental results on two datasets validate the effectiveness of the approach."_ -- R.dKqM

We address their questions in separate responses. A common concern was **reproducibility**: we can now commit to releasing the code, as open source (Apache 2.0 license). Note that we cannot release the training data, as we do not own it. Please note that a large-scale dataset of ground-level images must comply with regulations such as GDPR, including takedown requests, which require non-trivial infrastructure and maintenance and are impossible to enforce after the data has been released. Existing projects that distribute datasets crawled from StreetView do so illegally, without permission from the data owners, and do not comply with such regulations.

Another suggestion was evaluation on **open datasets**: we are working on this, along with open-sourcing of the code, but it is a major effort that cannot be completed during the rebuttal period.

Meta-review: This paper describes a method capable of generating 2D maps from two different types of sources, street view images and aerial photographs. This is performed through a contrastive map alignment objective.

The work has received a majority of positive ratings (5,6,7,4,6), but there was also an ongoing discussion with weaknesses raised by reviewer PKpW regarding comparisons with an aerial variant only. This could be addressed by the authors through the rebuttal and discussions. Other issues were simplifying assumptions, limited comparisons, 

This paper was flagged for privacy and security concerns, but the authors do address this issue.

The AC judges that the paper provides valuable contributions to the field and recommends acceptance.
Decision: Accept (poster)