Title: SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural maps

visual positioning

semantic mapping
Abstract: Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain, especially in an automated fashion. Can we use  _raw imagery_ to automatically create _better maps_ that can be easily interpreted by both humans and machines? We introduce SNAP, a deep network that learns rich 2D _neural_ maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps.

SNAP: Self-Supervised Neural Maps

Abstract

Semantic 2D maps are commonly used by humans and machines for navigation purposes, whether it's walking or driving. However, these maps have limitations: they lack detail, often contain inaccuracies, and are difficult to create and maintain, especially in an automated fashion. Can we use raw imagery to automatically create better maps that can be easily interpreted by both humans and machines? We introduce SNAP, a deep network that learns rich neural 2D maps from ground-level and overhead images. We train our model to align neural maps estimated from different inputs, supervised only with camera poses over tens of millions of StreetView images. SNAP can resolve the location of challenging image queries beyond the reach of traditional methods, outperforming the state of the art in localization by a large margin. Moreover, our neural maps encode not only geometry and appearance but also high-level semantics, discovered without explicit supervision. This enables effective pre-training for data-efficient semantic scene understanding, with the potential to unlock cost-efficient creation of more detailed maps. †

1 Introduction

Semantic 2D maps such as Google Maps are ubiquitous in our daily lives, used by billions of people. They offer compact, yet easily interpretable representations of the world from a bird's-eye view, allowing us to effectively navigate large outdoor environments by foot or vehicle. By contrast,

posed inputs

aerial image

sequence

query

image

overhead

encoder

ground-level

encoder

query encoder

neural map

alignment

multi-modal mapping

partial map

final map

query

BEV

fuse

machines position themselves in the real world through computer vision, which remains dominated by structure-based approaches [79, 83, 56, 74, 39, 65] relying on basic hand-crafted [54, 4] or learned [58, 111, 23, 69, 101, 75, 53] primitives, such as points or lines. These approaches build 3D maps with Structure-from-Motion (SfM) and then localize query images via 2D-3D registration. Their complexity and many components (feature extraction and matching, bundle adjustment, pose refinement, etc.) make it difficult to tune [42] or update [25] them, and to learn high-level priors end-to-end [10, 8, 76]. They are also costly to store and generally not reusable for other applications.

Recent works such as OrienterNet [78] instead learn planar, neural representations from the same 2D semantic maps that humans use. These maps encode scene geometry and semantics and can be used for visual positioning with sub-meter accuracy. This approach is however limited to a few semantic classes, and the maps it is based on can be inaccurate, costly to obtain, and difficult to maintain.

We argue that maps are most useful for figuring out where we are when they are abstract enough to be robust to temporal changes, yet preserve enough geometric and semantic information to yield high-quality correspondences with the physical world. Our work, SNAP, shows that, by learning 2D neural maps for localization, meaningful semantics emerge without explicitly supervising them. These semantics improve positioning accuracy and also make our maps usable for other tasks (Fig. 1).

SNAP leverages the complementary strengths of different input modalities, like ground-level and overhead imagery, by fusing them into a single 2D neural map (Fig. 2). It can flexibly and efficiently integrate arbitrary combinations of data captured at different points in time, which is key to continuously update maps in a changing world. We train it end-to-end to estimate the pose of a query image relative to the mapping images, by simply aligning their neural maps. This kind of contrastive learning requires only sensor poses, which can be easily obtained with photogrammetry [45, 37]. We train and evaluate SNAP on a dataset with 50M StreetView images ‡ from 5 continents, orders-of-magnitude larger and more diverse than comparable academic benchmarks.

Despite training only for a positioning objective, we observe that our neural maps learn easilyinterpretable, high-level semantics without the need for explicit semantic cues (Fig. 3), and demonstrate that they provide an effective pretraining for semantic understanding tasks by fine-tuning them on little labeled data. This can potentially unlock cost-efficient creation of more detailed and richer maps, readable by humans and machines alike, while providing state-of-the-art visual positioning.

Our main contributions are as follows. (i) We introduce a simple and lightweight encoder to estimate bird's-eye view maps from ground-level imagery, combining principles from multi-view geometry with strong monocular cues. (ii) We fuse different imaging modalities to integrate and benefit from complementary cues. (iii) We show how to train our model by aligning neural maps in a contrastive learning framework, using RANSAC to mine hard negatives. (iv) We outperform the state of the art on visual positioning and register image queries beyond the reach of traditional methods (Fig. 3). (v) We demonstrate that high-level semantics emerge by learning to align neural maps, without any explicit supervision, and fine-tune them on semantic understanding with few labels (Fig. 6).

map images

query

t=0.3m

R=0.3°

2

3

map images

query

t=0.7m

R=3.6°

map images

query

t=0.6m

R=1.1°

NANNND

0

1

2

3

map images

query

t=0.5m

R=1.4°

2 Mapping the world with neural maps

We now formalize neural maps, and describe a neural network architecture to infer them from raw sensor data. Our goal is to infer a more generic neural representation that can encode both the geometry, semantics, and appearance of a given point in the 2D world.

Problem formulation: For a 3D scene, such as a large outdoor environment, we consider a local, 3D Cartesian coordinate system such that the z axis points upwards along the gravity direction. A neural map M is defined over a regular grid that partitions the xy plane into I × J square cells of size ∆ . Each cell ( i, j ) is associated with a D -dimensional feature M ij ∈ R D . To infer such neural map, we leverage large quantities of raw imagery captured by diverse cameras.

Input modalities: Ground-level images are captured by cameras mounted on StreetView cars or backpacks [18]. They are often part of a sequence of multi-camera frames. As such, they are very unevenly distributed throughout space. Each image offers a high resolution view of a small area, mainly limited by the occlusion of static or dynamic objects like buildings or vehicles. On the other hand, overhead images are captured by cameras mounted on planes or satellites. These images benefit from high spatial coverage at a uniform but low resolution. Their visibility is mostly affected by vertical occluders like trees. Ground-level and overhead images capture different aspects of the environment and are thus complementary.

Assumptions: All images of either modality are calibrated and registered with respect to the map coordinate system. Each image n follows a projection function Π n : R 3 → R 2 that maps a 3D point in the world to a 2D point on the image plane. Π n combines the camera pose w T n ∈ SE(3) and the camera calibration, including lens distortions. Overhead images are ortho-rectified, such that world points along the z axis project onto the same pixel coordinate. As this process relies on a coarse digital surface model [27], fine details like poles are not rectified and may result in artifacts, which SNAP can however learn to account for.

2.1 Fusing multi-modal representations

Each location in the world is observed by an arbitrary number of images for each modality, captured at arbitrary points in time. We thus follow a late-fusion strategy that first encodes each modality separately and only finally fuses them (Fig. 2). This can flexibly adapt to the available inputs and efficiently handle arbitrary spatial distributions of data.

Encoding: We design two encoders that each combine a subset of observations n into a singlemodality neural map M n defined over the same grid as M . Φ OV encodes a single tile of overhead orthoimagery, while Φ SV encodes a single image or multiple covisible ground-level StreetView images, e.g ., a multi-view sequence. To best resolve the 3D information from perspective shots at arbitrary viewpoints, Φ SV leverages both multi-view observations and monocular cues. We describe its architecture in detail in Sec. 2.2. Φ OV , on the other hand, is a simple U-Net-style CNN [71] that computes a feature for each pixel of the overhead orthoimage, which is then resampled into the grid.

Fusion: We obtain the final neural map by fusing the set of encoded maps { M n } using a cell-wise max-pooling operation, i.e ., M ij = max n M n ij ∀ ( i, j ) ∈ I × J . This can combine maps with different spatial extents, which is essential to scale to large areas. The max aggregation picks the best estimate among all inputs for each feature channel and thus handles partial observations, such as when the road surface cannot be resolved in overhead images because it is occluded by trees.

2.2 Ground-level image encoder

We design a single module, Φ SV , that can arbitrarily encode one or multiple images, ordered or not. Φ SV first fuses the image data into 3D space and later projects it vertically into the map plane (Fig. 4). This design can handle arbitrary ground geometries and accurately resolve the 2D location of overhanging 3D structures, like street lights. The 3D fusion leverages both multi-view geometry and strong monocular cues learned end-to-end. Φ SV can thus resolve objects that are observed by a single image, while maximizing accuracy when multiple observations are available.

Monocular inference: We consider an unordered set of N images { I n } , N ≥ 1 . Each image n is encoded independently by a CNN Φ I into a C -dimensional feature image F n ∈ R H × W × C . Φ I also estimates a pixel-wise depth S n ∈ R H × W × D as a score over D depth planes along the ray of each

compressed

X

k

point

column

map cell

vertical

pooling

fusion

MLP

occ.

score

image

features

3D

occupancy

volume

view

#1

view

#2

neural cell

M

ij

CNN

F

1

k

S

1

k

k

+

pixel. S n is similar to a frustum-aligned occupancy volume [78, 67] but contains unnormalized logits of a depth distribution. Instead of regressing a single value, this encodes the full depth uncertainty along the ray and thus allows Φ I to provide meaningful multi-modal estimates. We distribute the depth planes uniformly in log space to correlate with the uncertainty of monocular depth estimation [3, 78].

Multi-view fusion: To fuse information in 3D, we define K horizontal planes at heights { z k } , which are uniformly distributed within a range of interest defined with respect to the height of the camera [85, 50], e.g ., from 4 m below to 8 m above. For a 2D map cell ( i, j ) ∈ I × J , we consider its center point ( x, y ) and a column of 3D points { P k = ( x, y, z k ) } . For each 3D point k , we define the subset of views that best observe it as N k ⊆ { 1 . . . N } , e.g ., those that are closest spatially. We project the point to each of these views, obtain a 2D observation p n k = Π n ( P k ) , and sample the corresponding feature image with bi-linear interpolation: F n k = F n [ p n k ] . Given the depth d n k of P k in the corresponding view, we also tri-linearily interpolate a score from the depth prior: S n k = S n [ p n k , d n k ] . Intuitively, S n k is low if the 3D point is in free space or is occluded in view n . Following common practice in learned multi-view stereo [110, 106], we then compute feature consistency statistics, as mean and variance ( µ k , σ k ) ∈ R C , weighted by the depth priors:

A Multi-Layer Perceptron (MLP) fuses this information into a feature X k , which is finally pooled across all points in the column, resulting in a neural map cell M ij :

Adding the maximum depth score differentiates free and occupied space when the point is observed by a single image. This makes it possible to use the same model for single images and sequences.

By tightly combining 3D geometry and representation learning, our approach leverages both monocular priors and multi-view information, while past research on 2D mapping or 3D reconstruction typically relies on only one of the two. Compared to expensive Transformers [102] or 3D CNNs [33], we show that a simpler, lightweight MLP is effective at fusing multi-view information, inspired by [82]. Compared to top-down 2D CNNs that squash the vertical dimension [34, 70], this MLP is more expressive and makes our neural maps equivariant to 2D translations and rotations and invariant to translations along the vertical axis. Overall, this simple design enables scaling to very large scenes, which is critical to provide hard negatives for contrastive learning and ultimately learn rich semantics.

3 Learning from pose supervision

Alignment as contrastive learning: We want neural maps to encode high-level semantic information about the environment. Given recent advances in self-supervised learning [14, 64], we hypothesize that this can emerge from learning distinctive features that distinguish one location from another and that are invariant to viewpoint and temporal appearance changes. Intuitively, good maps help us identify where we are . More generally, good maps are such that we can unambiguously align them

<latexi sh

1_b

64="TZ

yuJO

G

dn

>A

CQH

c

V

7

MwF

qY

S

oU

K

2

k5

0

f

/

D

B

8

R

9

X

z

3

m

j

N

I

P

<latexi sh

1_b

64="2u

z

j90

RAoGH

VP

gIYQ

>

C

n

c

DLT

MwENyU

vO

KJq

5

B

rk

F

7

/

3

X

m

Wp

Z

8

d

S

+

f

<latexi sh

1_b

64="Cmv

o

Nnwq

d

AHT

8S

BE9Q

>

R

c

VDL

M

yU

O

KJ

5

g

ru

2Y

f

X+

F

j

P

z

k

I

3

W

G

0

7

p

<latexi sh

1_b

64="S2LHogcuq zTy+r

Bm

G

JVU

>A

CR

D

MwE

v

kK

I

O

n

/

F

9

X

5

P

N

8

W

7

Q

Z

f

0

j

d

Y

p

<latexi sh

1_b

64="2m

9A

InY

Q

MRG

ykZuO8

CW

>

3

c

VDLSg

F

Ud

z

q

r

NJp

w

/

7

T

0

E

K

5H

o

B

+

P

vX

f

j

when inferred from partial inputs. Consider neural maps M Q and M R obtained from two disjoint subsets of inputs, the query Q and the reference R . In camera pose estimation, Q corresponds to a single ground-level image and R to a sequence of images with an aerial tile. Because our encoder is flexible, we can use the same shared model to encode Q and R (Fig. 2). M Q is defined over a grid G Q ∈ R I × J × 2 in a local coordinate frame, e.g ., aligned with the query camera, where G Q ij is the center point of cell ( i, j ) , while M R is defined in the world frame.

We define a score function E ( T ; M Q , M R ) : SE(2) → R that evaluates the consistency between M Q and M R given an estimate of their 3-DoF relative pose R T Q ∈ SE(2) . To distinguish the ground-truth pose R T ∗ Q from K other, incorrect poses { R T k Q } , we want to increase E ( R T ∗ Q ) and decrease E ( R T k Q ) (omitting M Q and M R for brevity). This corresponds to a contrastive learning problem, for which we minimize the InfoNCE loss [63]

where τ is a learnable temperature parameter. Neural maps are trained end-to-end and require only relative poses R T ∗ Q , which can be easily obtained at a large scale using photogrammetry [45, 37].

Featuremetric pose scoring: A linear layer projects each neural map M to a lower-dimensional, L2-normalized map ¯ M . This creates an information bottleneck that encourages compact features. The score E evaluates the consistency of two neural maps as the similarity of each cell after warping:

where R T Q transforms a grid point from coordinate frames Q to R and [ · ] interpolates the map at this location. max clips negative scores to zero to reduce the impact of outliers, as in robust optimization.

Negative sampling: A critical and well-studied aspect of contrastive learning is the selection of negative samples [35, 98, 108]. Hard negatives should be high-likelihood but incorrect predictions, so as to push the probability mass to the ground truth. Random poses can be easily distinguished and exhaustive voting in the 3-DoF pose space is computationally infeasible at high resolution [78, 6, 28]. Instead, we use RANSAC [29] to sample poses that are consistent with the predicted features. We sample pairs of 2D-2D correspondences between all cells of both neural maps and solve for the relative pose using the Kabsch algorithm [43]. Inspired by PROSAC [19], we sample a correspondence between cells ( i, j ) and ( k, l ) based on its feature similarity with probability P ijkl = softmax ijkl ( ¯ M Q ⊤ ij ¯ M R kl / τ ) . Unlike NG-RANSAC [10], gradients are propagated through the scoring rather than the sampling and are thus much smoother. Because the sampling and scoring mirror similar featuremetric errors, negative samples become harder as the learning proceeds.

Inference-time alignment: SNAP can estimate the unknown 3-DoF relative pose between any two neural maps. We estimate each map in the sensor coordinate frame, establish tentative correspondences by matching their cells, sample pose hypotheses, and select the pose with the highest score. This includes single-image positioning, where the query map M Q covers the camera frustum. The vertical pooling requires that the gravity direction is known, which is a reasonable assumption for applications like Augmented Reality (AR) and robotics [56, 113, 78]. Our framework also applies more generally to aligning any pair of inputs, including sequence-to-sequence and aerial-to-ground registration, which is required in the first place to pose mapping data in a common reference frame.

4 Related work

Visual positioning is most commonly tackled with geometric approaches [80, 40, 74] that rely on point correspondences across images and sparse 3D point clouds built with SfM [83, 2]. They then estimate the 6-DoF query pose with a robust solver [29, 20, 21, 19, 15, 5] from correspondences with the reference model or images. Such correspondences are most often estimated by sparse local features [54, 4]. This process is complex and end-to-end back-propagation is impractical [8]. Past works have thus focused on learning specific components like feature extraction [111, 58, 23, 24,

99, 69, 26, 101, 105, 55], matching [112, 115, 75, 91, 41, 117, 107, 53], and pose [104, 76] or point cloud refinement [52]. Coarse GPS location and gravity direction are commonly assumed to be known [113, 56, 93]. In AR and robotics, the height of the camera can be estimated as the distance to the ground in a local SLAM reconstruction [78]. These assumptions reduce the problem to 3-DoF estimation and make it more amenable to end-to-end learning. MapNet [38] also learns end-to-end 3-DoF visual mapping and localization but requires sequences of depth inputs. Recent works leverage overhead instead of ground-level images [87, 86, 109, 28]. They easily scale to large scenes but only in open-sky areas. Their accuracy is also limited by the low resolution of aerial imagery. Our work combines the strengths of both ground-level and overhead imagery by learning end-to-end how to best fuse them for 3-DoF positioning. Our differentiable pose estimation, based on RANSAC, is more efficient [38, 28, 78], robust [87], and stable [10, 8] than previous approaches.

Semantic representations can largely benefit loop closure [84] and pose estimation [100]. OrienterNet [78] learns 3-DoF positioning end-to-end from public 2D semantic maps that are more compact yet detailed enough for localization. Its accuracy is however limited because these maps have low spatial accuracy and are infrequently updated. It is also also restricted to few, explicit semantic classes that are often not discriminative. Differently, [49] learns finer-grained semantic classes for temporal and viewpoint consistency. Our work instead learns implicit semantics from posed imagery by combining end-to-end self-supervised learning with large amounts of data. This boosts the positioning accuracy and is an effective pre-training for semantic tasks.

Neural scene representation is an active topic of research. MLPs [57, 96] and tokens [73] are compact but lack geometric inductive bias. 3D voxel grids are more expressive and thus popular for reconstruction [66, 60, 92, 9, 119], rendering [59, 95], and semantic perception [17, 102, 13, 7] but are expensive to store and thus often restricted to small scenes. 2D grids, or Bird's-Eye Views (BEV), are more compact and thus scale to larger outdoor scenes by compressing the information along the vertical axis. Neural BEVs can be learned from images for supervised semantic tasks [72, 50, 34, 67, 70], 3D reconstruction [66], self-supervised view synthesis [85], and 3-DoF positioning [78, 28, 38]. These approaches assume planar scenes or rely on monocular priors only, even if multiple views are available. Instead, we combine these priors with multi-view fusion [110, 82, 106] to leverage information from image sequences and better resolve objects in large scenes.

Self-supervised learning leverages unlabeled datasets to learn representations useful for down-stream tasks. Many works focus on image- or pixel-level contrastive learning for semantic tasks [63, 35, 14, 62, 36]. View synthesis from few images typically learns lower-level representations [57, 85]. Some works [49, 89] learn features for image matching across appearance changes. CoCoNets [48] learns representations for 3D scenes but requires perfect, synthetic depth maps. We learn high-level contrastive scene representations from posed images and show that it translates to semantic mapping.

5 Experiments

Data: StreetView images are captured by rigs of 6 rolling-shutter cameras mounted on cars or on backpacks worn by pedestrians [18], which results in a wide diversity of viewpoints in street-level scenes. Multi-view 'frames' are captured synchronously every ∼ 5m. Sequences are captured between 2017 and 2022. We build mapping segments only from car sequences by partitioning each sequence into groups of 36 images that face either the left or right side of the road. We define each map grid as a 64 × 16 m tile aligned with the segment mid-frame, in which we render an aerial orthophoto with 20 cm ground sample distance. Query images are sampled from different sequences, captured from cars or backpacks, based on their frustum overlap, and are often taken years apart. We train with 2.5M segments and ∼ 50M queries from 11 cities across the world: Barcelona, London, Paris (Europe), New York, San Francisco (North America), Rio de Janeiro (South America), Manila, Singapore, Taipei, Tokyo (Asia), and Sydney (Oceania), reserving some areas in each city for validation. We test on 6 different cities (Amsterdam, Melbourne, Mexico City, Osaka, São Paulo, and Seattle), with 4k queries per city. This covers 5 continents, while academic localization benchmarks focus on tourism landmarks [42] or single cities in Europe or the US [81, 116, 77] - see details in Appendix D.

Training and implementation: In the ground-level encoder, Φ I is a U-Net [71] with a BiT ResNet backbone [46], pre-trained as in [114], and an FPN decoder [51], initialized randomly. We consider two models with different backbones: a 'large' R152x2 (353M parameters) and a 'small' R50x1 (84M parameters). Φ OV is a similarly-defined R50x1+FPN. In multi-view fusion (Sec. 2.2) we use

0

1

2

3

4

5

0

25

50

75

Recall [%]

easy (25%)

0

1

2

3

4

5

Position error [meter]

medium (50%)

0

1

2

3

4

5

hard (25%)

0

2

4

6

8

10

0

25

50

75

Recall [%]

0

2

4

6

8

10

Orientation error [degree]

0

2

4

6

8

10

SfM + SuperGlue

Semantic maps (OrienterNet)

Ours

D = 32 depth planes and K = 60 height planes { z k } uniformly distributed within 12 m. Neural maps M and matching maps ¯ M have dimensions 128 and 32, respectively, and are defined over 64 × 16 m grids with 20 cm ground sample distance. Query BEVs have a maximum depth of 16 m. At training time, neural maps are built from one aerial tile and one SV segment, with each of the two randomly dropped, similarly to dropout [90]. We use a subset of N = 20 views, some of them at a ± 60° angle, which we empirically found provides a good coverage/memory trade-off. See details in Appendix E.

Visual positioning: We build a map for each segment using all 36 views and evaluate the 3-DoF query pose in terms of position and orientation errors. While many academic benchmarks use much larger mapping areas, we argue that GPS and motion priors often make this unnecessary for practical applications [56]. We slice the results by difficulty in terms of query-scene overlap based on the distance between the query and its closest map view, in position ∆ t , and orientation ∆ θ . We split in the data into 3 groups: 'easy' ( ∆ t< 10 m and ∆ θ< 45°, ∼ 25% of the data), 'hard' ( ∆ t> 10 m and ∆ θ> 60°, ∼ 25%), and 'medium' (the remaining ∼ 50%). We compare our approach to hloc [74], a state-of-the-art [81, 77] structure-based 6-DoF localization system based on COLMAP [83], a popular Structure-from-Motion framework, with correspondences estimated by either RootSIFT [54, 4] or SuperPoint+SuperGlue [23, 75], a learned feature and matcher. Note that these approaches can only leverage ground-level imagery. We match the query to all map images, without using hloc's

road building

road

building

street light

pole

tree

street light

pole

tree

road

building

street light

pole

tree

50

100

Recall [%]

pre-trained

scratch

road

sidewalk

building

tree

crosswalk

fence

pole

traffic sign

street light

traffic light

0.1

1

10

Distribution [%]

Ground truth

Prediction

crosswalk

sidewalk

road

terrain

building

fence

pole

tree

traffic_sign

traffic_light

street_light

retrieval component, and estimate the query's pose using RANSAC and a P2P solver with gravity constraint [94]. We evaluate the 6-DoF pose projected to 3-DoF. We also evaluate OrienterNet [78], which matches a query BEV with a semantic map. We re-implement and train it on overhead semantic rasters derived from SfM points with semantic labels obtained by fusing 2D image segmentations. Note that OrienterNet was originally trained on OpenStreetMap, which has limited coverage of small objects. While our rasters are noisy, they provide a consistent, global coverage of fine-grained classes like tree, streetlight, poles, etc. We checked our implementation with the authors. We also evaluate versions of SNAP trained with only either ground-level or aerial imagery - the latter is an extreme case of cross-view localization, similar to [28].

Fig. 5 and Tab. 1 show that SNAP outperforms the state of the art, COLMAP with SuperPoint+SuperGlue, by a large margin: 25% relative. Structure-based approaches are more accurate for easy queries but significantly worse for hard ones (Fig. 3). Using ground-level imagery is crucial in most localization scenarios and performs ∼ 46% relative better than using aerial imagery, whereas our multi-modal model performs ∼ 8% relative better than the StreetView-only variant. We justify our design decisions using an ablation study in Appendix A. We report detailed results per city and per sequence type in Appendix C. Our framework is also efficient, as mapping takes 223 ms per segment and 6 ms per aerial tile, estimating a query BEV takes 14 ms and localizing it takes 86 ms, on an A100 GPU. In comparison, matching with SuperGlue takes 100ms per pair for 36 pairs per query, and is thus 36 times slower. Each tile of our matching maps has size 1.6 MB in fp16, while storing SuperPoint descriptors requires 5.3 MB on average.

Semantic mapping: We show that SNAP's neural maps are an effective pre-training for 2D semantic mapping. Existing approaches rely on ground truth 2D semantic rasters derived from the segmentation of LiDAR 3D point clouds. These are manually labeled, which is too expensive to generate enough data to train from scratch models that generalize across countries, sensors, seasons, and times of the day. Existing datasets [12, 7] thus rarely span more than a few cities and overfit supervised models to the local appearance. Instead, our self-supervised pre-training learns better features from a much larger dataset of posed imagery, which is much cheaper to acquire at scale. The information bottleneck forces SNAP to learn unified representations for objects, like street crossings or lights, that look very different across countries, and would require larger amounts of labeled data. Fig. 6-left shows a 2D t-SNE [103] visualization of SNAP's neural maps at points sampled on a few types of objects common in street scenes, according to their ground-truth semantic label. Points of the same class are clustered together. This clearly shows that neural maps learn to distinguish these objects without any semantic supervision, even if they are geometrically similar, e.g ., tree vs pole.

To evaluate the pre-training, we train a tiny CNN to predict semantic rasters from pre-trained neural maps, keeping SNAP frozen. We compare this to training the entire model from scratch (with the same backbones initialization [46, 114]). We derive 3k 64 × 16 m ground truth rasters from LiDAR point clouds captured by StreetView cars in 84 cities across the world. We train with 2k examples and report the recall of both approaches on 1k test examples in Fig. 6-center. Pre-training consistently

London

Paris

Rio de Janeiro

Manhattan

Taipei

Tokyo

(a) image

(b) expected log-depth

(c) confidence

(d) BEV

yields better results for every class, with larger gains on more difficult/infrequent classes. While training from scratch massively overfits to such small dataset, our neural maps encode enough information to reach recalls over 70%. We show qualitative examples in Fig. 6-right and Appendix B.

Monocular priors: We visualize in Fig. 7 the occupancy predicted by SNAP as depth and confidence maps. SNAP learns sensible priors over the geometry of street scenes from only pose supervision.

Limitations: Our approach is not as accurate as structure-based methods given easy queries closer to map images (Fig. 5). We hypothesize this is partly due to operating at lower image resolutions. It also assumes gravity direction and a location prior, which are reasonable assumptions but restrict its use.

6 Conclusion

We present SNAP, a novel approach to build semantic, 2D neural maps from multi-modal inputs and train it by simply learning to align two neural maps in a contrastive framework. This simple objective yields a model that can localize queries beyond the reach of the state of the art in structure-based matching by discovering high-level semantics from self-supervision. Our neural maps are easily interpretable and provide an effective pre-training towards unlocking semantic understanding at scale.

Broader impact: This work has implications to privacy and surveillance. However, our 2D maps are too compact to preserve personal identifiable information, and likely more difficult to invert than point clouds [68], which despite ongoing efforts [88, 31, 32, 61, 118] remain susceptible to attacks [16].

Acknowledgements: We thank Bernhard Zeisl, Songyou Peng, Rémi Pautrat, and Michał Tyszkiewicz for their valuable feedback, Manuel Cabral and Johann Volz for providing useful code reviews, Tianqi Fan for helping processing the data, Lucas Beyer for providing the pre-trained ResNet backbones, Thomas Funkhouser and Kyle Genova for providing the ground truth semantic labels, and Arjun Karpur for helping run SuperGlue.

References

Ammar Abbas and Andrew Zisserman. A Geometric Approach to Obtain a Bird's Eye View From an Image. ICCV workshop , 2019. 17, 18

Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle Adjustment in the Large. ECCV , 2010. 6

Manuel López Antequera, Pau Gargallo, Markus Hofinger, Samuel Rota Bulò, Yubin Kuang, and Peter Kontschieder. Mapillary Planet-Scale Depth Dataset. ECCV , 2020. 5

Relja Arandjelovi´ c and Andrew Zisserman. Three things everyone should know to improve object retrieval. CVPR , 2012. 2, 6, 8

Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiˇ rí Matas. MAGSAC++, a Fast, Reliable and Accurate Robust Estimator. CVPR , 2020. 6

Ioan Andrei Barsan, Shenlong Wang, Andrei Pokrovsky, and Raquel Urtasun. Learning to Localize Using a LiDAR Intensity Map. CoRL , 2018. 6

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. ICCV , 2019. 7, 9

Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced Feature Points: Optimizing Feature Detection and Description for a High-Level Task. CVPR , 2020. 2, 6, 7

Aljaž Božiˇ c, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. TransformerFusion: Monocular RGB Scene Reconstruction using Transformers. NeurIPS , 2021. 7

Eric Brachmann and Carsten Rother. Neural-Guided RANSAC: Learning Where to Sample Model Hypotheses. ICCV , 2019. 2, 6, 7

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. http://github. com/google/jax , 2018. 27

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A Multimodal Dataset for Autonomous Driving. CVPR , 2020. 9

Anh-Quan Cao and Raoul de Charette. MonoScene: Monocular 3D Semantic Scene Completion. CVPR , 2022. 7

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV , 2021. 5, 7

Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, and Marc Pollefeys. AdaLAM: Revisiting Handcrafted Outlier Detection. ECCV , 2020. 6

Kunal Chelani, Fredrik Kahl, and Torsten Sattler. How Privacy-Preserving are Line Clouds? Recovering Scene Details from 3D Lines. CVPR , 2021. 10

Ricson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry-Aware Recurrent Neural Networks for Active Visual Recognition. NeurIPS , 2018. 7

Danny Cheung. Mapping stories with a new Street View Trekker. https://blog.google/ products/maps/mapping-stories-new-street-view-trekker/ , 2018. 4, 7

Ondˇ rej Chum and Jiˇ rí Matas. Matching with PROSAC - Progressive Sample Consensus. CVPR , 2005. 6

Ondˇ rej Chum, Jiˇ rí Matas, and Josef Kittler. Locally Optimized RANSAC. GCPR / DAGM Symposium , 2003. 6

Ondˇ rej Chum, Tomas Werner, and Jiˇ rí Matas. Two-view geometry estimation unaffected by a dominant plane. CVPR , 2005. 6

Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A JAX Library for Computer Vision Research and Beyond. CVPR , 2022. 27

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-Supervised Interest Point Detection and Description. CVPR workshop , 2018. 2, 6, 8

Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-Net: A Trainable CNN for Joint Detection and Description of Local Features. CVPR , 2019. 6

Mihai Dusmanu, Ondrej Miksik, Johannes L Schönberger, and Marc Pollefeys. CrossDescriptor Visual Localization and Mapping. ICCV , 2021. 2

Patrick Ebel, Anastasiia Mishchuk, Kwang Moo Yi, Pascal Fua, and Eduard Trulls. Beyond Cartesian Representations for Local Descriptors. ICCV , 2019. 7

Esri. Introduction to Ortho Mapping - ArcGIS Pro documentation. https://pro.arcgis. com/en/pro-app/latest/help/data/imagery/introduction-to-ortho-mapping. htm , 2023. 4

Florian Fervers, Sebastian Bullinger, Christoph Bodensteiner, Michael Arens, and Rainer Stiefelhagen. Uncertainty-aware Vision-based Metric Cross-view Geolocalization. CVPR , 2023. 6, 7, 9

Martin A Fischler and Robert C Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM , 1981. 6

Kyle Genova, Xiaoqi Yin, Abhijit Kundu, Caroline Pantofaru, Forrester Cole, Avneesh Sud, Brian Brewington, Brian Shucker, and Thomas Funkhouser. Learning 3D Semantic Segmentation with only 2D Image Supervision. 3DV , 2021. 27

Marcel Geppert, Viktor Larsson, Pablo Speciale, Johannes L Schönberger, and Marc Pollefeys. Privacy Preserving Structure-from-Motion. ECCV , 2020. 10

Marcel Geppert, Viktor Larsson, Johannes L Schönberger, and Marc Pollefeys. Privacy Preserving Partial Localization. CVPR , 2022. 10

Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. CVPR , 2018. 5

Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. SimpleBEV: What Really Matters for Multi-Sensor BEV Perception? ICRA , 2023. 5, 7, 17, 18

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. CVPR , 2020. 6, 7

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. CVPR , 2022. 7

Jared Heinly, Johannes L. Schonberger, Enrique Dunn, and Jan-Michael Frahm. Reconstructing the World* in Six Days *(As Captured by the Yahoo 100 Million Image Dataset). CVPR , 2015. 2, 6

Joao F Henriques and Andrea Vedaldi. MapNet: An Allocentric Spatial Memory for Mapping Environments. CVPR , 2018. 7

Martin Humenberger, Yohann Cabon, Nicolas Guerin, Julien Morat, Vincent Leroy, Jérôme Revaud, Philippe Rerole, Noé Pion, Cesar de Souza, and Gabriela Csurka. Robust Image Retrieval-based Visual Localization using Kapture. arXiv preprint , 2020. 2

Arnold Irschara, Christopher Zach, Jan-Michael Frahm, and Horst Bischof. From Structurefrom-Motion Point Clouds to Fast Location Recognition. CVPR , 2009. 6

Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. COTR: Correspondence Transformer for Matching Across Images. ICCV , 2021. 7

Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiˇ rí Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls. Image Matching across Wide Baselines: From Paper to Practice. IJCV , 2021. 2, 7

Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A , 1976. 6

Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint , 2014. 27

Bryan Klingner, David Martin, and James Roseborough. Street View Motion-from-Structurefrom-Motion. ICCV , 2013. 2, 6

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. ECCV , 2020. 7, 9

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the Carbon Emissions of Machine Learning. NeurIPS workshop on Climate Change AI , 2019. 27

Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. CoCoNets: Continuous Contrastive 3D Scene Representations. CVPR , 2021. 7

Måns Larsson, Erik Stenborg, Carl Toft, Lars Hammarstrand, Torsten Sattler, and Fredrik Kahl. Fine-Grained Segmentation Networks: Self-Supervised Segmentation for Improved Long-Term Visual Localization. ICCV , 2019. 7

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning Bird's-Eye-View Representation From Multi-Camera Images Via Spatiotemporal Transformers. ECCV , 2022. 5, 7

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. CVPR , 2017. 7

Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Marc Pollefeys. Pixel-Perfect Structure-from-Motion with Featuremetric Refinement. ICCV , 2021. 7

Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. ICCV , 2023. 2, 7

David G Lowe. Distinctive Image Features from Scale-Invariant Keypoints. IJCV , 2004. 2, 6, 8

Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan. ASLFeat: Learning Local Features of Accurate Shape and Localization. CVPR , 2020. 7

Simon Lynen, Bernhard Zeisl, Dror Aiger, Michael Bosse, Joel Hesch, Marc Pollefeys, Roland Siegwart, and Torsten Sattler. Large-scale, real-time visual-inertial localization revisited. IJRR , 2020. 2, 6, 7, 8

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV , 2020. 7

Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiˇ rí Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. NeurIPS , 2017. 2, 6

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. , 2022. 7

Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich. Atlas: End-to-End 3D Scene Reconstruction from Posed Images. ECCV , 2020. 7

Tony Ng, Hyo Jin Kim, Vincent T Lee, Daniel DeTone, Tsun-Yi Yang, Tianwei Shen, Eddy Ilg, Vassileios Balntas, Krystian Mikolajczyk, and Chris Sweeney. NinjaDesc: Content-Concealing

Visual Descriptors via Adversarial Learning. CVPR , 2022. 10

Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised Learning of Dense Visual Representations. NeurIPS , 2020. 7

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv preprint , 2018. 6, 7

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint , 2023. 5

Shuxue Peng, Zihang He, Haotian Zhang, Ran Yan, Chuting Wang, Qingtian Zhu, and Xiao Liu. MegLoc: A Robust and Accurate Visual Localization Pipeline. arXiv preprint , 2021. 2

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional Occupancy Networks. ECCV , 2020. 7

Jonah Philion and Sanja Fidler. Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV , 2020. 5, 7

Francesco Pittaluga, Sanjeev J Koppal, Sing Bing Kang, and Sudipta N Sinha. Revealing Scenes by Inverting Structure from Motion Reconstructions. CVPR , 2019. 10

Jerome Revaud, Philippe Weinzaepfel, César Roberto de Souza, and Martin Humenberger. R2D2: Repeatable and Reliable Detector and Descriptor. NeurIPS , 2019. 2, 7

Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic Feature Transform for Monocular 3D Object Detection. BMVC , 2019. 5, 7

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. MICCAI , 2015. 4, 7

Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden. Translating Images into Maps. ICRA , 2022. 7

Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, and Andrea Tagliasacchi. Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations. CVPR , 2022. 7

Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From Coarse to Fine: Robust Hierarchical Localization at Large Scale. CVPR , 2019. 2, 6, 8

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning Feature Matching with Graph Neural Networks. CVPR , 2020. 2, 7, 8

Paul-Edouard Sarlin, Ajaykumar Unagar, Måns Larsson, Hugo Germain, Carl Toft, Victor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the Feature: Learning Robust Camera Localization from Pixels to Pose. CVPR , 2021. 2, 7

Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L. Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Marc Pollefeys. LaMAR: Benchmarking Localization and Mapping for Augmented Reality. ECCV , 2022. 7, 8

Paul-Edouard Sarlin, Daniel DeTone, Tsun-Yi Yang, Armen Avetisyan, Julian Straub, Tomasz Malisiewicz, Samuel Rota Bulo, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. OrienterNet: Visual Localization in 2D Public Maps with Neural Matching. CVPR , 2023. 2, 5, 6, 7, 8, 9, 27

Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Fast Image-Based Localization using Direct 2D-to-3D Matching. ICCV , 2011. 2

Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Improving Image-Based Localization by Active Correspondence Search. ECCV , 2012. 6

Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, Fredrik Kahl, and Tomas Pajdla. Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions. CVPR ,

2018. 7, 8

Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. SimpleRecon: 3D reconstruction without 3D convolutions. ECCV , 2022. 5, 7

Johannes Schönberger and Jan-Michael Frahm. Structure-from-Motion Revisited. CVPR , 2016. 2, 6, 8

Johannes L Schönberger, Marc Pollefeys, Andreas Geiger, and Torsten Sattler. Semantic Visual Localization. CVPR , 2018. 7

Prafull Sharma, Ayush Tewari, Yilun Du, Sergey Zakharov, Rares Ambrus, Adrien Gaidon, William T Freeman, Fredo Durand, Joshua B Tenenbaum, and Vincent Sitzmann. Seeing 3D Objects in a Single Image via Self-Supervised Static-Dynamic Disentanglement. NeurIPS , 2022. 5, 7, 17, 18

Yujiao Shi and Hongdong Li. Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image. CVPR , 2022. 7, 17, 18

Yujiao Shi, Xin Yu, Dylan Campbell, and Hongdong Li. Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching. CVPR , 2020. 7

Pablo Speciale, Johannes L Schonberger, Sing Bing Kang, Sudipta N Sinha, and Marc Pollefeys. Privacy Preserving Image-Based Localization. CVPR , 2019. 10

Jaime Spencer, Richard Bowden, and Simon Hadfield. Same Features, Different Day: Weakly Supervised Feature Learning for Seasonal Invariance. CVPR , 2020. 7

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. JMLR , 2014. 8

Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: DetectorFree Local Feature Matching with Transformers. CVPR , 2021. 7

Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video. CVPR , 2021. 7

Linus Svärm, Olof Enqvist, Fredrik Kahl, and Magnus Oskarsson. City-Scale Localization for Cameras with Known Vertical Direction. TPAMI , 2017. 7

Chris Sweeney, John Flynn, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. Efficient Computation of Absolute Pose for Gravity-Aware Augmented Reality. ISMAR , 2015. 9

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes. CVPR , 2021. 7

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-NeRF: Scalable Large Scene Neural View Synthesis. CVPR , 2022. 7

TensorFlow Datasets. TensorFlow Datasets, a collection of ready-to-use datasets. https: //www.tensorflow.org/datasets , 2019. 27

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What Makes for Good Views for Contrastive Learning? NeurIPS , 2020. 6

Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas. SOSNet: Second Order Similarity Regularization for Local Descriptor Learning. CVPR , 2019. 7

Carl Toft, Erik Stenborg, Lars Hammarstrand, Lucas Brynte, Marc Pollefeys, Torsten Sattler, and Fredrik Kahl. Semantic Match Consistency for Long-Term Visual Localization. ECCV , 2018. 7

Michał Tyszkiewicz, Pascal Fua, and Eduard Trulls. DISK: Learning local features with policy gradient. NeurIPS , 2020. 2, 7

Michał J Tyszkiewicz, Kevis-Kokitsi Maninis, Stefan Popov, and Vittorio Ferrari. RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. ECCV , 2022. 5, 7

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR , 2008. 9

Lukas Von Stumberg, Patrick Wenzel, Qadeer Khan, and Daniel Cremers. GN-Net: The Gauss-Newton loss for multi-weather relocalization. RA-L , 2020. 7

Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely. Learning Feature Descriptors using Camera Pose Supervision. ECCV , 2020. 7

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. IBRNet: Learning Multi-View Image-Based Rendering. CVPR , 2021. 5, 7, 17, 18

Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen. MatchFormer: Interleaving Attention in Transformers for Feature Matching. ACCV , 2022. 7

Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling Matters in Deep Embedding Learning. ICCV , 2017. 6

Zimin Xia, Olaf Booij, Marco Manfredi, and Julian FP Kooij. Visual Cross-View Metric Localization with Dense Uncertainty Estimates. ECCV , 2022. 7

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth Inference for Unstructured Multi-view Stereo. ECCV , 2018. 5, 7, 17, 18

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. LIFT: Learned Invariant Feature Transform. ECCV , 2016. 2, 6

Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to Find Good Correspondences. CVPR , 2018. 7

Bernhard Zeisl, Torsten Sattler, and Marc Pollefeys. Camera Pose Voting for Large-Scale Image-Based Localization. ICCV , 2015. 6, 7

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. CVPR , 2022. 7, 9

Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning Two-View Correspondences and Geometry Using Order-Aware Network. ICCV , 2019. 7

Zichao Zhang, Torsten Sattler, and Davide Scaramuzza. Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis. IJCV , 2021. 7

Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive Correspondence Pruning by Consensus Learning. ICCV , 2021. 7

Qunjie Zhou, Sérgio Agostinho, Aljoša Ošep, and Laura Leal-Taixé. Is Geometry Enough for Matching in Visual Localization? ECCV , 2022. 10

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. CVPR , 2022. 7

Appendix

A Design decisions

In this section, we explain our design decisions and support them with an ablation study.

Constraints: Learning features that are discriminative requires sufficiently challenging negative pose samples. These arise from viewpoints that are visually similar to the ground truth viewpoint, for example due to repeated patterns, lack of distinctive features, or occlusions. In order to find sufficiently difficult negative samples in a training example, the map should be as large as possible. The size of the map and the number of mapping images are however limited by the amount of GPU memory available. We therefore found it critical to find the right balance between the size of the map, its spatial resolution, the number of views, and the batch size, as also reported in [34].

We found that a simple and lightweight model that saves memory is beneficial over a complex model that requires reducing the size of the map, the number of views, or the batch size. We thus favor explicit constraints by geometry (camera projection, 3D occupancy) rather than flexible but heavy mechanisms like attention or 3D convolutions. The effectiveness of our design shows that such complexity is not required. This enables the training of high-capacity models with maps of 64 × 16 m, at a resolution of 20 cm ( ∼ 25k cells per map), from 20 images, and within 20 GB of GPU memory per example. Despite extensive memory optimizations, including the use of gradient checkpointing and mixed-precision training, using larger maps for training remains challenging.

Ground-level encoder: We consider four alternative designs for the ground-level encoder Φ SV .

(a) Fixed-height plane. Instead of lifting the image information to 3D and pooling it vertically, the geometry of the scene can be approximated from a ground plane [1]. Driving applications commonly assume that this plane is horizontal and at a fixed height below the camera [86]. We train a variant that aggregates the multi-view information on a plane 2.5 m below the camera. This approach can hardly resolve the spatial location of overhanging structures, like street lights. It furthermore introduces distortions in the BEV if the image is not gravity aligned, which is the case for many capture platforms (such as StreetView backpacks or consumer phones), or the ground is not planar, as in many real environments.

(b) No monocular occupancy. Multi-view stereo [106, 110] typically does not explicitly leverage monocular geometry priors. We thus train a variant that omits the weighting by occupancy scores. Image features are thus painted identically along all depth planes for each ray and their mean and variance across all views is fed to the fusion MLP. This model can only leverage the bearing angles of point features (like poles) to disambiguate a pose but cannot resolve the location of line features.

(c) No multi-view variance. SimpleBEV [34] simply averages feature volumes obtained by painting image features along each ray. This corresponds to removing both variance and monocular priors from SNAP, making it even harder to resolve surfaces.

(d) Ray conditioning. Close to our approach, Sharma et al . [85] augment multi-view fusion with monocular cues, but do so by conditioning each ray feature by its 3D location, using an MLP that encodes the ray direction and camera-space coordinate. This requires evaluating the MLP for each observation of each point. Our approach, based on an occupancy volume, requires only performing a tri-linear interpolation for each observation, which is significantly cheaper. We train a variant based on this MLP conditioning, replacing the weighting by occupancy score. It increases the memory requirement by 4, since we use 4 observing views per point ( |N k | =4 ). We thus need to halve the batch size and the number of height planes.

We train all variants, including our model, for an identical number of steps. To save compute resources, we train for fewer steps than in the main paper (200k vs 400k) and we use the smaller ResNet-50 architecture with only ground-level inputs. Tab. 2 shows that each of these variants yields a lower positioning accuracy than our model.

Vertical pooling: In our design, the vertical pooling is performed with max pooling. Tab. 3-top shows that average pooling is significantly less effective. We hypothesize that averaging makes it harder to ignore features of points located in empty space. We found that pooling with an attention mechanism [85] performs similarly as max pooling despite the increase in computation. Harley

et al . [34] flatten the vertical elements with a space-to-depth (or pixel-shuffling) operation, followed by an MLP. This makes the model sensitive to a translation along the vertical axis, which rarely occurs in small driving datasets based on a few cars with identical specifications, but matters for heterogeneous data captured by backpacks and cars with widely different setups. We thus found that this approach yields a lower performance than a simpler pooling. This also makes it impossible to adjust the number of height planes at inference time.

Multi-modal fusion: Tab. 3-bottom shows that the choice of pooling operator makes little difference when fusing neural maps inferred from StreetView and aerial inputs.

B Qualitative examples

Visual positioning: Due to space constraints, we showcase only a handful of visual positioning examples in the main paper. We show additional visualizations of successfully localized queries in Figs. 8 to 10 and failure cases in Figs. 17 and 18. We also show some examples with non-linear mapping trajectories in Fig. 19, which our method is robust to. As in Fig. 3, we show the 3-DoF poses of map images (red) , the GT query pose (black) , and the pose predicted by SNAP (green) with its error ( ∆ t , ∆ R ). We visualize neural maps by projecting them to RGB using PCA.

Sequence to sequence alignment In the paper we focus on the alignment of single-image queries to maps built from multiple views, but SNAP can arbitrarily align any pair of neural maps. We show two examples of sequence to sequence alignment in Figs. 11 and 12. These maps are built from five views. For ease of understanding, we lift our neural maps into 3D using LiDAR, color-coding each LiDAR point with the neural map values of the cell it belongs to - note that this is stricly for visualization purposes, and our algorithm does not rely on LiDAR.

Semantic mapping: We show additional visualizations of the semantic maps predicted by our model after fine-tuning it in only 2k scenes in Figs. 14 and 15.

query

map images

t=0.6m

R=0.1°

O

query

map images

t=0.7m

R=1.3°

1

3

1

3

map images

query

t=0.2m

R=2.1°

2

A

0

1

query

map images

t=0.3m

R=1.2°

2

1

3

map images

query

t=0.5m

R=2.5°

map images

query

t=0.1m

R=0.1°

1

2

3

2

3

1

2

3

mu

map images

query

t=0.2m

R=1.1°

AVA

2

0

3

AVA

ANAAVAAVAAVAAV

map images

query

t=0.2m

R=0.5°

1

2

3

2

1

3

2

D

2

2

map images

3

map images

1

1

map images

3

query d) The query is localized from a difficult angle (medium).

map images

t=0.3m

R=1.4°

24

0

1

3

1

2

3

query t=1.6m

R=2.5°

t=0.6m

2

1

3

query

R=0.2°

query

R=1.8°

t=0.2m

Alignment error:

∆

t

=0.5m,

∆

R

=0.33

o

0'ST

SL

0'S

00

2.5

15

Large-scale mapping: We can easily build large tiles by 'stitching' smaller neural maps together via cell-wise max-pooling, similarly to how we fuse aerial and ground-level neural maps. Starting from multiple sequences posed in a common reference frame, Fig. 13 shows how neural maps are inferened for each of them and finally combined together.

C Detailed results

Fig. 21 shows the single-image positioning recall performance for each for the 6 test cities. The performance of the different approaches is fairly consistent across cities, with the exception of Osaka. We observe that Osaka has a large number of small, narrow streets, and StreetView sequences often feature close-ups of building facades: see Fig. 18-(c-d). This is somewhat out of distribution with the training set and could be solved by training on more data. It also explains why structured-based matching approaches are either very accurate (when there is sufficient visual overlap between query and reference views) or fail to register the image (when there isn't).

Fig. 22 shows the recall for queries taken by cameras mounted on either backpacks or cars. Backpack queries have a lower recall because they are typically taken from a sidewalk and thus have larger viewpoint differences (Fig. 16).

Alignment error:

∆

t

=0.3m,

∆

R

=0.05

o

15.0

12.5

10.0

7.5

5.0

2.5

0.0

-10

-5

0

10

15

20

fuse

2019

2022

aerial

2021

neural map tile

inputs

Ground truth

Prediction

crosswalk

sidewalk

road

terrain

building

fence

pole

tree

traffic_sign

traffic_light

street_light

The curb is mostly occluded by the parked cars and has very sparse ground truth labels. Notice how our neural maps encode the location of street lights that are hanging above the street.

Ground truth

Prediction

crosswalk

sidewalk

road

terrain

building

fence

pole

tree

traffic_sign

traffic_light

street_light

Ground truth

Prediction

crosswalk

sidewalk

road

terrain

building

fence

pole

tree

traffic_sign

traffic_light

street_light

Ground truth

Prediction

crosswalk

sidewalk

road

terrain

building

fence

pole

tree

traffic_sign

traffic_light

street_light

easy

medium

hard

all

0

20

40

60

Ratios of query platform in the test splits [%]

backpack

car

1 day

1 month

1 year

3 years

10 2

10 3

Number of test examples

Time difference between query and mapping data

D Dataset

We provide more details about the data used for the experiments conducted in the paper.

Platform type: Ground-level mapping sequences are selected from sequences captured by cars only. Query images are sampled from sequences captured by either cars or backpacks. For each map tile, we sample as many car queries as backpack queries, when possible. As cities are only partially covered by backpack sequences, the final ratio of backpack queries varies per city, from 20% to 40%. Fig. 16-left shows the ratio of both types for the different difficulty splits of the test set.

Map grid: We found that maps of size 64 × 16 m provide a good trade-off between visual diversity and memory consumption. Since multi-view frames are provided every ∼ 5 m, segments of 14 frames generally cover a length of 64 m. We therefore sample segments of 14 frames and define the map grid to be aligned with the average pose of the 7 th and 8 th side cameras. Given that most car segments are rectilinear, the map is most often aligned with the mapping views. In turns, the coverage of the map varies and can make the localization more challenging, which is generally a good property for both training and evaluation. Fig. 19 shows examples of maps with partial coverage.

Spatial distribution: For each of 11 training cities, we define disjoint training and validation regions. We show these regions in Fig. 20. We sample at most 5M training examples from each training region. The validation region corresponds to 5 (10 for New York) randomly-sampled level-14 S2 cells § , from which we sample 1024 examples. 4k test queries are sampled from 20 random S2 cells in each of the 6 test cities, as shown in Fig. 23. This corresponds to an average area of 5.6 km 2 per city.

Temporal distribution: Sequences used for mapping or localization are randomly drawn from the pool of available StreetView data. Fig. 16-right shows the distribution of temporal differences. This yields challenging queries with large appearance changes, e.g ., due to the seasons, or structural changes, e.g ., due to construction work. Aerial images are selected as captured as close as possible to the ground-level mapping images. The average time difference varies from 6 months to 3 years.

Ground truth semantic rasters: In the semantic mapping experiments, we used a small labeled dataset to train and evaluate our fine-tuning. We start with 1.5k segments of data captured in 84 cities across the world by StreetView cars equipped with LiDARs. We aggregate the measurements into point clouds that we label into 10 classes spanning objects encountered in urban environments. These classes include surfaces with a direct ground footprint (road, sidewalk, crosswalk, building, tree) and small overhanging objects (fence, pole, street light, traffic sign, traffic light). As each segment is about 80 m long, we derived two maps of size 64 × 16 m from each, facing either side of the street. We derived semantic rasters by binning the points into each map cell and inferring two types of labels: a) Class surfaces are mutually-exclusive in 2D so we compute multi-class surface labels corresponding to the class most represented in each map cell. b) Differently, objects might appear on top of each other so they are not mutually-exclusive. We thus compute binary labels for each object class using a threshold on the number of points in each map cell. We obtain 3k semantic rasters which are split between training and test sets, each including data from all 84 cities.

Semantic rasters for OrienterNet: To obtain labels at the scale required to train OrienterNet [78], we apply this rasterization process to sparse Structure-from-Motion points that have similar semantic labels. Those labels are obtained fully automatically by fusing 2D image segmentations, without any human work [30]. While the resulting rasters are noisier than those derived from LiDAR, they cover our entire training set rather than only 3k maps. They also more consistently cover small objects (like trees and street lights) than OpenStreetMap.

E Implementation details

Training: We develop our models with JAX [11] and Scenic [22], and format our dataset with TFDS [97]. All models are trained on 16 A100 GPUs over 3-4 days with a total batch size of 32 (2 examples per GPU). We use the ADAM [44] optimizer over 400k iterations for the small model and 200k iterations for the larger one. This corresponds to 24% and 12% of the training data, respectively, but is sufficient for convergence. We use a constant learning rate of 10 -4 for the first half and then decay it to zero with a cosine decay. Streetview images are resized to 684 × 456 . We train our models with mixed precision and use gradient checkpointing in order to reduce GPU memory usage as much as possible. The temperature τ is parameterized as log τ , initialized to 0, and optimized alongside other model parameters. All hyperparameters are tuned on the validation set.

In the multi-view fusion, we fuse observations from the 4 closest views that observe a point. This saves a significant amount of memory compared to fusing observations from all views. Using fewer views (20 instead of 36 at inference time) also makes the problem artificially harder.

Pose estimation: We use 10k RANSAC samples during training and 20k during inference. Since only 3 constraints are required to solve for a 3-DoF pose, 2 2D correspondences over-constrain the problem. Randomly sampling each correspondence independently thus yields a large number of invalid poses. To increase the ratio of valid poses, we sample 8 × the number of desired correspondences and pick the ones with the ratio of distances closest to 1. This improves the pose quality, at both training and test time, and is well amenable to parallelized GPU processing, without requiring an expensive sorting. When computing the featuremetric score, query points that project outside the map are given the value of the map boundary. We found this to be more robust than ignoring these points. At inference time, we refine the pose by exhaustive search in a grid centered at the maximum-score pose, with size 4 m × 4 m × 5° and resolution 20 cm × 0.5°.

Architecture: The decoder of the image encoder is an FPN with bilinear upsampling. We progressively upsample features from stages 4, 3, and 2 to the resolution of stage-1 ResNet features, which corresponds to an output stride of 4. For the aerial encoder, we remove the root block (7 × 7 convolution and max pooling) such that the output stride is 1.

Semantic fine-tuning: We train a small 2-layer CNN with two heads: one for the multi-class classification of ground surfaces, using a softmax with cross-entropy loss, and the other for the multi-label classification of objects, using sigmoids and binary cross-entropy losses. We train for 5k iterations and retain the checkpoint with the lowest validation loss.

F Social impact

CO2 emissions: Experiments were conducted using Google Cloud in region us-central1, which has a carbon efficiency of 0.57 kgCO 2 eq/kWh. A cumulative of 88.8k hours of computation was performed on NVidia A100 GPUs. Total emissions are estimated to be 12654 kgCO 2 eq, of which 100 percent were directly offset by Google. Had this model been run in region europe-west6, the emissions would have been of 355.31 kgCO 2 eq. Estimations were conducted using the ML Impact calculator [47].

map images

query

t=32.0m

R=3.1°

2

13

44

2

3

T

query

map images

t=22.6m

R=179.6°

0

1

3

2

3

map images

query

t=24.8m

R=4.3°

2

0

1

34

1

3

.

map images

query

t=7.9m

R=0.5°

2

3

AAAAA

A

3

map images

query

t=5.1m

R=0.1°

2

0

1

3

MAA

map images

query

t=27.7m

R=1.1°

2

1

3

map images

query

t=15.2m

R=80.0°

1

3

map images

query

t=20.2m

R=179.9°

213

3

map images

query

t=1.3m

R=3.6°

0

1

2

3

query

map images

t=0.1m

R=1.1°

LEFT

1

2

3

大

query

map images

t=0.1m

R=1.2°

3

Q

2

map images

query

t=0.8m

R=0.1°

3A

1

3

Barcelona

San Francisco

Manila

New York

Sydney

Paris

London

Rio de Janeiro

Singapore

Taipei

Tokyo

SANTMARI

Barcelona

COLLBLAN

L'Hospitalet

deLlobregat

Singapoe

VAON3OYO:

Rio deJaneiro

Taipei

台

中国区

ano City

台来区

0

20

40

60

80

Recall [%]

Amsterdam

Melbourne

Mexico City

0

1

2

3

4

5

0

20

40

60

80

Recall [%]

Osaka

0

1

2

3

4

5

Position error [meter]

Sao Paulo

0

1

2

3

4

5

Seattle

0

20

40

60

80

Recall [%]

Amsterdam

Melbourne

Mexico City

0

2

4

6

8

10

0

20

40

60

80

Recall [%]

Osaka

0

2

4

6

8

10

Orientation error [degree]

Sao Paulo

0

2

4

6

8

10

Seattle

SfM + SuperGlue

Semantic maps (OrienterNet)

Ours

0

1

2

3

4

5

0

20

40

60

80

Recall [%]

0

2

4

6

8

10

0

1

2

3

4

5

Position error [meter]

0

20

40

60

80

Recall [%]

0

2

4

6

8

10

Orientation error [degree]

BACKPACK

CAR

Amsterdam

Melbourne

Mexico City

New York City

Sao Paulo

Seattle

terdam

IC4

AnneFran

[5109)

s101)

EAPABK[S100

Noord

AMSTERDAM-NOORD

0116

116)

S114

A10E

Flemington

Melbourne Museum

Melbourne

Abbotsford Convent

IKEA

SpO

umBuenavista

rio

DE LAS SALINAS

MAGDALENA

LINDAVISTA

口

IKEA Oceania

日

日

SAN FELIPE

Juan de Aragon

ESUS

Facultad

Estudic

uperior

Aragon

Amsterdam

Hawthorn

oSoumaya

catieVUmc

tembrandtp

rdam.UMC

Vondelparki

H

ZUIDAS

109)

Amstelpark

A10

110

S100

AMSTERDA

Duivendrecht

RCRAAFSMEEH

Diemen

1S0

A10

DESNIEP

IthMelhourneMarket

wn

lens

n

Hobsors Bay

irne

Prahran Mark

StKilda

62

Prahra

24

2

V

M

dPark

Malvi

GlE

BAYA

CONDESA

Hundido

Parque Delta

Parque

ROMANT

NARVART

PONIENTE

Mexico City

losVenados)

(Parque de

Vill

AN

Rodriguez Sam'sClub

Autodromo

Internacional

Aeropuerto

de Mexico

PANTITLAN

AGRICOLA

AZ

Paraue

PlazaShopping

cuchiPark

绿地公园

ragura

RIHGA Royal

大阪市

Osak

Osakaic

国

Morin

大

SUMINODO STATION

大

口

东大

rginaleTiete:

Osasco

nopping Uniao

aVestcasa

116

oso

Villa-Lobos

Parque

BUTANTA

PINHEIROS

nerica Lati

JdoFutebol

BRANCA

Temorialda

Shopping Center Norte

SaoPaulo

116

de Sho Paulo

Atacadac

BRAS

Shopp

Mu

do

ILIA

WoodlandParkZoo

ANNE

WALLIN

NIVERSITY

ISTRICT

Washington

Arboretum

Park

513

renga

Kumata ShrineH

杭全神社

da

DKintet

21

anta Shopping

ledo(Morumbi)

116

VIL

ITAIMBIBI

PIA

OLEMENTINO

-Museu Paulista

VILA

Parque Aclimacao

Space Needle

Seattle

CAPITOL HILL

Madro

naPark

M

口

Sumiyoshi/TaishaR

住大社

479

NaaaiPark

14

HIRANO WARD

平野区

25

八尾

BROOKLIN

BROOKLIN

VILA DA SAUDE

Viewpoint

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrently，2D semantic maps have limitations:  they lack detail, often contain inaccuracies, and are difficult to create and maintain， in order to solve these problem,  the author proposes fusing features from different input modalities to obtain a 2D neural map that is robust over time and retains sufficient geometric and semantic information. The effectiveness of the method was demonstrated by using the 2D neural map generated by SNAP for other tasks.\n\nSTRENGTHS:\nThis paper proposes to fuse various input modalities and use camera pose estimation to generate a 2D neural map, which is the originality. Moreover, the paper delves into this methodology, related work, and conducts a plethora of experiments, culminating in high-quality output. However, the clarity of the paper requires further refinement, and its significance remains somewhat limited.\n\nWEAKNESSES:\nThe author demonstrated the effectiveness of using self-supervised pre-training in the article, but there is no relevant experimental evidence to prove whether the fusion of multi-mode features and the proposed supervision method are reasonable. In addition, it is unfair to only compare with OrienterNet in Experiment 1, as the generation of neural graphs involves aerial images, which are more similar to [1][2].\n[1] Shi Y, Li H. Beyond cross-view image retrieval: Highly accurate vehicle localization using satellite image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 17010-17020.\n[2] Lentsch T, Xia Z, Caesar H, et al. SliceMatch: Geometry-guided Aggregation for Cross-View Pose Estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17225-17234.\n\nQUESTIONS:\n1.It is necessary to supplement experimental evidence for the effectiveness of fuse and contrastive learning supervision in the generation process of neural maps.\n2.Can other satellite-based localization methods generate neural maps? The authors need to conduct experiments to demonstrate the advantages of their method compared to other satellite-based localization methods in generating neural maps.\n3.Does the method also perform well on other general datasets such as KITTI?\n\nLIMITATIONS:\nThe author discussed the limitations of the method on accuracy and the impact of the method on privacy.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised learning approach to create a neural representation of the 3D environment from aerial-view and ground-view images. It utilizes contrastive learning to align neural maps based on the same pose and distinguish maps with inconsistent poses. Large-scale pretraining enhances the model's capabilities for visual positioning and semantic mapping. Experimental results on two datasets validate the effectiveness of the approach.\n\nSTRENGTHS:\n1. The overall idea of learning a neural scene representation from multimodal images via pose contrast is both elegant and effective. This approach allows for a comprehensive understanding of scenes by leveraging different modalities, resulting in a robust representation.\n\n2. The implementation of the overall framework is detailed and non-trivial. The monocular inference, multiview fusion, multimodal fusion, and differentiable pose estimation are well executed and contribute together to the final performance.\n\n3. The large-scale pretraining enhances the neural map with high-level semantics. This pretrained model can be further fine-tuned for tasks such as visual positioning and semantic mapping. This versatility makes it a strong candidate for serving as the foundation model in areas like robotic localization and mapping.\n\n4. The writing in the paper is clear and easy to follow. Additionally, the visualization provided is informative.\n\nWEAKNESSES:\n1. **Pose supervision**: The authors should discuss the influence of pose accuracy on SNAP's performance and address how reliable pose information can be obtained to improve SNAP's learning capabilities. Exploring the impact of pose accuracy on the model's performance would provide valuable insights.\n\n\n2. **Overall illustration**: It would be beneficial for the authors to include a pseudo-algorithm to better illustrate the notations used and clarify the training and inference procedures of SNAP. This addition would enhance the understanding of the model's implementation.\n\n\n3. **Training and inference time**: Given the large scale of the training set, it would be helpful for the authors to discuss the training time of SNAP and justify the necessity of such a large-scale dataset (will reducing the training data significantly reduce the performance of SNAP?). Additionally, considering the practicality for mobile robotics, it would be appropriate to provide information on the inference time on a more affordable GPU. This discussion would provide insights into the computational requirements of the model.\n\n\n4. **Depth estimation**: The authors should clarify how the depth model handles the large distortions present in the input images. Providing details on how these distortions are addressed and mitigated would enhance the understanding of the model's performance.\n\n\n5. **Ablation studies**: The reviewer suggests conducting ablation studies to investigate the influence of Equation (1) and the max-pooling used for multimodal fusion. Including these studies would demonstrate the effectiveness of different modules and contribute to a deeper understanding of the model's performance.\n\n\n6. **Details of semantic mapping**: More information is needed regarding the input, scale, and output of the semantic mapping process. Additionally, since the semantic classes appear to be limited, the authors are encouraged to test the model on publicly available datasets such as SemanticKITTI SSCBenchmark to provide a comprehensive evaluation.\n\n\n7. **Dataset and code availability**: It is essential for the authors to make the datasets used for pretraining and semantic mapping publicly available. Furthermore, open-sourcing the code and data would greatly benefit the field, allowing researchers to replicate and build upon the presented work.\n\n\n**Minor**\n1. The authors are suggested to show easy, medium, and hard examples’ visualizations respectively.\n\n2. In the literature review section (line 196-line 204), it would be beneficial for the authors to include references to works related to semantic occupancy mapping, specifically addressing voxel representations for large-scale scene understanding (not only just object-centric applications). Adding these related works will provide a broader context. Some relevant works to consider include MonoScene [1], VoxFormer [2], TPVFormer [3], and SemanticKITTI [4]. \n\n[1] Cao, A.Q. and de Charette, R., 2022. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3991-4001).\n\n[2] Li, Y., Yu, Z., Choy, C., Xiao, C., Alvarez, J.M., Fidler, S., Feng, C. and Anandkumar, A., 2023. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9087-9098).\n\n[3] Huang, Y., Zheng, W., Zhang, Y., Zhou, J. and Lu, J., 2023. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9223-9232).\n\n[4] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C. and Gall, J., 2019. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9297-9307).\n\nQUESTIONS:\nOverall, the paper presents a novel and well-executed approach to learning neural scene representations from multimodal images. Its technical implementation, combined with the potential for practical applications, makes it a valuable contribution to the field. However, there are some concerns listed in the weakness section. This reviewer will consider raising the score if the concerns are well addressed.\n\nLIMITATIONS:\nThere seems no checklist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for self-supervised learning of BEV features, by using a contrastive loss that links features computed from perspective-view images to features computed from aerial images, where BEV correspondences are given by poses. Interesting method details include depth-weighted splatting in the style of Lift-Splat-Shoot, local decoding/fusion with an MLP, and hard negative sampling via RANSAC. The method provides good localization especially in \"hard\" cases where the camera moves a lot, and the features are useful pre-training for downstream semantic segmentation tasks.\n\nSTRENGTHS:\nThis paper is very well written. The problem and the method are very clearly articulated, the figures and sectioning and style are all helpful to getting the key ideas across efficiently. \n\nSelf-supervised feature learning from the combination of car-mounted and aerial data is a new and exciting area, and the approach here is well motivated. The method makes sense. \n\nThe approach achieves fairly good results, and outperforms SfM methods in some cases. \n\n\nUsing featuremetric similarity to guide the sampling is a very nice detail.\n\nWEAKNESSES:\nThe pre-training experiment would be more convincing if it were benchmarked agaginst some existing approach. I am particularly interested to see if the pre-trained model outperforms some SOTA on some BEV segmentation task (e.g., on nuScenes). The danger is something like: as you add more supervised data, the impact of the pre-training goes to zero.\n\nThe interpolation operator used in eq 4 is confusing for me. Can you write it or explain it in another way? (One additional question here, beyond the main confusion of that equation, is: are you really doing a plain sum here? Typically a mean makes more sense here, so that the scale of the term does not depend on a hyperparameter like resolution.)\n\nThe overall map seems like it may be huge. I need more implementation details on this aspect to really understand what is going on, especially with respect to generating pairs (and non-pairs) for the contrastive loss.\n\nQUESTIONS:\n\"Overhead images are ortho-rectified, such that all world points along the z axis project onto the same pixel coordinate.\"\n\nThis sounds odd to me, so I want to double check. I am imagining the aerial imagery is captured from very high up, and the images cover a wide FOV, and only a few pixels will be well-aligned with the z axis (defined as the gravity direction). Fixing this seems like it should require depth information (e.g., especially for tall buildings).\n\n\n\"we define K horizontal planes at heights {zk}, which are uniformly distributed around the camera heights and points of interest\"\n\nWhat exactly are the \"points of interest\"?\n\n\nIt's interesting that the architecture does not have any BEV CNN involved. To my understanding, it is often this BEV computation which resolves much of the 3D structure of the scene, since depth information is so difficult to estimate from RGB alone (without supervision). Is it really just an MLP applied on each cell independently? The citation here (SimpleRecon) seems not as simple as what's presented here -- it uses a cost volume, various geometric \"metadata\", and another 2D CNN. Can you comment a bit more here, to help me get a sense of what's going on in the architecture? \n\n\n\"Our approach tightly combines 3D geometry and representation learning instead of relying on expensive Transformer [60] or 3D CNNs [25].\"\n\nOK but I think the more reasonable baseline is a 2D CNN applied in BEV (e.g., [26]). This is lightweight also. \n\n\n\"We build mapping segments only from car sequences\"\n\nWhy? And why discuss the data collected from pedestrians/backpacks, if it is never used?\n\nLIMITATIONS:\nLooks good", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops neural 2D maps in an self-supervised manner by matching multi-view ground level imagery with aerial images.\nTo construct a neural map (2D map of features), they encode an image into a feature frustum (similar to lift-splat-shoot), then pool vertically - for multi-view images, they perform a similar fusion method and interpolate between image features at multiple defined heights to construct the map.  \nFor supervision, they use known 3-dof poses to align two neural maps. They use a contrastive loss and mine for negative samples using RANSAC. \nThey train on a large scale streetview dataset with aerial imagery and evaluate performance on a visual positioning task - given 36 views, predict the 3-dof pose of where a query image was taken. Results show improvements over traditional SFM and more recent learning based methods.\n\nSTRENGTHS:\n* Innovative and novel idea with real world applications.  \n* Technical section fairly clear to follow.\n* Overall the work is well written and descriptive.\n* The multi-view fusion could be applied to other fields (3D object detection)\n\nWEAKNESSES:\n* There are many simplifying assumptions (which the authors have provided discussion for) - relative poses which they \"claim can easily be obtained\", gravity direction, pose in only 3-Dof.\n* Quantitative evaluation is not as fair as I would like. Especially when comparing streetview inputs only, the performance improvement is not particularly strong.  \n* All ablations are in the supplementary. The system is consists of many design choices and it would be useful to present which choices were critical and which did not matter much.  \n* All baselines are fairly strong in the \"Easy\" group. I imagine this is actually the most important category for real world applications since the pose estimation would be applied in the loop.\n\nQUESTIONS:\n* Are the neural maps biased towards features that are greater in height? During training with multi-modal inputs, the aerial tiles are biased towards features that are maximum in height (all other stuff is occluded) and so in theory there is no need for the feature encoder to learn anything below the highest cell.\n* Line 105: I didn't understand this. I thought lift-splat-shoot normalized the distribution.  \n* Is the data used public?\n\nLIMITATIONS:\nLimitations are provided (accuracy + method assumptions).  \nCO2 usage is specified in supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SNAP, a novel architecture to obtain 2D neural maps from street-view and top-down images. The model works by encoding street-view and top-down images separately using corresponding encoders, projecting features to the top-down 2D space, and fusing them via max-pooling to obtain the final features. The paper also proposes to train the model using a contrastive map alignment objective, where a query map is aligned with a reference map at the ground-truth pose, and differentiated from negative poses. The learned representations are evaluated on single-image positioning and semantic mapping applications. The results show strong improvement over prior approaches.\n\nSTRENGTHS:\n* The problem of learning semantic map representations in a self-supervised way from raw images and camera poses is interesting and well-motivated. While the approach shares some technical similarities with prior work in terms of the mapping functions, the large-scale nature of training, self-supervised learning, and use of top-down + street-view images is exciting. \n* The proposed approach is sensible, the training methodology is simple yet effective, and is clearly described in the paper.\n* The experiments are well designed and show strong improvements over baselines, and demonstrate the semantic quality of the learned representations.\n\nWEAKNESSES:\n* L41 - 42 - Will the \"50M street view images\" dataset be shared publicly for reproducibility?\n* L73 - 77 - How reasonable are the assumptions made? E.g., Is the projection function accurate and noise-free in practice? How easy is it to get ortho-rectified overhead images? How are they obtained?\n* Clarifications needed\n    * L92 - \"picks the best estimate ... handles partial observability\" -- how exactly is this achieved?\n    * L163 - 165 - How is inference performed to generate pose hypotheses? How scalable is this as map sizes grow larger?\n    * Figure 6 - The classification setup for the center plot is not clearly described. Is there quantitative evaluation for the segmentation shown in the right?\n* Similarity to prior work not discussed. Please compare and contrast.\n    * The mapping approach described in  L99 - 101 is very similar to [R1, R2]. \n    * The objective of learning features by localization is demonstrated in R1, albeit in a classification approach instead of the proposed contrastive learning.\n\n\n[R1] Henriques, Joao F., and Andrea Vedaldi. \"Mapnet: An allocentric spatial memory for mapping environments.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n[R2] Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. \"Geometry-aware recurrent neural networks for active visual recognition.\" Advances in Neural Information Processing Systems 31 (2018).\n\nQUESTIONS:\nPlease address the weaknesses listed.\n\nLIMITATIONS:\nYes, this is discussed well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}