Lean Images for Geo-Localization
Authors: Moti Kadosh
Yael Moses
Ariel Shamir
Venue:  (2020)
Keywords: Geo Localization
Deep Learning
Computer Vision
Camera Localization

Abstract: Most computer vision tasks use textured images. In this paper we consider the geo-localization task - finding the pose of a camera in a large 3D scene from a single lean image, i.e. an image with no texture.  We aim to experimentally explore whether texture and correlation between nearby images are necessary in a CNN-based solution for this task. Our results may give insight to the role of geometry (as opposed to textures) in a CNN-based geo-localization solution.  Lean images are projections of a simple 3D model of a city. They contain solely information that relates to the geometry of the scene viewed (edges, faces, or relative depth). We find that the network is capable of estimating the camera pose from lean images for a relatively large number of locations (order of hundreds of thousands of images). The main contributions of this paper are: (i) demonstrating the power of CNNs for recovering camera pose using lean images; and (ii) providing insight into the role of geometry in the CNN learning process;

Under review as a conference paper at ICLR 2020
LEAN IMAGES FOR GEO-LOCALIZATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Most computer vision tasks use textured images. this paper we consider the geo-localization task finding the pose of camera in large 3D scene from a single lean image, i.c. an image with no texture. We aim to experimentally explore whether texture and correlation between nearby images are necessary in CNN- based solution for this task. Our results may give insight to the role of geometry (as opposed to textures) in CNN-based geo-localization solution. Lean images are projections of a simple 3D model of a city. They contain solely information that relates the geometry of the scene viewed (edges, faces, relative depth). We find that the network is capable of estimating the camera pose from lean images for a relatively large number of locations (order of hundreds of thousands of images). The main contributions of this paper are: (i) demonstrating the power of CNNs for recovering camera pose using lean images; and (ii) providing insight into the role of geometry in the CNN learning process;
INTRODUCTION
Imagine you are brought blindfolded to street corner of city you know well. Now, you remove the blindfold, Can you tell where you are? This the geo-localization task. In computer vision, this amounts to estimating the position (and sometimes the orientation) of a camera given its current view. Although localization devices such as Global Positioning Systems (GPS) have improved sig- nificantly over the last years, they often do not work well in city scenes and do not provide highly accurate results. Autonomous cars, drones, and IOT devices are expected to benefit tremendously from the ability to determine their pose (position & orientation) accurately in their environment.
solution for geo-localization, either by a human or by a machine, can use appearance cues (e.g.. texture of a unique building), geometric cues (e.g., a unique shape of a building), or both. Significant improvements were obtained object recognition, scene recognition, and localization tasks, largely by exploiting the appearance of the scene rather than only its edges (e.g.. color and texture and image features such as SIFT (Se et al. (2002); Lowel (2004): Li et al. (2010)) Later, these methods were improved by adding coarse geometric constraints to the image features (e.g., |Ramalingam eet al. (201T): [ansal & Danilidii (2014)). Nowadays, methods are based on machine learning, in particular convolutional neural networks (CNNs), where the input is typically the unprocessed textured image. Clearly, both appearance and geometry play an important role in these methods.
Figure 1: Left: lean images contain mostly geometric features: edges (left), faces (center), and depth information (right), which are projections of a 3D) model. Center: Bird's s-dds view of one of the areas we used. Right: a top view of city area (buildings are marked as white) where color indicates the localization success rate of the network from red (high) to blue (low). Note how open spaces more distinct than narrow streets.

---

Under review as a conference paper at ICLR 2020
The main goal of this paper is to study the role of geometry in a CNN solution to the geo-localization task rather than propose a working system for application purposes. We study an end-to-end deep CNN for geo-localization, by ignoring the often available texture of the scene. To do this, we con- sider only lean images, which contain mostly information that relates to the geometry of the scene while lacking texture rich geometric details. In practice, types of binary images that consist of the edges of the buildings' outline and the buildings' facades. In addition, we also consider depth images that contain more geometric information. Examples of the three types of lean images are shown in Fig. | (left). Note that in the first row, the view contains dominant landmarks. while the second row shows very little distinct information that might be ex- pected to assist localization. Such non-distinct views are very common in large environments such as a city, making localization with lean images very challenging. Further note that we deliberately do not consider real images or synthetic images with texture, since our goal is to study only the information available from pure geometric data.
city scene and consider two
Our lean- lean-images are obtained simply by projecting an untextured 3D mesh model of Berlin onto various positions in the scene (See Sec. 3. A bird's s-ede view of one of the areas is shown in Fig. (middle). Using such model allows us to study the role of geometry for geo-localization in controlled manner and in a larger scale than ever before, both in terms of the area covered (many city streets) and in terms of the number of images (up hundreds of thousands).
Neural networks were shown to be effective in geo-localization tasks (e.g.. Kendall et al. 2015); Walch et al.. (2017); Melekhov et al. (2017); Sattler et al.] (2017)) A typical geo-localization lution will return the pose of a camera in novel views. That is, the image query is not part of the training set. In this paper we ask (i) can the CNN generalize and support geo-localization in large environments using lean images that contains only geometric and spatial data? (ii) is it likely that geometric information is used by the CNN to solve this task? However, geo-localization task, we also consider the memorization capacity of the network for previously seen images. This can be regarded as image retrieval from database of all available views of the city. A na've solution would store all images and then perform a brute-force search in the database. However, this is inefficient and can become infeasible as the database gets larger. Defining a com- pact representation and an efficient search is clearly desired. The question we address in this case is (iii) whether CNNs can be used to solve the meomorization task using lean images.
better understand the
As discussed in the results (Sec. 6) depend on the number of images and their sampling density. We believe this indicates that networks can learn some sort of a spatial map for area using only geometric data, since no colors or textures are available in our data. The success of geo-localization also depends on the specific position. Fig. (right) shows how certain positions in the streets of a city are more recognizable than others.
we found positive answers to all these questions, but the results
Our paper presents an empirical study regarding the information that can be used by CNNs to build an internal representation; we do not propose a practical solution based on lean images. The main contributions of our study are: (i) showing that lean images contain sufficient information for sol sing the memorization and the gco-localization tasks (ii) proposing a systematic method to study the role of geometry in CNNs for geo-localization tasks: (iii) demonstrating the power of CNNs to use the geometric information to build internal represenations of data.
2 RELATED WORK
Place recognition (e.g., the Eiffel Tower) can be regarded as a coarse geo-localization task. Classic approaches use visual features to represent each image in set of images of a given location (e.g.. by a bag of words) and then match a target image with the stored representations (e.g., [Se et al. (2002); Sivic & Zisserman (2003); Lowe (2004); Robertson & Cipolla (2004)). Hays & Efros (2008) were the first to address the place recognition task using millions of geo-tagged images, based on various visual features (e.g., tiny images, color histograms, line features, gist descriptors).
Memorization of image data-sets is often performed by manually engineered image representation (e.g., dictionary of image features) and an image retrieval approach, including the metric between the stored representation and a target one. Te.g..Sivic & Zisserman (2003); Lowe (Roberton
& Cipolla (2004); /Zhang & Kosecka (2006); Nister & Stewenius (2006); Schindler al. (2007);
2

---

Under review as a conference paper at ICLR 2020
Hays & Efros ( Li al. (2010). We show that neural networks are able to efficiently create such a representation in various experiments.
A possible solution for geo-localization task, where both position and orientation of a camera with respect to a scene should be estimated, is by using triangulation with images with known pose (e.g.,
Zhang & Kosecka (2006)). In most studies, 3D models of the scene used by means of point-
clouds (e.g.. Irschara et al. (2009): [Stttler et al.] (2011); Matei et al.] y013); Svarm et al.] (2014). Digital Elevation Maps (DEM) (e.g.. Baatz et al. (2012); Bansal & Daniilidis (2014)),
full 3D models (e.g., 1as.aaaaaing et al. (2011). One of the main challenges of these works is to develop an
efficient computation of 2D 3D feature matching. New 3D feature representation have also been developed (e.g.. Irschara et al. (2009); [Sattee et al. (2011). Bansal & Danillidis (2014) introduce a feature more closely related the lean images we consider. It consists of 3D corners and direction vectors extracted from a Digital Elevation Map (DEM) to be matched geometrically to the corners and roof-line edges of buildings visible in street-level query image. Efficiency and robustness become even more important when dealing with city-scale 3D model. A fast method for inliers detection that enables solving the correspondence problem on such a scale was suggested by Svärm et al. 2017). Recent survey existing localization methods can be found in Piasco al. 2018).
One of the key ideas that bypasses the challenge of defining an efficient and robust 2D-3D feature matching required by the abovementioned methods is to use end-to-end CNN solution that per- forms both feature extraction and matching. PoseNet Kendall et al. (2015) is an impressive CNN based approach for solving the pose of real images. A dataset of images was used for training Google LeNet Szegedy et al. (2015) where the 6-DoF pose of the camera was used as ground truth. Walch al. (2077) suggested an improvement to the PoseNet by adding an LSTM, which reduces the dimensionality of the feature vector. [Meekhhvv et al.] (2017) used ResNet34 He et al. (2016), which uses encoder-decoder structure to improve model accuracy. Kendall Cipolla (2016) im- proved their earlier work Kendall et al.] (2015) by applying an uncertainty framework to the CNN pose regressor, In another work, Kendall et al. (2017) studied the affect of various loss functions on the result of PoseNet.
In our study we assume a 3D model of a city is given. Our setup is very challenging since the model and the images consists of only coarse 3D structure of the scene without texture for computing image features. On the other hand, our images are noise-free and there are no object-level occlusions such as trees, cars and people. In addition, we were not limited by data size, we projected as many images as we chose. Most importantly, our goal differs from that of the aforementioned methods: whereas they focus obtaining better and faster solution for geo-localization, to understand the role of geometry, alone, in geo-localization by systematically training and testing the same neural network controlled datasets.
focus trying
DATA
Using 3DD model the data source orientation, and resolution. In our study we used a simplified 3D model of Berlin für Wirischaft und Technologie GmbH (2016). The model contains only the geometry of building walls and rooftops, and does not contain any tine geometric details such as window frames or doors or texture (see bird-eyes view in Fig. D. We consider three types of images which are projections from this model: edge, face, and depth map, see Fig. [ and we call them lean images since they contain no texture or structural details. The face images contain the buildings' planar facades.
sample as many images necessary in any position,
Each image is defined by its camera pose. That is, (x, 0, Φ). where (z, y) is the position on the ground plane and (0, d) is the camera Y Yow and Pitch angles. We assume for simplicity that the picture is taken at a fixed height, and the roll angle is fixed as horizontal. We generated several image datasets that are sampled uniformly along this 4D grid. The density in the (x, y) domain varies between the data-sets but fixed in the (0, domain. A 6D pose vector (instead of 4D) in the form of (I. y. q1, 92. 93, 9a). is used to represent the camera pose. The q1, 92, 93. 44 are the quaternions representation of the (8, Ø) Euler angles. Each set of images is created in predefined area of the city. The number of images in the set is determined by the size of the area and the grid sampling density. All three types of lean images were generated for each sample pose.
3

---

Under review as a conference paper at ICLR 2020
ThE puride
Tan Sorado
Mire330 Tyoar cLDm
Ted 30104
(a) Nearest Neighbor
(b) Manhattan Distance
(c) Sampling Map
Figure 2: (a)-(b) Illustration in 2D of the evaluation measures for geo-matching (a) and geo- interpolation (b). The real measures are 4D in nature. (c) Example of sampling positions on a area of the map. For the training set: green indicates valid samples and red invalid samples. For the test set: blue indicates valid samples and orange invalid samples (please zoon-in on screen).
When dealing with lean images, care must be taken not include empty images. For example when the camera is facing a building wall from a short distance. Such images contain almost no visual information and do not contribute to the learning process. We discard an image that has less than 8 edges, or an image that does not contain skyline (at least 50% of its top-most pixel row is sky). Moreover, images of positions inside a building are irrelevant to geo-localization, and are also discarded from training and test (see Fig. 2 right).
4 TASKS & HYPOTHESIS
The geo-localization task we consider is to recover the camera pose of an unseen image (not in the training set). We also consider a memorization task of retrieving the camera pose of an image from the training set. For both tasks we consider lean images several configuration of regression and classification networks (see Sec. 5 Our goal is to answer the following questions: (i) Can CNN be trained solve these tasks from lean images? (ii) Does geometry play a role when training the CNN for geo-localization?
input to the network, and test
4.1 MEMORIZATION TASK
The memorization task could be defined as a classification task where each (ar, y, ()))) considered as class. We examined whether CNN can solve the memorization task using lean images. Given an image from the training set, we tested whether the correct camera pose could be determined. In sense, the network is trained to overfit, and actually will not generalize. On the other hand, this would mean that the network managed to encode very large set of images (order of hundreds of the thousand) in some feature space as well as compute efficient matching function between the features of the images to find the right pose.
(A) Geometrically Correlated: In this test, the camera pose for generating the image was used ground truth for training. Hence, when the pose of nearby images is correlated, the network has access to this geometric information.
We aim to test whether the network used the correlation be- tween pose and image, in order to learn the training data. To do so, we randomly shuffled the pose information between images so that poses were not spatially correlated with respect to the images. If no geometry is used by the CNN, the results on this training data are expected to be similar to those obtained with the real pose as a ground truth. Previous study of random labels for object clas- sification task Zhang et al. (2016) also considered randomized labeled for classification, and showed that random labels only affect the time to converge.
(B) Geometrically Decorrelated:
Evaluation: For a classification network, the evaluation simply the number of correct classifi- cation. For regression network, the computed pose does not necessarily match exactly pose of an image from the training set (see Fig. Eb). We used the nearest neighbor (nn) grid sample to the computed pose as the pose retrieval. We report the percentage of images whose correct pose is the
4

---

Under review as a conference paper at ICLR 2020
Geo-Matching
Gieo-Interpolation
(B) Arbitrary Pose
(A) Correct Pose
(C) Correct Pose
Input type
2D
(z, y)
4D
[x, V. U.O, 0,6)
()
Inn
Inn
3nn
D<1
D<3
D<1
D<3
Area 400x400
Edges
0.45
0.97
0.99
0.64
0.82
0.58
0.75
step 20
Faces
0.35
0.99
0.99
0.56
0.76
0.51
0.69
37K images
Depth
0.23
0.99
0.99
0.61
0.79
0.55
0.72
Edges+Facess
0.29
0.98
0,99
0.72
0.88
0.65
0,82
0.24
0.98
0.99
0.71
0.88
0.64
0.81
Area 400x400
Edges
0.11
0.98
0.98
0.85
0.94
0.84
0.93
step 10 140K images
Faces
0.05
0.97
0,97
0.80
0.90
0.79
0.88
Depth
0.06
0.97
0,97
0.83
0.92
0.82
0.91
Edges++aaces
0.09
0.97
0.97
0.88
0.96 0.87
0.95
Edges+Faces+DDppp
0.08
0,94
0.95
0.88
0.96 0.87
0,95
Area 800x800 step 20 170K images
Edges
0.06
0.96
0,96
0.62
0.78 0.59
0.75
Faces
0.01
0.96
0.96
0.51
0.68 0.48
0.65
Depth
0.01
0.96
0.97
0.61
0.77
0.59
0.73
Edges+Faces
0.04
0.92
0.93
0.70
0.86
0.67
0.83
0.03
0.95
0.96
0.70
0.85 0.67
0.81
Table 1: Results of our experiments. The fraction of images on which a correct estimation was obtained out of the total number of valid images evaluated (the higher the better). For geo-matching we use the nearest neighbor measure (nn) and for geo-interpolation the Manhattan distance (D).
nearest neighbor (Inn) and also report the percentage of images whose correct pose is among the three nearest neighbors (3nn) of the computed pose. These evaluations were used for both (A) and (B) tasks. An additional advantage of using this measure is that it is given in grid steps and not in meter/angle units, circumventing the difficulty of comparing distances and angles and enabling a comparison of results with different grid densities (we do provide numerical lg errors in Table 2
4.2 GRO-LLOAALIZATION TASK (C)
We tested whether the network can generalize and estimate the pose of an image that is not in the training set. To avoid over-fitting and allow generalization, the network was trained until best result was achieved on validation set. We do not expect the network to return correct position that is outside the learned area. For this reason our test is comprised of images sampled at midpoints of the training grid. These are images that are farthest from the training set samples.
Evaluation: In this task we only use regression network. A computed pose is considered correct if it lies within the same grid hyper-cube the test sample. We report the number of correctly computed poses (D < 1). In addition, we considered the Manhattan distance between the hyper- cubes of the computed pose and the test sample (see Fig. Ap.. We report the number of images for which this distance is smaller than 3 (D < 3). Note that these measurements are also invariant to the sampling step size. Thus, we are able to compare results of experiments that were sampled with
different step sizes. For completeness, we also provide the standard & errors in Table 2
5 NETWORK
For the geo-localization task, a regression network is more natural than classification network. It allows the network exploiting the geometric structure and information, and to use the same trained CNN for the geo-localization task. It directly returns the pose of the unseen images. Because the considered CNNs were designed for classification tasks, we follow Kendall et al. (205)) and modify them to solve regression task by simply removing the last softmax layer and replacing by fully connected layer of our result vector (z, y, Qı, q2, Q3, 44). Although position and orientation are considered different tasks that should have some weighting factor during the learning process, we noticed that normalizing (I, y) with respect to the total area size eliminates the need for such weighting. Our loss function is lg for the position (xr, y) and &g for the orientation (q1 42, Z3. 94).
We examined several CNN architectures that proved to be successful on object recognition tasks. Specifically, tested VGG, Google LeNet (Szegedy al. (2015) and ResNet50 He et al.) (0016).
5

---

Under review as a conference paper at ICLR 2020
(A) Gen-MeeeMi Matching
(C) Gieo-Interpolation
Table Examples of the & errors for an experi- ment with Edges+Paces im- age types in each sub-space: spatial (I, y) errors (ap- prox.) meters, and orienta- tion (0, Φ)) errors in degrees. Similar to this example, usu- ally the errors show a long- tail distribution: many im- ages have small errors and a few have very large errors.
[T, y)
(0, ())
(x, y)
(0,
mean
median
mean median
mean median
mean median
Area 400x400
step 20
3.65
3.26
0.84
0.69
26.30
11.26
10.95
1.84
37K images
Area 400x400
step 10
2.37
2.10
0.57
0.48
7.99
3.67
2.65
0.67
140K images
Area 800x800
step 20
5.43
4.71
0.67
0.54
40.23
12.28
9.80
1.40
170K images
built for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) Russakovsky et al. (015)): Deng et al. (2009). In set of preliminary experiments we found that ResNer50 had the combination of smallest network size in terms of parameters and the best training and testing results. Therefore, we report our experiments using only the ResNet50 architecture. We trained the CNNs from scratch rather than use the pre-trained weights, since the networks we tested were trained with ImageNet, which contains real photographs. Our assumption is that the pre-trained models are tuned for texture information that not available in lean images. Transfer learning shows to improve the learning rate, when moving from one AOI to another.
EXPERIMENTS & RESULTS
We tested and evaluated the regression ResNet50 network for the three tasks described in Sec.
The datasets, which are described in Sec. 3 are defined by the following parameters. (i) Area
of interest (AOI): (z, y, width, height); (ii) Grid-step, ô: the distance between adjacent (a, y) position of the sampling grid. That is, adjacent to (z, y, 0, 6) are ( 0, Φ) and (I, y d, 0, Φ). The grid density in (8, Φ) domain was fixed: (iii) Input type: edges, faces, depth, edges + faces, edges faces + depth. For the last two input types the images were fed to the network by stacking them channel-wise; (iv) Validation set created by randomly choosing 10% of the training samples; and (v) Test set created by images that were sampled at midpoints of the training grid.
We used various step sizes for the camera position on the grid: å 10, 20, 40 in model units (1 unit ~ 1 meter). 0 (yaw) was sampled at 5 steps between 0 and 360 and ø (pitch) was sampled at 3 steps between 0 and 15". The height was set to a fixed value of 1.7 above ground.
The main results of the regression CNN are summarized in Table The results for tasks (A)-CC), each block of three rows consists a different dataset, defined by the area size and For each block considered the different types of lean images and evaluated them the three tasks as described in Sec. 4 Each entry is an average three different AOIs. For completeness. Table[]showw an example of the mean and median l errors of the pose estimation for edges+faces experiment. Similar results were obtained other experiments. Table [ the results of testing the limitations of the CNN with respect the sparsity of the grid (6 40) and the size of the datasets (> 630K images). We next discuss the obtained results.
Memorization Task
Very poor results were obtained for the memorization task when arbitrary poses were used as ground truth (Table Task (B)). i.e. when no geometric correlation between the images and their ground truth was available. The highest percentage of correct matches (45%) was obtained for the smallest set of considered images (37K images). For largest set (170K images), the percentage of correct matches was less than 10%. As can be seen, the quality of the results decreases as the number training samples increases. contrast, when the correct poses were used as ground truth (Table [ Task (A)), the CNN succeeded in Inn localization of more than 92% of the training samples in all cases. We believe the significant differences between the two geo-matching tasks (A) and (B) is due to the network exploiting the geometric correlations when learning metric between images. This true, even when we consider only the memorization task, where we evaluate the over fitting of the network.

---

Under review as a conference paper at ICLR 2020
(A)
(C)
Table 3: Testing network learning capacity. These results are from single experiment where the image input type is only edges. The net- D<3
Geo- Matching
Geo- Geo-Interpolation
2D
(x, y)
4D
(x, y, 0, Φ)
Inn
3nn
D<1
D<
D<l
Area 800x800
work ability to learn drops when number of images grows beyond certain point.
step 10
0.82
0.82
0.80
0.92
0.79
0.92
636K images
Area 1600x 1600
step 20 666K images
0.58
0.59
0.46
0.69
0.44
0.67
We also considered much larger datasets with more than 600K images (Table The percentage of correct matches dropped 82% for dense grid, 8 10, and 56% for a sparser grid, 8 20. For = 20 and > 600 images, the network capacity is probably saturated. A comparison of these results to those reported in Table [ (Task (A)) for the same 5 values, indicates that both the number of images and the grid size determine how successfully the CNN models the data.
In addition, we tested datasets with sparser sampling the position domain (Table blocks), and in both the position and the orientation domains (Table Taable@ootoon bottom 2 blocks). For sparse sampling only the position domain, the percentage of correct matches is reduced marginally. However, when reducing the sampling also in the orientation domain, the percentage of correct matches is dramatically dropped. This indicates that casier for the CNN to model a denser grid (probably because of higher geometric correlation between images), and it is easier model fewer images (probably because of network capacity).
(A) Geo-Matching
(C) Geo- Interpolation
2D
(x. y)
4D (x. y, 0,
Input type
Inn
3nn
D<l
D<3
D<l
D<3
Area 800x800
Edges
0.90
0.96
0.39
0.62
0.30
0.49
step 40
Edges+Faces
0.91
0.97
0.48
0.72
0.38
0.59
61K images
Fdges+Faces++epll
0.96
0.98
0.48
0.72
0.38
0.58
Area 1600x160X
Edges
0.89
0.90
0.22
0.39
0.19
0.32
step 40
Eages+Faces
0.94
0.95
0.30
0.50
0.24
0.41
174K images
Edges
0,94
0.96
0.32
0.51
0.26
0.43
Area 800x800
Edges
0.40
0.41
step 40 sparse ang.
Edges+Faaces
0.37
0.38
Failed
2.5K images
Edges+Faces+Depthh
0.26
0.27
Area 1600x 1600
Edges
0.16
0.18
step 40 / sparse ang.
Edges+Faces
0.17
0.19
Failed
7K images
Edges+Faces+Depth
0.13
0.14
Table 4: Low grid density results. Datasets (single experiment each) with sparser spatial sampling (top two blocks), and sparser spatial and orientation sampling (bottom two blocks) where the pitch angle is E [6, 12]. and yaw (45m) The sparser the data, the worse the results. Geo-interpolation could not succeed very sparse and very small datasets.
Geo-localization task
Here we tested whether the pose estimation by the CNN generalizes to unseen images. We used the same training as in the memorization task with the correct pose as ground truth, and we tested on images sampled from the mid point of each grid cell. We report our results with respect to the 2D position as well with respect to the 4D parameters of a pose (T.
The network was able to generalize image position with good accuracy where 70% of images are correctly positioned in their grid cell, and above 80% the computed poses are within three cells of the correct one. As expected, this task achieves better results on a sparse grid (6 = 20, ~ 70%). The 4D position error is lower bounded by the 2D position error, and hence is greater. Moreover, the sampling rate in the orientation domain is much higher that in the location. Hence a small error in orientation estimation has a greater effect on the 4D errors. Still, the accuracy in 4D for 8 10 is 87%.
tighter grid (6 10, 88%) than
7

---

Under review as a conference paper at ICLR 2020
It is clear from Table | Task(C) that for 5 = 10 the results are better than for 8 = 20, even if the number of images is larger. We further explore the effect of the grid density for a sparser grid, 8 40, where the percentage of correct estimations dropped significantly below 50% and 30% for 61 K' and 174 images, respectively (Table (Table(TTakk(C)) Task(C). For 5 = 10 for 636K images, 80% of the estimations were correct (Table 3 Tas)(C). For this task, sparser sampling is more critical than the memorization task as can be seen in Table For very sparse sampling of the 4D space the network cannot really generalize to positions not seen before. Here again we believe that not only the number of images play a role but also their density. The denser the grid, the higher the correlation between images, and hence better generalization can be obtained.
A nice application of our results is the ability to rate the distinctiveness of positions in the city. In Fig. we we illustrate how certain places can be easily recognized (high geo-interploation success rate) while other are more difficult. Note, for instance, how open spaces are more distinct than narrow streets.
Effeet of Data Type
Faces alone provides the least geometric information, and indeed in most cases was inferior to edges or depth results. Surprisingly, edges alone provides better information than depth alone for all tasks. Similar results were obtained for all data types for the memorization task for å s 20 (Table [ but for very sparse grid 6 40 (Table 4 richer geometric information improves the results. We believe it is because the results on 8 S 20 were very high to begin with only edges.
For the geo-localization task (C), adding the faces information significantly improved the results, as expected. Surprisingly, the depth information did not show any significant performance gain when d < 20. This may indicate that edges+faces provide sufficient information for these cases. However, for a very sparse grid, 5 = 40, with a relatively small number of images, adding the depth significantly improves the results (Table 174K images).
For the data with geometrically decorrelated pose (Task B) and for the very sparse sampling (Table bottom 2 blocks), the more information we add, the worse results were. The reason for this still unclear to us. A possible explanation is that as the problem becomes more of a memorization task, the increase of information makes it harder for the CNN to find discriminant features.
Classification
For completeness, we also considered a brute-force classifier networks. Using the number of images in the training set as an output (~ 100k images), creates a huge FC last layer, which results in a much larger number of parameters network with a binary representation of 100k indexes (17 output parameters). Experiments showed that the classification CNN is able to memorize the training set extremely well as also shown by Zhang et al. (2016). On all experiments, on both step 10/20, the result exceeded 97% accuracy. On the other hand, as expected, this network gave very poor results for the geo-loclization task even for 8 = 10 (below 15% and 21% D < 1 and D < 3. respectively) compared to the regression network (above 85%). The test sample location estimation is done by feeding the CNN with a test image and than taking the location of the training sample that matches the model output. We believe this proves that such classification supervision is inadequate for the Geo-Localization task, the network has no incentive for creating any meaningful features that would later be applied to new similar images.
comparison to the regression network. Hence we test a
CONCLUSIONS
In this work we showed that (i) CNN can achieve good results in geo-localization tasks using only lean images taken from a very simple 3D model, and (ii) that geometry plays an important role in geo-localization, by achieving good results while ignoring texture and scene details. The results indicates that noise-free lean images are sufficient for solving the memorization task using a CNN, and that the use of uncorrelated images makes it nearly impossible. In addition, our results indicate that (iii) the generalization task, can also be solved by CNNs when using lean images.
From more practical perspective. it is of interest to explore whether geometric information can be used for real life geo-localization tasks, also because 3D models, e.g., the Open Street Map project Wiki (2018) readily available.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Georges Baatz, Olivier Saurer, Kevin Köser, and Mare Pollefeys. Large scale visual geo-localization of images in mountainous terrain, In European Conference on Computer Vision (ECCV), pp. 517- 530. Springer, 2012.
Mayank Bansal and Kostas Daniilidis. Geometric urban geo-localization. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3978-3985, IEEE, 2014.
J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Berlin Partner für Wirtschaft und Technologie GmbH. https://www.busitt busshsssocaiooeeenerr///////ssss
Berlin 3d city model, 2016.
James Hays and Alexei A. Efros. im2gps: estimating geographic information from single image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
A. Irschara. C. Zach, J. Frahm, and H. Bischof. From structure- senccture-from--ne from-motion point clouds fast location recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2599-2606, June 2009. doi: 10.1109/CVPRR2009 250.5.55...9
Alex Kendall and Roberto Cipolla. Modelling uncertainty in deep learning for camera relocalization. In IEEE International Conference on Robotics and Automation (ICRA), pp. 4762-4769. IEEE, 2016.
Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real- time 6-dof camera relocalization. In IEEE International Conference Computer Vision (ICCV), pp. 2938-2946. IERE, 2015.
Alex Kendall, Roberto Cipolla, et al. Geometric loss functions for camera pose regression with deep learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 3, pp. 8, 2017.
Yunpeng Li, Noah Snavely, and Daniel P Huttenlocher. Location recognition using prioritized fea- ture matching. In European Conference on Computer Vision (ECCV), pp. 791-804. Springer, 2010.
David G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-10,, Nov 2004. ISSN 1573-1405, doi: 10. 0001002//BVISS1.00002266 99615.94. URthttps::/ /doi. org//c.......oooooo.oooooo..o... 10. 1023 VISII.IIII................IIIIII w0..0000002... 99619.9699999
B. C. Matei, N. Vander Valk, Zhu, H. Cheng, and H. S. Sawhney. Image to lidar matching for geotagging in urban environments. In IEEE Workshop Applications of Computer Vision (WACV), pp. 413-420, Jan 2013. doi: 10. (199/VACV.2013.64755500
laroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu. Image-based localization using hourglass networks. arXiv preprint arXiv: 17030077,, 2017.
D. Nister and H. Stewenius. Scalable recognition with vocabulary tree. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pp. 2161-2168, June 2006. doi: 10.1 10011099/VVPR.20006.6...
Nathan Piasco, Désiré Sidibé, Cédric Demonceaux, and Valérie Gouet-Brunet. A survey on visual- based localization: On the benefit of heterogeneous data. Pattern Recognition, 74:90-109, 2018. 8.
Srikumar Ramalingam, Sofien Bouaziz, and Peter Sturm. Pose estimation using both points and lines for geo-localization. IEEE International Conference on Robotics and Automation (ICRA), pp. 471 4716-4723. IEEE, 2011.

---

Under review as a conference paper at ICLR 2020
Duncan P Robertson and Roberto Cipolla. An image- baage-bbseddee system for urban navigation. In British Machine Vision Conference (BMVC), volume 19, pp. 165, 2004.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211225) 252, 2015. doi: 10.1007/s1 1263-015-08166->
Sattler, B. Leibe, and Kobbelt. Fast image-based based localization using direct 2d-to-3d matching. In IEEE International Conference Computer Vision (ICCV), pp. 667-674, Nov 2011. doi: 10. 10.1100/...2211..0 1.6126600...2
Torsten Sattler, Akihiko Torii, Josef Sivic, Mare Pollefeys, Hajime Taira, Masatoshi Okutomi, and Tomas Pajdla. Are large-scale 3d models really necessary for accurate visual localization? In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
G. Schindler, M. Brown, and R. Szeliski. City-scale location recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-7, June 2007. doi: 10.1109/CVPR. 2007.383150.
Stephen Se, David Lowe, and Jim Little. Mobile robot localization and mapping with uncertainty using scale-invariant visual landmarks. The International Journal of Robotics Research, 21(8): 735-758, 2002. doi: 10.1 00.1177007836402764442222 02783640276141222
URL https:
pestpp::/dci.orgg./ htt. http://tt...oo.////////o///////e 11.777
1177/
Sivic and Zisserman. Video google: a text retrieval approach to object matching in videos. In IEEE International Conference on Computer Vision (ICCV), pp. 1470-1477 vol.2, Oct 2003. doi: 10. 10.1109//CCV22003. 1238663.
Linus Svarm, Olof Enqvist, Magnus Oskarsson, and Fredrik Kahl. Accurate localization and pose estimation for large 3d models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 532-539, 2014.
Linus Svärm, Olof Enqvist, Fredrik Kahl, and Magnus Oskarsson. City-scale localization for cam- eras with known vertical direction. IEEE Transactions on Pottern Analysis and Machine Intelli- gence (TPAMI), 39(7):145511461. 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolu- tions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. URL,
http:/ fettp://arxian www./rggg///./oooooo//ooo/...e/e abs 1409.440000 4842
Florian Walch, Caner Hazirbas, Laura Leal-Taixe, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers. Image- localization using Istms for structured feature correlation. In IEEE Inter- national Conference Computer Vision (ICCV), volume 1, pp. 3, 2017.
OpenStreetMap Wiki. Osm-3d.org openstreetmap wiki,, 2018. URL http://wiki:
openst wwwp///g///oooo.....o iineexxw/n//x////////e
[On
line; accessed I-November-20188]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 1otiv:6611.0350 2016.
W. Zhang and J. Kosecka. Image based localization in urban environments. In Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), pp. 33-40, June 2006. doi: 10. 119093DPVT...00..0....
10


Reviews:
[{"review_id": "SklaSE3uKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper evaluates the performances of Deep Learning for geo-localization tasks in a world of images without textures (\"lean images\"). More exactly, the lean images are images rendered from a 3D model of a city. made of the depth and/or the buildings' edges and/or the buildings' faces. For the purpose of the evaluation, no real images are used,  only lean images both at training and test times. Many lean images are generated for training a network to predict the camera pose (2 translation parameters,  two angles), as a classification problem or a regression problem.\n\nThe motivation for the study is to understand the use of geometric aspects by a deep network to solve the task. However, as the paper presents mostly the results of the quantitative evaluation, it is not clear what we really learn about the mechanisms of the deep network. I think the empirical results can be interesting for people working on geo-localization, but probably not for the audience of ICLR.\n\nI suspect the authors have in mind for the future to learn to predict a lean image from a real image. I would find this aspect very interesting. It would also be interesting to evaluate the performance on the  geo-localization task when noisy images are used as input.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gCdj5T9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper evaluates a geo-localization task based on \"lean\" images only, obtained by projection of 3d models without texture information. Multiple levels of granularity of the lean images (edges/edges+faces/edges+faces+depth) are compared for the learning, both in a \"memorization\" setting and in a \"generalization to unseen poses\" setting. Moreover, the behavior of the learning when labels are shuffled is evaluated.\n\nThe paper is motivated by identifying lean images with \"pure geometrical\" representation, and argues that the study is a demonstration that the network can learn some spatial map. I do not share this analysis: the edges, shapes and colors of the lean images still create some characteristic shapes that do not have to be linked with the 3D map in order to be detected; it is not clear why the detection from lean images is more geometrical in essence w.r.t. textured images. The motivation of the paper is therefore not compelling, and the study would have more relevance as an ablation study of a method using both geometry and texture. Can you give more compelling arguments for the problem setup, rather than a mere thought experiment?\n\nOther than this particular setup of geo-localizing based on lean images only, the novelty of the paper and the approach seem limited and does not offer new insights in geo-interpolation methods, geometry-aware neural networks, or memorization. Therefore, I believe the paper does not meet the standards of ICLR.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJxZfI20KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of geolocating an image from a 3D model of a city. This is an interesting and challenging problem since it involves solving the matching under 3D transformations at large scale.\n\nThe proposed approach is to use geometric features such as edges, faces and depths of the image, which is called \"lean image\" by the paper. The paper demonstrate that by using the \"lean image\" a neural network can recognize its geolocation with substantial accuracy. However, the idea of using geometric features has been investigated in quite a few previous literature and the paper fails to discuss relevant works, e.g., [1,2,3] and distinguish their differences.\n[1] Li et al. Planar Structure Matching Under Projective Uncertainty for Geolocation. ECCV 2014.\n[2] Mousavian and Kosecka. Semantic Image Based Geolocation Given a Map. ArXiv 2016.\n[3] Kim et al. Learned Contextual Feature Reweighting for Image Geo-Localization. CVPR 2018.\n\nThe paper contains mostly empirical evaluations however the provided experiments do not well support the claim that CNN works well with geometric structures in geolocalization. The paper should compare with CNN on RGB images on the same data in order to demonstrate its effectiveness.\n\nReview:\n## Contribution of the paper:\n1. The paper argues that \"lean images\" (geometric structure of a 2D scene) contains sufficient information for solving the memorization of geolocation tasks.\n2. The paper provide an approach to evaluate the argument.\n3. The paper provide experimental validation that the proposed method works fairly on a certain Berlin dataset.\n\n## Feedback\nThe paper is interesting but it does not meet the standard of this conference. There are some key reasons:\n1. The writing is poor. The paper is hard to follow. There are also missing references. The importance of geometry in geolocation has been studied in many previous works.\n2. The paper is empirical so it has to provide strong empirical evidence to support the claim. The experiment is only conducted in Berlin data whose scope is fairly small. Without large scale experiments, it is hard to convince readers that CNN is able to \"memorize\" all the locations using just geometric features. Without comparing to RGB baseline or other existing approaches, it is hard to convince readers that the proposed approach works fairly well. It also seems the input images are just 2D projections so without experiments using real 2D images, it is hard to say the approach is effective.\n\n## Improvements that could be done\n1. Please make sure everything in the tables are clearly explained. The metrics should be explained somewhere in the caption. It is hard to find the meaning of some notations in the table such as \"37K images\", \"Area 400x400\". Please clearly state the unit.\n2. Please include some baseline results into the tables so that readers know how much improvments achieved by your approach.\n3. Please reorganize the paper to make it more clear. \n4. Please provide more literature review and discuss the difference of your work to existing ones.\n\n\n## Questions\n1. How large is the Berlin data?\n2. How would your method work when there is only a street image? How would you method scale when 3D model is not always available.\n3. How would you method work when there is some places without much structure? Rural area could be an example.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission studies the problem of geolocalizing a city based on geometric information encoded in so called "lean" images.  The reviewers were unanimous in their opinion that the submission does not meet the threshold for publication at ICLR.  Concerns included quality of writing, novelty with respect to existing literature (in particular see Review #2), and limited validation on one geographic area.  No rebuttal was provided.
Decision: Reject