Lean Images for Geo-Localization

1 INTRODUCTION . Imagine you are brought blindfolded to a street corner of a city you know well . Now , you remove the blindfold . Can you tell where you are ? This is the geo-localization task . In computer vision , this amounts to estimating the position ( and sometimes the orientation ) of a camera given its current view . Although localization devices such as Global Positioning Systems ( GPS ) have improved significantly over the last years , they often do not work well in city scenes and do not provide highly accurate results . Autonomous cars , drones , and IOT devices are expected to benefit tremendously from the ability to determine their pose ( position & orientation ) accurately in their environment . A solution for geo-localization , either by a human or by a machine , can use appearance cues ( e.g. , texture of a unique building ) , geometric cues ( e.g. , a unique shape of a building ) , or both . Significant improvements were obtained in object recognition , scene recognition , and localization tasks , largely by exploiting the appearance of the scene rather than only its edges ( e.g. , color and texture and image features such as SIFT Se et al . ( 2002 ) ; Lowe ( 2004 ) ; Li et al . ( 2010 ) ) . Later , these methods were improved by adding coarse geometric constraints to the image features ( e.g. , Ramalingam et al . ( 2011 ) ; Bansal & Daniilidis ( 2014 ) ) . Nowadays , methods are based on machine learning , in particular convolutional neural networks ( CNNs ) , where the input is typically the unprocessed textured image . Clearly , both appearance and geometry play an important role in these methods . The main goal of this paper is to study the role of geometry in a CNN solution to the geo-localization task rather than propose a working system for application purposes . We study an end-to-end deep CNN for geo-localization , by ignoring the often available texture of the scene . To do this , we consider only lean images , which contain mostly information that relates to the geometry of the scene while lacking texture or rich geometric details . In practice , we use a city scene and consider two types of binary images that consist of the edges of the buildings ’ outline and the buildings ’ facades . In addition , we also consider depth images that contain more geometric information . Examples of the three types of lean images are shown in Fig . 1 ( left ) . Note that in the first row , the view contains dominant landmarks , while the second row shows very little distinct information that might be expected to assist localization . Such non-distinct views are very common in large environments such as a city , making localization with lean images very challenging . Further note that we deliberately do not consider real images or synthetic images with texture , since our goal is to study only the information available from pure geometric data . Our lean-images are obtained simply by projecting an untextured 3D mesh model of Berlin onto various positions in the scene ( See Sec . 3 ) . A bird ’ s-eye view of one of the areas is shown in Fig . 1 ( middle ) . Using such a model allows us to study the role of geometry for geo-localization in a controlled manner and in a larger scale than ever before , both in terms of the area covered ( many city streets ) and in terms of the number of images ( up to hundreds of thousands ) . Neural networks were shown to be effective in geo-localization tasks ( e.g. , Kendall et al . ( 2015 ) ; Walch et al . ( 2017 ) ; Melekhov et al . ( 2017 ) ; Sattler et al . ( 2017 ) ) . A typical geo-localization solution will return the pose of a camera in novel views . That is , the image query is not part of the training set . In this paper we ask ( i ) can the CNN generalize and support geo-localization in a large environments using lean images that contains only geometric and spatial data ? ( ii ) is it likely that geometric information is used by the CNN to solve this task ? However , to better understand the geo-localization task , we also consider the memorization capacity of the network for previously seen images . This can be regarded as image retrieval from a database of all available views of the city . A naı̈ve solution would store all images and then perform a brute-force search in the database . However , this is inefficient and can become infeasible as the database gets larger . Defining a compact representation and an efficient search is clearly desired . The question we address in this case is ( iii ) whether CNNs can be used to solve the meomorization task using lean images . As discussed in the results ( Sec . 6 ) , we found positive answers to all these questions , but the results depend on the number of images and their sampling density . We believe this indicates that networks can learn some sort of a spatial map for an area using only geometric data , since no colors or textures are available in our data . The success of geo-localization also depends on the specific position . Fig . 1 ( right ) shows how certain positions in the streets of a city are more recognizable than others . Our paper presents an empirical study regarding the information that can be used by CNNs to build an internal representation ; we do not propose a practical solution based on lean images . The main contributions of our study are : ( i ) showing that lean images contain sufficient information for solving the memorization and the geo-localization tasks ( ii ) proposing a systematic method to study the role of geometry in CNNs for geo-localization tasks ; ( iii ) demonstrating the power of CNNs to use the geometric information to build internal represenations of data . 2 RELATED WORK . Place recognition ( e.g. , the Eiffel Tower ) can be regarded as a coarse geo-localization task.Classic approaches use visual features to represent each image in a set of images of a given location ( e.g. , by a bag of words ) and then match a target image with the stored representations ( e.g. , Se et al . ( 2002 ) ; Sivic & Zisserman ( 2003 ) ; Lowe ( 2004 ) ; Robertson & Cipolla ( 2004 ) ) . Hays & Efros ( 2008 ) were the first to address the place recognition task using millions of geo-tagged images , based on various visual features ( e.g. , tiny images , color histograms , line features , gist descriptors ) . Memorization of image data-sets is often performed by manually engineered image representation ( e.g. , a dictionary of image features ) and an image retrieval approach , including the metric between the stored representation and a target one . ( e.g. , Sivic & Zisserman ( 2003 ) ; Lowe ( 2004 ) ; Robertson & Cipolla ( 2004 ) ; Zhang & Kosecka ( 2006 ) ; Nister & Stewenius ( 2006 ) ; Schindler et al . ( 2007 ) ; Hays & Efros ( 2008 ) ; Li et al . ( 2010 ) ) . We show that neural networks are able to efficiently create such a representation in various experiments . A possible solution for geo-localization task , where both position and orientation of a camera with respect to a scene should be estimated , is by using triangulation with images with known pose ( e.g. , Zhang & Kosecka ( 2006 ) ) . In most studies , 3D models of the scene are used by means of pointclouds ( e.g. , Irschara et al . ( 2009 ) ; Sattler et al . ( 2011 ) ; Matei et al . ( 2013 ) ; Svarm et al . ( 2014 ) ) , Digital Elevation Maps ( DEM ) ( e.g. , Baatz et al . ( 2012 ) ; Bansal & Daniilidis ( 2014 ) ) , or full 3D models ( e.g. , Ramalingam et al . ( 2011 ) ) . One of the main challenges of these works is to develop an efficient computation of 2D to 3D feature matching . New 3D feature representation have also been developed ( e.g. , Irschara et al . ( 2009 ) ; Sattler et al . ( 2011 ) ) . Bansal & Daniilidis ( 2014 ) introduce a feature more closely related to the lean images we consider . It consists of 3D corners and direction vectors extracted from a Digital Elevation Map ( DEM ) to be matched geometrically to the corners and roof-line edges of buildings visible in a street-level query image . Efficiency and robustness become even more important when dealing with a city-scale 3D model . A fast method for inliers detection that enables solving the correspondence problem on such a scale was suggested by Svärm et al . ( 2017 ) . Recent survey on existing localization methods can be found in Piasco et al . ( 2018 ) . One of the key ideas that bypasses the challenge of defining an efficient and robust 2D-3D feature matching required by the abovementioned methods is to use an end-to-end CNN solution that performs both feature extraction and matching . PoseNet Kendall et al . ( 2015 ) is an impressive CNN based approach for solving the pose of real images . A dataset of images was used for training Google LeNet Szegedy et al . ( 2015 ) where the 6-DoF pose of the camera was used as ground truth . Walch et al . ( 2017 ) suggested an improvement to the PoseNet by adding an LSTM , which reduces the dimensionality of the feature vector . Melekhov et al . ( 2017 ) used ResNet34 He et al . ( 2016 ) , which uses encoder-decoder structure to improve model accuracy . Kendall & Cipolla ( 2016 ) improved their earlier work Kendall et al . ( 2015 ) by applying an uncertainty framework to the CNN pose regressor . In another work , Kendall et al . ( 2017 ) studied the affect of various loss functions on the result of PoseNet . In our study we assume a 3D model of a city is given . Our setup is very challenging since the model and the images consists of only coarse 3D structure of the scene without texture for computing image features . On the other hand , our images are noise-free and there are no object-level occlusions such as trees , cars and people . In addition , we were not limited by data size , as we projected as many images as we chose . Most importantly , our goal differs from that of the aforementioned methods : whereas they focus on obtaining a better and faster solution for geo-localization , we focus on trying to understand the role of geometry , alone , in geo-localization , by systematically training and testing the same neural network on controlled datasets . 3 DATA . Using a 3D model as the data source we can sample as many images as necessary in any position , orientation , and resolution . In our study we used a simplified 3D model of Berlin für Wirtschaft und Technologie GmbH ( 2016 ) . The model contains only the geometry of building walls and rooftops , and does not contain any fine geometric details such as window frames or doors or texture ( see bird-eyes view in Fig . 1 ) . We consider three types of images which are projections from this model : edge , face , and depth map , see Fig . 1 , and we call them lean images since they contain no texture or structural details . The face images contain the buildings ’ planar facades . Each image is defined by its camera pose . That is , ( x , y , θ , φ ) , where ( x , y ) is the position on the ground plane and ( θ , φ ) is the camera Y aw and Pitch angles . We assume for simplicity that the picture is taken at a fixed height , and the roll angle is fixed as horizontal . We generated several image datasets that are sampled uniformly along this 4D grid . The density in the ( x , y ) domain varies between the data-sets but fixed in the ( θ , φ ) domain . A 6D pose vector ( instead of 4D ) in the form of ( x , y , q1 , q2 , q3 , q4 ) , is used to represent the camera pose . The q1 , q2 , q3 , q4 are the quaternions representation of the ( θ , φ ) Euler angles . Each set of images is created in a predefined area of the city . The number of images in the set is determined by the size of the area and the grid sampling density . All three types of lean images were generated for each sample pose . When dealing with lean images , care must be taken not to include empty images . For example when the camera is facing a building wall from a short distance . Such images contain almost no visual information and do not contribute to the learning process . We discard an image that has less than 8 edges , or an image that does not contain a skyline ( at least 50 % of its top-most pixel row is sky ) . Moreover , images of positions inside a building are irrelevant to geo-localization , and are also discarded from training and test ( see Fig . 2 , right ) .

Summary: The paper evaluates a geo-localization task based on "lean" images only, obtained by projection of 3d models without texture information. Multiple levels of granularity of the lean images (edges/edges+faces/edges+faces+depth) are compared for the learning, both in a "memorization" setting and in a "generalization to unseen poses" setting. Moreover, the behavior of the learning when labels are shuffled is evaluated.