Learning models for visual 3D localization with implicit mapping

We consider learning based methods for visual localization that do not require the construction of explicit maps in the form of point clouds or voxels . The goal is to learn an implicit representation of the environment at a higher , more abstract level , for instance that of objects . We propose to use a generative approach based on Generative Query Networks ( GQNs , Eslami et al. , 2018 ) , asking the following questions : 1 ) Can GQN capture more complex scenes than those it was originally demonstrated on ? 2 ) Can GQN be used for localization in those scenes ? To study this approach we consider procedurally generated Minecraft worlds , for which we can generate images of complex 3D scenes along with camera pose coordinates . We first show that GQNs , enhanced with a novel attention mechanism can capture the structure of 3D scenes in Minecraft , as evidenced by their samples . We then apply the models to the localization problem , comparing the results to a discriminative baseline , and comparing the ways each approach captures the task uncertainty . 1 INTRODUCTION . The problem of identifying the position of the camera that captured an image of a scene has been studied extensively for decades ( Triggs et al. , 1999 ; Thrun et al. , 2005 ; Cadena et al. , 2016 ) . With applications in domains such as robotics and autonomous driving , a considerable amount of engineering effort has been dedicated to developing systems for different versions of the problem . One formulation , often referred to as simply ‘ localization ’ , assumes that a map of the 3D scene is provided in advance and the goal is to localize any new image of the scene relative to this map . A second formulation , commonly referred to as ‘ Simultaneous Localization and Mapping ’ ( SLAM ) , assumes that there is no prespecified map of the scene , and that it should be estimated concurrently with the locations of each observed image . Research on this topic has focused on different aspects and challenges including : estimating the displacement between frames in small time scales , correcting accumulated drifts in large time scales ( also known as ‘ loop closure ’ ) , extracting and tracking features from the observed images ( e.g . Lowe , 2004 ; Mur-Artal et al. , 2015 ) , reducing computational costs of inference ( graph-based SLAM , Grisetti et al. , 2010 ) and more . Although this field has seen huge progress , performance is still limited by several factors ( Cadena et al. , 2016 ) . One key limitation stems from reliance on hand-engineered representations of various components of the problem ( e.g . key-point descriptors as representations of images and occupancy grids as representations of the map ) , and it has been argued that moving towards systems that operate using more abstract representations is likely to be beneficial ( Salas-Moreno et al. , 2013 ; McCormac et al. , 2017 ) . Considering the map , it is typically specified as either part of the input provided to the system ( in localization ) or part of its expected output ( in SLAM ) . This forces algorithm designers to define its structure explicitly in advance , e.g . either as 3D positions of key-points , or an occupancy grid of a pre-specified resolution . It is not always clear what the optimal representation is , and the need to pre-define this structure is restricting and can lead to sub-optimal performance . In this work , we investigate the problem of localization with implicit mapping , by considering the ‘ re-localization ’ task , using learned models with no explicit form of map . Given a collection of ‘ context ’ images with known camera poses , ‘ re-localization ’ is defined as finding the relative camera pose of a new image which we call the ‘ target ’ . This task can also be viewed as loop closure without an explicit map . We consider deep models that learn implicit representations of the map , where at the price of making the map less interpretable , the models can capture abstract descriptions of the environment and exploit abstract cues for the localization problem . In recent years , several methods for localization that are based on machine learning have been proposed . Those methods include models that are trained for navigation tasks that require localization . In some cases the models do not deal with the localization explicitly , and show that implicit localization and mapping can emerge using learning approaches ( Mirowski et al. , 2016 ; Cueva & Wei , 2018 ; Banino et al. , 2018 ; Wayne et al. , 2018 ) . In other cases , the models are equipped with specially designed spatial memories , that can be thought of as explicit maps ( Zhang et al. , 2017 ; Gupta et al. , 2017a ; Parisotto & Salakhutdinov , 2017 ) . Other types of methods tackle the localization problem more directly using models that are either trained to output the relative pose between two images ( Zamir et al. , 2016 ; Ummenhofer et al. , 2017 ) , or models like PoseNet ( Kendall et al. , 2015 ) , trained for re-localization , where the model outputs the camera pose of a new image , given a context of images with known poses . Models like PoseNet do not have an explicit map , and therefore have an opportunity to learn more abstract mapping cues that can lead to better performance . However , while machine learning approaches outperform hand-crafted methods in many computer vision tasks , they are still far behind in visual localization . A recent study ( Walch et al. , 2017 ) shows that a method based on traditional structure-from-motion approaches ( Sattler et al. , 2017a ) significantly outperforms PoseNet . One potential reason for this , is that most machine learning methods for localization , like PoseNet , are discriminative , i.e . they are trained to directly output the camera pose . In contrast , hand-crafted methods are usually constructed in a generative approach , where the model encodes geometrical assumptions in the causal direction , concerning the reconstruction of the images given the poses of the camera and some key-points . Since localization relies heavily on reasoning with uncertainty , modeling the data in the generative direction could be key in capturing the true uncertainty in the task . Therefore , we investigate here the use of a generative learning approach , where on one hand the data is modeled in the same direction as hand-crafted methods , and on the other hand there is no explicit map structure , and the model can learn implicit and abstract representations . A recent generative model that has shown promise in learning representations for 3D scene structure is the Generative Query Network ( GQN ) ( Eslami et al. , 2018 ) . The GQN is conditioned on different views of a 3D scene , and trained to generate images from new views of the same scene . Although the model demonstrates generalization of the representation and rendering capabilities to held-out 3D scenes , the experiments are in simple environments that only exhibit relatively well-defined , small-scale structure , e.g . a few objects in a room , or small mazes with a few rooms . In this paper , we ask two questions : 1 ) Can the GQN scale to more complex , visually rich environments ? 2 ) Can the model be used for localization ? To this end , we propose a dataset of random walks in the game Minecraft ( figure 1 ) , using the Malmo platform ( Johnson et al. , 2016 ) . We show that coupled with a novel attention mechanism , the GQN can capture the 3D structure of complex Minecraft scenes , and can use this for accurate localization . In order to analyze the advantages and shortcomings of the model , we compare to a discriminative version of it , similar to previously proposed methods . In summary , our contributions are the following : 1 ) We suggest a generative approach to localization with implicit mapping and propose a dataset for it . 2 ) We enhance the GQN using a novel sequential attention model , showing it can capture the complexity of 3D scenes in Minecraft . 3 ) We show that GQN can be used for localization , finding that it performs comparably to our discriminative baseline when used for point estimates , and investigate how the uncertainty is captured in each approach . 2 DATA FOR LOCALIZATION WITH IMPLICIT MAPPING . In order to study machine learning approaches to visual localization with implicit mapping , we create a dataset of random walks in a visually rich simulated environment . We use the Minecraft environment through the open source Malmo platform ( Johnson et al. , 2016 ) . Minecraft is a popular computer game where a player can wander in a virtual world , and the Malmo platform allows us to connect to the game engine , control the player and record data . Since the environment is procedurally generated , we can generate sequences of images in an unlimited number of different scenes . Figure 1 shows the visual richness of the environment and the diversity of the scene types including forests , lakes , deserts , mountains etc . The images are not realistic but nevertheless contain many details such as trees , leaves , grass , clouds , and also different lighting conditions . Our dataset consists of 6000 sequences of 100 images each generated by a blind exploration policy based on simple heuristics ( see appendix A for details ) . We record images at a resolution of 128×128 although for all the experiments in this paper we downscale to 32× 32 . Each image is recorded along with a 5 dimensional vector of camera pose coordinates consisting of the x , y , z position and yaw and pitch angles ( omitting the roll as it is constant ) . Figure 1 also demonstrates the localization task : For every sequence we are given a context of images along with their camera poses , and an additional target image with unknown pose . The task is to find the camera pose of the target image . While other datasets for localization with more realistic images have been recently proposed ( Armeni et al. , 2016 ; Savva et al. , 2017 ; Chang et al. , 2017 ; Song et al. , 2017 ) , we choose to use Minecraft data for several reasons . First , in terms of the visual complexity , it is a smaller step to take from the original GQN data , which still poses interesting questions on the model ’ s ability to scale . The generative approach is known to be hard to scale but nevertheless has advantages and this intermediate dataset allows us to demonstrate them . The second reason , is that Minecraft consists of outdoors scenes which compared to indoor environments , contain a 3D structure that is larger in scale , and where localization should rely on more diverse clues such as using both distant key-points ( e.g . mountains in the horizon ) and closer structure ( e.g . trees and bushes ) . Indoor datasets also tend to be oriented towards sequential navigation tasks relying much on strong priors of motion . Although in practice , agents moving in an environment usually have access to dense observations , and can exploit the fact that the motion between subsequent observations is very small , we focus here on localization with a sparse observation set , since we are interested in testing the model ’ s ability to capture the underlying structure of the scene only from observations and without any prior on the camera ’ s motion . One example for this setting is when multiple agents can broadcast observations to help each other localize . In this case there are usually constraints on the network bandwidth and therefore on the number of observations , and there is also no strong prior on the camera poses which all come from different cameras .

Summary: Eslami et al. (2018) proposed a deep neuronal framework for a scene representation and renderer (the Generative Query Networks: GQN), which generate an image from a scene representation and a query camera pose. In this work, the authors use the GQN to estimate the camera pose from a target image. Existing learning approaches are discriminative, meaning that they are trained to output the camera pose in an end-to-end fashion, while this paper proposes a generative method more in the line of hand-crafted methods which still largely outperform learning approaches. Using the GQN with the proposed attention mechanism, the method captures an implicit mapping of the environment at a more abstract level. This implicit representation is then used to optimize the likelihood of the target pose in a probabilistic graphical model framing. They compare their solution to a discriminative baseline, based on a reversed GQN.