EgoMap: Projective mapping and structured egocentric memory for Deep RL

1 INTRODUCTION . A critical part of intelligence is navigation , memory and planning . An animal that is able to store and recall pertinent information about their environment is likely to exceed the performance of an animal whose behavior is purely reactive . Many control problems in partially observed 3D environments involve long term dependencies and planning . Solving these problems requires agents to learn several key capacities : spatial reasoning — to explore the environment in an efficient manner and to learn spatio-temporal regularities and affordances . The agent needs to autonomously navigate , discover relevant objects , store their positions for later use , their possible interactions and the eventual relationships between the objects and the task at hand . Semantic mapping is a key feature in these tasks . A second feature is discovering semantics from interactions — while solutions exist for semantic mapping and semantic SLAM Civera et al . ( 2011 ) ; Tateno et al . ( 2017 ) , a more interesting problem arises when the semantics of objects and their affordances are not supervised , but defined through the task and thus learned from reward . A typical approach for these types of problems are agents based on deep neural networks including recurrent hidden states , which encode the relevant information of the history of observations Mirowski et al . ( 2017 ) ; Jaderberg et al . ( 2017 ) . If the task requires navigation , the hidden state will naturally be required to store spatially structured information . It has been recently reported that spatial structure as inductive bias can improve the performance on these tasks . In Parisotto & Salakhutdinov ( 2018 ) , for instance , different cells in a neural map correspond to different positions of the agent . In our work , we go beyond structuring the agent ’ s memory with respect to the agent ’ s position . We use projective geometry as an inductive bias to neural networks , allowing the agent to structure its memory with respect to the locations of objects it perceives , as illustrated in Figure 1b . The model performs an inverse projection of CNN feature vectors in order to map and store observations in an egocentric ( bird ’ s eye view ) spatially structured memory . The EgoMap is complementary to the hidden state vector of the agent and is read with a combination of a global convolutional read operation and an attention model allowing the agent to query the presence of specific content . We show that incorporating projective spatial memory enables the agent to learn policies that exceed the performance of a standard recurrent agent . Two different objects visible in the same input image could be at very different places in the environment . In contrast to Parisotto & Salakhutdinov ( 2018 ) , our model will map these observations to their respective locations , and not to cells corresponding to the agent ’ s position , as shown in Figure 1a . The model bears a certain structural resemblance with Bayesian occupancy grids ( BOG ) , which have been used in mobile robotics for many years Moravec ( 1988 ) ; Rummelhard et al . ( 2015 ) . As in BOGs , we perform inverse projections of observations and dynamically resample the map to take into account ego-motion . However , in contrast to BOGs , our model does not require a handcrafted observation model and it learns semantics directly from interactions with the environment through reward . It is fully differentiable and trained end-to-end with backpropagation of derivatives calculated with policy gradient methods . Our contributions are as follows : • To our knowledge , we present the first method using a differentiable SLAM-like mapping of visual features into a top-down egocentric feature map using projective geometry while at the same time training this representation using RL from reward . • Our spatial map can be translated and rotated through a differentiable affine transform and read globally and through self-attention . • We show that the mapping , spatial memory and self-attention can be learned end-to-end with RL , avoiding the cost of labelling trajectories from domain experts , auxiliary supervision or pre-training specific parts of the architecture . • We demonstrate the improvement in performance over recurrent and spatial structured baselines without projective geometry . • We illustrate the reasoning abilities of the agent by visualizing the content of the spatial memory and the self-attention process , tying it to the different objects and affordances related to the task . • Experiments with noisy actions demonstrate the agent is robust to actions tolerances of up to 10 % . The code will be made publicly available on acceptance . 2 RELATED WORK . Reinforcement learning — In recent years the field of Deep Reinforcement Learning ( RL ) has gained attention with successes on board games Silver et al . ( 2018 ) and Atari games Mnih et al . ( 2015 ) . One key component was the application of deep neural networks Lecun et al . ( 1998 ) to frames from the environment or game board states . Recent works that have applied Deep RL for the control of an agent in 3D environments such as maze navigation are Mirowski et al . ( 2017 ) and Jaderberg et al . ( 2017 ) which explored the use of auxiliary tasks such as depth prediction , loop detection and reward prediction to accelerate learning . Meta RL approaches for 3D navigation have been applied by Wang et al . ( 2016 ) and Lample & Chaplot ( 2017 ) also accelerated the learning process in 3D environments by prediction of tailored game features . There has also been recent work in the use of street-view scenes to train an agent to navigate in city environments Mirowski et al . ( 2018 ) . In order to infer long term dependencies and store pertinent information about the partially observable environment ; network architectures typically incorporate recurrent memory such as Gated Recurrent Units Chung et al . ( 2015 ) or Long Short-Term Memory Hochreiter & Schmidhuber ( 1997 ) . Differentiable memory — Differentiable memory such as Neural Turing Machines Graves et al . ( 2014 ) and Differential Neural Computers Graves et al . ( 2016 ) have shown promise where long term dependencies and storage are required . Neural Networks augmented with these memory structures have been shown to learn tasks such as copying , repeating and sorting . Some recent works for control in 2D and 3D environments have included structured memory-based architectures and mapping of observations . Neural SLAM Zhang et al . ( 2017 ) aims to incorporate a SLAM-like mapping module as part of the network architecture , but uses simulated sensor data rather than RGB observations from the environment , so the agent is unable to extract semantic meaning from its observations . The experimental results focus on 2D environments and the 3D results are limited . Playing Doom with SLAM augmented memory Bhatti et al . ( 2016 ) implements a non-differentiable inverse projective mapping with a fixed feature extractor based on Faster-RCNN Ren et al . ( 2015 ) , pre-trained in a supervised manner . A downside of this approach is that the network does not learn to extract features pertinent to the task at hand as it is not trained end-to-end with RL . Fang et al . ( 2019 ) replace recurrent memory with a transformer ( Vaswani et al . ( 2017 ) ) attention distribution over previous observation embeddings , to highlight that recurrent architectures can struggle to capture long term dependencies . The downside is the storage of previous observations grows linearly with each step in the environment and the agent can not chose to discard redundant information . Grid cells — there is evidence that biological agents learn to encode spatial structure . Rats develop grid cells/neurons , which fire at different locations with different frequencies and phases , a discovery that led to the 2014 Nobel prize in medicine O ’ Keefe & Dostrovsky ( 1971 ) ; Hafting et al . ( 2005 ) . A similar structure seems to emerge in artificial neural networks trained to localize themselves in a maze , discovered independently in 2018 by two different research groups Cueva & Wei ( 2018 ) ; Banino et al . ( 2018 ) . Projective geometry and spatial memory — Our work encodes spatial structure directly into the agent as additional inductive bias . We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations , justifying a fully differentiable model of perception . To our knowledge , we present the first method which uses projective geometry as inductive bias while at the same time learning spatial semantic features with RL from reward . The past decade has seen an influx of affordable depth sensors . This has led to a many works in the domain reconstruction of 3D environments , which can be incorporated into robotic systems . Seminal works in this field include Izadi et al . ( 2011 ) who performed 3D reconstruction scenes using a moving Kinect sensor and Henry et al . ( 2014 ) who created dense 3D maps using RGB-D cameras . Neural Map Parisotto & Salakhutdinov ( 2018 ) implements a structured 2D differentiable memory which was tested in both egocentric and world reference frames , but does not map observations in a SLAM-like manner and instead stores a single feature vector at the agent ’ s current location . The agent ’ s position is also discretized to fixed cells and orientation quantized to four angles ( North , South , East , West ) . A further downside is that the movement of the memory is fixed to discrete translations and the map is not rotated to the agent ’ s current viewpoint . MapNet Henriques & Vedaldi ( 2018 ) includes an inverse mapping of CNN features , is trained in a supervised manner to predict x , y position and rotation from human trajectories , but does not use the map for control in an environment . Visual Question Answering in Interactive Environments Gordon et al . ( 2018 ) creates semantic maps from 3D observations for planning and question answering and is applied in a discrete state space . Unsupervised Predictive Memory in a Goal-Directed Agent Wayne et al . ( 2018 ) incorporates a Differential Neural Computer in an RL agent ’ s architecture and was applied to simulated memorybased tasks . The architecture achieves improved performance over a typical LSTM Hochreiter & Schmidhuber ( 1997 ) based RL agent , but does not include spatial structure or projective mapping . In addition , visual features and neural memory are learned through the reconstruction of observations and actions , rather than for a specific task . Cognitive Mapping and Planning for Visual Navigation Gupta et al . ( 2017 ) applies a differentiable mapping process on 3D viewpoints in a discrete grid-world , trained with imitation learning which provides supervision on expert trajectories . The downside of discretization is that affine sampling is trivial for rotations of 90-degree increments , and this motion is not representative of the real world . Their tasks are simple point-goal problems of up to 32 time-steps , whereas our work focused on complex multi-step objectives in a continuous state space . Their reliance on imitation learning highlights the challenge of training complex neural architectures with reward alone , particular on tasks with sparse reward such as the ones presented in this paper . Learning Exploration Policies for Navigation Chen et al . ( 2019 ) , do not learn a perception module but instead map the depth buffer to a 2D map to provide a map-based exploration reward . Our work learns the features that can be mapped so the agent can query not only occupancy , but task-related semantic content . Our work greatly exceeds the performance of Neural Map Parisotto & Salakhutdinov ( 2018 ) , by embedding a differentiable inverse projective transform and a continuous egocentric map into the agent ’ s network architecture . The mapping of the environment is in the agent ’ s reference frame , including translation and rotation with a differentiable affine transform . We demonstrate stable training with reinforcement learning alone , over several challenging tasks and random initializations , and do not require the expense of acquiring expert trajectories . We detail the key similarities and differences with related work in table 1 .

Summary: This paper studies how to build semantic spatial maps for the purpose of navigation in 3D environments. The paper presents a differentiable policy network that pastes together semantic map predictions into a spatial map. Information is read out from this map using a global read operation (that looks at the entire map) and a self-attention read operation. This information is used to produce actions. The paper presents experimental results in 3D VizDoom scenarios and reports improvements over a vanilla LSTM, and another spatial memory based method (Neural Map).