EgoMap: Projective mapping and structured egocentric memory for Deep RL

1 INTRODUCTION . A critical part of intelligence is navigation , memory and planning . An animal that is able to store and recall pertinent information about their environment is likely to exceed the performance of an animal whose behavior is purely reactive . Many control problems in partially observed 3D environments involve long term dependencies and planning . Solving these problems requires agents to learn several key capacities : spatial reasoning — to explore the environment in an efficient manner and to learn spatio-temporal regularities and affordances . The agent needs to autonomously navigate , discover relevant objects , store their positions for later use , their possible interactions and the eventual relationships between the objects and the task at hand . Semantic mapping is a key feature in these tasks . A second feature is discovering semantics from interactions — while solutions exist for semantic mapping and semantic SLAM Civera et al . ( 2011 ) ; Tateno et al . ( 2017 ) , a more interesting problem arises when the semantics of objects and their affordances are not supervised , but defined through the task and thus learned from reward . A typical approach for these types of problems are agents based on deep neural networks including recurrent hidden states , which encode the relevant information of the history of observations Mirowski et al . ( 2017 ) ; Jaderberg et al . ( 2017 ) . If the task requires navigation , the hidden state will naturally be required to store spatially structured information . It has been recently reported that spatial structure as inductive bias can improve the performance on these tasks . In Parisotto & Salakhutdinov ( 2018 ) , for instance , different cells in a neural map correspond to different positions of the agent . In our work , we go beyond structuring the agent ’ s memory with respect to the agent ’ s position . We use projective geometry as an inductive bias to neural networks , allowing the agent to structure its memory with respect to the locations of objects it perceives , as illustrated in Figure 1b . The model performs an inverse projection of CNN feature vectors in order to map and store observations in an egocentric ( bird ’ s eye view ) spatially structured memory . The EgoMap is complementary to the hidden state vector of the agent and is read with a combination of a global convolutional read operation and an attention model allowing the agent to query the presence of specific content . We show that incorporating projective spatial memory enables the agent to learn policies that exceed the performance of a standard recurrent agent . Two different objects visible in the same input image could be at very different places in the environment . In contrast to Parisotto & Salakhutdinov ( 2018 ) , our model will map these observations to their respective locations , and not to cells corresponding to the agent ’ s position , as shown in Figure 1a . The model bears a certain structural resemblance with Bayesian occupancy grids ( BOG ) , which have been used in mobile robotics for many years Moravec ( 1988 ) ; Rummelhard et al . ( 2015 ) . As in BOGs , we perform inverse projections of observations and dynamically resample the map to take into account ego-motion . However , in contrast to BOGs , our model does not require a handcrafted observation model and it learns semantics directly from interactions with the environment through reward . It is fully differentiable and trained end-to-end with backpropagation of derivatives calculated with policy gradient methods . Our contributions are as follows : • To our knowledge , we present the first method using a differentiable SLAM-like mapping of visual features into a top-down egocentric feature map using projective geometry while at the same time training this representation using RL from reward . • Our spatial map can be translated and rotated through a differentiable affine transform and read globally and through self-attention . • We show that the mapping , spatial memory and self-attention can be learned end-to-end with RL , avoiding the cost of labelling trajectories from domain experts , auxiliary supervision or pre-training specific parts of the architecture . • We demonstrate the improvement in performance over recurrent and spatial structured baselines without projective geometry . • We illustrate the reasoning abilities of the agent by visualizing the content of the spatial memory and the self-attention process , tying it to the different objects and affordances related to the task . • Experiments with noisy actions demonstrate the agent is robust to actions tolerances of up to 10 % . The code will be made publicly available on acceptance . 2 RELATED WORK . Reinforcement learning — In recent years the field of Deep Reinforcement Learning ( RL ) has gained attention with successes on board games Silver et al . ( 2018 ) and Atari games Mnih et al . ( 2015 ) . One key component was the application of deep neural networks Lecun et al . ( 1998 ) to frames from the environment or game board states . Recent works that have applied Deep RL for the control of an agent in 3D environments such as maze navigation are Mirowski et al . ( 2017 ) and Jaderberg et al . ( 2017 ) which explored the use of auxiliary tasks such as depth prediction , loop detection and reward prediction to accelerate learning . Meta RL approaches for 3D navigation have been applied by Wang et al . ( 2016 ) and Lample & Chaplot ( 2017 ) also accelerated the learning process in 3D environments by prediction of tailored game features . There has also been recent work in the use of street-view scenes to train an agent to navigate in city environments Mirowski et al . ( 2018 ) . In order to infer long term dependencies and store pertinent information about the partially observable environment ; network architectures typically incorporate recurrent memory such as Gated Recurrent Units Chung et al . ( 2015 ) or Long Short-Term Memory Hochreiter & Schmidhuber ( 1997 ) . Differentiable memory — Differentiable memory such as Neural Turing Machines Graves et al . ( 2014 ) and Differential Neural Computers Graves et al . ( 2016 ) have shown promise where long term dependencies and storage are required . Neural Networks augmented with these memory structures have been shown to learn tasks such as copying , repeating and sorting . Some recent works for control in 2D and 3D environments have included structured memory-based architectures and mapping of observations . Neural SLAM Zhang et al . ( 2017 ) aims to incorporate a SLAM-like mapping module as part of the network architecture , but uses simulated sensor data rather than RGB observations from the environment , so the agent is unable to extract semantic meaning from its observations . The experimental results focus on 2D environments and the 3D results are limited . Playing Doom with SLAM augmented memory Bhatti et al . ( 2016 ) implements a non-differentiable inverse projective mapping with a fixed feature extractor based on Faster-RCNN Ren et al . ( 2015 ) , pre-trained in a supervised manner . A downside of this approach is that the network does not learn to extract features pertinent to the task at hand as it is not trained end-to-end with RL . Fang et al . ( 2019 ) replace recurrent memory with a transformer ( Vaswani et al . ( 2017 ) ) attention distribution over previous observation embeddings , to highlight that recurrent architectures can struggle to capture long term dependencies . The downside is the storage of previous observations grows linearly with each step in the environment and the agent can not chose to discard redundant information . Grid cells — there is evidence that biological agents learn to encode spatial structure . Rats develop grid cells/neurons , which fire at different locations with different frequencies and phases , a discovery that led to the 2014 Nobel prize in medicine O ’ Keefe & Dostrovsky ( 1971 ) ; Hafting et al . ( 2005 ) . A similar structure seems to emerge in artificial neural networks trained to localize themselves in a maze , discovered independently in 2018 by two different research groups Cueva & Wei ( 2018 ) ; Banino et al . ( 2018 ) . Projective geometry and spatial memory — Our work encodes spatial structure directly into the agent as additional inductive bias . We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations , justifying a fully differentiable model of perception . To our knowledge , we present the first method which uses projective geometry as inductive bias while at the same time learning spatial semantic features with RL from reward . The past decade has seen an influx of affordable depth sensors . This has led to a many works in the domain reconstruction of 3D environments , which can be incorporated into robotic systems . Seminal works in this field include Izadi et al . ( 2011 ) who performed 3D reconstruction scenes using a moving Kinect sensor and Henry et al . ( 2014 ) who created dense 3D maps using RGB-D cameras . Neural Map Parisotto & Salakhutdinov ( 2018 ) implements a structured 2D differentiable memory which was tested in both egocentric and world reference frames , but does not map observations in a SLAM-like manner and instead stores a single feature vector at the agent ’ s current location . The agent ’ s position is also discretized to fixed cells and orientation quantized to four angles ( North , South , East , West ) . A further downside is that the movement of the memory is fixed to discrete translations and the map is not rotated to the agent ’ s current viewpoint . MapNet Henriques & Vedaldi ( 2018 ) includes an inverse mapping of CNN features , is trained in a supervised manner to predict x , y position and rotation from human trajectories , but does not use the map for control in an environment . Visual Question Answering in Interactive Environments Gordon et al . ( 2018 ) creates semantic maps from 3D observations for planning and question answering and is applied in a discrete state space . Unsupervised Predictive Memory in a Goal-Directed Agent Wayne et al . ( 2018 ) incorporates a Differential Neural Computer in an RL agent ’ s architecture and was applied to simulated memorybased tasks . The architecture achieves improved performance over a typical LSTM Hochreiter & Schmidhuber ( 1997 ) based RL agent , but does not include spatial structure or projective mapping . In addition , visual features and neural memory are learned through the reconstruction of observations and actions , rather than for a specific task . Cognitive Mapping and Planning for Visual Navigation Gupta et al . ( 2017 ) applies a differentiable mapping process on 3D viewpoints in a discrete grid-world , trained with imitation learning which provides supervision on expert trajectories . The downside of discretization is that affine sampling is trivial for rotations of 90-degree increments , and this motion is not representative of the real world . Their tasks are simple point-goal problems of up to 32 time-steps , whereas our work focused on complex multi-step objectives in a continuous state space . Their reliance on imitation learning highlights the challenge of training complex neural architectures with reward alone , particular on tasks with sparse reward such as the ones presented in this paper . Learning Exploration Policies for Navigation Chen et al . ( 2019 ) , do not learn a perception module but instead map the depth buffer to a 2D map to provide a map-based exploration reward . Our work learns the features that can be mapped so the agent can query not only occupancy , but task-related semantic content . Our work greatly exceeds the performance of Neural Map Parisotto & Salakhutdinov ( 2018 ) , by embedding a differentiable inverse projective transform and a continuous egocentric map into the agent ’ s network architecture . The mapping of the environment is in the agent ’ s reference frame , including translation and rotation with a differentiable affine transform . We demonstrate stable training with reinforcement learning alone , over several challenging tasks and random initializations , and do not require the expense of acquiring expert trajectories . We detail the key similarities and differences with related work in table 1 .

Summary: The paper proposes a novel architecture for spatially structured memory. The main idea is to incorporate inductive bias/invariance derived from projective geometry arguments. The experiments seem to clearly show that this new architecture improves previous approaches to tasks which require spatial reasoning and memory, and the ablations studies and visualizations provide useful insights into the workings of the agent. One thing I'm missing is an experiment showing that this inductive bias also doesn't degrade performance on tasks where spatial reasoning is not necessary (as compared to vanilla GRU/LSTM).