EgoMap: Projective mapping and structured egocentric memory for Deep RL
Authors: Edward Beeching
Christian Wolf
Jilles Dibangoye
Olivier Simonin
Venue:  (2020)
Keywords: Reinforcement Learning
Deep Learning
Computer Vision
Robotics
Neural Memory

Abstract: Tasks involving localization, memorization and planning in partially observable 3D environments are an ongoing challenge in Deep Reinforcement Learning. We present EgoMap, a spatially structured neural memory architecture. EgoMap augments a deep reinforcement learning agent’s performance in 3D environments on challenging tasks with multi-step objectives. The EgoMap architecture incorporates several inductive biases including a differentiable inverse projection of CNN feature vectors onto a top-down spatially structured map. The map is updated with ego-motion measurements through a differentiable affine transform. We show this architecture outperforms both standard recurrent agents and state of the art agents with structured memory. We demonstrate that incorporating these inductive biases into an agent’s architecture allows for stable training with reward alone, circumventing the expense of acquiring and labelling expert trajectories. A detailed ablation study demonstrates the impact of key aspects of the architecture and through extensive qualitative analysis, we show how the agent exploits its structured internal memory to achieve higher performance. 

Under review as a conference paper at ICLR 2020
EGOMAP: PROJECTIVE MAPPING AND STRUCTURED EGOCENTRIC MEMORY FOR DEEP RL
Anonymous authors Paper under double-bling review
ABSTRACT
Tasks involving localization, memorization and planning in partially observable 3D environments are an ongoing challenge in Deep Reinforcement Learning. We present EgoMap, spatially structured neural memory architecture. EgoMap aug- ments a deep reinforcement learning agent's performance 3D environments on challenging tasks with multi-step objectives. The EgoMap architecture incorpo- rates several inductive biases including differentiable inverse projection of CNN feature vectors onto top-down spatially structured map. The map is updated with ego-motion measurements through a differentiable affine transform. We show this architecture outperforms both standard recurrent agents and state of the art agents with structured memory. We demonstrate that incorporating these inductive biases into agent's architecture allows for stable training with reward alone, circumventing the expense of acquiring and labelling expert trajectories. A de- tailed ablation study demonstrates the impact of key aspects of the architecture and through extensive qualitative analysis, we show how the agent exploits its structured internal memory to achieve higher performance.
INTRODUCTION
critical part of intelligence is navigation, memory and planning. An animal that is able to store and recall pertinent information about their environment is likely to exceed the performance of an animal whose behavior is purely reactive. Many control problems in partially observed 3D environments involve long term dependencies and planning. Solving these problems requires agents to learn several key capacities: spatial reasoning
explore the environment an efficient manner and to learn spatio-temporal regularities and affordances. The agent needs to autonomously navigate, discover relevant objects, store their positions for later use, their possible interactions and the eventual relationships between the objects and the task at hand. Semantic mapping is key feature in these tasks. A second feature is discovering semantics from interactions semantic mapping and semantic SLAM Civera al. (201I); Tateno et al. (2017), a more interesting problem arises when the semantics of objects and their affordances are not supervised, but defined through the task and thus learned from reward.
while solutions exist for
typical approach for these types of problems are agents based on deep neural networks including recurrent hidden states, which encode the relevant information of the history of observations Mirowski et al. (2017); Jaderberg et al. (2017). If the task requires navigation, the hidden state will naturally be required to store spatially structured information. It has been recently reported that spatial structure as inductive bias can improve the performance on these tasks. In Parisotto & Salakhutdinov (2018). for instance, different cells in a neural map correspond to different positions the agent.
In our work, we beyond structuring the agent's 's memory with respect to the agent's position. We use projective geometry as an inductive bias to neural networks, allowing the agent to structure its memory with respect to the locations of objects it perceives, as illustrated in Figure [b] The model performs an inverse projection of CNN feature vectors in order to map and store observations in an egocentric (bird's eye view) spatially structured memory. The EgoMap is complementary to the hidden state vector of the agent and is read with a combination operation and an attention model allowing the agent to query the presence of specific content. We show that incorporating projective spatial memory enables the agent to learn policies that exceed the performance of standard recurrent agent. Two different objects visible in the same input image
global convolutional read

---

Under review as a conference paper at ICLR 2020
Controller O
Controller
Differentiable severs protective mapping
Spotial map (bird's eye view?
Ditoorenalle allie
Featurc HEAp Unovecttie
Temere prodettve
mapplig
Perception module
Perception module
Perception module
Observation
(prejersivo view
Time-step
Time-step t+1
(a) Visual features are mapped to the top-down egocentric map. Observations from a first-pee person viewpoint passed through a perception mod- ule, extracted features projected with the in- verse camera matrix and depth buffer to their 3D coordinates. The operations are implemented a differentiable manner, so error derivatives can be back propagated update the weights of the perception module, Different objects the same image are mapped different locations the map.
(b) An agent exploring in 3D environment per- ceives from a projective egocentric viewpoint. Our model learns to unproject learned task-oriented mantic embeddings of observations and map the positions of relevant objects a spatially structured (bird's s eye view) neural memory, where different objects in the same input image are stored in dif- ferent map locations. Using the ego-motion of the agent, the map updated by differentiable affine resampling at each step in the environment.
Figure 1: Overview of the perception and mapping module (a) and EgoMap agent architecture (b).
could be at very different places in the environment. In contrast to Parisotto & Salakhutdinov (2018), our model will map these observations their respective locations, and not cells corresponding to the agent's position, shown in Figure la
The model bears a certain structural resemblance with Bayesian occupancy grids (BOG), which have been used in mobile robotics for many years Moravec (1888) Rummelhard al. (2015) As in BOGs, we perform inverse projections of observations and dynamically resample the map to take into account ego-motion. However, in contrast to BOGs, model does not require handcrafted observation model and it learns semantics directly from interactions with the environment through reward. It is fully differentiable and trained end-to-end with backpropagation of derivatives calculated with policy gradient methods. Our contributions are as follows:
To our knowledge, we present the first method using a differentiable SLAM-like mapping of visual features into a top-down egocentric feature map using projective geometry while at the same time training this representation using RL from reward.
Our spatial map can be translated and rotated through differentiable affine transform and read globally and through self-attention.
We show that the mapping, spatial memory and self-attention can be learned end-to-end with RL, avoiding the cost of labelling trajectories from domain experts, auxiliary supervision or pre-training specific parts of the architecture.
We demonstrate the improvement performance over recurrent and spatial structured baselines without projective geometry.
We illustrate the reasoning abilities the agent by visualizing the content of the spatial memory and the self-attention process, tying it to the different objects and affordances related to the task. Experiments with noisy actions demonstrate the agent robust to actions tolerances of up to 10%.
The code will be made publicly available acceptance.
2

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Reinforcement learning - In recent years the field of Deep Reinforcement Learning (RL.) has gained attention with successes on board games Silver et al. (2018) and Atari games Mnih al.) (2013). One key component was the application of deep neural networks Lecun et al. (1998) to frames from the environment or game board states. Recent works that have applied Deep RL for the control of an agent in 3D environments such as maze navigation are Mirowski et (2017) and Jaderbergg al. (2017) which explored the use of auxiliary tasks such as depth prediction, loop detection and reward prediction to accelerate learning. Meta RL approaches for 3D navigation have been applied by Wang al. 2016) and Lample & Chaplot (2017) also accelerated the learning process in 3D environments by prediction of tailored game features. There has also been recent work in the use of street-view scenes to train an agent to navigate in city environments M Mirowski et al. (2018). In order to infer long term dependencies and store pertinent information about the partially observable environment; network architectures typically incorporate recurrent memory such as Gated Recurrent Units Chung et al. (2015) Long Short-Term Memory Hochreiter & Schmidhuber (1997).
Differentiable memory - Differentiable memory such as Neural Turing Machines Graves et al. (0114) and Differential Neural Computers Graves et al. (2016) have shown promise where long term dependencies and storage are required. Neural Networks augmented with these memory structures have been shown to learn tasks such as copying. repeating and sorting. Some recent works for control in 2D and 3D environments have included structured memory- architectures and mapping of observations. Neural SLAM ZZang et al. (2017) aims incorporate SLAM-like mapping module as part of the network architecture, but uses simulated sensor data rather than RGB observations from the environment, so the agent is unable extract semantic meaning from its observations. The experimental results focus on 2D environments and the 3D results are limited. Playing Doom with con-differentiable inverse projective
SLAM augmented memory Bhatti et al. (2016) implements mapping with fixed feature extractor based on Faster-RCNN Ren et al.] (2015), pre-trained in supervised manner. A downside of this approach is that the network does not learn to extract features pertinent to the task at hand as it is not trained end-to-end with Fang et al. (2019) replace recurrent memory with a transformer [Vaswani et al. al. (2017) attention distribution over previous observation embeddings, to highlight that recurrent architectures can struggle to capture long term dependencies. The downside is the storage of previous observations grows linearly with each step in the environment and the agent cannot chose to discard redundant information.
Grid cells - there is evidence that biological agents learn to encode spatial structure. Rats develop grid cells/neurons, which fire at different locations with different frequencies and phases. a discovery that led to the 2014 Nobel prize in medicine O' Keefe & Dostrovsky [) |Hafting et al. (2055).
A similar structure seems to emerge in artificial neural networks trained to localize themselves in a maze, discovered independently in 2018 by two different research groups Cueva & Weil (2018);
Banino et al. (2018).
Projective geometry and spatial memory - Our work encodes spatial structure directly into the agent as additional inductive bias. We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations, justifying fully differentiable model of perception. To our knowledge, we present the first method which uses projective geometry as inductive bias while at the same time learning spatial semantic features with RI. from reward.
The past decade has seen an influx of affordable depth sensors. This has led to a many works in the domain reconstruction of 3D environments, which can be incorporated into robotic systems. Seminal works in this field include Izadi et al. T011T) who performed 3D reconstruction scenes using a moving Kinect sensor and Henry et al. (2014) who created dense 3D maps using RGB-D cameras.
Neural Map Parisotto & Salakhutdinoy (2018) implements a structured 2D differentiable memory which was tested in both egocentric and world reference frames, but does not map observations in a SLAM-like manner and instead stores a single feature vector at the agent's 'ss current location. The agent's position is also discretized to fixed cells and orientation quantized to four angles (North, South, East, West). A further downside is that the movement of the memory is fixed to discrete translations and the map is not rotated to the agent's current viewpoint.
Heeniquuee & Vedaldi (2018) includes an inverse mapping of CNN features, is trained in a supervised manner predict x.y position and rotation from human trajectories, but does not use the
3

---

Under review as a conference paper at ICLR 2020
Spurial map leamne from reward
Tank- aak-speciffcc somontic features
Continueus Affine transform
Reinfor Leamiing
Continuaus State Space
End to-ean-d differentiabee
Continaous translaioon.
Projective Geonerty
Imiidtation learning
Sparially Straztured
Continous rotations
Ead 11--0-o 1o-----odd training
Intrinsic rewards
Maltplo goals
Visual input
Related work
Somantic mapping|Civera Civera et
2011
Playing Doom SLAM Bhalit
Neural SL.M (Zhng al.
Cog. Guplla al. 201
Semantic Lateno
2017
10A Gordon Y2018
MapNe[[Hennquus & Vedaldi 2018
Neural Map
huldmoy 2018
MERLIN Wayne et 2018
Learning exploration polictes Chen et al. 2019
EgeMap
Table I: Comparison of key features of related works
map for control in an environment. Visual Question Answering Interactive Environments Gordon it al. (2018) creates semantic maps from 3D observations for planning and question answering and is applied in a discrete state space.
Unsupervised Predictive Memory in Goal-Directed Agent Wayne et al. (2018) incorporates Differential Neural Computer in an RL agent's architecture and was applied to simulated memory- based tasks. The architecture achieves improved performance over a typical LSTM Hochreiter & Schmidhuber (1997) based RL agent, but does not include spatial structure or projective mapping. In addition, visual features and neural memory are learned through the reconstruction of observations and actions, rather than for a specific task.
Cognitive Mapping and Planning for Visual Navigation Gupta et al. (017)) applies differentiable mapping process on 3D viewpoints discrete grid-world, trained with imitation learning which provides supervision on expert trajectories. The downside of discretization is that affine sampling trivial for rotations 90-degree increments, and this motion not representative of the real world. Their tasks simple point-goal problems of up to 32 time-steps, whereas our work focused on complex multi-step objectives in a continuous state space. Their reliance on imitation learning highlights the challenge of training complex neural architectures with reward alone, particular on tasks with sparse reward such as the ones presented in this paper.
Learning Exploration Policies for Navigation Chen et al. (2019), do learn perception module but instead map the depth buffer to 2D map to provide a map- based exploration reward. Our work learns the features that can be mapped so the agent can query not only occupancy, but task-related semantic content.
Our work greatly exceeds the performance Neural Map Parisotto Salakhutdinov (018)), by embedding differentiable inverse projective transform and continuous egocentric map into the agent's network architecture. The mapping of the environment is the agent's reference frame, including translation and rotation with differentiable affine transform. We demonstrate stable training with reinforcement learning alone, over several challenging tasks and random initializations, and do not require the expense of acquiring expert trajectories. We detail the key similarities and differences with related work in table
EGOMAP
We consider partially observable Markov decision processes (POMDPs) Kaelbling al. (1998) in 3D environments and extend recent Deep-RL. models, which include a recurrent hidden layer to store pertinent long term information Mirowski et al. (2017): Jaderberg et al. (2017). In particular, RGBD observations I at time step are passed through perception module extracting features st, which are used to update the recurrent state:
8t = fp((:;pp)
h4 f=((--1,8650.)
4

---

Under review as a conference paper at ICLR 2020
where fg is a convolutional neural network and f. is recurrent neural network in the Gated Recurrent Unit variant Chung et al. )2015). Gates and their equations have been omitted for simplicity. Above and in the rest of this paper. 6, are trainable parameters, exact architectures are provided in the appendix. The controller outputs an estimate of the policy (the action distribution) and the value function given its hidden state:
(2) The proposed model is motivated by the regularities which govern 3D physical environments. When an agent perceives an observation of the 3D world, it observes 2D planar perspective projection of the world based on its current viewpoint. This projection is well understood physical process, aim to imbue the agent's 's architecture with an inductive bias based on inverting the 3D to 2D planar projective process. This inverse mapping operation appears to be second nature to many organisms, with the initial step of depth estimation being well studied in the field of Physiology FFrégnac et al. (2004). We believe that providing this mechanism implicitly the agent's architecture will improve its reasoning capabilities in new environments bypass a large part of the learning process.
Te T== fn (he: 0x)
V fb)
The overall concept is that as the agent explores the environment, the perception module fp produces a 2D feature map St. in which each feature vector represents learned semantic representation of small receptive field from the agent's ' egocentric observation. While they are integrated into the flat (not spatially structured) recurrent hidden state h, through function fr (Equation I we propose its integration into a second tensor Mr top-down egocentric memory, which we call EgoMap. The feature vectors are mapped to their egocentric positions using the inverse projection matrix and depth estimates. This requires an agent with a calibrated camera (known intrinsic parameters), which is soft constraint easily satisfied. The map can then be read by the agent in two ways: global convolutional read operation and a self-attention operation.
Formally, let the agent's 's position and angle at time be (I1, y1) and d, respectively, M, is the current EgoMap. & are the feature vectors extracted by the perception module, D, are the depth buffer values. The change in agent position and orientation in the agent's 's frame of reference between time-step I and 1-1 are (dæ,, dyt. déx)-
There are three key steps the operation of the EgoMap:
1. Transform the map to the agent's 's egocentric frame of reference:
M, Affine (M,, dyc, dé.))
(3)
2. Update the map to include new observations:
M, InverseProjectsss. D.)
Mr Combine M,)
(4)
3. Perform global read and attention based read, the outputs of which are fed into the policy and value heads:
Ft (M,)
Ct Context( st,
These three operations will be further detailed below in individual subsections. Projective mapping and spatially structured memory should augment the agent's performance where spatial reasoning and long term recollection are required. On simpler tasks the network can still perform well the baseline, assuming the extra parameters do not cause further instability in the RL training process.
Affine transform- At each time-step we wish translate and rotate the map into the agent's frame of reference, this is achieved with a differentiable affine transform, popularized by the well known Spatial Transformer Networks Jaderberg al. (201)). Relying on the simulator to be an oracle and provide the change in position (dz, dy) and orientation do, we convert the deltas to the agent's s egocentric frame of reference and transform the map with differentiable affine transform. The effect of noise on the change in position on the agent's performance is analysed in the experimental section.
Inverse projective mapping- We take the agent's current observation, extract relevant semantic embeddings and map them from 2D planar projection to their 3D positions in an egocentric frame of reference. At each time-step, the agent's egocentric observation is encoded by the perception module (a convolutional neural network) to produce feature vectors, this step is a mapping from R4 x4x 112 + 116x4x10 10 Given the inverse camera projection matrix and the depth buffer provided by the simulator, we can compute the approximate location of the features in the agent's egocentric frame of reference. As the mapping is a many to one operation, several features can be mapped to the same location. Features that share the same spatial location averaged element-wise.
5

---

Under review as a conference paper at ICLR 2020
The newly mapped features must then be combined with the translated map from the previous time-step. We found that the use of a momentum new and previously observed features. We use value of 0.9 for the tests presented in the paper. We ensured that the blending only occurs where the locations of new projected features and the map from the previous time-step are co-located, this criterion is detailed in Equation [
a, enabled a smooth blending of
M = =mMe (a.y)
+ (1 1,
1.0, if aM
0 & M, (#.w)
#
0.0, if M, (2.0) # #0& M((..)
Ox,
otherwise
Sampling from global map - A naive approach to the storage and transformation of the egocentric feature map would be to apply an affine transformation to the map at each time-step. fundamental downside of applying repeated affine transforms is that at each step bilinear interpolation operation is applicd, which causes smearing and degradation of the features in the map. We mitigated this issuc by storing the map in global reference frame and mapping the agent's 's observations to the global reference frame. For the read operation an offline affine transform is applied. For further details see the appendix
Read operations We wanted the agent to be able to summarize the whole spatial map and also selectively query for pertinent information. This achieved by incorporating two types of read operation into the agent's architecture, Global Read operation and a Self-attention Read.
The global read operation is CNN that takes as input the egocentric map and outputs 32- dimensional feature vector that summarizes the map's contents. The output of the global read concatenated with the visual CNN output.
To query for relevant features in the map, the agent's controller network can output a query vector yt. the network then compares this vector each location in the map with cosine similarity function in order to produce scores, which are the same width and height as the map. The scores are normalized with softmax operation to produce soft-attention in the lines Bahdanau et al. (2014) and used to compute a weighted average of the map, allowing the agent to selectively query and focus on parts of the map. This querying mechanism was used in both the Neural Map Parisotto & Salakhutdinov (2018) and MERLIN MERLIN[Wayne et al. (2018) RL agents. We made the following improvements: Attention Temperature and Query Position.
Sz.
a
Σ E0e
Query Position: A limitation of self-attention is that the agent can query what it has observed but not where had observed it. To improve the spatial reasoning performance the agent we augmented the neural memory with two fixed additional coordinate planes representing the x.y egocentric coordinate system normalized to (-1.0, 1.0), as introduced for segmentation in Liang et al. (2018). The agent still queries based the features in the map, but the returned context vector includes two extra scalar quantities which are the weighted averages of the x.y planes. The impacts of these additions are discussed and quantified in the ablation study, in Section 4
Attention Temperature: To provide the agent with the ability to learn to modify the attention dis- tribution, the query includes additional learnable temperature parameter, 3, which can adjust the softmax distribution detailed in Equation 7 This parameter can vary query by query and is constrained to be one greater by Oneplus function. The use of temperature in neural memory architectures was first introduced in Neural Turing Machines Graves et al. (0014).
4 EXPERIMENTS
The EgoMap and baseline architectures were evaluated on four challenging 3D scenarios, which require navigation and different levels of spatial memory, The scenarios are taken from Beeching et al. (2019) who extended the 3D ViZDoom environment Kempka et al. (2017) with various scenarios that are partially observable, require memory, spatial understanding, have long horizons and sparse rewards. Whilst more visually realistic simulators are available such as Gibson Xia et al. (2018), Matterport Anderson et al. (2018), Home Brodeur al. (018)) and Habitat Manolis Savva* et al.

---

Under review as a conference paper at ICLR 2020
Scenarão
4 item
item
Find and Return
Labyrinth
Ageut
Train
Test
Train
TGst
Train
Test
Trala
Test
Randon
-0.79
-0.206
-0.21
-0.21
-0.21
21.21
-0.115
-0,085
Raseline
2.341 0.006
2.266
0.055
2,855
0.164
2.545 - 0.226
0.661 + 0.003
0.633 - 0.027
0.73 + 0.2
0.694
0.009
Neural Map
2.339
0.038
2.223
0.040
2,750
+ 0.062
2.465 0.004
0.825
+ 0.070
0.723 = 0.026
0.769 + 0.042
0.706
0.018
EgoMap
2.398
0.014
2.291
0.021
3.214
0.007
2,801
0.048
0.893 4: 0.007
0.848
0.017
0.753 0.002
0,732
0.016
(Upper Bound)
Opeimum
2.5
2.5
3,5
3.5
1.0
1.0
1.0
1.0
Table 2: Results of the baseline and EgoMap architectures trained on four scenarios for 1.2 B environment steps. We show the mean and std. of the final agent performance, evaluated for three independent experiments on a held-out testing set of scenario configurations.
(2019), the tasks available are simple point-goal tasks which do not require long term memory and recollection. We target the following three tasks;
Labyrinth: The agent must find the exit in the fastest time possible, the reward is a sparse positive reward for finding the exit. This tests an agent's 's ability to explore in an efficient manner.
Ordered k-item: An agent must find l objects in fixed order. It tests three aspects of an agent: its ability to explore the environment efficiently, the ability to learn to collect items a predefined order and its ability to store as part of its hidden state where items were located so they can be retrieved in the correct order. We tested two versions of this scenario with 4-items or 6-items.
Find and return: The agent starts next to green totem, must explore the environment to find red totem and then return to the starting point. This is our implementation of "Minotaur" scenario from Parisottoo & Salakhutdinov (2018). The scenario tests an agent's 's ability to navigate and retain information over long time periods.
All the tasks require different levels of spatial memory and reasoning. For example, if an agent observes an item out of order can store the item's 's location in its spatial memory and navigate back to it later. We observe that scenarios that require more spatial reasoning, long term planning and recollection are where the agent achieves the greatest improvement in performance. In all scenarios there a small negative reward for each time-step to encourage the agent to complete the task quickly.
Experimental strategy and generalization to unseen environments - Many configurations of each scenario were created through procedural generation and partitioned into separated training and testing sets of size 256 and 64 respectively for each scenario type. Although the task in scenario is fixed, we vary the locations of the walls, item locations, and start and end points; thus we ensure a diverse range of possible scenario configurations. A limited hyper-parameter sweep was undertaken which were fixed for both the baseline,
with the baseline architecture to select the Neural Map and EgoMap agents. Three independent experiments were conducted per task to evaluate the algorithmic stability of the training process. To avoid information asymmetry, we provide the baseline agent with dr, dy, sin(d0),
cos))dd)))(()))))) (d0) concatenated with its visual features.
Training Details - The model parameters were optimized with an on-policy, policy gradient algorithm; batched Advantage Actor Critic (A2C) Mnih et al. al.) (2016), used the popular Py PyTorch Paszke et al. (0017) implementation of A2C Kostrikov (2018). We sampled trajectories from 16 parallel agents and updated every 128 steps in the environment with discounted returns hootstrapped from value estimates for non-terminal states. The gamma factor was 0.99, the entropy weight was 0.001, the RMSProp Tieleman & Hinton (012)) optimizer was used with learning rate of 7c-4. The EgoMap agent map size was 16x24x24 with a grid sampling chosen to cover the environment size with a 20% padding. The agent's 's policy was updated over 1.2B environment steps, with a frame skip of 4. Training took 36 hours for the baseline and 8 days for the EgoMap, on 4 Xeon E5-2640v3 CPUs, with 32GB of memory and one NVIDIA GK210 GPU.
Results - Results from the baseline and EgoMap policies evaluated on the 4 scenarios are shown in table 2 all tasks benefit from the inclusion of inverse projective mapping and spatial memory in the agent's 's network architecture, with the largest improvement on the Find and Return scenario. We postulate that the greater improvement in performance is due to two factors; firstly this scenario always requires spatial memory as the agent must return to its starting point and secondly the objects in this scenario are larger and occupy more space in the map. We also compared to the state of the art in spatially structured neural memory, Neural Map Parisotto & Salakhutdino (2018). Figure shows agent training curves for the recurrent baseline, Neural Map and EgoMap, on the Find and Return test set configurations.
7

---

Under review as a conference paper at ICLR 2020
Ablation
Train
Test
Baseline
0.668 = 0.028
0.662 = 0.036
glohal
0.787
0.029
04
No query
0.838 = 0.003
0.811 = 0.013
No query temperature
0.845 = 0.014
0.815 = 0.019
query position
EgoMap
0.847 = 0.011
0.814 = 0.017
Baseline
EgoMap (1.1 query)
0.851
0.014
0.828
0.011
Neural Map
EgaMap
-12
Envierment Flames
Figure 2: Left: Agent performance on unseen test configurations of the Find and Return scenario. Right: Ablation study on the Find and Return scenario conducted after 800M environment steps.
Ablation study - An ablation study was carried out on the improvements made by the EgoMap architecture. We were interested to see the influence of key options such as the global and attention- based reads, the similarity function used when querying the map, the learnable temperature parameter and the incorporation of location-baat based querying. The Cartesian product of these options is large and it was feasible test them all, therefore decided to selectively switch off key options understand which aspects contribute to the improvement in performance. The results of the ablation study are shown in Table 2 Both the global and self-attentio reads provide large improvements in performance over the baseline recurrent agent. The position-based query provides a small improve- ment. A comparison of the similarity metric of the attention mechanism highlights the LI Li-similarity achieved higher performance than cosine, A qualitative analysis of the attentonn mechanism is shown in the next section.
ANALYSIS
Noisy Actions One common criticism of agents trained in simulation is that the agent can query its environment for information that would not be readily available in the real world. In the case of EgoMap, the agent is trained with ground truth ego-motion measurements. Real-world robots have noisy estimates of ego-motion due to the tolerances of available hardware. We performed an analysis of the EgoMap agent trained in the presence of a noisy oracle, which adds noise the ego-motion measurement. Noise is drawn from normal distribution centred at one and is multiplied by the agent's ground-truth motion, the effect the noise is cumulative but unbiased. Tests were conducted with standard deviations of up to 0.2 which is tolerance of more than 20% the agent's ego-motion ego-motion measurements, results are shown in Figure We observed retain the performance increase over the baseline for up 10% of noisy actions, the performance degrades that of the baseline agent.
Noise std.
Average return
0.00
0.702 + 0.026
0.02
0.686 1 0.027
0.04
0.669 + 0.031
0.06
0.623 + 0.018
3.4
6.1
0.08
0.585 = 0.017
0.10
0.568 + 0.023
6.2
0.12
0.575 0.020
No std. 0.0.5
0.14
0.578 1 0.023
No std. 0.244
0.16
0.546 + 0.017
No us 18 010.66
No vld 010.00
Baselnn agent (so naley achinss
0.20
0.537 0.031
No 2t3 of0.66
EgoMtpp Agents with nolay actions
Baseline
0.527 E 0.047
66
400
35k
120M
188N 2009 2434 2004 3236
3500 6.05 0.00 1os edoo 1925
e350 cirs sa03
Figure 3: Test set performance of the EgoMap agent during training with noisy ego-motion measure- ments, conducted for 320 M environment frames. Shown test set performance during training (left), final performance for a range noise values (centre) which are tabulated (right). retain an improvement in performance over the baseline agent for noisy actions of up to 10%.
8

---

Under review as a conference paper at ICLR 2020
Stop
Projectiv mapping
Object features retained map
Projectvve
tuiere un starting point
olject Suatones
Ktog: 101
Stap:
Collection of priggers attvation to object
tnatiee
Stapp: 134
querying
object
Stop: 140
Stop: 165
object
Postion querying locates atjec in sparial memary
(a) Analysis: Ordered 6-item scenario
(b) Analysis: Find and Return scenario
Figure 4: Analysis of the Ego for key steps (different rows) during an episode from the Ordered 6-item and Find and Return scenarios. Within each sub-figure: Left column- RGB observations, central column the three largest PCA components of features mapped in the spatially structured memory, right attention heat map (result of the query) and x.y query position vector. We observe that the agent maps and stores features from key objects and attends to them when they are pertinent to the current stage of the task. For example, for the left figure on the first row at time-step the blue spherical object, which is ordered 4 of 6, is mapped into the agent's s spatial memory. The agent explores the environment collecting the items in order, it collects item 3 of 6 between time-step 105 and 108, shown rows and 3. As soon the agent has collected item it queries its internal memory for the presence of item 4, which shown by the attention distribution on rows 3 and 4. On the last row, time-step 140, the agent observes the item and longer attempts to query for it, the item is in the agent's field of view.
Visualization - The EgoMap architecture is highly interpretable and provides insights about how the agent reasons in 3D environments. In Figures 4a and 4b we show analysis of the spatially structured memory and how the agent has learned to query and self self-attend to recall pertinent information. The Figures show key steps during an episode the Ordered 6-item and Find and Return scenarios, including the first three principal components of dimensionality reduction of the 16-dimensional EgoMap, the attention distribution and the vector returned from position queries. Refer to the caption for further details. The agent is seen to attend to key objects at certain phases of the task, in the Ordered 6-item scenario the agent attends the next item in the sequence and in the Find and Return scenario the agent attends to the green totem located at the start/return point once it has found the intermediate goal.
CONCLUSION
We have presented EgoMap, an egocentric spatially structured neural memory that augments an RL agent's performance in 3D navigation, spatial reasoning and control tasks. EgoMap includes a differentiable inverse projective transform that maps learned task-specific semantic embeddings of agent observations their world positions. We have shown that through the use of global and self-attentive read mechanisms an agent can learn to focus on important features from the environment, We demonstrate that an RL agent can benefit from spatial memory, particularly in 3D scenarios with sparse rewards that require localization and memorization of objects. EgoMap out-performs existing state of the art baselines, including Neural Map, spatial memory architecture. The increase in performance compared to Neural Map is due to two aspects. 1) The differential projective transform maps what the objects are to where they are the map, which allows for direct localization with attention queries and global reads. In comparison, Neural Map writes what the agent observes to where the agent on the map. this means that the same object viewed from two different directions will be written to different locations the map, which leads poorer localization of the object. 2) Neural Map splits the map to four 90-degree angles, which alleviates the blurring highlighted in the appendix, our novel solution this issue stores single unified map in allocentric frame of

---

Under review as a conference paper at ICLR 2020
reference and performs an offline egocentric read, which allows an agent to act in states spaces where the angle is continuous, without the need to quantize the agent's 's angle to 90-degree increments.
We have shown, with detailed analysis, how the agent has learned to interact with its structured internal memory with self-attention. The ablation study has shown that the agent benefits from both the global and self-attention operations and that these be augmented with temperature, position querying and other similarity metrics. We have demonstrated that the EgoMap architecture robust actions with tolerances of up 10%. Future work in this area of research would be apply the mapping and memory architecture in more realistic-looking domains and aim to incorporate both dynamic and static objects into the agent's network architecture and update mechanisms.
REFERENCES
Peter Anderson, Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton den Hengel. Vision-ane -nd---n navigation: Interpreting visually- grounded navigation instructions in real environments. CVPR, 2018
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXhv:1409.0473, 2014.
A. Banino, C. Barry, B. Uria, C. Blundell, T. Lillicrap, P. Mirowski, A. Pritzel, M.J. Chadwick, T. Degris, J. Modayil, G. Wayne, H. Soyer, F. Viola, B. Zhang, R. Goroshin, N. Rabinowitz, R. Pascanu, C. Beattie, S. Petersen, A. Sadik, S. Gaffney, H. King, K. Kavukcuoglu, D. Hassabis,
R. Hadsell, and D. Kumaran. Vector-based navigation using grid-like representations in artificial agents. Nature, 557, 2018.
Edward Beeching, Christian Wolf, Jilles Dibangoye, and Olivier Simonin. Deep reinforcement learning budget: 3d control and reasoning without supercomputer. CoRR, Mtss/190.018000 2019. URL http::///a.xxv.ooo/aa/////// wwt./o/.//aaa.ooooooo.oooooaaa/a 11904. C1806
Shehroze Bhatti, Alban Desmaison, Ondrej Miksik, Nantas Nardelli, N. Siddharth, and Philip H. S. Torr. Playing doom with slam-augmented deep reinforcement learning. arxiv preprint 1672. 00380, 2016.
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. HoME: Household Multimodal Environment. In ICLR, 2018.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In ICLR, 03 2019.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated Feedback Recurrent Neural Networks. In ICML,, 2015.
Civera, Galvez-Lopez, L. Riazuelo, D. Tardós, and M. M. Montiel. Towards semantic SLAM using a monocular camera. In IROS, 2011.
C.J. Cueva and X.-X. Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In ICLR, 2018.
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Yves Frégnac, Alice René, Jean Baptiste Durand, and Yves Trotter. Brain encoding and representation of 3d-space using different senses, in different species. Journal of physiology, Paris, 98(1-3):1, 2004.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. CVPR. IEEE, 2018.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. urXiv:1410.5.4 2014.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626):471, 2016.
S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7272-7281, July 2017. doi: 10.1199/CVPR..211..66.
Torkel Hafting, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Edvard Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436:801-6, 09 2005. doi: 10. 1038/nature003211
Henriques and A. Vedaldi. Mapnet: An allocentric spatial memory for mapping environments. In CVPR, 2018.
Peter Henry, Michael Krainin, Evan Herbst, Xiaofeng Ren, and Dieter Fox. Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In Experimental robotics, pp. 477-491. Springer, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, 1997.
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 559-568. ACM, 2011.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In NIPS. 2015.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-13, 1998.
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. ViZ- Doom: A Doom-based AI research platform for visual reinforcement learning. JEEE Conference Computatonal Intelligence and Games, CIG, 2017. ISSN 23254289.
Ilya Kostrikov. inptemcntmml of reinforcement learning algorithms. hatps:://ithhu
com/ http://otiiioootttrrree a2c ppo--pppoaaee acktr 2018.
Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. AAAI, 2017.
Yann Lecun, Eon Bottou, Yoshua Bengio, and Patrick Haaner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
X. Liang, Y. Wei, X. Shen, J. Yang, L.. Lin, and S. Yan. Proposal-Free Network for Instance-Level Object Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2018.
Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, and Dhruv Batra. Habitat: platform for embodied ai research. arXiv, 2019.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to Navigate in Complex Environments. In ICLR, 2017.
Piotr Mirowski, Matthew Koichi Grimes, Mateusz Malinowski, Karl Moritz Hermann. Keith An- derson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learning to Navigate in Cities Without Map. In NeurIPS2018. 2018

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540). 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML., 2016.
H. Moravec. Sensor fusion in certainty grids for mobile robots. Al magazine, 9(2), 1988.
J. O' Keefe and Dostrovsky. The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Research, 34(1):171 175, 1971.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In ICLR, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito,, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPN-ISSII W, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), NIPS. 2015.
L. Rummelhard, A. Nègre, and C. Laugier. Conditional Monte Carlo Dense Occupancy Tracker. In International Conference on Intelligent Transportation Systems, 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 114001144,, 2018.
Keisuke Tateno, Federico Tombari, Iro Laina, and Nassir Navab. Cnn-slam: Real-time dense monocular slam with learned depth prediction. In CVPR, 2017.
Tieleman and G. Hinton. Lecture 6.5 RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL http: //papers poppers.nips.o wtw.app..cpppppee//p// 7181 attention is all you need. pdf
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. arxiv pre-print 1611,05763, 2016.
Greg Wayne, Chia-Chus Hung, David Amos, Mchdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack W. Rae, Piotr W. Mirowski, Joel Z. Leibo, Adam Santoro. Mevlana Gemici, Malcolm Reynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Jimenez Rezende, David Saxton, Adam Cain, Chloe Hillier, David Silver, Koray Kavukcuoglu, Matthew Botvinick, Demis Hassabis, and Timothy P. Lillicrap. Unsupervised predictive memory goal-directed agent. arxiv preprint 1803. 10760, 2018
Fei Xia, Amir R. Zamir, Zhi-Yas Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In CVPR, 2018
Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural SLAM. arxiv preprint $706.09520, 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
B AFFINE TRANSFORM
A naive implementation of the repeated affine transforms leads to smearing of features Figure S demonstrates the degradation of features on synthetic RGB images with repeated rotations and translations. We should how storing the features in allocentric frame of reference and performing offline transforms for read operations can greatly mitigate this issue.
Miyid
Ma
Mky
Wey 140
Viny
140
Figure 5: A synthetic 3-channel top down map. Rotations of two degrees per time-step with naive and global reference frame (rows & 2) shown over 180 time-steps. Translations with naive and global reference frame (rows 3 & 4) shown over 180 time-steps. Distinct degradation of the map features observed by time-step 100 and features are barely visible after 180 steps, leading to poor localization with self-attention queries. In tasks that span up 500 time-steps, the blurring of features could have greatly restricted the EgoMap agent's performance.
C ARCHITECTURES
To encourage reproducibility, we detail the exact architectures of the agents. Figure 6 shows an overview.
Baseline Model is comprised of the following: Perception Module fp: A 3 layer CNN with kernel sizes of 8,4.3, strides of 4,2,1, 1, no padding and filter sizes of 16,32,16, respectively and ReLU activation. fp is a mapping from RGBD input observation of R1x6ix112 Module f+: A FC layer reduces the output of fp from 640 values to vector of size 128 and includes ReLU activation; we then use a GRU layer with 128 hidden units. Policy and Value Heads fa & fo: These layers receive as input the output of fr. The policy layer is a FC layer with output units corresponding the discrete actions available the agent (through softmax activation). The value layer is a FC layer with one output unit.
R16x1x10. Recurrent
comprised of the following: Perception Module fp: The same as the baseline architecture, apart from that the mapping operation is applied before the final ReLU activation function. EgoMap Global Read Module: 3 layer CNN with kernel sizes of 3,4,4, strides of 1 1,2.2 and filter sizes of 16,16,16 respectively, and no padding. Followed by two linear layers of 256 and 32 hidden units, and ReLU activations, apart from the last which was tanh. Recurrent Module fr: The output of fp and the global read module were concatenated to form vector of size 672 and fed into a FC layer later with 128 output units. The recurrent module was a GRU with 128 units. Self-Attention Read Head: The query head is linear layer with 17 output units, 16 for the calculation of the EgoMap similarity scores and one for the 3 temperature parameter. The query head returns a vector of size 18 which includes two more scalar values for the average position of the query. Policy
EgoMap Model
13

---

Under review as a conference paper at ICLR 2020
and Value Heads fo & fo: Are the same the baseline but their input is the concatenation of the output of the fr and the attention head.
CNN
FC
GRU
v(A,)
h2
Depth dx, dy, d0
Egocentric Mapping
Context Read
M
M,
Figure 6: The baseline agent architecture (light blue) augmented with the EgoMap architecture (green), the agent's hidden state is h2 and the spatially structured neural memory M,. We rely on an oracle (the simulator) to provide the depth buffer D, and the agent deltas (dz, d0)
D READ MECHANISMS
In figure we provide further details of the operation of the global read, context read and xy-querying.
ADDITIONAL RESULTS
In figure we provide the curves of agent performance on held test configurations for three scenarios: 4-item, 6-item and labyrinth.
14

---

Under review as a conference paper at ICLR 2020
32-D Vector, &t
Souvee
2
Sparit
Fixed planes
Query vedes
Figure 7: Schematics detailing the operation of the read mechanisms, with global read (left), context reiddeid and xy-querying (right).
2.5
Uppar bound en optimal Padial
2.5
Auerage Return
Rewrage Return
3.0
1.0
Baseline
Baseline
Neural Map
Neural Map EgoMap
EgoMap
Environment Frames
Environment Frames
Ulper exopiimal th
Average
Baseline
0.0
Neural Map
EgoMap
68
2004
400M
GOOM
50OM
3OECM
1300M
Figure 8: Training curves of held out test set performance on 4-item (top left), 6-item(top-right) and labyrinth scenarios (bottom).
15


Reviews:
[{"review_id": "rke0Lprc5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This very well written and executed paper synthesizes several ideas recently published in the field of deep reinforcement learning-based goal-driven navigation. It elegantly combines these ideas together by presenting a neural agent architecture that consists of:\n* a perception module (e.g. a convnet) that extracts coarse visual feature maps s_t from an RGBD image\n* a differentiable map canvas M_t that is rotated at each step based on affine egocentric velocity (dx_t, dy_y, d \\phi_t)\n* differentiable inverse projection mapping, which uses known camera parameters, projective geometry and the depth channel of the image to project the visual feature vectors s_t onto a 2D map and add it to the existing canvas M_t\n* a recurrent module (GRU) for update a state h_t that is used for computing the policy distribution and value function\n* additional inputs to the policy and value function, that include a global map read r_t, as well as a query q_t (produced by the policy head) based retrieval of features from the map\n* position indexing of features retrieved from the map\n\nThe algorithm is trained end-to-end, without extra supervision, using Advantage Actor-Critic (A2C) RL. Based on the strong inductive biases regarding the map, namely affine transforms of the map given information about relative movement, and projective geometry transformations of visual features in the map frame, it seems that the question of where to write is solved, and that the network only needs to learn what to write in the differentiable map. Evaluation is done on 3 games in VizDoom: finding the exit of the Labyrinth, object retrieval and find and return / Minotaur. \n\nCriticism:\n\nThe authors could justify better the choice of using the projective geometry inductive prior. They use sentences like \"We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations\" (not quite related to grid and place cells, despite being in that section) and \"this inverse mapping operation is second nature to many organisms\" without giving any reference.\n\nSeveral papers have been published in the last two years, focusing on differential memory architectures with a 2D map structure, projective geometry. This paper goes further by building and iteratively updating a 2D occupancy map using visual features and image geometry, just like RGBD-SLAM (which would merit a citation, e.g., [1] and [2]). This paper essentially combines existing ideas (see table 1): projective geometry, reward-based learning of M_t, RL, multitask navigation, semantic features. While this is not novel, seeing all this combined in a single technique does have merit.\n\nWhat is disappointing, given that this is a combination paper, is that the environment is so simply, and that photorealistic environments were not tested. For example, the VizDoom environment uses 2D sprites for objects, making the visual feature extraction from objects much simpler. Would the method work equally well with the objects in DeepMind Lab, which are seen from multiple view points? And what in an environment like AdobeIndoorNav?\n\n[1] Henry et al (2010) \"RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments\"\n[2] Izadi et al (2011) \"KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera\"", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1xgNJJAFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel architecture for spatially structured memory. The main idea is to incorporate inductive bias/invariance derived from projective geometry arguments. The experiments seem to clearly show that this new architecture improves previous approaches to tasks which require spatial reasoning and memory, and the ablations studies and visualizations provide useful insights into the workings of the agent. One thing I'm missing is an experiment showing that this inductive bias also doesn't degrade performance on tasks where spatial reasoning is not necessary (as compared to vanilla GRU/LSTM).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxJAE3CKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to build semantic spatial maps for the purpose of navigation in 3D environments. The paper presents a differentiable policy network that pastes together semantic map predictions into a spatial map. Information is read out from this map using a global read operation (that looks at the entire map) and a self-attention read operation. This information is used to produce actions. The paper presents experimental results in 3D VizDoom scenarios and reports improvements over a vanilla LSTM, and another spatial memory based method (Neural Map).\n\nStrengths:\n1. I very much like the proposed formulation for tackling navigation problems. Using learning to leverage semantic reasoning, and structuring the computation spatially makes a lot of sense.\n2. In my view, the proposed formulation advances current models in the following ways:\na. Maintaining and updating allocentric maps, and reading off egocentric maps. This alleviates need for repeated rotations of the map, and thus prevents aliasing.\n3. The paper provides ablations for the various parts of the system and provides qualitative analysis of the learned spatial representations.\n4. Very good placement of work in current literature. I really like Table 1.\n\nShortcomings:\n1. The central contribution of the paper is the design of the egocentric spatial memory, how to build and maintain it over time, and its use in deep RL. The paper does this by using components from previous papers and presents a very nice summary of this in Table 1. Unfortunately, modulo the component described above (that of maintaining allocentric maps and reading off egocentric maps as and when needed), all other components are borrowed from existing papers, as can be seen in Table 1 already. The paper lists its contributions in Introduction on page 2, and each of those contributions has been studied in previous papers (though I will that admit no single paper does all these things together). Thus, I believe the paper falls short in terms of technical contributions.\n\n2. Following on from point above, putting everything together and showing that it works, could also be a reasonable contribution, though it would warrant more extensive and systematic experiments for the different design choices, possibly in more realistic environments. For example, a) is the projective projection important, or could that have been learned, b) do repeated rotations indeed lead to blurred representations, c) what is critical to get such models work with RL, that past models that used imitation learning couldn't, d) other claimed differences from past works in this space.\n\n3. Experiments and analysis:\na. The paper compares against NeuralMap, and reports improvements, but doesn't give a reason as to why this happens.\nb. Past works have demonstrated these ideas in visually realistic environments (similar to those in Gibson / Habitat, see semantic tasks in CogMap). Current paper only investigates proposed ideas in VizDoom environments.\n\nThus, while I like the direction of research and the fact that the paper presents an architectures that uses latest techniques in the area, I believe the paper doesn't have enough technical contribution of its own, and experiments are limited to synthetic VizDoom environments.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a spatially structured neural memory architecture that supports navigation tasks.  The paper describes a complex neural architecture that integrates visual information, camera parameters, egocentric velocities, and a differentiable 2D map canvas.  This structure is trained end-to-end with A2C in the VizDoom environment.  The strong inductive priors captured by these geometric transformations is demonstrated to be effective on navigation-related tasks in the experiments in this environment.

The reviewers found many strengths and a few weaknesses in this paper.  One strength is that the paper pulls together many related ideas in the mapping literature and combines them in one integrated system.  The reviewers liked the method's ability to leverage semantic reasoning and spatial computation.  They liked the careful updating of the maps and the use of projective geometry.  

The reviewers were less convinced of the generality of this method.  The lack of realism in these simulated environments left the reviewers unconvinced that the benefits observed from using projective geometry in this setting will continue to hold in more realistic environments.   The use of fixed geometric transformations with RGBD inputs instead of learned transformations also makes this approach less general than a system that could handle RGB inputs.  Finally, the reviewers noted that the contributions of this paper are not well aligned with the paper's claims.

This paper is not yet ready for publication as the paper's claims and experiments were not sufficiently convincing to the reviewers. 
Decision: Reject