End-to-End Egospheric Spatial Memory
Authors: Daniel James Lenton
Stephen James
Ronald Clark
Andrew Davison
Venue: ICLR 2021 Poster (2021)
Keywords: egocentric
differentiable memory
spatial awareness
mapping
image-to-action learning

Abstract: Spatial memory, or the ability to remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments. However, most existing artificial memory modules are not very adept at storing spatial information. We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations. ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance. We further show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities. Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures. Implementation at: https://github.com/ivy-dl/memory.

Published as a conference paper at ICLR 2021
END-TO-END EGOSPHERIC SPATIAL MEMORY
Daniel Lenton ', Stephen James' Ronald Clark 2, Audrew J. Davison Dyson Robotics Lab, Department of Computing, Imperial College London {dj1ll, s1 j12, ronald, clark, a. davison @ic.ac.......
ABSTRACT
Spatial memory, the ability remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments. However, most existing artificial memory modules are not very adept storing spatial information. We propose parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations. ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance. We further show applications to semantic segmentation the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities. Through our broad set of experiments, we show that ESM provides general computation graph for embodied spatial reasoning, and the mod- ule forms bridge between real-time mapping systems and differentiable memory architectures. Implementation at: https: //tp:/gggtttttttt com/ Livy-ddd/eenee
INTRODUCTION
Egocentric spatial memory is central to our understanding of spatial reasoning in biology (Klatzky 1998, Burgess 2006), where an embodied agent constantly carries with it local map its sur- rounding geometry. Such representations have particular significance for action selection and motor control (Hinman al. 2019). For robotics and embodied AI, the benefits of a persistent local spatial memory are also clear. Such system has the potential to run for long periods, and bypass both the memory and runtime complexities of large scale world-centric mapping. |Pters et al. (200T) propose an EgoSphere as being a particularly suitable representation for roboties, and more recent works have utilized ego-centric formulations for planar robot mapping (Fankhauser et al. 2014), drone obstacle avoidance (Fragoso al. 2018) and mono-to-depth (Liu al. 2019).
In parallel with these ego-centric mapping systems, à new paradigm of differentiable memory architectures has arisen, where memory bank is augmented learn read and write operations (Weston et al.) 2014 Graves et al. 2014 Sukhbaatar et al. 2015). When compared to Recurrent Neural Networks (RNNs), the persistent memory circumvents issues of vanishing or exploding gradients, enabling solutions to long-horizon tasks. These have also been applied to visuomotor control and navigation tasks (Wayne et al., 2018), surpassing baselines such as the ubiquitous Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber 1997).
neural network, which then
We focus on the intersection of these two branches of research, and propose Egospheric Spatial Memory (ESM), parameter-free module which encodes geometric and semantic information about the scene in an ego-sphere around the agent. To the best of our knowledge, FSSM is the first end-to-end trainable egocentric memory with a full panoramic representation, enabling direct encoding of the surrounding scene in a 2.5D image.
We also show that by propagating gradients through the ESM computation graph we can learn features to be stored in the memory. We demonstrate the superiority of learning features through the ESM module on both target shape reaching and object segmentation tasks. For other visuomotor control tasks, we show that even without learning features through the module, and instead directly projecting image color values into memory, ESM consistently outperforms other memory baselines.

---

Published as a conference paper at ICLR 2021
Through these experiments, show that the applications of our parameter-free free ESM module widespread, where can either be dropped into existing pipelines as a non-leamndd module, or end-to-end trained in a larger computation graph, depending the task requirements.
RELATED WORK
2.1 MAPPING
Geometric mapping is mature field, with many solutions available for constructing high quality maps. Such systems typically maintain an allocentric map, either by projecting points into a global world co-ordinate system (Newcombe et al. 2011 Whelan et al. 2015), or by maintaining a certain number of keyframes in the trajectory history (Zhou et al. 2018: Bloesch et al.) (2018). If these systems are to be applied to life-long embodied AI, then strategies are required to effectively select the parts of the map which useful, and discard the rest from memory (Cadena et al. (016)).
For robotics applications, prioritizing geometry in the immediate vicinity is a sensible prior. Rather than taking world-vier to map construction, such systems often formulate the mapping problem in a purely ego-centric manner, performing continual re- re-projection to the newest frame and pose with fixed-sized storage. Unlike allocentric formulations, the memory indexing is then fully coupled to the agent pose, resulting in an ordered representation particularly well suited for downstream egocentric tasks, such as action selection. |Peters et al. (20T)) outline an EgoSphere memory structure as being suitable for humanoid robotics, with indexing via polar and azimuthal angles, [aanhhauser al. (2014) use ego-centric height maps, and demonstrate on quadrupedal robot walking over obstacles. Cigla et al. (2017) use per-pixel depth Gaussian Mixture Models (GMMs) to maintain an ego-cylinder of belief around a drone, with applications to collision avoidance (Fragoso et al. 2018). In a different application, [iu et al. (2019) learn to predict depth images from sequence of RGB images, again using ego reprojections. These systems are all designed to represent only at the level of depth and RGB features. For mapping more expressive implicit features via end-to-end training, a fully differentiable long-horizon computation graph is required. Any computation graph which satisfies this requirement is generally referred to memory in the neural network literature.
2.2 MEMORY
The concept of memory in neural networks is deeply coupled with recurrence. Naive recurrent net- works have vanishing and exploding gradient problems (Hochreiter) 1998), which LSTMs (Hochreiter) Schmidhuber 1997) and Gated Recurrent Units (GRUs) (Cho et al. [0144) mediate using additive gated structures. More recently, dedicated differentiable memory blocks have become popular alter- native. WWeston et al. (2014) applied Memory Networks (MemNN) to question answering, using hard read-writes and separate training of components. Graves et al. (2014) and Sukhbaatar et al. (2015) instead made the read and writes 'soft' with the proposal of Turinl Machines (NTM) and End-to-En Memory Networks (MemN2N) respectively, enabling joint training with the controller. Other works have since conditioned dynamic memory on images, for tasks such as visual question answering (Xiong et al.; (0166) and object segmentation (Oh et al. (019). Another distinct but closely related approach is self attention (Vaswani et al. (017). These approaches also use key-based content retrieval, but do so on history of previous observations with adjacent connectivity. Despite the lack of geometric inductive bias, recent results demonstrate the amenability of general memory (Wayne) al. 2018 and attention (Parisotto al. 2019 to visuomotor control and navigation tasks.
Other authors have explored the intersection of network memory and spatial mapping for navigation, but have generally been limited to 2D aerial-view maps, focusing on planar navigation tasks, Gupta et (201)) used an implicit ego-centric memory which was updated with warping and confidence maps for discrete action navigation problems. Parisotto & Salakhutdinov (2017) proposed a similar setup, but used dedicated learned read and write operations for updates, and tested on simulated Doom environments. Without consideration for action selection, |enriques & Vedaldi (2018) proposed a similar system, but instead used an allocentric formulation, and tested free-form trajectories of real images. Zhang et al. (2018) also propose a similar system, but with the inclusion of loop closure. Our memory instead focuses local perception, with the ability to represent detailed 3D geometry in all directions around the agent. The benefits of our module are complementary to existing 2D methods, which instead focus on occlusion-aware planar understanding suitable for navigation.
2

---

Published as a conference paper at ICLR 2021
3 METHOD
In this section, we describe our main contribution, the egospheric spatial memory (ESM) module, shown in Figure The module operates as an Extended Kalman Filter (EKF), with an egosphere image f4 E RR, x ((2+++)) and its diagonal covariance Eg ((h.x.,((+nn) representing the state. The egosphere image consists of 2 channels for the polar and azimuthal angles, 1 for radial depth, and n for encoded features. The angles are not included in the covariance, as their values are implicit in the egosphere image pixel indices. The covariance only represents the uncertainty in depth and features these fixed equidistant indices, and diagonal covariance is assumed due the large state size of the images. Image measurements are assumed to come from projective depth cameras, which similarly store I channel for depth and n for encoded features. We also assume incremental agent pose measurements W, I with covariance Σx, E _6x6 are available, in the form of translation and rotation vector. The algorithm overview is presented Algorithm 1.
14-1
Et-
200
VeO 04O
Vi 135m
11
Vies
quantize
Fuse
Vem Utn
jtt Σt
Σ
H4-1
h, x ws x + 2n)
E
3
h,
x
(4 + 2n)
ESM
ESM
Overview of the ESM module. The module consists of projection and quantization steps, used bring the belief from the previous agent frame to the current agent frame.
Figure 1:
First, the motion step takes the state from the previous frame 14 E,-1 and transforms this into a predicted state for the current frame jı, E, via functions fm, Fm
Algorithm 1: ESM Step
1 Given: fmn Fm for Fo
and the incremental pose measurement i, with covariance Em.. Then in the observation step. we use measured vi-
2 j4 Sm (u4,)4-1)
Σn Fm(us, Pt- Ex-1, En,)
sual features (wt E TeA (1+ (1+a)) (iE(.....m) with diag-
a fr fo((Uri,Pei)
onal covariances (Vi Rh (1+1)) )E((1.... orig-
EN Fo((uti, Pti: Vea, Pu)ier)
inated from m arbitrary vision sensors, and associated pose measurements (Pa E (L.,......) with covariances (Phs E IR ( to produce new observation of the state tu E Xph.xw,x(2+110 ance Eg E x(1+x) via functions fo and Fo. The measured poses also take the form of translation and rotation vector.
Ke = E, E Σ
7 = pr + Kel Pel
Ee [[ - Ki] Eg
again with diagonal covari-
return He Ee
Finally, the update step takes our state prediction Fe, Σ, and state observation Ar, Er, and fuses them to produce our new state belief μt, E. We spend the remainder of this section explaining the form of the constituent functions. All functions in Algorithm involve re-projections across different image frames, using forward warping. Functions fm. Fm. fo and F, are therefore all built using the same core functions. While the re-projections could solved using a typical rendering pipeline mesh construction followed by rasterization, we instead choose simpler approach and directly quantize the pixel projections with variance- image smoothing to fill in quantization holes. An overview of the projection and quantization operations for single ESM update step is shown in Fig. [
3.1 FORWARD WARPING
Forward warping projects ordered equidistant homogeneous pixel co-ordinates p6y1 from frame fi to non-ordered non-equidistant homogeneous pixel co-ordinates pc,z in frame f2. We use uyz (ō12, 02 dr2. ē to denote the loss of ordering following projection from μy1 = {oy1. By2, dr1, ),2), where d. 6, d and represent polar angles, azimuthal angles, depth and encoded features respectively. We only consider warping from projective omni cameras, which corresponds
3

---

Published as a conference paper at ICLR 2021
to functions for Fo. but the omni-to- ommi case as in fmy Fm is identical except with the inclusion of another polar co-ordinate transformation.
The encoded features transform the values to the new frame in polar co-ordinates, which is a composition of a linear trans- formation and non-linear polar conversion. Using the camera intrinsic matrix K1, the full projection is composed of a scalar multiplication with homogeneous pixel co-ordinates pCyı- transformation by camera inverse matrix Kj and frame-to- trame T12 matrices, and polar conversion fp:
assumed constant during projection ēy2 eʃ1. For depth, we must
(012.0 12. dr2) = =f (Ti2KK [pey1 @ dr1l)
Combined, this provides with both the forward warped image ju (ō)2, , fy,, dy2, εy2). and the newly projected homogeneous pixel co-ordinates pc (kippr 0 kiprr B 11, where kppr denotes the pixels-pe mixel--pe--radiin per-radian resolution constant. The variances are also projected using the full analytic Jacobians, which are efficiently implemented as tensor operations, avoiding costly autograd usage.
Σ = JvVi + JpP12JP
3.2 QUANTIZATION, FUSION AND SMOOTHIING
Following projection, first quantize the floating point pixel coordinates pc,z into integer pixel co-ordinates pCf2- This in general leads to quantization holes and duplicates. The duplicates are handled with variance conditioned depth buffer, such that the closest projected depth is used, provided that it's variance is lower than a set threshold. This in general prevents highly uncertain close depth values from overwriting highly certain far values. We then perform per pixel fusion based on lines 6 and 7 in Algorithm I provided the depths fall within a set relative threshold, otherwise the minimum depth with sufficiently low variance is taken. This again acts as a depth buffer.
Finally, we perform variance based image smoothing, whereby we treat each N x N image patch (pR,))EE (1) (E...N),E
as collection of independent measurements of the central pixel, and combine their variance values based on central limit theory, resulting in smoothed values for each pixel in the image Mi.jj. Although we use this to update the mean belief, we do not smooth the variance values, meaning projection holes remain at prior variance. This prevents the smoothing from distorting our belicf during subsequent projections, and makes the smoothing inherently local to the current frame only. The smoothing formula is follows, with variance here denoted as o²:
Σx K
[4. = E.E
(3)
Given that the quantization is discrete operation, we cannot compute it's analytic jacobian for uncertainty propagation. We therefore approximate the added quantization uncertainty using the numerical pixel gradients of the newly smoothed image Gij. and assume additive noise proportional to the and y quantization distances Apes,j:
Eng = Eng + Gi.
(4)
3.3 NEURAL NETWORK INTEGRATION
The ESM module can be integrated anywhere into a wider CNN stack, forming an Egospheric Spatial Memory Network (ESMN). Throughout this paper we consider two variants, ESMN and ESMN-RGB see Figure 2 ESMN-RGE RGB into memory, while ESMN projects CNN encoded features into memory. The inclusion of polar angles, azimuthal angles and depth means the full relative polar coordinates are explicitly represented for each pixel in memory. Although the formulation described in Algorithm [ and Fig allows for m vision sensors, the experiments in this paper all involve only a single acquiring sensor, meaning m = We also only consider cases with constant variance in the acquired images V22 kvars and so we omit the variance images from the ESM input in Fig 2 for simplicity. For baseline approaches, we compute image of camera-relative coordinates via K image along with the tiled incremental poses before input to the networks. All values are normalized to before passing convolutions, based on the permitted range for each channel.
special case of ESMN, where RGB features are directly projected
and then concatenate this to the RGB
4

---

Published as a conference paper at ICLR 2021
EXCOxZe
In/Out
RGB img
CNN
output
img
CNN
momory module
output
Layers
Sam
cam coord
Inc. cam pose
Inc. cam pose INC
Inc. poss vectors
Memory
Inc.
32x32x18
Mono
32x32x18
LSTM/NTM
Inc. pose
ESMN-RGB
inc. pose vectors
ESMN
vectors
32x33
3
CNN
output
CNN
CNN
output
RGB img
Img
depth img
depth
32x32x4
90x18 1101180((4440 +(3)))
32x32x4
90x180x
Figure 2: High level schematics of the ESM-integrated network architectures ESMN-RGB and ESMN, as well as other baseline architectures used in the experiments: Mono, LSTM and NTM.
EXPERIMENTS
The goal of our experiments is show the wide applicability of ESM to different embodied 3D learning tasks. We test two different applications:
1. Image-to-action learning for multi-DOF control (Sec 4.1
Here consider drone and robot manipulator target reacher tasks using either ego-centric or scene-centric cameras. We then assess the ability for ESMN policies to generalize between these different camera modalities, and assess the utility of the ESM geometry for obstacle avoidance. We train policies both using imitation learning (IL) and reinforcement learning (RL).
2.
Object segmentation (Sec 4.2h. Here explore the task of constructing semantic map, and the effect of changing the ESM module location in the computation graph on performance.
4.1 MULTI-DOF VISUOMOTOR CONTROL.
While ego-centric cameras are typically used when learning to navigate planar scenes from images
JJaeeberg et al. 2016 Zhu et al.
2017 Gupta et al.
2017 Parisotto & Salakhutdinov 2017), ) static
scene-centric cameras the de facto when learning multi-DOF controllers for robot manipulators James et al. 2019b). We consider the more challenging and less explored setup of learning multi-DOl visuomotor controllers from ego-centric cameras, and also from moving scene-centric cameras. LSTMs are the de facto memory architecture in the RL literature (Jaderberg et al., 2016 Espeholt al.) 2018 Kapturowski al.) [018] MMrowwki et al. 2018; BBuce et al. 2018), making this a suitable baseline. NT'Ms represent another suitable baseline, which have outperformed L.STMs on visual navigation tasks (Wayne al. 2018). Many other works exist which outperform LSTMs for planar navigation in 2D maze- like environments (Gupta et al. 2017 Parisotto & Salakhutdinoy 2017; Henriques Vedaldi 2018), but the top-down representation means these methods not readily applicable our multi-DOF control tasks. LSTM and NTM are therefore selected as competitive baselines for comparison.
(Levine et al. 2016 James et al. 2017 Matas et al. 2018
4.1.1 IMITATION LEARNING
For imitation learning experiments, test the utility of the ESM module on simulated visual reacher tasks, which we refer to as Drone Reacher (DR) and Manipulator Reacher (MR). Both are implemented using the CoppeliaSim robot simulator (Rohmer al. 2013), and its Python extension PyRep (James et al. 2019a). We implement DR ourselves, while MR is modification of the reacher task RLBench (James al. 2020). Both tasks consist of targets placed randomly in simulated arena, and colors are newly randomized for each episode. The targets consist of a cylinder, sphere, and "star", see Figure
5

---

Published as a conference paper at ICLR 2021
In both tasks, the target locations remain fixed for the duration of an episode, and the agent must continually navigate to newly specified tar- gets, reaching as many as possible in a fixed time frame of 100 steps. The targets are spec- ified to the agent either as RGB color values or shape class id, depending on the experiment. The agent does not know in advance which tar- get will next be specified, meaning a memory of all targets and their location in the scene must be maintained for the full duration of an episode. Both environments have single body- fixed camera, as shown in Figure 3 and also an external camera with freeform motion, which we use separately for different experiments.
+
Figure 3: Visualization of (a) Drone Reacher and (b) Manipulator Reacher tasks.
For training, we generate offline dataset of 100k 16-step sequences from random motions for both environments, and train the agents using imitation learning from known expert actions. Action spaces of joint velocities q RT and cartesian velocities à E R6 are used for MR and DR respectively. Expert translations move the end-effector or drone directly towards the target, and expert rotations rotate the egocentric camera towards the target via shortest rotation. Expert joint velocities are calculated for linear end-effector motion via the manipulator Jacobian. For all experiments, we compare baselines of single-frame, dual-stacked LSTM with and without spatial auxiliary losses, and NTM. We also compare against a network trained on partial oracle omni-directional images, masked at unobserved pixels, which we refer to as Partial-Oracle- Omni (PO2), as well as random and expert policies. PO2 cannot see regions where the monocular camera has not looked, but it maintains a pixel-perfect memory of anywhere it has looked. Full details of the training setups are provided in Appendix A.I The results for all experiments presented in Table I
Drone Reacher
Manipulator Reacher
Ego Acy
Freeform Aeq
Ego Acg
Freeform Acy
Color
Shape
Color
Shape
Color
Shape
Color
Shape
Mono
0.6(0.7)
0.9(1.7)
2.4(5.0)
0,5(1.6)
1.8(1.5)
1.6(11))
0.1(0.2)
0.1(0.2)
LSTM
12.7(3.4)
4.1(2.3)
1.0(10)
0.6(0.8)
1.00.5)
0.1(0.2)
0.1(0.2)
0.1(0.4)
LSTM Aux
1.3(0.8)
0.4(0.8)
2.4(2.2)
1.9(1.7)
1.00..7)
0.1(0.3)
0.3(0.6)
0.0(0.2)
NTM
10.5(4.2)
2.5(1.9)
3,2(2.9)
1.6(1.5)
L.0(0.6)
0.2(0.4)
0.1(0.3)
0.1(0.2)
ESMN RGB
20.6(7.3)
4.1(3.8)
16.1(12.7)
1.1(2.6)
11.4(5.1) 1)
3.1(322
10.5(5.7)
0.9(1.)
ESMN
20.8(7.8)
18.3(6.4)
16.6(12.9)
8.5(11.2)
11.7(5.3)
4.7(4.0)
11.0(5.8)
1.0(1.))
Random
0.1(0.2)
0.1(0.2)
0.1(0.2)
0.1(0.2)
0.1(0.2)
0.1(0.2)
0.1(0.2)
0.1(0.2)
PO2
21.0(8.6)
14.4(6.1)
19. 1(127)
3.9(8.1)
13.0(5.9)
4. (3355))
12.5(6.1)
2.6(2.5)
Expert
21.3(8.4)
21.3(8.4)
21.3(8.4)
21.3(8.4)
16.5(5.2)
16.5(5.2)
16.5(5.2)
16.5(5.2)
Table 1: Final policy performances on the various drone reacher (DR) and manipulator reacher (MR) tasks, from egocentric acquired (Ego Aeq) or freeform acquired (Freeform Aeq) cameras, with the network conditioned on either target color or shape. The values indicate the mean number of targets reached in the 100 time-step episode, and the standard deviation, when averaged over 256 runs. ESMN-RGB stores RGB features in memory, while ESMN stores learnt features.
Ego-Centric Observations: In this configuration we take observations from body-mounted cameras. We can see in Table that for both DR and MR, our module significantly outperforms other memory baselines, which do not explicitly incorporate geometric inductive bias. Clearly, the baselines have difficulty in optimally interpreting the stream of incremental pose measurements and depth. In contrast, ESM by design stores the encoded features memory with meaningful indexing. The ESM structure ensures that the encoded features for each pixel are aligned with the associated relative polar translation, represented as an additional feature in memory. When fed to the PSst-ESM convolutions, action selection can then in principle be simplified to target feature matching, reading the associated relative translations, and then transforming to the required action space. A collection of short sequences of the features in memory for the various tasks are presented in Figure 4] with (a), (b) and (d) coming from egocentric observations. In all three cases we see the agent reach one target by the third frame, before re-orienting reach the next.

---

Published as a conference paper at ICLR 2021
We also observe that ESMN-RGE performs well when the network is conditioned target color, but fails when conditioned on target shape id. This is to be expected, as the ability to discern shape from the memory is strongly influenced by the ESM resolution, quantization holes, and angular distortion. For example, the "star" shape in Figure 4 (a) not apparent until 15- However, ESMN is able to succeed, and starts motion towards this star iz. The pre-ESM convolutional encoder enables ESMN to store useful encoded features in the ESM module from monocular images, within which the shape was discernible. Figure (a) shows the most dominant ESM feature channels projected RGB.
t2
b)
to
t2
14
15
<)
t2
d)
10
t3
t5
e)
t0
t5
Figure 4: Sample trajectories through the memory for (a) ESMN DR-Ego-Shape, and ESMN-RGB on (b) OR-Ego-Color, (c) DR- CC-Freeform-Clooo (d) MR-Ego-Color (c) MRRFreeform--CC Coloor The images each correspond features in the full 90 x 180 memory at that particular timestep
Scene-Centric Observations: Here explore the ability of ESM generalize to unseen camera poses and motion, from cameras external to the agent. The poses of these cameras are randomized for each episode during training, and follow random freeform rotations. with bias face towards the centre of the scene, and linear translations. Again, we see that the baselines fail to learn successful policies, while ESM- -agganemetedd networks are able solve the task, see Table The memories in these tasks take on a different profile, as can be seen in Fig 4 egocentric observations always contain information recent monocular frame projects with high density, this is not the case for projections from arbitrarily positioned cameras which can move far from the agent, resulting in sparse projections into memory. The targets in Fig (e) are all represented by only I or 2 pixels. The large apparent area in memory is a result of the variance-bar based smoothing, where the low-variance colored target pixels are surrounded by high-variance unobserved pixels in the ego-sphere.
(c) and (e). While the memories from the memory image centre, where the most
Obstacle Avoidance: To further demonstrate the benefits of a local spa- tial geometric memory, augment the standard drone reacher task with obstacles, see Figure Rather than learning the avoidance in the pol- icy, exploit the interpretable geometric structure in ESM, and instead augment the policy output with a local avoidance component. We then compare targets reached and collisions for different avoidance baselines, and test these avoidance strategies on random, ESMN- RGB and expert target reacher policies. We see that the ESM geometry enables superior avoidance over using the most recent depth frame alone. The obstacle avoidance results are presented in Table 2, and further details of the experiment presented in Appendix A.2
Figure 5: Avoidance task
We now explore the extent to which policies trained from egocentric observations can generalize to cameras moving freely in the scene, and vice-versa. The results of these transfer learning experiments are presented in Table Rows not labelled "Transferred" are taken directly from Table [ and repeated for clarity. Example image trajectories for both egocentric and free- free-form observations are presented Figure [ The trained networks were not modified in any way, with further training fine-tuning applied before evaluation the new image modality.
Camera Generalization:
7

---

Published as a conference paper at ICLR 2021
Policy
Random
ESMN-RGB
Expert
Reached
Collisions
Reached
Collisions
Reached
Collisions
Avoidance
No Avoidance
0.0(0.2)
29.4(27.9)
9.9(7.0)
64.8(67.4)
21.3(2.1)
121.7(15.8)
Single Depth Frame
0.1(0.2)
9.6(118)
11.1(4.3)
10.8(13.0)
15.3(2.0)
7.7(6.9)
ESM Depth Map
0.1(0.3)
4.2(7.1)
9.6(4.5)
2.7(5.5)
10.8(4.4)
2.2(3.8)
Ground Truth
0.1(0.3)
0.1(0.5)
9.6(5.1)
0.0(0.0)
15.2(3.9)
0.0(0.0)
Table Targets reached and number of collision for the obstacle avoidance drone task. Full details
of this experiment are provided in Appendix A.2
Irej
cam traj
1
Figure 6: Example image sequences from both egocentric (E) and freeform (F) cameras, on both
reacher tasks. The images are all time-aligned, and correspond to the same agent motion in the scene.
Drone Reacher
Manipulator Reacher
Ego Acq
Freeform Acq
Ego Acq
Freeform Acq
Color
Shape
Color
Shape
Color
Shape
Color
Shape
LSTT
12.7(3.4)
4.1(2.3)
1.0(100)
0.6(0.8)
1.00.5)
0.1(0.2)
0.T(0.2)
0.1(0.4)
Transferred
0.3(0.6)
0.5(0.7)
0.1(0.3)
0.3(0.7)
0.1(0.3)
0.0(0.2)
0.0(0.0)
0.0(0.2)
ESMN-RGB
20.6(7.3)
4.1(3.8)
16.1(12.7)
1.1(2.6)
11.4(5.1)
3.1(3.2)
10.5(5.7)
0.9(1.6)
Transferred
20.3(11.3)
5.8(4.7)
15.3(10.9)
0.8(1.6)
11.2(5.6)
3.9(4.3)
9.8(388
1.1(1.3)
ESMN
20.8(7.8)
18.3(6.4)
16.6(12.9)
8,5(11.2)
11.7(5.3)
4.7(4.0)
11.0(5.8)
1.0(1.)
Transferred
10.4(5.5)
0.7(0.9)
8.9(6.5)
1.0(1.4)
7.5(4.9)
6.3(6.2)
6.8(4.7)
0.6(1.0)
Table 3: Reacher performances both with and without camera transfer, using the same notation and Successful transfers are highlighted in bold.
setup as deseribed in T
4.1.2 REINFORCEMENT LEARNING
Assuming expert actions in partially observable (PO) environments is inherently limited. is not necessarily true that the best action always rotates the camera directly the next target for example. In general, for finding optimal policies in PO environments, methods such as reinforcement learning (RL) must be used. We therefore train both ESM networks and all the baselines on simpler variant of the HR-EEo-Coloo task via DQN (Mnih et al.) 2015). The manipulator must reach red, blue and then yellow spherical targets from egocentric observations, after which the episode terminates. We refer to this variant as MR Ee--Se-FFgo--oon
due to the sequential nature. The only other difference to MR that MR-Seq uses 128 x 128 images as opposed to 32 x 32. The ESM-integrated networks again outperform all baselines, learning to reach all three targets, while the baseline policies all only succeed reaching one. Full details the RL setup and learning curves are given in Appendix
4.2 OBJECT SEGMENTATION
We now explore the suitability of the ESM module for object segmentation. One approach is to perform image-level segmentation in individual monocular frames, and then perform probabilistic fusion when projecting into the map (McCormac et al. 2017). We refer to this approach as Mono. Another approach is to first construct an RGB map, and then pass this map as input network. This has the benefit of wider context, but lower resolution is necessary to store large map in memory, meaning details be lost. ESMN-RGB adopts this approach. Another approach is to combine monocular predictions with multi view optimization to gain the benefits of wider surrounding context in (Zhi et al. 2019). Similarly, the ESMN architecture is able combine monocular inference with
8

---

Published as a conference paper at ICLR 2021
the wider map context, but does so by constructing a network with both image-level and map-level convolutions. These ESM variants adopt the same broad architectures as shown in F with the full networks specified in Appendix AA.4 We do not attempt quote state-of-ther the-art results, but rather to further demonstrate the wide applications of the ESM module, and to explore the effect of placing the ESM module at different locations in the convolutional stack. We evaluate segmentation accuracy based on the predictions projected to the ego-centric map. With fixed network capacity between methods, we see that ESMN outperforms both baselines, see Table for the results, and Figure 7 for example predictions in ScanNet reconstruction. Further details are given in Appendix A.4
HSNN
Mislabelled colling and table from mono fusiin
Bed
Books
Cailing
Chair
Floor
Furniture
Objects
Picture
Sofa
Table
Will
Window
Figure 7: Top: Segmentation predictions ESM memory for Mono and ESMN, and ground truth. Middle: Point cloud rendering of the predictions. Bottom: Monocular input image sequence.
Images 60 x 80
Images 120 x 160
Images 60 x 80
Memory 90 x 180
Memory x 180
Memory 180 x 360
1-frame
16-frame
1-frame
16-frame
1-frame
16-frame
Mono
54.5(19.0)
55.1(13.6)
54.9(19.3)
55.6(13.9)
54.6(19.2)
55.3(13.7)
ESMN-RGB
51.4(19.6)
54.1(13.8)
51.7(19.3)
54.4(13.2)
54.0(19.2)
57.1(13.5)
ESMN
55.0(19.0)
59.4(12.7)
55.3(19.1)
59.8(13.0)
55.2(19.4)
59.7(12.9)
Table 4: Object segmentation segmentation accuracies on the ScanNet test set for a monocular fusion baseline (Mono), well ESMN-RGB and ESMN.
5 CONCLUSION
Through diverse set of demonstrations, have shown that ESM represents widely applicable computation graph and trainable module for tasks requiring general spatial reasoning. When compared other memory baselines for mmgeeto--action learning, our module outperforms these dramatically when learning both from ego-centric and scene-centric images. One weakness of our method is that assumes the availability of both incremental pose measurements of all scene cameras, and depth measurements, which is not a constraint of the other memory baselines. However, with the ever increasing ubiquity of commercial depth sensors, and with there being plentiful streams for incremental pose measurements including visual odometry, robot kinematics, and inertial sensors, we argue that such measurements are likely to be available in any real-world application of embodied AI. We leave it to future work to investigate the extent to which ESM performance deteriorates with highly uncertain roal-world measurements, but with full uncertainty propagation and probabilistic per pixel fusion, ESM is well suited for accumulating noisy measurements in principled manner.

---

Published as a conference paper at ICLR 2021
REFERENCES
Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison. Codeslam learning compact, optimisable representation for dense visual slam. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2560-2568, 2018.
Jake Bruce, Niko Sünderhauf, Piotr Mirowski, Raia Hadsell, and Michael Milford. Learning deploy- able navigation policies at kilometer scale from a single traversal. arXiv preprint arXiv:1807.05211, 2018.
Neil Burgess. Spatial memory: how egocentric and allocentric combine. Trends in cognitive sciences, 10(12):551-557, 2006.
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, lan Reid, and John J Leonard. Past, present, and future of simultancous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics, 32(6):1309 1att1139933233 2016.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: /259, 2014.
Cevahir Cigla, Roland Brockers, and Larry Matthies. Gaussian mixture models for temporal depth fusion. 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 889-897. IEEE, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint 1002.01800.155550 2018.
Péter Fankhauser, Michael Bloesch, Christian Gehring, Marco Hutter, and Roland Siegwart. Robot- centric elevation mapping with uncertainty estimates. In Mobile Service Robotics, pp. 433 World Scientific, 2014.
Anthony T Fragoso, Cevahir Cigla, Roland Brockers, and Larry H Matthies. Dynamically feasible motion planning for micro air vehicles using an egocylinder. Field and Service Robotics, pp. 433-447. Springer, 2018.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint 1rXiv:1410.5400 2014.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2616-2625, 2017.
Joao F Henriques and Andrea Vedaldi. Mapnet: An allocentric spatial memory for mapping environ- ments. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8476-8484, 2018
James R Hinman, G William Chapman, and Michael E Hasselmo. Neuronal representation of environmental boundaries in egocentric coordinates. Nature communications, 10(1): 1-11 2019.
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge- Basedd Systems, 6(02): 107-116, 1998.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint wxXiv:161..0539. 2016.
Stephen James, Andrew Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation real world for multi-stage task. arXiv preprint arXiv: 0700 2017.
10

---

John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference Published as a conference paper at ICLR 2021
Stephen James, Mare Freese, and Andrew J Davison. Pyrep: Bringing v-rep to deep robot learning. arXiv preprint
2019a.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- efficient robotic grasping via randomized-to comanonnca adaptation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12627-12637, 2019b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RIbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Roberta Klatzky. Allocentric and egocentric spatial representations: Definitions, distinctions, and interconnections. In Spatial cognition, pp. 1-17. 1- Springer, 1998.
Daniel Lenton, Fabio Pardo, Fabian Falck, Stephen James, and Ronald Clark. Ivy: Templated deep learning for nnterfframwoor portability. arXiv preprint arXiv:2102.02886, 2021.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa G Narasimhan, and Jan Kautz. Neural rgb (r) d sensing: Depth and uncertainty from video camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10986-10995, 2019.
Jan Matas, Stephen James, and Andrew Davison. Simatt-real reinforcement learning for deformable object manipulation. arXiv preprint arXiv:1806.07881 2018.
on Robotics and automation (ICRA), pp. 4628 0628-4635.. IEEE, 2017.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. In Advances in Neural Information Processing Systems, pp. 2419-2493 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):599-533 2015.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium Mixed and Augmented Reality, pp. 127-136. IEEE, 2011.
Seoung Wug Oh, Joon- Young Lee, Ning Xu, and Scon Joo Kim. Video object segmentation using space-time memory networks. Proceedings of the IEEE International Conference on Computer Vision, pp.
Joon-Young
2019.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. arXiv preprint 1eX.v:1702.08360 2017.
Emilio Parisotto, H Francis Song, Jack W Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. arXiv preprint 1910::110000664 2019.
Richard Alan Peters, Kimberly E Hambuchen. Kazuhiko Kawamura, and D Mitchell Wilkes. The sensory ego-sphere as a short-term memory for humanoids. In Proceedings of the IEEE-RAS international conference humanoid robots, number 1, pp. 451-459 2001.
11

---

Published as a conference paper at ICLR 2021
Eric Rohmer, Surya PN Singh, and Mare Freese. V-rep: A versatile and scalable robot simulation framework. In International Conference on Intelligent Robots and Systems. IEEE, 2013.
Fereshteh Sadeghi, Alexander Toshev, Eric Jang, and Sergey Levine. Sim2real view invariant visual servoing by recurrent control. arXiv preprint arXiv:1712.07642, 2017.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, al. End-to-end memory networks, In Advances in neural information processing systems, pp. 2440-2448, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Greg Wayne, Chia-Chun Hung. David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, et al. Unsupervised predictive memory in goal-directed agent. arXiv preprint arXiv: 1803.16000 2018.
Jason Weston, Sumit Chopra, and Antoine Bordes.
Memory networks. arXiv preprint
arXiv: 14X1::1400.316, 2014.
Thomas Whelan, Stefan Leutenegger, R Salas-Moreno, Ben Glocker, and Andrew Davison. Elastic- fusion: Dense slam without pose graph. Robotics: Science and Systems, 2015.
Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference machine learning, pp. 2397-2406, 2016.
Mengmi Zhang, Keng Teck Ma, Shih-Cheng Yen, Joo Hwee Lim, Qi Zhao, and Jiashi Feng. Ego- centric spatial memory. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 137-144. IEEE, 2018.
Shuaifeng Zhi, Michael Bloesch, Stefan Leutenegger, and Andrew Davison. Scenecode: Monocular dense semantic reconstruction using learned encoded scene representations, In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp- 11776-11785, 2019.
Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox. Deeptam: Deep tracking and mapping. In Proceedings of the European conference on computer vision (ECCV), pp. 822-838, 2018
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Fei-Fei, and Ali Farhadi. darget-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
A APPENDIX
A.1 MULTI-DOF IMITATION LEARNING
A.1.1 OFFLINE DATASETS
The image sequences for the offline datasets are captured following random motion of the agent in both the DR and MR tasks, but known expert actions for each of the three possible targets in the scene are stored at every timestep. The drone reacher is initialized in random locations at the start of each episode, whereas the manipulator reacher always started in the same robot configuration overlooking the workspace, as in the original RL Bench reacher task.
For the scene-centric acquisition, we instantiate three separate scene-centric cameras in the scene. In order to maximise variation in the dataset to encourage network generalization to arbitrary motions at test-time, we reset the pose of each of these scene-centric cameras at every step of the episode, rather than having each camera follow smooth motions. Each new random pose has rotational bias to face towards the scene-centre, to ensure objects are likely to be seen frequently. By resetting the cameras poses every time-step. encourage the networks learn to make sense of the pose information given to the network, rather than learning policies which fully rely on smoothly and slowly varying images.
12

---

Published as a conference paper at ICLR 2021
Both tasks use ego-centric cameras with wider field of view (FOV) than the scene-centris cameras. This is a common choice in robotics, where wide angle perception is especially necessary for body- mounted cameras. RLBench by default uses ego-centric FOV of 60 degrees and scene-centric FOV of 40 degrees, and we use the same values for our Rl.Bechh-derive MR task. For the drone reacher task, we use ego-centric FOV of 90 degrees and scene-centric FOV of 55 degrees, enable all methods to more quickly explore the perceptual ego-sphere.
For the manipulator reacher dataset, we also store robot mask image, which shows the pixels corresponding to the robot for all ego-centric and scene-centric images acquired. Known robot forward kinematics are used for generating the masking image. All images in the offline dataset are 32 x 32 resolution.
A.1.2 TRAINING
To maximize the diversity from the offline datasets, new target is randomly chosen from each of the three possible targets for each successive step in the unrolled time dimension in the training batch. This ensures maximum variation in sequences train-time, despite relatively small number of 100k 16-frame sequences stored in the offline dataset. This also strongly encourages each of the networks to learn remember the location of every target seen so far, because any target can effectively be requested from the training loss at any time.
Similarly, for the scene-centric cameras we randomly choose one of the three scene cameras each successive time-step in the unrolled time dimension for maximum variation. Again this forces the networks to make use of the camera pose information to make sense of the scene, and prevents overfitting on particular repeated sequences in the training set, instead encouraging generalization to fully arbitrary motions. For these experiments, the baseline methods of Mono, LSTM, LSTM-Aux and NTM also receive the full absolute camera pose at each step rather than just incremental poses received by ESM, as we found this to improve the performance of the baselines.
For training the manipulator reacher policies, we additionally use the robot mask images to set high variance pixel values before feeding to the ESM module. This prevents the motion of the robot from breaking the static-scene assumption adopted by ESM during re-projections. We also provide the mask image as input to the baselines.
All networks use batch size of 16, unroll size of 16, and are trained for 250k steps using an ADAM optimizer with 1e- 4 learning rate. None of the memory baselines cache the the internal state between training batches, and so the networks must learn utilize the memory during the 16-frame unroll. 16 frames is on average enough steps to reach all 3 of the targets for both tasks.
A.1.3 NETWORK ARCHITECTURES
The network architectures used in the imitation learning experiments are provided in Fig 8 Both LSTT baselines use dual stacked architectures with hidden and cell state sizes of 1024. For NTM we use similar variant to that used by Wayne et al. (2018), namely, we use sequential writing to the memory, and retroactive updates. Regarding the 16-frame unroll, we again emphasize that 16 steps is on average enough time to reach all targets once. In order to encourage generalization to longer sequences than only 16-steps, we limit the writable memory size to 10, and track the usage of these 10 memory cells with usage indicator such that subsequent writes can preferentially overwrite the least used of the 10 cells. This again is the same approach used by Wayne et al.) (2018). which is one of very few works successfully apply NTM-style architectures image-to-action domains. It's important to note that the use of retroactive updates makes the total memory size actually 20, as half of the cells are always reserved for the retroactive memory updates. Regarding image padding the borders for input to the convolutions, the Mono and LSTM/NTM basclines use standard zero padding, whereas ESMN-RGB and ESMN pad the outer borders with the wrapped umni-directional image.
A.1.4 AUXILIARY LOSSES
Motivated by the fact that many successful applications of LSTMs to image- image-to-actions to-actions learning involve the use of spatial auxiliary losses (Jaderberg et al.) [0166] James et al. [2017: (Sadeghi et al.) 2017; Mirowski et al. 2018). we also compare to an LSTM which uses two such auxilarr proposals, namely the attention loss proposed in (Sadeghi et al. 2017) and 3-dimensional Fuler-based variant
13

---

Published as a conference paper at ICLR 2021
Mono
LSTM / NTM
InDDut
Emiaaiia MA 7
Layers
EXTRA S2 301.
30
Con P
Memory
To Carn pana
52x32x18
37-12x18
ESMN-RGB
ESMN
Coote Caroy 321. 31
Comte 315
31
3x3t
Sx5 5
DOTN CC
comni 11
3x3
Sx5
Comee 2x0
COoTHe S2.
32x32x4
9bx180x10
E
3263224
90x180x16
Figure 8: Image-to-action imitation learning network architectures for Mono, LSTM/NTM, ESMN- RGB and ESMN.
of the heading loss proposed in (Mirowski et al. 2018), which itself only considers 1D rotations normal to the plane of navigation. Our heading loss does not compute the ID rotational offset from North, as this not detectable from the image input. Instead, the networks are trained to predict the 3D Euler offset from the orientation of the first frame in the sequence. The modified LSTM network architecture is presented in Fig 9 and example images and classification targets for the auxiliary attention loss are presented in Fig 10 We emphasize that we did not attempt to tune these auxiliary losses, and applied them unmodified to the total loss function, taking the mean of the cross entropy loss for each, and linearly scaling so that the total auxiliary loss is roughly the same magnitude as the imitation loss at the start of training. Tuning auxiliary losses on different tasks is known to be challenging, and the losses can worsen performance without time-consii tnne-consuming manual tuning, as evidenced in the performance of the UNREAL agent (Jaderberg et al. 2016) compared to a vanilla RL-LSTM network demonstrated in (Wayne et al.] 2018). We reproduce this general finding, and see that the untuned auxiliary losses do not improve performance on our reacher tasks. To further investigate the failure mechanism, we plot the two auxiliary losses on the validation set during training for each task in Fig II We find that the heading loss over-fits on the training set in all tasks, without learning any useful notion of incremental agent orientation. This is particularly evidenced in the DR tasks, which start each sequence with random agent orientations. In contrast, predicting orientation relative to the first frame on the MR task much simpler because the starting pose always constant in the scene, and so cues for the relative orientation are available from individual frames. This is why we observe a lower heading loss for the MR task variants in Fig [T] We do however still observe all tasks helps to explain why LSTM-Aux performs
overfitting in the MR task. This overfitting worse than the vanilla LSTM baseline for some of the tasks in Table In contrast, the ESM module embeds strong spatial inductive bias into the computation graph itself, requires tuning at all, and consistently leads to successful policies on the different tasks, with no sign of overfitting on any of the datasets, further discuss in Section A.1.5.
A.1.5 FURTHER DISCUSSION OF RESULTS
The losses for each network evaluated on the training set and validation set during the course of training are presented in Fig I2 We first consider the results for the drone reacher task. Firstly, we can clearly see from the De-goo-rgb and DR- OR-frecform--gb tasks that the baselines struggle to interpret the stream of incremental pose measurements and depth, in order to select optimal actions in the training set, and this is replicated in the validation set, and in the final task performance in Tab I We can also see that ESMN is able to achieve lower training and validation losses than ESMN-RGB when conditioned on shape in the DR-ego-shape and DR-freeform-shape tasks, and also expectedly achieves higher policy performance, shown in in Tab I What we also observe is that the baselines have a higher propensity to over-fit on training data. Both the LSTM and NTM baselines achieve lower training set error than ESMN on the Dh-ego-shape task, but not lower validation error. In contrast, all curves for ESM-integrated networks are very similar between the training and validation set. The ESM module by design performs principled spatial computation, and so these networks are inherently much more robust to overfitting.
14

---

Published as a conference paper at ICLR 2021
Figure 10: Top: pixel-wise attention classification targets for the LSTM-Aux network on DR (left) and MR (right) tasks. Bottom: the corresponding monocular images from the DR (left) and MR (right) tasks.
Auctiarr Attention
Aual Heading Loss
validation loss
0008
0.0
onee
300
150
200
300
100
200
250
training steps (1000s)
DR- Ego-Color
DR-Ego-Shape
DR-Freeform-Co
DR-Freeform-Shaag
MR-Ego-Color
MR-Ego-Shape
MR-Freeform-Color
MR-Freeform-Shape
Figure 11: Auxiliary attention (left) and heading (right) losses on the validation during training for each of the different imitation learning reacher tasks.
Looking to the manipula- tor reacher task, we first notice that the LSTT and NTM baselines are actually able to achieve lower losses than the ESM-integrated networks both the train- ing set and validation set for the MR-ego-rgb and MR- ego-shape tasks. However, this does not translate higher policy performance in Table I The reason for this is that the RLBench reacher task always initializes the robot in the same configuration, and so the diversity in the offline dataset is less than that of the drone reacher offline dataset. The scope of possible robot config- urations each 16-step window in the dataset is more limited. In essence, the baselines are achieving well in both training and validation sets result of overfitting to the limited data distributions observed. What these curves again highlight is the strong generalization power ESM-integrated networks. Despite seeing relatively limited robot configurations in the dataset, the ESM policies
LSTM AuX
IryOut
ConNE32 3t3.1 11
Layers
Memory
ConeeT 1R,
PC SOME
32x32x18
Tistton outer logits
ellention
lugto
② suter
Bx0R2
per SONN
Figure 9: Network architecture for the LSTM-AuX baseline.
15

---

Published as a conference paper at ICLR 2021
do not overfit on these, and are still able use this data to learn general policies which succeed from unseen -ittof--istribution images at test-time. We also again observe the same superiority of ESMN over ESMN-RGB when conditioned on shape input in the training and validation losses for the MR-ego-shape task.
A final observation is that all methods fail to perform well the MR-freeform-shape task. We investigated this, and the weak performance is a combined result of low-resolution 32 x 32 images acquired in the training set and the large distance between the scene-centric cameras and the targets in the scene. The shapes are often difficult to discern from the monocular images acquired, and so little information is available for the methods to successfully learn a policy. We expect that with higher resolution images, or with an average lower distance between the scene-cameras and workspace in the offline dataset, we would again observe the ESM superiority observed for all other tasks.
Mono
LSTM
LSTM Aux
NTM
ESMN-RGB
ESMN
PO2
0.90
6.0
6:42
training
0.99
0.0
Drone Reacher
1.40
ade
6.9
CW ago 19b
Of ago shape
CM freeforn ago
DR freeforoom shape
validation
6:33
030
0300
4.000
100
180
UR ego rgo
H4 ego shope
freeform rzb
MR freeform stape
0.50
0.30
3.40
0:39
training
9452-
928
Manipulator Reacher
6.50
023
023
5.07
VR ago r9b
14 ogo shape
froeform rgb
MR. Freeforr mo Shape
044
0.43
0.03
validation
0.20
0:42
039
0110
250
750
200
300
250
300
250
100
180
330
250
130
350
200
250
training steps (1000s)
Figure 12: Network losses on the training (top 8) and validation set (bottom 8) during the course of training for imitation learning from the offline datasets, for the different reacher task.
16

---

Published as a conference paper at ICLR 2021
A.1.6 IMPLICIT FEATURE ANALYSIS
Here briefty explore the nature of the the features which the end-to-end ESM module learns store in memory for the different reacher tasks. For each task, we perform a Principal Component Analysis (PCA) on the features encoded by the pre-ESM encoders in the ESMN networks. We compute the principal components (PCs) using encoded features from all monocular images in the training dataset. We present example activations for each of the principal components for a sample of monocular images taken from each of the shape conditioned task variations in in Fig [3] with the most dominate principal components shown on the left in green, going through to the least dominant principal component on the right in purple. Each principal component is projected to different color- space for better visualization, with plus or minus one standard deviation of the principal component mapping to the full color-space. Lighter colors correspond to higher PC activations.
We can see that most dominant PC (shown in green) for the drone reacher tasks predominantly activate for the background, and the third PC (blue) appears activate most strongly for edges. The principal components also behave similarly on the Eek-Eg-Shappe task. However, on the TRX-Freeform-Shar task, which neither ESMN any of the baselines able to succeed on, the first PC appears to activate strongly on both the arm and the target shapes.
The main conclusion which we draw from Fig [3]ss is that the pre-ESM encoder does not directly encode shape classes as might be expected. Instead, the encoder learns to store other lower level features into ESM. However, as evidenced the results in Table []] the combination of these lower level features in ESM is clearly sufficient for the post-ESM convolutions to infer the shape id for selecting the correct actions the policy, least by using collection of the encoded features within a small receptive field, which is not possible when using pure RGB features.
(a) DR- OR-Ego-Shapp
(b) DR- Fee-Freef--Sheaee
(c) MR-Ego-Shape
(d) MR-Freeform--Pe Shape
RGB
Principal Component:
Figure 13: Principal Components (PCs) of the features from the pre-ESM encoder of the ESMN architecture on some example images for each of the four shape-conditioned reacher tasks, with each of the six the principal components mapped different colors to maximise clarity, PCs go from most dominant on the left (green) to least dominant on the right (purple). Lighter values correspond to higher PC activation, with black indicating low activation.
17

---

Published as a conference paper at ICLR 2021
A.2 OBSTACLE AVOIDANCE
For this experiment, we increase the drone reacher environment by 2x in all directions, resulting in an 8x increase in volume. We then also add 20 spherical obstacles into the scene with radius 0.1 For the avoidance, we consider a bubble around the agent with radius R = 0.2, and flag collision whenever any part an obstacle enters this bubble. Given the closest depth measurement available deloseat. the avoidance algorithm simply computes avoidant velocity vector whose magnitude is inversely proportional to the distance from collision, clipped to the maximum velocity [v] malz Equation 5 shows the calculation for the avoidance vector magnitude. We run the avoidance controller at 10x the rate of the ESM updates.
10-3
|val min
R. 10 1212 /0/man
(5)
max [closest
In order prevent avoidant motion away from the targets reach, retrain the ESMN-RGU networks on the drone reacher task, but we train the network to also predict the full relative target location additional auxiliary loss. When evaluating the obstacle avoidance task, prevent depth values within a fixed distance this predicted target location from influencing the obstacle avoidance. This has the negative effect of causing extra collisions when the agent erroneously predicts that the target is close, but it enables the agent to approach and reach the target without being pushed away by the avoidance algorithm. Regarding the performance against the baseline, all monocular images have a large field-of sildd-of-vewwww view of 90 degrees, and yet we still observe significant reductions in collisions when using the full ESM geometry for avoidance, see Tab
re-iterate that
A.3 MULTI-DOF REINFORCEMENT LEARNING
A.3.1 TRAINING
For the reinforcement learning experiment, train both ESMN and ESMN-RGB well as all baselines on a similar sequential target reacher task as defined in Section 4.1 via DQN (Mmih et al. 2015), where the manipulator must reach red, blue and then yellow targets from egocentric observations. We use (128 x 128) images in this experiment rather than (32 x 32) as used in the imitation learning experiments. We also use an unroll length of rather than 16. We use discrete delta end-effector translations, with +0.05 meters for each axis, with no rotation (resulting in an action size of 6). We use shaped reward of -len -len(rene)) (rwmaininggtaraaeeeeeeeeeee (arggttrresst) lle gll2, where and are the gripper and current target translation respectively.
MR Seq
shaped reward
20
60
80
100
training steps (1000s)
Mono
LSTM
LSTM Aux
NTM
ESMN-RGB
ESMN
PO2
Figure 14: Average return during RL training on sequential reacher task over 5 seeds. Shaded region represent the min and max across trials.
18

---

Published as a conference paper at ICLR 2021
A.3.2 NETWORK ARCHITECTURES
In order to make the RL more tractable, and enable larger batch sizes, we use smaller network than used in the imitation learning experiments. Both methods use Siamese network to process the RGB and coordinate- image inputs separately, and consist of 2 convolution (conv) layers with 16 and 32 channels (for each branch). We fuse these branches with 32 channel conv and Ixl kernel. The remainder of the architecture then follows the same as in the imitation learning experiments. but instead use channel sizes of 64 throughout. The network outputs 6 values corresponding the Q-values for each action. All methods use learning rate of 0.001, target Q-learning т = 0.001, batch size of 128, and leakyReLU activations. We use an epsilon greedy strategy for exploration that is decayed over 100k training steps from to 0.1. We show the average shaped return during RL training on the sequential reacher task over seeds in Fig 14 Both ESM policies succeed in reaching all 3 targets, whereas all baseline approaches generally only succeed in reaching 1 target. The Partial-Oracle--
(PO2) baseline also succeeds in reaching targets.
A.4 OBJECT SEGMENTATION
A.4.1 DATASET
For the object segmentation experiment, we use down- sampled 60 80 and 120 160 images from the Scan- Net dataset, which we first RGB-Depth align. We reduced dataset with frame skip of 30 to maximize di- versity whilst minimizing dataset memory. Many quences contain slow cam- motion, resulting in ad- jacent frames which vary very little. We the Eigen- 13 classification la- bels as training targets.
Mono
(2.33)
05001112 313.11
SDc120c10 memoey
SDs BDG33 pardittion
ESMN-RGB
PATT 24
2243204
90w180K13 senrrratioon prodecioon
memory
ESMN
A.4.2 NETWORK ARCHITECTURES
SE CIE Sa SOMIE CERE
The Mono, ESMN-RGB and ESMN networks all ex- hibit a U-Net architecture, and output object segmenta- tion predictions sphere map. ESMN-RGB and ESMN do so with a U-Net connecting the ESM output to the final predictions, and Mono does so by projecting and probabilistically fusing the monoc- ular segmentation predictions in a non-learnt manner. The network architectures are all presented in Fig 15 Regarding image padding at the borders for input to the convolutions, the Mono and LSTM/NTMM baselines use standard zero padding, whereas ESMN pads the outer borders with the wrapped omni-directional image.
Comtr Sel.
Camel3 203
cazhing
OCHABCA30 memory
D0x18bu13 prediciion
an ego-
Figure 15: Object segmentation network architectures for Mono, ESMN-RGB and ESMN.
A.4.3 TRAINING
For training, all losses are computed in the ego-sphere map frame of reference, either following convolutions for ESMN and ESMN-RGB, or following projection and probabilistic fusion for the Mono case. We compute ground-trui truth segmentation training target labels by projecting the ground truth monocular frames form ego-sphere target segmentation image, the right-hand-side of
19

---

Published as a conference paper at ICLR 2021
Fig 7 for example. We chose this approach over computing the ground truth segmentations from the complete ScanNet meshes for implementational simplicity. All experiments use batch size of 16, unroll size of in the time dimension, and Adam optimizer with learning rate le 4, trained for 250k steps.
A.5 RUNTIME ANALYSIS
In this section, we perform a runtime analysis of the ESM memory module. We explore the extent to which inference speed is affected both by monocular resolution and egosphere resolution, well the differences between CPU and GPU devices, and the choice of machine learning framework. Our ESM module is implemented using Ivy (Lenton et al. 2021). which is templated deep learning framework supporting multiple backend frameworks. The implementation our module is therefore jointly compatible with TensorFlow 2.0, Py Porch,, MXNet, Jax and Numpy. We analyse the runtimes of both the TensorFlow 2.0 and PyTorch implementations, the results are presented Tables [5] reconstruct 7 and [ All analysis was performed while using ESM with RGB projections
ScanNet scene 0002-00 shown in Fig [6 The timing is averaged over the course of the 260 frames in the frame-skipped image sequence, with frame skip of 30, for this scene. ESM steps with 960 x 1280 monocular images were unable to fit into the 1 IGB of GPU memory when using the PyTorch implementation, and so these results are omitted in Table g
Figure 16: Left: point cloud representation of the ego-centric memory around the camera after full rotation in ScanNet scene 0002-00, with RGB features. Mid: (top) Equivalent onni-directional RGB image, (bottom) equivalent omni-directional depth image, both without smoothing to better demonstrate the quantization holes. Right: (top) single RGB frame, (bottom) a single depth frame. This is the reconstruction produced during the time-analysis. This particular reconstruction used monocular resolution of 120 x 60, and a memory resolution of 180 x 360
Monocular Res
60 x 80
120 x 160
240
320
480 x 640
960
1280
Memory Res
45 90
245.4
162.6
83.7
24.4
6.3
90 x 180
140.1
126.5
70.8
23.3
6.1
180 x 360
63.9
64.0
47.5
19.2
5.8
360 x 720
16.3
14.3
14.5
11.1
4.7
720 1440
3.9
3.7
3.6
3.6
2.7
1440 x 2880
1.1
1.1
1.0
1.0
0.9
Table 5: Average frames-per-secont (fps) runtime for the TensorFlow 2 implemented ESM module on the ScanNet scene 0002-00, with RGB projections, running on CPU cores.
What we see from these runtime results is that the off-the- the--hh--hff ESM module is fully compatible as real-time mapping system. Compared to more computationally intensive mapping and fusion pipelines, the simplicity of ESM makes it particularly suitable for applications where depth and pose measurements are available, and highly responsive computationally cheap local mapping is a strong requirement, such on-board mapping for drones.
20

---

Published as a conference paper at ICLR 2021
Monocular Res
60 x 80
120 X 160
240 320
480 x 640
960 x 1280
Memory Res
45 x 90
262.1
223.8
166.5
76.2
24.1
90 x 180
193.5
214.8
164.4
73.5
23.6
180 360
184.6
181.6
147.6
71.5
23.0
360 X 720
91.0
89.1
83.6
56.8
21.6
720 1440
19.8
19.5
18.7
17.6
11.3
1440 X 2880
4.9
4.8
4.8
4.5
4.2
Table 6: Average frames-per-second (fps) runtime for the TensorFlow 2 implemented and pre-
compiled ESM module the ScanNet scene 0002-00, with RGB projections, running on Nvidia RTX 2080 GPU.
Monocular Res
80
120 x T60
240 320
480 x 640
960
1280
Memory Res
45 x 90
98.9
45.8
26.3
6.9
1.8
90 180
57.0
36.8
22.3
6.6
1.7
180 x 360
14.1
11.6
10.5
5.0
1.5
360 720
5.5
5.1
4.8
3.2
1.4
720 x 1440
1.5
1.4
1.4
1.3
0.8
1440 x 2880
0.4
0.4
0.4
0.4
0.3
Table 7: Average frames-per-second (fps) runtime for the PyTorch implemented ESM module on the ScanNet scene 0002-00, with RGB projections, running on 8 CPU cores.
Monocular Res
60 x 80
120 x 160
240 320
480 x 640
960
1280
Memory Res
x 90
108.2
105.5
92.5
86.1
90 x 180
102.0
92.2
85.0
83.8
180 x 360
81.8
79.9
76.2
72.0
360 x 720
44.1
43.0
42.3
41.7
720 x 1440
13.9
13.9
13.8
13.1
1440 x 2880
3.7
3.7
3.7
3.6
Table 8: Average frames-per-second (fps) runtime for the PyTorch implemented ESM module on the ScanNet scene 0002-00, with RGB projections, running on Nvidia RTX 2080 GPU.
21


Reviews:
[{"review_id": "m5Q_yRkj82", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of creating spatial memory representations, which play important roles in robotics and are crucial for real-world applications of intelligent agents. The paper proposes an ego-centric representation that stores depth values and features at each pixel in a panorama. Given the relative pose between frames, the representation from the previous frame is transformed via forward warping (using known depth values) to the viewpoint of the current frame. The proposed approach has no learnable parameters. Experiments on a wide range of tasks show that the proposed approach outperforms baselines such as LSTM and NTM.\n\nOn the positive side, the approach is positively simple, in the sense that it relies on known techniques (EKF, forward warping, etc.) that ensure that it is easy to implement while achieving good results in the experiments. Up to Sec. 4, I found the paper easy to follow, although some design choices could be better motivated (e.g., I assume that diagonal covariances are assumed for simplicity).\nThe paper evaluates the proposed approach on multiple tasks and in various configurations, which is another strength of the paper.\n\nWhile I like the proposed approach, I also see multiple significant weaknesses:\n1) I found the experimental evaluation nearly impossible to understand. My main problem is that I don't understand what the different method that are evaluated are:\n * Given the abbreviation ESMN introduced in the abstract, I assume that ESMN is the proposed approach. ESM seems to be a variant of ESMN, but I am not sure how ESM and ESMN differ as the difference is never clearly described (or if it is, I seem to have missed it). Sec. 4.1.1 mentions training with a convolutional encoder in the context of ESMN, Sec. 4.1.3 and Sec. 4.1.4 only evaluate ESM but not ESMN, while Sec. 4.2 states that \"ESM represents map-only inference, while ESMN includes convolutions for both the image-level and map-level inference\". Unfortunately, the term \"map-level\" inference is not well-defined. Overall, I don't understand the difference between ESM and ESMN. As a result, it is unclear to me why ESM performs worse than ESMN in Tab. 1 for DR-Ego-S but comparable for all other tasks in the table, or why ESM and not ESMN is used for some of the experiments. Similarly, what is the \"ESM-DepthAvoid\" baseline? Does it only use depth and no features?\n * Tab. 1 contains a baseline called \"PO\" and I don't understand how it works. The abstract introduces PO as an abbreviation for partial observability, but that does not seem to be an explanation for a baseline. The PO baseline performs similarly well as ESM and ESMN in Tab. 1, which makes me wonder whether Tab. 1 really shows the superiority of the proposed approach.\n * I am confused by the statement \"In contrast, ESM by design stores features in the memory with meaningful indexing. The inclusion of relative cartesian co-ordinates in the memory image also effectively aligns each pixel with an associated relative translation.\" since the inclusion of such coordinates is never mentioned before. I assume ESM uses some form of handcrafted features?\n2) Some of the statements made in the paper seem too strong:\n * I don't see how the claim \"Our memory is much more expressive than these 2D examples, with the ability to represent detailed 3D geometry in all directions around the agent.\" (Sec. 2.2)  holds. I agree that being able to store information in 2.5D (panorama and depth) is more powerful than storing only top-down 2D maps. However, the allocentric maps of the references can store larger and more complicated scene parts. E.g., if an agent would turn around a corner, ESMN would essentially be forced to forget about everything that is not directly visible anymore as it is occluded and thus not included in the memory structure anymore. As such, a point can be made that ESMN is much less expressive than for example Henriques & Vedaldi. Given that the latter evaluate on significantly more complex scenes compared to the ones used in this paper (which do not have strong occlusions) strengthens this impression. \n * Regarding the statement \"Although the most recent depth frame is also a strong signal for local obstacle avoidance, we show that the avoidance based on the full ESM geometry results in fewer collisions when tested on a variant of the drone task with the inclusion of 25 obstacles.\": Looking at Tab. 2, it seems to me that the standard deviation is so large for both ESM and ESM-DepthAvoid that it is unclear to me whether one is consistently better than the other. \n3) The CodeSLAM approach from Bloesch et al. also provides a form of memory representation and I don't understand why the paper, and its follow-up (Zhi et al., SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded\nScene Representations, CVPR 2019), is not discussed in the related work section.\n\nOverall, I believe that the paper has potential. My main criticism is that I do not feel able to properly understand the experimental evaluation. As such, it is hard to recommend acceptance. However, I am willing to increase my score if the information necessary to understand this part of the paper is provided.\n\n### After rebuttal phase ###\nThe answers provided by the authors and the revised version of the paper sufficiently address my concerns. As such, I recommend to accept the paper. I am still concerned that the experimental evaluation is very packed with multiple experiments while lacking details on the experimental setup and explanations of the baselines. Still, I feel that in the current form, the paper can be accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E74-ljiAYm4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n-------\nThis paper presents a method to build an ego-centric spatial memory map from an agent's viewpoint. This map module is differentiable and can be used for a variety of tasks, such as object segmentation, or image-to-action learning in different control tasks.\n\n# Pros\n----\n+ The idea of combining the ego-centric representation seems interesting and novel. \n+ Evaluation in Image-to-Action learning shows good performance compared to baselines. \n\n# Cons\n----\n   *  _[Major]_ The Method section is very confusing. The method is based on the EKF pipeline and modifies the particular parts of the pipeline. The authors only briefly outline the overview of the method and then jump into specific details.  The text mostly focuses on improvement, and hence it is hard to judge which parts are novel and what the contribution is. Furthermore, there are errors in notation and variables that are not properly explained. u_t is first defined as incremental pose measurements and then as a control vector. Kamera intrinsic matrix K_1 is never introduced. The method section would benefit from restructuring, especially giving a clear overview of the method would help the reader to understand the method better.\n\n   * _[Major]_ Experiments are not well explained, and more baselines are necessary for proper evaluation. In all experiments, the experimental set-up is not well defined. For example, in the first experiment, we find out that the authors use imitation learning only from the appendix. In the second experiment, the environment used for the evaluation is not introduced. The experimental set-up needs to be clearly outlined, and the goal of the experiment needs to be provided. The choice of baselines is not well motivated. Why were LSTM and NTM chosen in the first experiment? There is very little rationale for the choice in the submission. Do these approaches show the state-of-the-art performance on the examined tasks? In the second experiment, the authors only use the LSTM network to train the RL agent. Various methods tackle the navigation problem in the RL domain, and hence the authors should choose better baselines for comparison. I suppose that the PO task is some form of navigation task, although the authors never explain what the PO task is. \n\n   * _[Minor]_ The simple approach to directly quantize pixel projections leads to artifacts in the map. It is a bit unclear why this option was chosen, especially now that progress on differentiable renderers has been made.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BGlQS3nCO-V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced Egocentric Spatial Memory Networks (ESMN), a novel learning paradigm and architecture for encoding spatial memory in a sphere representation. The representation can be used for several downstream applications ranging from semantic segmentation as well as action learning for robotics applications.\n\nThe formulation of the proposed approach builds on a Kalman Filter setting and encodes memory in a spherical structure. Both the formulation and design makes sense, and the downstream applications show that the proposed approach is indeed plausible.\n\nWhile I think the value of this paper is well justified, I do have a few comments on the paper:\n- I think it'll be worthwhile to elaborate on the \"Mono\" method, including the network architectures as well as training details and such. It will also help understand which part of ESMN improves over this baseline method.\n- In the related work section, one argument made by authors is that ESMN and MemNN/NTM are two different paradigms for learning spatial memory. While the proposed approach ESMN is reasonable, I wonder if it is possible to have a concrete experiment comparing those two different designs.\n- Could the authors elaborate on Sec 4.1.1 Fig 5? What's the meaning of the colorings in this figure, and how to interpret that?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors combine ideas from SLAM (using an Extended Kalman Filter and a state with nonlinear transitions and warping) and differentiable memory networks that store a spherical representation of the state (from the ego-centric point of view of an RL agent moving in an environment) with depth and visual features stored at each pixel and dynamics transitions corresponding to warping.

The main idea in the paper is very simple and elegant, but I will concur with the reviewers that the writing of the first version of the paper was extremely hard to understand and that the experimental section was too dense. Two subsequent revisions of the paper have dramatically improved the paper.

Given the spread of scores (R1: 6, R2: 7 and R3: 4) and the fact that only R1 and R2 have acknowledged the revisions, I will veer towards acceptance.

Decision: Accept (Poster)