End-to-End Egospheric Spatial Memory

1 INTRODUCTION . Egocentric spatial memory is central to our understanding of spatial reasoning in biology ( Klatzky , 1998 ; Burgess , 2006 ) , where an embodied agent constantly carries with it a local map of its surrounding geometry . Such representations have particular significance for action selection and motor control ( Hinman et al. , 2019 ) . For robotics and embodied AI , the benefits of a persistent local spatial memory are also clear . Such a system has the potential to run for long periods , and bypass both the memory and runtime complexities of large scale world-centric mapping . Peters et al . ( 2001 ) propose an EgoSphere as being a particularly suitable representation for robotics , and more recent works have utilized ego-centric formulations for planar robot mapping ( Fankhauser et al. , 2014 ) , drone obstacle avoidance ( Fragoso et al. , 2018 ) and mono-to-depth ( Liu et al. , 2019 ) . In parallel with these ego-centric mapping systems , a new paradigm of differentiable memory architectures has arisen , where a memory bank is augmented to a neural network , which can then learn read and write operations ( Weston et al. , 2014 ; Graves et al. , 2014 ; Sukhbaatar et al. , 2015 ) . When compared to Recurrent Neural Networks ( RNNs ) , the persistent memory circumvents issues of vanishing or exploding gradients , enabling solutions to long-horizon tasks . These have also been applied to visuomotor control and navigation tasks ( Wayne et al. , 2018 ) , surpassing baselines such as the ubiquitous Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) . We focus on the intersection of these two branches of research , and propose Egospheric Spatial Memory ( ESM ) , a parameter-free module which encodes geometric and semantic information about the scene in an ego-sphere around the agent . To the best of our knowledge , ESM is the first end-to-end trainable egocentric memory with a full panoramic representation , enabling direct encoding of the surrounding scene in a 2.5D image . We also show that by propagating gradients through the ESM computation graph we can learn features to be stored in the memory . We demonstrate the superiority of learning features through the ESM module on both target shape reaching and object segmentation tasks . For other visuomotor control tasks , we show that even without learning features through the module , and instead directly projecting image color values into memory , ESM consistently outperforms other memory baselines . Through these experiments , we show that the applications of our parameter-free ESM module are widespread , where it can either be dropped into existing pipelines as a non-learned module , or end-to-end trained in a larger computation graph , depending on the task requirements . 2 RELATED WORK . 2.1 MAPPING . Geometric mapping is a mature field , with many solutions available for constructing high quality maps . Such systems typically maintain an allocentric map , either by projecting points into a global world co-ordinate system ( Newcombe et al. , 2011 ; Whelan et al. , 2015 ) , or by maintaining a certain number of keyframes in the trajectory history ( Zhou et al. , 2018 ; Bloesch et al. , 2018 ) . If these systems are to be applied to life-long embodied AI , then strategies are required to effectively select the parts of the map which are useful , and discard the rest from memory ( Cadena et al. , 2016 ) . For robotics applications , prioritizing geometry in the immediate vicinity is a sensible prior . Rather than taking a world-view to map construction , such systems often formulate the mapping problem in a purely ego-centric manner , performing continual re-projection to the newest frame and pose with fixed-sized storage . Unlike allocentric formulations , the memory indexing is then fully coupled to the agent pose , resulting in an ordered representation particularly well suited for downstream egocentric tasks , such as action selection . Peters et al . ( 2001 ) outline an EgoSphere memory structure as being suitable for humanoid robotics , with indexing via polar and azimuthal angles . Fankhauser et al . ( 2014 ) use ego-centric height maps , and demonstrate on a quadrupedal robot walking over obstacles . Cigla et al . ( 2017 ) use per-pixel depth Gaussian Mixture Models ( GMMs ) to maintain an ego-cylinder of belief around a drone , with applications to collision avoidance ( Fragoso et al. , 2018 ) . In a different application , Liu et al . ( 2019 ) learn to predict depth images from a sequence of RGB images , again using ego reprojections . These systems are all designed to represent only at the level of depth and RGB features . For mapping more expressive implicit features via end-to-end training , a fully differentiable long-horizon computation graph is required . Any computation graph which satisfies this requirement is generally referred to as memory in the neural network literature . 2.2 MEMORY . The concept of memory in neural networks is deeply coupled with recurrence . Naive recurrent networks have vanishing and exploding gradient problems ( Hochreiter , 1998 ) , which LSTMs ( Hochreiter & Schmidhuber , 1997 ) and Gated Recurrent Units ( GRUs ) ( Cho et al. , 2014 ) mediate using additive gated structures . More recently , dedicated differentiable memory blocks have become a popular alternative . Weston et al . ( 2014 ) applied Memory Networks ( MemNN ) to question answering , using hard read-writes and separate training of components . Graves et al . ( 2014 ) and Sukhbaatar et al . ( 2015 ) instead made the read and writes ‘ soft ’ with the proposal of Neural Turing Machines ( NTM ) and End-to-End Memory Networks ( MemN2N ) respectively , enabling joint training with the controller . Other works have since conditioned dynamic memory on images , for tasks such as visual question answering ( Xiong et al. , 2016 ) and object segmentation ( Oh et al. , 2019 ) . Another distinct but closely related approach is self attention ( Vaswani et al. , 2017 ) . These approaches also use key-based content retrieval , but do so on a history of previous observations with adjacent connectivity . Despite the lack of geometric inductive bias , recent results demonstrate the amenability of general memory ( Wayne et al. , 2018 ) and attention ( Parisotto et al. , 2019 ) to visuomotor control and navigation tasks . Other authors have explored the intersection of network memory and spatial mapping for navigation , but have generally been limited to 2D aerial-view maps , focusing on planar navigation tasks . Gupta et al . ( 2017 ) used an implicit ego-centric memory which was updated with warping and confidence maps for discrete action navigation problems . Parisotto & Salakhutdinov ( 2017 ) proposed a similar setup , but used dedicated learned read and write operations for updates , and tested on simulated Doom environments . Without consideration for action selection , Henriques & Vedaldi ( 2018 ) proposed a similar system , but instead used an allocentric formulation , and tested on free-form trajectories of real images . Zhang et al . ( 2018 ) also propose a similar system , but with the inclusion of loop closure . Our memory instead focuses on local perception , with the ability to represent detailed 3D geometry in all directions around the agent . The benefits of our module are complementary to existing 2D methods , which instead focus on occlusion-aware planar understanding suitable for navigation . 3 METHOD . In this section , we describe our main contribution , the egospheric spatial memory ( ESM ) module , shown in Figure 1 . The module operates as an Extended Kalman Filter ( EKF ) , with an egosphere image µt ∈ Rhs×ws× ( 2+1+n ) and its diagonal covariance Σt ∈ Rhs×ws× ( 1+n ) representing the state . The egosphere image consists of 2 channels for the polar and azimuthal angles , 1 for radial depth , and n for encoded features . The angles are not included in the covariance , as their values are implicit in the egosphere image pixel indices . The covariance only represents the uncertainty in depth and features at these fixed equidistant indices , and diagonal covariance is assumed due to the large state size of the images . Image measurements are assumed to come from projective depth cameras , which similarly store 1 channel for depth and n for encoded features . We also assume incremental agent pose measurements ut ∈ R6 with covariance Σut ∈ R6×6 are available , in the form of a translation and rotation vector . The algorithm overview is presented in Algorithm 1 . Finally , the update step takes our state prediction µ̄t , Σ̄t and state observation µ̂t , Σ̂t , and fuses them to produce our new state belief µt , Σt . We spend the remainder of this section explaining the form of the constituent functions . All functions in Algorithm 1 involve re-projections across different image frames , using forward warping . Functions fm , Fm , fo and Fo are therefore all built using the same core functions . While the re-projections could be solved using a typical rendering pipeline of mesh construction followed by rasterization , we instead choose a simpler approach and directly quantize the pixel projections with variance-based image smoothing to fill in quantization holes . An overview of the projection and quantization operations for a single ESM update step is shown in Fig . 1 . 3.1 FORWARD WARPING . Forward warping projects ordered equidistant homogeneous pixel co-ordinates pcf1 from frame f1 to non-ordered non-equidistant homogeneous pixel co-ordinates p̃cf2 in frame f2 . We use µ̃f2 = { φ̃f2 , θ̃f2 , d̃f2 , ẽf2 } to denote the loss of ordering following projection from µf1 = { φf1 , θf2 , df1 , ef2 } , where φ , θ , d and e represent polar angles , azimuthal angles , depth and encoded features respectively . We only consider warping from projective to omni cameras , which corresponds to functions fo , Fo , but the omni-to-omni case as in fm , Fm is identical except with the inclusion of another polar co-ordinate transformation . The encoded features are assumed constant during projection ẽf2 = ef1 . For depth , we must transform the values to the new frame in polar co-ordinates , which is a composition of a linear transformation and non-linear polar conversion . Using the camera intrinsic matrix K1 , the full projection is composed of a scalar multiplication with homogeneous pixel co-ordinates pcf1 , transformation by camera inverse matrix K−11 and frame-to-frame T12 matrices , and polar conversion fp : { φ̃f2 , θ̃f2 , d̃f2 } = fp ( T12K−11 [ pcf1 df1 ] ) ( 1 ) Combined , this provides us with both the forward warped image µ̃f2 = { φ̃f2 , θ̃f2 , d̃f2 , ẽf2 } , and the newly projected homogeneous pixel co-ordinates p̃cf2 = { kpprφ̃f2 , kppr θ̃f2 , 1 } , where kppr denotes the pixels-per-radian resolution constant . The variances are also projected using the full analytic Jacobians , which are efficiently implemented as tensor operations , avoiding costly autograd usage . ˆ̃Σ2 = JV V1J T V + JPP12J T P ( 2 ) 3.2 QUANTIZATION , FUSION AND SMOOTHING . Following projection , we first quantize the floating point pixel coordinates p̃cf2 into integer pixel co-ordinates pcf2 . This in general leads to quantization holes and duplicates . The duplicates are handled with a variance conditioned depth buffer , such that the closest projected depth is used , provided that it ’ s variance is lower than a set threshold . This in general prevents highly uncertain close depth values from overwriting highly certain far values . We then perform per pixel fusion based on lines 6 and 7 in Algorithm 1 provided the depths fall within a set relative threshold , otherwise the minimum depth with sufficiently low variance is taken . This again acts as a depth buffer . Finally , we perform variance based image smoothing , whereby we treat each N ×N image patch ( µk , l ) k∈ { 1 , .. , N } , l∈ { 1 , .. , N } as a collection of independent measurements of the central pixel , and combine their variance values based on central limit theory , resulting in smoothed values for each pixel in the image µi , j . Although we use this to update the mean belief , we do not smooth the variance values , meaning projection holes remain at prior variance . This prevents the smoothing from distorting our belief during subsequent projections , and makes the smoothing inherently local to the current frame only . The smoothing formula is as follows , with variance here denoted as σ2 : µi , j = ∑ k ∑ l µk , l · σ −2 k , l∑ k ∑ l σ −2 k , l ( 3 ) Given that the quantization is a discrete operation , we can not compute it ’ s analytic jacobian for uncertainty propagation . We therefore approximate the added quantization uncertainty using the numerical pixel gradients of the newly smoothed image Gi , j , and assume additive noise proportional to the x and y quantization distances ∆pci , j : Σi , j = Σ̃i , j +Gi , j∆pci , j ( 4 )

Summary: The paper considers the problem of creating spatial memory representations, which play important roles in robotics and are crucial for real-world applications of intelligent agents. The paper proposes an ego-centric representation that stores depth values and features at each pixel in a panorama. Given the relative pose between frames, the representation from the previous frame is transformed via forward warping (using known depth values) to the viewpoint of the current frame. The proposed approach has no learnable parameters. Experiments on a wide range of tasks show that the proposed approach outperforms baselines such as LSTM and NTM.