Environment Predictive Coding for Visual Navigation

1 INTRODUCTION . In visual navigation , an intelligent embodied agent must move around a 3D environment using its stream of egocentric observations to sense objects and obstacles , typically without the benefit of a pre-computed map . Significant recent progress on this problem can be attributed to the availability of large-scale visually rich 3D datasets ( Chang et al. , 2017 ; Xia et al. , 2018 ; Straub et al. , 2019 ) , and high-quality 3D simulators ( Kolve et al. , 2017 ; Savva et al. , 2019a ; Xia et al. , 2020 ) . End-to-end reinforcement learning ( RL ) has been shown to achieve state-of-the-art navigation performance ( Savva et al. , 2019a ; Wijmans et al. , 2020 ) . However , these approaches suffer from sample inefficiency and incur significant computational cost . Recent approaches try to mitigate these limitations by pre-training image representations offline and transferring them for navigation ( Mirowski et al. , 2016 ; Sax et al. , 2020 ) , or by performing auxiliary tasks and data augmentation in an online fashion during RL policy learning ( Gordon et al. , 2019 ; Kostrikov et al. , 2021 ; Ye et al. , 2020 ; 2021 ) . Current offline representation learning methods are flexible — once learned , the representations can be transferred to improve multiple embodied tasks . However , they are limited to learning image feature extractors ( Gupta et al. , 2017 ; Sax et al. , 2020 ) , or image-level proximity functions ( Savinov et al. , 2018 ; Chaplot et al. , 2020c ; Chang et al. , 2020 ) . Since embodied agents typically operate with limited field-of-view sensors , image representations only encode small parts of the scene in the nearby locality of the agent , and do not consider the broader context from the rest of the environment . We contend that embodied agents must learn higher-level semantic and geometric representations of the larger 3D environment around them , conditioned on their entire history of observations . To that end , we introduce environment predictive coding ( EPC ) , a self-supervised approach to learn environment-level representations that are transferrable to a variety of navigation-oriented tasks . The key idea is to learn an encoding of a 3D environment from a series of egocentric observations so as to be predictive of visual content that the agent has not yet observed . Consider the example in Fig . 1 . An agent has observed the living room , lounge , and the bedroom in an unfamiliar house . The agent ’ s encoding of the observed spaces ( i.e. , the red trajectory ) should be predictive of the visual features at an unseen location ( x , y , θ ) , e.g. , the green viewpoint , and enable inferences like “ there is a small kitchen ” , and “ it contains a sink and an oven ” . Learning such predictive representations can equip an agent with the ability to reason about 3D environments as it starts performing various 1 navigation-oriented tasks . The proposed EPC model aims to learn such representations that capture the natural statistics of real-world environments in a self-supervised fashion , by simply watching video walkthroughs recorded by other agents . To achieve this , we devise a self-supervised masked-zone prediction task in which the model learns environment embeddings in an offline fashion , by watching pre-collected video walkthroughs recorded by other agents navigating in 3D environments ( See Fig . 1 ) . The videos contain RGB-D and odometry . Specifically , we segment each video , into zones of temporally contiguous frames which capture local regions of the 3D environment . Then , we randomly mask out zones , and predict features for the masked zones conditioned on both the unmasked zones ’ views and the masked zones ’ camera poses . Since the overlap in scene content across zones is typically limited , the model needs to reason about the geometry and semantics of the environment to figure out what is missing . Our general strategy can be viewed as a context prediction task in sequential data ( Devlin et al. , 2018 ; Sun et al. , 2019b ; Han et al. , 2019 ) —but , very differently , aimed at learning high-level semantic and geometric representations of 3D environments to aid embodied agents acting in them . Unlike any prior video-based feature learning , our approach learns features conditioned on camera poses , explicitly grounding them in a 3D space ; we demonstrate the impact of this important distinction . Through simulated experiments in photorealistic scenes from Matterport3D ( Chang et al. , 2017 ) and Gibson ( Xia et al. , 2018 ) , we show that transferring the EPC environment-level representations leads to 2 - 6× higher sample-efficiency and up to 57 % better performance compared to only image-level transfer on 4 navigational tasks : room goal navigation , object visitation , flee , and area coverage . Our contributions are : ( 1 ) we propose environment-predictive coding ( EPC ) , a self-supervised approach to represent the underlying 3D environment given the observation sequences of an embodied agent , ( 2 ) we propose the proxy task of masked-zone prediction to learn environment-level representations from video walkthroughs captured by other agents , ( 3 ) we perform extensive experiments on Gibson and Matterport3D to demonstrate that EPC leads to good improvements on multiple navigation-oriented tasks , and study EPC ’ s design choices and noise robustness . 2 RELATED WORK . Self-supervised visual representation learning : Prior work leverages self-supervision to learn image and video representations from large unlabelled datasets . Image representation learning attempt proxy tasks such as inpainting ( Pathak et al. , 2016 ) and instance discrimination ( Oord et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ) , while video representation learning leverages signals such as temporal consistency ( Jayaraman & Grauman , 2015 ; Wei et al. , 2018 ; Kim et al. , 2019 ) and contrastive predictions ( Han et al. , 2019 ; Sun et al. , 2019a ) . VideoBERT ( Sun et al. , 2019a ; b ) jointly learns video and text representations from videos by filling in masked out information . Dense Predictive Coding ( Han et al. , 2019 ; 2020 ) learns video representations that capture the slow-moving semantics in videos . Whereas these methods tackle human activity recognition in videos , we aim to learn geometric and semantic cues in 3D spaces for embodied agents . Unlike the existing video models ( Sun et al. , 2019a ; b ; Han et al. , 2019 ) , which simply infer missing frame features conditioned on time , our approach explicitly grounds its predictions in the spatial structure of 3D environments . Representation learning via auxiliary tasks for RL : Reinforcement learning approaches often suffer from high sample complexity , sparse rewards , and unstable training . Prior work tackles these using auxiliary tasks for learning image-level representations during online RL training ( Mirowski et al. , 2016 ; Gordon et al. , 2019 ; Shen et al. , 2019 ; Ye et al. , 2020 ; 2021 ; Stooke et al. , 2021 ) . In contrast , we learn environment-level representations in an offline fashion from previously captured videos . Recent work also learns state representations via future prediction and implicit models ( Ha & Schmidhuber , 2018 ; Gregor et al. , 2019 ; Hafner et al. , 2019 ; Guo et al. , 2020 ) . In particular , neural rendering approaches achieve impressive reconstructions for arbitrary viewpoints ( Eslami et al. , 2018 ; Kumar et al. , 2018 ) . However , unlike our idea , they focus on pixelwise reconstruction , and their success has been limited to synthetically generated environments like DeepMind Lab ( Beattie et al. , 2016 ) . In contrast to any of the above , we use egocentric videos to learn predictive feature encodings that capture the naturally occurring regularities of photorealistic 3D environments . Scene completion : Prior scene completion work performs pixelwise ( Jayaraman & Grauman , 2018 ; Ramakrishnan et al. , 2019 ) or voxelwise ( Song et al. , 2017 ) reconstruction of 360 panoramas , image inpainting ( Pathak et al. , 2016 ) , and image-level extrapolation of depth and semantics ( Song et al. , 2018 ; Yang et al. , 2019 ) . Recent work on visual navigation extrapolates maps of roomtypes ( Wu et al. , 2019 ; Narasimhan et al. , 2020 ) and occupancy ( Ramakrishnan et al. , 2020 ) . While our approach is also motivated by anticipating unseen elements , we learn to extrapolate in a highdimensional feature space ( rather than pixels , voxels , or semantic categories ) and in a self-supervised manner without relying on human annotations . Further , the proposed model learns from egocentric video sequences captured by other agents without assuming access to detailed 3D reconstructions . Learning representations for visual navigation : Researchers are making steady advances in visual navigation architectures ( Gupta et al. , 2017 ; Chen et al. , 2019 ; Fang et al. , 2019 ; Wijmans et al. , 2020 ; Chaplot et al. , 2020b ; Lenton et al. , 2020 ) . Prior work further improves navigation by pretraining representations using supervised image annotations ( Gupta et al. , 2017 ; Anderson et al. , 2018b ; Chen et al. , 2019 ; Sax et al. , 2020 ; Chaplot et al. , 2020c ) , mined object relations ( Yang et al. , 2018 ) , unannotated videos ( Chang et al. , 2020 ) , and active exploration ( Du et al. , 2021 ) . However , these approaches are limited to learning image-level functions . In contrast , we learn environmentlevel representations of image sequences conditioned on the camera poses . We show that the two are complementary by augmenting a SoTA navigation architecture from Fang et al . ( 2019 ) . 3 APPROACH . We propose environment predictive coding ( EPC ) to learn environment-level representations via selfsupervision on video walkthroughs ( Sec . 3.1 ) . To demonstrate the utility of these representations , we integrate them into a transformer-based architecture and refine them for individual navigation tasks ( Sec . 3.2 ) . Finally , we describe our procedure for generating video walkthroughs ( Sec . 3.3 ) . 3.1 ENVIRONMENT PREDICTIVE CODING . Our hypothesis is that it is valuable for an embodied agent to learn a predictive coding of the environment . The agent must not just encode the individual views it observes , but also learn to leverage the encoded information to anticipate the unseen parts of the environment . We propose to train an encoder-decoder model that observes a subset of views from a video walkthrough in a 3D environment , and then infers the features of unobserved views conditioned on their camera poses . To successfully infer features from unobserved views , the encoder must build a predictive representation of the underlying physical environment using the observed views . By transferring this encoder to a navigation agent , we equip the agent with the structural and semantic priors of 3D environments to quickly perform new tasks in new spaces , like mapping the house or room goal navigation . We propose the self-supervised task of masked-zone prediction to achieve this goal ( see Fig . 2 ) . For this task , we use a dataset of egocentric video walkthroughs containing ( possibly noisy ) RGBD and odometer sensor readings collected by other agents deployed in various unseen simulated environments ( Fig . 2 , top ) . These environments are inaccessible for interactive RL training , and the agent policies are ignorant of our eventual downstream tasks ( see Sec . 3.3 ) Our method works as follows . First , we automatically segment each video into “ zones ” which contain temporally contiguous sets of frames . We then learn an environment encoder via the self-supervised maskedzone prediction task on the segmented videos . Finally , we transfer the learned environment encoder to an array of downstream navigation-oriented tasks . We explain each step in detail next . Zone generation At a glance , one might first consider masking arbitrary individual frames in the training videos . However , doing so can result in poor representation learning since shared content from nearby unmasked frames can make the prediction task trivial . Instead , our approach masks zones of frames at once . We define a zone to be a set of temporally contiguous frames in the video . By choosing a large-enough temporal window , we can reduce the amount of shared content with temporally adjacent zones . Given a video walkthrough of size L , we divide it into zones { Z0 , Z1 , · · · } of length N ( selected through validation ) : Zi = { ( ot , pt ) | ∀t ∈ [ ts , te ] } , ( 1 ) where ts = i×N , te = min ( ( i+ 1 ) ×N , L ) , ot is the RGB-D sensor reading , and pt is the camera pose obtained by accumulating odometer readings from time 0 to t ( see Fig . 2 , bottom left ) . While two zones may share visual content , we find that this simple approach works better than strictly limiting the overlap between zones ( see Appendix . A8 ) . Thus , the learning is guided by predicting parts of the environment that were never seen as well as those seen from different viewpoints . Masked-zone prediction Having segmented the video into zones , we next present our EPC masked-zone prediction task to learn environment embeddings ( see Fig . 2 ) . The main idea is to infer unseen zones in a video by previewing the context spanning multiple seen zones . We randomly divide the zones into seen zones { Zsi } ni=1 and unseen zones { Zui } mi=1 . Given the seen zones and the mean camera pose from an unseen zone pui , we need to infer a feature encoding of the unseen zone Zui . To perform this task efficiently , we first extract visual features xt from each RGB-D frame ot in the video using pretrained CNNs ( described in Sec . 3.2 ) . These features are concatenated with the corresponding pose pt and projected using an MLPM to obtain the image-level embedding . The target features for the unseen zone Zui are obtained by averaging 1 all the MLP projected features : fui = 1 |Zui | ∑ ∀x ∈ Zui M ( [ x , −→0 ] ) , ( 2 ) where we mask out the pose ( i.e. , p = −→ 0 ) in the target to avoid trivial solutions . We use a transformer encoder-decoder model ( Vaswani et al. , 2017 ) to infer the zone features ( see Fig . 2 , bottom ) . An environment encoder uses self-attention over the image-level embeddings from all the seen zones , i.e. , { M ( [ x , p ] ) | ∀ ( x , p ) ∈ Zsi , ∀i ∈ [ 1 , n ] } , to generate the environment embeddings E . A zone decoder then attends to E conditioned on the camera pose pui from the unseen zone and predicts the zone features : f̂ui = ZoneDecoder ( E , pui ) . ( 3 ) Following Fang et al . ( 2019 ) , we transform all poses in the input zones relative to pui before encoding , which provides the model an egocentric view of the world . As we will show in experiments , conditioning on pose is critical to learn useful representations . The environment encoder , zone decoder , and projection functionM are trained end-to-end using noise-contrastive estimation ( Gutmann & Hyvärinen , 2010 ) . We use f̂ui as the anchor and f u i from Eqn . 2 as the positive . We sample 1We found this to be better than randomly sampling features within a zone . negatives from other unseen zones in the same video2 and from all zones in other videos . The former discourages the model from simply learning video-specific textures and patterns ; the latter expands the pool of negatives which was found to be beneficial in prior work ( Chen et al. , 2020b ; a ) . The loss for the ith unseen zone in the video is : Li = −log sim ( f̂ u i , f u i ) ∑ j sim ( f̂ui , fj ) + ∑ k sim ( f̂ui , f ′ k ) , ( 4 ) where f ′ k are zone features from other videos , and sim ( q , k ) = exp ( q·k |q||k| 1 τ ) is a similarity measure with temperature τ = 0.1 . The idea is to predict zone representations that are closer to the ground truth , while being sufficiently different from the negative zones . Since the seen and unseen zones may only have limited overlap , the model needs to effectively reason about the geometric and semantic context in the seen zones to perform this task . We qualitatively analyse the masked-zone prediction results from the learned EPC model in Fig . 4 .

Summary: The work builds upon the scene memory transformer (SMT, Fang et al., 2019) for visual navigation, which encodes and adds past observations to a memory and uses attention to exploit spatio-temporal dependencies.  The authors propose a self-supervised pre-training for the observation encoder using an encoder-decoder architecture with the encoded memory as the bottleneck (the decoder is only used for pre-training). Self supervision is achieved by using sequences of observations (posed RGBD images) and splitting them into batches of temporally consecutive frames (termed “zones” in the paper). Considering a subset of the zones as not observed, i.e. not available as input for the network, the goal of the encoder-decoder is to predict the features of an unseen zone from the seen input zones, conditioned on the pose of the unseen zone.  Compared to a frame based masking, the zone (=spatio-temporal consecutive frames) based exclusion of data results in less overlap between seen and unseen scene parts, which is argued to learn higher-level semantic and geometric representation of the 3D environment  Contributions: - Approach for self supervision based on the prediction of visual-geometric features for unseen scenes. - Superior performance to SoTA on the navigation task.