Environment Predictive Coding for Visual Navigation

1 INTRODUCTION . In visual navigation , an intelligent embodied agent must move around a 3D environment using its stream of egocentric observations to sense objects and obstacles , typically without the benefit of a pre-computed map . Significant recent progress on this problem can be attributed to the availability of large-scale visually rich 3D datasets ( Chang et al. , 2017 ; Xia et al. , 2018 ; Straub et al. , 2019 ) , and high-quality 3D simulators ( Kolve et al. , 2017 ; Savva et al. , 2019a ; Xia et al. , 2020 ) . End-to-end reinforcement learning ( RL ) has been shown to achieve state-of-the-art navigation performance ( Savva et al. , 2019a ; Wijmans et al. , 2020 ) . However , these approaches suffer from sample inefficiency and incur significant computational cost . Recent approaches try to mitigate these limitations by pre-training image representations offline and transferring them for navigation ( Mirowski et al. , 2016 ; Sax et al. , 2020 ) , or by performing auxiliary tasks and data augmentation in an online fashion during RL policy learning ( Gordon et al. , 2019 ; Kostrikov et al. , 2021 ; Ye et al. , 2020 ; 2021 ) . Current offline representation learning methods are flexible — once learned , the representations can be transferred to improve multiple embodied tasks . However , they are limited to learning image feature extractors ( Gupta et al. , 2017 ; Sax et al. , 2020 ) , or image-level proximity functions ( Savinov et al. , 2018 ; Chaplot et al. , 2020c ; Chang et al. , 2020 ) . Since embodied agents typically operate with limited field-of-view sensors , image representations only encode small parts of the scene in the nearby locality of the agent , and do not consider the broader context from the rest of the environment . We contend that embodied agents must learn higher-level semantic and geometric representations of the larger 3D environment around them , conditioned on their entire history of observations . To that end , we introduce environment predictive coding ( EPC ) , a self-supervised approach to learn environment-level representations that are transferrable to a variety of navigation-oriented tasks . The key idea is to learn an encoding of a 3D environment from a series of egocentric observations so as to be predictive of visual content that the agent has not yet observed . Consider the example in Fig . 1 . An agent has observed the living room , lounge , and the bedroom in an unfamiliar house . The agent ’ s encoding of the observed spaces ( i.e. , the red trajectory ) should be predictive of the visual features at an unseen location ( x , y , θ ) , e.g. , the green viewpoint , and enable inferences like “ there is a small kitchen ” , and “ it contains a sink and an oven ” . Learning such predictive representations can equip an agent with the ability to reason about 3D environments as it starts performing various 1 navigation-oriented tasks . The proposed EPC model aims to learn such representations that capture the natural statistics of real-world environments in a self-supervised fashion , by simply watching video walkthroughs recorded by other agents . To achieve this , we devise a self-supervised masked-zone prediction task in which the model learns environment embeddings in an offline fashion , by watching pre-collected video walkthroughs recorded by other agents navigating in 3D environments ( See Fig . 1 ) . The videos contain RGB-D and odometry . Specifically , we segment each video , into zones of temporally contiguous frames which capture local regions of the 3D environment . Then , we randomly mask out zones , and predict features for the masked zones conditioned on both the unmasked zones ’ views and the masked zones ’ camera poses . Since the overlap in scene content across zones is typically limited , the model needs to reason about the geometry and semantics of the environment to figure out what is missing . Our general strategy can be viewed as a context prediction task in sequential data ( Devlin et al. , 2018 ; Sun et al. , 2019b ; Han et al. , 2019 ) —but , very differently , aimed at learning high-level semantic and geometric representations of 3D environments to aid embodied agents acting in them . Unlike any prior video-based feature learning , our approach learns features conditioned on camera poses , explicitly grounding them in a 3D space ; we demonstrate the impact of this important distinction . Through simulated experiments in photorealistic scenes from Matterport3D ( Chang et al. , 2017 ) and Gibson ( Xia et al. , 2018 ) , we show that transferring the EPC environment-level representations leads to 2 - 6× higher sample-efficiency and up to 57 % better performance compared to only image-level transfer on 4 navigational tasks : room goal navigation , object visitation , flee , and area coverage . Our contributions are : ( 1 ) we propose environment-predictive coding ( EPC ) , a self-supervised approach to represent the underlying 3D environment given the observation sequences of an embodied agent , ( 2 ) we propose the proxy task of masked-zone prediction to learn environment-level representations from video walkthroughs captured by other agents , ( 3 ) we perform extensive experiments on Gibson and Matterport3D to demonstrate that EPC leads to good improvements on multiple navigation-oriented tasks , and study EPC ’ s design choices and noise robustness . 2 RELATED WORK . Self-supervised visual representation learning : Prior work leverages self-supervision to learn image and video representations from large unlabelled datasets . Image representation learning attempt proxy tasks such as inpainting ( Pathak et al. , 2016 ) and instance discrimination ( Oord et al. , 2018 ; Chen et al. , 2020a ; He et al. , 2020 ) , while video representation learning leverages signals such as temporal consistency ( Jayaraman & Grauman , 2015 ; Wei et al. , 2018 ; Kim et al. , 2019 ) and contrastive predictions ( Han et al. , 2019 ; Sun et al. , 2019a ) . VideoBERT ( Sun et al. , 2019a ; b ) jointly learns video and text representations from videos by filling in masked out information . Dense Predictive Coding ( Han et al. , 2019 ; 2020 ) learns video representations that capture the slow-moving semantics in videos . Whereas these methods tackle human activity recognition in videos , we aim to learn geometric and semantic cues in 3D spaces for embodied agents . Unlike the existing video models ( Sun et al. , 2019a ; b ; Han et al. , 2019 ) , which simply infer missing frame features conditioned on time , our approach explicitly grounds its predictions in the spatial structure of 3D environments . Representation learning via auxiliary tasks for RL : Reinforcement learning approaches often suffer from high sample complexity , sparse rewards , and unstable training . Prior work tackles these using auxiliary tasks for learning image-level representations during online RL training ( Mirowski et al. , 2016 ; Gordon et al. , 2019 ; Shen et al. , 2019 ; Ye et al. , 2020 ; 2021 ; Stooke et al. , 2021 ) . In contrast , we learn environment-level representations in an offline fashion from previously captured videos . Recent work also learns state representations via future prediction and implicit models ( Ha & Schmidhuber , 2018 ; Gregor et al. , 2019 ; Hafner et al. , 2019 ; Guo et al. , 2020 ) . In particular , neural rendering approaches achieve impressive reconstructions for arbitrary viewpoints ( Eslami et al. , 2018 ; Kumar et al. , 2018 ) . However , unlike our idea , they focus on pixelwise reconstruction , and their success has been limited to synthetically generated environments like DeepMind Lab ( Beattie et al. , 2016 ) . In contrast to any of the above , we use egocentric videos to learn predictive feature encodings that capture the naturally occurring regularities of photorealistic 3D environments . Scene completion : Prior scene completion work performs pixelwise ( Jayaraman & Grauman , 2018 ; Ramakrishnan et al. , 2019 ) or voxelwise ( Song et al. , 2017 ) reconstruction of 360 panoramas , image inpainting ( Pathak et al. , 2016 ) , and image-level extrapolation of depth and semantics ( Song et al. , 2018 ; Yang et al. , 2019 ) . Recent work on visual navigation extrapolates maps of roomtypes ( Wu et al. , 2019 ; Narasimhan et al. , 2020 ) and occupancy ( Ramakrishnan et al. , 2020 ) . While our approach is also motivated by anticipating unseen elements , we learn to extrapolate in a highdimensional feature space ( rather than pixels , voxels , or semantic categories ) and in a self-supervised manner without relying on human annotations . Further , the proposed model learns from egocentric video sequences captured by other agents without assuming access to detailed 3D reconstructions . Learning representations for visual navigation : Researchers are making steady advances in visual navigation architectures ( Gupta et al. , 2017 ; Chen et al. , 2019 ; Fang et al. , 2019 ; Wijmans et al. , 2020 ; Chaplot et al. , 2020b ; Lenton et al. , 2020 ) . Prior work further improves navigation by pretraining representations using supervised image annotations ( Gupta et al. , 2017 ; Anderson et al. , 2018b ; Chen et al. , 2019 ; Sax et al. , 2020 ; Chaplot et al. , 2020c ) , mined object relations ( Yang et al. , 2018 ) , unannotated videos ( Chang et al. , 2020 ) , and active exploration ( Du et al. , 2021 ) . However , these approaches are limited to learning image-level functions . In contrast , we learn environmentlevel representations of image sequences conditioned on the camera poses . We show that the two are complementary by augmenting a SoTA navigation architecture from Fang et al . ( 2019 ) . 3 APPROACH . We propose environment predictive coding ( EPC ) to learn environment-level representations via selfsupervision on video walkthroughs ( Sec . 3.1 ) . To demonstrate the utility of these representations , we integrate them into a transformer-based architecture and refine them for individual navigation tasks ( Sec . 3.2 ) . Finally , we describe our procedure for generating video walkthroughs ( Sec . 3.3 ) . 3.1 ENVIRONMENT PREDICTIVE CODING . Our hypothesis is that it is valuable for an embodied agent to learn a predictive coding of the environment . The agent must not just encode the individual views it observes , but also learn to leverage the encoded information to anticipate the unseen parts of the environment . We propose to train an encoder-decoder model that observes a subset of views from a video walkthrough in a 3D environment , and then infers the features of unobserved views conditioned on their camera poses . To successfully infer features from unobserved views , the encoder must build a predictive representation of the underlying physical environment using the observed views . By transferring this encoder to a navigation agent , we equip the agent with the structural and semantic priors of 3D environments to quickly perform new tasks in new spaces , like mapping the house or room goal navigation . We propose the self-supervised task of masked-zone prediction to achieve this goal ( see Fig . 2 ) . For this task , we use a dataset of egocentric video walkthroughs containing ( possibly noisy ) RGBD and odometer sensor readings collected by other agents deployed in various unseen simulated environments ( Fig . 2 , top ) . These environments are inaccessible for interactive RL training , and the agent policies are ignorant of our eventual downstream tasks ( see Sec . 3.3 ) Our method works as follows . First , we automatically segment each video into “ zones ” which contain temporally contiguous sets of frames . We then learn an environment encoder via the self-supervised maskedzone prediction task on the segmented videos . Finally , we transfer the learned environment encoder to an array of downstream navigation-oriented tasks . We explain each step in detail next . Zone generation At a glance , one might first consider masking arbitrary individual frames in the training videos . However , doing so can result in poor representation learning since shared content from nearby unmasked frames can make the prediction task trivial . Instead , our approach masks zones of frames at once . We define a zone to be a set of temporally contiguous frames in the video . By choosing a large-enough temporal window , we can reduce the amount of shared content with temporally adjacent zones . Given a video walkthrough of size L , we divide it into zones { Z0 , Z1 , · · · } of length N ( selected through validation ) : Zi = { ( ot , pt ) | ∀t ∈ [ ts , te ] } , ( 1 ) where ts = i×N , te = min ( ( i+ 1 ) ×N , L ) , ot is the RGB-D sensor reading , and pt is the camera pose obtained by accumulating odometer readings from time 0 to t ( see Fig . 2 , bottom left ) . While two zones may share visual content , we find that this simple approach works better than strictly limiting the overlap between zones ( see Appendix . A8 ) . Thus , the learning is guided by predicting parts of the environment that were never seen as well as those seen from different viewpoints . Masked-zone prediction Having segmented the video into zones , we next present our EPC masked-zone prediction task to learn environment embeddings ( see Fig . 2 ) . The main idea is to infer unseen zones in a video by previewing the context spanning multiple seen zones . We randomly divide the zones into seen zones { Zsi } ni=1 and unseen zones { Zui } mi=1 . Given the seen zones and the mean camera pose from an unseen zone pui , we need to infer a feature encoding of the unseen zone Zui . To perform this task efficiently , we first extract visual features xt from each RGB-D frame ot in the video using pretrained CNNs ( described in Sec . 3.2 ) . These features are concatenated with the corresponding pose pt and projected using an MLPM to obtain the image-level embedding . The target features for the unseen zone Zui are obtained by averaging 1 all the MLP projected features : fui = 1 |Zui | ∑ ∀x ∈ Zui M ( [ x , −→0 ] ) , ( 2 ) where we mask out the pose ( i.e. , p = −→ 0 ) in the target to avoid trivial solutions . We use a transformer encoder-decoder model ( Vaswani et al. , 2017 ) to infer the zone features ( see Fig . 2 , bottom ) . An environment encoder uses self-attention over the image-level embeddings from all the seen zones , i.e. , { M ( [ x , p ] ) | ∀ ( x , p ) ∈ Zsi , ∀i ∈ [ 1 , n ] } , to generate the environment embeddings E . A zone decoder then attends to E conditioned on the camera pose pui from the unseen zone and predicts the zone features : f̂ui = ZoneDecoder ( E , pui ) . ( 3 ) Following Fang et al . ( 2019 ) , we transform all poses in the input zones relative to pui before encoding , which provides the model an egocentric view of the world . As we will show in experiments , conditioning on pose is critical to learn useful representations . The environment encoder , zone decoder , and projection functionM are trained end-to-end using noise-contrastive estimation ( Gutmann & Hyvärinen , 2010 ) . We use f̂ui as the anchor and f u i from Eqn . 2 as the positive . We sample 1We found this to be better than randomly sampling features within a zone . negatives from other unseen zones in the same video2 and from all zones in other videos . The former discourages the model from simply learning video-specific textures and patterns ; the latter expands the pool of negatives which was found to be beneficial in prior work ( Chen et al. , 2020b ; a ) . The loss for the ith unseen zone in the video is : Li = −log sim ( f̂ u i , f u i ) ∑ j sim ( f̂ui , fj ) + ∑ k sim ( f̂ui , f ′ k ) , ( 4 ) where f ′ k are zone features from other videos , and sim ( q , k ) = exp ( q·k |q||k| 1 τ ) is a similarity measure with temperature τ = 0.1 . The idea is to predict zone representations that are closer to the ground truth , while being sufficiently different from the negative zones . Since the seen and unseen zones may only have limited overlap , the model needs to effectively reason about the geometric and semantic context in the seen zones to perform this task . We qualitatively analyse the masked-zone prediction results from the learned EPC model in Fig . 4 .

Summary: This paper focuses on the problem of efficiently training an intelligent embodied agent for navigation purposes. Technically, the manuscript introduces a technique named as Environment-Predictive Coding (EPC). It is a model for learning environment-level representations via self-supervision, just using video walkthroughs with masked zones.  EPC proposes to learn an environment encoder, using a transformer-based encoding-decoding, that produces environment embeddings and that predicts feature encodings for the unseen areas. This is the main scientific contribution of the paper. This environment encoder is finally integrated in the Scene Memory Transformer (SMT) (Fang et al., 2019) for navigation purposes. A thorough experimental evaluation is offered, using Matterport3D (MP3D) (Chang et al., 2017) and Gibson (Xia et al., 2018) for 4 navigation tasks. Reported results seem to explain the benefits of incorporating such a self-supervision mechanism.