Environment Predictive Coding for Embodied Agents

1 INTRODUCTION . In visual navigation tasks , an intelligent embodied agent must move around a 3D environment using its stream of egocentric observations to sense objects and obstacles , typically without the benefit of a pre-computed map . Significant recent progress on this problem can be attributed to the availability of large-scale visually rich 3D datasets ( Chang et al. , 2017 ; Xia et al. , 2018 ; Straub et al. , 2019 ) , developments in high-quality 3D simulators ( Anderson et al. , 2018b ; Kolve et al. , 2017 ; Savva et al. , 2019a ; Xia et al. , 2020 ) , and research on deep memory-based architectures that combine geometry and semantics for learning representations of the 3D world ( Gupta et al. , 2017 ; Henriques & Vedaldi , 2018 ; Chen et al. , 2019 ; Fang et al. , 2019 ; Chaplot et al. , 2020b ; c ) . Deep reinforcement learning approaches to visual navigation often suffer from sample inefficiency , overfitting , and instability in training . Recent contributions work towards overcoming these limitations for various navigation and planning tasks . The key ingredients are learning good image-level representations ( Das et al. , 2018 ; Gordon et al. , 2019 ; Lin et al. , 2019 ; Sax et al. , 2020 ) , and using modular architectures that combine high-level reasoning , planning , and low-level navigation ( Gupta et al. , 2017 ; Chaplot et al. , 2020b ; Gan et al. , 2019 ; Ramakrishnan et al. , 2020a ) . Prior work uses supervised image annotations ( Mirowski et al. , 2016 ; Das et al. , 2018 ; Sax et al. , 2020 ) and self-supervision ( Gordon et al. , 2019 ; Lin et al. , 2019 ) to learn good image representations that are transferrable and improve sample efficiency for embodied tasks . While promising , such learned image representations only encode the scene in the nearby locality . However , embodied agents also need higher-level semantic and geometric representations of their history of observations , grounded in 3D space , in order to reason about the larger environment around them . Therefore , a key question remains : how should an agent moving through a visually rich 3D environment encode its series of egocentric observations ? Prior navigation methods build environment-level representations of observation sequences via memory models such as recurrent neural networks ( Wijmans et al. , 2020 ) , maps ( Henriques & Vedaldi , 2018 ; Chen et al. , 2019 ; Chaplot et al. , 2020b ) , episodic memory ( Fang et al. , 2019 ) , and topological graphs ( Savinov et al. , 2018 ; Chaplot et al. , 2020c ) . However , these approaches typically use hand-coded representations such as occupancy maps ( Chen et al. , 2019 ; Chaplot et al. , 2020b ; Ramakrishnan et al. , 2020a ; Karkus et al. , 2019 ; Gan et al. , 2019 ) and semantic labels ( Narasimhan et al. , 2020 ; Chaplot et al. , 2020a ) , or specialize them by learning end-to-end for solving a specific task ( Wijmans et al. , 2020 ; Henriques & Vedaldi , 2018 ; Parisotto & Salakhutdinov , 2018 ; Cheng et al. , 2018 ; Fang et al. , 2019 ) . 1 In this work , we introduce environment predictive coding ( EPC ) , a self-supervised approach to learn flexible representations of 3D environments that are transferrable to a variety of navigation-oriented tasks . The key idea is to learn to encode a series of egocentric observations in a 3D environment so as to be predictive of visual content that the agent has not yet observed . For example , consider an agent that just entered the living room in an unfamiliar house and is searching for a refrigerator . It must be able to predict where the kitchen is and reason that it is likely to contain a refrigerator . The proposed EPC model aims to learn representations that capture these natural statistics of real-world environments in a self-supervised fashion , by watching videos recorded by other agents . See Fig . 1 . To this end , we devise a self-supervised zone prediction task in which the model learns environment embeddings by watching egocentric view sequences from other agents navigating in 3D environments in pre-collected videos . Specifically , we segment each video into zones of visually and geometrically connected views , while ensuring limited overlap across zones in the same video . Then , we randomly mask out zones , and predict the masked views conditioned on both the unmasked zones ’ views and the masked zones ’ camera poses . Intuitively , to perform this task successfully , the model needs to reason about the geometry and semantics of the environment to figure out what is missing . We devise a transformer-based model to infer the masked visual features . Our general strategy can be viewed as a context prediction task in sequential data ( Devlin et al. , 2018 ; Sun et al. , 2019b ; Han et al. , 2019 ) —but , very differently , aimed at representing high-level semantic and geometric priors in 3D environments to aid embodied agents who act in them . Through extensive experiments on Gibson and Matterport3D , we show that our method achieves good improvements on multiple navigation-oriented tasks compared to state-of-the-art models and baselines that learn image-level embeddings . 2 RELATED WORK . Self-supervised visual representation learning : Prior work leverages self-supervision to learn image and video representations from large collections of unlabelled data . Image representations attempt proxy tasks such as inpainting ( Pathak et al. , 2016 ) and instance discrimination ( Oord et al. , 2018 ; Chen et al. , 2020 ; He et al. , 2020 ) , while video representation learning leverages signals such as temporal consistency ( Wei et al. , 2018 ; Fernando et al. , 2017 ; Kim et al. , 2019 ) and contrastive predictions ( Han et al. , 2019 ; Sun et al. , 2019a ) . The VideoBERT project ( Sun et al. , 2019a ; b ) jointly learns video and text representations from unannotated videos via filling in masked out information . Dense Predictive Coding ( Han et al. , 2019 ; 2020 ) learns video representations that capture the slow-moving semantics in videos . Whereas these methods focus on capturing human activity for video recognition , we aim to learn geometric and semantic cues in 3D spaces for embodied agents . Accordingly , unlike the existing video models ( Sun et al. , 2019a ; b ; Han et al. , 2019 ) , our approach is grounded in the 3D relationships between views . Representation learning via auxiliary tasks for RL : Reinforcement learning approaches often suffer from high sample complexity , sparse rewards , and unstable training . Prior work tackles these challenges by using auxiliary tasks for learning image representations ( Mirowski et al. , 2016 ; Gordon et al. , 2019 ; Lin et al. , 2019 ; Shen et al. , 2019 ; Ye et al. , 2020 ) . In contrast , we encode image sequences from embodied agents to obtain environment-level representations . Recent work also learns state representations via future prediction and implicit models ( Ha & Schmidhuber , 2018 ; Eslami et al. , 2018 ; Gregor et al. , 2019 ; Hafner et al. , 2019 ; Guo et al. , 2020 ) . In particular , neural rendering approaches achieve impressive reconstructions for arbitrary viewpoints ( Eslami et al. , 2018 ; Kumar et al. , 2018 ) . However , unlike our idea , they focus on pixelwise reconstruction , and their success has been limited to synthetically generated environments like DeepMind Lab ( Beattie et al. , 2016 ) . In contrast to any of the above , we use egocentric videos to learn predictive feature encodings of photorealistic 3D environments to capture their naturally occurring regularities . Scene completion : Past work in scene completion performs pixelwise reconstruction of 360 panoramas ( Jayaraman & Grauman , 2018 ; Ramakrishnan et al. , 2019 ) , image inpainting ( Pathak et al. , 2016 ) , voxelwise reconstructions of 3D structures and semantics ( Song et al. , 2017 ) , and imagelevel extrapolation of depth and semantics ( Song et al. , 2018 ; Yang et al. , 2019b ) . Recent work on visual navigation extrapolates maps of room-types ( Wu et al. , 2019 ; Narasimhan et al. , 2020 ) and occupancy ( Ramakrishnan et al. , 2020a ) . While our approach is also motivated by anticipating unseen elements , we learn to extrapolate in a high-dimensional feature space ( rather than pixels , voxels , or semantic categories ) and in a self-supervised manner without relying on human annotations . Further , the proposed model learns from egocentric video sequences captured by other agents , without assuming access to detailed scans of the full 3D environment as in past work . Learning image representations for navigation : Prior work exploits ImageNet pretraining ( Gupta et al. , 2017 ; Anderson et al. , 2018a ; Chen et al. , 2019 ) , mined object relations ( Yang et al. , 2019a ) , video ( Chang et al. , 2020 ) , and annotated datasets from various image tasks ( Sax et al. , 2020 ; Chaplot et al. , 2020c ) to aid navigation . While these methods also consider representation learning in the context of navigation tasks , they are limited to learning image-level functions for classification and proximity prediction . In contrast , we learn predictive representations for sequences of observations conditioned on the camera poses . 3 APPROACH . We propose environment predictive coding ( EPC ) to learn self-supervised environment-level representations ( Sec . 3.1 ) . To demonstrate the utility of these representations , we integrate them into a transformer-based navigation architecture and refine them for individual tasks ( Sec . 3.2 ) . As we will show in Sec . 4 , our approach leads to both better performance and better sample efficiency compared to existing approaches . 3.1 ENVIRONMENT PREDICTIVE CODING . Our hypothesis is that it is valuable for an embodied agent to learn a predictive coding of the environment . The agent must not just encode the individual views it observes , but also learn to leverage the encoded information to anticipate the unseen parts of the environment . Our key idea is that the environment embedding must be predictive of unobserved content , conditioned on the agent ’ s camera pose . This equips an agent with the natural priors of 3D environments to quickly perform new tasks , like finding the refrigerator or covering more area . We propose the proxy task of zone prediction to achieve this goal ( see Fig . 2 ) . For this task , we use a dataset of egocentric video walkthroughs collected parallely from other agents deployed in various unseen environments ( Fig . 2 , top ) . For each video , we assume access to RGB-D , egomotion data , and camera intrinsics . Specifically , our current implementation uses egocentric camera trajectories from photorealistic scanned indoor environments ( Gibson ( Xia et al. , 2018 ) ) to sample the training videos ; we leave leveraging in-the-wild consumer video as a challenge for future work . We do not assume that the agents who generated those training videos were acting to address a particular navigation task . In particular , their behavior need not be tied to the downstream navigationoriented tasks for which we test our learned representation . For example , a training video may show agents moving about to maximize their area coverage , whereas the encoder we learn is applicable to an array of navigation tasks ( as we will demonstrate in Sec . 4 ) . Furthermore , we assume that the environments seen in the videos are not accessible for interactive training . In practice , this means that we can parallelly collect data from different robots deployed in a large number of environments , without having to actually train our navigation policy on those environments . These assumptions are much weaker than those made by prior work on imitation learning and behavioral cloning that rely on task-specific data generated from experts ( Bojarski et al. , 2016 ; Giusti et al. , 2016 ) . Our method works as follows . First , we automatically segment videos into “ zones ” which contain frames with significant view overlaps . We then perform the self-supervised zone prediction task on the segmented videos . Finally , we incorporate the learned environment encoder into an array of downstream navigation-oriented tasks . We explain each step in detail next . Zone generation At a glance , one might first consider masking arbitrary individual frames in the training videos . However , doing so is inadequate for representation learning , since unmasked frames having high viewpoint overlap with the masked frame can make its prediction trivial . Instead , our approach masks zones of frames at once . We define a zone to be a set of frames in the video which share a significant overlap in their viewpoints . We also require that the frames across multiple zones share little to no overlap . To generate these zones , we first cluster frames in the videos based on the amount of pairwisegeometric overlap between views . We estimate the viewpoint overlap ψ ( oi , oj ) between two frames oi , oj by measuring their intersection in 3D point clouds obtained by backprojecting depth inputs into 3D space . See Appendix for more details . For a video of length L , we generate a distance matrix D ∈ RL×L where Di , j = 1 − ψ ( oi , oj ) . We then perform hierarchical agglomerative clustering ( Lukasová , 1979 ) to cluster the video frames into zones based on D ( see Fig . 2 , bottom left ) . While these zones naturally tend to overlap near their edges , they typically capture disjoint sets of content in the video . Note that the zones segment video trajectories , not floorplan maps , since we do not assume access to the full 3D environment . Zone prediction task Having segmented the video into zones , we next present our EPC zone prediction task to learn environment embeddings ( see Fig . 2 ) . We randomly divide the video v into seen zones { Zvs , i } ni=1 ( cyan ) and unseen zones { Zvu , i } mi=1 ( yellow ) , where a zone Z is a tuple of images and the corresponding camera poses Zi = { ( oj , pj ) } |Zi|1 . Given the seen zones , and the camera pose from an unseen zone pvu , i , we need to infer a feature encoding of the unseen zone Z v u , i . To perform this task , we first extract visual features x from each RGB-D frame o in the video using pretrained CNNs ( see Sec . 3.2 ) . These features are concatenated with the corresponding pose p and projected using an MLPM to obtain the image-level embedding . The target features for the unseen zone Zvu , i are obtained as follows : fvu , i = 1 |Zvu , i| ∑ [ x , p ] ∈Zvu , i M ( [ x , p ] ) . ( 1 ) The rationale behind the feature averaging is that we want to predict the high-level visual content of the zone , while ignoring viewpoint specific variations within the zone . We use a transformer-based encoder-decoder model to perform this task ( Vaswani et al. , 2017 ) . Our model consists of an environment encoder and a zone decoder which infers the zone features ( see Fig . 2 , bottom ) . The environment encoder takes in the image-level embeddingsM ( [ x , p ] ) from the input zones , and performs multi-headed self-attention to generate the environment embeddings E . The zone decoder attends to E using the average camera pose from the unseen zone pvu , i and predicts the zone features as follows : f̂u , i = ZoneDecoder ( E , pvu , i ) . ( 2 ) We transform all poses in the input zones relative to pvu , i before encoding , which provides the model an egocentric view of the world . The environment encoder , zone decoder , and the projection function M are jointly trained using noise-contrastive estimation ( Gutmann & Hyvärinen , 2010 ) . We use f̂u , i as the anchor and fvu , i from Eqn . 1 as the positive . We sample negatives from other unseen zones in the same video and from all zones in other videos . The loss for the ith unseen zone in video v is : Lvi = −log exp ( sim ( f̂u , i , fvu , i ) ) m∑ j=1 exp ( sim ( f̂u , i , fvu , j ) ) + ∑ w 6=v , k exp ( sim ( f̂u , i , fwk ) ) , ( 3 ) where sim ( q , k ) = q·k|q||k| 1 τ and τ is a temperature hyperparameter . The idea is to predict zone representations that are closer to the ground truth , while being sufficiently different from the negative zones . Since the unseen zones have only limited overlap with the seen zones , the model needs to effectively reason about the geometric and semantic context in the seen zones to differentiate the positive from the negatives . We discourage the model from simply capturing video-specific textures and patterns by sampling negatives from within the same video .

Summary: The paper proposes a self-supervised approach for learning environment-level representations for embodied agents. The idea is that agents collect images and their corresponding poses during a walk-through phase. The images are clustered into multiple "zones". The zones are divided into seen and unseen zones. Using contrastive learning, the model is trained to distinguish the features of an unseen zone from the rest of the zones. The paper shows this approach improves performance over a number of baselines for Area Coverage, Flee, and Object Coverage tasks.