Environment Predictive Coding for Visual Navigation
Authors: Santhosh Kumar Ramakrishnan
Tushar Nagarajan
Ziad Al-Halah
Kristen Grauman
Venue: ICLR 2022 Poster (2022)
Keywords: Self-supervised learning
visual navigation
representation learning

Abstract: We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for individual images, we aim to encode a 3D environment using a series of images observed by an agent moving in it. We learn these representations via a masked-zone prediction task, which segments an agent’s trajectory into zones and then predicts features of randomly masked zones, conditioned on the agent’s camera poses. This explicit spatial conditioning encourages learning representations that capture the geometric and semantic regularities of 3D environments. We learn such representations on a collection of video walkthroughs and demonstrate successful transfer to multiple downstream navigation tasks. Our experiments on the real-world scanned 3D environments of Gibson and Matterport3D show that our method obtains 2 - 6× higher sample-efﬁciency and up to 57% higher performance over standard image-representation learning.

Published as a conference paper at ICLR 2022
ENVIRONMENT PREDICTIVE CODING FOR VISUAL NAVIGATION
Santhosh K. Ramakrishnan" :22 Tushar Nagarajan" 1.2 Ziad Al-Halah' Kristen Grauman 1.2
'The University of Texas Austin farama, tushar]8cs utcxas. edu, ziadlhihe 8gmail cem, grauman8 fb.coom
Facebook Al Research
ABSTRACT
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for individual images, we aim to encode 3D en- vironment using a series of images observed by agent moving in it. We learn these representations via masked-zone prediction task, which segments agent's trajectory into zones and then predicts features of randomly masked zones, conditioned on the agent's camera poses. This explicit spatial condition- ing encourages learning representations that capture the geometric and seman- tic regularities of 3D environments. We learn the representations on a collec- tion of video walkthroughs and demonstrate successful transfer multiple down- stream navigation tasks. Our experiments on the real-world scanned 3D envi- ronments of Gibson and Matterport3I show that our method obtains 2 6x higher sample-efficiency and up to 57% higher performance over standard image- representation learning. Code and pre-trained models are publicly available: https:/ hatps://viiionn //viiiiiino.////// cs. utw.as.ee........... edu/pro jerts//e//ppc
INTRODUCTION
In visual navigation, an intelligent embodied agent must move around 3D enviromment using its stream of egocentric observations sense objects and obstacles, typically without the benefit of pre-computed map. Significant recent progress on this problem can be attributed to the availability of lcrge-scale visually rich 3D datasets (Chang et al., 2017; Xia al., 2018; Straub et al., 2019), and high-quality 3D simulators (Kolve et al., 2017; Savva et al., 2019a; Xia et al., 2020).
End-to-end reinforcement learning (RI.) has been shown to achieve state-of-the-an navigation per- formance (Savva et al., 2019a; Wijmans et al., 2020). However, these approaches suffer from sample inefficiency and incur significant computational cost. Recent approaches try to mitigate these limi- tations by pre- pre-training image representations offline and transferring them for navigation (Mirowski et al., 2016; Sax et al., 2020), or by performing auxiliary tasks and data augmentation in an online fashion during RL policy learning (Gordon et al., 2019; Kostrikov et al., 2021; Ye et al., 2020; 2021).
Current offline representation learning methods are flexible once learned, the representations be transferred to improve multiple embodied tasks. However, they are limited to learning image feature extractors (Gupta al., 2017; Sax et al., 2020), or image-level proximity functions (Savinov et al., 2018: 8; Chaplot et al., 2020c; Chang et al., 2020). Since embodied agents typically operate with limited field-of-view sensors, image representations only encode small parts of the scene the nearby locality of the agent, and do not consider the broader context from the rest of the environment. We contend that embodied agents must learn higher-level semantic and geometric representations of the larger 3D environment around them, conditioned on their entire history of observations.
To that end, we introduce environment predictive coding (EPC), self-supervised approach to learn environment-level representations that are transferrable to variety of -orientte-- tasks. The key idea to learn an encoding of a 3D environment from a series of egocentric observations so as to be predictive of visual content that the agent has not observed. Consider the example in Fig. An agent has observed the living room, lounge, and the bedroom in an unfamiliar house. The agent's 's encoding of the observed spaces (i.e., the red trajectory) should be predictive of the visual features at an unseen location (z. y. 8), e.g., the green viewpoint, and enable inferences like "there is small kitchen". and "it contains sink and oven". Learning such predictive representations can equip an agent with the ability to reason about 3D environments as it starts performing various navigation-oriented tasks. The proposed EPC model aims to learn such representations that capture

---

Published as a conference paper at ICLR 2022
Predict visual features (I, y,
?
Figure I: Environment Predictive Coding: During self superviedd learning. our model is given video walk- throughs of various 3D environments. We mask out portions of the trajectory (dotted lines) and learn to infer them from the unmasked parts (in red). The resulting EPC encoder builds environment-ler level representations of the seen content that are predictive of the unseen content (marked with a "?"", conditioned on the camera poses. We then transfer this learned encoder agents performing various navigation tasks in novel environments.
the natural statistics of real-world environments in self-supervised fashion, by simply watching video walkthroughs recorded by other agents.
To achieve this, we devise a self-superviser masked-zone prediction task in which the model learns environment embeddings in an offline fashion, by watching pre-collected video walkthroughs recorded by other agents navigating in 3D environments (See Fig. 1). The videos contain RGB-D and odometry. Specifically, we segment each video, into zones of temporally contiguous frames which capture local regions of the 3D environment. Then, randomly mask zones, and pre- dict features for the masked zones conditioned on both the unmasked zones' views and the masked zones' camera poses. Since the overlap in scene content across zones is typically limited, the model needs to reason about the geometry and semantics of the environment to figure out what is missing.
Our general strategy can be viewed as context prediction task in sequential data (Devlin et al., 2018; Sun et al., 2019b; Han et al., 2019)-but very differently, aimed at learning high-level semantic and geometric representations of 3D environments aid embodied agents acting in them. Unlike any prior video-based feature learning, our approach learns features conditioned on camera poses, explicitly grounding them in 3D space; we demonstrate the impact of this important distinction.
Through simulated experiments in photorealistic scenes from Matterport3D (Chang et al., 2017) and Gibson (Xia et al., 2018), we show that transferring the EPC environment-level representations leads to 2-266 6x higher transfer on 4 navigational tasks: room goal navigation, object visitation, flee, and area coverage.
and up to 57% better performance compared to only image-level
Our contributions are: (1) we propose catrronment-predicctin coding (EPC), self-supervised ap- proach to represent the underlying 3D environment given the observation sequences of an embodied agent, (2) we propose the proxy task of masked-zone prediction to learn environment-leve repre- sentations from video walkthroughs captured by other agents, (3) we perform extensive experiments on Gibson and Matterport3D demonstrate that EPC Ieads to good improvements multiple navigation-oriented tasks, and study EPC's design choices and noise robustness.
2 RELATED WORK
Self-supervised visual representation learning: Prior work leverages self-supervision to learn im- age and video representations from large unlabelled datasets. Image representation learning attempt proxy tasks such as inpainting (Pathak et al., 2016) and instance discrimination (Oord et al., 2018; Chen al., 2020a; He et al., 2020), while video representation learning leverages signals such as temporal consistency (Jayaraman & Grauman, 2015; Wei et al., 2018; Kim et al., 2019) and con- trastive predictions (Han al., 2019; Sun et al., 2019a). VideoBERT (Sun al., 2019a:b) jointly learns video and text representations from videos by filling in masked out information. Dense Pre- dictive Coding (Han et al., 2019; 2020) learns video representations that capture the slow-moving semantics in videos. Whereas these methods tackle human activity recognition in videos, we aim to learn geometric and semantic cues in 3D spaces for embodied agents. Unlike the existing video models (Sun et al., 201 9a;b; Han et al., 2019), which simply infer missing frame features conditioned on time, our approach explicitly grounds its predictions in the spatial structure of 3D environments.
Representation learning via auxiliary tasks for RL: Reinforcement learning approaches often suffer from high sample complexity, sparse rewards, and unstable training. Prior work tackles these using auxiliary tasks for learning image-level representations during online RL training (Mirowski
2

---

Published as a conference paper at ICLR 2022
9
2
Zi
Environment
encoder
A
Zone
similarity
devader
(o.p)
p
Contrastive loss
Figure 2: We propose the masked- zone prediction task for self- supervisd learning of environment embeddings. We learn from video walkthroughs generated by other agents moving under policies ignorant of our eventual downstream tasks. Each frame consists of the egocentric view and camera pose (top left). We group the frames the video into seen zones in cyan { [ Y and generated automatically by grouping temporally contiguous sets of frames (bottom left). Given camera pose p" sampled from the unseen zone 2,", we use a transformer based encoder-decoder architecture that generates environment embeddings from the seen zones, and predicts the feature encoding f conditioned on the pose p" (bottom center). The model is trained to distinguish the positive f? from negatives in the same video UGls # } as well from other videos [f') (bottom right).
in yellow {Z.),. Ji=o (top row). The zones
al., 2016; Gordon et al., 2019; Shen et al., 2019; Ye et al., 2020; 2021; Stooke et al.. 2021). In contrast, we learn environment-leve representations in an offline fashion from previously captured videos. Recent work also learns state representations via future prediction and implicit models (Ha & Schmidhuber, 2018; Gregor et al., 2019; Hafner et al., 2019; Guo et al., 2020). In particular, neural rendering approaches achieve impressive reconstructions for arbitrary viewpoints (Eslami et al., 2018; Kumar et al., 2018). However, unlike our idea, they focus pixelwise reconstruction, and their success has been limited to synthetically generated environments like DeepMind Lab (Beattic al., 2016). In contrast to any of the above, encodings that capture the naturally occurring regularities of photorealistic 3D environments.
egocentric videos to learn predictive feature
Scene completion: Prior scene completion work performs pixelwise (Jayaraman & Grauman, 2018; Ramakrishnan et al., 2019) or voxelwise (Song et al., 2017) 7) reconstruction of 360 panoramas, image inpainting (Pathak al., 2016), and image-level extrapolation of depth and semanties (Song al., 2018; Yang et al., 2019). Pathdreamer furthers extends this to navigation in large-scale 3D scenes by predicting future observations, with the help of supervision from semantic annotations (Koh et al., 2021). Recent work on visual navigation extrapolates maps of room-types (Wu et al., 2019; Narasimhan et al., 2020) and occupancy (Ramakrishnan et al., 2020). While our approach is also motivated by anticipating unseen elements, we learn to extrapolate in a high-dimensional feature space (rather than pixels, voxels, or semantic categories) and self-supervised manner without relying on human annotations. Further, the proposed model learns from egocentric video sequences captured by other agents without assuming access to detailed 3D reconstructions.
making steady advances in vi- sual navigation architectures (Gupta et al., 2017: Chen et al., 2019; Fang et al., 2019; Wijmans et al., 2020; Chaplot et al., 2020b; Lenton al., 2020). Prior work further improves navigation by pre- training representations using supervised image annotations (Gupta et al., 2017; Anderson et al., 201 8b; Chen et al., 2019; Sax et al., 2020; Chaplot et al., 2020c), mined object relations (Yang al., 2018), unannotated videos (Chang et al., 2020), and active exploration (Du et al., 2021). However, limited to learning image-level functions. In contrast, level representations of image sequences conditioned on the camera poses. We show that the two complementary by augmenting a SoTA navigation architecture from Fang et al. (2019).
Learning representations for visual navigation: Researchers
these approaches
learn environment-
APPROACH
We propose environment predictive coding (EPC) to learn environment- representations via self- supervision video walkthroughs (Sec. 3. To demonstrate the utility of these representations, we integrate them into a transformer based architecture and refine them for individual navigation tasks (Sec. 3.2). Finally, describe our procedure for generating video walkthroughs (Sec. 3.3).
3

---

Published as a conference paper at ICLR 2022
3.1 ENVIRONMENT PREDICTIVE CODING
Our hypothesis is that it is valuable for an embodied agent to learn a predictive coding of the envi- ronment. The agent must not just encode the individual views it observes, but also learn leverage the encoded information to anticipate the unseen parts of the environment. We propose to train an encoder-decoder model that observes subset of views from a video walkthrough in a 3D envi- ronment, and then infers the features of unobserved views conditioned on their camera poses. To successfully infer features from unobserved views, the encoder must build a predictive representa- tion of the underlying physical environment using the observed views. By transferring this encoder navigation agent, we equip the agent with the structural and semantic priors of 3D environments to quickly perform new tasks in new spaces, like mapping the house or room goal navigation.
We propose the sule-sup task of masked-zon zone prediction achieve this goal (see Fig. 2). For this task, we use a dataset of egocentric video walkthroughs containing (possibly noisy) RGB- D and odometer sensor readings collected by other agents deployed in various unseen simulated environments (Fig. 2, top). These environments are inaccessible for interactive RL training, and the agent policies are ignorant of our eventual downstream tasks (see Sec. 3.3) Our method works as follows. First, we automatically segment each video into "zones" which contain temporally contiguous sets of frames. We then learn an environment encoder via the self-supervised masked- zone prediction task on the segmented videos. Finally, we transfer the learned environment encoder to an array of downstream navigation-orientel tasks. We explain each step in detail next.
Zone generation Ät glance, one might first consider masking arbitrary individual frames in the training videos. However, doing so can result in poor representation learning since shared content from nearby unmasked frames can make the prediction task trivial. Instead, our approach masks zones of frames at once. We define a zone to be a set of temporally contiguous frames in the video. By choosing large-enough temporal window, we can reduce the amount of shared content with temporally adjacent zones. Given a video walkthrough of size L. we divide it into zones (Zo, ".. of length N (selected through validation):
Z, {(oc. (t) Vt E [ta.
where 1, N, L. min( (i 1) x L), of the RGB-D sensor reading, and pa is the camera pose obtained by accumulating odometer readings from time 0 to (see Fig. 2, bottom left). While two zones may share visual content, we find that this simple approach works better than strictly limiting the overlap between zones (see Appendix. A8). Thus, the learning is guided by predicting parts of the environment that were never seen as well as those seen from different viewpoints.
Masked -one prediction Having segmented the video into zones, we next present our EPC masked- zone prediction task to learn environment embeddings (see Fig. 2). The main idea to infer unseen zones in a video by previewing the context spanning multiple seen zones. We randomly divide the zones into seen zones { and unseen zones {2 1m Ji=1. Given the seen zones and the mean camera pose from unscen zone p.a we need to infer a feature encoding of the unseen zone 2 To perform this task efficiently, we first extract visual features Ia from each RGB-D frame og in the video using pretrained CNNs (described in Sec. 3.2). These features are concatenated with the corresponding pose pr and projected using an MLP M to obtain the image-level embedding. The target features for the unseen zone Z, are obtained by averaging' all the MLP projected features:
f
Σ 4Z 2
M(I d))
(2)
Z
where we mask out the pose (i.e., p= 0 in the target avoid trivial solutions. We trans- former encoder-decoder model (Vaswani et al., 2017) to infer the zone features (see Fig. 2, bot- tom). An environment encoder uses self-attention over the image-level embeddings from all the seen zones, i.e., {M([E,)) [|) v(x,p) E 2J, Vi E [1, n]). to generate the environment embeddings E. A zone decoder then attends to E conditioned on the camera pose p from the unseen zone and predicts the zone features:
ZoneDecoder({,, p.).
(3)
Following Fang et al. (2019), we transform all poses in the input zones relative to Pt before encod- ing, which provides the model an egocentric view of the world. We perform the same transformation
We found this to he hetter than randomly sampling features within zone (see Appendix A A12).
4

---

Published as a conference paper at ICLR 2022
Environment-levt pretraining
Scene memory trunsformer (SMT)
Image-level pretminiig
O
Policy decoder
RIB-D
Environment crabeddings
Vr
(1) Imaageleevel trans
Iange
Environment encoder
ceoecer
Environment--Pe level transfer
Image encoder
%
conrastive los
EPC masked-zone prediction
Pa
MoCo-2
memarry
Figure 3: Integrating environment- pretraining for navigation. Right: First, we transfer image-level representations to encode each RGB-D D image. We transfer weights from an image encoder pre-trained for the self suff-vuper MoCo-v2 task video walkthrough images. Left: Next, and most importantly, we transfer the environment-lovel representations encode the entire history of observations. We transfer weights from the environment encoder and projection function M pre-trained for the proposed EPC masked-zone prediction task on video walkthroughs. Center: Finally, we finetune the SMT end-to-end on each task using RI.
to P internally in the ZoneDecoder, effectively making it zero (i.e., no rotations or translations). As we will show in experiments, conditioning on pose critical to learn useful representations. The environment encoder, zone decoder, and projection function M are trained end-to-end using noise- contrastive estimation (Gutmann & Hyvärinen, 2010). We use fr as the anchor and f. from Equ. as the positive. We sample negatives from other unseen zones in the same video and from all zones in other vidcos. The former discourages the model from simply learning video-specific tex- tures and patterns; the latter expands the pool of negatives which was found to be beneficial in prior work (Chen et al., 2020b;a). The loss for the 7th unseen zone in the video is:
sim(f." f() C, = -log Σ sim(f?? f5) + simm f2)'
(4)
where f are zone features from other videos, and sim k) = exp( lg [8] ?) is a similarity measure with temperature т = 0.1. The idea is to predict zone representations that are closer to the ground truth, while being sufficiently different from the negative zones. Since the seen and unseen zones may only have limited overlap, the model needs to effectively reason about the geometric and se- mantic context in the seen zones to perform this task. We qualitatively analyse the masked-zone prediction results from the learned EPC model in Fig. 4.
3.2 ENVIRONMENT EMBEDDINGS FOR EMBODIED AGENTS
Once a self- supervised environment encoder trained, we transfer the encoder various agents for performing navigation-oriente tasks. To this end, we integrate our pretrained environment encoder into the Scene Memory Transformer (SMT) from Fang et al. (2019). While our idea potentially applicable to other memory models, our choice of SMT is motivated by the recent successes of transformers in NLP (Devlín et al., 2018) and vision (Sun al., 2019b; Fang et al., 2019).
We briefly overview the SMT architecture (Fig. 3, center). We extract visual features from each RGB-D input (concatenated along the channel dimension) using ResNet-18 18 image encoder (He et al., 2016). We store the visual features and agent poses { P.) P:) observed during the episode in scene memory. We then memory and generate a rich set of environment embeddings E. We use a policy decoder to attend to conditioned on the inputs or = [Zt, Pt] at time and use the decoder outputs sample an action a, and estimate the value ut. We detail each component in the Appendix A2.
an environment encoder to perform self-attention
the scene
To incorporate our EPC environment embeddings into SMT, we first initialize the image encoder using CNNs pre-trdi using a state-of-the-arr MoCo-v2 method (Chen et al.. 2020b) (see Fig. 3, right). The image encoder is pre- pre-trained for 7,000 epochs on RGB-D D images sampled from the video walkthroughs generated for EPC pre-training. Note that this the CNN used to pre-extract RGB-D image features during EPC masked-zone prediction. Next, and most importantly, we initialize the environment encoder with our EPC pre-training on masked-zone prediction (see Fig. 3, left). We then finetune the SMT model end-to-end on the downstream taet-whether that is room navigation,
While the agent may revisit the views in different zones, we find learning be robust to this.
5

---

Published as a conference paper at ICLR 2022
3D environment
Intra-video retrieval
Inter-video retrieval
0.954
0.044
0,002
0.818
0.148
0,033
Figure 4: Each row shows one zone prediction example. Left: Top-down view of the 3D environment from which the video was sampled. Cyan viewing frusta correspond to the average pose for three input zones, Given the images and camera poses from each input zone, and target camera pose (green frustum), the model predicts the corresponding zone feature (the masked green zone). Center: Given the inferred feature, we rank three masked (unobserved) zones from within the same video, where green is the positive zone and the red are the negatives. For each zone, we show four randomly sampled images along with the retrieval confidence. Our methed retrieves the positive with high confidence. The model correctly predicts the existence of the narrow corridor (top row) and kitchen counter (hottom row) given the target poses. Right: Top retrieved zones from other videos that are closest to the inferred feature. The predicted features are general enough to retrieve related concepts from other videos (here, narrow corridors and kitchens).
object visitation, flee, area coverage (cf. Sec. 4). As we will demonstrate in the experiments, initializing the environment encoder using EPC leads to 2-6x higher sample efficiency and better performance when compared to initializing only the image encoder.
3.3 VIDEO WALKTHROUGH GENERATION
As discussed previously, EPC relies on a dataset of video walkthroughs containing RGB-D and odometer readings for self-supervision. Since EPC aims to learn the natural statistics of real-world environments (and not agent behavior), it suffices to use walkthroughs recorded in diverse environ- ments by agents performing their day-to-day tasks. In particular, used to generate walkthroughs are available for interactive policy learning, nor that the agent behav- iors used for generating walkthroughs tied to the downstream navigation tasks of interest. This means that we can, in parallel, collect videos from different agents operating in a large number of environments across geographical locations. We now realize this process in photorealistic Gibson scenes; we leave leveraging in nn-the-wild consumer videos as a challenge for future work.
do not require that the scenes
We generate 2 sets of egocentric video walkthroughs by deploying simulated agents in photorealis- tic scanned indoor environments from Gibson 2 (Xia al., 2018). 1) Strong Exploration: an SMT agent that was trained for area coverage on MP3D, and 2) Weak Exploration: a heuristic forward- biased navigation agent that moves forward until colliding, then turns randomly, and repeats. The weak exploration agent generates less-informative walkthroughs as it tends to repeatedly explore the same regions. Experimenting with EPC on both types of walkthrough allows us to test its depen- dence walkthrough quality. Methods robust to this quality are desirable since it may be casier to collect lower quality walkthroughs on a large-scale. In both cases, the agents explore each Gibson environment starting from multiple locations for 500 steps per location and record the RGB-D and odometer readings. This results in 5,047 videos per agent, which we divide into an 80-20 train/val split (i.e.. ~2M training frames). As we will show in Sec. 4, the EPC encoders learned on these walkthroughs are applicable to navigation tasks not tied to the exploration agents.
4 EXPERIMENTS
First, we review the experimental setup (Sec. 4. 4... We then evaluate the pre-trained EPC embeddings on multiple downstream navigation-orien oriented tasks in unmapped environments (Sec. 4.2). Finally, we present an ablation study of the data and proxy task used for EPC (Sec. 4.3).
We use 332 Gibson environments from the Gibson 2+ training split (Wijmans al., 2020).

---

Published as a conference paper at ICLR 2022
4.1 EXPERIMENTAL SETUP FOR DOWNSTREAM TASKS
We perform experiments on the Habitat simulator (Savva et al., 2019b) with Matterport3D (MP3D) (Chang et al., 2017) and Gibson (Xia et al., 2018), two challenging and photorealistic 3D datasets with 90 and 572 scanned real-world indoor environments, respectively. Our observation space consists of 171 X 128 RGB-D images and agent pose p = (z.y. y. 0) relative to the starting pose & = 0 (obtained by accumulating odometer readings). Our action space consists of: MOVE- FORWARD by 25cm, TURN-LEFT by 30", TURN-RIGHT by 30', and STOP (only for RoomNav), For all methods, we assume noise-free sensing during training, then evaluate with both noise-free and noisy sensing (pose, depth).
We perform interactive RL training on 61 MP3D train scenes. We evaluate the learned policies on 11 val and 18 test scenes in MP3D, and 14 val scenes in Gibson. These are disjoint from the 332 Gibson train scenes used for walkthroughs. We use episode lengths of T = 1000 for MP3D, and T = 500 for Gibson since the scenes are smaller. We further divide Gibson results into small and large environments (Ramakrishnan et al., 2020; Chaplot et al., 2020b). We evaluate our approach on four standard navigation tasks from the literature:
1. Area coverage (Chen et al., 2019; Chaplot et al., 2020b; Ramakrishnan et al., 2021): Agent is trained maximize the area covered (in m²) within a fixed time budget.
2. Flee (Gordon et al., 2019): Agent is trained to maximize the 'flee distance', i... the geodesic distance (in m) between its start and end positions, within fixed time budget.
3. Object visitation (Fang et al., 2019; Ramakrishnan et al., 2021): Agent is trained to maximize the # object categories visited within a fixed time budget. An object is 'visited' if it is visible to the agent within 1.5m of the agent's position. We report both the # categories and # instances visited. 4. RoomNav (Savva et al., 2017; Wu et al., 2019; Narasimhan et al., 2020): Agent is trained to find the nearest room instance of a provided room category. We evaluate using the two standard metries: SPL and Success (Anderson et al., 2018a). Please see Appendix A4 for the task details.
We choose these tasks since they capture different forms of geometric (tasks & 2) and semantic (tasks 3 & 4) inference in 3D environments. All tasks are different from the agent behavior in the weak exploration trajectories. While task 1 is well-aligned with the strong exploration trajectories, all other tasks are distinct. Object visitation is challenging task since the agent must navigate within 1.5m of each object and directly observe it. RoomNav the most challenging task since the goal is to visit a specific room (not arbitrary exploration).
We compare our approach to several baselines.
Scratch baselines: We randomly initialize the visual encoders and policy, and train them end-to-ent for each task. Images are encoded using ResNet- 18. Agent pose and past actions are encoded using FC layers. Reactive (scratch) has no memory. RNN (scratch) uses 2-layer LSTM as the temporal memory. SMT (scratch) uses a Scene Memory Transformer (SMT) for aggregating observations. SMT (MidLevel): This SMT-based model uses image encoders pre-trained for various supervised visual-tasks. The image encoders are kept frozen during RL (Sax et al., 2020; Zamir et al., 2020). SMT (MoCo): This SMT-based model uses ResNet- 18 image encoder pre pre-trained using MoCo- v2 (Chen et al., 2020b) and finetunes it end-to-end during RL. This is an ablation of our model from Sec. 3.2 where only the image encoder is pre-trained
the video walkthroughs. This So'A image- level pre-training is critical to isolate the impact of our proposed environment- pre-training. SMT (Video): This SMT-based model is inspired by Dense Predictive Coding (Han et al., 2019). The image encoder is initialized using weights from SMT (MoCo). It pre-trains the environment encoder as 'video-level' model on the video walkthroughs. It takes 25 consecutive frames as input and predicts the features from the next 15 frames (following timespans used in Han et al. (2019)). During SSL, we mask the input camera poses and query based on time, unlike EPC which uses input poses and queries based on pose during SSL.
ANS: This is our implementation of the SoTA hierarchical policy from Active Neural SLAM (Chap- lot et al., 2020b), upgraded to perform depth-basee occupancy mapping (instead of using only RGB).
All RL agents receive RGB-D images and pose inputs. All models are trained for 13M-15M frames with 60 parallel processes. We train the scratch and ANS baselines for 2M more frames to account for the 2M frames in video walkthroughs used for other methods. Both SMT (MoCo) and SMT (Video) are given the advantage of using strong exploration walkthroughs for SSL (see Sec. 3.3). However, we test EPC with either strong or weak exploration walkthroughs (variants denoted as S.E and W.E, respectively). See Appendix A5 for optimization hyperparameters.
7

---

Published as a conference paper at ICLR 2022
Area coverage (m²)
Flee (m)
Object visitation (# (f17#c)
RoomNav (Suce. / SPL)
Method
Gib-S
Gib-L
MP3D
Gib-S
Gib-L
MP3D
Gibson
MP3D
Gibson
MP3D
Reactive (scratch)
25.2
50.1
121.8
3.0
4.0
6.6
6.4/40
12.1/4.9
1.5/ 0.4
0.9/0.2
RNN (scrutch)
30.8
57.6
130.1
5.3
8.1
12.0
6.7/4.2
13.8/5.3
6.8/ 2.2
5.0/166
SMT (scratch)
32.2
62.6
166.5
4.3
6.4
11.0
9.2/5.4
17.8/6.3
5.0/ 2.3
1.4
SMT (MidLevel)
31.4
57.1
147.2
4.8
6.7
11.2
7.67 4.6
15.2/ 5.7
10.4/ 4.8
5.5/2.3
SMT (MoCo)
32.4
66.4
177.5
5.1
8.0
13.1
9.17
18.87 6.5
20.1/ 8.9
1..7/
SMT (Video)
31.9
61.3
137.1
4.9
7.7
11.
8.71 5.1
16.47 6.0
18.1/ 7.1
12.0/
ANS
33.3
81.5
172.8
3.6
7.8
13.0
10.0/ 5.9
20.7/ 6.9
0.0% 0.0
0.0/
EPC (WLE) EPC (S.E)
33.9
76.1
195.1
5.9
10.6
16.6
1..9/6.5
24.0/7.6
28.9/12.0
21.1/9.7
33.7
75.8
198.2
6.0
10.3
15.8
1..0/6.6
24.5/7.7
31.5/13.9
19.
Table I: Downstream task performance at the end of the episode. The two tasks in blue are geometric, and the two tasks in green are semantic. Gib-S/L means Gibson small/large. (# 1/ # c) means number of object instancesscateggoris visited. All methods are trained and evaluated on 2 and 3 random sceds, respectively. In each column, the best methods are highlighted bold (using a one-sided T-test with p 0.05). We report only the mean due to space constraints. See Appendix All for performance vs. time step plots.
Area coverage
Flee
Object visitation
RoomNNav
160
14
covered (m²)
Flce distance (m)
object categories
140
18
170
Success
100
SMT (MoCC))
12
80
DPC (W.E)
FPC (S.F)
60
# training steps (millions)
training steps (millions)
training stops (millicns)
training steps (millions)
Figure 5: Sample efficiency Matterport3D val split: We compare the sample-efficiencyy of EPC that of the standard approach of pretraining only image-level features. Our environment- level pretraining leads to 2x to 6x better training sample efficiency on all four tasks. See Appendix A6 for corresponding Gibson plots.
4.2 DOWNSTREAM TASK PERFORMANCE
We transfer the EPC features to downstream navigation tasks. Tab. shows the complete results. For brevity in the text, we also report the change in a method's performance relative to another as 4 values: (AAreaCoverage%, AFlee%, AObjerVVistation%, ARoomNNv%)), where each value is the mean relative change over all datasets and metrics for the corresponding task in Tab. 1.
SMT (scratch) outperforms RNN (scratch) and Reactive (scratch) on area coverage and object visita- tion. This is in line with results reported by Fang al. (2019) and verifies our SMT implementation. Image-level pre-training from SMT (MoCo) provides good gains over SMT (scratch), particularly on Flee and RoomNav. It relatively improves over SMT (scratch) by (4%, 20%, 1%, 301%).
Environment Predictive Coding (EPC) effectively combines environment-level and image-level pre- training to substantially improve over only image-level pre-training in SMT (MoCo). EPC is most beneficial for the challenging semantic tasks of object visitation and RoomNav. For example, EPC (W.E) improves by (9%, 24%, 24%, 56%) relative to SMT (MoCo). Furthermore, SMT (Video)- the video-level pretraining spatial information during self- superviee learning. This highlights EPC's value in reasoning about the underlying 3D spaces of the walkthrough videos instead of treating them simply as video frames. We also don't It observe statistically significant difference between EPC (W.E) and EPC (S.E) on majority of the tasks. Thus, EPC is robust to the walkthrough quality, and does not require the agent behavior in walkthroughs to be tied downstream tasks. This is advantageous since diverse and lower quality walkthroughs may be casier to obtain in practice.
srnatey--sgnnifcaame
underperforms EPC, as it ignores the underlying
Finally, EPC matches or outperforms the state-of-the -hrt--rrt ANS (Chaplot et al., 2020b) on both geo- metric and semantic tasks. We relatively improve over ANS by (2%, 42% 13%, N/A) with EPC (W.E) and (2%, 40%, 15%, N/A) with EPC ES.E). On area coverage, while ANS performs better on Gibson, EPC outperforms it on the larger MP3D environments. On RoomNav, ANS fails to learn the STOP action resulting in 0% success. Note that ANS was not originally designed with STOP action. Un- like the other agents with 4 actions, ANS has 24 x 24 location actions (see Appendix A3). Adding STOP action to this huge action space makes RL optimization difficult.
In Fig. 5, plot the validation metrics of SMT (MoCo) and EPC over the course of training. We see that environment-level pretraining offers significantly better sample efficiency: both EPC variants reach the best performance of SMT (MoCo) up 6x faster. This confirms our hypothesis:
8

---

Published as a conference paper at ICLR 2022
Area coverage (m²)
Flee (m)
Object visitation (# i7
RoomNav (Suce. / /SLL)
Method
Gib-S
Gib-L
MP3D
Gib-S
Gib-L L
MP3D
Gibson
MP3D
Gibsoo
MP3D
SMT (MoCo)
32.4
66.4
177.5
8.0
13.1
9.1/5.3
18.8/ 6.5
20.1
1..7/5.5
ANS
33.3
81.5
172.8
3.6
7.8
13.0
10.0/ 5.9
20.7 / 6.9
0.0 0.0
0.0/0.0
A. EPC
33.9
76.1
195.1
5.9
10.6
16.6
1..9/6.5
24.0/7.6
28.9 12.6
21.1/9.7
B. EPC woo query-p
32.2
63.0
166.1
7.0
11.4
7.8/4.8
15.3/5.8
12.0
8,1/3.3
C. EPC w/ auisy-d
33.4
72.5
185.6
5.6
8.6
15.0
12.2
24.0/7.6
27.3/11.0
19.0/8.0
EPC noisy.di dp
33.3
73.6
198.2
8.5
14.1
1..5/6.4
23.7/7.5
27.5
16.9/7.5
E. EPC w/ noisy-devo
33.6
74.7
192.1
5.8
9.8
15.5
11.6/6.4
23.3/7.5
25.3/
10.5
8
E. EPC w/o global
33.1
72.0
187.2
5.0
7.9
12.9
11.2/663
22.7/7.4
28.1 /
12.4
20.4/ 9.2
Table 2:
Ablation study of EPC with weak exploration videos. Top 2 rows are the best baselines from
Tab. Row B uses time query the representation during masked-zone prediction (not pose). Rows C, D show the impact of sensory noise (depth and/or odometer) the video walkthroughs. Row E shows the impact of replacing GT pose with estimates from a visual-odometry model (vo). Row F re-defines the masked zonc prediction task to use local input context (instead of global). See Appendix A7 for ablations with EPC (S.E).
transferring environment-leve encoders learned via spatial reasoning helps embodied agents learn faster compared to the current approach of transferring image-level encoders alone (Sax ct al., 2020).
4.3 ABLATION STUDY
We now present an ablation study of EPC in Tab. 2 to answer three key questions.
1. Does EPC need spatial conditioning for SSL? One of our key contributions is infer the unseen zone features conditioned on pose p. from the unseen zone (see Eqn. 2). We now try conditioning only on the time to when the agent visited the unseen zone. Note that the dataset, inputs to the environment encoder, and the loss function still remain the same. Consider rows A and B in Tab. 2. The performance for EPC w/o query-p dramatically declines by (12%, 25%, 30,, 61%) relative to EPC. Thus, it is critical to use spatial queries for SSL with EPC.
2. Does EPC need noise-free sensors to gather videos for SSL? In Tab. 1, we assumed noise-free depth and odometer sensors for gathering video walkthroughs. We now inject noise into the depth and odometer readings from videos to analyze EPC's robustness. We use popular existing noise models: the depth noise uses disparity- quantization, high-frequency noise, and uw-frrquency distortion (Choi et al., 2015); the odometry noise is based on data collected from LoCoBot, and accumulates over time leading to drift in pose (Ramakrishnan et al., 2020; Chaplot et al., 2020b). Additionally, we replace the GT pose with pose estimates from RGB-D based visual odometry (Zhao et al., 2021). Consider rows A, C, D, and E in Tab. 2. EPC is robust to sensory noise in the videos and its transfer performance is stable. The performance declines on area coverage and flee when we inject depth noise (noisy-d), and on object visitation and RoomNav when we further inject odometry noise (noisy-dp). The performance remains similar when using estimated pose from visual odometry poiisy-4vv), Overall, EPC retains its advantages over the best baselines from Tab. 1.
3. Does EPC need global input context during SSL? We use the global input context from multiple seen zones for EPC masked-zone prediction (Fig. 2). We now reduce this context to a more localized part of the video. Specifically, we modify the task to use the local context from a randomly sampled chunk of 25 contiguous frames as input, and infer features from the next 15 frames conditioned on pose. Consider rows A and E in Tab. 2. When the performance deteriorates by (3%, 20%, 4%, 2%) relative to EPC. As shown in Appendix A7, we observe an even more drastic reduction of (7%, 17%, 8%, 21%) for EPC w/o global with the strong exploration videos. This suggests that the global input context is beneficial for SSL with EPC.
remove the global context (EPC w/o global),
In Appendix A9, we demonstrate the robustness of EPC's policies sensory noise during testing.
5 CONCLUSIONS
We introduced Environment Predictive Coding, a self-supervised approach to learn environment- level representations for embodied agents. By training on video walkthroughs generated by other agents, our model learns to infer missing content through masked-zone prediction task. When transferred to multiple downstream embodied agent tasks, the resulting embeddings lead to 2-6x higher sample efficiency and upto 57% better performance when compared to the current practice of transferring only image-level representations. Our work highlights the advantages of supf-suue learning in agent videos by predicting features grounded in the spatial structure of 3D environments. In future work, we aim to extend our idea to in-the-wild videos and apply it to multimodal tasks such embodied question answering, instruction following, and audio-visua navigation.
9

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENTS
UT Austin is supported in part by the IFML NSF AT Institute, the FRL Cognitive Science Consor- tium, and DARPA Lifelong Learning Machines. We thank members of the UT Austin Vision Lab for feedback on the project. We thank Dhruy Batra for feedback on the paper draft. We thank the ICLR reviewers and meta-reviewers for their valuable feedback and suggestions.
7 REPRODUCIBILIITY STATEMENT
We describe method in Sec. with architecture diagrams and equations to support the tex- tual description. We describe our experimental setup in Sec. 4.1, and provide the necessary hyperparameters in Appendix A5, We publicly release the code and pre-trained models here: https://visl on, cs www.eae.eeeeeeeo petpppojeeetess//epp...
ETHICS STATEMENT
Our idea for self-supervised learning relies on having a dataset of video walkthroughs captured in diverse set of environments. In our experiments, we demonstrate proof-of-concept by generating these walkthroughs in photorealistic Gibson scenes using simulated agents. However, obtaining such large-scale dataset of walkthroughs in real-world environments be challenging due to privacy concern.. To create such a dataset, we would need to carefully consider the privacy of the people and personal belongings recorded
part of this data.
We perform experiments on the publicly available Gibson and Matterport3D datasets. Since these datasets primarily contain data from work places and economically well-off residences, our results may be influenced by the social and geographical distribution of the scanned buildings in these datasets. In experiments, we demonstrate good generalization of embodied policies trained in MP3D train scenes to novel MP3D test scenes and novel Gibson val scenes. However, it is possible that the generalization may be impacted when the policies are evaluated in environments that are signif- icantly outside the distribution of these datasets.
REFERENCES
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozheh Mottaghi, Manolis Savva, et al. On evaluation of embodied naviga- tion agents. arXiv preprint arXiv: /87706677, 201 Sa.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson. Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-an and---nd--aa languag navigation: Interpreting grounde navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018b.
Iro Armeni, Zhi- Yang He, Young Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE International Conference on Computer Vision, pp.
2019.
Tim Bailey, Nebot, Rosenblatt, maps. In Proceedings of the International Conference on Field and Service Roboticx, pp. 347-352, 1999.
Durrant- Whyte.. Robust distinctive place recognition for topological
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Olcksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navi- gating to objects. arXiv preprint McXiv22006.1377 2020.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Vietor Valdés, Amir Sadik, Julian Schritwieserr, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801, 2016.
03801.
Angel Chang. Angela Dai, Tom Funkhouser, Matthias NieBner, Manolis Savva, Shuran Song. Andy Zeng. and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In Proceedings of the International Conference 3D Vision (3DV), 2017. MatterPort3D dataset license available at: http: ParPP_TSSppFFFf
//kaldir. ve. in. www.///teemmot////eeeeeeee
10

---

Published as a conference paper at ICLR 2022
Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. In NeurlPS, 2020.
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In In Neural Information Processing Systems, 2020a.
Devendra Singh Chaplot, Saurabh Gupta, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural mapping. 8th International Conference on Learning Representations, ICLR 2020, 2020b.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurahh Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recog- nition, pp. 12875-12884, 2020c.
Changan Chen, Sagnik Majumder, Al-Halah Ziad, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning set waypoints for audio-visual navigation. In ICLR, 2021.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In 7th Interna- rional Conference on Learning Representations, ICLR 2019, 2019.
Ting Chen, Simon Komblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint ar 2003.0000 2020b.
Zetao Chen, Adam Jacobson, Niko Sünderhauf, Ben Uperoft, Lingqiao Liu, Chunhua Shen, Ian Reid, and Michael Milford. Deep learning features scale for visual place recognition. In 2017 IEFF International Conference on Robotics and Automation (ICRA), pp. 3223-3230. IEFF, 2017.
Sungjoon Choi, Qian-Yr Yi Zhou, and Vladlen Koltun. Robust reconstruction indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5556-556 5565, 2015.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8/0.04805, 2018.
Yilun Du, Chuang Gan, and Phillip Isola. Curious representation learning for embodied intelligence. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 360(6394)):1044 1200- 2018
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 538-547, 2019.
Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffiman, and Dhruv Batra. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In ICCV, 2019.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping Belief States with Generative Environment Models for RL. In NeurlPS, 2019.
Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean-bastien Grill, Florent Altché, Rémi Munos, and Moham- mad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforcement learning. arXiv preprint arXiv:2004. 14646, 2020.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEF. Conference on Computer Vision and Pattern Recognition, pp. 2616-2625, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Proceedings of the Thirleenth International Conference on Artificial Intelli- gence and Statistics, pp. 297-304 2010.
David Ha and Jürgen Schmidhuher. Recurrent world models facilitate policy evolution, In Advances in Neural Information Processing Systems, pp. 2450-2462, 2018.

---

Published as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Tengda Han, Weidi Xic, and Andrew Zisserman. demory-augmennie dense predictive coding for vidco repre- sentation learning. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-7700777 (700-78, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-99999999 9738, 2020.
D. Jayaraman and Grauman, Learning image representations tied ego-motion. ICCV, 2015.
Dinesh Jayaraman and Kristen Grauman. Learning look around: Intelligently exploring unseen environments for unknown tasks. In Computer Vision and Pattern Recognition, 2018 IEEE Conference on, 2018.
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-supervised video representation learning with space-time cubic puzzles. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 8545-8552, 2019.
Jing Yu Koh, Honglak Lec, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
14738-14748, 2021.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep rein- forcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Benjamin Kuipers. The spatial semantic hierarchy. Artificial intelligence, 119(1-2):191-22211 2000.
Benjamin Kuipers and Yung-tai Byun, A robot exploration and mapping strategy based on semantic hierarchy of spatial representations. Journal of Rohotics and Autonomous Systems, 8:47-63, 1991.
Ananya Kumar, SM Eslami, Danilo Rezende, Marta Garnelo, Fabio Viola. Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint arXiv: 1807.00033 2018.
Daniel James Lenton, Stephen James, Ronald Clark, and Andrew Davison. End-to-end egospheric spatial memory. In International Conference on Learning Representations, 2020.
Stephanie Lowry, Niko Sünderhauf, Paul Newman, John Leonard, David Cox, Peter Corke, and Michael J Milford. Visual place recognition: survey. IEEE Transactions on Robotics, 32(1):1-19, 2015.
Alena Lukasová. Hicrarchical agglomerative clustering procedure. Pattern Recognition, 11(5-6):365-2 1979.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning navigate complex environments. CoRR, abs/1611.0366 1.36673,, 2016. URL bettp://axxiv.ee /bs/ 1611. 03673.
Medhini Narasimhan, Erik Wijmans, Xinlei Chen, Trevor Darrell, Dhruy Batra, Devi Parikh, and Amanpreet Singh. Seeing un-scene: Learning amodal semantic maps for room navigation. ECCV, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1oXv:/887.00345 2018
12

---

Published as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Ed- ward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: / http:/pppppp.e.ppppppppppp www.ris..ccppppe... 9015-pytorch- anwpreh-aaniiimnatiieeeeee settyll-high-prrreeeee pdr.
deep- learning-library
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. CVPR, 2016.
Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. Emergence of exploratory look-around behaviors through active observation completion. Science Robotics, 4(30), 2019, doi: 10. 10/sii//sssroooticss aaw6326. URL htpps: / http//bboticcccoo s sciencemeaa org/ content / 4/33 30 eeaw 6326.
Santhosh K. Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient explo- ration and navigation. In ECCV. 2020.
Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. An exploration of embodied visual
exploration. International Journal of Computer Vision, 2021. doi: 10.10077111263--21-114111 URL https:/ //cii http//cci.rrgg/////// crg/10. 1007/ / 126111666222 02100021-----101
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. cemi-parametric topological memory for naviga- tion. In International Conference on Learning Representations, 2018
Manolis Savva, Angel X Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. Minos: Mul- timodal indoor simulator for navigation in complex environments. arXiv preprint 17X1v:/7/2.03931, 2017.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao. Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019a.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision. pp. 0239999947,71 2019b.
Alexander Sax, Jeffrey Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik. Learning to navigate using mid-level visual priors. Conference on Robot Learning, pp. 791-812, 2020.
William Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situational fusion of visual representation for visual navigation. ICCV, 2019.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Shuran Song, Andy Zeng, Angel X Chang, Manolis Savva, Silvio Savarese, and Thomas Funkhouser. 1m2pano3d: Extrapolating 360 structure and semantics beyond the field of view. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3847-3856, 2018.
Adam Stooke, Kimin Lec, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from rein- PMLR, 2021.
forcement learning. In International Conference on Machine Learning, pp.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel. Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:/906.0 05797, 2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using con- trastive bidirectional transformer. arXiv preprini 1rX.v:1906.05543. 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBert: A joint model for video and language representation learning. In Proceedings of the IEF.E. International Conference on Computer Vision, pp. 7464-7473, 2019h.
13

---

Published as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Dongla Wei, Joseph Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow time. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8052-8060, 2018.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Ddoppo: Learning near-perfedd pointgoal navigators from 2.5 billion frames. In ICLR, 2020,
Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9068-9079, 2018. Gibson dataset license agreement available at https: / //sttraage... peccolllapii.. com/gi wtt.aaannntaraaa/aoaaaaa...a /twpo///to.eeee///orrrra//eaeeee
pdf.
Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martin-Martín, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):713-720, 2020.
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual semantic navigation using scone priors arXiv preprint arXiv: 1810.06543, 2018.
Zhenpei Yang, Jeffrey 7. Pan, Linjie Luo, Xiaowei Zhou, Kristen Grauman, and Qixing Huang. Extreme relative pose estimation rgh-d scans via scene completion. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Joel Ye, Dhruv Batra, Erik Wijmans, and Abhishek Das. Auxiliary tasks speed up learning pointgoal navigation. In Proceedings of the Conference on Robot Learning (CoRL), 2020.
Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectnav, 2021.
Amir 7amir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas Guihas. Robust learning through cross-task consistency. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11197-11200 2020.
Xiaoming Zhao, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation. In Proc. ICCV, 2021.
14

---

Published as a conference paper at ICLR 2022
Appendices
We provide additional information about the experimental settings as well as quantitative results to support the experiments from the main paper. Below is summary of the sections in the Appendix:
($A1) Supplementary visualization
($A2) Scene memory transformer
($A3) Model architectures
($A4) Additional task details
($A5) Hyperparameters
($A6) Sample efficiency on Gibson
($A7) Ablations of EPC with strong exploration videos
($A8) Complex zone creation schemes for EPC
($A9) Noise robustness of learned policies
($A10) ANS performance on Flee
($A11) Downstream task performance vs. time
($A12) Masked-zone task: Averaging random sampling features
($A13) Relation to visual place recognition
A1 SUPPLEMENTARY VIDEO
The supplementary visualization "epc_ www.vvuuuuzaiin...... supppippoo provides a brief overview of envi- ronment predictive coding and the masked-zone prediction task. We also show an example of inter- video retrievals (similar to Fig. 4 from main paper). In the demonstrated example, we show that our EPC model is able to accurately retrieve the masked-zen zone from the current scene, and even retrieve similar zones from other scenes.
A2 SCENE MEMORY TRANSFORMER
We provide more details about individual components of the Scene Memory Transformer (Fang al., 2019). As discussed in the main paper, the SMT model consists of ResNet-18 visual encoder that extracts features Ię from RGB-D inputs, and a scene memory for storing the visual features (r, and agent poses {p, 1 seen during episode. The environment encoder uses self-attention on the scene memory to generate a richer set of environment embeddings {ei) The policy decoder attends to the environment embeddings using the inputs o, time which consist of the visual feature 2t, the agent pose Pt. and optionally, the navigation goal gt. The outputs of the policy decoder are used sample an action a and estimate the value Ut. Next, we discuss the details of the individual components.
VISUAL ENCODER At each time step t, the visual encoder extracts features 2, from the RGB-D input.
VisualEncoder((rr. de))
(5)
where and d, are the RGB and depth inputs, respectively. We consider two visual encoders for our work. The first variant modified ResNet-18 18 encoder from (Wijmans al., 2020) where the number of output channels are halved and the BatchNorm layers are replaced by GroupNorm. We used this encoder for the scratch models, SMT (MoCo), SMT (Video), and the EPC variants in the main paper. Next, consider MidLevel features derived from various pre-trained CNNs to solve midlevel perception tasks (Sax et al., 2020; Zamir et al., 2020). For RGB inputs, we ex- tract features from the pre- pre-trained models in the max-coverage set proposed in Sax al. (2020). These include 4 encoders trained for predicting surface normals, keypoints, semantic segmentation, and 2.51) segmentation. For depth inputs, extract features from pre-trained models that predict
15

---

Published as a conference paper at ICLR 2022
surface normals and keypoints from depth (Zamir et al., 2020). These encoders kept frozen during reinforcement learning, following Sax et al. (2020). MidLevel features were used for SMT (MidLevel) in the main paper. We do not use the MidLevel encoder for EPC since we found the ResNet- 18 encoder to be more efficient and achieved hetter performance.
SCENE MEMORY It stores the visual features 2i derived from the input images and the agent poses Pt at each time-step t. Following Fang et al. (2019), the pose consists of the 2D coordinates (æ, y) of the agent. heading 0, and the time While we assumed that the ground ground-truth pose (Iz, Yt. (,) was available for downstream tasks in the main experiments, we evaluate the impact of noisy odometry in Appendix A9.
ATTENTION MECHANISM Following the notations from Vaswani et al. (2017), we define the attention mechanism used in the environment encoder and policy decoder. Given two inputs e Ra1xd, and Y R=axdy, the attention mechanism attends to Y using X as follows:
Qx dd
Attn(X, Y) = softmax
Vr
(6)
where Qx E Reixds Ky E Re2xds Vy R=2xd, are the queries, keys, and values computed from X and Y as follows: Qx XW Ky = YW and Vy- = YW", W4, W W are learned weight matrices. The multi-headed version of Attn generates multiple sets of queries, keys, and values to obtain the attended context E Ra1xd, xd.
MHAttn(X,, Y) FC([Att" (X,
We use the transformer implementation from PyTorch (Paszke et al., 2019). Here, the multi-headed attention block builds on top of MHAttn by using residual connections, LayerNorm (LN) and fully connected (FC) layers to further encode the inputs.
MHAutBlock X, Y) LN(MLP(H) + H)
where = LN(MHA (X, Y) + X), and MLP has FC layers with ReLU activations.
We now describe the memory attention mechanisms used in SMT. At time I, the agent recieves fea- tures It, pose Pt, and optionally the navigation goal 9t- First, we transform the pose vectors {p.}-- from the scene memory relative to the current agent pose Pt. This allows the agent to maintain an egocentric view of the past inputs (Fang et al., 2019). Next, the environment encoder performs self- attention between the features stored in the scene memory M obtain the environment encoding:
m EnvironmenEEncoee(()) = MHAttBBock(MMA. M)
Next, the policy decoder attends the environment encodings E ussing linear projection of the current observation f = M([x,, gt))
D PolieyDeccodr(((e,, Pt. 9.). E) MHAttBlock(ff.e)
(10)
POLICY It consists of two FC layers corresponding to the actor æ and critic V. Given the output D of the PocieeDecoder, the policy samples an action a₂ x (DD and predicts the value function 2i2 = (DD. Please see the next section for more details on the exact architectures used.
A3 MODEL ARCHITECTURES
We provide the architectures for the RNN (scratch), SMT (scratch), and ANS baselines Fig. A1. For Reactive (scratch), we remove the "Transformer Encoder" and "Transformer Decoder" blocks from SMT (scratch) and directly feed the input features to the policy. For SMT (MidLevel), we replace the ResNet encoder with the MidLevel encoders discussed in Sec. A2. For SMT (MoCo), SMT (Video), and EPC, we use the same architecture as SMT (scratch) and initialize pre-trained weights according to the method used.
For ANS, Fig. Al describes a global policy that samples as actions an (x, y) location or STOP. The input occupancy map is 300 x 300 2 egocentric map where the two channels track occupied
16

---

Published as a conference paper at ICLR 2022
RNN (scratoh)
(scraton)
PRR action
Thandorrer
Rechar 1
layer hoods
Erécdyy S21
RGB-O
354
120-
106 vedeo difc
Trandorer
120-
RGB-
Eread 32
2206
POR
Erboah 2t
Active Neurra SLAM (ANS)"
Occupancy CNN
20-d
ConuONi 15, 21
Co in 2
Coopancy Mep
Cono222
Brioah 80
12-4
room goel
ConvGNJM.N
Notations
Dospany
make comnaoor mumner room datagoria
Docupency
Gravniame Ponrramm M
FoR Wy
Figure A1: Baseline architectures: We show the detailed architectures for each of the key baselines. "Goa- |Encoder" is a 1-layer MLP. . is 2-layer MLP. Embed, FC, Conv, GroupNorm, and ReLU are PyTorch layers corresponding to nn. Embedding (). an Linear (), nn Conv2d (), Grr.pNornnnnoo......o (), and n (), respectively (Paszke et al., 2019). The architectures shown used for the RoomNav task. For ReLU
remaining tasks, we remove the room goal "Embed" "GoalEEncoder" layers and keep rest of architecture unchanged.
and explored regions. It captures 48m 48m region around the agent. A local policy, which low-level navigator, then navigates the (x, y) location for 25 steps. We mapper+planner as the local policy as this was found to perform as well as a learned policy (Chaplot et al., 2020b;a). For area coverage, flee, and object visitation, the global policy. We try both discrete (24 x 24 actions) and continuous action spaces, and pick the best choice based the validation performance. For RoomNav, use discrete action space and add STOP as an additional action, giving us 24 x 24 + 1 = 577 actions.
analytical
48m 48m action space for
A4 ADDITIONAL TASK DETAILS
We provide more details about the object visitation and RoomNav tasks below.
OВJECT VISITATION To determine if an object is visited, we check if it is within 1.5m of the agent, present in the agent's field view, and if it is not occluded Ramakrishnan et al. (2021). We use shaped reward function that rewards the agent for visiting new object category and cell-coverage reward to encourage exploration (similar to Fang et al. (2019)):
R = O, - 0.-1 + 0.02(C, Cc-1);
(11)
where Ot, Ce are the number of object categories and 2D grid-cells visited by time 1. For MP3D, use all 21 object categories defined for the ObjectNav task in Habitat (Batra et al., 2020: Savva et al., 2019a):
chair, table, picture, cabinet, cushion, sofa, bed, chest of dravers, plant, sink, toilet, stool, towel, tv monitor, shower, bathtub, counter, fireplace, gym equipment, seating, clothes
For Gibson, we use the semantic annotations from the 3D Scene Graph dataset (Armeni et al., 2019). We evaluate the following categories:
17

---

Published as a conference paper at ICLR 2022
chair, dining table, cook, vase, bottle, couch, bed, refrigerator, potted plant, sink, toilet, clock, tv, oven, cup, umbrella, bowl, bench
We report both the number of categories and instances of the above objects visited. For MP3D, evaluate on 550/900 validation/test episodes. For Gibson, we evaluate on 300 validation episodes. We do not split Gibson semantic results into small/large since there are only 6 val semantic scenes.
ROOMNAV For an agent to successfully reach a specified room category, it needs to within 0.1m of navigable location inside any room belonging to that category. We use a shaped reward function (Savva et al., 2019a):
R₁ = 1 - de 2.5S, 0.001
(12)
where de is the distance to the nearest target room location, indicates success time L, and 0.001 is a slack penalty. We evaluate on Gibson and MP3D rooms of the following categories: bathroom, bedroom, office, kitchen, living room, dining room
We measure both success rate and SPL, i.e., success weighted by path length (Anderson et al., 2018a). We calculate SPL similar to (Batra et al., 2020), where the shortest path length is de- fined based on the target room closest to the agent's starting point. We evaluate on 410/1600 val/test episodes on MP3D, and 598 val episodes on Gibson. We use 500-step episodes follow- ing Narasimhan et al. (2020) (unlike 1000-step episodes for other tasks on MP3D).
A5 HYPERPARAMETERS
All models are trained in PyTorch (Paszke et al., 2019) with DD-PPO (Wijmans et al., 2020) for 13M-15M frames with 60 parallel processes. Since SMT (MoCo), SMT (Video) and EPC benefit from 2M frames of off-policy experience in the video walkthroughs, we train the scratch and ANS baselines for 2M more frames account for this. Note that SMT (MidLevel) is already pre-trained on 4M frames of annotated data.
We detail the list of hyperparameter choices for different tasks and models in Tab. AI. For ANS, we use 4 PPO mini-batches, 4 PPO epochs (not 2), and entropy coefficient of 0.003. For SMT (Video), we randomly sample 40 consecutive frames in the video and predict the final 15 frames from the initial 25 frames (following time-spans from Han et al. (2019)). For EPC, we randomly mask out m zones of size N in the video and predict them from the remaining video. We selected these values based on a grid-search and the validation performance on downstream tasks. For area coverage, flee, and object visitation, we found N = 5, m = 6 work best. For RoomNav, we found N 40, = 6 to work best on weak exploration videos, and N = 40, m 6 to work best on strong exploration videos. The hyperparameter search results are shown in Figs. A2, A3.
A6 SAMPLE EFFICIENCY ON GIBSON
We plot the Gibson validation performance function of training experience in Fig. A4. EPC achieves 2 - 6x higher sample efficiency through env environment-level pre-training when compared to the only image-level pre-training from SMT (MoCo).
A7 ABLATIONS OF EPC WITH STRONG EXPLORATION VIDEOS
In Sec. 4.3 from the main paper, we presented an ablation study of EPC (W.E). We now provide the corresponding results for EPC (S.E) in Tab. A2. Our conclusions remain the same as in Sec. 4.3. (1) EPC needs spatial conditioning. i.e., to query using pose during SSL (instead of only time).
(2) EPC robust to noise in depth and odometer sensors used for collecting the video walkthroughs.
(3) EPC benefits from global input context during SSL (instead of a local input context of 25 frames).
A8 COMPLEX ZONE CREATION SCHEMES FOR EPC
Over the course arriving at our final EPC zone creation scheme, we tried an alternative scheme that attempted to strictly minimize the overlap between the and unseen zones. We measured
18

---

Published as a conference paper at ICLR 2022
Area DovaEage
Ples
Object vislation
Roomblay
15.3
1583
134
75
200
60 callgge
ADHA (197
640
Figure A2: EPC hyperparameter search on weak exploration videos: We show the MP3D validation results for different zone sizes on 3 tasks. The X-axis indicates the zone size as (N, m) where is the number of frames in each zone, and m is the number of zones masked in the video. For N 5-40, we randomly sample .V from 5 40 for each video during training.
Areo COTHENEM
Flee
Objcl valatio
Roomblee
158
1515
1915
141 1OE
FOOECE curgootes
137
244
21.
(00)
2.0
9
2.31 0544
8.5
21 PM DU 29
JUM
3-ML
154
Figure A3: EPC hyperparameter search on strong exploration videos: We show the MP3D validation results for different zone sizes 3 tasks. The X-axis indicates the zone size (N,m) where N the number frames in each zone, and m is number zones masked the video. 5-40, we randomly sample NV from 5 to 40 for each video during training.
RL Optimization
Optimizer
Adam
Learning rate
0.00025
# parallel actors
60/64
PPO mini-batches
2*
PPO epochs
2*
PPO clip param
0.2
Value loss coefficient
0.5
Entropy coefficient
0.01
Advantage estimation
GAE
Normalized ad advantage?
Yes
Training episode length
1000*
GRU history length
128
# training steps (in millions)
13-15
SMT hyperparameters
Hidden size
128
Scene memory length
500
Scene memory stride
2/4
# attention heads
8
# encoder layers
1
# decoder layers
Self-supervised learning optimization
Optimizer
Adam
Learning rate
0.0001
Video batch size
10
Temperature (7)
0.1
Table Al: Hyperparameters for training our RL and self-supervised learning models. we use 4 PPO mini- batches and 4 PPO epochs for ANS. We disable normalized advantage and use 500-step episodes for RoomNav
19

---

Published as a conference paper at ICLR 2022
Area coverage
Flee
Object visitation
RoomNav
45
32
Ares covered (m3)
40
Floe distance (m)
# objecc eategorics
24
85
16
30
SMT (MoCo)
(W.F)
(SE)
2 4 6 8 10 12 14
2 0
8
12
2
4
6
8
10
12
14
0
9
12
# training stops (millions)
# training
steps (millions)
# training steps (millions)
21 &training stegs (millioons)
Figure A4: Sample efficiency on Gibson val split. Our environment-levv pre-training leads to 2 6x higher sample efficiency when compared to SoTA image-level pre-training.
Area coverage (m²*)
Flee (m)
Object visitation (# i/#e)
RoomNav (Suce. / SPLL
Method
Gib-S
Gib-L
MP3D
Gib-S
Gib-L
MP3D
Gibson
MP3D
Gibson
MP3D
SMT (MoCo)
32.4
66.4
177.5
5.1
8.0
13.1
9,1
18,
20.1/ 8.9
12.7/5.5
ANS
33.3
81.5
172.8
7.7
13.0
10.0/5.9
6,9
0.0%
0.0/0.0
A. EPC
33.7
75.8
198.2
6.0
10.3
15.8
12.0/6.0
24.5/7.7
31.5/ 13.9
19.8/8.9
B. EPC wo query-p
31.9
62.7
162.9
7.8
12.5
8.0/
15.6/5,9
13.1 5.6
10.0/413
C. EPC w/ aoisy-d
33.8
74.9
196.4
5.7
10.2
15.9
11.8/62
3.73/7.7
29.4/ 12.7
19.6/8,4
D. EPC w/ noisy-dp
33.6
75.8
192.2
5.1
8.3
13.7
12.0
24.1/7.8
26.6/ 11.7
17.7 8.2
E. EPC w/o global
32.4
68.4
178.4
8.1
13.2
10.8/6.0
22.1/7.2
23.7
17.4/74
Table A2: Ablation study of EPC with strong exploration videos. Top 2 rows are the best baselines from
Tab. 1. Row B uses time to query the representation during masked zone prediction (not pose). Rows C, D show the impact of sensory noise (depth and/or odometer) in the vidco walkthroughs. Row E defines the masked- zone prediction task to use local input context (instead of global). All methods are trained and evaluated on and 3 random seeds, respectively, and we report the mean. In each column, the best methods are highlighted in bold (using a one-sided T-test with p = 0.05).
the 31D point-cloud overlap between every pair of frames in video walkthrough (projected using depth, camera pose, and camera intrinsics), and clustered the frames that have significant overlap together using hierarchical agglomerative clustering (Lukasová, 1979). We defined each cluster to be a zone, and used the same SSL training pipeline from Sec. 3. Tab. A3 shows the results on the MP3D validation split on tasks. The spatial overlap variant works reasonably well. However, is significantly more complex than our original proposal (requiring camera intrinsics. 3D point-cloud overlap computations and clustering), and performs worse. Since the videos are unconstrained, the agent might revisit the same areas multiple times, causing too many frames to be grouped together into a single zone. This results in very irregular zone sizes for masked-zon zone prediction across videos. This affects the quality of EPC SSL training. By not strictly limiting the overlap between zones, the agent learns by both inferring seen features (but from different viewpoints), as well as unseen features. Our current method is simpler, and works better.
A9 NOISE ROBUSTNESS OF LEARNED POLICIES
In the main paper, we assumed the availability of ground-truth depth and pose sensors for down- stream tasks (In Tab. we added pose and depth noise the walkthrough videos only). Now, we relax these assumptions and re-evaluate all methods by injecting noise in the depth and odometer sensors for downstream tasks (same noise models from prior work that we applied in Sec. 4.3), without any noise-correctin ness (Chen al., 2019; Ramakrishnan et al., 2021; Chen et al., 2021). Note that the noise in agent's pose estimate increases over time since the per-step noise in odometer readings are accumulated. We compare the noise-robustness of all methods in Tab. A4. As expected, the performance declines as we add noise to more sensors (depth, then pose). We observe larger deterioration in performance when noise is added to depth. Note that this is likely due to the domain shift in the depth inputs between noise-free training and noisy evaluation. Nevertheless, most approaches reasonably stable. EPC outperforms all methods when all noise sources are added. ANS declines rapidly in the absence of noise-correction due accumulated map errors.
This is a common evaluation protocol for assessing noise robust-
20

---

Published as a conference paper at ICLR 2022
Flee (m) MP3D
Object visitation
RoomNav (MP3D)
Method
categories
# instances
SPL.
Success
EPC (W.E) spatial
12.0
26.4
9.8
23.7
EPC (W.E) temporal
14.4
7.5
26.3
11.0
25.0
EPC (S.E) w/ spatial
12.5
7.3
25.7
9.3
22.5
EPC (S.E) w/ temporal
13.9
7.5
26.6
10.3
24.4
Table A3: Comparing zone creation schemes for EPC. 'EPC w/ spatial' is variant of EPC which uses 3D spatial overlap between frames to cluster them into zones. 'EPC w/ temporal" is our original method proposed Sec. 3. The results on the MP3D validation split. Our temporal scheme works better while being much simpler to implement.
A10 ANS PERFORMANCE ON FLEE
ANS relies on a global policy that samples a spatial goal location for navigation. A local navigation policy then executes series of low-level actions reach that goal. Our qualitative analyses indicate that the global policy overfit to large MP3D environments. often samples far away exploration targets, relying the local navigator explore the spaces along the sampled direction. However, this strategy fails the small Gibson-S environments (typically single room). Selecting far away targets results in the local navigator oscillating in place trying exit single-room environment. This does not affect area coverage much because it suffices to stand in the middle of a small room and look all sides.
A11 DOWNSTREAM TASK PERFORMANCE VS. TIME
We show the downstream task performance function of time in Fig. A5. For each model, train with 2 random seeds and evaluate with 3 random seeds. We report the mean and the 95% confidence interval the plots. EPC converges faster than using only image-level pretraining in SMT (MoCo), and outperforms state-of-the-e Active Neural SLAM (ANS) most tasks.
A12 MASKED-ZONE TASK: AVERAGING VS. RANDOM SAMPLING FEATURES
In Eqn. 2 from the main paper, we averaged the projected visual features for all frames within zone to get the ground-truth zone feature. Now, we compare an alternative strategy where we randomly sample visual feature from frame in the zone instead of averaging. Specifically,
ft M ([2,, d)), where I; Z,"
(13)
We compare this with our original feature averaging strategy the MP3D val for flee, object visitation, and roomnav in Tab. A5. As we can see, our current scheme of averaging the projected features over all frames in the zone works better.
A13 RELATION TO VISUAL PLACE RECOGNITION
Prior work on visual place recognition build maps and recognize whether the current visual in- formation is from a place recorded in the map (Lowry et al., 2015). A place could be an exact location (Kuipers, 2000), rooms (Kuipers, 2000), locally distinctive regions (Kuipers & Byun, 1991; Bailey et al., 1999). Zones in EPC can be viewed as specific 'place' in 3D environments. However, instead learning place-level features for recognizing previously seen places (Chen al., 2017), we learn encoders by inferring features for unseen places.
21

---

Published as a conference paper at ICLR 2022
Area covered (m=²)
Gibson-S N-D
Gibson-L N-D
MP3D
Method
NF
N-D.P
NF
N-D.P
NF
N-D
N-D.P
Reactive (scratch)
28.2
29.9
30.2
50.1
49.2
50.8
121.8
119.5
120.9
RNN (scratch)
30.8
33.0
33.1
67.6
56.4
56.9
130.1
106.6
108.8
SMT (scratch)
32.2
34.5
34.6
62.6
62.8
62.1
166.5
153.8
149.3
SMT (MidLevel)
31.4
33.2
33.0
67.1
54.5
53.7
147.2
143.0
138.3
SMT (MoCo)
32.4
34.5
35.1
66.4
66.4
66.2
177.5
162.5
148.9
SMT (Video)
31.9
34.3
33.7
61.3
61.9
59.4
157.1
138.5
137.4
ANS
33.3
35.6
33.3
81.5
84.4
62.7
172.8
171.7
129.6
(W.E)
33.9
36.4
36.0
76.1
77.0
75.6
195.1
179.7
175.2
(S.E)
33.7
36.4
75.8
77.1
75.7
198.2
183.2
178.9
Flee distance (t))
Gibson-S
Gibson-L N-D
MP3D
Method
NF
N-D.P
N-D,P
NF
N-D
N-D.P
Reactive (scratch)
3.0
3.0
3.1
4.0
4.1
4.1
6.6
7.8
7.9
RNN (scratch)
5.3
7.5
7.4
12.0
11.1
10.9
SMT (scratch)
4.3
6.4
6.5
6.5
11.0
10.8
(MidLevel)
4.8
6.1
6.1
11.2
10.8
10.5
SMT (MoCo)
5.1
5.1
8.0
8.0
13.1
12.3
12.1
SMT (Video)
4.9
7.6
7.0
11.8
10.9
10.2
ANS
3.6
3.5
3.0
7.7
7.8
6.3
13.0
12.7
11.2
EPC (W.E)
5.9
5.9
6.0
10.6
10.6
10.5
16.6
15.7
15.1
EPC (S.E)
6.0
5.9
6.9
10.3
10.2
10.0
15.8
15.2
14.7
Object visitation (# instances /# categories)
Gibson N-D
MP3D
Method
NF
N-D,P
NF
N-D
N-D.P
Reactive (scratch)
6.4/4.0
6.2/
6.3
12.1/4.9
11.5/477
1..6/4.8
RNN (scratch)
6.7
6.4/4.1
6.3/4.0
13.8/5.3
12.1/5.0
12.0/4.9
SMT (scratch)
9.2/5.4
8.8/5.1
8.6/5.1
1..8/6.
16.7/6.1
16.1
SMT (MidLevel)
7.6/4.6
7.4/4.4
7.4444
15.2/5.7
13.9/5.4
13.5/5.3
SMT (MoCo)
9.1
8.8/5.3
8.8/5.3
18.8/6.5
18.2/6.5
18.0/6.5
SMT (Video)
8.7/5.1
8.3/5.0
8.2/5.0
16.4/6.0
15.2/5.8
14.7/5.7
ANS
10.0/5.9
9.2/5.5
6.6
20.6/6.9
18.8/6.6
11.1/
EPC (W.F)
11.9/ 6.5
1..3/6.3
11.2 6.3
24.0/7.6
23.2/7.6
22.2 774
EPC (S.E)
12.0/ 6.6
11.9/6.5
11.8 6.5
24.5/74
23.6/7.6
23.2 / 7.5
RoomNav (SPL / Success)
Gibson N-D
MP3D
Methad
NF
N-D,P
NF
N-D
N-D,P
Reactive (scratch)
0.4/ L.5
0.3/ 1.0
0.3/ 1.1
0.2/ 0.9
0.2/ 0.6
0.2/ 0.7
RNN (scratch)
2.2/ 6.8
2.37 6.2
2.1/ 5.6
1.6 5.0
1.5/ 4.8
1.5/ 4.8
SMT (scratch)
2.3/ 5.0
4.9
1.7/
1.4/ 3.0
1.7/ 3.7
1.6/ 3.5
SMT (MidLevel)
4.8/10.4
4.3/
4.5/ 9.9
2.3/
1.9 4.4
1.7/ 3.9
SMT (MoCo)
8.9/
7.8/18.1
7.7/
5.5 12.7
4.9/110
4.0% 8.7
SMT (Video)
18.1
6.1/16.3
1.85
5.0 12.0
4.3/
3.6/ 8.4
ANS
0.0 0.0
0.07 0.0
0.07 0.0
0.07 0.0
0.07 0.0
0.0 0.0
EPC (W.E)
12.6 28.9
12.2 / 27.2
11.0/ 24.0
217/21.1 8.9 19.8
8.4/18.1
7.5 15.7
EPC (S.E)
13.9 31.5
12.5 / 28.2
10.9 23.7
7.6/
6.7 14.4
Table A4: Comparing robustness to sensor noise on downstream fasks in Gibson and Matterport3D. We inject noise into the depth sensor and/or the odometer sensor. The depth-noisc model consists of disparity- quantization, high-froe figh-frequency noise, and low-frequency distortion (Choi ct al., 2015). The odometry noise model is based on data collected from LoCoBot, and accumulates over time leading to a drift in pose (Ramakrishnan al., 2020; Chaplot et al.. 2020b). Note: NF denotes noise free sensing, N-D denotes noisy depth (and noise- free pose), and N-D.P denotes noisy depth and pose.
22

---

Published as a conference paper at ICLR 2022
coverage
Gibson-S
Gibson-L
Matterport3D
Reactive (scratch)
80
Arei covered (m²²
70
Area covered (m²)
175
(scratch)
60
150
(scrateh)
20
50
100
SMT (MoCa)
SMT (Midl.
15
30
75
(Video)
Area
20
ANS
EPC (W.E)
EPC (S.E)
100
200 300 400 500
100
200 300 400 500
500 250
750
1000
episode steps
Flee
Gibson-S
Gibson-L
Matterport3D
17.5
Reactive (scratch)
Flee distance (m))
Flee distance (m)
10
15.0
KNN (scratch)
(scratch)
4
12.5
SMI (MidLevel)
10.0
SMT (MoCo)
2
Flee
7.5
5.0
SMT (Video)
2.5
EPC (W.E)
0.0
EPC (S.F)
200
400
# episode sleps
Object visitation
Gibson
Matterport3D
12
25
categories
instances
# categories
8
15
10
#
#
100 200 300 400 500
100 200 300 400 500
0
250
500
750
1000
250
500
750
1000
episude sleps
RoomNav
Gibson
Matterport3D
10
20
SPL (x100)
Success %
SPL (x100)
6
%
15
20
6
1 4 6
15
10
10
100 200 300 400 500
100 200 300 400 500
100 200 300 400 500
100 200 300 400 500
# episode steps
Figure A5: We highlight the downstream task performance as a function of episode time on both Matterport3D and Gibson. The legend shown near the top two rows also applics to the bottom two rows. Note that the maximum episode length Gibson scenes is T 500. For MP3D, use maximum episode length of 1000 for all tasks except RoomNav where we use T 500 (following Narasimhan al. (2020)).
Flee (m) MP3D
Object visitation
RoomNav (MP3D)
Method
# categories
# instances
SPL
Success
EPC w/ feat-sampling
12.4
7.1
25.9
9.3
21.3
EPC
13.9
7.5
26.6
10.3
24.4
Table A5: Comparing zone feature averaging vs. sampling. 'EPC w/ fcat-sampling' is a variant of EPC which sets the target zone feature from Eqn. to be the projected visual feature of randomly frame from the transfer results than existing scheme which averages the projected features from all frames belonging to the target zone.
This leads
23


Reviews:
[{"review_id": "vURBXTwiT8", "reviewer": "Reviewer_q87t", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n***S1***: Pre-training aims at exploiting the higher level semantic & geometric features of an environment by predicting a scene representation \\hat(f)_i^u that should not have been observed before, such that lower, image-level interpolation of frames (without reasoning of semantics) is prevented. This is an interesting concept, which is applicable to many reasoning tasks.\n\n***S2***: Good experimental evaluation and ablation study including the SMT architecture with different pre-training techniques. This comparison demonstrates the usefulness of the proposed “environment-level” pretraining.\n\n***S3***: SoTA results on the chosen datasets for various robotic visual navigation tasks.\n\n\n**Weaknesses**\n\n***W1***: Insufficient argumentation and analysis for the proposed zone based masking approach.\n\nIt is argued that a (random) frame based masking “can result in poor representation learning since shared content from nearby unmasked frames can make the prediction task trivial”. A simple spatial sampling of frames would ensure that the visual overlap is minimized. The zone decoder would then become a frame decoder which predicts features for a masked frame, conditioned on the pose of the masked frame. This would still qualify as learning an “environment-level” representation as blending/inpainting is not possible. A comparison to such an approach is missing. The argument that zones result in a higher-level representation is not verified. Therefore, the advantage of using zones over frames in the pre-training is not at all clear.\n\n***W2***: The definition of zones solely on the number of frames does not account for the motion of the camera and thus does not avoid overlapping of zones.\n\nE.g. if the camera stands still for >N frames, then consecutive zones capture the same scene part and zone feature prediction based on the pose seems trivial. A definition of zones based on the camera pose (and not necessarily consecutive frames) would be more convincing, as it aims to minimize scene overlap.\n\nThe paper does not make any analysis of different zone selection strategies and the amount of overlap between seen and unseen zones is unknown. Though, this is important, as the authors argue that their zone based masking is superior to a frame based masking because of the reduced overlap (see W1).\n\n***W3***:  What prevents the zone decoder from ignoring the environment embeddings Epsilon and predicting \\hat(f)_i^u as only an embedding/projection of p_i^u?\nGiven that the supervising f_i^u is built from the RGBD frames AND their poses, see Eq. (2), this seems to be the trivial solution; i.e. the MLP would ignore the visual content X and just rely on the poses. The paper would benefit from explaining the choice of supervision better and in particular point out why the features f are a good choice for supervision.\n\n***W4***: The idea behind the usage of an egocentric view, i.e. usage of relative over absolute poses, is unclear.\n\nIn pre-training the poses of frames of seen zones are relative to the mean pose p_i^u of the unseen zone. When plugging the pre-trained encoder in the SMT architecture poses relative to the query o_t can still be used during training (unclear if this is actually done). Though, during inference o_t changes, which would require to update the relative poses of all previous observations and re-encode them into “environment embeddings”. This seems largely impractical. In contrast, one would only want to encode an observation once which argues for a scene-centric coordinate representation, e.g. use the first frame’s pose as reference.\n\n***W5***: Experimental evaluation on the very same dataset as the baseline SMT work is not provided.\n\n**Comments**\n\n***C1***: In Fig. 3 left it is illustrated that all frames of a zone are fed to a MLP, while it should be per frame to obtain the image-level embeddings. \n\n", "summary_of_the_paper": "The work builds upon the scene memory transformer (SMT, Fang et al., 2019) for visual navigation, which encodes and adds past observations to a memory and uses attention to exploit spatio-temporal dependencies.\n\nThe authors propose a self-supervised pre-training for the observation encoder using an encoder-decoder architecture with the encoded memory as the bottleneck (the decoder is only used for pre-training). Self supervision is achieved by using sequences of observations (posed RGBD images) and splitting them into batches of temporally consecutive frames (termed “zones” in the paper). Considering a subset of the zones as not observed, i.e. not available as input for the network, the goal of the encoder-decoder is to predict the features of an unseen zone from the seen input zones, conditioned on the pose of the unseen zone.\n\nCompared to a frame based masking, the zone (=spatio-temporal consecutive frames) based exclusion of data results in less overlap between seen and unseen scene parts, which is argued to learn higher-level semantic and geometric representation of the 3D environment\n\nContributions:\n- Approach for self supervision based on the prediction of visual-geometric features for unseen scenes.\n- Superior performance to SoTA on the navigation task.\n", "summary_of_the_review": "The paper presents an interesting idea for self-supervised pre-training of the encoder stage of the SMT architecture. The conceptual approach is not limited to visual navigation but applicable to a variety of (3D) reasoning tasks.\n\nThe paper lacks clarity in the details of the approach, see above weaknesses W1-W4. With that I’m concerned that the proposed masked-zone prediction task (which is the core contribution of the paper) is a good choice for handling the pre-training. While the approach achieves SoTA on the visual navigation task there are only limited learnings but numerous uncertainties for the reader why this is so.\n\nI would be willing to increase my rating, if the above weaknesses are addressed and it is shown that the proposed masked-zone pre-training is superior to alternative encoder pre-training approaches..\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZK9w68eeIvx", "reviewer": "Reviewer_dLzX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n+Related work section has been clearly written. It contains not only a complete review of the literature, but also a clear discussion on why the proposed ideas are different from previous works.\n\n+Clarity and the way the article is written are two of its strongest points. Introduction has been nicely written, positioning the work clearly, and highlighting the main contributions. The figures and diagrams are quite elaborate, and help to understand many of the ideas proposed. Sections 3 and 4 do not skimp on details. Overall: good work!\n\n+The experimental evaluation design allows the impact of the proposed ideas to be assessed. Two different datasets are used, multiple baselines are reported, an ablation study with clear questions about possible limitations of the model, etc. Both SMT (MoCo) and SMT(Video) baselines are really informative. I appreciate when the experiments focus on trying to confirm the hypothesis proposed, and not only on beating a state-of-the-art performance.\n\n+As for the results, they look promising. Both WE and SE strategies, combining SMT with EPC, seem to lead the SMT model to provide better results on several problems.\n\nWeaknesses\n\n-Paper's strongest point is not originality. \na) In this work, the scene completion problem is addressed simply using a standard transformer encoder-decoder model. An MLP is employed to obtain image-level embeddings. No significant novelty is found in the formulation of the zone decoder. All these elements are trained end-to-end following a previous contrastive loss learning approach (Gutmann & Hyvärinen, 2010). I focus on judging the novelty of the article, dissecting the EPC module, because this is the main contribution. Overall, EPC is a standard trasformer encoder-decoder trained to learn and embedding of some selected features (those belonging to the marked/masked zones).\nb)The other contribution is also incremental. The encoder learned by EPC has simply been integrated into the SMT model, which is the one that performs all the navigation tasks.\nI would like the authors to offer a clear argument about the novelty of their paper for a conference such as ICLR.\n\n\n-The application of the EPC model to other non-SMT architectures remains unresolved. Although this is claimed in different parts of the paper, NO experiments are provided to support this claim. In my opinion, the proposed model will fit only in transformer-based solutions such as SMT. It would be useful to hear from the authors on this issue (has it been tested with any other architecture?). Perhaps some of the assertions made in the article might be relaxed.\n\n-Something similar occurs with the type of videos used to train the model. To conjecture that any videos of building walks could be used to train the model, and not those provided by the datasets used, is also not supported by the experiments provided.\n\n-There is another aspect for which I also find no evidence in the experiments, and which appears throughout the manuscript. Overall, the paper sells the idea that the proposed module is able to \"predict the spatial structure of 3D environments\". This claim is too ambitious in my humble opinion. The model is able to predict features for those masked zones. From here to understanding that the model predicts the 3D structure of the environment is a big leap. Moreover, no experiments are provided to support this claim. My advise: to revise (and relax) all those affirmations for which no experimental evidence is provided.\n\n\nMinor comments:\n-Figure 2 is difficult to follow, too many details in it.\n-Please, punctuate all the equations. They are part of the text.", "summary_of_the_paper": "This paper focuses on the problem of efficiently training an intelligent embodied agent for navigation purposes.\nTechnically, the manuscript introduces a technique named as Environment-Predictive Coding (EPC). It is a model for learning environment-level representations via self-supervision, just using video walkthroughs with masked zones. \nEPC proposes to learn an environment encoder, using a transformer-based encoding-decoding, that produces environment embeddings and that predicts feature encodings for the unseen areas. This is the main scientific contribution of the paper.\nThis environment encoder is finally integrated in the Scene Memory Transformer (SMT) (Fang et al., 2019) for navigation purposes.\nA thorough experimental evaluation is offered, using Matterport3D (MP3D) (Chang et al., 2017) and Gibson (Xia et al., 2018) for 4 navigation tasks. Reported results seem to explain the benefits of incorporating such a self-supervision mechanism. ", "summary_of_the_review": "I like the work the authors have done with the article. I don't think we have a model of great originality, but rather of an incremental nature. However, the ideas are clearly stated, and the experiments are well designed and executed. I need the authors to defend their position on my criticisms of the novelty of their ideas, and to comment on those parts for which I have not found experimental evidence. Overall: I see here a borderline submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S3qN54Qw5vD", "reviewer": "Reviewer_JRbw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please find below a detailed review of the strengths and weaknesses of the paper.\n\n**Strengths**\n\n- The paper addresses a challenging and pertinent problem i.e. how to learn representations of large environments that can help high level tasks. I think the approach is well-designed. I strongly agree with the hypothesis that pose information should help with learning representations grounded in 3D space, and the paper does a good job at verifying this hypothesis.\n\n- The experimental evaluation is really well done in almost all aspects. The authors have picked very good datasets to evaluate on that are challenging and representative of real-world applications. The tasks they have picked clearly demonstrate the advantages of the epc method and have practical interest. The methods against which the proposed approach are compared are appropriate. The ablation study is also excellent. I think the questions that the authors raise and answer in the ablation study are very relevant and helps to show why the specific components of the method (such as the spatial conditioning) are necessary. Overall, the experimental evaluation is well thought out and demonstrates the advantages of the method.\n\n- The results are very promising. The pretraining using EPC seems to help across all the tasks tested on and the improvement is statistically significant.\n\n\n**Weaknesses**\n\n- The main weakness of the paper is that there are lots of works in computer vision on “place recognition” (see [A]). In general, these methods also groups areas into “zones” and learn representative features of each zone. How would the proposed EPC features compare to those learned by a place recognition method such as [B]?  Also, a very related method is that of [C]. It would be good to include a discussion on this approach and how it compares.\n-  It would be good to show how the zone segmentation affects the performance of the method. Does it matter where the zone boundaries are chosen?\n\n[A] Lowry, Stephanie, et al. \"Visual place recognition: A survey.\" IEEE Transactions on Robotics 32.1 (2015)\n\n[B] Chen, Zetao, et al. \"Deep learning features at scale for visual place recognition.\" 2017 IEEE International Conference on Robotics and Automation (ICRA)  2017.\n\n[C] Lenton, Daniel, et al. \"End-to-End Egospheric Spatial Memory.\" International Conference on Learning Representations (ICLR) 2021.\n\n", "summary_of_the_paper": "The main contribution of this paper is a method for learning a spatial representation of an environment, called environment predictive encoding, that can be used for pre-training a network in an unsupervised manner for downstream embodied tasks such as navigation. The results show that the method significantly improves the sample efficiency of four common downstream tasks.", "summary_of_the_review": "Overall I think this is an excellent paper that presents an idea that is both simple, practical and effective. The method is well validated on two challenging benchmark environments, MP3D and Gibson, and compared against numerous state of the art approaches and achieves great performance. In the response, i would like the authors to address the points in “weaknesses” above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QBCQm-tAskD", "reviewer": "Reviewer_MWw9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper is a resubmission from last years’ ICLR, from what I can see many of the concerns from previous reviews have been addressed, and the paper has been thoroughly revised. To highlight:\n  - Many new baselines are added, SMT(Moco) is an interesting baseline showing that just with image-based representation learning it doesn’t perform so well. ANS is a very competitive baseline for exploration tasks.\n  - More tasks are added, especially the semantic navigation tasks. \n  - Added different exploration strategies \n- With the added experiments and baselines, the experiments and analysis are very thorough\n- The paper is generally well written and easy to follow.\n- The ablation study is very informative, it shows that using spatial conditioning is important for SSL for embodied tasks. It can provide guideline for the general recipe for pretraining embodied agents. \n\nWeaknesses:\n\n- “We learn the representations on a collection of video walkthroughs” is somewhat misleading, since the video walkthrough contains poses and people would think of video walkthroughs as those “house touring videos”. The authors said later “We now realize this process in photorealistic Gibson scenes; we leave leveraging in-the-wild consumer videos as a challenge for future work.” \n- It would be interesting to use the poses from visual odometry, apart from adding noise in ablation study 2. This will further lift the assumptions on walkthrough videos with pose\n- The authors should probably adapt ANS to make it work with the stop action, instead of letting it trivially fail.\n- (minor) In figure 5 different baselines seem to have different steps. Also the SMT (MoCo) doesn't seem to be converged yet.", "summary_of_the_paper": "The authors propose a self-supervised representation learning method called environment predictive coding inspired by the context prediction in other representation learning works. This explicit spatial conditioning encourages learning representations that capture the geometric and semantic regularities of 3D environments. The learned representations can be used for downstream navigation tasks, achieving higher sample efficiency over standard image-representation learning.\n", "summary_of_the_review": "This is a good paper with thorough experimental analysis. It is significantly improved from the previous version and I would recommend accepting it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have done a good job methodologically addressing reviewer concerns. The empirical results are good, and the application impact is clear. There were some concerns about the technical heft of the approach, but there's overall agreement that the effective application to the domain is interesting and done very well. The AC is a bit concerned about the impact of the regularities of the domain used on the results, especially with regard to semantic regularities (homes have very particular regularities). But even without answering this question (it should be discussed in the camera ready though), this paper makes a solid contribution.
Decision: Accept (Poster)