Environment Predictive Coding for Embodied Agents
Authors: Santhosh Kumar Ramakrishnan
Tushar Nagarajan
Ziad Al-Halah
Kristen Grauman
Venue:  (2021)
Keywords: Self-supervised learning
visual navigation

Abstract:  We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work on self-supervised learning for images, we aim to jointly encode a series of images gathered by an agent as it moves about in 3D environments. We learn these representations via a zone prediction task, where we intelligently mask out portions of an agent's trajectory and predict them from the unmasked portions, conditioned on the agent's camera poses. By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigation-oriented tasks. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-art on challenging tasks with only a limited budget of experience.

Under review as a conference paper at ICLR 2021
ENVIRONMENT PREDICTIVE CODING FOR EMBODIED AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce environment predictive coding, a self-supervised approach to learn environment-level representations for embodied agents. In contrast to prior work self-supervised learning for images, aim jointly encode a series of images gathered by an agent as it moves about in 3D environments. We learn these repre- sentations via zone prediction task, where we intelligently mask out portions of an agent's trajectory and predict them from the unmasked portions, conditioned on the agent's 's camera poses. By learning such representations on a collection of videos, we demonstrate successful transfer to multiple downstream navigation- oriented tasks. Our experiments on the photorealistic 3D environments of Gibson and Matterport3D show that our method outperforms the state-of-the-nn chal- lenging tasks with only a limited budget of experience.
INTRODUCTION
In visual navigation tasks, an intelligent embodied agent must move around 3D environment using its stream of egocentric observations to sense objects and obstacles, typically without the benefit of a pre-computed map. Significant recent progress on this problem can be attributed to the availability
of large-scale visually rich 3D datasets (Chang et al. 2017 Xia al., 2018 Straub et al. 2019) developments in high-quality 3D simulators (Anderson et al. 2018b Kolve et al. 2017 Savva et al.)
2019a Xia et al., (020)) and research on deep memory- based architectures that combine geometry and semantics for learning representations of the 3D world (Gupta et al. 2017 Henriques & Vedaldi
2018 Chen et al. 2019, Fang al. 2019 Chaplot al. (020b]e).
Deep reinforcement learning approaches to visual navigation often suffer from sample inefficiency, overfitting, and instability in training. Recent contributions work towards overcoming these limita- tions for various navigation and planning tasks. The key ingredients are learning good image-level
representations (Das et al. 2018: Gordon et al. 2019 Lin et al. 2019 Sax et al., (020), and using modular architectures that combine high-level reasoning, planning, and low-level navigation (Gupta
et al. 2017 Chaplot et al. 2020b Gan et al. 2019 Ramakrishnan et al. 2020a).
Prior work uses supervised image annotations MMrowski al.. 2016 Das et al. 2018 Sax al., 2020) and self-supervision (Gordon et al. [019 Lin et al. 2019) to learn good image representations that are transferrable and improve sample efficiency for embodied tasks. While promising, such learned image representations only encode the scene in the nearby locality. However, embodied agents also need higher-level semantic and geometric representations of their history of observations. grounded in 3D space, in order to reason about the larger environment around them.
Therefore, key question remains: how should an agent moving through a visually rich 3D environ- ment encode its series of egocentric observations? Prior navigation methods build environment-levee representations of observation sequences via memory models such as recurrent neural networks (Wi-
jmans et al. 2020), maps (Henriques & Vedaldi 2018; Chen et al.] 2019 Chaplot et al.) 2020b). episodic memory (Fang et al.) 2019), and topological graphs (Savinov et al. 20188] Chaplot et al.
2020c). However, these approaches typically use hand-coded representations such as occupancy maps (Chen et al. [219 Chaplot et al. [2020b] Ramaakrshaan et 20.0a Karkus et al. 2019 Gan
aal/2000/Chhplot et al. 2019) and semantic labels (Narasimhan et 2020 Chaplot et al. 2020a), specialize them
by learning end-to-end for solving a specific task (Wijmans et al. 2020
Henriques & Vedaldi 2018
Parisotto & Salakhutdinov 2018 Cheng al. 2018 Fang et al.
2019

---

Under review as a conference paper at ICLR 2021
Figure 1: Environment Predictive Coding: During self supervise learning. model is given video walk- throughs various 3D environments. We mask portions out the trajectory (dotted lines) and learn to infer them from the unmasked parts (in red). We specifically mask out all overlapping views in a local neighborhood to limit the content shared with the unmasked views. The resulting EPC encoder builds environment-lewe resentations of the seen content that predictive of the unseen content (marked with "?"",, conditioned on the camera poses. The agent then uses this learned encoder multiple navigational tasks in novel environments.
rep-
In this work, we introduce environment predictive coding (EPC), a self-supervised approach to learn flexible representations of 3D environments that are transferrable to variety of irrigatiionooiented tasks. The key idea is to learn to encode a series of egocentric observations in a 3D environment so be predictive of visual content that the agent has not yet observed. For example, consider an agent that just entered the living room in an unfamiliar house and is searching for a refrigerator. It must be able to predict where the kitchen and reason that is likely contain refrigerator. The proposed EPC model aims to learn representations that capture these natural statistics of real-world environments in a self- supervised fashion, by watching videos recorded by other agents. See Fig.
To this end, we devise self-superviser zone prediction task in which the model learns environment embeddings by watching egocentric view sequences from other agents navigating in 3D environ- ments in pre-collected videos. Specifically, we segment each video into zones of visually and geo- metrically connected views, while ensuring limited overlap across zones in the same video. Then, we randomly mask out zones, and predict the masked views conditioned on both the unmasked zones' views and the masked zones' camera poses. Intuitively, to perform this task successfully, the model needs to reason about the geometry and semanties of the environment to figure out what is missing- We devise a transformerbasee model to infer the masked visual features. Our general strategy can be viewed context prediction task in sequential data (Devlin et al.) 2018 Sun al. 2019b Han et al. butt)bbut, but, very differently, aimed at representing high-level semantic and geometric priors 3D environments to aid embodied agents who in them.
Through extensive experiments on Gibson and Matterport3DD we show that our method achieves good improvements on multiple navigation-oriented tasks compared state-of-the-art models and baselines that learn image-level embeddings.
2 RELATED WORK
Self-supervised visual representation learning: Prior work leverages self-supervision learn image and video representations from large collections of unlabelled data. Image representations attempt proxy tasks such as inpainting (Pathak et al. 2016) and instance discrimination (Oord al. 2018 Chen et al. 2020 He et al.) (020)), while video representation learning leverages signals such as temporal consistency (Wei al.) al.[2018 Fernando et al. 2017 Kim al., 2019) and contrastive predictions (Han et al. 2019] Sun et al. 2019a). The VideoBERT project (Sun et al. 2019a b) jointly learns video and text representations from unannotated videos filling in masked out infor- mation. Dense Predictive Coding (Han et al., [219 2020) Iearns video representations that capture the slow-moving semantics in vidcos. Whereas these methods focus on capturing human activity for video recognition, aim to learn geometric and semantic cues in 3D spaces for embodied agents. Accordingly, unlike the existing video models (Sun et al. 2019a Han et al. 2019), our approach grounded in the 3D relationships between views.
Representation learning via auxiliary tasks for RL: Reinforcement learning approaches often suffer from high sample complexity, sparse rewards, and unstable training. Prior work tackles these
2

---

Under review as a conference paper at ICLR 2021
challenges by using auxiliary tasks for learning image representations (Mirowski al. 2016; Gor-
don et al. 2019 Lin et al. 2019; Shen et al. 2019: Ye et al. 2020). In contrast, we encode image sequences from embodied agents to obtain environment- level representations. Recent work also learns state representations via future prediction and implicit models (Ha & Schmidhuber, [0188] Eslami al., 2018: Gregor al.; 2019, Hatner et al.) 2019 Guo et al.) 2020). In particular, ral rendering approaches achieve impressive reconstructions for arbitrary viewpoints (Eslami et al, 2018 Kumar et al,, 2018). However, unlike our idea, they focus on pixelwise reconstruction, and their success has been limited to synthetically generated environments like DeepMind Lab (Beattie al. 2016). In contrast to any of the above, we use egocentric videos to learn predictive feature encodings of photorealistic 3D environments to capture their naturally occurring regularities.
Scene completion: Past work scene completion performs pixelwise reconstruction of 360 panora- mas (Jayaraman & Grauman 2018 [Raakkishhaan et al. 2019), image inpainting (Pathak et al.)
2016). voxelwise reconstructions of 3D structures and semantics (Song et al.] (017). and image- level extrapolation of depth and semantics (Song al., 2018 Yang et al.) 2019b). Recent work on visual navigation extrapolates maps of room-types (Wu et al. 2019; Narasimhan et al. 2020) and occupancy (Ramakrishnan et al. 2020a). While our approach also motivated by anticipating unseen elements, we learn to extrapolate in a high-dimensional feature space (rather than pixels, voxels, or semantic categories) and in self-supervised manner without relying human annota- tions. Further, the proposed model learns from egocentric video sequences captured by other agents, without assuming access detailed scans of the full 3D environment as in past work.
Learning image representations for navigation: Prior work exploits ImageNet pretraining GGupta al. 2017 Anderson et al. 2018a| Chen al. 2019), mined object relations (Yang et al. 20)9a), video (Chang et al.) 2020), and annotated datasets from various image tasks (Sax et al. [2000] Chap-
lot et al. | 2020cb to aid navigation. While these methods also consider representation learning the context of navigation tasks, they are limited to learning image-level functions for classification and proximity prediction. In contrast, we learn predictive representations for sequences of observations conditioned on the camera poses.
3 APPROACH
We propose environment predictive coding (EPC) to learn self-supervised environment-level repre- sentations (Sec. [.I). To demonstrate the utility of these representations, we integrate them into a transformer-basee navigation architecture and refine them for individual tasks (Sec. 3.2). As we will show in Sec. our approach leads to both better performance and better sample efficiency compared existing approaches.
3.1 ENVIRONMENT PREDICTIVE CODING
Our hypothesis is that it is valuable for embodied agent to learn predictive coding of the envi- ronment. The agent must not just encode the individual views it observes, but also learn to leverage the encoded information anticipate the unseen parts of the environment. Our key idea is that the environment embedding must be predictive of unobserved content, conditioned on the agent's cam- era pose. This equips an agent with the natural priors of 3D environments to quickly perform new tasks, like finding the refrigerator or covering more area.
We propose the proxy task of zone prediction to achieve this goal (see Fig. For this task, use a dataset of egocentric video walkthroughs collected parallely from other agents deployed in various unseen environments (Fig. - top). For each video, we assume access RGB-D, egomotion data, and camera intrinsics. Specifically, our current implementation uses egocentric camera trajectories from photorealistic scanned indoor environments (Gibson (Xia et al. 2018)) to sample the training videos; we leave leveraging in wildh consumer vidco challenge for future work.
We do not assume that the agents who generated those training videos were acting to address a par- ticular navigation task. In particular, their behavior need not be tied to the downstream navigation- oriented tasks for which we test our learned representation. For example, a training video may show agents moving about to maximize their area coverage, whereas the encoder we learn is applicable to an array navigation tasks (as we will demonstrate in Sec. 4.. Furthermore, we assume that the environments in the videos nol accessible for interactive training. In practice, this means
3

---

Under review as a conference paper at ICLR 2021
2,0
Z
Enviroument encoder
g
fuge
fu
f
Zone decoder
fuc
similarity
Zup
232
Pu.
Contrastive loss
Figure 2: We propose the zone prediction task for supe-supe learning of environment embeddings from vidco walkthroughs generated by other agents. Each frame consists of the egocentric view and camera posc (top left). We group the frames in vidco v into seen zones in cyan -Z"...... Z"." and unseen zones in (top row). The zones generated automatically based on viewpoint overlap in 3D transformer-basee
yellow {Z".o.... :.., Z", space (bottom left). Given camera pose pi,, sampled from the unseen zone Zz,; encoder-decoder architecture that generates environment embeddings & from the seen zones, and predicts the feature encoding fu, of Zü, conditioned on the pose pi,, (bottom center). The model is trained to distinguish the positive fü, from negatives the same video (Su.. Jiss well from other videos {LE Tese (bottom right).
that we can parallelly collect data from different robots deployed in a large number of environments, without having to actually train our navigation policy on those environments. These assumptions are much weaker than those made by prior work on imitation learning and behavioral cloning that rely on task-specific data generated from experts (Bojarski al. 2016 Giusti et al. 2016).
Our method works as follows. First, we automatically segment videos into "zones" which contain frames with significant view overlaps. We then perform the self-supervised zone prediction task on the segmented videos. Finally, we incorporate the learned environment encoder into an array of downstream navigation-oriente tasks. We explain each step in detail next.
Zone generation At a glance, one might first consider masking arbitrary individual frames in the training videos. However, doing so is inadequate for representation learning, since unmasked frames having high viewpoint overlap with the masked frame can make its prediction trivial. Instead, our approach masks zones of frames at once. We define a zone be a set of frames the video which share significant overlap in their viewpoints. We also require that the frames across multiple zones share little to no overlap.
To generate these zones, we first cluster frames in the videos based on the amount of pairwise- geometric overlap between views. We estimate the viewpoint overlap (0, 0y) between two frames Oi Oj by measuring their intersection in 3D point clouds obtained by backprojecting depth inputs into 3D space. See Appendix for more details. For a video of length L, we generate a distance matrix D RRLX where Di.j
<b(o,, 0y). We then perform hierarchical agglomerative clustering (Lukasová] 1979) to cluster the video frames into zones based on 7) (see Fig. bottom left). While these zones naturally tend to overlap near their edges, they typically capture disjoint sets of content in the video. Note that the zones segment video trajectories, not floorplan maps, since we do not assume access to the full 3D environment.
Zone prediction task Having segmented the video into zones, prediction task to learn environment embeddings (see Fig. 2 We randomly divide the video v into zones ( (cyan) and unseen zones { ( 1= (yellow), where zone Z is tuple of images and the corresponding camera poses 7, {(o), (1))) Given the zones, and the camera pose from an unseen zone p",;- we need to infer feature encoding of the unseen zone Z To perform this task, we first extract visual features from each RGB-D frame o in the video using pretrained CNNs (see Sec. 3.2). These features are concatenated with the corresponding pose p and projected using an MLP M to obtain the image-level embedding. The target features for the unseen zone Z" are obtained as follows:
present our EPC zone
f
Σ [x.p]eZz,
M(
Zur
4

---

Under review as a conference paper at ICLR 2021
The rationale behind the feature averaging is that we want to predict the high-level visual content of the zone, while ignoring viewpoint specific variations within the zone.
We use Transformer-basee encoder-decoder model to perform this task (Vaswani al. (017). Our model consists of an environment encoder and a zone decoder which infers the zone features (see Fig. 2 bottom). The environment encoder takes in the image-level embeddings M( pl) from the input zones, and performs multi-headed self-attention to generate the environment embeddings E. The zone decoder attends to & using the average camera pose from the unseen zone pa,, and predicts the zone features as follows:
furi ZoneDecoder(E,
We transform all poses in the input zones relative to Pū,; before encoding, which provides the model an egocentric view of the world. The environment encoder, zone decoder, and the projection function M are jointly trained using noise-contrastive estimation (Gutmann Hyvärinen 2010). We use fu,: as the anchor and fa,; from Eqn. as the positive. We sample negatives from other unseen zones in the same video and from all zones in other videos. The loss for the 7th unseen zone in video is: exp(sim(fu.ts (H..))
L = -log
Σ
exp Exp(sim(fussS... + E exp(sim(fus::J)) wifukk
[g] F and is temperature hyperparameter. The idea is predict zone representations that are closer to the ground truth, while being sufficiently different from the negative zones. Since the unseen zones have only limited overlap with the seen zones, the model needs to effectively reason about the geometric and semantic context in the seen zones to differentiate the positive from the negatives. We discourage the model from simply capturing video-specific textures and patterns by sampling negatives from within the same video.
where sim(q, Rt)
9.k
3.2 ENVIRONMENT EMBEDDINGS FOR EMBODIED AGENTS
Having introduced approach learn environment emheddings in self-supervised fashion, now briefly overview how these embeddings are used for agents performing navigation-oriented tasks. To this end, integrate our pre-trained environment encoder into the Scene Memory Trans- former (SMT) (Fang et al. 2019). Our choice of SMT is motivated by the recent successes of transformers in both NLP (Devlin al. 2018) and vision (Sun al., 2019b Fang et al. 2019) However, our idea is potentially applicable to other forms of memory models as well.
We briefly overview the SMT architecture (see Fig. center). consists of a scene memory that stores visual features o and agent poses {pi H generated from the observations seen during an episode. The environment encoder uses self-attention on the history of observations to generate richer set of environment embeddings {e.){=1- At given time-step t + 1, the policy decoder attends to the environment embeddings using the inputs Ot+1. which consist of the visual feature x and agent pose p at time t + 1. The outputs of the policy decoder are used to sample an action d,41 and estimate the value 11+1- We detail each component in the Appendix.
To incorporate our EPC environment embeddings, we modify two key components from the original SMT model. First, and most importantly, we initialize the environment encoder with our pre-trained EPC (see Fig. left). Second, replace the end-to-end trained image encoders with MidLevel features that are known to be useful across a variety of embodied tasks (Sax et al., 2020) (see Fig. 3 right). We consider two visual modalities inputs: RGB and depth. For RGB, we extract features from the pre-trained models in the max-coverage set proposed by Sax et al. (020)). These include surface normals, keypoints, semantic segmentation, and 2.5D segmentation. For depth, we extract features from pre-trained models that predict surface normals and keypoints from depth (Zamir al. 2020). For training the model on a navigation task, we keep the visual features frozen, and only finetune the environment encoder, policy decoder, policy ग, and value function V.
4 EXPERIMENTS
We validate our pre-trained EPC environment embeddings for zone prediction (Sec. 4.1 and mul- tiple downstream tasks that require embodied agent move intelligently through unmapped
5

---

Under review as a conference paper at ICLR 2021
Environment lovel pretraining
Scene memoory transformer
Image-level peetraining
Policy decoder
Environment eraheddings
(2) Image-leve transfer
Iragee chondar
Environment cneoder
kavironieeni-leeel transfer
Image encoder
IFC
Xg
X
Zone-pradiction task
Scene metsory
Mid-level visual tasks
The first level of transfer occurs for the environment-leww representations. We transfer the proposed EPC environment encoder and projection function M that are pre-trained for zone prediction. Right: The second level of transfer occurs for the image- level representations. We transfer pre-trained MidLevel image encoder (Sax el al. 2020 to generate visual features for each input the seene memory. Center: To train the SMT on a task, we keep the visual features frozen, and finetune the environment encoder and projection function M with the rest of the SMT model.
Figure 3: Integrating environment-level pre-training for navigation: Left:
environment (Sec. 4.2p. We evaluate the sensitivity of self--suprvvsed learning to noise in the video data (Sec. 4.3 and assess noise robustness of the learned policies on downstream tasks (Sec.
EXPERIMENTAL SETUP AND TASKS We perform experiments on the Habitat simulator Sav et al. 2019b) with Matterport3DD (MP3) (Chang et al. 2017) and Gibson (Xia et al. 2018)) two challenging and photorealistic 3D datasets with 90 and 500 scanned real-world indoor environ- ments, respectively. Our observation space consists of 171 x 128 RGB-D observations and odometry sensor readings that provide the relative agent pose p = (r. y, #) w.r.t the agent pose at = O. Our action space consists of: MOVE-FORWARD by 25cm, TURN-LEFT by 30", and PURN-RIGHT by 30". For all methods, we assume noise-free actuation and odometry for simplicity.
We use MP3D for interactive RL training, and reserve Gibson for evaluation. We the default train/val/test split for MP3I) (Savva et al. 2019h) for 1000-step episodes. For Gibson, which has smaller environments, lowing prior work (Ramakrishnan et al. 2020a Chaplot et al.) 22020)), we divide results on Gibson into small and large environments. We generate walkthroughs for sell-supervised learning from 332 Gibson training environments. We train SMT(scratch) agent to perform area-cos area-coverage on MP3D. explores starting from multiple locations and gathers the RGB-D and odometer readings for 500 steps per video. Note that this agent only collects data, and is not used for downstream tasks. This results in ~ 5000 videos, which we divide into an 80-20 train/val split.
evaluate the 14 validation environments for 500-step episodes. Fol-
We evaluate our approach on three standard tasks from the literature:
Area coverage (Chen et al. 2019 Chaplot et al., 2020b Ramakrishnan et al. 2020b): The agent is rewarded for maximizing the area covered (in m²) within fixed time budget.
Flee (Gordon et al. 2019): The agent is rewarded for maximizing the flee distance (in m), i.e., the
geodesic distance between its starting location and the terminal location, for fixed-length episodes.
3. Object coverage (Fang et al. 2019: Ramakrishnan et al. 2020by: The agent is rewarded for maximizing the number of categories of objects covered during exploration (see Appendix). Since Gibson lacks extensive object annotations, we evaluate this task only MP3D.
Together, these tasks capture different forms geometric and semantic inference in 3D environ- ments (e.g., area/object coverage encourage finding large open spaces/new objects, respectively).
BASELINES
We compare to the following basclines:
Scratch baselines: We randomly initialize the visual encoders and policy and train them end-to- end for each task. Images are encoded using ResNet-18. Agent pose and past actions are encoded using FC layers. These are concatenated to obtain the features at each time step. We use three temporal aggregation schemes. Reactive (scratch) has memory. RNN (scratch) uses 2-layer LSTM as the temporal memory. SMT (scratch) uses a Scene Memory Transformer for aggregating observations (Fang et al. 2019).

---

Under review as a conference paper at ICLR 2021
4-2 split
2-4 split
Method
w/ inputs
w/o imputs
w/ inputs
w/o inputs
Nearest neighbors
0.033
0.591
0.011
0.295
Rundom masking
0.332
0.760
0.209
0.449
EPC (bilincar)
0.452
0.806
0.321 0.368
0,518
EPC (f2-norm)
0.528
0.825
0.556
Table I: Zone prediction: Top-1 zone prediction accuracy on the validation walkthrough videos.
3D environment
Intra-video retrieval
Inter-video retrieval
0L954
0.044
0.002
0.818
6.148
0.033
Figure 4: Each row shows one zone prediction example. Left: Top-down view of the 3D environment from which the video sampled. The cyan viewing frusta correspond to the average pose for three input zones. Given the images and camera poses from each input zone, and target camera pose (green frustum), the model predicts the corresponding zone feature (the masked green zone). Center: Given the inferred feature, we rank three masked (unobserved) zones from within the same vidco, where green is the positive zone and the red are the negatives. For each zone, show four randomly sampled images along with the retrieval confidence. Our method retrieves the positive with high confidence. The model correctly predicts the existence of the narrow corridor (top row) and a kitchen counter (bottom row) given the target poses. Right: Top two retrieved zones from other videos that are closest to the inferred feature. The features predicted by the model are general enough to retrieve related concepts from other videos (narrow corridors and kitchens).
SMT (MidLevel): extracts image features from pre-trained encoders that solve various mid-level perceptual tasks (Sax et al. 2020). This is an ablation of our model from Sec. 3.2 that uses the same image features, but randomly initializes the environment encoder. This SoTA image-level encoder is a critical baseline to show the impact of our proposed EPC environment-leve encoder.
SMT (Video): Inspired by Dense Predictive Coding (Han et al.) 2019), this baseline uses MidLevel features and pre-trains the environment encoder as a level model using the same training videos as our model. For pre-training, we randomly sample 25 consecutive frames as inputs and predict the average features corresponding to the next 15 frames. We query based on the time (not pose) and train the model using the NCE loss in Eqn.
DccupancyMemory, This similar the SoTA Active Neural SLAM model (Chaplot et al. 2020b) that maximizes area coverage, but using ground truund-truth truth depth to build the map (instead of RGB) and a state-of-the-art pointnav agent (Wijmans et al. 2020) for low-level navigation (instead of planner). It represents the environment as a top-down occupancy map.
All models are trained in PyTorch (Paszke et al., (019) with DD-PPO (Wijmans et al. 2020) for 15M frames with 64 parallel processes and the Adam optimizer. See Appendix.
4.1 ZONE PREDICTION PERFORMANCE
First we evaluate the EPC embedding quality in terms of zone prediction on the validation videos. We divide each video into m seen and n unseen zones and infer the features for each unseen zone, given its average camera pose. We rank the features from the unseen zones based on their similarity with the inferred feature, and measure the top-1 retrieval accuracy. We evaluate with (m = 4, n = 2) and (m 2, n 4) splits. The larger the value of m, the easier the task, since more information available as input. We also test two simple baselines. Nearest neighbors uses the query pose to retrieve the 50 closest frames in the input zones, and outputs their averaged features. Random
7

---

Under review as a conference paper at ICLR 2021
Area coverage (m²)
Flee (m)
coverage (#obj)
Method
Gibson-S
Gibson-L
MP3D
Gibson-8
Gibson-L
MP3D
MP3D-cat.
MP3D-inst.
Reactive (scratch)
17.4
0.6
68.0
1.1
2.5
6.2 6
19.0
RNN (scratch)
20.6 +: 0.5
0.3
2.11
2.3
2.4
5.9
6.1
1..
SMT (scratch)
23.0 0.7
0.8
3.2
4.4
6.9
7.0
23.2
SMT (MidLevel)
0.1
1.7
15
4.200.
6.0
0.3
7.6
2.6
SMT (Video)
2.5
50.6
129.7
4.1
5.0
1..9
7.3
25.4 1.0
OccmpancyMemoory
29.4
0.9
155.6+14
2.8
0.0
14.1
7.8
27.8
Ours (EPC)
29.9
2.14
18
4.1
7.1
1.6
8.6
0.8
Table 2: Downstream task performance at the end of the episode. Gibson means small/large. MP3D- cat./inst. means categories/instnnes. All methods evaluated three random seeds. See Appendix for performance time step plots.
Area coverage
Flee
Object coverage
SMT (Midl evell
Area covered (m²)
Flee distance (m)
10
Ours (EPC)
10
# categories
instances
80
30
26
60
8
12
8 12
0
8
12
22
12
training experience (in million frumes)
Figure 5: Sample efficiency Matterport3D val split. Our environment-lovel pre-training leads to 2-4x
training sample efficiency when compared SoTA image-level pre-training. See Appendix for Gihson plots.
masking uses a different proxy task to learn the environment representations, randomly masking 10 consecutive frames in the video and predicting their averaged feature from the rest. EPC (bilinear) uses bilinear product similarity (Oord et al. 2018) instead of the f--nrrm below Equ. 3 We report retrieval from only the unseen zones (w/o inputs) well as the more challenging case where input zones are also candidates (w/inputs).
Tab. D shows the results. The EPC (f2-norm) model obtains superior retrieval performance on both settings. It retrieves the positive zones with high confidence (see Fig. 4and Appendix). EPC's gain over random masking shows the value of the proposed zone generation step. Therefore, we select this model for downstream task transfer.
4.2 DOWNSTREAM TASK PERFORMANCE
Now we transfer these features to downstream navigation tasks. Tab. (lb.22shen shows the results. On both datasets, observe the following ordering:
Reactive (scratch) RNN (scratch)
SMT (scratch).
(4)
This is in line with results reported by Fang et al. (2019) and verifies our implementation of SMT. Using MidLevel features for SMT leads to significant gains in performance versus training image encoders from scratch.
Our environment-level pre-training provides substantial improvements compared SMT (Mi- dLevel), particularly for larger environments. Furthermore, SMT (Video)-the the video-level pre- often training strategy- often deteriorates performance compared to using only image-level trainin.. This highlights EPC's value in representing the underlying 3D spaces of the walkthroughs instead of treating them simply as video frames. EPC competes closely and even slightly outperforms the state-of-the-art MecupancyMemory metrics. Thus, our model competes strongly with a task-specific representation model on the tasks that the latter was designed for, while outperforming it significantly other tasks.
these tasks, with significant gain on the object coverage
Finally, Fig. 5 shows that EPC offers better sample efficiency than image-only pre- ve-training: our method reaches the best performance of SMT (MidLevel) 2-4x faster. This confirms our hypothesis: transferring environment-lovw level representations learned via contextual reasoning can help embodied agents learn faster compared the current approach of transferring image-level encoders alone.
8

---

Under review as a conference paper at ICLR 2021
Arca coverage (m")
Floe (m)
Object coverage (#ob))
Method
Gibson-S
Gibson-L
MP3D
Gibson-S
Gibson-L
MP3D
MP3D-car.
MP3D-inst.
SMT (MidLevel)
2.1
47.2 1.7
135.7
4.2
6.4
10.6 0.3
7.6
26,8 = 0.6
EPC
0.3
56.4 2.1
2.
0.1
7.1
128=06
8.1
0..
EPC w/ noisy depth
30.6 + 0.4
58.0 3.8
163.2+33.3
4.8
+.7
1300005
8.3+0.0
31.8 0.0
EPC w/ noisy depth and pose
0.3
56.4 1.8
0.
44+00
0.7
124=04
78
28.9
EPC w/ simple heuristic
30.1 0.2
50.9 + 1.6
108.2 3.1
4.6
7
12.3 0.4
8.2 0.1
32.5 + 0.2
Table 3: Impact of noisy video data and non-lcarned policy for video generation on EPC self-supervised learning.
coverage (m")
Flee (m)
Object (fcat.)
Method
NF
N-D
N-D.P
NF
N-D
N-D.P
NF
N-D
N-D.P
SMT (MidLevel)
155.7 = 2.0
145.1 = 23
134.2=12
10.+ 0.3
1066 0.6
76-02
7.3+0.1
73++02
MemmpnncyMmmory
155.6
24
10.9 146
1.
78=01
5.8000
0.00000
EPC
165.6 d 2.8
148.4 +
152.3 2:2
12.8 0.6
12.2 0.1
11.4
8.6 0.1
8.4
8.4 0.2
Table 4: Comparing robustness sensor noise on downstrcam tasks in Matterport3D. Note: NF denotes noise free sensing, N-D denotes noisy depth (and noise-free pose), and N-D,P denotes noisy depth pose. See Appendix G for full results.
4.3 SENSITIVITY ANALYSIS OF SUPERVIDDD LEARNING
We analyze the sensitivity EPC to sensory noise in the videos, and the exploration strategy used for video data collection. Specifically, we inject noise in the depth and pose data from the videos using existing noise models from Choi et al. (015)) and [Ramakrishnan the video walkthroughs from the area-coverage agent with equivalent amount of data collected by a simple heuristic used prior work (Chen et al. 2019 Ramakrishnan et al., 2020b). works as follows: move forward until colliding, then turn left right by random amount, then continue moving forward. We evaluate the impact of these changes on the downstream task performance.
al.] (2020ap. We also replace
See Tab. 3 Our approach EPC is reasonably robust changes in the video data during SSL training. The performance remains stable when noise is injected into depth inputs. While it starts to decline on MP3D when further inject noise into pose inputs, EPC still generally outperforms the random initialization of environment-encodee in SMT (MidLevel). Note that we do not employ any noise- correction mechanism, which could better limit this decline (Chaplot et al.) 2020b Ramakrishnan ett al. 2020a). Finally, the performance is not significantly impacted when we use video data from a simple exploration heuristic, showing that EPC does not require a strong exploration policy for the agent that generates the self-supervised training videos. nor does it require tight similarity between the tasks demonstrated in the videos and the downstream tasks.
4.4 ROBUSTNESS OF LEARNED POLICIES TO SENSOR NOISE
In previous experiments, we assumed the availability of ground-truth depth and pose sensors for downstream tasks. Now, we relax these assumptions and re-evaluate all methods by injecting noise in the depth and pose sensors for downstream tasks (same noise models as Sec. 4.3 without any noise-correction. This is a common evaluation protocol for assessing noise robustness (Chen al. 2019 Ramakrishnan et al. 2020b). We compare the top three methods on MP3D in Tab. 4 and provide the complete set of results in Appendix[G] As expected, the performance declines slightly as we add noise to more sensors (depth, then pose). However, most learned approaches are reasonably stable. EPC outperforms all methods when all noise sources are added. OccupancyMemory declines rapidly in the absence of noise-correction due to accumulated errors in the map.
CONCLUSIONS
We introduced Environment Predictive Coding, a self-supervised approach to learn environment- level representations for embodied agents. By training video walkthroughs generated by other agents, our model learns to infer missing content through a zone-preed pone-iredittion task. When transferred to multiple downstream embodied agent tasks, the resulting embeddings lead to better performance and sample-efficiena compared to the current practice of transferring only image-level representations. In future work, plan to extend our idea for goal-driven tasks like PointNav and ObjectNav.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton den Hengel. Vision-and- languuge navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. rision-anddlanguaee navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2018b.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.0380) 2016. URL http:://rxxt http://wwwwo.////a//////o/ abs/ / /6/ 612.038011
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint 160..0360 2016.
Angel Chang. Angela Dai, Tom Funkhouser, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In Proceedings of the International Conference on 3D Vision (3DV), 2017. MatterPort3D dataset vc. in. tum /t/ptttteeertttt//////// /MP_TOSPPPPPPP
license available at: http::/kakll
_OS _
Matthew Chang, Arjun Gupta, and Saurabh Gupta. Semantic visual navigation by watching youtube videos. arXiv preprint arXiv:2006. 10034, 2020.
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. arXiv preprint arXiv:2007.00643 2020a.
Devendra Singh Chaplot, Saurabh Gupta, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural mapping. 8th International Conference Learning Representations, ICLR 2020, 2020b.
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topo- logical slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12875-12884, 2020c.
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.0557 05709, 2020.
Ricson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry aware recurrent neural networks for active visual recognition. In Advances in Neural Information Processing Systems, pp. 5081- 5091, 2018.
Sungjoon Choi, Qian-Yi Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5556 5565, 2015.
Abhishek Das. Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruy Batra. Embod- ied question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2054-2063, 2018
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 04800, 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 12446199) 2018.
10

---

Under review as a conference paper at ICLR 2021
Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 538-547, 2019.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video rep- resentation learning with odd-one-out networks. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 3636-3645, 2017.
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. arXiv preprini arXiv: arXiv:1912.11664 2019.
Alessandro Giusti, Jéröme Guzzi, Dan C Cireşan, Fang-Lin He, Juan Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 2016.
Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffman, and Dhruv Batra. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. 2019.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping Belief States with Generative Environment Models for RL. 2019.
Daniel Guo, Bernardo Avila Pires, Bilal Piot, Mohammad Gheshlaghi Azar. Bootstrap latent-predictive representations for multitask reinforce- ment learning. arXiv preprint arXiv:2004. 14646, 2020.
Grill, Florent Altché, Rémi Munos, and
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625,, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2460-2462, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Tengda Han, Weidi Xie, and Andrew Zisserman. augmete- dense predictive coding for video representation learning. Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Joao F Henriques and Andrea Vedaldi. Mapnet: allocentric spatial memory for mapping envi- ronments. In proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8476-8484, 2018
Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. In Computer Vision and Pattern Recognition, 2018 IEEE Con- ference on, 2018.
Peter Karkus, Xiao Ma, David Hsu, Leslie Pack Kaelbling, Wee Sun Lee, and Tomás Lozano- Pérez. Differentiable algorithm networks for composable robot learning. arXiv preprint arXiv: 1905. 1602, 2019.
11

---

Under review as a conference paper at ICLR 2021
Dahun Kim, Donghyeon Cho, and In So Kweon. Self-super video representation learning with space-time cubic puzzles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8545- 2019.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Ananya Kumar, SM Eslami, Danilo J Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint 2018.
Xingyu Lin, Harjatin Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. 2019.
Alena Lukasová. Hierarchical agglomerative clustering procedure. Pattern Recognition, 11(5-6): 365-381, 1979.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. CoRR, 0bs/1611.03673 2016. URL
http://arxiv. Btrgg/bb/11110 03673
Medhini Narasimhan, Erik Wijmans, Xinlei Chen, Trevor Darrell, Dhruv Batra, Devi Parikh, and Amanpreet Singh. Seeing the un-scene: Learning amodal semantic maps for room navigation. arXiv preprint Xiiv:2007.09841, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. In International Conference on Learning Representations, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: : / wwwpppaapprspssppppp wtw.::s..nnrrpss.cc.sss.
901 http:p/www....oot.e..eeeeeaaaaaa
Aibrar..
pdf
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. 2016.
Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. Emergence of exploratory
look-a behaviors through active observation completion. Science Roboties, 4(30), 2019. doi: 10. 0..1126/scirobotissaawwii URL
4/30/eaaw6326
Santhosh K. Ramakrishnan, Ziad AI-Halah, and Kristen Grauman. Occupancy anticipation for effi cient exploration and navigation, 2020a.
Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. An exploration of embodied visual exploration, 2020b.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-paraments topological memory for navigation. In International Conference on Learning Representations, 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2019a.
12

---

Under review as a conference paper at ICLR 2021
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9339- 2019b.
Alexander Sax, Jeffrey O Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik. Learning to navigate using mid-level visual priors. In Conference on Robot Learning, pp. 791-812, 2020.
William B Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situa- tional fusion of visual representation for visual navigation. 2019.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser.
Semantic scene completion from single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Shuran Song, Andy Zeng, Angel X Chang, Manolis Savva, Silvio Savarese, and Thomas Funkhouser. Im2pano3d: Extrapolating 360 structure and semantics beyond the field of view. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3847- 3856, 2018
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv: 1906.05777 2019.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint 13Xi::9906005455 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBert: joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7464-7473, 2019b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the ar- row of time. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 0052-8060, 2018.
Erik Wijmans, Abhishek Kadian, Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. 2020.
Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian relational memory for semantic visual navigation. In Proceedings of the IEEE International Con- ference Computer Vision, 2019.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9068 9068-9079, 2018. Gibson dataset license agreement available httpt:://// htttps://storage www.eeoooogggggoooooooo www gibsor.
material/ naterial//grremmntt w2w.rreeeeennttttsssssss S20GDS 04-18.pd1.
Fei Xia, William Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martin and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):
713-720, 2020.
13

---

Under review as a conference paper at ICLR 2021
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual seman- tic navigation using scene priors, 2019a. URL https://openrevieww net tareeeffoummi forum?ir HJeRkh05Km
Zhenpei Yang, Jeffrey Z. Pan, Linjie Luo, Xiaowei Zhou, Kristen Grauman, and Qixing Huang. Extreme relative pose estimation for rgb-d scans via scene completion. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019b.
Joel Ye, Dhruv Batra, Erik Wijmans, and Abhishek Das. Auxiliary tasks speed up learning pointgoal navigation. arXiv preprint wwXiv:2007.04561, 2020.
Amir R Zamir, Alexander Sax, Nikhil Cheerla, Rohan Suri, Zhangjie Cao, Jitendra Malik, and Leonidas Guibas. Robust learning through cross-task consistency. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 11 11197-11206, 2020.
Appendix
A ZONE GENERATION
As discussed in the main paper, we generate zones by first clustering frames in the video based on their geometric overlap. Here, we provide details on how this overlap estimated. First, we project pixels in the image to 3D point-clouds using the camera intrinsics and the agent pose. Let Di. Pi be the depth map and agent pose for frame in the video. The agent's 's pose frame can be expressed as p4 (R, f;), with R. la representing the agent's camera rotation and translation in the world coordinates. Let K E R3x3 be the intrinsic camera matrix, which assumed to be provided for each video. We then project each pixel Zij in the depth map D; to the 3D point cloud follows:
1 R. o
Wy =
Tigs Vj E {1....,
(5)
where St is the total number of pixels in Dj. By doing this operation for each pixel, we can obtain the point-cloud W; corresponding to the depth map D, To compute the geometric overlap between two frames and j. we estimate the overlap their point-clouds W; and W,. Specifically, for each point w; E W, we retrieve the nearest neighbor from w/ E W, and check whether the pairwise distance in 3D space within a threshold т: I|w; - will2 || T If this condition satisfied, then match exists for wi. Then, we define the overlap fraction (bDD,, D,) the fraction of points in W; which have a match in W,. This overlap fraction computed pairwise between all frames in the video, and hierarchical agglomerative clustering is performed using this similarity measure.
B
TASK DETAILS
For the object coverage task, determine if an object is covered, we check if it is within 3m of the agent, present in the agent's 's field of view, and if it is not occluded (Ramakrishnan et al. (0200). We shaped reward function:
R - - 0.02 - -C-1),
(6)
where O., C. are the number of object categories and 2D grid-cells visited by time t (similar to Fang et al. 2219)).
C SCENE MEMORY TRANSFORMER
We provide more details about individual components of the Scene Memory Transformer Fang et al. 2019). As discussed in the main paper, the SMT model consists of a scene memory for storing the visual features {2: H and agent poses {p, {p.)--m J seen during an episode. The environment encoder uses self-attention on the scene memory to generate a richer set environment embeddings {e, M= The policy decoder attends the environment embeddings using the inputs ot+1, which consist of
14

---

Under review as a conference paper at ICLR 2021
the visual feature 2, and agent pose time t + 1. The outputs of the policy decoder are used to sample an action a1+1 and estimate the value 0141- Next, we discuss the details of the individual components.
SCENE MEMORY stores the visual features derived from the input images and the agent poses at each time-step. Motivated by the ideas from Sax al. (2020), we mid-level features derived from various pre-trained CNNs for each input modality. In this work, we consider two imput modalities: RGB, and depth. For RGB inputs, coverage set proposed in Sax et al. (2020). These include surface normals, keypoints, semantic segmentation, and 2.5D segmentation. For depth inputs, extract features from pre- treitrained models that predict surface normals and keypoints from depth (Zamir et al.) 2020). For simplicity, we assume that the ground-truth pose is available to the agent in the form of (24,, yt. Ze; 0t) at each time-step, where 6g is the agent heading. While this can be relaxed by following ideas from state-of- the-art approaches to Neural SLAM (Chaplot al. 2020b Ramakrishnan al. 2020a), we reserve this for future work as it is orthogonal to our primary contributions.
extract features from the pre-trained models in the max-
ATTENTION MECHANISM Following the notations from Vaswani al. (2017), define the attention mechanism used in the environment encoder and policy decoder. Given two inputs X E Rmixdd and Y Ruzxdy, the attention mechanism attends Y using X follows:
(xKY ddz
Attn(X, Y) = softmax
Vy
where Qx E Rn1xde Ky E Reaxde Vy R are the queries, keys, and values computed from X and Y as follows: Qx XW", Ky YW* and Vy = YW". W9, W W are learned weight matrices. The multi-headed version of Attn generates multiple sets of queries, keys, and values to obtain the attended context C' E 1 Xd.
MHAttn X, Y) = (X,
(8)
We use the transformer implementation from PyTorch (Paszke et al. 2019). Here, the multi-headed attention block builds on top of MHAttn by using residual connections, LayerNorm (LN) and fully connected (FC) layers to further encode the inputs.
MHAunBlock(X, Y) = LN(MLP(H) + H)
(9)
where // LN(MHAmn(X. Y) + X), and MLP has 2 FC layers with ReLU activations. The environment encoder performs self-attention between the features stored in the scene memory to obtain the environment encoding E.
E EnniroomeentFncoder((MM MHAutBlock((M M).
(10)
The policy decoder attends the environment encodings E using the current observation Zt, Pt.
PolicyDecoder((1z, pt), E) MHAAtnBlook(FF((((rr.)))) E)
(11)
We transform the pose vectors {pr | from the scene memory relative to the current agent pose Pr
this allows the agent maintain egocentric view of past inputs Fang et al. (2019).
15

---

Under review as a conference paper at ICLR 2021
D HYPERPARAMETERS
We detail the list of hyperparameter choices for different tasks and models in Tab. [ For the random masking baseline in Tab. [ we tried masking out 10, 20 and 50 frames and picked 10 frames based on the zone prediction performance. For SMT (Video), we choose 25 frames as inputs and frames as output based on Dense Predictive Coding Han et al. (2019).
RL Optimization
Optimizer
Adam
Learning rate # parallel actors
0.00025
0.001
64
PPO mini-batches
2
PPO epochs
2
PPO clip param
0,2
Value loss coefficient
0.5
Entropy coefficient
0.01
Advantage estimation
GAE
Normalized advantage?
Yes
Training episode length
1000
GRU history length
128
# training steps (in millions)
15
RNN hyperparameters
Hidden size
128
RNN type
LSTM 2
Num recurrent layers
SMT hyperparameters
Hidden size
128
Scene memory length
500
#
attention heads
8
# encoder layers
1
# decoder layers
1
Occupancy memory hyperparameters
Action space range
48m x 48m
# global action sampling interval
25
Reward scaling factors for different tasks
Task
Reward scale
Area coverage
0.3
Flee
1.0
Object coverage
1.0
Self-supervised learning optimization
Optimizer
Adam
Learning rate
0.0001
Video batch size
20
Temperature (r)
0.1
Table 5: Hyperparameters for training our RL and self-supervised learning models.
E DOWNSTREAM TASK PERFORMANCE VS. EPISODE TIME
We show the downstream task performance as function of time in Fig. 6 We evaluate each model with 3 different random seeds and report the mean and the 95% confidence interval in the plots.
16

---

Under review as a conference paper at ICLR 2021
MP3D
Area coverage
Flee
Object coverage
Arca covered (m²)
175
# categories
# instances
34
125
Flee distance
100
20
250 500 1000
500 250
750 1800
750
250
Gibson-1.
Gibson-S
Area coverage
Flee
Arca coverage
Flee
Arca covered (ml)
60
Area covered (m=)
Flee distance (m)
50
Flee distance
25
40
20
.30
15
10
100 200 400 500
100 300-0000000000000000000000000000 400 50X
100 200 300 400 500
cpisode steps
Reactive (scratch)
SMT (scratch)
SMT (Video)
Ours (EPC)
(scratch)
SMT (MidLevel)
OccupancyMemorry
Figure 6: We highlight the downstream task performance as a function of episode time on both Matterport3 and Gibson.
Area coverage
Flee
covered (m²
37.5
Flee distance (m)
5.5
SMT (MidLevel)
5.0
Ours (EPC)
32.5
30.0
4,0
27.5
25.0
3.0
22.5
20.0
2.0
12
12
training experience million frames)
Figure 7: Sample efficiency on Gibson val split. Our environment-level pre-training leads to 2-4x training sample efficiency when compared to SoTA image-level pre-training.
F SAMPLE EFFICIENCY CURVES ON GIBSON
We plot the Gibson validation performance function of training experience in Fig. EPC achieves better sample efficiency through environment-level pre-training when compared to the image-level pre-training baseline SMT (MidLevel).
G COMPLETE ANALYSIS OF NOISE ROBUSTNESS IN DOWNSTREAM TASKS
In Tab, from the main paper, we compared the noise robustness of top three approaches on MP3D. Here, we present the complete set of results for all methods on Gibson and MP3D in Tab. 6
H EXPLORING SPATIAL CONTEXT FOR SELF-SUPERVISION IN EPC
Originally, our EPC proposal previews context from large parts of the video (spanning several zones) to fill in the content for missing zone (termed "EPC-global") However, we can also leverage local spatial context spanning limited set of frames. The SMT (Video) baseline exploits local temporal context which spanned 25 + 15 frames to derive self-supervision. Now, we consider a local variant of EPC that performs spatial reasoning within similar context, i.e., takes 25 frames their poses
17

---

Under review as a conference paper at ICLR 2021
Matterport3D Flce (m)
Arca coverage (m²)
Object (Acat.)
N-D
Reactive (scratch)
68.0 1.3
1.4
65.7
1.5
5.1
5.3
5.3
6.2
6.0
6,0 010
RNN (scratch)
2.0
7..0
T3.4
5.9
5.9
6.0
6.0000
5.9
1.9
SMT (scratih)
1018 222
10
99.2 2.9
6.9
6.6
7.4002
6880..1
6.7
SMT (MidLevel)
155.7 = 2.0
/45.14:23
1
10.6
10.6.10..6
10.8
7.8
7.340..1
7.3
SMT (Video)
2.8
1.88
10.9004
9.
1.2000
7.2+
SecupancMeemng
155.6 1.4
86.6 2.2
85.2 =
14.1 0.6
4..9
1..
7.8
0.1
5.0+0.0
5.8 0.0
EPC
165.6
148.4 1.4
152.8 +
128
12.2
11.4002
8.6 0.1
8.4
8.4 8.4
Gibson-S
Area coverage (m²)
Flee (m)
Object cov. (ftat.)
Method
NF
N-D
NF
N-D
N-D.P
Reactive (scratch))
17.4 0.2
17.8 0.4
17.8 = 0.4
191
181
1810.1
RNN (scratch)
20.6 0.1
21.5
216 0.2
232
2.2002
22=022
SMT (scratcc)
23.0=
23
23.40.4
33=022
33
28 28=0.1
SMT (MidLevel)
29.1
30.5
30.8 066
1.2000
4.1
3.4
SMT (Video)
28.7
30.0
0.4
4.1
3.6
3.=
OccupuncMMemrry
29.4 0.0
30.8
0.3
90.6 0.2
28
3.1
3.0 0.2
EPC
29.9 + 0.3
31.8 0.1
31.6
0.2
4.5 0.1
4.2 0.2
4.2
Area coverage (m²)
Flee (m)
Object cov. (fcat)
Method
NF
N-D
N-D.P
N-D
N-D.P
NF
N-D
N-DP
Renctive (sematch)
22.8
0.6
22.4 0.2
22.4
2
2.6
= 0.4
RNN (scratch)
28.6 0.3
27.9
26.2 2.
2.
2.7
28=044
(scritch)
32.3
32.6
4.6+0.2
SMT (MidLevel)
17.2 1.6
19.2
162 28
6.0
5.
5.1
SMT (Video)
50.6
50.41:14
47.5
5.0
5.210.4
1.4
OccupancyMemory
67.4=09
5.
56.9
0.8
70
6.9
6.9
EPC
2..4
55.8 1 0.6
55.0 - 0.8
7.1
7.0 0.6
7.6.
Table Comparing robustness to sensor noise on downstream tasks in Gibson Matterport3D, Note: denotes noise free sensing, N-D denotes noisy depth (and noise-free pose), and N-D,P denotes noisy depth and pose.
Area coverage (m?)
Flee (m)
Object coverage (Nobj)
Method
GibssonSS
Gibson-L 50.6 2.6
MP3D
Gibson-S
Gibson-L
MP3D
MP3D-cat.
MP3D-inst.
SMT (Video)
28.7 + 0.5
129.7 + 2.8
4.1 1.1+00
+.0
10.9 0..4
7.3 + 0.2
25.4 1.0
EPC-local
30.9 0.2
165.0 58.0
4.7
7.8
1.9
8.2
32.6 0.6
EPC-global
3..0
56.12.1
165.6
4.110.1
7.1
1.6
8.6
34.5 0.8
EPC-local
0.2
1.
1.
4.1
7.5
1226:02
8.4
32.9
EPC-global S
31.5 0.1
62.2 1.0
17244206
4.4
8.0
12.6
8.9
36.4
EPC aug. +
31.8 = 0.2
68.0 + 1.6
179.8 +: 1.4
4.4 1.2::.2
7.4 0.2
13.2 2: 0.3
9.0 0.0
37.3 + 0.0
Table 7: Exploring spatial context for sclf-supervision in EPC
inputs, and predicts the average feature for the next 15 frames conditioned the pose. We term this as "EPC-local". We compare the two EPC variants with SMT (Video) in Tab. 7
As expected, both EPC variants outperform SMT (Video) by large margin, validating the main hy- pothesis in EPC that spatial reasoning during self-supervision is critical. However, at a first glance, appears that EPC-global only offers limited advantage over EPC-local. Our analysis reveals that EPC-global is bottlenecked by the averaging of zone features during self-supervison (Eqn. []. Each zone typically contains anywhere from 305 frames (mean of 48 frames), and averaging them reduces the self-supervision available per video. To test this hypothesis, we make a simple change where we replace feature averaging with sampling, i.e., we sample a random frame from the masked zone as the prediction target in Eqn. [ The performance of this new "sampling-based" zone rep- resentation is shown in Tab. 7 (denoted as "+ S*'). As expected, removing the feature averaging improves both the EPC variants. We see larger improvements in EPC-global since lot more frames were averaged over in this case (i.c., more information lost). As we noted, these two variants capture two types of contextual cues: local and global which could be complementary. To test this, combine the two losses during SSL training (EPC aug. + S). This model generally outperforms the individual methods, confirming intuition that we derive complementary cues from local and global context.
18


Reviews:
[{"review_id": "vWNhTVVmD9J", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a self-supervised method to learn environment-level representations for embodied agents. The representations are learned using a zone prediction task. This task involves predicting features of the masked portions of input trajectories based on unmasked portions. Results on downstream navigation tasks show an improvement over a range of baselines.\n\nStrengths:\n- The method is novel and original to the best of my knowledge. Although it is demonstrated to be useful for exploration tasks, it can potentially be useful for different types of embodied tasks.\n- The paper is written well and easy to follow in most places.\n- The authors have done an incredible job of reviewing related work comprehensively and discussing the proposed approach in the context of prior work.\n- The authors also compare the proposed method with a wide range of useful baselines.\n\nWeaknesses:\n- I believe the main weakness of the method is the requirement of depth and pose as input during the self-supervised pretraining part. This severely limits the applicability of the method. As mentioned in the paper, the motivation is to use `in-the-wild consumer videos’ that are readily available for self-supervised pre-training. However, consumer videos typically do not have pose or depth. The zone generation process relies heavily on accurate depth and pose input. I believe it is not easy to adapt the method when to videos where depth and pose are not available or are noisy.\n- The downstream navigation tasks also assume access to the true depth and pose. This setup is also unrealistic as pose and depth from sensors are often noisy.\n- The trajectories used for pre-training are generated using a model trained on an exploration task and the learned representations are tested for different exploration tasks. This is counter-intuitive, the representations should be tested on a different task such as object goal or image goal navigation. These tasks provide a better variety as compared to different variants of the exploration task even otherwise.\n- Except for the SMT (video) baseline, I believe the comparison to the rest of the baselines is slightly unfair as the proposed method uses 4000 videos for pretraining which is not available to the baselines.\n- OccupancyMemory baseline is based on the Active Neural SLAM model, which I believe uses RGB frames as input as compared to RGBD used in EPC. This seems to be another unfair comparison.\n- The environment and dataset splits are not clear. It is mentioned that the video trajectories are generated using an agent trained on MP3D. It seems unfair to use MP3D environments for pretraining and then use the same environments for testing on downstream tasks. This is especially problematic because the proposed method leads to large gains mostly in the MP3D environment.\n\nQuestions/Comments/Suggestions:\n- What is the average length of 4000 videos used of pretraining?\n- Why was Occupancy Anticipation (Ramakrishnan et al. 2020b) not used as a baseline?\n- Why is the performance of OccupancyMemory unusually low on Gibson-S Flee as compared to other test sets?\n- It would interesting to see if the proposed EPC method can be used for pretraining with OccupancyMemory, possibly leading to even better performance.\n- It would be good to see the performance of the best baseline when trained for x frames where x = 15M + number of frames in 4000 pretraining videos. This would be a good upper bound for self-supervised learning.\n\n-- Update after author response\nThe authors have addressed some of my concerns, I have revised my score accordingly. I am not convinced that the method is significantly better than the baselines as it performs worse on some tasks/datasets and all the tasks in the paper are similar. Even the heuristic policy used for gathering the data is similar to the downstream tasks. The addition of other types of downstream navigation tasks such as objectnav or imagenav would make the paper much stronger. I like the noisy pose and depth experiments but there's no information on the type and amount of noise. I am assuming a zero-mean gaussian noise, which is not very realistic in my opinion. The standard deviation of the noise is not reported. I encourage the authors to add some other downstream navigation task and add realistic noise with relevant details to the camera-ready version if accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "R78LqaSJCp", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the embodied agent problem. The paper provides a self-supervised approach to learn environment-level representations for embodied agents. For the self-supervised module, the paper considers a series of images. The paper is well written and clear. The problem studied in the paper is an important problem. The approach of adding signals is reasonable. The experimental results show nice improvement compared to existing approaches.\n\nThe motivation of using self-supervised learning seems to be clear but not very strong for this problem. Because, taking searching for a refrigerator for example, it is not clear how these self-supervised tasks enhance the reasoning ability. It is a bit of brute-force solution to have this auxiliary task. It is better to have ablation study for leave-one-out testing. In addition, if there are more auxiliary tasks, will the performance continue to be improved? How these tasks help the embedded agent is not clear. It would be more interesting to have analysis on that. \n\nThe training part is not very clear. What is the policy for navigation tasks and what is the setup?  There is no study about the effect of EPC designs. It is better to have more ablation study for EPC. The paper shows it helps improve the performance. However, how can it improve the RL training is not clear. In other words, it is better to provide more details at least experimental results about the connection between the designs of EPC and navigation tasks.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2otjyMGOtI6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#############################################################################################\nSummary:\n\nThis paper proposes Environment Predictive Coding which leverages predictable information from video trajectories of egocentric movements to learn the environment features in a self-supervised manner. They then show the learned environment feature encoder can be useful for downstream navigation tasks.\n\n#############################################################################################\n\nPros:\n1.\tLearning the predictive features from the environment is an interesting idea for using self-supervised signals from the video trajectories.\n2.\tDifferent components and design decisions are well-motivated, such as noise-contrastive estimation, STM, etc.\n\n#############################################################################################\n\nCons:\n1.\tThe approach assumes training data of video trajectories from other agents in the environment. However, it is not clear to me how this dataset was generated. Could you please provide more details on how this is generated? What policy do you use? What is the task or reward these agents use?\n2.\tFollowing on the question above, another concern that I have is that the training data requires a strong exploratory policy which will be hard to achieve with a random policy. This layer of complexity is ignored in the paper. Moreover, the authors state that their approach is much less constrained than previous methods that generate data with task-specific algorithms. If this paper requires a similar level of computation and strong task similarity with the test task, then this argument should be weakened. Or at least, an analysis should be provided to detail the difference.\n\n#############################################################################################\n\nRecommendation and Explanation:\n\nI recommend acceptance since the proposed self-supervised task is interesting and could be useful for the embodiment navigation community. However, I strongly recommend the authors to resolve my questions above.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1W80Cdop2Ii", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe paper proposes a self-supervised approach for learning environment-level representations for embodied agents. The idea is that agents collect images and their corresponding poses during a walk-through phase. The images are clustered into multiple \"zones\". The zones are divided into seen and unseen zones. Using contrastive learning, the model is trained to distinguish the features of an unseen zone from the rest of the zones. The paper shows this approach improves performance over a number of baselines for Area Coverage, Flee, and Object Coverage tasks.\n\n**Strengths**\n\n- The idea of environment-level representation learning is interesting.\n\n- The paper shows improvements over a number of strong baselines for three tasks.\n\n- The experiments show sample efficiency compared to image-level representation learning.\n\n**Weaknesses**\n\n- It is not clear if it is the zone prediction task that provides the improvements. While the baselines are very helpful to better understand the model, a few important baselines are missing. The following baselines clarify if the improvements can be attributed to zone prediction or not:\n\n    (a) Train with random frames from the training environments in a contrastive learning-based framework such as MoCo (He et al., CVPR 2020) and use that as the environment encoder.\n\n    (b) Augment SMT (Video) baseline with pose.\n\n- There are a number of statements that are not correct and should be removed. For example,\n\n    (a) In Section 3-zone generation, it is mentioned that \"Note that the zones segment video trajectories, not floorplan maps, since we do not assume access to the full 3D environment\", while the approach uses all features of a full 3D environment: computing the 3D intersection of point clouds, knowing which poses are valid and which are not, etc.\n\n   (b) It is mentioned that \"Our model competes strongly with a task-specific representation model on the tasks that the latter was designed for, while outperforming it significantly on other tasks\". This model has been trained using area coverage data, which is the same or very similar to the proposed end tasks.\n\n**Score justification**\n\nOverall, I am positive about this paper. The main issue is that it is not clear what provides performance improvements. This should be clarified in the rebuttal.\n\n**After rebuttal**\n\nThe rebuttal does not address my concerns so I lower my rating due to the following reason:\nIt was not clear where the improvement comes from so I asked for an image-level baseline, which is trained using contrastive methods. The rebuttal does not provide that. It is mentioned that adding the image-level baseline to the proposed approach even improves the results without providing any evidence. My concern was that an image-based method trained in a similar way might provide the same results. I cannot really judge if the proposed method is effective or not due to lack of this baseline. Several previous embodied representation learning works are outperformed by simple image-level baselines. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a self-supervised method to predict the gist features of image frames during navigation of an agent supervised by depth and egomotion. The features are retargeted to train navigation policies and outperform previous methods or other pretraining schemes. The idea is related to self-supervised by feature prediction but is employed in a zone level as opposed to isolated image level. Though reasonable, in the context of the recent abundance of self-supervised prediction papers in various level of spatial visual granularity, the paper may not be of sufficient novelty to present a sizable contribution for ICLR acceptance.

Decision: Reject