Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a self-supervised method to learn environment-level representations for embodied agents. The representations are learned using a zone prediction task. This task involves predicting features of the masked portions of input trajectories based on unmasked portions. Results on downstream navigation tasks show an improvement over a range of baselines.

Strengths:
- The method is novel and original to the best of my knowledge. Although it is demonstrated to be useful for exploration tasks, it can potentially be useful for different types of embodied tasks.
- The paper is written well and easy to follow in most places.
- The authors have done an incredible job of reviewing related work comprehensively and discussing the proposed approach in the context of prior work.
- The authors also compare the proposed method with a wide range of useful baselines.

Weaknesses:
- I believe the main weakness of the method is the requirement of depth and pose as input during the self-supervised pretraining part. This severely limits the applicability of the method. As mentioned in the paper, the motivation is to use `in-the-wild consumer videos’ that are readily available for self-supervised pre-training. However, consumer videos typically do not have pose or depth. The zone generation process relies heavily on accurate depth and pose input. I believe it is not easy to adapt the method when to videos where depth and pose are not available or are noisy.
- The downstream navigation tasks also assume access to the true depth and pose. This setup is also unrealistic as pose and depth from sensors are often noisy.
- The trajectories used for pre-training are generated using a model trained on an exploration task and the learned representations are tested for different exploration tasks. This is counter-intuitive, the representations should be tested on a different task such as object goal or image goal navigation. These tasks provide a better variety as compared to different variants of the exploration task even otherwise.
- Except for the SMT (video) baseline, I believe the comparison to the rest of the baselines is slightly unfair as the proposed method uses 4000 videos for pretraining which is not available to the baselines.
- OccupancyMemory baseline is based on the Active Neural SLAM model, which I believe uses RGB frames as input as compared to RGBD used in EPC. This seems to be another unfair comparison.
- The environment and dataset splits are not clear. It is mentioned that the video trajectories are generated using an agent trained on MP3D. It seems unfair to use MP3D environments for pretraining and then use the same environments for testing on downstream tasks. This is especially problematic because the proposed method leads to large gains mostly in the MP3D environment.

Questions/Comments/Suggestions:
- What is the average length of 4000 videos used of pretraining?
- Why was Occupancy Anticipation (Ramakrishnan et al. 2020b) not used as a baseline?
- Why is the performance of OccupancyMemory unusually low on Gibson-S Flee as compared to other test sets?
- It would interesting to see if the proposed EPC method can be used for pretraining with OccupancyMemory, possibly leading to even better performance.
- It would be good to see the performance of the best baseline when trained for x frames where x = 15M + number of frames in 4000 pretraining videos. This would be a good upper bound for self-supervised learning.

-- Update after author response
The authors have addressed some of my concerns, I have revised my score accordingly. I am not convinced that the method is significantly better than the baselines as it performs worse on some tasks/datasets and all the tasks in the paper are similar. Even the heuristic policy used for gathering the data is similar to the downstream tasks. The addition of other types of downstream navigation tasks such as objectnav or imagenav would make the paper much stronger. I like the noisy pose and depth experiments but there's no information on the type and amount of noise. I am assuming a zero-mean gaussian noise, which is not very realistic in my opinion. The standard deviation of the noise is not reported. I encourage the authors to add some other downstream navigation task and add realistic noise with relevant details to the camera-ready version if accepted.
review_title: Interesting method, some issues with evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: #############################################################################################
Summary:

This paper proposes Environment Predictive Coding which leverages predictable information from video trajectories of egocentric movements to learn the environment features in a self-supervised manner. They then show the learned environment feature encoder can be useful for downstream navigation tasks.

#############################################################################################

Pros:
1.	Learning the predictive features from the environment is an interesting idea for using self-supervised signals from the video trajectories.
2.	Different components and design decisions are well-motivated, such as noise-contrastive estimation, STM, etc.

#############################################################################################

Cons:
1.	The approach assumes training data of video trajectories from other agents in the environment. However, it is not clear to me how this dataset was generated. Could you please provide more details on how this is generated? What policy do you use? What is the task or reward these agents use?
2.	Following on the question above, another concern that I have is that the training data requires a strong exploratory policy which will be hard to achieve with a random policy. This layer of complexity is ignored in the paper. Moreover, the authors state that their approach is much less constrained than previous methods that generate data with task-specific algorithms. If this paper requires a similar level of computation and strong task similarity with the test task, then this argument should be weakened. Or at least, an analysis should be provided to detail the difference.

#############################################################################################

Recommendation and Explanation:

I recommend acceptance since the proposed self-supervised task is interesting and could be useful for the embodiment navigation community. However, I strongly recommend the authors to resolve my questions above.

review_title: Official Blind Review R2

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper studies the embodied agent problem. The paper provides a self-supervised approach to learn environment-level representations for embodied agents. For the self-supervised module, the paper considers a series of images. The paper is well written and clear. The problem studied in the paper is an important problem. The approach of adding signals is reasonable. The experimental results show nice improvement compared to existing approaches.

The motivation of using self-supervised learning seems to be clear but not very strong for this problem. Because, taking searching for a refrigerator for example, it is not clear how these self-supervised tasks enhance the reasoning ability. It is a bit of brute-force solution to have this auxiliary task. It is better to have ablation study for leave-one-out testing. In addition, if there are more auxiliary tasks, will the performance continue to be improved? How these tasks help the embedded agent is not clear. It would be more interesting to have analysis on that. 

The training part is not very clear. What is the policy for navigation tasks and what is the setup?  There is no study about the effect of EPC designs. It is better to have more ablation study for EPC. The paper shows it helps improve the performance. However, how can it improve the RL training is not clear. In other words, it is better to provide more details at least experimental results about the connection between the designs of EPC and navigation tasks.

review_title: Self-supervised approach is useful. However, how it helps embodied agents is not clear.

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**

The paper proposes a self-supervised approach for learning environment-level representations for embodied agents. The idea is that agents collect images and their corresponding poses during a walk-through phase. The images are clustered into multiple "zones". The zones are divided into seen and unseen zones. Using contrastive learning, the model is trained to distinguish the features of an unseen zone from the rest of the zones. The paper shows this approach improves performance over a number of baselines for Area Coverage, Flee, and Object Coverage tasks.

**Strengths**

- The idea of environment-level representation learning is interesting.

- The paper shows improvements over a number of strong baselines for three tasks.

- The experiments show sample efficiency compared to image-level representation learning.

**Weaknesses**

- It is not clear if it is the zone prediction task that provides the improvements. While the baselines are very helpful to better understand the model, a few important baselines are missing. The following baselines clarify if the improvements can be attributed to zone prediction or not:

    (a) Train with random frames from the training environments in a contrastive learning-based framework such as MoCo (He et al., CVPR 2020) and use that as the environment encoder.

    (b) Augment SMT (Video) baseline with pose.

- There are a number of statements that are not correct and should be removed. For example,

    (a) In Section 3-zone generation, it is mentioned that "Note that the zones segment video trajectories, not floorplan maps, since we do not assume access to the full 3D environment", while the approach uses all features of a full 3D environment: computing the 3D intersection of point clouds, knowing which poses are valid and which are not, etc.

   (b) It is mentioned that "Our model competes strongly with a task-specific representation model on the tasks that the latter was designed for, while outperforming it significantly on other tasks". This model has been trained using area coverage data, which is the same or very similar to the proposed end tasks.

**Score justification**

Overall, I am positive about this paper. The main issue is that it is not clear what provides performance improvements. This should be clarified in the rebuttal.

**After rebuttal**

The rebuttal does not address my concerns so I lower my rating due to the following reason:
It was not clear where the improvement comes from so I asked for an image-level baseline, which is trained using contrastive methods. The rebuttal does not provide that. It is mentioned that adding the image-level baseline to the proposed approach even improves the results without providing any evidence. My concern was that an image-based method trained in a similar way might provide the same results. I cannot really judge if the proposed method is effective or not due to lack of this baseline. Several previous embodied representation learning works are outperformed by simple image-level baselines. 


review_title: Good paper, interesting idea, needs more baselines

Meta-review: The paper proposes a self-supervised method to predict the gist features of image frames during navigation of an agent supervised by depth and egomotion. The features are retargeted to train navigation policies and outperform previous methods or other pretraining schemes. The idea is related to self-supervised by feature prediction but is employed in a zone level as opposed to isolated image level. Though reasonable, in the context of the recent abundance of self-supervised prediction papers in various level of spatial visual granularity, the paper may not be of sufficient novelty to present a sizable contribution for ICLR acceptance.

Decision: Reject