Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper tries to analyze the similarities and transferring abilities of learned visual representations for embodied navigation tasks. It uses PWCCA to measure the similarity.  There are some interesting observations by smart experimental designing. 

I have several concerns.

- for the non-disjoint experiments, the difference between A and B is that the subsets contain different instances. The objects in subsets A and B may have the same category. The objects with the same category may share similar surrounding environment. Thus, the visual inputs for the training model on A and B may just have minor differences. This point is also related to the spatial coverage used in the paper. Since the visual input is similar, why is the conclusion in Figure1(b) non-trivial?

- for the transferring experiments, in the beginning, the finetuning way is better than the new training makes sense. But, why do the results of learning a new policy from scratch will inferior to the finetuning way when training to convergence? The two experiments are both performed on the same fixed visual encoder.

- I think the experiments can not support the argument that residual connections help networks learn more similar representations. Will other structures such as VGG also learn similar representations? Will the degrees of similar representations be proportional to the accuracy of the classification tasks and the modified residual network still outperforms the squeezenet? The more straightforward ablation studies might be that we remove all shortcuts of the ResNet as the plain version.

=========================================================
After Rebuttal:

I thank the author for the response. I still think the evaluations and experimental settings cannot fully support the conclusions. So I keep the original score.

I hope the comments are useful for preparing a future version of this work.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This article studies the similarities between the learned representations for different tasks when trained using reinforcement learning algorithms. The ultimate question that this study tries to answer is an interesting one. Namely, how much can representations learned by training on one task be beneficial for learning other tasks? A high interdependence between the representations can lead to a more successful transfer of knowledge between tasks.
The authors are further interested in studying the properties that influence this relationship, which depends on the elements of training as well as attributes of the tasks themselves.

However, I believe that the results are not strong enough to support the claims that are stated in the paper and the limited scope of the environments tested does not make a convincing case that the results will be generalizable much beyond these scenarios. Therefore, in the current state, I think this paper should be rejected.

Firstly, the paper states that:
> ".. if the distance between models trained on different tasks is the same as that between models trained on the same task, representations are task-agnostic."
This seems to be a key argument in the paper. However, I believe that this argument is based on the assumption that representations learned for a single task are indeed highly similar. I think this assumption requires some sort of support as reinforcement learning algorithms are known to be highly inconsistent even in reaching similar solutions. Therefore, one cannot take for granted that the learned representations would be similar in any way.

Second, the authors claim in Section 5 that the random splits have little impact on the learned representation, but in Section 6 claim that the spatially disjoint splits have a noticeable impact on the representations. Without any measure of the impact, looking at figures 1b and 3a, I'm not convinced that this distinction is so obvious. The situation is worse when looking at the figures in the appendix, namely figures A3 and A5. If someone were to swap these two figures, my untrained eye would not be able to tell the difference.

I suggest that the authors spend more time explaining their reasoning as to why these results are significant enough to support the claims. Also, the text should be improved if it is to be accepted. There exist many problems ranging from small typos (simlate -> simulate, reuse-ability -> reusability, and "?." -> "?") to sentences that need to be reworked. Some figure legends/captions can also be improved to include more information, such as explaining what exactly the shaded regions represent (I'm guessing one standard deviation from the mean over some unknown replicates), or in Fig 3b making clear whether "A -> B" is done with "New Policy" or "Fine-tuned Policy". I am also curious as to why only one of these scenarios was experimented with in sections 6 and 7.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper tests generality and transfer in visual navigation tasks. It's an interesting question and a well-executed study. I applaud the use of a complex high-dimensional environment.

The experiments are well done, but I am not sure what we learn. Each experiment compares two highly similar tasks - as the authors themselves acknowledge - in ways that do not obviously connect to realistic transfer scenarios, such as transferring to a new environment. I would have liked to see experiments that compare networks trained on different environments or on different sets of environments.

The paper is called "Insights on visual representations for embodied navigation tasks," but I am left wondering what those insights are. The authors state that "Our work provides valuable and actionable insight into how the task influences the representation for embodied navigation tasks," but it is light on specifics and on the general discussion of the results. Not much is offered beyond a suggestion to use ResNets. I would consider revising my assessment if the authors write a more thorough discussion and specify clear implications of their findings.

The paper is generally well-written although important details are left out:
- I think the target objects have fixed rather than randomized locations in the environments, but this isn't stated in the paper. Please specify.
- Where does the agent start during an episode?
- How large are the environments? e.g., one room or multiple rooms?
- How many objects per environment are there e.g., in sets A and B?
- The "permutations of an environment" setting isn't very clearly written and it took me a few readings to understand what you mean.
- Finally, I suggest revising the very vague title to the paper
review_title: Official Blind Review #3

Meta-review: The general consensus amongst the reviewers is that this paper is not quite ready for publication, and needs to dig a little deeper in some areas.  Some reviewers thought the contributions are unclear, or unsupported.  I hope these reviews will help you as you work towards finding a home for this work.
Decision: Reject