Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a method for the navigation task. The proposed method is inspired by the concept of cognitive map in human and animal. 
A recurrent neural network is incorporated and training is divided in two steps of (1) task-agnostic pre-training and (2) task-speciﬁc Q learning.

The paper is well-written and clear.

While the idea of using a representation inspired by cognitive maps is interesting, the paper does not offer much technical novelty. (e.g no technical novelty in eq. 1 and eq. 2)

The experimental results are weak and only a simple domain is tested.  

It is not clear how efficient the method would be compared to other approaches.

Visualizations can be improved. As an example, Fig. 4 is not quite self-representative.

I see the paper has a large room for improvement and the current manuscript is not convincing for publication.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: ## Overview
This paper explores how pre-training a recurrent network on different navigational objectives confers different benefits when it comes to solving downstream tasks. First, networks are pretrained on an objective that either emphasizes position (path integration) or landmark memory (identity of the last wall encountered). This pretraining generates recurrent networks of two classes, called PosNets and MemNets (in addition to no pre-training, called RandNets). Surprisingly, the authors found that pre-training confers different benefits that manifests as differential performance of PosNets and MemNets across the suite. Some evidence is provided that this difference has to do with the requirements of the task. Moreover, the authors show how the different pretraining manifests as different dynamical structures (measured using fixed point analyses) present in the networks after pre-training. In particular, the PosNets contained a 2D plane attractor (used to readout position), whereas the MemNets contained clusters of fixed points (corresponding to the previously encountered landmark).

Overall, I thought this was a very interesting paper--it is one of the first papers I have seen that demonstrates how different pre-training requirements both change network dynamics (as measured by fixed points), and how those differences can yield different benefits on downstream navigational tasks.

## Major comments/concerns
- I think the presentation the pretraining objective (eq 3) could be clearer. Is eq 3 what is minimized during pre-training? How are \alpha, \beta, and \gamma chosen? \alpha is used to separate the two types of networks (MemNet from PosNet), which is the critical difference studied in the paper, so it would helpful to go into more detail about what \alpha controls and how it was chosen.

- For the first task, I am surprised that the agent is able to navigate the environment using only the eight neighboring locations. What is the size of the arena? What fraction of the states are simply surrounded on all sides by empty space? It would be informative to show some trajectories of agents solving the basic task.

- For Fig 3A and 3B, it would be nice to show the other network's performance (i.e. show the PosNet on the scaling task in 3A, and the MemNet on the bar task in 3B).

- How come there are no networks that are able to solve both sets of tasks? That is, how come there are no networks in the upper right region of Fig 3C? Does this suggest that an agent needs to combine two separate RNNs to solve the whole suite of tasks?

- What happens if you train recurrent networks with more sophisticated cell architectures (e.g. a GRU or an LSTM)? These are typically easier to train (and using automatic differentiation techniques are also amenable to fixed point analysis).

## Minor comments
- In eq. (1), use `\left(` and `\right)` to make the first set of parentheses have an appropriate height.
- Typo on the first line after eq. (6) (matrices)
- Relevant reference on comparing networks using dynamics around approximate fixed points: https://arxiv.org/abs/1907.08549.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies the internal representations of recurrent neural networks trained on navigation tasks. By varying the weight of different terms in an objective used for supervised pre-training, RNNs are created that either use path integration or landmark memory for navigation. The paper shows that the pretraining method leads to differential performance when the readout layer of these networks networks is trained using Q-learning on different variants of a navigation task. The main result of the paper is obtained by finding the slow points of the dynamics of the trained RNNs. The paper finds that the RNNs pre-trained to use path integration contain 2D continuous attractors, allowing position memory. On the other hand, the RNNs pre-trained for landmark memory contain discrete attractors corresponding to the different landmarks.

An interesting implication of the findings for neuroscience is that the same underlying network architecture can learn different dynamics, explaining diverse types of navigation-related signals found in the mammalian brain (place cells, border cells etc.).

I am not entirely sure about the novelty or impact of the presented results. However, the exposition and the results are clear and it is interesting how pre-training can shape the dynamics of a network. I therefore recommend acceptance.

Minor comments:
- Please describe how the networks 1, 13, 20 used for Figure 4 were selected. Were they selected at random or selected according to some criteria?
- It may be interesting to study the effect of more modern RNNs, e.g. LSTMs or GRUs, on the dynamics.

review_title: Official Blind Review #1

Meta-review: Navigation is learned in a two-stage process, where the (recurrent) network is first pre-trained in a task-agnostic stage and then fine-tuned using Q-learning. The analysis of the learned network confirms that what has been learned in the task-agnostic pre-training stage takes the form of attractors.

The reviewers generally liked this work, but complained about lack of comparison studies / baselines. The authors then carried out such studies and did a major update of the paper.

Given that the extensive update of the paper seems to have addressed the reviewers' complaints, I think this paper can be accepted.
Decision: Accept (Poster)