Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors train a recurrent neural network on a navigation task, and observe the emergence of several phenomena reminiscent of the hippocampus: appearance of place cells with a secondary receptive field at task-relevant locations; anticipation of possible future paths in the activity of the model, with alternation in time between possible future paths; a high proportion of neurons tuned to task variables rather than animal trajectory.

Strong points:
- these findings are compelling, they account for some hallmark properties of the activity of hippocampus, and they could lead to a better understanding of the role and function of the hippocampus.
- the experiments are rigorous and convincing.

Weak points:
- some technical aspects of the paper could be clarified (see below)
- it is unclear how this model improved our understanding of the hippocampus function, and whether the model makes any testable predictions about the hippocampus.

I recommend to accept this paper because of its strengths listed above.

Clarification questions:
1) I did not understand the role of secondary cue point. Why were these required in addition to the primary cue point and choice point?
2) vocabulary: What are "metric" and "non-metric" representations?
3) How reliable is the alternative path visiting phenomenon? Can this phenomenon be observed reliably in networks trained from different initial conditions?
4) I did not understand the consequences/take-homes of the second and third paragraph discussions. 

Additional feedback:
1) It would be interesting to see a discussion on what we learned about the function of the hippocampus from this model, and/or what predictions this model makes about neural activity in hippocampus. 
2) Are there any oscillations and phase-precessions phenomena in the model? If not, it would be interesting to discuss why these oscillations might be present in the brain but not in the model.
3) Could this network be used to simulate difficult experiments, e.g. understand how future paths exploration works in an open-field setting?
4) The first sentence of the abstract is difficult to understand. In general, shorter sentences could improve clarity.
review_title: A compelling model of some hallmark properties of the hippocampus

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
 
The authors trained a recurrent network to perform a sensory prediction task and this gave rise to units that resembled hippocampal place fields. Then they augmented the network with a Q-learning objective and shown that the activity in the network sweep forward in space if the agent is fixed at a decision point. 

##########################################################################

Reasons for score: 
 
Overall, I think the paper should be rejected. The work is interesting, but the clarity of the paper is not at the level of the findings. I think the authors should enhance exposition, and strengthen some analysis. The findings are really interesting ,but at the current stage I don’t think the paper is ready to be published. However, I’m happy to revise my score if authors addresses my comments. 

##########################################################################

Pros: 
 
1. Combination of unsupervised learning and RL to show that units in a recurrent network can be used to understand spatial and non-spatial firing patterns in the hippocampus

2. Figure 4 and 5 are convincing.


Cons:

1. While explaining the task the authors claim that the colours are chosen at random, however it is not clear whether these stay fixed across episodes or they changed. Intuitively it looks like, once the colour are generated, then they stay fixed, but in this case it is difficult to understand why a simple Q-leaning objective won’t be able to solve this task (as claimed by the author). The only reason I can think about is if the steps between the cue and the reward are longer that the ones allowed by the discount factor chosen (0.8). However, in the paper there are no details about the number of steps or how the discount affect the results. I think this is a serious issue.

2. The authors claim that they first pre-train on the predictive task, but then in the loss of eq. 3 they report a combined loss. Does it mean that the loss_{rgb} is also fine-tuned while training with the Q-learning objective? This point need clarification. 

3. The paper doesn’t report any details about the learning rates or the sizes of the linear layers used in eq. 1 and 2. This way is impossible to replicate this results. This is a serious issue.

4. How the threshold for place cells are defined? Why 30%? How many units show firing above that threshold? This needs further analysis to support the decision, which otherwise seem very arbitrary. 

5. Are figure 4 and 5 just cherry picked run or are these averaged across several testing runs? Also in the captions of these figures the authors are using the singular “observation ” in figure 4 and the plural “observations” in figure 5. Does it means that the analysis have been performed differently? Or is it just a mistake?. This is an important point as I think the results will be more powerful with the same image fed as input, or even better with no image, just with 0ed input to simulate pondering.	

Minors:

1. 380 units seems quite an unconventional number of units, why not 256 or 512? Can you please explain. Also is this number affecting the representations? Have you done a sweep and settle on this number because it support your findings better? If so, it would be important to mention it.

2. It would have been nice to have analysis to support the following sentence on page 4: Generally, when the network loses its ability to self-localise the agent, state-action values are no longer reliable indicators of future reward potential as the current environmental state is not clearly discernible. Otherwise please correct it.


review_title: Interesting findings, bad explanation.

final_score: rating: 7: Good paper, accept
review_content: Motivated by biological considerations, this paper shows that recurrent networks trained with a predictive and goal-based objective on a maze finding task, qualitatively recapitulate experimental findings in hippocampal recordings in rodents trained on the same task. In particular, these LSTM networks demonstrate both metric representations of their environment and nonlocal extrafield firing at decision points along the maze (anticipating the future trajectory of the agent).

Strengths:
+ I like that the authors take a normative approach that exhibits both metric and non-metric place cell representations of the environment, unifying prior findings in one model.
+ I appreciate that no velocity input is given to their model, in contrast to prior approaches.
+ I also liked the qualitative comparisons to hippocampal recordings from rodents trained on the same task (especially Figures 6 and 7).

Weaknesses:
- The primary conclusion, namely, training an RNN on a maze-like environment gives you place cells, is really not all that new, especially considering that the network is still supervised to predict position and landmarks.
- Given that lack of novelty in the modeling conclusion, it would have therefore been nice to have seen more quantitative comparisons to hippocampal recordings in rodents. Does their approach explain more variance in these neurons than prior approaches? Otherwise, it seems that they simply recapitulate prior qualitative comparisons.

Minor comments: “Recurrence based” should be changed everywhere to “recurrent” (e.g. on pg. 1), and “Neuroscience” is not capitalized. The motivation to use double Q learning should be expanded on in pg. 3 prior to equation 3.

Question: The authors mention that Q-learning performs poorly on tasks in dynamic environments – however, I do not see any evidence of this in the paper, it would be imperative to show this explicitly for the environments they consider. Suppose this is in fact the case, could you clarify what makes your approach more successful at this task than others? Is it because of the pretraining to predict the subsequent observation of wall colors from the current wall color observations?

As it stands, I think the ideas of this paper are interesting and think it unifies prior approaches, but I do not think the conclusions from the modeling add all that much novel insight from prior approaches. Therefore, I recommend a weak accept.

review_title: Unifies prior approaches though not many new insights

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Paper summary**

The main goal of this paper is to show that LSTM units in a network trained to solve a T-maze task, show similar activity patterns as neurons in rats solving a similar task. Specifically, the authors make the following claims: (1) an RNN learning the task by  a combination of reinforcement and predictive learning produces internal representations with consistent extrafield firing associated with consequential decision points, (2) the network’s representation, once trained, follows a forward sweeping pattern similar to those found in rats and (3) a higher proportion of units in the trained network show strong selectivity for the choice phase of the task than for spatial topology, as seen in rats.

**Pros**
1. The submission is clear, well-written and the execution is competent. 
2. I find the approach well motivated and the problem interesting for the current state of the field.
3. The authors provide a sufficient amount of details so that reproducibility should be possible.

**Cons**

I have concerns about key points of the paper and the interpretation of the results:

1. One of the main results of the paper is the observation that the network produces a forward-moving representation similar to the one observed in rats at the decision point. However, the way the authors simulate this is by freezing the agent at such point and keeping the LSTM running, repeating the same constant observation. The LSTM was trained on trajectories on the maze, so in this trajectory (which was never used during training), the network is completely out of distribution. The activity of any network in this situation is difficult to interpret. Because the network was trained with a predictive loss on two very specific sequences of observations, it seems plausible that it is robust to the change of input statistics and follows the same sequence, maybe with some instabilities. Note that the cue was present, which explains why the correct sequence is followed. 
In the network trained also by RL in particular, the authors interpret this as “The agent appears to be sampling the trajectory concerning the alternate return arm of the maze before ultimately settling on the rewarding return arm”. But this is, in my opinion, an over-interpretation, as the agent has no sampling capability in the first place (there is no generative model of observations), nor any particular planning mechanism. Alternatively, the authors may be claiming that this jumping behavior happens only after the RL training and not before, in which case they should emphasize this difference and quantify it explicitly. Although in this case, a simple explanation for this could be that due to the epsilon-greedy, only during the RL training the network is exposed to the wrong cue-arm combination. Therefore, the LSTM would be less able to rely on the cue, which could explain the jump between attractors in the out-of-distribution case. In the discussion section, the authors claim “We demonstrate that extrafield firing activity [..] emerges when a simulated agent [...] pauses at decision points - suggesting intrinsic dynamics are encoding the future planned trajectory of the agent.”. I find this to be an over-claim, as the agent doesn’t pause (it can’t) and doesn’t plan (for any common definition of planning). I would be more convinced if the agent was able to pause (as an additional action) and this behavior was observed in the LSTM activity in this situation, which is closer to the biological case.

2. The pre-train stage is done on trajectories that correspond to the solved task. This means that the LSTM trained by the predictive loss is not exposed to the general structure of the environment but to the specific solution of the task, including the cue-choice association and the exact sequence of observations in each of the two correct trajectories. The authors draw a parallel with the pre-training phase in behavioral experiments (Johnson & Redish, 2007) in which rats usually run each trajectory separately (by having the other one blocked). However, in my opinion, this is problematic for their analysis. First it’s unclear to what extent the network is learning by RL as during the pre-training it has already learnt to predict the observation corresponding to the correct turn (wich corresponds to one of the two actions). Second, the LSTM is exposed only to the correct cue-arm trajectories, which I think is the reason why the forward-looking sweeps only follow these trajectories (see previous point). This is more similar to a demonstration than to pre-training. A more conventional pre-training would leave the agent to explore freely to implicitly learn the structure of the environment (a la Tollman). On the other hand, the argument of following the protocol of the behavioral experiments also doesn’t fully work as the rats are still producing motor outputs and even being rewarded during the pre-training phase (Johnson & Redish, 2007).

3. Finally, a more general concern is the main point of the paper. If I understand correctly, the main claim is the similarity of the observations between the RNN agent and the experimental findings in rats. However, given that there are plenty of arbitrary choices when training an RNN, I believe the results are not particularly explanatory.  I would encourage the authors to formulate better alternative hypothesis and controlled experiments. For example, I would find it interesting to show that the forward-sweeping observations done in rats, which is often interpreted as a signature of planning or prediction of the consequences of future actions, arises simply from a next-step prediction loss in an overtrained rat.

Minor concerns: 

- “As such, a network of Gated Recurrent Units [...] or vanilla RNN units was unable to perform well in either the pre-training or joint RL task due to these prevalent long term dependencies.”  How many steps are there between cue and choice, and between choice and reward? 

- Related to the previous point: “We attempted to run the reinforcement learning task alone in a maze with no wall colours or environment statistics except the cue. In this scenario the network is not able to learn the task due to a lack of self-localisation.” If I understand correctly, there is a constant number of steps between the cue and the moment where the choice has to be made. I would tend to believe that an LSTM can learn to make a prediction only based on the number of timesteps, regardless of the lack of wall observations (e.g. 2 sequence problem in Hochreiter and Schmidhuber, 1997).

Details: 

- It would be good to clarify what exactly is the action set of the agent. 
- I would like to know how exactly are activity maps obtained. The authors mention “Place fields determined by contiguous locality with average activity exceeding 30% peak unit activity during a single left trajectory followed by a right trajectory” but I don’t find this particularly clear. I also found it difficult to understand the bottom row of Fig 3. 
- Fig 5 should be referred to in the paragraph starting with “In stark contrast to the dynamics of the LSTM network following predictive pre-training...”
- Why is the return to start representation in Fig. 6 different for right and left trajectories?



review_title: Official review

Meta-review: This paper analyses a recurrent neural network model trained to perform a simple maze task, and reports that the network exhibits multiple hallmarks of neural selectivity reported in neurophysiological recordings from the hippocampus— in particular, they find place cells which also are tuned to task-relevant locations, cells which anticipate possible future paths, and a high proportion of neurons tuned to task variables.  

The reviewers appreciated the interesting empirical analysis, and the demonstration that multiple such features could arise in the same neural network— to the best of my knowledge, this had not been demonstrated explicitly before. However, there were also multiple concerns, which lead to this paper beeing discussed extensively and controversially. In particular, it is not clear which features arise from which learning objective, for example, for place cells to arise, do we  just need sensory prediction, or do we need q-learning? In addition, there were some points in which the tightness of the analogy between model and biology is questionable— in particular, this refers to the comprising between hippocampal recordings and the evaluation of the network.  Finally, it is also clear that  some of these observations reported in the paper are, indeed, empirical observations rather than explanations. Because of these shortcomings, there was no consensus and strong support from the reviewers for acceptance of the paper.

After extensive discussion between both the reviewers, the AC and the program chair, the final decision was to not accept the paper. We do hope that the reviews will help you in improving the study and its presentation. It clearly has potential to be a valuable contribution to the literature. 

Decision: Reject