Hippocampal representations emerge when training recurrent neural networks on a memory dependent maze navigation task
Authors: Justin Jude
Matthias Hennig
Venue:  (2021)
Keywords: recurrent neural network
place cell
hippocampus
neural dynamics

Abstract: Can neural networks learn goal-directed behaviour using similar strategies to the brain, by combining the relationships between the current state of the organism and the consequences of future actions? Recent work has shown that recurrent neural networks trained on goal based tasks can develop representations resembling those found in the brain, entorhinal cortex grid cells, for instance. Here we explore the evolution of the dynamics of their internal representations and compare this with experimental data. We observe that once a recurrent network is trained to learn the structure of its environment solely based on sensory prediction, an attractor based landscape forms in the network's representation, which parallels hippocampal place cells in structure and function. Next, we extend the predictive objective to include Q-learning for a reward task, where rewarding actions are dependent on delayed cue modulation. Mirroring experimental findings in hippocampus recordings in rodents performing the same task, this training paradigm causes nonlocal neural activity to sweep forward in space at decision points, anticipating the future path to a rewarded location. Moreover, prevalent choice and cue-selective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that combining predictive, unsupervised learning of the structure of an environment with reinforcement learning can help understand the formation of hippocampus-like representations containing both spatial and task-relevant information.

Under review as a conference paper at ICLR 2021
HiPPOCAMPAL REPRESENTATIONS EMERGE WHEN TRAINING RECURRENT NEURAL NETWORKS ON A MEMORY DEPENDENT MAZE NAVIGATION TASK
Anonymous authors Paper under double-blind review
ABSTRACT
Can neural networks learn goal-directed behaviour using similar strategies to the brain, by combining the relationships between the current state the organism and the consequences of future actions? Recent work has shown that recurrent neural networks trained on goal based tasks can develop representations resem- bling those found in the brain, entorhinal cortex grid cells, for instance. Here we explore the evolution of the dynamics of their internal representations and compare this with experimental data. We observe that once recurrent network is trained to learn the structure of its environment solely based on sensory prediction, attractor based landscape forms in the network's is representation, which parallels hippocampal place cells in structure and function. Next, we extend the predictive objective to include Q-learning for a reward task, where rewarding actions are dependent on delayed cue modulation. Mirroring experimental findings in hip- pocampus recordings rodents performing the same task, this training paradigm causes nonlocal neural activity to sweep forward in space at decision points, antic- ipating the future path to rewarded location. Moreover, prevalent choice and cue- selective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that combining predictive, unsupervised learning of the structure of environment with reinforcement learning help under- stand the formation of hippocampus-like representations containing both spatial and task-relevant information.
INTRODUCTION
Recurrent neural networks have been used to perform spatial navigation tasks and the subsequent study of their internal representations has yielded dynamics and structures that are strikingly bio- logical. Metric (Cueva & Wei 2018 Banino et al., 2018) and non-metric (Recanatesi et al. 2019) representations mimicking grid (Fyhn et al. 2004) and place cells (O Keefe & Nadel 1978) respec- tively form once the recurrent network has learned a predictive task in the context of a complex environment. Cueva al. (2020) demonstrates not only the emergence of characteristic neural rep- resentations, but also hallmarks of head direction system cells when training a recurrent network on simple angular velocity integration task. Biologically, non-metric representations are associated with landmark spatial memory, in which place cells within the mammalian hippocampus fire when the associated organism is present in a corresponding place field. Extrafield firing of place cells occurs when these neurons spike outside of these contiguous place field regions. Here we show that recurrent neural networks (RNNs) not only form corresponding attractor landscapes, but also produce representations with internal dynamics that closely resemble those found experimentally in the hippocampus when performing gial-directed behaviour.
Research in neuroscience such as that Johnson & Redish (2007). shows that spatial representations mice in the CA3 region of the hippocampus frequently fire nonlocally, Griffin et al. (2007) show that far higher proportion of hippocampal neurons in the CAI region in rats performing episodic task in a T-shaped maze encode the phase of the task rather than spatial information (in this case trajectory direction). Ainge et al. (2007) show CA1 place cells encode destination location at the start position of a maze. Lee et al. (2006) demonstrate that place fields of CAI neurons gradually drift toward reward locations throughout reward training T-shaped maze.

---

Under review as a conference paper at ICLR 2021
In this work show that recurrent neural network learning choice-reward based task using inforcement learning, in conjunction with predictive sensory learning in a T-shaped maze produces an internal representation with consistent extrafield firing associated with consequential decision points. In addition we find that the network's ' representation, once trained, follows a forward sweep- ing pattern identified by JoJnsson Redish (2007). We then show that higher proportion of units in the trained network show strong selectivity for the encoding or choice phase of the task than the proportion showing selectivity for spatial topology. Importantly, these properties only emerge during predictive learning, where task learning is much faster compared to traditional deep Q learning.
METHOD
N
Observations (t)
Cue Tone (*)
Cue tone
LSTM
Predicted Observations 1)
Figure 1: Left, the wall observation and cue received by the network at each timestep. Right, the entangled predictive task the LSTM network pre-trained in order to generate non-metric map of the maze environment.
We use a form of the cued-choice maze used by Johnson & Redish (2007) which has a central structure with returning arms, shown in Figure I All walls of the maze tiled with distinct RGB colours which are generated at random and remain fixed throughout. An agent is initially learning to predict the next sensory stimulus given its movement. This combination of unsupervised learning and exploration has been shown previously to produce place cell-like encoding of the agent's posi- tion (Recanatesi al. 2019). Next, rewards four possible locations are introduced and the agent is tasked with associating cue with the rewarding trajectory. The agent has four vision sensors, one in each cardinal direction, reading the wall RGB colours they intersect. The cue tone is played the agent as it passes the halfway point of the central maze stem. A low frequency cue indicates that the agent will turn left at the top of the maze stem and a high frequency cue indicates right turn. These cue tones take the form of high or low valued scalar perturbed with normally distributed noise if at a cue point, with a zero value given all other locations. These four RGB colours as well as the cue frequency the current location make up the total input received by the agent.
The agent is controlled by recurrent neural network comprised of 380 unit Long-Short term memory (Hochreiter & Schmidhuber 1997) (LSTM) network with single layered readout for the prediction RGB values. We first pre- train the network by tasking it with predicting the subsequent observation of wall colours from the currently observable wall colours given its trajectory through the maze. The agent's starting location at the bottom of the central stem of the T maze and a trajectory of left right the top of the central stem is chosen pseudorandomly. depicted with red and blue arrows respectively in Figure and corresponding to the low (red trajectory) or high (blue trajectory) cue tone value given halfway up the stem. As in the experiments by Johnson & Redish (2007), during pre-trate pre-training the agent does not choose any of its actions and is only learning predict the sequence of wall colors it encounters. In a given pre- pre-training iteration, we collect all observations as the agent traverses the maze until it returns to the start location at the bottom of the central stem and finally train the L.STM on the entire collected trajectory. The network trained with mean-squared error loss of predicted and target wall colours (Eq. 1), with model parameters optimised using Adam (Kingma & Ba 2015) and a learning rate 0.001.
lossrgb n " (Urgb (Wigthy + brgb))²
(1)
i=1
2

---

Under review as a conference paper at ICLR 2021
To solve this task, the network has maintain the tone played in its internal memory for several time steps in order to predict subsequent wall colours from the top of the central stem. our model, this is achieved through the network forming a non-metric representation (attractor landscape) of the maze environment, as also demonstrated by Xu & Barak (2220). Similarly, behavioural experiments typically have comparable familiarisation phase with the environment before reward-based tasks are introduced (Johnson & Redish 2007 Griffin et al. 2007).
Secondary Cue/Choice points
Reward
Observations (t
Linear
Choice Point
Potential Reward sites
Potential Reward sites
LSTM
Observations (t)
No gradient
Linear
Primary point
Agent
Q Values
FC layer
Action
Start Po nt
Agent position (t+1)
π
Agent chooses turn direction according to RL pclicy
Figure 2: For the joint task to be learned by the LSTM network, we introduce secondary cue points, where the same cue tone as that played the primary cue point will be repeated and only if the agent has proceeded in turning in the direction corresponding to the cue tone frequency given at the primary cue location. The agent is free choose the next action be taken when traversing the maze at either the choice point at the top of the stem of the maze or at the secondary cue locations. There are two potential reward sites on both returning arms, with the reward sites being active if the agent is on the returning arm corresponding to the cue tone frequency.
Once the LSTM has formed an internal representation of the maze, the agent is tasked with navigat- ing towards potential reward sites whose location is indicated by the signal: low frequency cue indicates active reward sites on the left return arm and a high frequency cue indicates active reward sites the right return chosen randomly at each iteration with a secondary cue given if the agent has turned correctly. In this phase there are three choice points wherein the agent is able choose its next action and is con- strained to follow the forward maze direction elsewhere: at the top of the maze stem and at the two secondary choice points (Figure 2). with initially random movement these points during reward training. There are 5 steps between the cue and choice points and 7 steps from the choice point to first reward site on either return arm. The inclusion of the secondary cues as additional choice points was motivated by the experimental set up used by Johnson & Redish (2007), to compare the network activity at these points to experimental data. These secondary points also give the agent the opportunity backtrack on its decision made at the primary choice point in light of further environ- mental observation (the presentation or lack thereof of the secondary cue), and make learning more efficient in model. This may explain how it speeds up training the animals in the same task.
the tone and corresponding side of active reward sites are together
We additionally introduce a new single layered readout for the LSTM network which predicts state- action values associated with the four cardinal directions in relation to the agent's current position and direction. At each timestep. this ensemble receives the agent's 's environment observation and the agent follows an resilon-greedy policy (starting with fully random movement choice points and a decaying epsilon thereafter) for choosing optimal actions of those available at each of the three choice points. The recurrent network controlling the agent trained on weighted combined loss of reinforcement learning (RL) task loss and the previously described predictive wall colour loss:
losScombined (Q(s,a) - (r + + (s's,, arg max Q(s', ('))) A losSrgb
(2)
o'
3

---

Under review as a conference paper at ICLR 2021
The first component of this loss is the difference between predicted and observed state-action values which are represented by Q-values (Watkins & Dayan 1992), which are prediction of future global reward:
((s,a) Woh, + bq
(3)
We use double-Q learning (Van Hasselt et al.) 2016) to train the agent on the task, updating the target Q value predictor (Q' a LSTM with same number of units) every 15 training iterations. Double-Q learning allows for optimal performance on the reward task in drastically fewer agent maze traversals and network training iterations than with standard DQN (Mnih et al. 2013) based Q-lcarning which suffers from overestimation of Q-values. We settle on a discount factor (7) of 0.8 as values higher than this regularly cause the network to converge on solutions wherein the agent does not take the most direct path to reward locations, with backtracking at secondary choice points. The second loss component is the sensory prediction task which we used pre-train the network (A = 0.5). After the network has been pre-trained (optimising to minimise Eq. D. the network achieves perfect performance on the predictive task. This loss component is included when training the network on the reward task so that the spatial map (non-metric attractor landscape) of the maze environment formed during pre- gre-training maintained throughout Q-learning. This ensures the map is not overwritten as would happen when Q-learning is performed alone, and leads to faster task learning (see results). We optimise the network for this joint task using Adam and a learning rate of 0.0005, which we find improves the rate of convergence with optimal task performance, opposed to higher learning rates which still converge but with backtracking often inherent in task solutions.
In contrast to much of the previous work spatial representations in recurrent networks, we do not give the network any indication of the agent's 's location or movement. This makes the task considerably more difficult due to the umpredictable movement possible at choice points during the reward task. The network is coerced into storing the current movement direction of the agent in its representation, in addition storing the cue frequency. As such, network of Gated Recurrent Units (Cho et al., (2014) (GRUs) or vanilla RNN units was unable to perform well in either the pre- training or joint RL task due to these prevalent long term dependencies (18 steps between cue and final reward).
To analyse the representations formed by the network, we train a further single layered fully con- nected network with a softmax layer (shown in green in Figure 2 to predict the agent's next location using the activity of the LSTM. There no backpropagation of gradients between this predictor and the LSTM network, and the predictor is trained at the end of reward training. The plots in Figures 5 and 6 are distributions indicating the probability of agent location inferred from LSTT activity by the predictor. This is used in place of the decoding algorithm used by Johnson & Redish (2007) to predict the neuronally inferred maze location of rats when performing a cue based task.
3 RESULTS
The agent learns the sensory prediction task to high degree of recall and after around a thousand training itcrations (combined loss with pre-training in Figure()). the agent was able to achieve perfect performance on the reward task when the LSTM network had 380 or more units (Fig. 3 right). We trained the reinforcement learning (Eq. 2 portion of the task in an epsilon greedy manner, with a steadily decaying epsilon ensure that the agent would choose the rewarding path consistently once actions were chosen at choice points completely by the network. Notably, the agent did not turn at either of the secondary choice points once training had completed only the primary choice point.
We attempted to run the reinforcement learning task alone in a maze with no sensory input except the reward cue. In this scenario the network not able learn the task due to a lack of self-localisation and is unable to perform the task based on step counting between the cue and choice point. In addition, the reward based reinforcement learning task was attempted using Q-learning alone with a loss function that did not include the wall colour prediction error, both with and without pre-training (shown in Fig. 3 left). In both cases we find that the reward task is not learnable with the same higher rate of epsilon decay we use for the combined loss function with pre-training, as the network quickly forgets the attractor landscape of the maze formed during pre- pre-training, which we maintain through the combined loss (Eq. 2 We also find the network can solve the reward task using the combined loss without pre- pre-training, albeit in around times the number of maze traversals as with the use of the spatial map formed in the pre-trained case (Fig. 3 left).
4

---

Under review as a conference paper at ICLR 2021
Convergence speed
Convergence speed
1.0
1.0
0.8
Success Rate
Success Rate
0.6
Network Size
Training Paradigm
128
0.4
Combined loss with pre-training
0.4
256
Q-learning without pre-training
380
Q-learning with pre-training
0.2
512
Combined loss without pre-training
1024
1000 1500 2000
2500
3000
500
1000
1500
Maze Traversals
Maze Traversals
Figure 3: Left: Success rate (proportion of direct traversals to reward locations) of each set of training paradigms on the reward task, averaged over initial conditions and random wall colours using optimal rate of epsilon decay for each paradigm, each shown with a 95% confidence interval. Attractor landscape formed during pre-training alongside combined loss allows network to achieve perfect performance pre-training also achieves perfect performance in more than twice the number of maze traversals. Q-learning alone with pre-training takes far more maze traversals converge (and is less likely be optimal) due to the non-random initial state of network and inability to utilise the spatial map formed. Combined training without pre-training also takes relatively many maze traversals to converge due to relatively difficult joint task with no biased initial state. Right: Pre-trained network optimised with combined loss converges at similar rates with different network sizes above 380 units.
reward task in relatively few maze traversals. Q-learning alone without
3.1 EXTRAFIELD PLACE CELL FIRING
First, we investigate the representation learned by the the network during these two stages of training. Pre rre-training causes the formation of discrete attractors that resemble place cells in the hippocampus. Individual units in the network generally have well isolated place fields, which together cover the whole maze and therefore allow reliable decoding of agent location. In addition to an increase in activity in a particular unit when the agent moves across its respective place field, we also observe substantial extrafield firing of these units. This activity occurs mainly at the primary cue location and at the first choice point after pre-training. After training on the reward task, in addition to the place fields, the network also has units with extrafield activity at the secondary choice points (Fig. 4E).
In the top row of Figure 3(A-D) we show activity in 4 reward trained LSTM units obtained through the collection of unit activity from a full left sided trajectory from the maze start point returning to the start point with cues presented, together with full right sided trajectory. We show all activity from this activity collection in the top row of Figure 3(ADD) and proceed to outline the maze areas for each unit with activity higher than 30% of the peak activity of that particular unit (mirroring the experimental threshold used by Johnson & Redish| (2007)), denoting them as place fields corre- sponding these LSTM units. In experiments, rodents seem to pause at high consequence decision points (Johnson & Redish 2007) with alternating head movement behaviour signifying vicarious trial and error (VTE) (Muenzinger] 1938, Hu & Amsel 1995). In the activity plots in the bottom row of Figure 3(AD)), we simulate this using our reward trained model by running the agent from the start position at the bottom of the maze stem, then pausing it at the top of the stem, with left cue presented halfway up. We show activity above 60% of unit peak activity (identified with the previously collected aggregated activity) shown in addition to the previously identified place fields.
The network representation seems to sample both return arms, with surprisingly high extrafield activity in the shown LSTM units when the agent is paused at the maze choice point, a location for which these units do not usually have corresponding activity (Fig. 4A-D). We define nonlocal firing as unit activity above 60% of peak averaged unit activity when running the agent along the central stem (bottom row Fig. 4A-D) observe only the most poignant extrafield behaviour.
5

---

Under review as a conference paper at ICLR 2021
E
Figure 4: A-D) Top Activity maps showing well isolated place fields of four LSTM units (acting as place cells) indicated in dotted regions after the reward task. Place fields determined by contiguous locality with average activity exceeding 30% peak unit activity during single left tra- jectory followed by right trajectory. Bottom row: LSTM unit activity exceeding 60% of previously averaged peak unit activity for the given neuron when agent run from bottom of maze stem to top of stem and given a low frequency (left) cue tone halfway up the stem, then stationary at choice point with LSTT network repeatedly receiving observation from choice point for timesteps there- after (shown in addition previously determined unit place fields in dotted regions). A, B) Strong extrafield firing contiguously from cue to choice point. C. D) High extrafield firing choice point while agent is paused top of stem. E) Place fields (determined from average activity on both trajectories) of four LSTM units outlined in dotted areas after reward based task. High levels of consistent extrafield firing at primary and secondary points in 56% of LSTM units.
3.2 FORWARD MOVING REPRESENTATION
The internal dynamics of the LSTM network has an inherently forward looking representation the maze pre-trained in predictive manner. As depicted in Fig. 5 whilst the agent is stationary, the dynamics of the LSTM network moves forward through the maze, incorporating the trajectory modulation of the cue played halfway up the maze stem. The forward movement of the represen- tation is also notable for having an inconsistent velocity, where the LSTM inferred agent location jumps (Hasselmo 2009) from the top of the maze lower down the arm (timestep 16 in Fig.
L=
t= (choice) agent stops
t= 15
(cue)
t= Inforred position jumps
t= 26
Inferred position returns start
Figure 5: LSTM inferred agent position after pre-training on maze. The agent is from the start timestep 1 to timestep 4 where receives a low frequency cue (indicating left turn). At timestep 9 the agent is stopped the top of the maze stem and the LSTM is given the environment observation from this location for the remainder of the shown timesteps. The inferred position then moves left according to the cue with the position seeming to jump abruptly between timesteps 15 and 16. The inferred position then moves back to the starting position at timestep 26. We observe an analogous inferred forward moving representation on the right side of the maze with a high frequency cue.

---

Under review as a conference paper at ICLR 2021
t=
= (cue)
t= 9 (choice) agent stops
t= 14
t= 15 Inferred position switches
=
t= 31 Inferred position start location
= 32 Inferred position switches again
== 34
t= 38 Inferred position to start location
Figure 6: LSTM representation after reward training. As previously, we run the agent from the start position the top of the stem of the maze at timestep with a low frequency (left) cue tone at timestep 4. Again, the agent is stopped at this position with the LSTM network receiving the environment observation from this position for the remainder the shown timesteps. As with the network purely trained on the predictive task, the representation moves in the direction correspond- ing to the frequency the given cue tone. Then between timesteps 14 and 15, the inferred position jumps from the return arm with active reward sites to the alternate arm, with the inferred position moving from this position to the start location fairly consistently. Then the inferred position jumps again timestep 32 the rewarding return and moves constantly to the start position.
In stark contrast to the dynamics of the LSTM network after predictive pre-training, following train- ing the reward task the forward representation of the LSTM is still looking ahead of the agent but now displaying sweeping behaviour (Fig. 6 which identified experimentally in rats by Johnson & Redish (2007) when performing cue based tasks. When the agent is stationary the choice point, we observe the representation moving ahead the agent first in the direction corresponding to the cue given at the first cue point and then abruptly down the opposing arm of the maze towards the starting location, thereafter the representation moves down the correct arm (corresponding to the cue) and becomes stationary the maze start location. This path switching behaviour reliably ob- served in networks trained on the combined loss (Eq. 2 with and without pre-training, with differing numbers of units and initial conditions long as the reward task solved without backtracking at forward moving representation when
secondary cue locations. The network lacks a sweeping trained on the reward task with Q-learning alone, regardless of pre-training. Thus pre-training does not contribute sweeping path switching behaviour.
Right: Trajectory
night: with pause
Frimary Cue
Choice Point
Retur Start
Figure 7: UMAP manifold of LSTM network dynamics of complete left trajectory (dark blue) and complete right trajectory (red) shown along with manifold of dynamics when agent from start location to choice point with left cue (light blue) and right cue (pink) given at cue point and agent paused in place at the top of the maze stem. few timesteps after the agent is paused, the dynamics of the left cue paused agent (light blue) switches manifold path abruptly from running alongside the complete left trajectory path (blue) and joins the right trajectory path (red), following this for many timesteps before ultimately resulting at the same manifold end position as the complete left trajectory manifold path (blue). This analogous for the right cue paths (red and pink).

---

Under review as a conference paper at ICLR 2021
We further investigate the network representation using Uniform Manifold Approximation and Pro- jection (UMAP) (Melnnes et al. 2018). Figure 7 shows generally connected manifolds, with closer inspection revealing the dynamics which leads to the sweeping arm behaviour in Figure when the agent is stationary at the primary choice point. Zeroing visual input while the agent is paused at the choice point gives comparable representation dynamics to that observed in Figures[oande
Separately, we find that place fields of particular LSTM units drift forwards from their original firing positions after pre-training, towards the reward locations the return arms throughout reward training, as shown experimentally in CAI neurons in Lee et al. (2006). We observe this behaviour in 50 out of 380 network units (13%), with final resting locations place fields at reward locations (seen in Appendix Figure TO). This is possibly explained by the gradient of Q values (prediction of predicted reward) spreading backwards from reward locations (Hasselmo) 2005 and becoming stronger throughout training.
3.3 SELECTIVITY OF NEURONAL UNITS
In addition to a forward sweeping representation, this trained network also exhibits neural selectivity that closely matches hippocampal circuits. Griffin et al. (2007) reported that after reward learning, hippocampal neurons were more strongly selective for the encoding or choice phase a task rather than the direction of the organism's 's trajectory. We garner the preference of selectivity of each neuronal unit in our network using a discrimination index used by Griffin et al. (0077) for the turn direction selectivity (DTum) and the phase selectivity (IIbhwe):
F Rright
F Riee
F FReue F F Pibhaice
D tum
DIphue
Riight + FlR
F Feee + F Rehoice
where F Rright for a particular LSTM unit is the mean firing rate from the cue point on the central stem the choice point the top of the stem on trajectories where the agent turns right the choice point. Similarly F Pfaa is the mean stem firing rate when the agent turns left. FRe is the firing rate the cue (encoding) point averaged over both left and right trajectories and similarly F Reboice is the averaged firing rate at the choice (sampling) point.
80
Turn Direction
Task Phase
50
Number of LSTM units
Rummber of LSTM units
20
30
20
10
08 Right Seloctive
Loft Soloctive
Choice Selective
So octiviive
Figure 8: Histograms showing LSTM unit discrimination index for turn direction selectivity (D4w)) vs task phase selectivity (hppse)). A highly negative selectivity index for turn direction indicates neuronal unit which exhibits high levels of selectivity (uniquely high network activity) for leftward trajectory and a highly positive selectivity index indicates selectivity for rightward trajectory. A negative selectivity for task phase indicates a neuron which is highly selective for the choice (re- trieval) phase of the goal based task whereas positive index indicates a neuron which is highly selective for the cue (encoding) phase of the task.
The firing areas used for selectivity measurement are insets in Figure g the cue point to assess turn direction selectivity, and the cue/choice points to assess encoding and sampling (DI Figure & shows higher proportion of LSTM units strongly task selective rather than turn selective, with significantly more units having large absolute DIphase indices than DIum indices.
We use the stem above
In addition, the reward trained network is found to have a disproportionately high number of units (163 out of 380 LST units) with place fields the start location of the maze. Moreover, find
8

---

Under review as a conference paper at ICLR 2021
evidence of conditional destination encoding in these units which were heavily differentiated in their firing with respect to particular rewarding locations, as shown experimentally in CA1 hippocampal
place cells {Ainge et al. 2007| Wood et al., 2000; Ferbinteanu & Shapiro 2003). 59.5% of units with a place field at the maze start location fired uniquely at this point for rewarding locations on a particular return arm.
4 DISCUSSION
In this work we show that networks trained with a combined predictive and goal-based objective exhibit functional dynamics and selectivity behaviour coinciding with that of hippocampal neurons. We demonstrate that extrafield firing activity of network units emerge when a simulated agent, which is trained on goal based reward task in T-shaped maze, pauses decision points suggesting intrinsic dynamics are encoding the future trajectory of the agent. This mirrors experimental results in hippocampal place cells in rats (Johnson & Redish 2007; Frank al. 2000). At the same time, we find that networks using this combined objective, following pre-training only on sensory prediction task, can learn the correct goal direcciee behaviour much faster than an equivalent network with only Q learning objective.
Previous work shows that metric neural representations of environments form when an RNN is optimised to predict agent position from agent velocity (Cueva & Weil 2018 Banino et al., 2018) and non-metric representations form when an RNN is trained to predict future sensory events given direction of movement (Recanatesi et al., 2019). When training
model we do not provide the LSTM network with any explicit information about location or direction, only receives sensory information. This is similar to the purely contextual input received by the model pre-trained by Xu & Barak (020) where no velocity input is given, however, the network used by these authors is still trained on position and landmark prediction in supervised way.
Instead, our training paradigm forces the LSTM to maintain an implicit notion of movement within its internal state in relation environmental observations. This, in conjunction with the consider- ation that model-free RL methods such as Q-learning perform poorly on tasks in dynamic environ- ments such ours (Dolan & Dayan 2013), and the long term dependency on the delayed cue in perspective of the choice location, makes the task outlined in Figure particullary challenging.
Training on a sensory predictive task causes the formation of non-metri place cell like representa- tion in the activations of network units, similarly tolRecanatesi et al. (2019). These units demonstrate nonlocal extrafield firing (Appendix Figure ( and after reward training (Figure 4 Johnson Re- dish (2007) find that this extrafield firing is particularly striking at consequential decision points where rats usually pause in order to sample previously seen trajectories. We observe that cue or choice point extrafield activity is evident in most LSTM units after training the reward task. This is likely due to the increased precedence these points have in the agent reaching reward lo- cations. Together the trained LSTM network units form a representation which sweeps along the paths available to the agent, first down the reward path and then the other, as shown in Figure 6 and demonstrated in rats in Johnson & Redish (2007).
Although hippocampal place cells are critical for spatial memory Nakazawa et al. 2002 Florian &
Roullet 2004. (andi et al.] 2003) Redish & Touretzky 1998 Miller et al. 2020). currently un- clear by what mechanism an ensemble of place cells contributes to representation of goal-directed
behaviour (Morris 1990). Our model and training paradigm is in keeping with the hypothesis that the hippocampus is involved in maintaining a conjunctive representation of cognitive maps and sen- sory information (Whitiintton
al. 2019). We show that this paradigm can be extended with pre- dictive learning of Q-values of anticipated future reward, and show that the resulting representation well suited for learning actions leading from cue to a reward. Importantly, this representation emerges solely from sampling sensory inputs and predicted rewards, while reinforcement learning itself remains model-free and is initially random. The surprising similarity of the task-dependent activity in our simulations and experimentally recorded neural activity in similar tasks suggests that the model may replicate central aspects of learning and planning in the hippocampus. Our trained model could improve understanding of hippocampal function by testing hypotheses regarding pre- viously unobserved dynamics inexpensively. This could be performed on maze environments such as this work, or open arena settings once the model is retrained.

---

Under review as a conference paper at ICLR 2021
REFERENCES
James A. Ainge, Minija l'amosiunaite, Florentin Woergoetter, and Paul A. Dudchenko. Hippocam- pal CAI place cells encode intended destination on maze with multiple choice points. Journal of Neuroscience, 27(36), 2007. ISSN 02706474. doi: (0.152//NEUROSCL.20 1-07.2007.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel. Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne. Hubert Soyer, Fabio Viola. Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Charlic Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukeuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid-like repre- sentations artificial agents. Nature, 2018. ISSN 14764687. doi: 10. N0.1038/s41586-011 0180000266.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Damitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP 2014 2014 Conference on Empirical Methods in Nattural Language Processing, Proceedings of the Conference, 2014. ISBN 9781937284961. doi: 10.3115/v/d114-117
Christopher J. Cueva and Xue Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In 6th International Conference Learning Rep- resentations, ICLR 2018 Conference Track Proceedings. International Conference on Learning Representations, ICLR, 2018
Christopher J. Cueva, Peter Y. Wang, Matthew Chin, and Xue-Xin Wei. Emergence of functional and structural properties the head direction system by optimization of recurrent neural networks. In International Conference on Learning Representations, 2020. URL http:::/ 2 HK1SeREtPB
net /utt/Eo-ee Eorum ?ice-ummiidiii
Ray J. Dolan and Peter Dayan. Goals and habits in the brain, 2013. ISSN 08966273.
Janina Ferbinteanu and Matthew L. Shapiro. Prospective and retrospective memory coding in the hippocampus. Neuron, 40(6), 2003. ISSN 08966273. doi: 10.1016/50896-62730000
Cédrick Florian and Pascal Roullet. Hippocampal CA3-region is crucial for acquisition and memory consolidation in Morris water maze task in mice. Behavioural Brain Research, 154(2), 2004. ISSN 01664328. doi: 10. 00.1006/.bbb::2440003
Loren M. Frank, Emery N. Brown, and Matthew Wilson. Trajectory encoding in the hippocampus and entorhinal cortex. Neuron, 27(1), 2000. ISSN 08966273. doi: 10. (0.1016/50896-627((00) 00018-0.
Marianne Fyhn, Sturla Molden, Menno P. Witter, Edvard I. Moser, and May Britt Moser. Spatial representation in the entorhinal cortex. Science, 2004. ISSN 00368075. doi: 10. 126/science. 1099901.
Amy L. Griffin, Howard Eichenbaum, and Michael E. Hasselmo. Spatial representations of hip- pocampal CA1 neurons are modulated by behavioral context in a hippocampus-dependen mem- ory task. Journal of Neuroscience, 27(9), 2007. ISSN 02706474. doi: 10. 10.URRS/ 4083-06.2007.
Michael E. Hasselmo. A model of prefrontal cortical mechanisms for goal-directed behavior. Jour- nal of Cognitive Neuroscience, 17(7), 2005. ISSN 0898929X. doi: 10. 00011622099899954475199999
Michael E. Hasselmo. A model of episodic memory: Mental time travel along encoded trajectories using grid cells. Neurobiology of Learning and Memory, 92(4), 2009. ISSN 10747427. doi:
10.1016//..lm.20099 0t0.0/0....000000000000 .0.00000............
Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Memory. Neural Computation, 1997. ISSN 08997667. doi: 10. 116/neco.19977777 1735..
Dan Hu and Abram Amsel. A simple test of the vicarious trial-and-error hypothesis of hippocampal function. Proceedings of the National Academy of Sciences of the United States of America, 92
(12), 1995. ISSN 00278424, doi: 10. 1...003/pna..22.12.55.
10

---

Under review as a conference paper at ICLR 2021
Adam Johnson and A. David Redish. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience, 275) 2007. ISSN 02706474. doi: 10.152//NEUROSCL376...000
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Inah Lee, Amy L. Griffin, Eric A Zilli, Howard Eichenbaum, and Michael E Hasselmo. Gradual Translocation of Spatial Correlates of Neuronal Firing in the Hippocampus toward Prospective Reward Locations. Neuron, 51(5):639-650, 2006. ISSN 0896-6273, doi: http:///do.oor/100.1011 Ciceuron.206.06.000 URL http::ttt//// //tttp://ww..r sci wwweeniieeeeeeeeeeeoeoeeeeeeeeee rect com coom/sclence/arttilee
pi 211/80896627306000
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software, 3(29), 2018. ISSN 2475-9066.
doi: 10.21 105/joss.008 00861.
Thomas D. Miller, Trevor T.J. Chong, Anne M.Aimola Davies, Michael R. Johnson, Sarosh R. Irani, Masud Husain, Tammy W.C. Ng. Saiju Jacob, Paul Maddison, Christopher Kennard, Penny A. Gowland, and Clive R. Rosenthal. Human hippocampal CA3 damage disrupts both recent and remote episodic memories. eLife, 9, 2020. ISSN 2050084X. doi: 10.7554/eLife.411853
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
RGM Morris. Does the hippocampus play disproportionate role spatial memory. Discussions in Neuroscience, $:39-45, 1990.
Karl F. Muenzinger. Vicarious Trial and Error at a Point of Choice: I. A General Survey of its Relation to Learning Efficiency. Pedagogical Seminary and Journal of Genetic Psychology, 53 (1), 1938. ISSN 08856559. doi: 10. 10.1080/08856559. 1938. 10533799.
Kazu Nakazawa, Michael C. Quirk, Raymond A. Chitwood, Masahiko Watanabe, Mark F. Yeckel, Linus Sun, Akira Kato, Candice A. Carr, Daniel Johnston, Matthew A. Wilson, and Susumu Tonegawa. Requirement for hippocampal CA3 NMDA receptors in associative memory recall. Science, 297(5579), 2002. ISSN 00368075.
doi: 10.1 10.1166/ciencce100777333
OeKee and I.. Nadel. The hippocampus as a cognitive map. Clarendon Press, Oxford, United Kingdom, 1978.
Stefano Recanatesi, Matthew Farrell, Guillaume Lajoie, Sophie Deneve, Mattia Rigotti, and Eric Shea-Brown. Signatures of low-dimensional neural predictive manifolds. Cosyne Abstracts 2019, Lisbon, PT., 2019.
A. David Redish and David S. Touretzky. The Role of the Hippocampus in Solving the Morris Water Maze. Neural Computation, 10(1), 1998. ISSN 08997667. doi: 10.1 00008666709776660000000110010000
Carmen Sandi, Heather Davies, M. Isabel Cordero, Jose J. Rodriguez, Victor I. Popov, and Michael G. Stewart. Rapid reversal of stress induced loss of synapses in CA3 of rat hippocam- pus following water maze training. European Journal of Neuroscience, 17(11), 2003. ISSN 0953816X. doi: 10.1046/j.1460--0 11w.6/.1440-956822000000 9568.2003.02675.x
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-Learning. In 30th AAAI Conference on Artificial Intelligence, AAAI 2016, 2016. ISBN 9781577357605..
Christopher J. C. H. Watkins and Peter Dayan. Q-learning Machine Learning, 8(3-4), 1992. ISSN 0885-6125, doi: 10.1100766009992698.
James CR Whittington, Timothy H. Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens. The folman-EEchenboe Machine: Unifying space and relational memory through generalisation in the hippocampal formation. bioRxiv, pp. 770495, September 2019. doi: 10.1 101/70197
11

---

Under review as a conference paper at ICLR 2021
Emma R. Wood, Paul A. Dudchenko, R. Jonathan Robitsek, and Howard Eichenbaum. Hippocampal neurons encode information about different types of memory episodes occurring in the same location. Neuron, 27(3), 2000. ISSN 08966273, doi: 10.
Tie Xu and Omri Barak. Implementing inductive bias for different navigation tasks through diverse mn attrractors. In International Conference on Learning Representations, 2020. URL https:
A APPENDIX
A.1 EXTRAFIELD PLACE CELL FIRING AFTER SENSORY PREDICTION TASK
After pre-training on the sensory prediction task outlined in Figure I we observe that when the agent is paused at the top of the stem of the maze the network representation moves far ahead of the agent, caused by extrafield activity of many neurons the network.
B
C
Figure A, B, C) Top row: well isolated place fields of three LSTM units indicated dotted regions after pre-training Place fields determined by contiguous locality with average activity exceeding 30% peak field activity during a single left trajectory followed by right trajectory. Bottom row: agent from bottom of maze
top of stem (and given low frequency tone halfway up the stem) and paused at choice point with LSTM network repeatedly receiving observations from choice point for timesteps thereafter. A) Strong extrafield firing choice point with some activity at the cue point. B) Extrafield activity at choice point and at position below. C) High extrafield firing cue point before agent pauses top of stem.
12

---

Under review as a conference paper at ICLR 2021
i 0
== 200
400
i= 600
i= 800
== 1000
i= 1200
1400
Figure 10: Place fields of four LSTM units, starting from where the network has been pre- trained on the sensory prediction task, drifting forwards towards reward locations throughout reward training (where is the number of training iterations). The place fields ultimately rest at maze reward locations at the end of reward training (i 1400).
13


Reviews:
[{"review_id": "rNajhMVKXIu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n \nThe authors trained a recurrent network to perform a sensory prediction task and this gave rise to units that resembled hippocampal place fields. Then they augmented the network with a Q-learning objective and shown that the activity in the network sweep forward in space if the agent is fixed at a decision point. \n\n##########################################################################\n\nReasons for score: \n \nOverall, I think the paper should be rejected. The work is interesting, but the clarity of the paper is not at the level of the findings. I think the authors should enhance exposition, and strengthen some analysis. The findings are really interesting ,but at the current stage I don’t think the paper is ready to be published. However, I’m happy to revise my score if authors addresses my comments. \n\n##########################################################################\n\nPros: \n \n1. Combination of unsupervised learning and RL to show that units in a recurrent network can be used to understand spatial and non-spatial firing patterns in the hippocampus\n\n2. Figure 4 and 5 are convincing.\n\n\nCons:\n\n1. While explaining the task the authors claim that the colours are chosen at random, however it is not clear whether these stay fixed across episodes or they changed. Intuitively it looks like, once the colour are generated, then they stay fixed, but in this case it is difficult to understand why a simple Q-leaning objective won’t be able to solve this task (as claimed by the author). The only reason I can think about is if the steps between the cue and the reward are longer that the ones allowed by the discount factor chosen (0.8). However, in the paper there are no details about the number of steps or how the discount affect the results. I think this is a serious issue.\n\n2. The authors claim that they first pre-train on the predictive task, but then in the loss of eq. 3 they report a combined loss. Does it mean that the loss_{rgb} is also fine-tuned while training with the Q-learning objective? This point need clarification. \n\n3. The paper doesn’t report any details about the learning rates or the sizes of the linear layers used in eq. 1 and 2. This way is impossible to replicate this results. This is a serious issue.\n\n4. How the threshold for place cells are defined? Why 30%? How many units show firing above that threshold? This needs further analysis to support the decision, which otherwise seem very arbitrary. \n\n5. Are figure 4 and 5 just cherry picked run or are these averaged across several testing runs? Also in the captions of these figures the authors are using the singular “observation ” in figure 4 and the plural “observations” in figure 5. Does it means that the analysis have been performed differently? Or is it just a mistake?. This is an important point as I think the results will be more powerful with the same image fed as input, or even better with no image, just with 0ed input to simulate pondering.\t\n\nMinors:\n\n1. 380 units seems quite an unconventional number of units, why not 256 or 512? Can you please explain. Also is this number affecting the representations? Have you done a sweep and settle on this number because it support your findings better? If so, it would be important to mention it.\n\n2. It would have been nice to have analysis to support the following sentence on page 4: Generally, when the network loses its ability to self-localise the agent, state-action values are no longer reliable indicators of future reward potential as the current environmental state is not clearly discernible. Otherwise please correct it.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pOReP3ueqT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by biological considerations, this paper shows that recurrent networks trained with a predictive and goal-based objective on a maze finding task, qualitatively recapitulate experimental findings in hippocampal recordings in rodents trained on the same task. In particular, these LSTM networks demonstrate both metric representations of their environment and nonlocal extrafield firing at decision points along the maze (anticipating the future trajectory of the agent).\n\nStrengths:\n+ I like that the authors take a normative approach that exhibits both metric and non-metric place cell representations of the environment, unifying prior findings in one model.\n+ I appreciate that no velocity input is given to their model, in contrast to prior approaches.\n+ I also liked the qualitative comparisons to hippocampal recordings from rodents trained on the same task (especially Figures 6 and 7).\n\nWeaknesses:\n- The primary conclusion, namely, training an RNN on a maze-like environment gives you place cells, is really not all that new, especially considering that the network is still supervised to predict position and landmarks.\n- Given that lack of novelty in the modeling conclusion, it would have therefore been nice to have seen more quantitative comparisons to hippocampal recordings in rodents. Does their approach explain more variance in these neurons than prior approaches? Otherwise, it seems that they simply recapitulate prior qualitative comparisons.\n\nMinor comments: “Recurrence based” should be changed everywhere to “recurrent” (e.g. on pg. 1), and “Neuroscience” is not capitalized. The motivation to use double Q learning should be expanded on in pg. 3 prior to equation 3.\n\nQuestion: The authors mention that Q-learning performs poorly on tasks in dynamic environments – however, I do not see any evidence of this in the paper, it would be imperative to show this explicitly for the environments they consider. Suppose this is in fact the case, could you clarify what makes your approach more successful at this task than others? Is it because of the pretraining to predict the subsequent observation of wall colors from the current wall color observations?\n\nAs it stands, I think the ideas of this paper are interesting and think it unifies prior approaches, but I do not think the conclusions from the modeling add all that much novel insight from prior approaches. Therefore, I recommend a weak accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lF-NjoDyDrY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\n\nThe main goal of this paper is to show that LSTM units in a network trained to solve a T-maze task, show similar activity patterns as neurons in rats solving a similar task. Specifically, the authors make the following claims: (1) an RNN learning the task by  a combination of reinforcement and predictive learning produces internal representations with consistent extrafield firing associated with consequential decision points, (2) the network’s representation, once trained, follows a forward sweeping pattern similar to those found in rats and (3) a higher proportion of units in the trained network show strong selectivity for the choice phase of the task than for spatial topology, as seen in rats.\n\n**Pros**\n1. The submission is clear, well-written and the execution is competent. \n2. I find the approach well motivated and the problem interesting for the current state of the field.\n3. The authors provide a sufficient amount of details so that reproducibility should be possible.\n\n**Cons**\n\nI have concerns about key points of the paper and the interpretation of the results:\n\n1. One of the main results of the paper is the observation that the network produces a forward-moving representation similar to the one observed in rats at the decision point. However, the way the authors simulate this is by freezing the agent at such point and keeping the LSTM running, repeating the same constant observation. The LSTM was trained on trajectories on the maze, so in this trajectory (which was never used during training), the network is completely out of distribution. The activity of any network in this situation is difficult to interpret. Because the network was trained with a predictive loss on two very specific sequences of observations, it seems plausible that it is robust to the change of input statistics and follows the same sequence, maybe with some instabilities. Note that the cue was present, which explains why the correct sequence is followed. \nIn the network trained also by RL in particular, the authors interpret this as “The agent appears to be sampling the trajectory concerning the alternate return arm of the maze before ultimately settling on the rewarding return arm”. But this is, in my opinion, an over-interpretation, as the agent has no sampling capability in the first place (there is no generative model of observations), nor any particular planning mechanism. Alternatively, the authors may be claiming that this jumping behavior happens only after the RL training and not before, in which case they should emphasize this difference and quantify it explicitly. Although in this case, a simple explanation for this could be that due to the epsilon-greedy, only during the RL training the network is exposed to the wrong cue-arm combination. Therefore, the LSTM would be less able to rely on the cue, which could explain the jump between attractors in the out-of-distribution case. In the discussion section, the authors claim “We demonstrate that extrafield firing activity [..] emerges when a simulated agent [...] pauses at decision points - suggesting intrinsic dynamics are encoding the future planned trajectory of the agent.”. I find this to be an over-claim, as the agent doesn’t pause (it can’t) and doesn’t plan (for any common definition of planning). I would be more convinced if the agent was able to pause (as an additional action) and this behavior was observed in the LSTM activity in this situation, which is closer to the biological case.\n\n2. The pre-train stage is done on trajectories that correspond to the solved task. This means that the LSTM trained by the predictive loss is not exposed to the general structure of the environment but to the specific solution of the task, including the cue-choice association and the exact sequence of observations in each of the two correct trajectories. The authors draw a parallel with the pre-training phase in behavioral experiments (Johnson & Redish, 2007) in which rats usually run each trajectory separately (by having the other one blocked). However, in my opinion, this is problematic for their analysis. First it’s unclear to what extent the network is learning by RL as during the pre-training it has already learnt to predict the observation corresponding to the correct turn (wich corresponds to one of the two actions). Second, the LSTM is exposed only to the correct cue-arm trajectories, which I think is the reason why the forward-looking sweeps only follow these trajectories (see previous point). This is more similar to a demonstration than to pre-training. A more conventional pre-training would leave the agent to explore freely to implicitly learn the structure of the environment (a la Tollman). On the other hand, the argument of following the protocol of the behavioral experiments also doesn’t fully work as the rats are still producing motor outputs and even being rewarded during the pre-training phase (Johnson & Redish, 2007).\n\n3. Finally, a more general concern is the main point of the paper. If I understand correctly, the main claim is the similarity of the observations between the RNN agent and the experimental findings in rats. However, given that there are plenty of arbitrary choices when training an RNN, I believe the results are not particularly explanatory.  I would encourage the authors to formulate better alternative hypothesis and controlled experiments. For example, I would find it interesting to show that the forward-sweeping observations done in rats, which is often interpreted as a signature of planning or prediction of the consequences of future actions, arises simply from a next-step prediction loss in an overtrained rat.\n\nMinor concerns: \n\n- “As such, a network of Gated Recurrent Units [...] or vanilla RNN units was unable to perform well in either the pre-training or joint RL task due to these prevalent long term dependencies.”  How many steps are there between cue and choice, and between choice and reward? \n\n- Related to the previous point: “We attempted to run the reinforcement learning task alone in a maze with no wall colours or environment statistics except the cue. In this scenario the network is not able to learn the task due to a lack of self-localisation.” If I understand correctly, there is a constant number of steps between the cue and the moment where the choice has to be made. I would tend to believe that an LSTM can learn to make a prediction only based on the number of timesteps, regardless of the lack of wall observations (e.g. 2 sequence problem in Hochreiter and Schmidhuber, 1997).\n\nDetails: \n\n- It would be good to clarify what exactly is the action set of the agent. \n- I would like to know how exactly are activity maps obtained. The authors mention “Place fields determined by contiguous locality with average activity exceeding 30% peak unit activity during a single left trajectory followed by a right trajectory” but I don’t find this particularly clear. I also found it difficult to understand the bottom row of Fig 3. \n- Fig 5 should be referred to in the paragraph starting with “In stark contrast to the dynamics of the LSTM network following predictive pre-training...”\n- Why is the return to start representation in Fig. 6 different for right and left trajectories?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_Sf3NTtx0zd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors train a recurrent neural network on a navigation task, and observe the emergence of several phenomena reminiscent of the hippocampus: appearance of place cells with a secondary receptive field at task-relevant locations; anticipation of possible future paths in the activity of the model, with alternation in time between possible future paths; a high proportion of neurons tuned to task variables rather than animal trajectory.\n\nStrong points:\n- these findings are compelling, they account for some hallmark properties of the activity of hippocampus, and they could lead to a better understanding of the role and function of the hippocampus.\n- the experiments are rigorous and convincing.\n\nWeak points:\n- some technical aspects of the paper could be clarified (see below)\n- it is unclear how this model improved our understanding of the hippocampus function, and whether the model makes any testable predictions about the hippocampus.\n\nI recommend to accept this paper because of its strengths listed above.\n\nClarification questions:\n1) I did not understand the role of secondary cue point. Why were these required in addition to the primary cue point and choice point?\n2) vocabulary: What are \"metric\" and \"non-metric\" representations?\n3) How reliable is the alternative path visiting phenomenon? Can this phenomenon be observed reliably in networks trained from different initial conditions?\n4) I did not understand the consequences/take-homes of the second and third paragraph discussions. \n\nAdditional feedback:\n1) It would be interesting to see a discussion on what we learned about the function of the hippocampus from this model, and/or what predictions this model makes about neural activity in hippocampus. \n2) Are there any oscillations and phase-precessions phenomena in the model? If not, it would be interesting to discuss why these oscillations might be present in the brain but not in the model.\n3) Could this network be used to simulate difficult experiments, e.g. understand how future paths exploration works in an open-field setting?\n4) The first sentence of the abstract is difficult to understand. In general, shorter sentences could improve clarity.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyses a recurrent neural network model trained to perform a simple maze task, and reports that the network exhibits multiple hallmarks of neural selectivity reported in neurophysiological recordings from the hippocampus— in particular, they find place cells which also are tuned to task-relevant locations, cells which anticipate possible future paths, and a high proportion of neurons tuned to task variables.  

The reviewers appreciated the interesting empirical analysis, and the demonstration that multiple such features could arise in the same neural network— to the best of my knowledge, this had not been demonstrated explicitly before. However, there were also multiple concerns, which lead to this paper beeing discussed extensively and controversially. In particular, it is not clear which features arise from which learning objective, for example, for place cells to arise, do we  just need sensory prediction, or do we need q-learning? In addition, there were some points in which the tightness of the analogy between model and biology is questionable— in particular, this refers to the comprising between hippocampal recordings and the evaluation of the network.  Finally, it is also clear that  some of these observations reported in the paper are, indeed, empirical observations rather than explanations. Because of these shortcomings, there was no consensus and strong support from the reviewers for acceptance of the paper.

After extensive discussion between both the reviewers, the AC and the program chair, the final decision was to not accept the paper. We do hope that the reviews will help you in improving the study and its presentation. It clearly has potential to be a valuable contribution to the literature. 

Decision: Reject