HIPPOCAMPAL NEURONAL REPRESENTATIONS IN CONTINUAL LEARNING
Authors: Samia Mohinta
Rui Ponte Costa
Stephane Ciocchi
Venue:  (2020)

Abstract: The hippocampus has long been associated with spatial memory and goal-directed
spatial navigation. However, the region’s independent role in continual learning of
navigational strategies has seldom been investigated. Here we analyse populationlevel
activity of hippocampal CA1 neurons in the context of continual learning of
two different spatial navigation strategies. Demixed Principal Component Analysis
(dPCA) is applied on neuronal recordings from 612 hippocampal CA1 neurons
of rodents learning to perform allocentric and egocentric spatial tasks. The components
uncovered using dPCA from the firing activity reveal that hippocampal
neurons encode relevant task variables such decisions, navigational strategies and
reward location. We compare this hippocampal features with standard reinforcement
learning algorithms, highlighting similarities and differences. Finally, we
demonstrate that a standard deep reinforcement learning model achieves similar
average performance when compared to animal learning, but fails to mimic animals
during task switching. Overall, our results gives insights into how the hippocampus
solves reinforced spatial continual learning, and puts forward a framework
to explicitly compare biological and machine learning during spatial continual
learning.

Under review as a conference paper at ICLR 2020
HIPPOCAMPAL NEURONAL REPRESENTATIONS IN CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The hippocampus has long been associated with spatial memory and goal-directed spatial navigation. However, the region's independent role in continual learning of navigational strategies has seldom been investigated. Here we analyse population- level activity of hippocampal CA1 neurons the context of continual learning of two different spatial navigation strategies. Demixed Principal Component Analy- sis (dPCA) is applied on neuronal recordings from 612 hippocampal CAI neurons of rodents learning perform allocentric and egocentric spatial tasks. The com- ponents uncovered using dPCA from the firing activity reveal that hippocampal neurons encode relevant task variables such decisions, navigational strategies and reward location. We compare this hippocampal features with standard reinforce- ment learning algorithms, highlighting similarities and differences. Finally, we demonstrate that standard deep reinforcement learning model achieves similar average performance when compared to animal learning, but fails to mimic ani- mals during task switching. Overall, our results gives insights into how the hip- pocampus solves reinforced spatial continual learning, and puts forward a frame- work to explicitly compare biological and machine learning during spatial contin- ual learning.
INTRODUCTION
The hippocampus has been long known to play a crucial role in spatial navigation (O'keefe & Nadel (1978); Tolman (1948). However, there still an ongoing debate in the field on how exactly the hippocampus is involved in the encoding and learning of multiple tasks, such allocentric and egocentric spatial tasks (Ekstrom et al. (2014); Feigenbaum & Rolls (1991); Fidalgo & Martin (2016)) In particular, it is not clear how the hippocampal neural population may encode such tasks and how these representations relate to the ability of animals for continual spatial learning.
Fidaloo
In reinforcement learning (RL), despite the growing interest in the topic, how to solve the continual learning problem is still unresolved (Parisi et al. (2019)). Previous theoretical work has explored how animals may rely on the hippocampus learn explore given environment based by combining RL algorithms and spatial coding (Foster et al. (2000)). More recently, there has been growing interest understanding how solve continual learning in deep neural networks (Zenke et al. (017); Kirkpatrick et al.) (2017)). But how both biological and artificial neural networks encode and solve spatial tasks in a continual reinforcement learning setting is not known.
Here, we first focus on analysing neural data from hippocampus CA1 population-level activity to extract meaningful latent features. We find representations that are directly related how the neural population encodes not only multiple tasks, but also reward signals, such as reward prediction errors. Next, we contrast these experimental observations with standard reinforcement learning algorithms, namely temporal difference learning and Q-learning theory. Finally, show that deep Q networks can achieve an average performance similar to animals, but fails to achieve the same relearning speed.
2 RESULTS
Our results are based on a dataset of neural activity from total of 612 hippocampal CAI neurons, recorded during bewerd-based T-mazee traversal task (Ciocchi al. (015)). For the behavioural

---

Under review as a conference paper at ICLR 2020
10 Plu mose conwent tl Timeue North
Rute to
Trai-avereged firing redea 50 neuroma
Start
age trap sao
Gpecomiicaule Chunoight N
East
Aloocmniic
Time Cit ssconce.
1o most")
right)
Figure I: Experimental setup and neuronal data. (A) Experimental task setup. (i) The experimental results are based plus maze which can be converted to T-maze by temporarily blocking the opposite start arm (yellow box) after pseudo-random selection of a start location during an allocentric egocentric spatial task. There are two possible start points (North and South) and two reward arms (West and East). During a training session the rules (or tasks) are switched multiple times. (ii) Representation of the allocentric task in which the animal has navigate the where a reward is located. (iii) Representation of the egocentric task in which the animal has to turn right get reward irrespective of start point. Figure adapted from Ciocchi al. (2015). (B) Example of trial-averaged z-scored firing rates of 30 randomly selected CAl neurons out of 612 around the reward point (vertical dashed line).
arm
experiment five food-deprived rats were trained to get sugar pellet rewards located at the extremities of rewarded arms in T-maze (Fig. [ Rats were trained to learn two behavioral rules in order to be rewarded. The first is based on allocentric spatial references (Fig. ( and the second on egocentric spatial references (Fig. [Aii). We analyse a total of 6 seconds of neuronal activity per trial across several days (around 500 trials in total). For illustration, trial-averaged and normalized firing activity of 30 randomly selected neurons from the recorded CA1 ensemble is presented in Fig. DB. From the raw firing rates is it virtually impossible to uncover meaningful trends around the reward delivery zone (dotted vertical line). As result, we decided focus on population-level analysis of the neuronal activity.
Recently a task-specific linear dimensionality reduction method, Demixed Principal Component Analysis (dPCA), has been introduced. This algorithm has previously been tested on population- level data from the prefrontal and orbifrontal cortices by Kobak et al. (Kobak et al. (2016)). but not on data from hippocampal spatial tasks. Next, we describe our results obtained using dPCA and how these may be related to reinforcement learning algorithms.
2.1 BEHAVIOUR- COMPONENTS IN CA1 NEURONAL POPULATIONS
We use demixed PCA to extract meaningful features from the data. Like PCA, dPCA (Kobak et al.) (2016)) allows us extract a lower dimensional representation of the data, but in contrast to PCA in dPCA this representation is constrained such that the components extracted encode behaviourally relevant features (e.g. spatial strategy or correct vs incorrect decisions; see Supplementary Mate- rial for more details on dPCA). The dPCA components extracted from the hippocampal CA1 data explained above are given in Figure 2
Although individual neurons do not exhibit a clear tuning for decision (i.c. correct or incorrect trials)
or strategy (i.e. allocentric or egocentric tasks; ef. Fig. B),, at the population level we observe clear separation of the neural representations for decisions and strategies (Fig. - Interestingly, we observe clear separation in population activity patterns for correct and incorrect decisions in the leading demixed decision component (Fig. 23. first row, component 3). It is interesting to note that the separation between correct and incorrect decisions increases gradually over time. This increased separation starts before the reward delivery and increases substantially afterwards, being its maximum where we believe to be the moment during which animals consume the actual reward (i.e. about second of crossing the reward sensor represented by vertical dashed line). This
2

---

Under review as a conference paper at ICLR 2020
rewarddrepressentation appears to be related but also different from standard reward prediction errors predicted by RL algorithms (see more below) and classical observations in dopaminergic neurons (Schultz et al. (199)); Wirth et al. (2009)).
In terms of multi-task representation, we find a clean separation at the population- between allocentric and egocentric tasks (Fig. F3 second row, component 4, suggesting that the hippocampus is directly involved in the learning and representation of these two tasks.
Temporal components capture the most variance in the data and their population activities reveal reward-specific or
responses (Fig. 23, third row). We interpret the first temporal component (1) as being analogous an eligibility trace, the second temporal component (2) as being encoding reward location and the third temporal component (5) as exhibiting traces of reward prediction error signals as seen in classical conditioning experiments (Schultz et al. (1997); Fig. 2B, third row). Note that the first decision component (3) may be also interpreted as some form of (weak) reward prediction error as discussed above.
A
Component [18.0 (%)
Component [1.5%]
(1.2%)
Eglend
Prodi Prod rouerct
(PCA
(3.%)
Nome
18/5
Component (1245]
Component 92 (12.855
(341)
Rwas
Encading meard lcoaion
Component (1.25]
Correct Decision
Alcoootis
T0me60
Ziee
Figure 2: Demixed principal component analysis (dPCA) reveals hidden features from CA1 hip- pocampal recordings. (A) Cumulative variance explained by dPCA and PCA components. (B) Demixed components for each experimental variable: decision (incorrect in black and correct in brown), strategy (allocentric in solid line and egocentric in dashed line), time and mixed deci- sion/strategy. Vertical dashed line represents the crossing of the reward sensor. We expect that the actual reward to be consumed ~1 afterwards. (C) The variance explained by each demixed principal component and total variance per experimental condition (pie chart). Figure follows same structure as in the original dPCA (Kobak et al. (0166).
These population features we observe should become better defined over learning. That is consistent with what we find when comparing earlier trials with later trials (Fig. 3). For example. the separation between the two tasks (egocentric vs allocentric) increases in later trials. A similar trend is observed between correct and incorrect trials. Finally, the eligibility trace-like component also becomes more pronounced in later trials.
2.2 COMPARISON BETWEEN STANDARD RI. ALGORITHMS AND HIPPOCAMPAL. REPRESENTATIONS
Here contrast the features observed in the hippocampus with ideas commonly found in RL theo- ries.
We first use simple temporal difference (TD) learning algorithm with eligibility traces in 1D environment (see SM for more details). Presence of eligibility traces postulated by TD models is similar to the ones we found in our experimental data (Fig. Bi). Regarding evidence for TD
3

---

Under review as a conference paper at ICLR 2020
A Initial Experimental Sessions
Final Experimental Sessions
Compomst (1a4
Figure 3: Demixed PCA for carlier and later training sessions. The first decision, strategy and interaction components along with the first three temporal components are plotted from dPCA on neuronal firing rates from the initial (A) and final (B) experimental sessions. The components from later days cleaner and highlight the influence of learning the tasks over time. Vertical dashed line represents the crossing of the reward sensor.
errors, our results only loosely support standard TD errors, in that the 3rd component distinguishes between correct and incorrect trials (Fig. Aii. However, from Fig. 3 we can see that this component is initially stronger (i.e. in earlier trials) and becomes weaker with learning around the expected reward location (i.e. Is after crossing the sensor), consistent with changes in TD errors over learning.
Next we used epsilon-greedy Q-learning in a grid-world setting to model the experimental task (see SM for details) we observe separation between the two tasks in the value function (i.e. allocentric and egocentric; Fig. (C,D), which appears to be similar to the one observed experimentally. How- ever, as expected, this is not sufficient for the model to learn the two tasks in continual learning setting as we highlight next.
Indeed, our Q-learning model fails to learn the navigational strategies consecutively (Fig. 5J3 in contrast to animals that can sustain performance above chance level (Fig. We show the per- formance of animal SC03 demonstrate that animals can learn both allocentric and egocentric strategies as is noticeable from the mean performance for both tasks being above chance level. The model, on the other hand, is unable to switch over and recall policies learned for the allocentric task at trials 212-216. This is because, expected, the policy information pertaining to the tasks is over- ridden as tasks or starting locations change over. This clearly highlights the continual forgetting in standard RL systems. However, it should be noted that some forgetting can be observed in animals as well ( A).
The incapability of standard Tabular Q-learning algorithm to reach a performance similar to animals lead us to develop a deep Q-Network that may be able to better solve the two tasks.
2.3 COMPARING DEEP Q-NETWORKS AND ANIMAL PERFORMANCE
We implement a deep Q network (DQN) with task-specific information provided to the neural net- work as input. Such task-specific information the DQN allows the network to learn the different rules of reaching reward locations and this method draws inspiration from the works of Anonymous et al. and Serra et al. (Authors) [eerá et al. (2018)) In addition, there is evidence that specific neuromodulators (Acetylcholine) may provide such task-specific information. Nevertheless, unlike earlier works our DQN does not require a separate memory layer to store the task-specific infor- mation corresponding to the different tasks. Salient task-specific information for inclusion in the blocked cells of the input maze matrix are chosen with the help of a trial and error mechanism for training the network with different positive and negative values of information. We used the first 250 trials from our animal data (Fig. (5) to train and test the network ability to learn and remember the different tasks. We used Adam optimizer to train the network.
4

---

Under review as a conference paper at ICLR 2020
Simple farming elightey T908
Norrokiss fang rate
-00
0
giliftyy traeo
Ogpeerbiic
noemolledd fingras
Component 43
Component th
MYTAABIN:: Fins aat
0
Sesarate Strstagis
(e)
Figure 4: Comparing RL simulation results with low-dimensional dPCA components. (Ai) Value and eligibility trace from TD(X) model simulated for a ID environment. (Bi) Firing activity pattern in dPCA component is akin to the simulated eligibility trace pattern. (Aii) TD errors before (light brown) and after learning (dark brown) from TD(A) model simulation. The model uses look-ahead strategy for error estimation. (Bi) Weak reward prediction signals around the reward delivery point. Error prediction signals signalling trial outcomes after reward point. (C) Separation of strategies based on Q-value estimates. (D) Distinct firing activity in dPCA component 4 pertaining allocentric and egocentric tasks.
The DQN consists of two hidden layers of 100 neuron units each. The input layer is of the size of the input maze, while the output layer contains four neurons with the Q-values for the four different actions that agent can take to change its state in the maze. Additionally, we found PRelu to be the most suitable activation function to solve this reinforced spatial navigation task involving allocentric and egocentric navigational strategies.
Interestingly, we observe that this network with task-specific memory achieves a performance for both allocentric and egocentric tasks comparable animals (animal SCO3 is given as typical example (A. In particular, the DQN learns the egocentric task more efficiently than the allocentric task as demonstrated by the higher mean performance (dashed blue) line in Fig. 6 Even though the network struggles with the allocentric, it shows a mean performance above the chance level. It should also be noted that the DQN performs better in the egocentric task, while the animal finds it slightly more difficult to learn the egocentric spatial navigation task rules. In other words, animal SC03 performs marginally better in the allocentric tasks than in the egocentric tasks (Fig. (AA)
We compared our results from this DQN with task-specific information with a DQN without task- specific information and this network performed significantly better than the other one (results not shown), whose performance accuracies for both allocentric and egocentric tasks were below the chance level.
Mean performance across all Q-leaming models including deep Q-networks are compared in the following Fig. A against the mean performance of animal SC03 over learning the allocentric and egocentric tasks for 250 trials. To highlight the ability of the different systems for continual learning we plotted the re-learning performance of the Q-learning models and the animals after the net- works/modelss and the animal has already been exposed to the two different task types (Fig. 7 B,C)).
A DQN that learns both tasks consecutively as the animal does in the behavioural experiment achieves mean performance comparable to the animal (green and orange bars in Fig. 7A). A DQN learning only learning the egocentric task but with a reversal of reward locations (see SM Fig. (i.ivv) performs better (peach bar) than the animal. On the other hand, a DQN learning only
5

---

Under review as a conference paper at ICLR 2020
Contineel lourning of strutegies acrose days unimal 5033
strategies
Triar
Alccoer9CC Moollin Pig. 1MBcaamiO
Egine MFF
(6goorati1)
Cla Laval
Alocserric Pirtt
Epccetthc
Figure 5: Behavioural performance in animals and Q-learning model during task switching. (A) Performance accuracy for animal SC03. (B) Performance accuracy for simulated model. perfor- mance of I represents correct and incorrect for both egocentric (blue) and allocentric (red). Solid darker lines represent moving averages. Abrupt changes in both model and animal occur within given strategy because of changes the start location (e.g. around the 80th trial). Color-coded vertical dashed lines indicate moments of task switches. Dashed yellow lines denote a change of reward locations while following particular strategy. Dashed purple lines denote the change from allocentric to egocentric task or vice versa.
Bariigg locomdyy
Maryling ((gunarddr)
Lovel
Tark markir
Figure 6: Task switching simulation using deep Q network. Solid darker lines represent moving averages. Task-specific information is added in the input maze to facilitate differentiation of the allocentric and egocentric rules by the network. Mean performance in both tasks is above the chance level. The neural network achieves improved continual learning of both tasks.
the allocentric task with reward location reversals fails to acquire a performance close to both the DQN-Ego and the animal (peru colored bar). Tabular Q-learning model shows the worst mean per- formance amongst all the models as well against the animal's 's mean performance. This is expected because a standard Tabular Q-learning model is inefficient in incorporating and sustaining complete information corresponding to the different strategies.
Additionally, we speculate that the DQN-Ego acquires a better performance than the DON-Allo since the DQN-Ego encounters far less reward location change overs (dashed 'Strategy Switch' lines) when compared to the DQN-Allo model. Providing task-specific information in the DQN also shows clear improvement in continual learning of tasks. We postulate that the DQNs somehow learn to utilize this extra information (memory weights for tasks), perhaps internally segregating the activity patterns (even weights) corresponding to the task types. In contrast, Tabular Q-learning model without any extra information about the tasks struggles to reach comparable performance as the animal does. This also leads to the hypothesis that the hippocampus CAI might also be using

---

Under review as a conference paper at ICLR 2020
similar mechanism to encode the rules involved with the allocentric and egocentric navigational strategies and consequently, remember the reward locations as the task change over trials.
(A) Comparing performance accuracies Q-learning models
1.0
Performence Actumay
0.8
I
I
I
0.6
0.4
0.2
0.0
Animal SC03
Tab-Q
DON-Alle
DQN-Ego
DQN
Re-learning switching Egocentric
Re-learning after switching Allocentric
1.0
re-terning Performane
Pe-tarning Parformanee
0.8
0.8
M MEW
M
0.4
0.4
0.0
0.0
106
116
126
136
146
156
166
206
21.6
226
236
246
Chance
Tab-Q
DoN--go
DON
Avuraga
Clancc Tea-0
DON-Aja
D4M
-Pr.. Avw.nve
Figure 7: Comparing mean performance of all Q-learning model simulations with the animal's 's mean performance. Re-learning performance on allocentric and egocentric tasks are also plotted from the Q-learning models and the animal. Color-coded dashed lines at trials 106-109 and 206-209 display average performance the networks and animal in the last few trials of the previous strategy was learning. (A) Even the animal is unable to achieve perfect learning of both tasks (orange bar). A DQN learning both tasks achieves mean performance almost at par with the animal. A DQN learning only the egocentric task shows better mean performance opposite to the animal SC03's actual behaviour. Mean performance a DQN model learning only the allocentric task is superior to Tabular Q-learning model approximating optimal policies for both task types. Error bars represent the standard error of the mean for each model. (B) Re-learning performance of each Q-learning model and the animal the point of reintroduction of previously learned egocentric task. The learning speeds are color-coded similar to the bar plots. The animal has the best re-learning speed (yellow), while the DQN takes while catch up (green). (C) Re-learning performance of each Qelearning model and the animal at the point of reintroduction of a previously learned allocentric task. The animal has an above chance level re-lnnng performance, but the DQN shows slow re-learning speed. Even the animal starts from a better prior re- re-learning performance value, while DQN does not show accurate prediction immediately after the switch.
Performance re-learning the tasks in each Q-learning model and the animal is plotted against trials after a task type is switched to another in Fig. BCC.) The animal starts with higher re- learning performance value than any of the models at both allocentric and egocentric task switch points. It also achieves rapidly relearns the switched-to task and this is observed from the orange line values above chance in Fig. 3 and Fig- 7 The DQN learning both tasks together shows a slow re-learning speed and fails to immediately re-learn the switched-to task. However, this network on average attains a performance close to the animal as can be noticed from comparing the green and orange bars in Fig. A for the switch to the egocentric task (brown line in Fig. 7B) than when the task is changed to allocentric rule (brown line in Fig. 7B). Networks learning only egocentric (DQN- Ego in Fig. (B) and allocentric (DQN-Allo in Fig. 7C have better re- re-learning speeds than the DQN learning both together. However, the re-learning performance for both DQN-All and DQN-Ego is inconsistent across trials.

---

Under review as a conference paper at ICLR 2020
Overall, standard deep RL models can achieve performance comparable to animals, but fail to continually learn. Although animals relearn faster, they are still not perfect relearners.
3 CONCLUSION AND DISCUSSION
Overall, our results using dPCA show that the hippocampus encodes task variables such as decisions, rewards and strategies, which suggests that the hippocampus plays key role in the ability for animals to perform continual spatial learning. We also demonstrate that some the components find are consistent with theoretical models in reinforcement learning. Our results suggest the existence of a non-standard form of TD learning implemented or used by the hippocampus. This raises exciting possibilities for future research. One avenue to develop model that gradually learns to separate the representations of the two tasks inspired by recent developments in continual learning where different groups of neurons may specialize in different tasks explaining the features and the ability for continual learning that we observe in experimental neuroscience (Zenke et al. (2017); Kirkpatrick et al. (2017). Our work also puts forward a framework that can be used to better compare how both machines and animals learn to solve continual learning problems.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anonymous Authors. OVERCOMING CATASTROPHIC FORGETTING THROUGH WEIGHT CONSOLIDATION AND LONG-TERM MEMORY.
S Ciocchi, J Passecker, H Malagon-Vina, N Mikus, and T Klausberger. Brain computation. selective information routing by ventral hippocampal CAI projection neurons. Science, 348(6234):560- 563, May 2015.
Arne D Ekstrom, Aiden G F Arnold, and Giuseppe laria. A critical review of the allocentric spatial representation and its neural underpinnings: toward a network-based perspective. Front. Hum. Neurosci., 8:803, October 2014.
Janet D Feigenbaum and Edmund T Rolls. Allocentric and egocentric spatial information processing in the hippocampal formation of the behaving primate. Psychobiology. 19(1):11-40, March 1991.
Celia Fidalgo and Chris B Martin. The hippocampus contributes to allocentric spatial memory
through coherent scene representations. J. Neurosci. 36(9):2555-2557, March 2016.
D Foster, G Morris, and P Dayan. model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus, 10(1):1-16, 2000.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinnse Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting March 2017.
in neural networks. Proc. Natl. Acad. Sci. U. S. A., 114(13)352113322
Dmitry Kobak, Wieland Brendel, Christos Constantinidis, Claudia E Feierstein, Adam Kepecs, Zachary F Mainen, Xue-Lian Qi, Ranulfo Romo, Naoshige Uchida, and Christian K Machens. Demixed principal component analysis of neural population data. Elife, 5, April 2016.
John O'keefe and Lynn Nadel. The hippocampus as cognitive map. Oxford: Clarendon Press, 1978.
German Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Netw., 113:54-71, May 2019.
W Schultzz Dayan, and R Montague, neural substrate of prediction and reward. Science, 275 (5306):1593-1599 March 1997.
Joan Serrà, Dídac Surís. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. arXiv [cs.LG], January 2018.
Richard S Sutton and Andrew G Barto. Introduction Reinforcement Learning. MIT Press, Cam- bridge, MA, USA, 1st edition, 1998.
E C Tolman. Cognitive maps in rats and men. Psychol. Rev., 55(4):189-208, July 1948.
Sylvia Wirth, Emin Avsar, Cindy C Chiu, Varun Sharma, Anne C Smith, Emery Brown, and Wendy A Suzuki. Trial outcome and associative learning signals in the monkey hippocampus. Neuron, 6 61())933,9940, March 2009.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. March 2017.
A SUPPLEMENTARY MATERIAL
METHODS OVERVIEW
A.1 DEMIXED PRINCIPAL COMPONENT ANALYSIS
Demixed Principal Component Analysis (dPCA) (Kobak et al. (2016) not only reduces the dimen- sionality of the data without losing too much information, but also facilitates visualization of neural

---

Under review as a conference paper at ICLR 2020
responses tuning experimental parameters like decisions, rewards and navigation rules. Another advantage of demixed PCA over standard Principal Component Analysis (PCA) methods is that it does not enforce orthogonality of the components and hence, enables successful demixing of population-Ievel activity. We applied the recent version of the demixed PCA algorithm Kobak et al. (2016) trial-averaged neuronal firing rates from the hippocampus CAT of all five rats across all experimental days. The trial-averaged data was organised into a matrix with dimensions Neurons Decision Strategy X Time. Note that 'reward' was not used as a demixing category (or task variable) in the algorithmic implementation since the influences of rewards could be detected from the firing patterns in demixed decision and strategies components itself. Additionally, we used a regularisation term in dPCA's cost function to prevent the model from overfitting.
A.2 TASK SIMULATION AND TEMPORAL DIFFERENCE LEARNING
The task is simulated in 5 x 5 grid (Fig [). with rewards being placed only at the terminal states. The simulations consider rule-switching and random start- location switching conditions in the original experiment. However, the simulation design does not incorporate use of spatial cues for the allocentric task.
TD(O) and TD(A) algorithms are employed in the simulations to analyse the back propagation of estimated value and TD error from the rewarding states/time-poins the states/time-ponnt coming before the reward states. We used two value functions at this stage for the two different strategies to see how each compares against the other. We also use eligibility traces to verify the notion of the hippocampus employing decaying memory trace of any important spatiotemporal event. Eligibility traces enable the model to distribute credit efficiently to past states that can predict future rewards. These traces have been shown to augment learning in reinforcement learning models (Sutton & Barto 1998)).
Furthermore, greedy online Q-learning algorithm to model policy learning (what actions to take when and where) of animals the allocentric and egocentric tasks is implemented. We add to the e-greedy approach the following Q-value update function in Equation I ensure rapid convergence of the model.
Q(Sr, = Q(S,, a R[441
+ a mar, -((,,,
(1)
Here, is the learning rate, y is discount factor and A() is the Q-value of a state-action pair at time t.
We think of this slight modification in the algorithm as combining best of the two worlds (SARSA and offline Q-learning) and this variant as tice-as-greedd online Q-learning. Nevertheless, we also verify our results from this modified model with a standard SARSA model to observe any discrepancies in learning policies, which we do not find. Additionally, this model does seem to converge slightly faster than a standard SARSA model(perfomaance metrics not shown).
A.3 SEPARATE DQN NETWORKS TO LEARN ALLOCENTRIC AND EGOCENTRIC STRATEGIES
We use deep Q neural networks simulate the behavioural experiment. Figs A and 9 show the results from using two deep Q-networks to learn the allocentric and egocentric tasks separately. We include task-specific information into the input maze to assist the network in recognizing changing reward positions while following single type of navigational rule. It seems that Deep Q neural networks can learn more efficiently than the Tabular Q-learning algorithms. Overall performance accuracies of both the networks are above chance levels, which shows that the network is able to cope with changing start and reward positions within a particular type of strategy (allocentric or egocentric). However, moving averages in the allocentric task show the occurrence of catastrophic forgetting as the network struggles to cope with changing reward locations while starting from the same start positions. On the other hand. the DQN that learns only the egocentric task performs perfectly after 130 trials and is even able to deal with a change in reward locations at trial 153 (dashed line). Changing reward locations within a task type are also denoted as strategy switches in the figure.
10

---

Under review as a conference paper at ICLR 2020
Egocentric Spatial Rules
B Allocentric Spatial Rules
-1
-1
-1
-1
-1
-1
-1
-1
0
1
0
-1
-
-1
-1
-1
-1
-1
-1
(i) Start South
Start South
-1
-1
-1
-1
-1
-1
-1
-1
0
0
0
1
0
0
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
[ii Start North
Start North
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
0
-1
0
0
1
0
0
0
2
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
(iii Start North
(iii Start North
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
1
0
0
0
0
0
0
0
0
-1
-1
0
-1
-1
-1
-1
0
-1
-1
-1
-1
-1
-1
-1
-1
0
-1
-1
(iv) Start South
(iv] Start South
Figure 8: T-maze layout in 5 5 matrix grid for the online Q-learning simulation. Grid layouts mimic the original allocentric and egocentric rules in the behavioural experiment. The value -I denotes un-walkable cells because it is an elevated plus maze (convertible to T-maze). The value denotes walkable cells. The middle row has trajectories that should be chosen by the animal/agent are marked with color-coded arrows.
either of its extremes represent reward. Ideal
11

---

Under review as a conference paper at ICLR 2020
Contisual loamning Allecentric Strotogy acress trialsMays with similated DON
MAWA
leaming the Egocentric Stradagy acroas trialcdays simulated DON
Plar
Place
Figure 9: Task switching simulation using separate deep Q networks. Total number of trials over which the networks learn the tasks independently is 250. Solid darker lines represent moving averages. Dashed lines denote change of reward locations during a type of task and can be called strategy (witch.(A) Performance accuracy of DQN learning the allocentric task. (B) Performance accuracy of DQN learning the egocentric task.
12


Reviews:
[{"review_id": "r1xep1EJ9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an analysis of hippocampal neuronal activity in rodents during spatial navigation tasks. Features were extracted from the data using a component analysis technique, and these features were then compared to quantities which arise during training of the DQN reinforcement learning algorithm. \n\nUnfortunately I am not well versed enough in this literature to assess the merits of this submission. However, I could not find any glaring issues, unclear sentences or any other obvious sign of incompetence. \n\nMy apologies for the inadequacy or this review (and/or the paper assignment algorithm).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gnCnjpYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on the problem of continual learning. It first proposes an analysis of neural recordings from rodents hippocampus that perform a task related to continual learning. The authors then claim to identify similarities in representations of behaviorally relevant variables between biological networks and standard artificial RL systems. Finally the authors propose a DQN implementation of the task reaching performance similar to rodents, although the implementation fail to perform continual learning.\n\nI have found the results of the analysis of neural data interesting and well explained. However I haven’t been able to judge positively the main part of the manuscript pertaining to the comparison with artificial agents, maybe because of the lack of clarity in the exposition of the results (see detailed comments below). As such I think this manuscript is not ready yet for publication in ICLR. \n\nDetailed comments:\n\nIntroduction:\nThe introduction reads well, it would be useful to have a short paragraph explaining the vast problem of continual learning, and situating the approach of the authors in this vast field. In particular it would be useful to explain in what sense the task considered is related to continual learning (switches between allocentric and egocentric tasks that are not informed by experimenter, but that the rodent has to figure out), because for now it is only briefly stated in the legend of figure 1.\n\nAnalysis of neural data: \nAnalysis of neural data using dPCA reveals interesting results regarding the representations of correct/incorrect trials, allocentric/egocentric tasks, temporal structure. Globally these results are well presented. Regarding figure 3, it would be nice to define what are early and late trials, is it a distinction inside a block where late trials are just before a switch. Or are late trials after weeks of training, in which case I do not really see the link with continual learning ?\n\n\nComparison between standard RL and hippocampal representations:\n\nThe idea to compare hippocampal representations and standard RL implementations is original and interesting. The exposition of the results, however, lacks important information for me to assess the relevance of these comparisons:\n\nFig4A(i): could the author explain what is on the x-axis ? What is the 1-D environment mentioned in the caption (in SM a 5*5 grid world is mentioned) ? Why not run TD on the same setting as Q-learning ? What is the black curve « value after learning » ? Why value depends on the x-axis ?\nFig4C: How are the two curves obtained ? As both tasks share the same Q function, it might be worth explaining how the two curves are obtained. \n\nThe authors mention that « policy information is override as starting location changes over » ? Could this fact be explained ? It would seem to me that starting points north s_north and south s_south would have distinct Q(s_north,.) and Q(s_south,.) avoiding overriding.\n\nAlso I am quite surprised that Q-learning fails on such a simple task (cf Fig5B), could the authors explain how many trials would be required to reach perfect performance by focusing on either of the two tasks ?\n\nIn this section a natural extension could be to use hierarchical RL, or options to model behavioral strategies such as allocentric and egocentric and compare with the hippocampal recordings.\n\n\nComparing DQN and animal performance:\n\nA first crucial point to clarify is whether the DQN receives information about what task to perform (it is mentioned: « task-specific information provided to the neural network as input«, «this network with task-specific memory«, then a DQN « without task-specific information » is mentioned). This seems rather crucial, because if information about allocentric VS egocentric task is provided, contrary to the behavioral task for rats, then the network is asked to do strategy switches based on some cues, which is a different problem than continual learning.\n\nIf the DQN receives information about whether it is in a allocentric or egocentric trial, I am very surprised it is not able to perform this simple task. \nCould the authors explain why the DQN has similar performance as the animal while it is shown to relearn slower than animals (is it because it learns better before task switch). \nIn figure 7, the curves for the model are very noisy and hence difficult to interpret, it would be nice to show averages over many same task switches, to get a clearer picture.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyedURFjFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyses a dataset of representations in the CA1 region of the hippocampus of a rat conducting a spatial plus maze task that switches between allocentric and egocentric versions. In the allocentric version of the task, the rat must always go from north or south arms to the west arm to receive a reward. In the egocentric version the rat must always turn right, regardless of whether they are starting in the north or south arm. These two versions of the task are switched after some period of time.\n\nThe researchers conducted demixed principal components analysis (dPCA) on the data, then compared the activity of the components to aspects of a tabular Q-learner. They argue that some of the components match variables used by the Q-learner. Following this, they examined the ability of both the tabular Q-learner and a deep Q-network to perform this task, and compared their performance to the rat. They find that the rat is better than the Q-learners at continually updating from the egocentric to allocentric task, and vice-versa.\n\nThe goals of this paper are fantastic. I really like the attempt to link hippocampal activity with RL representations. But, this study is very muddled, and there are very serious problems with the paper that render it inappropriate for acceptance to ICLR. The five most major issues are:\n\n(1) The choice of the demixing categories is missing a crucial category: space! Given the importance of the hippocampus for spatial representations, surely a big component of the variance can be explained by the animal's location in space. Why not include this? It might in fact be that some of the \"time\" components are just reflecting the usual spatial location of the animal during different components of the task.\n\n(2) The identification of components from the neural data with things like reward prediction errors, eligibility traces, etc., is all done in a qualitative manner with no statistical controls. This lack of quantitative assessment for some of the key claims in the paper is just not okay for an academic paper. Moreover, even the qualitative claims are underwhelming. The authors' claim, for example, that time component #1 is an eligibility trace is a real stretch, in my opinion (see more below in point 3).\n\n(3) The comparison between the animal data and the Q-learner is also done in a qualitative fashion that was extremely underwhelming. Fig 4 A & B were the most egregious. Are the authors really attempting to claim that the curves in 4A are clearly related to the curves in 4B? That's a real stretch, and to provide no quantitative assessment of this claim renders it completely unconvincing.\n\n(4) The fact that model-free reinforcement learning algorithms cannot adapt in changing environments/tasks has been known for a long time. As such, the result showing that the Q-learners cannot switch easily between the tasks is not novel. See, for example, this paper: https://www.sciencedirect.com/science/article/pii/S0896627313008052\n\n(5) Even if we accept the central claims from this paper, there is very little provided for machine learning researchers at ICLR to benefit from. What about the hippocampal representations makes the rats better at continual learning? What inductive biases or memory mechanisms might we glean from this work? Nothing like that is provided. As it stands, even being charitable, this paper really only speaks to a neuroscience audience, since even the Q-learning components are used only to understand the neuroscience data, not to think about how this could inform new ML systems or theories.\n\nIn addition to these problems, there are several small ones:\n\n- Was only one animal included in this analysis? That's never stated, but Figs. 5 & 7 seem to suggest that. Not only does this 100% need to be stated, but it's very problematic from a generality standpoint.\n\n- What type of recordings were these? How were individual cells identified (e.g. spike sorting)?\n\n- How was the Deep Q-net trained exactly? The authors say it was trained on the first 250 trials from the animal, but that's confusing? Was it not trained to perform the task itself? Also, where is all the info on memory buffers, hyperparameters, etc. There is no way to reproduce these simulations given the lack of detail here.\n\n- Some of the plots are confusing and hard to follow. For example, in figure 7 B and C, what determines the X-axis? What should I be looking for in the curves? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1lC7WG5iS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work sits at the intersection between neuroscience and machine learning. It proposes to use neural recordings from the rodent hippocampus to shed light into how biological agents achieve continual learning. The approach involves (i) analyzing neural data from the rodent hippocampus with the goal of identifying how neurons encode various behavioral variables; (ii) training different RL agents to solve a computer version of the same task, including tabular and DQN agents; (iii) contrasting the performance and representations of the artificial agents with those of animals.\n\nThe paper has an admirable goal: to find links between neuroscience and machine learning, using tools from one to promote advances in the other. When done correctly, this interaction can be fruitful and produce incredible insights for both fields. However, I believe that this paper does not deliver on either end due to major methodological and analytical flaws, rendering it innapropriate for ICLR. Below I list my major critiques:\n\n1. The interpretations of the dPCA components seems very preliminary and lacks methological rigor. In particular, many alternative interpretations of the components are possible. For instance: Component #3 might represent outcome (rather than decision); Component #4 might represent greater engagement of the hippocampus in the egocentric task (e.g. Packard and McGaugh, 1996); etc. Moreover, the reported analyses do not include animal position, which will almost certainly be a major driver of population activity in the hippocampus (see work on place cells). Since the animal position is not reported, it is impossible to know if the dPCA components (e.g. Component #2) are, instead, representing alocentric position. Including animal position (perhaps also as an explanatory variable in the dPCA) might help, but the authors should also do a more thorough job testing their specific hypotheses beyond simply reporting them based on visual inspection.\n\n2. The comparison between early and later training sessions is also rather crude and qualitative. The authors makes several claims about differences that are not tested directly with a proper hypothesis test.\n\n3. The comparison between RL quantities and dPCA component is also very weak. For instance, the claim that a given dPCA component represents an eligibility trace needs much more evidence than simply showing that this component decays over space when the eligibility trace also decays. With respect to Component #4 (a critical component presumably related to the authors' hypothesis of multi-task representation), the authors report that, for an e-greedy agent, the average value is higher for alocentric vs. egocentric task. Yet, this difference is not investigated further and, again the authors claim victory based on a simple visual comparison between two plots (4C vs. 4D). \n\n4. For most of the paper, the authors report the results from a single (typical?) animal. Ideally, the results for all animals would be reported (or some statistically sound aggregate of all animals).\n\n5. Finally, the authors compare the performance of animal 3 with the performance of different RL agents. Again, this comparison is incredibly superficial and neglects many confound variables. The fact that the accuracy of both the animal and DQN is around 70% is not sufficient to claim that DQN is a good model for the animal's behavior, or that it is superior to other models that achieve slightly worse or better performance. Much more work needs to be done to properly compare RL and animals (e.g. comparing, in addition to performance, representations across various layers, prediction error signals, reward signals, etc). \n\nOverall, while I commend the authors for an intriguing idea, the execution of the idea is so poor that I consider the paper to be of little interest to either the neuroscience or the machine learning communities. Therefore, I cannot recommend the paper for publication at ICLR.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes neural recording data taken from rodents performing a continual learning task using demixed principal component analysis, and aims to find representations for behaviorally relevant variables. They compare these features with those of a deep RL agent.

I am a big fan of papers like this that try to bridge between neuroscience and machine learning. It seems to have a great motivation and there are some interesting results presented. However the reviewers pointed out many issues that lead me to believe this work is not quite ready for publication. In particular, not considering space when analyzing hippocampal rodent data, as R2 points out, seems to be a major oversight. In addition, the sample size is incredibly small (5 rats, only 1 of which was used for the continual learning simulation). This seems to me like more of an exploratory, pilot study than a full experiment that is ready for publication, and therefore I am unfortunately recommending reject.

Reviewer comments were very thorough and on point. Sounds like the authors are already working on the next version of the paper with these points in mind, so I look forward to it.

Decision: Reject