universal Hopfield network framework has recently been introduced to en-
compass the traditional Hopfield network, modern variants and related
models (Millidge et al. 2022). These models recall memories with a common
sequence of operations: similarity, separation and projection. It has been
shown that retrieval from a DND is also done with such operations. Hence,
a DND is an instance of the universal Hopfield network framework.

This novel link contributes to the field of associative memory. The sim-
ilarity function of DND is Euclidean and has already been shown to yield
high capacity (Millidge et al. 2022). On the other hand, the *k*-nearest neighbor is not commonly used as a separation function for associative memory.
Thus, one of the present objectives was to assess its performance. Interestingly, *k* controls the separation function, and setting *k* = 1 is equivalent
to using the max function studied by Millidge et al. (2022). While having
theoretically unbounded capacity, the max function can transition sharply
from one memory attractor to another when noise of increasing amplitude
is added to the query. With Figure 6.1, it was hypothesized that higher *k*-
values could be better at modelling datasets and improve the performance
assessed in absolute terms. Simulations indeed revealed higher capacity
and better retrieval from noisy queries with *k* > 1, especially with simple
datasets like MNIST. However, these results depend on the way performance is evaluated. In Millidge et al. (2022), the evaluation of retrieval is
based on some distance evaluation of the memory output and the actual
image, which must not exceed some threshold fixed by the experimenter.
This is a widespread method for evaluating associative memory models,
but one must choose the threshold wisely, as setting it too high can result in false positives with the model reproducing statistics of the dataset
grossly. This for example seems to be the case when retrieval is assessed
using the MNIST dataset. The performance of 50-Max (and even the identity function) remains high despite very strong noise. Therefore, another
way of evaluating retrieval was used which does not consider the output in
absolute terms, but rather compares it to the whole memory set. Retrieval
is deemed correct if and only if the output resembles the actual image more
than any other stored memory. Memory models thus cannot benefit from
modeling statistics of the dataset, and must rather focus on recalling the
distinguishing characteristics of the query. Using this method, the max
function outperforms the others. Ideally, performance should be evaluated
in both absolute and relative terms to ensure that recall is accurate and
stands out from other memories.

This raises the question of what is the function of associative memory.
Modeling statistics of a dataset is related to generalization, which is typi-
cally the main goal of machine learning. The objective of associative mem-
ory is somewhat different. Instead of generalizing, an associative mem-
ory aims to recall the exact information corresponding to the individual
memory. This is reminiscent of the division of labor between episodic and
semantic memory. When it comes to episodic control however, that is the
use of episodic memories for action control, some generalization is desirable. This is especially the case in neural episodic control in which the
selection of actions only relies on episodes, the hippocampal module con-
stituting a bottleneck. Initially, episodic control (not to be confused with its
implementation in neural episodic control) has been introduced as a way
of speeding up the learning of reinforcement agent and, after the initial
episodic control phase, it is desirable that more robust controllers can take
over (Lengyel and Dayan 2007). A biologically inspired alternative to neural
episodic control would be to supplement episodic memory with other con-
trollers whose function is to generalize. The hippocampus would then no
longer be a bottleneck, and could instead be devoted to memorizing the
specifics of situations.

Another thing that was briefly explored is the use of different functions
for neural episodic control. The simulations were done in simple environ-
ments, as a proof of concept. It is possible that the Manhattan function
could consistently outperform the Euclidean kernel, as it does on the asso-
ciative memory tasks. As their biological plausibility has yet to be proven,
the Manhattan and Euclidean functions will not be used in the remain-
der of this thesis. The softmax function, which has been proven powerful
in transformers (Vaswani et al. 2017), could also improve episodic control
agents with a more informed choice of $\beta$. More conclusive evidence could
come from training agents to play Atari games, as in the original NEC study, and optimizing hyperparameters such as *k* and *β*. Unfortunately, a proper benchmark of the new functions on Atari games would require far greater computing resources than are presently available in the laboratory.

The next chapter focuses on how modern Hopfield networks can be con-
textually modulated. Then, Chapter 8 will present a model of how they
can be implemented in the hippocampus.
<figure><img src="image_47.png" /></figure>
# 7 CONTEXTUAL CONTROL OVER MEMORY RECALL

Before presenting the new sequence learning model, the connections from the prefrontal cortex to the hippocampus are studied. In order to guide behavior efficiently, recalled memories should be relevant to the current situation. More precisely, the memory selection process should be influenced by both contextual and sensory information. In contrast to the information that is directly accessible to the agent through its perception of the environment, contextual information encompasses any internal representation that is not directly perceptible, but is susceptible to help the agent predict future rewards and states. This includes information about the temporal unfolding of events, spatial integration, but also the goals pursued by the agent. While stimulus-driven retrieval has been studied both empirically and theoretically, much less is known about the contextual modulation of episodic recall. A better comprehension of the mechanisms involved is necessary to develop a complete theory of episodic memory, but also to develop machine learning algorithms that can comprehend a wider variety of spatiotemporal contexts and direct their behavior towards goals. As reviewed in Chapter 5, the prefrontal cortex controls the hippocampus to recall contextually relevant memories. In the standard framework (Chapter 4.5), episodic memories are thought to be stored in recurrent collaterals of CA3. However, there is no direct anatomical connection from the prefrontal cortex to CA3 (Andrianova et al. 2023), and it is thus not clear how retrieval in the latter can be modulated by the former.

In a modeling study (Pilly et al. 2018), contextual information was fed to the dentate gyrus to modulate recall. This is at odds with the current un-
derstanding of prefrontal-to-hippocampus projections (Chapter 5; Howard Eichenbaum 2017b). The aim is not to compete with their model, but rather to study how prefrontal signals can modulate memory through the entorhinal cortex. An artificial neural network with Hopfield associative memory is used to study performance and contextual representations based on the memory locus and the target of contextual information.

## 7.1 METHODS

### 7.1.1 TASK

Like the model of Pilly et al. (2018), our model performs the task described in Peters et al. (2013). Object-discrimination problems are presented in two different contexts and used to evaluate context-guided memory recall. In a given task $t_i$ and context $c$, the network learns to choose a rewarded object $R_{i,c}$ over a non-rewarded object $\bar{R}_{i,c}$. In the alternative context $c'$, it learns to choose between the same objects as in context $c$, except that the non-rewarded object in context $c$ is rewarded in $c'$, and conversely (i.e. $\bar{R}_i^c = R_i^{c'}$ and $R_i^c = \bar{R}_i^{c'}$). Objects are represented in the network as 6 × 4 matrices with 6 active entries of value 1 placed at random locations, and other entries with value 0 (Figure 7.2).

### 7.1.2 MODEL

The hippocampus model is a fully-connected neural network which follows the anatomy of the biological medial temporal lobe (Figure 7.1). Information flows through a trisynaptic pathway (Input → EC<sub>II</sub> → DG → CA3 → CA1 → EC<sub>V</sub> → Output) and a monosynaptic pathway (Input → EC<sub>III</sub> → CA1 → EC<sub>V</sub> → Output). The number of neurons in each region is set as the number of neurons in the analog rat region (as retrieved from Cutsuridis et al. 2019), scaled down by a factor N = 0.003 (Table 7.1). The network input consists of three pattern slots: two for the choice objects and one for the rewarded object (Table 7.2). The assignment of $R_i^c$ and $\bar{R}_i^c$ to the first
<figure><img src="image_48.png" /><figcaption>Figure 7.1: Architecture of the model. The deep learning network is trained to autoencode its input (green), and the context when indicated (red). The associative memory is modified during the encoding phase, and used to recall memories during the test phase. In the standard framework, it is in CA3 (blue), but other locations are also tested here (transparent blue). Context information can target various layers (transparent red), but the current understanding (Howard Eichenbaum 2017b) is that it is provided to the entorhinal cortex (red).</figcaption></figure>

or second slot is random to prevent the network from learning to always output the same slot. ReLU is used as the activation function of hidden layers and sigmoid is used in the output layer to map latent activity back to the pattern space. An additional input is used to modulate layers of the hippocampal network contextually. It represents the two possible contexts as a two-dimensional one-hot vector. When a layer is targeted by this con-textual layer, this simply means that it receives it as an additional input, in a feedforward way. Note that the output layer is never modulated, since it does not model a medial temporal lobe region.

One of the hippocampal layers (CA3, unless otherwise mentioned) is augmented with an autoassociative memory similar to the modern Hopfield network of Ramsauer et al. (2020). During the memory encoding phase, activity vectors of the hippocampal layer are manually added to the au-
<table><caption>Table 7.1: Network summary.</caption><thead><tr><th>Layer</th><th>Output size</th><th>Function</th></tr></thead><tbody><tr><td>EC<sub>II</sub></td><td>110,000N</td><td>ReLU</td></tr><tr><td>EC<sub>III</sub></td><td>250,000N</td><td>ReLU</td></tr><tr><td>DG</td><td>1,200,000N</td><td>ReLU</td></tr><tr><td>CA3</td><td>250,000N</td><td>ReLU</td></tr><tr><td>CA1</td><td>390,000N</td><td>ReLU</td></tr><tr><td>EC<sub>V</sub></td><td>330,000N</td><td>ReLU</td></tr><tr><td>Output</td><td>3 × 6 × 4</td><td>sigmoid</td></tr></tbody></table>

<table><caption>Table 7.2: Input summary. The "Objects" input size corresponds to the product of the number, width and height of input patterns.</caption><thead><tr><th>Input</th><th>Size</th></tr></thead><tbody><tr><td>Context</td><td>2</td></tr><tr><td>Objects</td><td>3 × 6 × 4</td></tr></tbody></table>

toassociative weight matrix $X$. During the test phase, activity of the layer $a'$ is completed with memory recall before being sent to the next layer:

$$a = X\mathrm{softmax}(\beta X^\top a'), \qquad (7.1)$$

where $\beta$ is set to 1,000 to strongly separate memories.

## 7.1.3 SIMULATION DETAILS

All task-context combinations are presented once during each epoch, in random order. Similarly to the recently proposed autoencoder model of the hippocampus (Santos-Pata et al. 2021), the network learns to output its input, namely the two input object patterns and the rewarded one, as well as the context vector (unless otherwise mentioned). This is done by updating feedforward weights using backpropagation, in a pretraining phase. The loss function to be minimized is the mean squared error between the input and the output. Learning is online, as training examples are presented one by one. Subsequently, in the acquisition phase, weights are fixed and memories are stored in the associative memory. Two training regimes are tested. In the many-shot regime, pretraining lasts 20 epochs with 50 tasks. The same 50 tasks are then acquired in the memory and used to test performance. Contrastingly, in the one-shot regime, pretraining lasts a single epoch with 1000 tasks, such that networks are updated as many times as
in the many-shot regime. In the acquisition phase of the one-shot regime,
50 novel tasks are presented to the network and used to test performance.

While the rewarded pattern is provided as input to the network during
the pretraining and training phases, it is masked during test (i.e. values
of the third input slot are set to 0). In the test phases, a trial is considered
correct when the pattern stored in the third slot of the output is closer to
the rewarded pattern than the non-rewarded one, as assessed by the mean
squared error. Thus, the expected performance of a random network, or
a network with no contextual target, is 50% and serves as baseline. The
learning rate is a sensible parameter, as its optimal value varies among
the conditions tested in this work. Therefore, for each tested condition,
the learning rate is set as the average of the five learning rates that best
performed during a random search. This purely stochastic optimization
process evaluated the performance of the model in each condition, with 300
random learning rate values. After this process, 50 different simulations
were performed with the optimal learning rate of each condition.

7.2 RESULTS

7.2.1 BEHAVIORAL PERFORMANCE

The need for associative memory is first assessed to ensure that the task
requires episodic processing. In the many-shot task, the results with CA3
memory (M = 65.06, SD = 7.85) and without modern Hopfield network
(M = 70.64, SD = 4.05) indicate that memory impairs performance with
this training regime (t(49) = -4.97, p < .001), as shown in Figure 7.3b.
On the other hand, in the one-shot task, results with CA3 memory (M =
75.06, SD = 4.64) and without modern Hopfield network (M = 49.54, SD =
3.19) indicate that memory improves performance when test tasks are pre-
sented once during encoding (t(49) = 34.89, p < .001). In these conditions,
CA3 is directly targeted by context information. To test whether the model
can perform the task in accordance with the model of Howard Eichenbaum
(2017b), context information is then provided to entorhinal layers. The per-
formance with entorhinal target (M = 51.06, SD = 3.05) and CA3 target (M = 75.06, SD = 4.64) indicate that the model is much less performant when context information is given to the entorhinal cortex (t(49) = 28.2, p < .001), almost no better than chance (Figure 7.3c). It thus appears that context information does not reach the memory layer in absence of a direct connection. This constraint is however alleviated by placing the memory in other subfields, such as the dentate gyrus and CA1. When the dentate gyrus is the memory-augmented layer, the model almost performs equally well whether the dentate gyrus is directly targeted or contextual information transits through superficial (input) layers of the entorhinal cortex (Figure 7.3a). Similarly, when the associative memory is in CA1, the model performs well when context is provided to upstream layers (Figure 7.3e). When superficial layers of the entorhinal cortex are targeted and memory is in CA1, the performance of the model with the monosynaptic pathway (M = 70.14, SD = 4.71) and without it (M = 56.28, SD = 4.05) indicate that this pathway is necessary (t(49) = 17.55, p < .001), as shown in Figure 7.3f. By combining results of when the dentate gyrus, CA3 or CA1 (no monosynaptic pathway for simplicity) is memory-augmented, it is possible to evaluate performance according to the position of the targeted layer relative to the memory layer. This analysis reveals that targeting regions downstream memory has no effect (Figure 7.3d). Spearman's rank correlation was computed to assess the relationship between the relative position of the target (with positions less than or equal to 0) and performance. There was a positive correlation between the two variables, r(898) = 0.78, p < .001. Furthermore, a two-way ANOVA was performed to evaluate the interaction effects of relative position (less than or equal to 0) and context reconstruction on performance. The means and standard deviations for performance are presented in Table 7.3. The results indicated a significant main effect for relative position (F(3,899) = 545.59, p < .001), main effect for context reconstruction (F(1,899) = 24.29, p < .001), and interaction between relative position and context reconstruction (F(3,899) = 9.06, p < .001). Figure 7.3d indeed shows that the steepness of performance decline, as target is located more and more upstream memory, is greater when the model only
<table><caption>Table 7.3: Descriptive statistics for performance.</caption><thead><tr><th>Objective</th><th>Target position</th><th>M</th><th>SD</th></tr></thead><tbody><tr><td rowspan="4">Sensory</td><td>-3</td><td>50.54</td><td>3.33</td></tr><tr><td>-2</td><td>52</td><td>3.43</td></tr><tr><td>-1</td><td>60.69</td><td>6.93</td></tr><tr><td>0</td><td>71.91</td><td>5.27</td></tr><tr><td rowspan="4">Sensory + Context</td><td>-3</td><td>56.28</td><td>4.05</td></tr><tr><td>-2</td><td>53.91</td><td>4.53</td></tr><tr><td>-1</td><td>63.53</td><td>6.96</td></tr><tr><td>0</td><td>71.43</td><td>5.86</td></tr></tbody></table>

tries to reconstruct sensory input than when it also tries to reconstruct context information in the pretraining phase.

## 7.2.2 NEURAL CODING OF CONTEXT

In order to understand how the model either succeeds or fails at performing the task, the representations employed in the different layers can be analyzed. Principal component projections reveal that contexts are well separated at the targeted EC<sub>II</sub> population (Figure 7.4a), but this separation diminishes in subsequent layers DG (Figure 7.4b) and CA3 (Figure 7.4c), to such an extent that the associative memory in CA3 then struggles to separate memories contextually (Figure 7.4d). Downstream layers thus cannot know the context to answer correctly.

One possibility is that some neurons are specifically modulated by context identity and underlie the separation in the principal component space. In order to find such context cells, paired t-tests are used to assess whether the activity of each cell is dependent on the context (either stronger or weaker), treating the 50 test tasks as samples. Neurons with *p* < .001 are considered context cells. This analysis reveals that 68% of EC<sub>II</sub> neurons are context cells, and that this number decreases in the next populations (Figure 7.4).

The presence of context-modulated cells is very likely to help the model performing the tasks. In order to ensure that context cells indeed have
<figure><img src="image_49.png" /><figcaption>Figure 7.2: Example tasks in which the network succeeds (first two rows) and fails (last two rows) to recall the correct memory during the test phase. The two choice patterns that are placed in the first slots of the input are shown on the left. The rewarded patterns of contexts *c* and *c'* are shown in the middle. On the right, the choice of the network (i.e. third slot of the output) is shown for both contexts. In the first two tasks, the output of the network successfully corresponds to the rewarded pattern in both contexts. In the third task, however, the network chooses the same pattern in both contexts and thus fails to complete the task. In the fourth task, the network outputs the correct pattern in context *c* but outputs a pattern that is not in the choice set for that task in *c'*. However, the trial is still considered a success, as the output is closer to *R*<sub>*i*</sub><sup>*c*'</sup> than *R*<sub>*i*</sub><sup>*c*</sup>. For these examples, the network was pretrained on 10000 tasks with a learning rate of 10<sup>-4.5</sup>.</figcaption></figure>
<figure><img src="image_50.png" /></figure>

(a) Performance of the model when the associative memory is in DG (blue).

(b) Performance of training regimes with and without associative memory (target is CA3).

<figure><img src="image_51.png" /></figure>

(c) Performance of the model when the associative memory is in CA3 (blue).

(d) Performance as a function of the location of the target relative to the associative memory, using sensory only and full reconstruction losses. No monosynaptic pathway was used with CA1 memory. Relative positions P and -P indicate that the target and the memory are P layers apart. The target is located before the memory with -P, and after with P.

<figure><img src="image_52.png" /></figure>

(e) Performance of the model when the associative memory is in CA1 (blue).

(f) Performance of the model when the associative memory is in CA1 (blue) and there is no monosynaptic pathway. 123

**Figure 7.3:** Behavioral results. Gray dashed lines indicate the baseline performance of 50%.
7 Contextual Control Over Memory Recall

<figure><img src="image_53.png" /><figcaption>Figure 7.4: Principal component analysis of activity in different layers. Here, EC<sub>II</sub> is the target and CA3 is the memory.</figcaption></figure>

causal influence on behavioral performance and the contextual separation
of activity, a model with EC<sub>II</sub> as the target and DG as the associative mem-
ory was used. After the test phase, 1664 context cells were identified in DG
(46%), and were then inactivated during an additional test phase by clamp-
ing their activity to 0 in all 50 test tasks. With this protocol, the contexts
are no longer separated in the principal component space (Figure 7.5a), as
compared to when 1664 neurons are randomly selected for inactivation in-
stead (Figure 7.5b). Furthermore, while random inactivation yielded 72%
correct answers (similar to results of Figure 7.3a), performance dropped to
52% when context cells were inactivated.
<figure><img src="image_54.png" /><figcaption>Figure 7.5: Principal component analysis of DG with EC<sub>II</sub> as a target and DG as a memory, with context cell and random cell inactivation.</figcaption></figure>

7.3 DISCUSSION

A task requiring one-shot episodic memory was used to study context-
guided retrieval in a simple hippocampus-inspired network. The task was
introduced by Peters et al. (2013), used in the computational study of Pilly
et al. (2018), and is very similar to the Think/No-Think paradigm used to
study cognitive control over memory in humans (Anderson et al. 2016). The
behavioral results presented here reveal that modern Hopfield networks
can encode and retrieve contextual memories. The fast associative mem-
ory is necessary to perform the task when stimuli are only presented once.
Furthermore, it is found that the position of the target of contextual in-
formation is a key factor in correctly performing the task. The locus of
the associative memory is also important. In fact, performance seems to
be governed by the distance between these two elements, such that the
target must be close, but not downstream, the associative memory layer.
Thus, the model that follows the standard framework of hippocampal pro-
cessing and the model of prefrontal-hippocampal interactions proposed
by Howard Eichenbaum (2017b) does not perform well. This is due to the
associative memory in CA3 being too far from prefrontal projections to su-
perficial layers of the entorhinal cortex. While there is a direct connection
from the entorhinal cortex to CA3 in the anatomy, which could improve
performance, it is most often thought to be silenced during encoding in the
standard framework.

Some alternative models are much better at performing the task. This is
for example the case of models whose associative memory is located in a
layer directly targeted by the prefrontal context. Alternatively, models with
associative memory in DG and CA1 work well with the entorhinal cortex
as the target. This is due to DG and CA1 being connected to the entorhinal
cortex through the trisynaptic and monosynaptic pathways respectively.
Another factor influencing performance is whether the networks learn to
reconstruct context during the pretraining phase. Models that take con-
text into account in the reconstruction error perform better. This is not
surprising, as backpropagation then favors the transmission of context in-
formation across more layers to reconstruct it in the output layer. Since
the hippocampus has been hypothesized to learn to autoencode its input
(Santos-Pata et al. 2021; Ketz et al. 2013), and since it both receives input from
and projects back to the prefrontal cortex, it is reasonable to assume that
context reconstruction is a training objective of the hippocampus.

Beyond behavioral performance, the model has been found to encode
context explicitly, as suggested by the separation of contexts in the princi-
pal component space and the discovery of context-modulated cells. Most
importantly, these context cells have been found to be necessary to separate
contexts and perform the task correctly. Context information must indeed
be encoded in the input of the associative memory for it to encode and
recall memories contextually. Context neurons are reminiscent of splitter
cells discovered in the hippocampus (Wood et al. 2000; L. M. Frank et al. 2000;
Duvelle et al. 2023), which split place representation according to the past or
future trajectory. In a sense, the recent past and expected future constitute
valuable context information to the hippocampus. The context cells found
in this work can thus be considered splitter cells without explicit temporal
and spatial components.
# 8 ONE-SHOT SEQUENCE LEARNING

## 8.1 INTRODUCTION

In this chapter, the most important contribution of this thesis is introduced: the CONSEQUENCE model. With the renewal of interest in Hopfield networks and the advent of powerful modern variants, it is worth reevaluating the way the hippocampus stores memories. The standard framework of hippocampal memory processing postulates that memories are stored in recurrent collaterals of CA3, and this view was strongly influenced by Hopfield networks which were originally recurrent. The best associative memory models are now feedforward and implemented in two-layer feedforward networks that successfully compute the similarity between the query and stored memories, separate similarity scores to only recall specific memories, and project the resulting values back to the sensory space. Since these models are made of separate key and value matrices, they can store both autoassociations and heteroassociations, and thus learn sequences. The question then arises as to how the hippocampus could perform such operations.

## 8.2 REQUIREMENTS FOR HIPPOCAMPAL ASSOCIATIVE MEMORY

A biological neural network implementing a two-layer associative memory must satisfy certain requirements (Table 8.1). First, in order for memory re-
call to be specific as is the case for episodic memory, similarity scores must
be strongly separated. If memory coding is local, this means that the activ-
ity of the population in which memories are stored must be sparse during
retrieval. Furthermore, it must also have enough neurons to encode many
memories. The theoretical capacity of a population can be approximated
as the number of possible combinations of active cells. The dentate gyrus,
CA3 and CA1 contain 1,200,000, 250,000 and 390,000 neurons respectively,
in the rat hippocampus (Cutsuridis et al. 2019). Furthermore, approximately
0.8%, 3% and 9% of dentate gyrus, CA3 and CA1 cells are active at a time
(Neher et al. 2015). Hence, the dentate gyrus is likely to have a much big-
ger capacity than the other subfields, which makes it a good candidate for
storing memories. A mechanism to recruit neurons is also needed to store
novel information. Neurogenesis is a unique property of the dentate gyrus
and could enable the storage of new memories.

The population reconstructing memories, or in other words the one stor-
ing values, also needs to meet requirements. This output layer must be
targeted by the memory population and also contain the features of the
value to encode, for example with a residual connection from the sensory
input. It then needs to adjust synaptic weights to minimize the discrep-
ancy between the information reconstructed from memory and the actual
information to store. For sequence learning, heteroassociations needs to be
performed successively, thus the output either needs to be connected back
to the input, or directly to the memory neurons. CA3 is well suited for the
role of ouput layer, as it receives information from the dentate gyrus, and
the hilar backprojection connects it back to dentate granule cells.

The associative memory must recall information according to incoming
sensory inputs and contextual information. As shown previously, the den-
tate gyrus is directly targeted by the entorhinal cortex which receives top-
down prefrontal cues. Furthermore, CA3 is recurrently connected and can
thus accumulate information over time to contextually guide sequence re-
trieval and disambiguate overlapping trajectories. These elements converge
to a comprehensive model of how the hippocampus could implement a
two-layer associative memory that can store and retrieve sequences contex-
tually (Figure 8.1, upper left model). Sensory input is sent from $EC_{II}$ to DG, which compares it to stored memories and selects a sparse subset of relevant neurons. Memories are then decoded by $CA3$ and compared to the sensory input sent through the residual connection from the entorhinal cortex. This constitutes the autoassociative circuit. In the $CA3$-DG backprojection, associations can be learned between $CA3$ activity and the neurons of DG active in the following time step, as proposed by Lisman, Talamini, et al. (2005) in the SOCRATIC model. As $CA3$ accumulates information over multiple time steps, the heteroassociative keys stored in this pathway are conditioned on the recent past.

Alternative implementations are compared in Table 8.1 and shown in Figure 8.1. The $EC_{II}$-$CA3$-DG circuit could implement the associative memory similarly. However, $CA3$ is less sparse and contains less neurons than DG. Furthermore, there is no report of neurogenesis in $CA3$ and it is not clear whether DG could keep track of the temporal context. Finally, it has been shown in the preceding chapter that prefrontal information sent through the entorhinal cortex fails to reach $CA3$, at least without the direct $EC_{II}$-$CA3$ connection. Furthermore, it does not reach $CA1$ in absence of the monosynaptic pathway. The $EC_{II}$-$CA3$-$CA1$, DG-$CA3$-$CA1$, $CA3$-$CA1$-$EC_{V}$ and $EC_{III}$-$CA1$-$EC_{V}$ options all lack an intrahippocampal backprojection necessary for sequence learning. On top of that, $EC_{II}$-$CA3$-$CA1$, DG-$CA3$-$CA1$ and $EC_{III}$-$CA1$-$EC_{V}$ do not have a clear residual connection, as $EC_{III}$ stores information of a different kind than $EC_{II}$ and $EC_{V}$ (Hafting, Fyhn, Bonnevie, et al. 2008). It is a possibility, however, that the $CA3$-$CA1$-$EC_{V}$ circuit could learn to reconstruct the input from $EC_{II}$ in $EC_{V}$, since these two entorhinal layers represent similar information. Yet, this model could not learn sequences in absence of intrahippocampal backprojection.

## 8.3 COMPARISON OF SEQUENCE LEARNING MODELS

The $EC_{II}$-DG-$CA3$ circuit underlies the proposed CONSEQUENCE model (standing for contextual sequence). It exploits the unique characteristics of the
<table><caption>Table 8.1: Candidate circuits (Input-Memory-Output) for implementing episodic memory using a two-layer associative memory within the hippocampus. Capacity refers to the number of possible combinations of active neurons in the memory population. Neurogenesis and contextual modulation also refer to the memory population layer. The residual connection is a way to convey target information to the output layer. Backprojection from the output to the input or to the memory, and temporal context formation in the heteroassociative input, are both necessary for contextual sequence learning. Finally, contextual modulation refers to the ability for prefrontal information to bias retrieval from the memory. According to these criteria, the EC<sub>II</sub>-DG-CA3 circuit (bold) is the best candidate for implementing episodic memory with a modern associative memory in the hippocampus.</caption><thead><tr><th>Features</th><th>EC<sub>II</sub>-DG-CA3</th><th>EC<sub>II</sub>-CA3-DG</th><th>EC<sub>II</sub>-CA3-CA1</th><th>DG-CA3-CA1</th><th>CA3-CA1-EC<sub>V</sub></th><th>EC<sub>III</sub>-CA1-EC<sub>V</sub></th></tr></thead><tbody><tr><td>Capacity</td><td>+++</td><td>+</td><td>+</td><td>+</td><td>++</td><td>++</td></tr><tr><td>Neurogenesis</td><td>✓</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Residual Connection</td><td>+++</td><td>+++</td><td>+</td><td>+</td><td>++</td><td>+</td></tr><tr><td>Backprojection</td><td>✓</td><td>✓</td><td></td><td></td><td></td><td></td></tr><tr><td>Contextual Modulation</td><td>++</td><td>+(+)</td><td>+(+)</td><td>+</td><td>+</td><td>++</td></tr><tr><td>Temporal Context</td><td>✓</td><td>?</td><td></td><td></td><td></td><td></td></tr></tbody></table>
<figure><img src="image_55.png" /><figcaption>Figure 8.1: Candidate circuits (Input-Memory-Output) for implementing episodic memory using a two-layer associative memory within the hippocampus. Associative memory connections are shown in blue.</figcaption></figure>
dentate gyrus to implement a key-value associative memory which learns to associate the current state of the episode to its successor. Like in the SOCRATIC model, the CA3-to-DG backprojection learns heteroassociations while the EC<sub>II</sub>-to-DG connection can implement autoassociations. Inspired by the sparsity of granule cells, the activation function of DG is used to separate memories. The resulting DG activation is projected back to CA3 with a value matrix that learns to reproduce EC<sub>II</sub>-driven patterns of CA3 activity. By predicting successive successors, CONSEQUENCE can recall complete sequences of experienced states.

This new model will be compared to the standard framework of sequence learning in the hippocampus (Levy 1996), which will be called the Collateral model in reference to the role of CA3 recurrent connections. CONSEQUENCE will also be compared to the SOCRATIC model (Lisman, Talamini, et al. 2005). In order to encode sequences of complex information, these models are embedded within a deep learning network of the whole hippocampus which learns in an autoencoding fashion. As this network learns information slowly, like semantic memory in the complementary learning systems theory (Section 3.3.1), it is called the semantic pathway. It should be noted that this name serves as a conceptual shortcut rather than a formal technical term. Each model can be divided into an encoding and decoding semantic pathway, and a fast learning episodic pathway (Figure 8.2).

### 8.3.1 SEMANTIC PATHWAYS

The semantic pathway of all models is a deep learning neural network trained to encode and decode information for episodic storage and retrieval in the episodic pathway, with a learning rate of $2 \times 10^{-4}$. The number of neurons in each region is set as the number of neurons in the analog rat region, as retrieved from Cutsuridis et al. (2019), scaled down by a factor $N = 5 \times 10^{-4}$ (Table 7.1). The activation function is the softmax function in DG (for comparison with CONSEQUENCE; see Section 8.3.2) and ReLU in other layers. During training, information flows from Input to Output.
<figure><img src="image_56.png" /><figcaption>Figure 8.2: Candidate models of sequence learning. In all three models, the semantic pathway (green) develops by minimizing the discrepancy between Input($t$) and its reconstruction Output($t$) through backpropagation. The fast episodic pathway (blue) learns to recall sequences of CA3 activity. Left: architecture of the Collateral model. This episodic network consists of a single heteroassociative CA3-to-CA3 connection $H$, the target of which is the CA3 pattern of the next time step. The density of DG-to-CA3 connections is determined by probability $c_{in}$, as partial connectivity was proposed as a mechanism for history-dependent modulation of sequence recall in the Collateral model Levy (1996). Center: architecture of the SOCRATIC model. In this candidate, the CA3-to-CA3 connection $A$ is autoassociative and is used to complete the current pattern of CA3. Heteroassociations are performed by the CA3-to-DG backprojection $H$, the goal of which is to reconstruct the next DG pattern. The recalled pattern is then “cleaned up” using $A$ in order to avoid the concatenation of heteroassociation errors. Right: architecture of the CONSEQUENCE sequence learner. In addition to using the same heteroassociative mechanism as SOCRATIC, the episodic network of CONSEQUENCE employs autoassociative $EC_{II}$-to-DG ($A$) and DG-to-CA3 ($V$) projections. $V$ is analog to the value matrix of a key-value memory network, in that it is used to decode memory patterns, which can come from the environment through $EC_{II}$ (pattern completion) or from CA3 (sequence completion). Instead of flowing through DG, information in the semantic network goes from $EC_{II}$ directly to CA3 through matrix $T$. This skip connection is crucial for training the episodic network of CONSEQUENCE, as $V$ learns to mimick the output of connection $T$ which serves as a target. Since the CA3-to-CA3 recurrence is not necessary for simple sequence learning in the CONSEQUENCE model, multiple implementations are explored for enabling history-dependent sequence recall. In all models, the semantic network is used to reconstruct Output from recalled CA3 patterns.</figcaption></figure>
A reconstruction error is computed and used to update connections of the
network using the backpropagation algorithm (Rumelhart et al. 1986; Lin-
nainmaa 1976). The semantic pathways of the candidate models differ in
three ways. First, in the Collateral model, DG does not fully connect to
CA3. A Bernoulli sampling process is used such that each CA3 neuron
has probability *c*<sub>in</sub> to receive connections from DG, while the entries of
the DG-to-CA3 weight matrix corresponding to nonselected CA3 neurons
are zeroed at each forward propagation. This has been proposed to en-
able contextual modulation of sequence learning, as neurons not receiving
external input but connected to other CA3 neurons through the recurrent
connections encode contextual representations (Levy 1996). Secondly, the
CA3-to-CA3 recurrence of CONSEQUENCE is not part of the episodic path-
way like in other models. Memories in this model are instead encoded in
the dentate gyrus. This leaves room for other kinds of processing. In fact, it
will become clear later that recurrent collaterals of CA3 can be dedicated to
encoding a history of past states to disambiguate sequences with common
bouts like the model of Levy (1996). The last difference is the skip connec-
tion *T* of CONSEQUENCE. The dentate gyrus is indeed bypassed and only
used by the episodic network. This is done for three reasons. First, while
this connection is not always ascribed to a specific function in computa-
tional models of the hippocampus, it is present in the biology as reviewed
earlier. Secondly, it is expected that error gradients flow better from CA3
directly back to EC<sub>II</sub> than through DG if the activity of the latter is sparse
with strong lateral inhibition. While DG activity profile has been proposed
to subserve functions like pattern separation, it is not easily included in
deep learning architectures. For example, if the activation function is set to
a softmax with high steepness *β*, the window of nonzero derivative is small
as exemplified in Figure 8.3, which can lead to a problem analog to vanish-
ing gradient (Bengio et al. 1994). Using a softmax as the activation function
for DG therefore makes training upstream layers harder. By skipping DG
in the semantic pathway, we can allow it to be very sparse for efficient
episodic processing without affecting backpropagation training. Last but
<figure><img src="image_57.png" /><figcaption>Figure 8.3: Gradient of the sigmoid function vanishes with steepness $\beta$. Area where gradient falls under $\epsilon = 1e-2$ is highlighted for each $\beta$ value.</figcaption></figure>

not least, the skip connection $T$ acts as a target for teaching the DG-to-CA3 projection of the episodic pathway, as detailed below.

### 8.3.2 FAST EPISODIC PATHWAY OF CONSEQUENCE

In the CONSEQUENCE model, DG implements a key-value memory network with two sets of keys (autoassociative keys $A$ and heteroassociative keys $H$) and a set of values $V$ which are learned using the biologically plausible rules proposed by Tyulmankov et al. (2021). In order to write keys, a local third factor is sequentially active for each postsynaptic DG neuron $j$:

$$ \gamma_j(t) = \begin{cases} 1 & \text{if } t = j \mod N_{\text{DG}} \\ 0 & \text{otherwise} \end{cases} \qquad (8.1) $$

where $N_{\text{DG}}$ is the number of DG neurons. This factor is combined to a scalar global learning rate $\eta_e$ for determining the learning rate of each synapse:

$$ \eta_{ij}^{A} = \eta_{ij}^{H} = \eta_{e} \gamma_{j}, \qquad (8.2) $$
which is then used to update A and H weights:

$$A(t+1) = (1 - \eta^A) \odot A(t) + \eta^A \odot (\text{EC}_{\text{II}}(t)\mathbf{1}) \quad (8.3)$$

$$H(t+1) = (1 - \eta^H) \odot H(t) + \eta^H \odot (\text{CA3}(t-1)\mathbf{1}) \quad (8.4)$$

where $\mathbf{1} = (1, 1, ..., 1)$. Note that vectors are regularized with the L2 norm before encoding and retrieving them to ensure there is no bias in favor of certain memories. This is done for the other candidate models too. After storing the normalized keys, the new weights are used to compute the state of DG:

$$\text{DG}^{\text{in}}(t) = \begin{cases} A^{\top}(t+1)\text{EC}_{\text{II}}(t) & \text{for encoding and pattern recall} \\ H^{\top}(t+1)\text{CA3}(t-1) & \text{for sequence recall} \end{cases} \quad (8.5)$$

$$\text{DG}(t) = \text{softmax}(\beta_{\text{DG}}\text{DG}^{\text{in}}(t)) \quad (8.6)$$

Just as in some modern Hopfield networks (Ramsauer et al. 2020), $\beta_{\text{DG}}$ is the inverse temperature controlling the amount of memory blending. As $\beta_{\text{DG}}$ approaches infinity, DG activity sparsifies since the softmax gets closer and closer to a max function giving a result resembling a one-hot vector (i.e. with one entry being one and the others being zero), given that all input entries are different. The predicted pattern is then the selected column of $V$. Conversely with $\beta_{\text{DG}} = 0$, the output of the softmax is a uniform distribution that results in a prediction that is an average of all values. During encoding, the state of DG is then used to update the value matrix:

$$V(t+1) = (1 - \eta_e \text{DG}(t)\mathbf{1}) \odot A(t) + \eta_e \text{DG}(T^{\top} \text{EC}_{\text{II}}(t))^{\top} \quad (8.7)$$

where $T^{\top} \text{EC}_{\text{II}}(t)$ is the input from $\text{EC}_{\text{II}}$ to CA3 which serves as a target matrix for $V$. In other words, $V$ learns to replace the input from the environment for recalling CA3 states.
Depending on the processing phase, the input to CA3 either comes from the entorhinal cortex or DG recall:

$$ CA3^{in}(t) = \begin{cases} V^{\top}DG(t) & \text{during recall} \\ T^{\top}EC_{II}(t) & \text{during encoding.} \end{cases} \qquad (8.9) $$

The state of CA3 is computed as:

$$ CA3(t) = \mathcal{R}(CA3^{in}(t)) \qquad (8.10) $$

where $\mathcal{R}$ is the recurrence function of CA3 which will be described in Section 8.3.5.

Once the network has been trained and memories have been stored, Equations 8.6 and 8.10 can be chained to recall a static pattern (using entorhinal input to DG) or a sequence transition (using the CA3-to-DG backprojection). Full sequences can be retrieved by computing Equation 8.10 multiple times. The reinstantiated CA3 states can be decoded using the semantic pathway in order to recover the sequence of incoming sensory (or cortical) stimuli. Now that the CONSEQUENCE model has been described, the episodic pathway of the other candidate models will be outlined.

### 8.3.3 FAST EPISODIC PATHWAY OF THE COLLATERAL MODEL

The first candidate to which CONSEQUENCE is compared is a model inspired by the standard framework. In the standard framework, memories are implemented in CA3 recurrent collaterals. While this connection is often proposed to encode autoassociations, this does not allow for the storage of sequences. Here, the CA3-to-CA3 connection encodes heteroassociations like in the model of Levy (1996): it learns to reconstruct $CA3(t + 1)$ from $CA3(t)$. To this end, heteroassociative weights are updated with a postsynaptic rule (Levy 1982):

$$ H_{ij}(t) = H_{ij}(t-1) + \eta_e CA3_j(t) [CA3_i(t-1) - H_{ij}(t-1)] \qquad (8.11) $$
where $H_{ij}$ is the synaptic weight from neuron $i$ to $j$ and $\eta_e$ is the rate of episodic learning. The activity of CA3 is:

$$ \mathrm{CA3}(t) = \mathrm{ReLU} \left( \left[ (\mathbf{M}_{in} \odot \mathbf{W})^{\top} \mathrm{DG}(t) + (\mathbf{M}_{r} \odot \mathbf{H})^{\top} \mathrm{CA3}(t-1) \right] \right) \quad (8.12) $$

where $M_{in}$ and $M_r$ are connection masks, $\odot$ is the Hadamard (element-wise) product and $W$ is the DG-to-CA3 weight matrix. While $M_{in}$ is the result of the neuron-wise sampling procedure described in section 8.3.1, the recurrent mask $M_r$ is constructed with a synaptic-wise process. Each entry has probability $c_r$ of being 1 and is set to 0 otherwise. Note that contrary to other models, the episodic pathway contributes to information processing even in the pretraining phase in which the semantic pathway develops. Weights $H$ are thus initialized with nonzero values and left unchanged until memories are encoded in the model. This is necessary to train the semantic network to cope with CA3 neurons that do not receive external inputs (i.e. $\mathrm{col}_j(M_{in}) = \vec{0}$) and would otherwise only become active once the semantic pathway is trained and memories are encoded.

### 8.3.4 FAST EPISODIC PATHWAY OF SOCRATIC

The SOCRATIC model employs two associative connections of different kinds. The CA3-to-DG backprojection learns to predict $\mathrm{DG}(t+1)$ from $\mathrm{CA3}(t)$ using Equation 8.4 of CONSEQUENCE with a learning rate $\eta_{ij}^H = \eta_{e1}\gamma_j$ (to distinguish $\eta_{e1}$ from the CA3-to-CA3 rate $\eta_{e2}$ introduced below).

The second associative connection is the CA3-to-CA3 recurrence which is used to encode autoassociations and cleanup the result of the heteroassociation in order to prevent the concatenation of errors. It is updated similarly to the connection of the Collateral model:

$$ A_{ij}(t) = A_{ij}(t-1) + \eta_{e2} \mathrm{CA3}_j(t) [\mathrm{CA3}_i(t) - A_{ij}(t)] \quad (8.13) $$

with a second episodic learning rate $\eta_{e2}$.

Unlike the Collateral model, the episodic network of SOCRATIC needs not be active during the training of the semantic pathway. Thus, the activity of
DG is governed by external inputs during training and for the recall of a single autoassociation, and driven by CA3 during heteroassociative recall:

$$ \mathrm{DG}^{in}(t) = \begin{cases} \mathbf{W}_{\mathrm{EC}_{\mathrm{II}} \to \mathrm{DG}}^{\top} \mathrm{EC}_{\mathrm{II}}(t) & \text{for encoding and pattern recall} \\ \mathbf{H}^{\top} \mathrm{CA3}(t-1) & \text{for sequence recall} \end{cases} \qquad (8.14) $$

$$ \mathrm{DG}(t) = \mathrm{softmax}(\beta_{\mathrm{DG}} \mathrm{DG}^{in}) \qquad (8.15) $$

Similarly, the autoassociative recurrence is only used during recall:

$$ \mathrm{CA3}^{in}(t) = \mathbf{W}_{\mathrm{DG} \to \mathrm{CA3}}^{\top} \mathrm{DG}(t) \qquad (8.16) $$

$$ \mathrm{CA3}(t) = \begin{cases} \mathrm{ReLU}(\mathrm{CA3}^{in}(t)) & \text{for encoding} \\ \mathrm{ReLU}(\mathbf{A}^{\top} \mathrm{CA3}^{in}(t) + \mathrm{CA3}^{in}(t)) & \text{for recall} \end{cases} \qquad (8.17) $$

(8.18)

### 8.3.5 TEMPORAL CONTEXT FORMATION IN CONSEQUENCE

With discrete sequence learning as a primary function, the hippocampus must learn how the input states follow one another. A naive approach would be to learn a set of associations between current states and subsequent states. However, this approach fails to account for the learning of temporal dependencies. Therefore, the naive approach is incapable of disambiguating sequences that share the same states.

A common solution to this problem is to use a representation of the temporal context as additional information to make a prediction about the next state. In neural networks, this is typically implemented with recurrent connections that convey information over time (Elman 1990). With its recurrent collaterals, CA3 is well suited to play this role, and the recurrence function $\mathcal{R}$ of Equation 8.10 can be:

$$ \mathcal{R}(\mathbf{x}_t) = \tanh(\mathbf{x}_t + \mathbf{R}^{\top} \mathrm{CA3}_{t-1} + \mathbf{b}) \qquad (8.19) $$
where **R** are recurrent weights and **b** is the bias vector. A simpler alternative is to model CA3 as a leaky feedforward layer:

$$ \mathcal{R}(\mathbf{x}_t) = \alpha \mathrm{ReLU}(\mathbf{x}_t + \mathbf{b}) + (1 - \alpha) \mathrm{CA3}_{t-1} \quad (8.20) $$

where $\alpha \in [0, 1]$ is the leak rate. Both leaky and recurrent weights versions of CA3 are initialized without recurrence in the first step of each sequence. In principle, these two recurrent functions could enable the model to dis-ambiguate overlapping sequences by taking past information into account for episodic storage and retrieval.

## 8.4 METHODS

### 8.4.1 DATASETS

Two datasets are used throughout this study. The binary dataset consists of sequences of random binary patterns. Each bit is either 1 with probability .35 and 0 with probability .65, as motivated by average activity levels of the entorhinal cortex reported by Neher et al. (2015). When using this dataset, the binary cross-entropy is used to train the semantic pathway.

The spatial dataset consists of sequences of spatial positions represented by 30 cells with grid activity profiles. The sequences and grid cell responses are generated with the RatInABox Python library (George, Cothi, Clopath, et al. 2022). The firing of a grid cell is modelled as:

$$ F_i(t) = \frac{1}{3} \max \left( 0, \sum_{j=0}^{2} \cos \left( 2\pi \frac{\mathbf{x}(t) \cdot \mathbf{e}_{\theta_i + j\pi/3}}{\lambda_i} + \phi_i \right) \right) \quad (8.21) $$

where $\phi_i \in [0, 2\pi]$ and $\theta_i \in [0, 2\pi]$ are the phase and orientation respectively, $\lambda_i$ is the scale, $\mathbf{x}(t)$ is the current position and $\mathbf{e}_{\theta_i}$ is the unit vector pointing in the direction $\theta_i$. Phases, orientations and scales are sampled uniformly. Example grid fields and trajectories generated with RatInABox
<figure><img src="image_58.png" /><figcaption>Figure 8.4: Examples of biologically realistic data generated with RatInABox (George, Cothi, Clopath, et al. 2022).</figcaption></figure>

are shown in Figure 8.4. The mean square error loss is used to train the semantic pathway with this dataset.

## 8.4.2 HYPERPARAMETER OPTIMIZATION

The models have some hyperparameters that need to be tuned for fair comparison. These include $\beta_{DG}$, $\eta_e$ and $\alpha$ for CONSEQUENCE, $\beta_{DG}$, $\eta_{e1}$ and $\eta_{e2}$ for SOCRATIC, and $\beta_{DG}$, $c_{in}$, $c_r$ and $\eta_e$ for the Collateral model. The parameters are thus optimized using Gaussian processes with the scikit-optimize library (Head et al. 2021). First, 100 parameter sets are generated with a Halton sequence (Halton 1960), then tested. After this initial phase, 100 additional evaluations are made guided by the Bayesian surrogate model. Each evaluation is made with 3 different runs.

# 8.5 EXPERIMENTS

## 8.5.1 ONE-SHOT SEQUENCE LEARNING

Here the ability of the candidate models to learn sequences in one-shot is tested. In the pretraining phase, the semantic pathways are trained to auto-encode 1000 independent spatial positions or binary patterns during
50 epochs. Grid scales are uniformly sampled between 0.1 and 0.5. The spatial environment is an empty 1 × 1 meter box. Then, 25 sequences of 20 steps are presented once for encoding in the episodic pathways. Spatial sequences are initialized and generated randomly. After this one-shot exposure phase, the ability of the models to recall the sequences is tested. In order to do that, the first half of each sequence is given as a cue, during which the episodic pathway is silenced. Then, the episodic pathway is used to recall the second half of the sequence in absence of additional external input. Performance is calculated as the R² score of the regression between the recalled and actual second half of the sequence. Similarly, the hyperparameter optimization process seeks to minimize the reconstruction loss of the second half of the sequences. Note that performance and loss in the spatial dataset are computed in the grid cell space. Here, the simple leaky recurrent function was used in CONSEQUENCE.

The results of the hyperparameter optimization procedures are shown in Figures 8.5, 8.6, 8.7, 8.8, 8.9 and 8.10. For CONSEQUENCE, the most important parameter is βDG in that it influences performance the most (Figures 8.5 and 8.6). It must be large to properly separate memories during retrieval. The optimization of ηe reveals a slightly more complex pattern. At first sight, it seems that the model works best with slower episodic learning rates, which is at odds with the one-shot learning regime. However, when βDG is large, the best performance is obtained with fast learning rates. Furthermore, CONSEQUENCE performs best with leak rate α = 1. When it comes to the SOCRATIC model, ηe2 (learning rate of CA3 recurrent collaterals) and βDG are the most important, even though the model did not perform well overall (Figures 8.7 and 8.8). Finally, the Collateral model was most influenced by the density of recurrent weights cr, but did not perform as well as CONSEQUENCE either (Figures 8.9 and 8.10). Performance of the three optimized models is shown in Figure 8.11. CONSEQUENCE performs best in both spatial and binary datasets. In comparison, the performance of the collateral and SOCRATIC models did not reach baseline despite hyperparameter optimization. In order to test the robustness of CONSEQUENCE, Gaussian noise is added to the recall cues. Since the sequences have values
between 0 and 1, a Gaussian noise of $\xi = 1$ is very strong. The model
nonetheless performs well (Figure 8.12).

8.5.2 PLACE CELLS AND REMAPPING

Now that CONSEQUENCE has been shown to perform well, it is worth in-
vestigating how biologically plausible the representations it employs are.
When spatial sequences are encoded in the model, the response profile of
recruited dentate gyrus cells resembles that of hippocampal place cells (Fig-
ure 8.13a; left). When placed in a novel environment (an adjacent box of
equal shape) in which no sequences were stored, the place field of dentate
gyrus cells remaps (Figure 8.13a; right). It has been shown that remap-
ping in the biological hippocampus reflects attractor dynamics, such that
continuously morphing between a circle and a square environment results
in abrupt and simultaneous changes in the response profile of place cells
(Wills et al. 2005; Figure 8.14). In order to test whether this can occur in the
dentate gyrus of CONSEQUENCE, it is assumed that the shape of the envi-
ronment is (at least partially) represented in external regions and provided
as input to the hippocampus. In the model, this is done by providing se-
mantic information through 30 lateral entorhinal cortex cells, in addition
to the encoding of space of the 30 grid cells. Semantic vectors A and B
are created independently, such that Aᵢ, Bᵢ ∼ U(-1, 1), with i = 1, 2, ..., 30.
Prior knowledge of these vectors in the first environment is encoded by
associating either A or B to each pretraining position, and to each posi-
tion of the sequences learned. Then, the response profile of dentate gyrus
is constructed when morphing between vectors A and B in the novel ad-
jacent environment (Figure 8.13b). Neurons of the CONSEQUENCE dentate
gyrus remap abruptly and synchronously as a result of morphing. Simi-
larity analyses are also reminiscent of attractor dynamics observed by Wills
et al. (2005), as shown in Figure 8.15.
<figure><img src="image_59.png" /><figcaption>Figure 8.5: Results of hyperparameter optimization of the CONSEQUENCE model with the spatial dataset. Partial dependence plots show how individual hyperparameters influence the loss function by marginalizing over the values of all other parameters. Similarly, contour plots show interactions of parameter pairs by marginalizing over the remaining variables.</figcaption></figure>
<figure><img src="image_60.png" /><figcaption>Figure 8.6: Results of hyperparameter optimization of the CONSEQUENCE model with the binary dataset. See Figure 8.5 for details regarding the plots.</figcaption></figure>
<figure><img src="image_61.png" /><figcaption>Figure 8.7: Results of hyperparameter optimization of the SOCRATIC model with the spatial dataset. See Figure 8.5 for details regarding the plots.</figcaption></figure>
<figure><img src="image_62.png" /><figcaption>Figure 8.8: Results of hyperparameter optimization of the SOCRATIC model with the binary dataset. See Figure 8.5 for details regarding the plots.</figcaption></figure>
<figure><img src="image_63.png" /><figcaption>Figure 8.9: Results of hyperparameter optimization of the Collateral model with the spatial dataset. See Figure 8.5 for details regarding the plots.</figcaption></figure>
<figure><img src="image_64.png" /><figcaption>Figure 8.10: Results of hyperparameter optimization of the Collateral model with the binary dataset. See Figure 8.5 for details regarding the plots.</figcaption></figure>
<figure><img src="image_65.png" /><figcaption>Figure 8.11: Performance of optimized candidate models. Performance of a model that would always output the average of the dataset is shown as baseline.</figcaption></figure>

<figure><img src="image_66.png" /><figcaption>Figure 8.12: Performance of CONSEQUENCE with various levels of cue noise. Gaussian noise of standard deviation $\xi$ is added to the input during recall.</figcaption></figure>
A to B morphing

<figure><img src="image_67.png" /></figure>

Figure 8.13: Place response profiles of dentate gyrus cells of the CONSEQUENCE model.
<figure><img src="image_68.png" /><figcaption>Figure 8.14: Abrupt and coherent expression of square or circle representation during probe trials in intermediate octagonal environments in Rat 4. The 17 of 20 place cells simultaneously recorded from rat 4 with different (remapped) firing patterns in the square and the circle almost all switch from the square-like to circle-like pattern between the 2:6 and 3:5 octagons. 8 cells had fields in the circle but not the square (cells 1-8); 4 in the square but not the circle (9-12); 5 fired in both but in different places (13-17), and 3 did not reach criterion for remapping (18-20). Taken from Wills et al. (2005).</figcaption></figure>
<figure><img src="image_69.png" /></figure>

(a) Coordinated shift in square-to-circle switch point between the first and second octagon series. The plot shows the similarity of place cells' firing patterns in probe trials of varying shape to their firing patterns in square (red) or circle (blue) baseline trials (mean and standard error across cells). Taken from Wills et al. (2005).

(b) Similarity between dentate place fields of CONSEQUENCE during morphing and baseline trial of A (red) or B (blue). Similarity is computed as the negative mean squared error. Plotted are the mean and standard error across cells.

Figure 8.15: Analysis of place fields during morphing in the biological and simulated hippocampus.
8.5.3 DISAMBIGUATION OF OVERLAPPING SEQUENCES

The sequences encoded so far were not generated to overlap, so there is no
need to keep track of past states such that a leak rate of $\alpha = 1$ was found
to work best (Figures 8.5 and 8.6). However, rodents can also perform
alternation tasks in which the hippocampus encodes the recent past, as
suggested by the presence of retrospective splitter cells (Wood et al. 2000;
L. M. Frank et al. 2000), to predict the location of future rewards. In order
to model this process with CONSEQUENCE, trajectories of 10 positions are
generated in an eight-shaped maze. 50 trajectories start at the bottom left
and end at the top right, and 50 trajectories start at the bottom right and end
at the top left (Figure 8.16; left). The number of DG neurons is multiplied
by 1.7 to ensure that the episodic pathway has sufficient capacity to encode
these sequences. Importantly, all trajectories pass through a common arm
and meet in the center before separating. This is done to prevent the model
from predicting future positions based on the current state. For accurate
prediction of the second half of the sequences, the model must keep a trace
of where it comes from.

The model is pretrained with positions spanning the environment as
described before. Grid scales are uniformly sampled between 0.5 and 1.
A Gaussian process regression model with radial basis function kernel is
trained to decode the positions retrieved by the model<fnref n="1" />. Example decoded
sequences are shown in Figure 8.16 (right).

Figure 8.17 shows the performance of the model in this task. With $\alpha = 0.5$, the model ends its recall in the correct arm in 95% and 72% of trials with a cue noise of $\xi = 0.1$ and $\xi = 0.3$ respectively. Performance approaches 50% with higher $\alpha$ as the model loses track of where it comes from, and approaches 0% with $\alpha < 0.2$ because recall does not reach any upper arm if CA3 is updating too slowly. Performance is reflected in the separation of sequences in principal component space (Figure 8.18). While the center is encoded differently according to the starting arm with $\alpha = 0.5$

<footnote n="1">¹This process is detailed in https://github.com/RatInABox-Lab/RatInABox/blob/main/demos/decoding_position_example.ipynb</footnote>
<figure><img src="image_70.png" /><figcaption>Figure 8.16: 10 example actual (left) and decoded (right) trajectories of 10 positions generated in the eight maze with RatInABox (George, Cothi, Clopath, et al. 2022). Blue trajectories start in the lower left arm and end in the upper right arm. Orange trajectories start in the lower right arm and end in the upper left arm. All trajectories meet at the center before diverging.</figcaption></figure>

(Figure 8.18d), it is not separated with $\alpha = 1$ (Figure 8.18b). In comparison, the average performance of the model with recurrent CA3 weights (Equation 8.19) is 82% ± 19 and 59% ± 13 with $\zeta = 0.1$ and $\zeta = 0.3$ respectively. The state of the recurrent neural network is initialized without the recurrent connection during the first step of each sequence. Therefore, the recurrent weights are not used nor updated when training the semantic pathway with individual positions (sequences of length 1). To test whether the autoencoding objective function can indeed recruit recurrent weights, an additional pretraining phase is performed. A long random sequence of 2509 steps is generated and transformed into 2500 sequences of 10 steps using a sliding window mimicking theta phase precession. A horizontally flipped duplicate of these sequences is created to avoid any bias towards right or left arms. During that second pretraining phase, the recurrent weights of CA3 are trained to autoencode spatial positions of the 5000 sequences before the model encodes the 100 test sequences. 50 runs are performed and compared to 50 control runs lacking the second pretraining phase. With $\zeta = 0.1$, the outcomes from the second pretraining test (M = 0.81, SD = 0.03) and the control test (M = 0.85, SD = 0.14) indicate that pretraining CA3 with sequences does not significantly change the average performance ($t(49) = 1.93, p = .06$). The Brown-Forsythe test indicates that second pretraining induces changes in variance (statistic = 15.55; $p < .001$).
Similarly, using $\xi = 0.3$, the results of the second pretraining test (M = 0.59, SD = 0.05) and the control test (M = 0.58, SD = 0.11) suggest that pretraining CA3 does not significantly alter the mean performance ($t(49) = -0.64, p = .53$), while changing variance (statistic = 19.78; $p < .001$). An additional experiment is carried out to ensure that the recurrent weights can be learned and used to perform the task rather than just decreasing the variance in performance. The initial recurrent weights are divided by 50 to penalize the model with weights left random. In this case, with $\xi = 0.1$, the results obtained with random (M = 0.52, SD = 0.06) and trained weights (M = 0.6, SD = 0.04) indicate that training recurrent weights is indeed beneficial to performance ($t(49) = -7.57, p < 0.001$). With $\xi = 0.3$, the results of random (M = 0.49, SD = 0.06) and trained weights (M = 0.52, SD = 0.05) lead to the same conclusion ($t(49) = -2.75, p < 0.01$).

So far, eight-maze results have proven that the model can store and recall many experienced overlapping trajectories. In order to show that the model can also perform the spatial alternation task in one-shot, a new experiment is conducted in which a single sequence for each starting arm is encoded. CONSEQUENCE is used with recurrent weights (initial weights left unchanged) and the second pretraining phase introduced in the previous paragraph. The 98 remaining sequences cue the network during the test phase. 69% of these sequences end in the correct arm despite not being encoded. With leaky CA3 ($\alpha = 0.5$), 92% of the unexperienced sequences end correctly.

## 8.6 DISCUSSION

In this chapter, the possibility that the hippocampus implements a two-layer associative memory was explored. While it has been proposed several times that the hippocampus could implement one (Krotov and J. Hopfield 2020; Chandra et al. 2023; Spens and Burgess 2023), the precise anatomical mapping had not been discovered. After reviewing all possible circuits, a new model was proposed in which the dentate gyrus plays a central role.
<figure><img src="image_71.png" /><figcaption>Figure 8.17: Performance of CONSEQUENCE with leaky CA3 in the eight maze task. Gaussian noise of standard deviation ξ is added to the input during recall. A trial is correct if the final decoded position is in the upper right arm for trajectories starting on the left and in the upper left arm for trajectories starting on the right. The mean and standard deviation of 5 runs are shown.</figcaption></figure>

With its large number of neurons and unique properties like sparse activ-
ity and continuous neurogenesis, this region is well suited to store properly
separated memories. Furthermore, it is ideally situated with projections to
and from CA3, and direct afferent connections from the entorhinal cortex.
In this new model, CA3 also plays a central role, as it teaches the dentate
gyrus to reconstruct memories with a distinct projections from entorhinal
neurons. It also sends information back to the dentate gyrus, closing an
ideal loop for sequence learning. Finally, recurrent collaterals can carry
information over time to disambiguate overlapping sequences.

The proposed model learns sequences perfectly. Implementations of the
models of Levy (1996) and Lisman, Talamini, et al. (2005) were proposed that
matched the implementation of CONSEQUENCE as closely as possible, while
respecting the details of the original publications. The new model out-
performed its predecessors in one-shot learning of spatial trajectories and
random sequences. The SOCRATIC model has inspired the development of
CONSEQUENCE in its use of the CA3-to-DG backprojection. However, no
implementation of SOCRATIC has previously been published and it failed to
learn the tasks in the present work. This is likely due to a tradeoff between
a high beta allowing the separation of memories and a lower beta allowing
<figure><img src="image_72.png" /><figcaption>Figure 8.18: Principal component analyses of CA3 states in the eight maze task ($\zeta$ = 0.1). The analyses are made on 100 trajectories (50 for each starting arm) and 20 trajectories are shown.</figcaption></figure>
the error gradients of the semantic pathway training to go up the afferent connections to the dentate gyrus. This limitation is overcome by the connection of the entorhinal cortex to CA3 in the CONSEQUENCE model. The Collateral model of Levy (1996) did not perform well either. While it could learn simple sequences in original publications, the general idea of storing sequences has been shown to be suboptimal (Melchior et al. 2019). As they cannot implement a separation function (Millidge et al. 2022), recurrent connections are not sufficient to learn information in one-shot. The more recent CRISP model of sequence learning in the hippocampus (Cheng 2013) could also be compared to the proposed model, as it has been shown to outperform the Collateral model (Melchior et al. 2019). However, its implementation would require training CA3 separately and would not fit straightforwardly into the semantic pathway.

CONSEQUENCE has also been shown to be able to recall sequential memories from noisy cues. Furthermore, biological properties emerged from the storage of spatial trajectories. Activation of granular cells in which memories have been stored resembles the receptive fields of hippocampal place cells, analogous to the model of Benna and Fusi (2021). More strikingly, the cells remap in new unexperienced environments. This remapping has been shown to follow similar attractor dynamics to hippocampal place cells (Wills et al. 2005). Place cells have not yet been found in the semantic pathway, but since it is very similar to the autoencoder model of Santos-Pata et al. (2021), it is expected that they emerge from hyperparameter tuning or small adjustments. While the semantic pathway is not the main focus of the present work, this topic is under active development. Finally, the model successfully learned the eight maze task in which similar sequence memories must be disambiguated according to the recent past. Retrospective splitting representations (Duvelle et al. 2023) emerged, as suggested by the differential encoding of merging sequences. The model performs well with random recurrent CA3 connections too, which is consistent with the success of reservoir computing (Schrauwen et al. 2007), but the performance is more variable over runs than with leaky CA3. Pretraining recurrent weights with phase precessing sequences reduces variance without
altering average performance when initial weights are left unchanged, and
improves performance when the magnitude of initial weights is reduced.
With contextual modulation similar to that investigated in the previous
chapter, prospective splitter cells could probably be observed too. The pre-
frontal cortex could indeed contextually bias the hippocampus to recall
trajectories leading to a certain state, which should be reflected in the rep-
resentation of space. The CONSEQUENCE model has also been shown to be
capable of efficiently using its experience. The model can indeed perform
the spatial alternation task of the eight-maze in one-shot, after experiencing
one sequence starting in each lower arm only.

The operations of the CONSEQUENCE model are biologically plausible.
The semantic pathway is trained with backpropagation, and it has been ar-
gued that the hippocampus is capable of implementing such error-driven
learning (Santos-Pata et al. 2021; Y. Zheng et al. 2022; Ketz et al. 2013). The simi-
larity function is the dot product which is simply implemented by synaptic
transmission in artificial neural networks. The softmax function is very
efficient to separate memories, and it has recently been shown that it can
be implemented in biological neural networks too (Snow and Orchard 2022).
This implementation involves interneurons, of which the biological dentate
gyrus is populated with. It remains to be determined whether interneuron
computation in the dentate gyrus is indeed linked to softmax calculation,
as suggested by the steep sigmoidal response of dentate gyrus neurons
to entorhinal inputs (Renata Bartesaghi et al. 1995), or whether the dentate
gyrus implements a slightly different separation function through its spars-
ity. The episodic pathway of CONSEQUENCE is trained using a learning
rule that was recently proposed as a biologically plausible way of encod-
ing memories in modern associative memory networks (Tyulmankov et al.
2021). The authors showed that storing keys with random local third factor
is more plausible but less efficient than sequentially activating the factor
of each neuron as done here (Equation 8.1). With random selection of
neurons for memory encoding, stored information can be arbitrarily over-
written. The continuous birth of highly plastic granule cells in the dentate
gyrus seems to reflect a biological implementation of the sequential factor,
<figure><img src="image_73.png" /><figcaption>Figure 8.19: Performance of CONSEQUENCE as a function of the number of neurons in DG in the spatial dataset. The number of successfully stored sequences scales linearly with the size of DG.</figcaption></figure>

in which newborn neurons are recruited to store new information and then become less plastic to preserve the memory. The current capacity of CONSEQUENCE is linear in the number of granule cells, as each neuron stores one autoassociation and one heteroassociation (Figure 8.19). In the biological hippocampus, dentate gyrus activity is sparse, but involves more than a single neuron. The dentate gyrus neurons in the model therefore represent populations of neurons, but these populations must be independent to avoid confusing memories.

It has been suggested that the dentate gyrus is necessary to recall recent memories before information gets consolidated in CA3 collaterals (Hainmueller and Bartos 2020). It is also a prediction of the CONSEQUENCE model that the (initial) storage of one-shot experience is dependent on the integrity of granule cells. Once the unique experience has been captured by the dentate gyrus, replays could be generated to encode the information in CA3. CA3 recurrences could thus be trained in many-shot fashion, with a learning rule that also enables CA3 to keep track of the temporal context. Hence, the CONSEQUENCE model is compatible with the view that there are complementary learning systems within the hippocampus but, unlike the model of Schapiro, Turk-Browne, M. M. Botvinick, et al. (2017), CA3 is a sta-
tistical learner in the new model. There is mixed evidence regarding the
role of neurogenesis in learning, due to the diversity of tasks and methods.
Another prediction of CONSEQUENCE is that the birth of new granule cells
is necessary for real one-shot learning in the hippocampus, even though
fast learning can potentially occur in CA3. This is compatible with reports
that neurogenesis in the dentate gyrus is increased by novelty (J. Brown et al.
2003) and necessary for learning temporal associations (Shors et al. 2001). As
the dentate gyrus

Overall, the CONSEQUENCE model constitutes a powerful alternative to
the standard framework postulating that episodic learning mainly involves
CA3 recurrent collaterals, a proposal that lacks empirical evidence (Cheng
2013) and is computationally suboptimal (Millidge et al. 2022; Melchior et al.
2019).
# 9 GENERAL DISCUSSION

This thesis began with three seemingly simple questions revolving around
Marcel Proust’s famous novel excerpt:

* How can the simple flavor of a madeleine transport us through time and space, allowing us to vividly revisit a specific moment from our past?

* What makes this particular event stand out amidst the myriad of others?

* In what way is this memory intended to guide behavior?

The review of the state of the art and the computational work carried out have been structured around these questions. First, a cognitive science approach was taken to describe episodic memory in contrast to other cognitive functions (Chapter 3). Episodic memory is associative, consisting of associations between elements. Crucially, these associations can link elements separated in time while preserving temporal order. Episodic memory is known as explicit memory because its content has privileged access to consciousness, even though it may operate unconsciously. It differs from semantic memory in the specificity of its memories. Since episodic memory often learns in one-shot, it encodes the characteristics of each situation, rather than extracting abstract regularities from multiple experiences, as semantic memory does. These two memories are complementary. The first allows us to integrate new information very quickly, while the second allows us to generalize our knowledge to new situations. Initial episode encoding requires the hippocampus. The involvement of this structure in episodic recall varies from theory to theory, depending in particular on how old the memory is.
Chapter 4 provided a more in-depth overview of the biological imple-
mentation of episodic memory, focusing on the hippocampus. It has been
shown that the hippocampus is made up of several subfields. One that
stands out is the dentate gyrus with characteristic properties like adult
neurogenesis, sparsity, and steep sigmoidal response. It is also the most
populated region of the hippocampus. The medial temporal lobe employs
specific representations, including spatial ones with entorhinal grid cells
and hippocampal place cells. They serve as structural scaffolding for the
encoding of episodes. These cells also reflect memory operations during
different oscillation regimes. In the theta-gamma regime, the firing of grid
and place cells is organized in phase precessing sequences. During sharp-
wave ripples, experienced spatial trajectories are replayed at a faster pace.
This process underlies the transfer of information from episodic memory
to semantic memory. It has also been shown that the standard framework
of hippocampal computation posits that memories are stored in recurrent
collaterals of area CA3. Several criticisms of this model have been outlined,
one of which is the lack of emphasis on sequence learning. Furthermore,
the Hopfield network on which the framework is based on, is attracting
renewed interest. Modern, high-performance versions that integrate into
deep learning have been proposed, but they do not involve recurrent con-
nections. Overall, this calls for a new model of the hippocampus that is
able to learn sequences in one-shot, and modern Hopfield networks can
provide computational tools for its development.

In the first review chapter (Chapter 3), cognitive control has also been
introduced as the ability to learn explicit rules and modulate inappropriate
automatic responses. The interactions with episodic memory are threefold.
First, episodic memory can replay experiences to train cognitive control
offline. Secondly, episodic memory can supply information to the cogni-
tive controller online. For example, it can inform the controller about the
appropriateness of actions by extracting information from situations expe-
rienced once or only a few times. Finally, cognitive control can in turn
modulate episodic memory retrieval to guide memory search and obtain
contextually relevant information. These interactions have been described
in Chapter 5 as resulting from interactions between the prefrontal cortex
and the medial temporal lobe. As with other cortical areas, entorhinal neu-
rons exchange information with the prefrontal cortex. When memories are
replayed in the hippocampus, information can train the prefrontal cortex
offline. There is also a direct connection from CA1 to the medial prefrontal
cortex, which can provide episodic memory context for online decision-
making. Conversely, hippocampal neurons encode prefrontal information
sent by the entorhinal cortex. As a result, hippocampal representations re-
flect the current context. Prospective splitter cells disambiguate spatial po-
sitions according to future trajectory. It is likely that they emerge because
of the prefrontal cortex sending planning information. Similarly, retrospec-
tive splitter cells disambiguate spatial positions according to past trajectory.
These emerge internally, as the hippocampus keeps track of the tempo-
ral context. This way, recall is biased towards memories that are relevant
with respect to the current temporal context and ongoing goals. Prefrontal-
hippocampal interactions are scheduled by subcortical structures like the
nucleus reuniens of the thalamus, such that the prefrontal cortex controls
the hippocampus to bias memory retrieval, and the hippocampus influ-
ences the prefrontal cortex when episodic context is provided.

Some machine learning algorithms are inspired by these interactions,
mostly by the hippocampus-to-prefrontal direction. One of them is neu-
ral episodic control (Pritzel et al. 2017). It comprises a differentiable neu-
ral dictionary supposedly playing the role of the hippocampus to guide
actions, but this parallel has so far been made on a purely functional ba-
sis. It is unclear whether the hippocampus stores experiences in the same
way as the dictionary. Chapter 6 presents a formal proof that the dif-
ferentiable neural dictionary belongs to the same framework as Hopfield
networks: the recently proposed universal Hopfield framework (Millidge
et al. 2022). Just as the dictionary, models of this framework recall mem-
ories by performing similarity, separation and projection operations. The
implications are twofold. It has been shown that the dictionary can recall
arbitrary information like images. The second implication is that neural
episodic control could potentially be improved by using other instances
of the framework. This novel connection between reinforcement learning
and associative memory paves the way for further research. It also adds
a whole new dimension to a hippocampal model based on Hopfield net-
works, like the one presented in Chapter 8, as it can easily be integrated
into a reinforcement learning paradigm.

Most machine learning algorithms of memory-augmented control ignore
the contextual modulation of memory operations by the prefrontal cortex.
In Chapter 7, this topic was studied. It was demonstrated that modern
Hopfield networks can learn contextual memories in one-shot. For the first
time, a model of the hippocampus was modulated with context informa-
tion passing through multiple layers, instead of targeting memory directly.
The ability to separate semantic representations and to recall the correct
memory contextually was assessed as a function of the position of the Hop-
field network and the prefrontal target. The prefrontal cortex can influence
memory recall if memory is not too far downstream the targeted layer. Ex-
plicitly teaching the network to reconstruct context increases the distance
from which the context can influence memory. This study provides design
principles for the development of reinforcement learning models that can
use their knowledge more efficiently. In addition, the results further call
into question the standard framework, as targeting the entorhinal cortex is
inefficient when memories are stored in CA3. The model performs much
better when memories are stored in the dentate gyrus or CA1.

The CONSEQUENCE model, standing for contextual sequence, was pre-
sented in Chapter 8. All operations of the model are biologically plausi-
ble. Since it is based on a modern Hopfield network located in the den-
tate gyrus, it can be integrated into reinforcement learning and modulated
through prefrontal connections to the entorhinal cortex. Of all the possi-
ble implementations of a modern Hopfield network in the hippocampus,
CONSEQUENCE meets the most criteria (table 8.1). It largely outperforms
alternative models for one-shot sequence learning. Furthermore, it can ac-
count for biological findings, such as the formation of place cells whose
fields remap in novel and morphing environments. Finally, it is able to
learn overlapping sequences in one-shot too, by keeping track of the tem-
poral context in the same way as hippocampal retrospective splitter cells. This allows the model to perform the spatial alternation task after a single exposure. All in all, these results argue in favor of bringing together the study of the hippocampus and the development of modern Hopfield networks. Y. Zheng et al. (2022) called for *correcting the hebbian mistake* by using error-driven algorithms, similarly, the success of feedforward associative memory models sheds new light on the functioning of the hippocampus and questions the original proposal of J. J. Hopfield (1982).

Beyond scientific contributions, a lot of code was produced to perform the experiments. The code of Millidge et al. (2022) was extended to test the capacity of the differentiable neural dictionary in associative memory tasks<fnref n="1" />. The code of Sarrico et al. (2019) was augmented with new similarity and separation functions<fnref n="2" />. A testbed was created to investigate the contextual modulation of modern Hopfield networks in the hippocampus<fnref n="3" />. Finally, another testbed was developed to benchmark and optimize models on a series of tasks, including the new CONSEQUENCE model<fnref n="4" />.

PERSPECTIVES

One of the aims of this thesis was to improve machine learning algorithms with functions such as episodic memory and cognitive control. We urgently need to develop models capable of hindsight and aligned with our values. Just as *deep learning needs a prefrontal cortex* (Russin et al. 2020), it probably needs a hippocampus that bidirectionally interacts with its prefrontal cortex too (Chateau-Laurent and Alexandre 2021a).

The prospects opened up by the CONSEQUENCE model are manifold. The first step will be to tune the model to bring out place cells elsewhere than in the dentate gyrus, which is currently being done. Another direction is to study the model's generative capacities. The softmax function used

<footnote n="1">¹https://github.com/HugoChateauLaurent/DND_AssociativeMemory</footnote>

<footnote n="2">²https://github.com/HugoChateauLaurent/NEC_UHN</footnote>

<footnote n="3">³https://github.com/HugoChateauLaurent/ContextMHN</footnote>

<footnote n="4">⁴https://github.com/HugoChateauLaurent/OneShotSequenceLearning</footnote>
to separate memories outputs a probability distribution which could be
sampled instead of being used as is. Memory recall would then be stochas-
tic, and the generative capacities of the hippocampus could be explored.
Hippocampal contribution to imagination indeed relies on the ability to
generate unexperienced yet plausible sequences of events. CONSEQUENCE
could model this by sampling from the softmax, and the β parameter would
control the extent to which the model generates sequences that match its
experience or, on the contrary, generates novel transitions. This is in line
with the proposal that the study of the hippocampus can help in the de-
velopment of artificial creativity (Chateau-Laurent and Alexandre 2021b). An-
other important step for the longevity of the model would be to propose
a more precise description of the ordering of processing modes and how
they fit into hippocampal oscillation patterns. For the sake of simplicity,
the semantic pathway was trained prior to the encoding of memories in
the episodic pathway, but how the model as a whole develops with experi-
ence will need to be clarified in the future.

Ideally, the elements studied separately here should be integrated in a
common architecture (Figure 9.1). CONSEQUENCE could store and retrieve
experiences resembling current observations to feed a PFC-like LSTM, like
in Fortunato et al. (2019). The hippocampus would receive projections from
the LSTM to modulate encoding and retrieval in order for the LSTM to
receive helpful information. More biologically plausible models of the pre-
frontal cortex could also be used (e.g. Dagar 2023; W. H. Alexander and J. W.
Brown 2015; O’Reilly and M. J. Frank 2006). With this kind of architecture, it
would be possible to test whether hippocampal neurons projecting to the
prefrontal cortex have broad spatial representations (ventral) and neurons
targeted by the prefrontal cortex have small spatial representations (dorsal),
as predicted by Howard Eichenbaum (2017b). Interestingly, CONSEQUENCE
would add a temporal dimension to episodic control, as it stores sequences.
Just as Dreamer (Hafner et al. 2019) generates model-based imaginative se-
quences, CONSEQUENCE could generate prospective episodic sequences in
collaboration with the prefrontal cortex. This would bring machine learn-
ing closer to the construction system of the brain (Hassabis and Maguire
<figure><img src="image_74.png" /><figcaption>Figure 9.1: Proposed architecture for generative episodic control. Action is selected through sensorimotor associations controlled by the prefrontal cortex. The prefrontal controller interacts with the hippocampus to generate prospective episodic sequences. Offline, the hippocampus is used to consolidate new information in semantic memory and train the prefrontal cortex with simulated situations.</figcaption></figure>

2009). To encode all the specificity of memories and generalize efficiently at the same time, the hippocampus should not be a bottleneck like in neural episodic control, but rather be complemented by a semantic memory processing sensory inputs and selecting actions in a model-free way. Replays generated in the hippocampus would be used to train the controller and consolidate semantic memory offline.
It now appears more feasible to provide answers to the three initial ques-
tions posed in this thesis. The reason the narrator of *Swann's Way* is trans-
ported to the distant past is that the madeleine's flavor acts as a key con-
nected to specific neurons. These neurons, once selectively chosen among
irrelevant ones, reactivate the value of the association and trigger a cascade
of information that replaces cortical representations with those dedicated
to a singular experience: Sunday mornings at his aunt Léonie's house in
Combray. How is this recollection meant to influence behavior? It does
not, it is just a madeleine.
BIBLIOGRAPHY

Agster, Kara L and Rebecca D Burwell (2009). "Cortical efferents of the perirhinal, postrhinal, and entorhinal cortices of the rat". *Hippocampus* 19:12, pp. 1159-1186.

Agster, Kara L, Norbert J Fortin, and Howard Eichenbaum (2002). "The hippocampus and disambiguation of overlapping sequences". *Journal of Neuroscience* 22:13, pp. 5760-5768.

Aguilera, Matthieu, Vincent Douchamps, Demian Battaglia, and Romain Goutagny (2022). "How Many Gammas? Redefining Hippocampal Theta-Gamma Dynamic During Spatial Learning". *Frontiers in behavioral neuroscience* 16, p. 811278.

Alexander, William H and Joshua W Brown (2015). "Hierarchical error representation: a computational model of anterior cingulate and dorsolateral prefrontal cortex". *Neural Computation* 27:11, pp. 2354-2410.

Allen, Gary V and David A Hopkins (1989). "Mamillary body in the rat: topography and synaptology of projections from the subicular complex, prefrontal cortex, and midbrain tegmentum". *Journal of Comparative Neurology* 286:3, pp. 311-336.

Alvernhe, Alice, Etienne Save, and Bruno Poucet (2011). "Local remapping of place cell firing in the Tolman detour task". *European Journal of Neuroscience* 33:9, pp. 1696-1705.

Ambrose, R Ellen, Brad E Pfeiffer, and David J Foster (2016). "Reverse replay of hippocampal place cells is uniquely modulated by changing reward". *Neuron* 91:5, pp. 1124-1136.

Andersen, Per (1975). "Organization of hippocampal neurons and their interconnections". In: *The Hippocampus: Volume 1: Structure and Development*. Springer, pp. 155-175.
Anderson, Michael C, Jamie G Bunce, and Helen Barbas (2016). "Prefrontal-hippocampal pathways underlying inhibitory control over memory". *Neurobiology of learning and memory* 134, pp. 145-161.

Andrianova, Lilya, Steliana Yanakieva, Gabriella Margetts-Smith, Shivali Kohli, Erica S Brady, John P Aggleton, and Michael T Craig (2023). "No evidence from complementary data sources of a direct glutamatergic projection from the mouse anterior cingulate area to the hippocampal formation". *Elife* 12, e77364.

Apergis-Schoute, John, Aline Pinto, and Denis Paré (2006). "Ultrastructural organization of medial prefrontal inputs to the rhinal cortices". *European Journal of Neuroscience* 24:1, pp. 135-144.

Aronov, Dmitriy, Rhino Nevers, and David W Tank (2017). "Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit". *Nature* 543:7647, pp. 719-722.

Atkinson, Richard C and Richard M Shiffrin (1968). "Human memory: A proposed system and its control processes". In: *Psychology of learning and motivation*. Vol. 2. Elsevier, pp. 89-195.

Augusto-Oliveira, Marcus, Gabriela PF Arrifano, João O Malva, and Maria Elena Crespo-Lopez (2019). "Adult hippocampal neurogenesis in different taxonomic groups: Possible functional similarities and striking controversies". *Cells* 8:2, p. 125.

Backus, Alexander R, Jan-Mathijs Schoffelen, Szabolcs Szebényi, Simon Hanslmayr, and Christian F Doeller (2016). "Hippocampal-prefrontal theta oscillations support memory integration". *Current Biology* 26:4, pp. 450-457.

Baddeley, Alan (2000). "The episodic buffer: a new component of working memory?" *Trends in cognitive sciences* 4:11, pp. 417-423.

Baddeley, Alan D and J Graham (1974). "Working memory". *The psychology of learning and motivation* 8, pp. 47-89.

Badre, David and Mark D'Esposito (2007). "Functional magnetic resonance imaging evidence for a hierarchical organization of the prefrontal cortex". *Journal of cognitive neuroscience* 19:12, pp. 2082-2099.
Baeg, EH, YB Kim, K Huh, I Mook-Jung, HT Kim, and MW Jung (2003).
"Dynamics of population code for working memory in the prefrontal cortex". *Neuron* 40:1, pp. 177-188.

Bakker, Hanna den, Marie Van Dijck, Jyh-Jang Sun, and Fabian Kloosterman (2023). "Sharp-wave-ripple-associated activity in the medial prefrontal cortex supports spatial rule switching". *Cell Reports* 42:8.

Banino, Andrea, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. (2018). "Vector-based navigation using grid-like representations in artificial agents". *Nature* 557:7705, pp. 429-433.

Barnes, Carol A, Bruce L McNaughton, Shed JY Mizumori, Brian W Leonard, and Lei-Huey Lin (1990). "Chapter Comparison of spatial and temporal characteristics of neuronal activity in sequential stages of hippocampal processing". *Progress in brain research* 83, pp. 287-300.

Barnes, Jean M and Benton J Underwood (1959). "" Fate" of first-list associations in transfer theory." *Journal of experimental psychology* 58:2, p. 97.

Bartesaghi, R, M Migliore, and T Gessi (2006). "Input-output relations in the entorhinal cortex–dentate–hippocampal system: Evidence for a non-linear transfer of signals". *Neuroscience* 142:1, pp. 247–265.

Bartesaghi, Renata, Tiziana Gessi, and Michele Migliore (1995). "Input-output relations in the entorhinal-hippocampal-entorhinal loop: Entorhinal cortex and dentate gyrus". *Hippocampus* 5:5, pp. 440-451.

Battaglia, Francesco P, Gary R Sutherland, and Bruce L McNaughton (2004).
"Local sensory cues and place cell directionality: additional evidence of
prospective coding in the hippocampus". *Journal of Neuroscience* 24:19,
pp. 4541-4550.

Bauer, Richard H and Joaquin M Fuster (1976). "Delayed-matching and delayed-response deficit from cooling dorsolateral prefrontal cortex in monkeys." *Journal of comparative and physiological psychology* 90:3, p. 293.

Becker, Suzanna (2005). "A computational principle for hippocampal learning and neurogenesis". *Hippocampus* 15:6, pp. 722-738.
Benchenane, Karim, Adrien Peyrache, Mehdi Khamassi, Patrick L Tierney, Yves Gioanni, Francesco P Battaglia, and Sidney I Wiener (2010). "Coherent theta oscillations and reorganization of spike timing in the hippocampal-prefrontal network upon learning". *Neuron* 66:6, pp. 921-936.

Bengio, Yoshua, Patrice Simard, and Paolo Frasconi (1994). "Learning long-term dependencies with gradient descent is difficult". *IEEE transactions on neural networks* 5:2, pp. 157-166.

Benna, Marcus K and Stefano Fusi (2021). "Place cells may simply be memory cells: Memory compression leads to spatial tuning and history dependence". *Proceedings of the National Academy of Sciences* 118:51, e2018422118.

Bentley, Jon Louis (1975). "Multidimensional binary search trees used for associative searching". *Communications of the ACM* 18:9, pp. 509-517.

Berners-Lee, Alice, Xiaojing Wu, and David J Foster (2021). "Prefrontal cortical neurons are selective for non-local hippocampal representations during replay and behavior". *Journal of Neuroscience* 41:27, pp. 5894-5908.

Bernier, Brian E, Anthony F Lacagnina, Adam Ayoub, Francis Shue, Boris V Zemelman, Franklin B Krasne, and Michael R Drew (2017). "Dentate gyrus contributes to retrieval as well as encoding: evidence from context fear conditioning, recall, and extinction". *Journal of Neuroscience* 37:26, pp. 6359-6371.

Beukers, Andre O, Timothy J Buschman, Jonathan D Cohen, and Kenneth A Norman (2021). "Is activity silent working memory simply episodic memory?" *Trends in cognitive sciences* 25:4, pp. 284-293.

Bian, Xin-Lan, Cheng Qin, Cheng-Yun Cai, Ying Zhou, Yan Tao, Yu-Hui Lin, Hai-Yin Wu, Lei Chang, Chun-Xia Luo, and Dong-Ya Zhu (2019). "Anterior cingulate cortex to ventral hippocampus circuit mediates context fear generalization". *Journal of Neuroscience* 39:29, pp. 5728-5739.

Bieri, Kevin Wood, Katelyn N Bobbitt, and Laura Lee Colgin (2014). "Slow and fast gamma rhythms coordinate different spatial coding modes in hippocampal place cells". *Neuron* 82:3, pp. 670-681.

Blundell, Charles, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis (2016). "Model-free episodic control". *arXiv preprint arXiv:1606.04460*.
Botvinick, Matthew, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis (2019). "Reinforcement learning, fast and slow". *Trends in cognitive sciences* 23:5, pp. 408-422.

Brankačk, Jurij, Mark Stewart, and Steven E Fox (1993). "Current source density analysis of the hippocampal theta rhythm: associated sustained potentials and candidate synaptic generators". *Brain research* 615:2, pp. 310-327.

Brasted, Peter J and Steven P Wise (2004). "Comparison of learning-related neuronal activity in the dorsal premotor cortex and striatum". *European Journal of Neuroscience* 19:3, pp. 721-740.

Brockman, Greg, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba (2016). "OpenAI gym". *arXiv preprint arXiv:1606.01540*.

Brown, Jason, Christiana M Cooper-Kuhn, Gerd Kempermann, Henriette Van Praag, Jürgen Winkler, Fred H Gage, and H Georg Kuhn (2003). "Enriched environment and physical activity stimulate hippocampal but not olfactory bulb neurogenesis". *European journal of Neuroscience* 17:10, pp. 2042-2046.

Bruck, Jehoshua (1990). "On the convergence properties of the Hopfield model". *Proceedings of the IEEE* 78:10, pp. 1579-1585.

Burak, Yoram and Ila R Fiete (2009). "Accurate path integration in continuous attractor network models of grid cells". *PLoS computational biology* 5:2, e1000291.

Burwell, Rebecca D and David G Amaral (1998). "Cortical afferents of the perirhinal, postrhinal, and entorhinal cortices of the rat". *Journal of comparative neurology* 398:2, pp. 179-205.

Butler, James L and Ole Paulsen (2014). "The hippocampal cacophony: multiple layers of communication". *Neuron* 84:2, pp. 251-253.

Buzsáki, György (1989). "Two-stage model of memory trace formation: a role for "noisy" brain states". *Neuroscience* 31:3, pp. 551-570.

Buzsáki, György and David Tingley (2018). "Space and time: the hippocampus as a sequence generator". *Trends in cognitive sciences* 22:10, pp. 853-869.
Buzsáki, György and Brendon O Watson (2012). "Brain rhythms and neural syntax: implications for efficient coding of cognitive content and neuropsychiatric disease." *Dialogues in clinical neuroscience*.

Cassel, Jean-Christophe, Anne Pereira De Vasconcelos, Michaël Loureiro, Thibault Cholvin, John C Dalrymple-Alford, and Robert P Vertes (2013). "The reuniens and rhomboid nuclei: neuroanatomy, electrophysiological characteristics and behavioral implications". *Progress in neurobiology* 111, pp. 34–52.

Caucheteux, Charlotte and Jean-Rémi King (2022). "Brains and algorithms partially converge in natural language processing". *Communications biology* 5:1, p. 134.

Cazé, Romain, Mehdi Khamassi, Lise Aubin, and Benoît Girard (2018). "Hippocampal replays under the scrutiny of reinforcement learning models". *Journal of neurophysiology* 120:6, pp. 2877–2896.

Cazin, Nicolas (2018). "A replay driven model of spatial sequence learning in the hippocampus-prefrontal cortex network using reservoir computing". PhD thesis. Université de Lyon.

Chandra, Sarthak, Sugandha Sharma, Rishidev Chaudhuri, and Ila Fiete (2023). "High-capacity flexible hippocampal associative and episodic memory enabled by prestructured"spatial"representations". *bioRxiv*, pp. 2023-11.

Chateau-Laurent, Hugo and Frédéric Alexandre (2021a). "Augmenting machine learning with flexible episodic memory". In: *13th International Joint Conference on Computational Intelligence*.

- (2021b). "Towards a Computational Cognitive Neuroscience Model of Creativity". In: *2021 IEEE 20th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*\* CC). IEEE, pp. 67–71.

Chaudhry, Hamza Tahir, Jacob A Zavatone-Veth, Dmitry Krotov, and Cenziz Pehlevan (2023). "Long Sequence Hopfield Memory". *arXiv preprint arXiv:2306.04532*.

Cheng, Sen (2013). "The CRISP theory of hippocampal function in episodic memory". *Frontiers in neural circuits* 7, p. 88.
Chudasama, Yogita, Victoria M Doobay, and Yuchen Liu (2012). "Hippocampal-prefrontal cortical circuit mediates inhibitory response control in the rat". *Journal of Neuroscience* 32:32, pp. 10915-10924.

Cipolotti, Lisa, Tim Shallice, Dennis Chan, Nick Fox, Rachel Scahill, Gail Harrison, John Stevens, and Peter Rudge (2001). "Long-term retrograde amnesia... the crucial role of the hippocampus". *Neuropsychologia* 39:2, pp. 151-172.

Clark, Andy (2008). *Supersizing the mind: Embodiment, action, and cognitive extension*. OUP USA.

Clark, Andy and David Chalmers (1998). "The extended mind". *analysis* 58:1, pp. 7-19.

Cohen, Jonathan D and Randall C O'Reilly (1996). "A preliminary theory of the interactions between prefrontal cortex and hippocampus that contribute to planning and prospective memory". *Prospective memory: Theory and applications*, pp. 267-295.

Cohen, Neal J, Jennifer Ryan, Caroline Hunt, Lorene Romine, Tracey Wszalek, and Courtney Nash (1999). "Hippocampal system and declarative (relational) memory: summarizing the data from functional neuroimaging studies". *Hippocampus* 9:1, pp. 83-98.

Colgin, Laura Lee (2015). "Do slow and fast gamma rhythms correspond to distinct functional states in the hippocampal network?" *Brain research* 1621, pp. 309-315.

Colgin, Laura Lee, Tobias Denninger, Marianne Fyhn, Torkel Hafting, Tora Bonnevie, Ole Jensen, May-Britt Moser, and Edvard I Moser (2009). "Frequency of gamma oscillations routes flow of information in the hippocampus". *Nature* 462:7271, pp. 353-357.

Cowen, Stephen L, Glen A Davis, and Douglas A Nitz (2012). "Anterior cingulate neurons in the rat map anticipated effort and reward to their associated action sequences". *Journal of neurophysiology* 107:9, pp. 2393-2407.

Cravens, Catherine J, Noelia Vargas-Pinto, Kimberly M Christian, and Kazu Nakazawa (2006). "CA3 NMDA receptors are crucial for rapid and automatic representation of context memory". *European Journal of Neuroscience* 24:6, pp. 1771-1780.
Cueva, Christopher J and Xue-Xin Wei (2018). "Emergence of grid-like representations by training recurrent neural networks to perform spatial localization". *arXiv preprint arXiv:1803.07770*.

Cutsuridis, Vassilis, Bruce P Graham, Stuart Cobb, and Imre Vida (2019). *Hippocampal microcircuits: a computational modeler's resource book*. Springer.

Dagar, Snigdha (2023). "Computational modeling of Cognitive Control for rule-guided behavior". PhD thesis. Université de Bordeaux.

Dayan, Peter (1993). "Improving generalization for temporal difference learning: The successor representation". *Neural computation* 5:4, pp. 613-624.

De Lavilléon, Gaetan, Marie Masako Lacroix, Laure Rondi-Reig, and Karim Benchenane (2015). "Explicit memory creation during sleep demonstrates a causal role of place cells in navigation". *Nature neuroscience* 18:4, pp. 493-495.

Degonda, Nadia, Christian RA Mondadori, Simone Bosshardt, Conny F Schmidt, Peter Boesiger, Roger M Nitsch, Christoph Hock, and Katharina Henke (2005). "Implicit associative learning engages the hippocampus and interacts with explicit associative learning". *Neuron* 46:3, pp. 505-520.

Dehaene, Stanislas, Jean-Pierre Changeux, Lionel Naccache, Jérôme Sackur, and Claire Sergent (2006). "Conscious, preconscious, and subliminal processing: a testable taxonomy". *Trends in cognitive sciences* 10:5, pp. 204-211.

Demircigil, Mete, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet (2017). "On a model of associative memory with huge storage capacity". *Journal of Statistical Physics* 168, pp. 288-299.

Denny, Christine A, Mazen A Kheirbek, Eva L Alba, Kenji F Tanaka, Rebecca A Brachman, Kimberly B Laughman, Nicole K Tomm, Gergely F Turi, Attila Losonczy, and René Hen (2014). "Hippocampal memory traces are differentially modulated by experience, time, and adult neurogenesis". *Neuron* 83:1, pp. 189-201.

Deshmukh, Sachin S and James J Knierim (2013). "Influence of local objects on hippocampal representations: Landmark vectors and memory". *Hippocampus* 23:4, pp. 253-267.
Diamantaki, Maria, Markus Frey, Philipp Berens, Patricia Preston-Ferrer, and Andrea Burgalossi (2016). "Sparse activity of identified dentate granule cells during spatial exploration". *Elife* 5, e20252.

Diba, Kamran and György Buzsáki (2007). "Forward and reverse hippocampal place-cell sequences during ripples". *Nature neuroscience* 10:10, pp. 1241–1242.

Dickinson, Anthony and Bernard Balleine (2002). "The role of learning in the operation of motivational systems". *Stevens' handbook of experimental psychology* 3, pp. 497–533.

Dobbins, Ian G, Heather Foley, Daniel L Schacter, and Anthony D Wagner (2002). "Executive control during episodic retrieval: multiple prefrontal processes subserve source memory". *Neuron* 35:5, pp. 989–996.

Dolleman-van der Weel, Margriet J, Amy L Griffin, Hiroshi T Ito, Matthew L Shapiro, Menno P Witter, Robert P Vertes, and Timothy A Allen (2019). "The nucleus reuniens of the thalamus sits at the nexus of a hippocampus and medial prefrontal cortex circuit enabling memory and behavior". *Learning & Memory* 26:7, p. 191.

Dolleman-Van Der Weel, MJ and MP Witter (1996). "Projections from the nucleus reuniens thalami to the entorhinal cortex, hippocampal field CA1, and the subiculum in the rat arise from different populations of neurons". *Journal of comparative neurology* 364:4, pp. 637–650.

Douchamps, Vincent, Ali Jeewajee, Pam Blundell, Neil Burgess, and Colin Lever (2013). "Evidence for encoding versus retrieval scheduling in the hippocampus by theta phase and acetylcholine". *Journal of Neuroscience* 33:20, pp. 8689–8704.

Drieu, Céline and Michaël Zugaro (2019). "Hippocampal sequences during exploration: mechanisms and functions". *Frontiers in cellular neuroscience* 13, p. 232.

Dupret, David, Joseph O'Neill, Barty Pleydell-Bouverie, and Jozsef Csicsvari (2010). "The reorganization and reactivation of hippocampal maps predict spatial memory performance". *Nature neuroscience* 13:8, pp. 995–1002.
Dusek, Jeffery A and Howard Eichenbaum (1997). "The hippocampus and memory for orderly stimulus relations". *Proceedings of the National Academy of Sciences* 94:13, pp. 7109-7114.

Duvelle, Éléonore, Roddy M Grieves, and Matthijs AA Van der Meer (2023). "Temporal context and latent state inference in the hippocampal splitter signal". *ELife* 12, e82357.

Ego-Stengel, Valérie and Matthew A Wilson (2010). "Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat". *Hippocampus* 20:1, pp. 1-10.

Egorov, Alexei V, Bassam N Hamam, Erik Fransén, Michael E Hasselmo, and Angel A Alonso (2002). "Graded persistent activity in entorhinal cortex neurons". *Nature* 420:6912, pp. 173-178.

Eichenbaum, H, M Kuperstein, A Fagan, and J Nagode (1987). "Cue-sampling and goal-approach correlates of hippocampal unit activity in rats performing an odor-discrimination task". *Journal of Neuroscience* 7:3, pp. 716-732.

Eichenbaum, Howard (2017a). "Memory: organization and control". *Annual review of psychology* 68, pp. 19-45.

- (2017b). "Prefrontal-hippocampal interactions in episodic memory". *Nature Reviews Neuroscience* 18:9, pp. 547-558.

Eichenbaum, Howard, Paul Dudchenko, Emma Wood, Matthew Shapiro, and Heikki Tanila (1999). "The hippocampus, memory, and place cells: is it spatial memory or a memory space?" *Neuron* 23:2, pp. 209-226.

Eichenbaum, Howard, Andrew P Yonelinas, and Charan Ranganath (2007). "The medial temporal lobe and recognition memory". *Annu. Rev. Neurosci.* 30, pp. 123-152.

Elman, Jeffrey L (1990). "Finding structure in time". *Cognitive science* 14:2, pp. 179-211.

Ergorul, Ceren and Howard Eichenbaum (2004). "The hippocampus and memory for "what", "where", and "when"". *Learning & Memory* 11:4, pp. 397-405.
Farovik, Anja, Laura M Dupont, Miguel Arce, and Howard Eichenbaum (2008). "Medial prefrontal cortex supports recollection, but not familiarity, in the rat". *Journal of Neuroscience* 28:50, pp. 13428-13434.

Feng, Ting, Delia Silva, and David J Foster (2015). "Dissociation between the experience-dependent development of hippocampal theta sequences and single-trial phase precession". *Journal of Neuroscience* 35:12, pp. 4890-4902.

Fernández-Ruiz, Antonio, Azahara Oliva, Eliezyer Fermino de Oliveira, Flor-bela Rocha-Almeida, David Tingley, and György Buzsáki (2019). "Long-duration hippocampal sharp wave ripples improve memory". *Science* 364:6445, pp. 1082-1086.

Fernández-Ruiz, Antonio, Azahara Oliva, Gergő A Nagy, Andrew P Maur-rer, Antal Berényi, and György Buzsáki (2017). "Entorhinal-CA3 dual-input control of spike timing in the hippocampus by theta-gamma cou-pling". *Neuron* 93:5, pp. 1213-1226.

Fortin, Norbert J, Kara L Agster, and Howard Eichenbaum (2002). "Critical role of the hippocampus in memory for sequences of events". *Nature neuroscience* 5:5, pp. 458-462.

Fortunato, Meire, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puig-domènech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell (2019). "Generalization of reinforcement learners with working and episodic memory". *Advances in neural information processing systems* 32.

Foster, David J and Matthew A Wilson (2006). "Reverse replay of behavioural sequences in hippocampal place cells during the awake state". *Nature* 440:7084, pp. 680-683.

Francisco, Varela, Thompson Evan, and Rosch Eleanor (1991). "The embodied mind: Cognitive science and human experience". *Massachusetts Institute of Technology*.

Frank, Loren M, Emery N Brown, and Matthew Wilson (2000). "Trajectory encoding in the hippocampus and entorhinal cortex". *Neuron* 27:1, pp. 169-178.
Frankland, Paul W and Bruno Bontempi (2005). "The organization of recent and remote memories". *Nature reviews neuroscience* 6:2, pp. 119–130.

Fransén, Erik, Babak Tahvildari, Alexei V Egorov, Michael E Hasselmo, and Angel A Alonso (2006). "Mechanism of graded persistent cellular activity of entorhinal cortex layer v neurons". *Neuron* 49:5, pp. 735–746.

Fried, Itzhak, Katherine A MacDonald, and Charles L Wilson (1997). "Single neuron activity in human hippocampus and amygdala during recognition of faces and objects". *Neuron* 18:5, pp. 753–765.

Fuhs, Mark C and David S Touretzky (2006). "A spin glass model of path integration in rat medial entorhinal cortex". *Journal of Neuroscience* 26:16, pp. 4266–4276.

- (2007). "Context learning in the rodent hippocampus". *Neural computation* 19:12, pp. 3173–3215.

Funahashi, Shintaro, Charles J Bruce, and Patricia S Goldman-Rakic (1989). "Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex". *Journal of neurophysiology* 61:2, pp. 331–349.

- (1993). "Dorsolateral prefrontal lesions and oculomotor delayed-response performance: evidence for mnemonic scotomas". *Journal of Neuroscience* 13:4, pp. 1479–1497.

Fuster, Joaquin M and Garrett E Alexander (1971). "Neuron activity related to short-term memory". *Science* 173:3997, pp. 652–654.

Gaal, Simon van, Floris P De Lange, and Michael X Cohen (2012). "The role of consciousness in cognitive control and decision making". *Frontiers in human neuroscience* 6, p. 121.

Gabrieli, John DE, Suzanne Corkin, Susan F Mickel, and John H Growdon (1993). "Intact acquisition and long-term retention of mirror-tracing skill in Alzheimer's disease and in global amnesia." *Behavioral neuroscience* 107:6, p. 899.

Gardner, Richard J, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A Baas, Benjamin A Dunn, May-Britt Moser, and Edvard I Moser (2022). "Toroidal topology of population activity in grid cells". *Nature* 602:7895, pp. 123–128.
George, Tom M, William de Cothi, Claudia Clopath, Kimberly Stachenfeld, and Caswell Barry (2022). "RatInABox: A toolkit for modelling locomotion and neuronal activity in continuous environments". bioRxiv, pp. 2022-08.

George, Tom M, William de Cothi, Kimberly L Stachenfeld, and Caswell Barry (2023). "Rapid learning of predictive maps with STDP and theta phase precession". Elife 12, e80663.

Gershman, Samuel J and Nathaniel D Daw (2017). "Reinforcement learning and episodic memory in humans and animals: an integrative framework". Annual review of psychology 68, pp. 101-128.

Gershman, Samuel J, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg (2012). "The successor representation and temporal context". Neural Computation 24:6, pp. 1553-1568.

Gerstner, Wulfram, Marco Lehmann, Vasiliki Liakoni, Dane Corneil, and Johanni Brea (2018). "Eligibility traces and plasticity on behavioral time scales: experimental support of neohebbian three-factor learning rules". Frontiers in neural circuits 12, p. 53.

Gilbert, Daniel T and Timothy D Wilson (2007). "Prospection: Experiencing the future". Science 317:5843, pp. 1351-1354.

Gilbert, Paul E, Raymond P Kesner, and Inah Lee (2001). "Dissociating hippocampal subregions: A double dissociation between dentate gyrus and CA1". Hippocampus 11:6, pp. 626-636.

Girardeau, Gabrielle, Karim Benchenane, Sidney I Wiener, György Buzsáki, and Michaël B Zugaro (2009). "Selective suppression of hippocampal ripples impairs spatial memory". Nature neuroscience 12:10, pp. 1222-1223.

Giustino, Thomas F and Stephen Maren (2015). "The role of the medial prefrontal cortex in the conditioning and extinction of fear". Frontiers in behavioral neuroscience 9, p. 298.

Goldman-Rakic, Patricia S (1996). "The prefrontal landscape: implications of functional architecture for understanding human mentation and the central executive". Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 351:1346, pp. 1445-1453.
Gonzalo-Ruiz, A, A Alonso, JM Sanz, and RR Llinas (1992). "Afferent projections to the mammillary complex of the rat, with special reference to those from surrounding hypothalamic regions". *Journal of Comparative Neurology* 321:2, pp. 277-299.

Goyal, Anirudh, Abram Friesen, Andrea Banino, Theophane Weber, Nan Rosemary Ke, Adria Puigdomenech Badia, Arthur Guez, Mehdi Mirza, Peter C Humphreys, Ksenia Konyushova, et al. (2022). "Retrieval-augmented reinforcement learning". In: *International Conference on Machine Learning*. PMLR, pp. 7740-7765.

Graves, Alex, Greg Wayne, and Ivo Danihelka (2014). "Neural turing machines". *arXiv preprint arXiv:1410.5401*.

Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. (2016). "Hybrid computing using a neural network with dynamic external memory". *Nature* 538:7626, pp. 471-476.

Gridchyn, Igor, Philipp Schoenenberger, Joseph O'Neill, and Jozsef Csicsvari (2020). "Assembly-specific disruption of hippocampal replay leads to selective memory deficit". *Neuron* 106:2, pp. 291-300.

Guanella, Alexis, Daniel Kiper, and Paul Verschure (2007). "A model of grid cells based on a twisted torus topology". *International journal of neural systems* 17:04, pp. 231-240.

Guise, Kevin G and Matthew L Shapiro (2017). "Medial prefrontal cortex reduces memory interference by modifying hippocampal encoding". *Neuron* 94:1, pp. 183-192.

Gupta, Anoopum S, Matthijs AA Van Der Meer, David S Touretzky, and A David Redish (2010). "Hippocampal replay is not a simple function of experience". *Neuron* 65:5, pp. 695-705.

Hafner, Danijar, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi (2019). "Dream to control: Learning behaviors by latent imagination". *arXiv preprint arXiv:1912.01603*.
Hafting, Torkel, Marianne Fyhn, Tora Bonnevie, May-Britt Moser, and Ed-
vard I Moser (2008). "Hippocampus-independent phase precession in en-
torhinal grid cells". *Nature* 453:7199, pp. 1248–1252.

Hafting, Torkel, Marianne Fyhn, Sturla Molden, May-Britt Moser, and Ed-
vard I Moser (2005). "Microstructure of a spatial map in the entorhinal
cortex". *Nature* 436:7052, pp. 801–806.

Hahnloser, Richard HR, Alexay A Kozhevnikov, and Michale S Fee (2003).
“Erratum: An ultra-sparse code underlies the generation of neural se-
quences in a songbird”. *Nature* 421:6920, pp. 294–294.

Hainmueller, Thomas and Marlene Bartos (2020). "Dentate gyrus circuits for encoding, retrieval and discrimination of episodic memories". *Nature Reviews Neuroscience* 21:3, pp. 153–168.

Hallock, Henry L, Arick Wang, and Amy L Griffin (2016). "Ventral midline thalamus is critical for hippocampal–prefrontal synchrony and spatial working memory". *Journal of Neuroscience* 36:32, pp. 8372–8389.

Halton, John H (1960). "On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals". *Numerische Mathematik* 2, pp. 84–90.

Hannula, Deborah E and Anthony J Greene (2012). "The hippocampus reeval-
uated in unconscious learning and memory: at a tipping point?" *Frontiers
in human neuroscience* 6, p. 80.

Hannula, Deborah E and Charan Ranganath (2009). "The eyes have it: hip-
pocampal activity predicts expression of memory in eye movements".
*Neuron* 63:5, pp. 592–599.

Hannula, Deborah E, Jennifer D Ryan, Daniel Tranel, and Neal J Cohen (2007).
"Rapid onset relational memory effects are evident in eye movement be-
havior, but not in hippocampal amnesia". *Journal of cognitive neuroscience*
19:10, pp. 1690–1705.

Hassabis, Demis, Dharshan Kumaran, and Eleanor A Maguire (2007). "Us-
ing imagination to understand the neural basis of episodic memory".
*Journal of neuroscience* 27:52, pp. 14365–14374.

Hassabis, Demis, Dharshan Kumaran, Seralynne D Vann, and Eleanor A Maguire (2007). "Patients with hippocampal amnesia cannot imagine
new experiences”. *Proceedings of the National Academy of Sciences* 104:5, pp. 1726–1731.

Hassabis, Demis and Eleanor A Maguire (2009). "The construction system of the brain". *Philosophical Transactions of the Royal Society B: Biological Sciences* 364:1521, pp. 1263–1271.

Hasselmo, Michael E, Clara Bodelón, and Bradley P Wyble (2002). "A proposed function for hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning". *Neural computation* 14:4, pp. 793–817.

Hasselmo, Michael E and Howard Eichenbaum (2005). "Hippocampal mechanisms for the context-dependent retrieval of episodes". *Neural networks* 18:9, pp. 1172–1190.

Hasselmo, Michael E and Chantal E Stern (2006). "Mechanisms underlying working memory for novel information". *Trends in cognitive sciences* 10:11, pp. 487–493.

Hayden, Benjamin Y and Michael L Platt (2010). "Neurons in anterior cingulate cortex multiplex information about reward and action". *Journal of Neuroscience* 30:9, pp. 3339–3346.

Head, Tim, Manoj Kumar, Holger Nahrstaedt, Gilles Louppe, and Iaroslav Shcherbatyi (2021). *scikit-optimize/scikit-optimize*. Version v0.9.0. DOI: 10.5281/zenodo.5565057. URL: https://doi.org/10.5281/zenodo.5565057.

Hebb, Donald Olding (1949). *The organisation of behaviour: a neuropsychological theory*. Science Editions New York.

Heckers, Stephan, Martin Zalesak, Anthony P Weiss, Tali Ditman, and Debra Titone (2004). "Hippocampal activation during transitive inference in humans". *Hippocampus* 14:2, pp. 153–162.

Henke, Katharina (2010). "A model for memory systems based on processing modes rather than consciousness". *Nature Reviews Neuroscience* 11:7, pp. 523–532.

Henke, Katharina, Christian RA Mondadori, Valerie Treyer, Roger M Nitsch, Alfred Buck, and Christoph Hock (2003). "Nonconscious formation and reactivation of semantic associations by way of the medial temporal lobe". *Neuropsychologia* 41:8, pp. 863–876.
Henke, Katharina, Valerie Treyer, Eva Turi Nagy, Stefan Kneifel, Max Dürsteler, Roger M Nitsch, and Alfred Buck (2003). "Active hippocampus during nonconscious memories". *Consciousness and cognition* 12:1, pp. 31–48.

Henze, Darrell A, Nathaniel N Urban, and German Barrionuevo (2000). "The multifarious hippocampal mossy fiber pathway: a review". *Neuroscience* 98:3, pp. 407–427.

Heys, James G, Krsna V Rangarajan, and Daniel A Dombeck (2014). "The functional micro-organization of grid cells revealed by cellular-resolution imaging". *Neuron* 84:5, pp. 1079–1090.

Hochreiter, Sepp and Jürgen Schmidhuber (1997). "Long short-term memory". *Neural computation* 9:8, pp. 1735–1780.

Holroyd, Clay B and Nick Yeung (2012). "Motivation of extended behaviors by anterior cingulate cortex". *Trends in cognitive sciences* 16:2, pp. 122–128.

Hölscher, Christian, Roger Anwyl, and Michael J Rowan (1997). "Stimulation on the positive phase of hippocampal theta rhythm induces long-term potentiation that can be depotentiated by stimulation on the negative phase in area CA1 in vivo". *Journal of Neuroscience* 17:16, pp. 6470–6477.

Hoover, Walter B and Robert P Vertes (2007). "Anatomical analysis of afferent projections to the medial prefrontal cortex in the rat". *Brain Structure and Function* 212, pp. 149–179.

Hopfield, John J (1982). "Neural networks and physical systems with emergent collective computational abilities." *Proceedings of the national academy of sciences* 79:8, pp. 2554–2558.

- (1984). "Neurons with graded response have collective computational properties like those of two-state neurons." *Proceedings of the national academy of sciences* 81:10, pp. 3088–3092.

Howard, Marc W and Michael J Kahana (2002). "A distributed representation of temporal context". *Journal of mathematical psychology* 46:3, pp. 269–299.

Hyman, James M, Eric A Zilli, Amanda M Paley, and Michael E Hasselmo (2005). "Medial prefrontal cortex cells show dynamic modulation with
the hippocampal theta rhythm dependent on behavior". *Hippocampus* 15:6, pp. 739-749.

Hyman, James M, Eric A Zilli, Amanda M Paley, and Michael E Hasselmo (2010). "Working memory performance correlates with prefrontal-hippocampal theta interactions but not with prefrontal neuron firing rates". *Frontiers in integrative neuroscience* 4, p. 1162.

Ising, Ernst (1925). "Beitrag zur theorie des ferromagnetismus". *Zeitschrift für Physik* 31:1, pp. 253–258.

Ito, Hiroshi T, Edvard I Moser, and May-Britt Moser (2018). "Supramillary nucleus modulates spike-time coordination in the prefrontal-thalamo-hippocampal circuit during navigation". *Neuron* 99:3, pp. 576–587.

Ito, Hiroshi T, Sheng-Jia Zhang, Menno P Witter, Edvard I Moser, and May-Britt Moser (2015). "A prefrontal-thalamo-hippocampal circuit for goal-directed spatial navigation". *Nature* 522:7554, pp. 50–55.

Itskov, Vladimir, Carina Curto, Eva Pastalkova, and György Buzsáki (2011). "Cell assembly sequences arising from spike threshold adaptation keep track of time in the hippocampus". *Journal of Neuroscience* 31:8, pp. 2828–2834.

Jadhav, Shantanu P, Caleb Kemere, P Walter German, and Loren M Frank (2012). "Awake hippocampal sharp-wave ripples support spatial memory". *Science* 336:6087, pp. 1454–1458.

Jadhav, Shantanu P, Gideon Rothschild, Demetris K Roumis, and Loren M Frank (2016). "Coordinated excitation and inhibition of prefrontal ensembles during awake hippocampal sharp-wave ripple events". *Neuron* 90:1, pp. 113–127.

Janowsky, Jeri S, Arthur P Shimamura, and Larry R Squire (1989). "Source memory impairment in patients with frontal lobe lesions". *Neuropsychologia* 27:8, pp. 1043–1056.

Jay, Thérèse M, Jacques Glowinski, and Anne-Marie Thierry (1989). "Selectivity of the hippocampal projection to the prelimbic area of the prefrontal cortex in the rat". *Brain research* 505:2, pp. 337-340.
Jay, Thérèse M and Menno P Witter (1991). "Distribution of hippocampal CA1 and subicular efferents in the prefrontal cortex of the rat studied by means of anterograde transport of Phaseolus vulgaris-leucoagglutinin". *Journal of Comparative Neurology* 313:4, pp. 574-586.

Jayachandran, Maanasa, Stephanie B Linley, Maximilian Schlecht, Stephen V Mahler, Robert P Vertes, and Timothy A Allen (2019). "Prefrontal pathways provide top-down control of memory for sequences of events". *Cell reports* 28:3, pp. 640-654.

Jensen, Kristopher T, Guillaume Hennequin, and Marcelo G Mattar (2023). "A recurrent network model of planning explains hippocampal replay and human behavior". *bioRxiv*, pp. 2023-01.

Jensen, Ole, John E Lisman, et al. (1996). "Novel lists of 7+/-2 known items can be reliably stored in an oscillatory short-term memory network: interaction with long-term memory". *Learn Mem* 3:2-3, pp. 257-263.

Jensen, Ole and John E Lisman (1996). "Theta/gamma networks with slow NMDA channels learn sequences and encode episodic memory: role of NMDA channels in recall." *Learning & Memory* 3:2-3, pp. 264-278.

- (2000). "Position reconstruction from an ensemble of hippocampal place cells: contribution of theta phase coding". *Journal of neurophysiology* 83:5, pp. 2602-2609.

- (2005). "Hippocampal sequence-encoding driven by a cortical multi-item working memory buffer". *Trends in neurosciences* 28:2, pp. 67-72.

Ji, Daoyun and Matthew A Wilson (2007). "Coordinated memory replay in the visual cortex and hippocampus during sleep". *Nature neuroscience* 10:1, pp. 100-107.

Jocham, Gerhard, Tilmann A Klein, and Markus Ullsperger (2011). "Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices". *Journal of Neuroscience* 31:5, pp. 1606-1613.

Jones, Matthew W and Matthew A Wilson (2005). "Theta rhythms coordinate hippocampal-prefrontal interactions in a spatial memory task". *PLoS biology* 3:12, e402.
Jung, Min W, Sidney I Wiener, and Bruce L McNaughton (1994). "Comparison of spatial firing characteristics of units in dorsal and ventral hippocampus of the rat". *Journal of Neuroscience* 14:12, pp. 7347-7356.

Jung, MW and BL McNaughton (1993). "Spatial selectivity of unit activity in the hippocampal granular layer". *Hippocampus* 3:2, pp. 165-182.

Kanerva, Pentti (1988). *Sparse distributed memory*. MIT press.

Kassab, Randa and Frédéric Alexandre (2018). "Pattern separation in the hippocampus: distinct circuits under different conditions". *Brain Structure and Function* 223:6, pp. 2785-2808.

Keinath, Alexandra T, Andrés Nieto-Posadas, Jennifer C Robinson, and Mark P Brandon (2020). "DG-CA3 circuitry mediates hippocampal representations of latent information". *Nature communications* 11:1, p. 3026.

Kennerley, Steven W, Timothy EJ Behrens, and Jonathan D Wallis (2011). "Double dissociation of value computations in orbitofrontal and anterior cingulate neurons". *Nature neuroscience* 14:12, pp. 1581-1589.

Kesner, Raymond P, Paul E Gilbert, and Lindsay A Barua (2002). "The role of the hippocampus in memory for the temporal order of a sequence of odors." *Behavioral neuroscience* 116:2, p. 286.

Ketz, Nicholas, Srinimisha G Morkonda, and Randall C O'Reilly (2013). "Theta coordinated error-driven learning in the hippocampus". *PLoS computational biology* 9:6, e1003067.

Killian, Nathaniel J, Michael J Jutras, and Elizabeth A Buffalo (2012). "A map of visual space in the primate entorhinal cortex". *Nature* 491:7426, pp. 761-764.

Kishimoto, Yasushi, Kazu Nakazawa, Susumu Tonegawa, Yutaka Kirino, and Masanobu Kano (2006). "Hippocampal CA3 NMDA receptors are crucial for adaptive timing of trace eyeblink conditioned response". *Journal of Neuroscience* 26:5, pp. 1562-1570.

Kjelstrup, Kirsten Brun, Trygve Solstad, Vegard Heimly Brun, Torkel Hafting, Stefan Leutgeb, Menno P Witter, Edvard I Moser, and May-Britt Moser (2008). "Finite scale of spatial representation in the hippocampus". *Science* 321:5885, pp. 140-143.
Koechlin, Etienne, Chrystele Ody, and Frédérique Kouneiher (2003). "The architecture of cognitive control in the human prefrontal cortex". *Science* 302:5648, pp. 1181–1185.

Komorowski, Robert W, Carolyn G Garcia, Alix Wilson, Shoai Hattori, Marc W Howard, and Howard Eichenbaum (2013). "Ventral hippocampal neurons are shaped by experience to represent behaviorally relevant contexts". *Journal of Neuroscience* 33:18, pp. 8079–8087.

Konkel, Alex and Neal J Cohen (2009). "Relational memory and the hippocampus: representations and methods". *Frontiers in neuroscience* 3, p. 679.

Kouneiher, Frédérique, Sylvain Charron, and Etienne Koechlin (2009). "Motivation and cognitive control in the human prefrontal cortex". *Nature neuroscience* 12:7, pp. 939–945.

Krotov, Dmitry (2023). "A new frontier for Hopfield networks". *Nature Reviews Physics*, pp. 1–2.

Krotov, Dmitry and John Hopfield (2020). "Large associative memory problem in neurobiology and machine learning". *arXiv preprint arXiv:2008.06996*.

Krotov, Dmitry and John J Hopfield (2016). "Dense associative memory for pattern recognition". *Advances in neural information processing systems* 29.

Kubota, Kisou and Hiroaki Niki (1971). "Prefrontal cortical unit activity and delayed alternation performance in monkeys." *Journal of neurophysiology* 34:3, pp. 337–347.

Kudrimoti, Hemant S, Carol A Barnes, and Bruce L McNaughton (1999). "Reactivation of hippocampal cell assemblies: effects of behavioral state, experience, and EEG dynamics". *Journal of Neuroscience* 19:10, pp. 4090–4101.

Kumaran, Dharshan, Demis Hassabis, and James L McClelland (2016). "What learning systems do intelligent agents need? Complementary learning systems theory updated". *Trends in cognitive sciences* 20:7, pp. 512–534.

Kumaran, Dharshan, Hans Ludwig Melo, and Emrah Duzel (2012). "The emergence and representation of knowledge about social and nonsocial hierarchies". *Neuron* 76:3, pp. 653–666.
Kunec, Steve, Michael E Hasselmo, and Nancy Kopell (2005). "Encoding and retrieval in the CA3 region of the hippocampus: a model of theta-phase separation". *Journal of neurophysiology* 94:1, pp. 70-82.

Lassalle, Jean-Michel, Thierry Bataille, and Hélène Halley (2000). "Reversible inactivation of the hippocampal mossy fiber synapses in mice impairs spatial learning, but neither consolidation nor memory retrieval, in the Morris navigation task". *Neurobiology of learning and memory* 73:3, pp. 243-257.

Lau, Hakwan C and Richard E Passingham (2007). "Unconscious activation of the cognitive control system in the human prefrontal cortex". *Journal of Neuroscience* 27:21, pp. 5805-5811.

Le, Hung, Thommen Karimpanal George, Majid Abdolshah, Truyen Tran, and Svetha Venkatesh (2021). "Model-based episodic memory induces dynamic hybrid controls". *Advances in Neural Information Processing Systems* 34, pp. 30313-30325.

Lee, Albert K and Matthew A Wilson (2002). "Memory of sequential experience in the hippocampus during slow wave sleep". *Neuron* 36:6, pp. 1183-1194.

Lengyel, Máté and Peter Dayan (2007). "Hippocampal contributions to control: the third way". *Advances in neural information processing systems* 20.

Lenz, Wilhelm (1920). "Beitrag zum Verständnis der magnetischen Erscheinungen in festen Körpern". *Z. Phys.* 21, pp. 613-615.

Levy, William B (1982). "Associative encoding at synapses". In: *Proceedings of the Fourth Annual Conference of Cognitive Science Society*, pp. 135-136.

- (1996). "A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal-like tasks". *Hippocampus* 6:6, pp. 579-590.

Li, Minghua, Jessica L Kinney, Yu-Qiu Jiang, Daniel K Lee, Qiwen Wu, Dae-hoon Lee, Wen-Cheng Xiong, and Qian Sun (2023). "Hypothalamic supra-mammillary nucleus selectively excites hippocampal CA3 interneurons to suppress CA3 pyramidal neuron activity". *Journal of Neuroscience* 43:25, pp. 4612-4624.
Lim, Seung-Lark, John P O'Doherty, and Antonio Rangel (2011). "The decision value computations in the vmPFC and striatum use a relative value code that is guided by visual attention". *Journal of Neuroscience* 31:37, pp. 13214-13223.

Lin, Long-Ji (1992). "Self-improving reactive agents based on reinforcement learning, planning and teaching". *Machine learning* 8, pp. 293-321.

Linnainmaa, Seppo (1976). "Taylor expansion of the accumulated rounding error". *BIT Numerical Mathematics* 16:2, pp. 146-160.

Lisman, John E (1999). "Relating hippocampal circuitry to function: recall of memory sequences by reciprocal dentate-CA3 interactions". *Neuron* 22:2, pp. 233-242.

Lisman, John E and Anthony A Grace (2005). "The hippocampal-VTA loop: controlling the entry of information into long-term memory". *Neuron* 46:5, pp. 703-713.

Lisman, John E and Marco AP Idiart (1995). "Storage of $7 \pm 2$ short-term memories in oscillatory subcycles". *Science* 267:5203, pp. 1512-1515.

Lisman, John E and Ole Jensen (2013). "The theta-gamma neural code". *Neuron* 77:6, pp. 1002-1016.

Lisman, John E and Nonna A Otmakhova (2001). "Storage, recall, and novelty detection of sequences by the hippocampus: elaborating on the SO-CRATIC model to account for normal and aberrant effects of dopamine". *Hippocampus* 11:5, pp. 551-568.

Lisman, John E, Lucia M Talamini, and Antonino Raffone (2005). "Recall of memory sequences by interaction of the dentate and CA3: a revised model of the phase precession". *Neural Networks* 18:9, pp. 1191-1201.

Long, Michael A and Michale S Fee (2008). "Using temperature to analyse temporal dynamics in the songbird motor pathway". *Nature* 456:7219, pp. 189-194.

Lopez-Rojas, Jeffrey and Michael R Kreutz (2016). "Mature granule cells of the dentate gyrus - Passive bystanders or principal performers in hippocampal function?" *Neuroscience & Biobehavioral Reviews* 64, pp. 167-174.
Lu, Qihong, Uri Hasson, and Kenneth A Norman (2022). "A neural network model of when to retrieve and encode episodic memories". *elife* 11, e74445.

MacDonald, Christopher J, Kyle Q Lepage, Uri T Eden, and Howard Eichenbaum (2011). "Hippocampal "time cells" bridge the gap in memory for discontiguous events". *Neuron* 71:4, pp. 737-749.

MacKay, Donald G, Deborah M Burke, and Rachel Stewart (1998). "HM's language production deficits: Implications for relations between memory, semantic binding, and the hippocampal system". *Journal of Memory and Language* 38:1, pp. 28-69.

Manns, Joseph R and Howard Eichenbaum (2006). "Evolution of declarative memory". *Hippocampus* 16:9, pp. 795-808.

Marr, D (1971). "Simple memory: a theory for archicortex". *Philosophical Transactions of the Royal Society of London. B, Biological Sciences* 262:841, pp. 23-81.

Marr, David and W Thomas Thach (1991). "A theory of cerebellar cortex". *From the Retina to the Neocortex: Selected Papers of David Marr*, pp. 11-50.

Mattar, Marcelo G and Nathaniel D Daw (2018). "Prioritized memory access explains planning and hippocampal replay". *Nature neuroscience* 21:11, pp. 1609-1617.

Maurer, Andrew P, Shea R VanRhoads, Gary R Sutherland, Peter Lipa, and Bruce L McNaughton (2005). "Self-motion and the origin of differential spatial scaling along the septo-temporal axis of the hippocampus". *Hippocampus* 15:7, pp. 841-852.

McClelland, James L (2013). "Incorporating rapid neocortical learning of new schema-consistent information into complementary learning systems theory." *Journal of Experimental Psychology: General* 142:4, p. 1190.

McClelland, James L, Bruce L McNaughton, and Randall C O'Reilly (1995). "Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory." *Psychological review* 102:3, p. 419.
McCloskey, Michael and Neal J Cohen (1989). "Catastrophic interference in connectionist networks: The sequential learning problem". In: *Psychology of learning and motivation*. Vol. 24. Elsevier, pp. 109–165.

McHugh, Thomas J, Matthew W Jones, Jennifer J Quinn, Nina Balthasar, Roberto Coppari, Joel K Elmquist, Bradford B Lowell, Michael S Fanselow, Matthew A Wilson, and Susumu Tonegawa (2007). "Dentate gyrus NMDA receptors mediate rapid pattern separation in the hippocampal network". *Science* 317:5834, pp. 94–99.

McNaughton, Bruce L, Francesco P Battaglia, Ole Jensen, Edvard I Moser, and May-Britt Moser (2006). "Path integration and the neural basis of the 'cognitive map'". *Nature Reviews Neuroscience* 7:8, pp. 663–678.

McNaughton, Bruce L and Richard GM Morris (1987). "Hippocampal synap-
tic enhancement and information storage within a distributed memory
system". *Trends in neurosciences* 10:10, pp. 408–415.

Mehta, Mayank R, Carol A Barnes, and Bruce L McNaughton (1997). "Experience-dependent, asymmetric expansion of hippocampal place fields". *Proceedings of the National Academy of Sciences* 94:16, pp. 8918–8921.

Melchior, Jan, Mehdi Bayati, Amir Azizi, Sen Cheng, and Laurenz Wiskott (2019). "A hippocampus model for online one-shot storage of pattern sequences". *arXiv preprint arXiv:1905.12937*.

Melo, Luckeciano C (2022). "Transformers are meta-reinforcement learners". In: *international conference on machine learning*. PMLR, pp. 15340–15359.

Menzel, Randolf (2023). "Navigation and dance communication in honeybees: a cognitive perspective". *Journal of Comparative Physiology A*, pp. 1–13.

Miller, Earl K and Jonathan D Cohen (2001). "An integrative theory of prefrontal cortex function". *Annual review of neuroscience* 24:1, pp. 167–202.

Miller, Michael H and J Orbach (1972). "Retention of spatial alternation following frontal lobe resections in stump-tailed macaques". *Neuropsychologia* 10:3, pp. 291–298.

Millidge, Beren, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz (2022). "Universal hopfield networks: A general frame-
work for single-shot associative memory models". In: *International Conference on Machine Learning*. PMLR, pp. 15561–15583.

Milner, Brenda (1962). "Les troubles de la memoire accompagnant des le-sions hippocampiques bilaterales". *Physiologie de l'hippocampe* 107, pp. 257-272.

Milner, Brenda, Suzanne Corkin, and H-L Teuber (1968). "Further analysis of the hippocampal amnesic syndrome: 14-year follow-up study of HM". *Neuropsychologia* 6:3, pp. 215–234.

Miyashita, Teiko, Stepan Kubik, Nahideh Haghighi, Oswald Steward, and John F Guzowski (2009). "Rapid activation of plasticity-associated gene transcription in hippocampal neurons provides a mechanism for encod-ing of one-trial experience". *Journal of Neuroscience* 29:4, pp. 898–906.

Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. (2015). "Human-level control through deep reinforcement learning". *nature* 518:7540, pp. 529–533.

Momennejad, Ida and Marc W Howard (2018). "Predicting the future with multi-scale successor representations". *BioRxiv*, p. 449470.

Moore, Andrew W and Christopher G Atkeson (1993). "Prioritized sweep-ing: Reinforcement learning with less data and less time". *Machine learn-ing* 13, pp. 103–130.

Moscovitch, Morris (1992). "Memory and working-with-memory: A com-ponent process model based on modules and central systems". *Journal of cognitive neuroscience* 4:3, pp. 257–267.

Moser, May-Britt, Edvard I Moser, Elma Forrest, Per Andersen, and RG40869 Morris (1995). "Spatial learning with a minislab in the dorsal hippocam-pus." *Proceedings of the National Academy of Sciences* 92:21, pp. 9697–9701.

Muller, Robert U and John L Kubie (1987). "The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells". *Journal of Neuroscience* 7:7, pp. 1951–1968.

Nádasdy, Zoltán, Hajime Hirase, András Czurkó, Jozsef Csicsvari, and György Buzsáki (1999). "Replay and time compression of recurring spike
sequences in the hippocampus". *Journal of Neuroscience* 19:21, pp. 9497-9507.

Nadel, Lynn and Morris Moscovitch (1997). "Memory consolidation, retrograde amnesia and the hippocampal complex". *Current opinion in neurobiology* 7:2, pp. 217-227.

Nadel, Lynn, A Samsonovich, Lee Ryan, and Morris Moscovitch (2000).
"Multiple trace theory of human memory: computational, neuroimaging,
and neuropsychological results". *Hippocampus* 10:4, pp. 352-368.

Nakamura, Nozomu H, Masaki Fukunaga, Keith T Akama, Tomoko Soga, Sonoko Ogawa, and Constantine Pavlides (2010). "Hippocampal cells encode places by forming small anatomical clusters". *Neuroscience* 166:3, pp. 994-1007.

Nakazawa, Kazu, Linus D Sun, Michael C Quirk, Laure Rondi-Reig, Matthew A Wilson, and Susumu Tonegawa (2003). "Hippocampal CA3 NMDA receptors are crucial for memory acquisition of one-time experience". *Neuron* 38:2, pp. 305-315.

Navawongse, Rapeechai and Howard Eichenbaum (2013). "Distinct pathways for rule-based retrieval and spatial mapping of memory representations in hippocampal neurons". *Journal of Neuroscience* 33:3, pp. 1002-1013.

Nayebi, Aran, Alexander Attinger, Malcolm Campbell, Kiah Hardcastle, Isabel Low, Caitlin S Mallory, Gabriel Mel, Ben Sorscher, Alex H Williams, Surya Ganguli, et al. (2021). "Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks". *Advances in Neural Information Processing Systems* 34, pp. 12167-12179.

Nee, Derek Evan and Joshua W Brown (2012). "Rostral-caudal gradients of abstraction revealed by multi-variate pattern analysis of working memory". *Neuroimage* 63:3, pp. 1285-1294.

Neher, Torsten, Sen Cheng, and Laurenz Wiskott (2015). "Memory storage fidelity in the hippocampal circuit: the role of subregions and input statistics". *PLoS computational biology* 11:5, e1004250.
Niki, Hiroaki and Masataka Watanabe (1979). "Prefrontal and cingulate unit activity during timing behavior in the monkey". *Brain research* 171:2, pp. 213-224.

Noonan, MP, RB Mars, and MFS Rushworth (2011). "Distinct roles of three frontal cortical areas in reward-guided behavior". *Journal of Neuroscience* 31:40, pp. 14399-14412.

O'Keefe, John and Jonathan Dostrovsky (1971). "The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat." *Brain research*.

O'Mara, Shane (2005). "The subiculum: what it does, what it might do, and what neuroanatomy has yet to tell us". *Journal of anatomy* 207:3, pp. 271-282.

O'Mara, Shane M, Sean Commins, Michael Anderson, and John Gigg (2001). "The subiculum: a review of form, physiology and function". *Progress in neurobiology* 64:2, pp. 129-155.

O'Neill, Pia-Kelsey, Joshua A Gordon, and Torfi Sigurdsson (2013). "Theta oscillations in the medial prefrontal cortex are modulated by spatial working memory and synchronize with the hippocampus through its ventral subregion". *Journal of Neuroscience* 33:35, pp. 14211-14224.

O'Reilly, Randall C and Michael J Frank (2006). "Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia". *Neural computation* 18:2, pp. 283-328.

O'Reilly, Randall C, Michael J Frank, Thomas E Hazy, and Brandon Watz (2007). "PVLV: the primary value and learned value Pavlovian learning algorithm." *Behavioral neuroscience* 121:1, p. 31.

O'Reilly, Randall C and James L McClelland (1994). "Hippocampal conjunctive encoding, storage, and recall: Avoiding a trade-off". *Hippocampus* 4:6, pp. 661-682.

Park, Seongmin A, Maryam Zolfaghar, Jacob Russin, Douglas S Miller, Randall C O'Reilly, and Erie D Boorman (2023). "The representational geometry of cognitive maps under dynamic cognitive control". *bioRxiv*, pp. 2023-02.
Park, Sungmo, Emily E Kramer, Valentina Mercaldo, Asim J Rashid, Nathan Insel, Paul W Frankland, and Sheena A Josselyn (2016). "Neuronal allocation to a hippocampal engram". *Neuropsychopharmacology* 41:13, pp. 2987-2993.

Pasquier, Daniel A and Fernando Reinoso-Suarez (1976). "Direct projections from hypothalamus to hippocampus in the rat demonstrated by retrograde transport of horseradish peroxidase". *Brain Research* 108:1, pp. 165-169.

Pavlides, Constantine and Jonathan Winson (1989). "Influences of hippocampal place cell firing in the awake state on the activity of these cells during subsequent sleep episodes". *Journal of neuroscience* 9:8, pp. 2907-2918.

Persuh, Marjan, Eric LaRock, and Jacob Berger (2018). "Working memory and consciousness: The current state of play". *Frontiers in human neuroscience* 12, p. 78.

Peters, Gregory J, Christopher N David, Madison D Marcus, and David M Smith (2013). "The medial prefrontal cortex is critical for memory retrieval and resolving interference". *Learning & memory* 20:4, pp. 201-209.

Peyrache, Adrien, Mehdi Khamassi, Karim Benchenane, Sidney I Wiener, and Francesco P Battaglia (2009). "Replay of rule-learning related neural patterns in the prefrontal cortex during sleep". *Nature neuroscience* 12:7, pp. 919-926.

Pezzulo, Giovanni, Caleb Kemere, and Matthijs AA Van Der Meer (2017). "Internally generated hippocampal sequences as a vantage point to probe future-oriented cognition". *Annals of the New York Academy of Sciences* 1396:1, pp. 144-165.

Pfeiffer, Brad E and David J Foster (2013). "Hippocampal place-cell sequences depict future paths to remembered goals". *Nature* 497:7447, pp. 74-79.

Pilly, Praveen, Michael Howard, and Rajan Bhattacharyya (2018). "Modeling contextual modulation of memory associations in the hippocampus". *Frontiers in Human Neuroscience* 12, p. 442.

Place, Ryan, Anja Farovik, Marco Brockmann, and Howard Eichenbaum (2016). "Bidirectional prefrontal-hippocampal interactions support context-guided memory". *Nature neuroscience* 19:8, pp. 992-994.
Posner, Michael I, Charles R Snyder, and R Solso (2004). "Attention and cognitive control". *Cognitive psychology: Key readings* 205, pp. 55–85.

Preston, Alison R and Howard Eichenbaum (2013). "Interplay of hippocampus and prefrontal cortex in memory". *Current biology* 23:17, R764–R773.

Preston, Alison R, Yael Shrager, Nicole M Dudukovic, and John DE Gabrieli (2004). "Hippocampal contribution to the novel use of relational information in declarative memory". *Hippocampus* 14:2, pp. 148–152.

Pritzel, Alexander, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell (2017). "Neural episodic control". In: *International conference on machine learning*. PMLR, pp. 2827–2836.

Proffitt, Dennis R (2006). "Embodied perception and the economy of action". *Perspectives on psychological science* 1:2, pp. 110–122.

Purves, Dale, Elizabeth M Brannon, Roberto Cabeza, Scott A Huettel, Kevin S LaBar, Michael L Platt, and Marty G Woldorff (2013). *Principles of cognitive neuroscience*. Vol. 83. 3. Sinauer Associates Sunderland.

Quirk, Gregory J, Robert U Muller, and John L Kubie (1990). "The firing of hippocampal place cells in the dark depends on the rat's recent experience". *Journal of Neuroscience* 10:6, pp. 2008–2017.

Rajasethupathy, Priyamvada, Sethuraman Sankaran, James H Marshel, Christina K Kim, Emily Ferenczi, Soo Yeun Lee, Andre Berndt, Charu Ramakrishnan, Anna Jaffe, Maisie Lo, et al. (2015). "Projections from neocortex mediate top-down control of memory retrieval". *Nature* 526:7575, pp. 653–659.

Ramsauer, Hubert, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. (2020). "Hopfield networks is all you need". *arXiv preprint arXiv:2008.02217*.

Reser, Jared Edward (2022). "A Cognitive Architecture for Machine Consciousness and Artificial Superintelligence: Updating Working Memory Iteratively". *arXiv preprint arXiv:2203.17255*.

Reynolds, Jeremy R, Randall C O'Reilly, Jonathan D Cohen, and Todd S Braver (2012). "The function and organization of lateral prefrontal cortex: a test of competing hypotheses". *PloS one* 7:2, e30284.
Ritter, Samuel, Jane Wang, Zeb Kurth-Nelson, Siddhant Jayakumar, Charles Blundell, Razvan Pascanu, and Matthew Botvinick (2018). "Been there, done that: Meta-learning with episodic recall". In: *International conference on machine learning*. PMLR, pp. 4354-4363.

Rosenbaum, R Shayna, Sandra Priselac, Stefan Köhler, Sandra E Black, Fuqiang Gao, Lynn Nadel, and Morris Moscovitch (2000). "Remote spatial memory in an amnesic person with extensive bilateral hippocampal lesions". *Nature neuroscience* 3:10, pp. 1044-1048.

Rumelhart, David E (1990). "Brain style computation: Learning and generalization". In: *An introduction to neural and electronic networks*, pp. 405-420.

Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams (1986). "Learning representations by back-propagating errors". *nature* 323:6088, pp. 533-536.

Rushworth, Matthew FS, Mark J Buckley, Timothy EJ Behrens, Mark E Walton, and David M Bannerman (2007). "Functional organization of the medial frontal cortex". *Current opinion in neurobiology* 17:2, pp. 220-227.

Rushworth, MFS, Mark E Walton, Steve W Kennerley, and DM Bannerman (2004). "Action sets and decisions in the medial frontal cortex". *Trends in cognitive sciences* 8:9, pp. 410-417.

Russin, Jacob, Randall C O'Reilly, and Yoshua Bengio (2020). "Deep learning needs a prefrontal cortex". *Work Bridging AI Cogn Sci* 107:603-616, p. 1.

Sabariego, Marta, Antonia Schönwald, Brittney L Boublil, David T Zimman, Siavash Ahmadi, Nailea Gonzalez, Christian Leibold, Robert E Clark, Jill K Leutgeb, and Stefan Leutgeb (2019). "Time cells in the hippocampus are neither dependent on medial entorhinal cortex inputs nor necessary for spatial working memory". *Neuron* 102:6, pp. 1235-1248.

Sakurai, Y (2002). "Coding of auditory temporal and pitch information by hippocampal individual cells and cell assemblies in the rat". *Neuroscience* 115:4, pp. 1153-1163.
Sankar, Remya (2022). "Dual pathway architecture underlying vocal learning in songbirds". Theses. Université de Bordeaux. URL: https://hal.science/tel-03910872.

Santos-Pata, Diogo, Adrián F Amil, Ivan Georgiev Raikov, César Rennó-Costa, Anna Mura, Ivan Soltesz, and Paul FMJ Verschure (2021). "Entorhinal mismatch: A model of self-supervised learning in the hippocampus". *Iscience* 24:4.

Sarrico, Marta, Kai Arulkumaran, Andrea Agostinelli, Pierre Richmond, and Anil Anthony Bharath (2019). "Sample-efficient reinforcement learning with maximum entropy mellowmax episodic control". *arXiv preprint arXiv:1911.09615*.

Schacter, Daniel L, Donna Rose Addis, and Randy L Buckner (2007). "Remembering the past to imagine the future: the prospective brain". *Nature reviews neuroscience* 8:9, pp. 657–661.

Schacter, Daniel L, Donna Rose Addis, Demis Hassabis, Victoria C Martin, R Nathan Spreng, and Karl K Szpunar (2012). "The future of memory: remembering, imagining, and the brain". *Neuron* 76:4, pp. 677–694.

Schaeffer, Rylan, Mikail Khona, and Ila Fiete (2022). "No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit". *Advances in Neural Information Processing Systems* 35, pp. 16052–16067.

Schapiro, Anna C, Lauren V Kustner, and Nicholas B Turk-Browne (2012). "Shaping of object representations in the human medial temporal lobe based on temporal regularities". *Current biology* 22:17, pp. 1622–1627.

Schapiro, Anna C, Nicholas B Turk-Browne, Matthew M Botvinick, and Kenneth A Norman (2017). "Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning". *Philosophical Transactions of the Royal Society B: Biological Sciences* 372:1711, p. 20160049.

Schapiro, Anna C, Nicholas B Turk-Browne, Kenneth A Norman, and Matthew M Botvinick (2016). "Statistical learning of temporal community structure in the hippocampus". *Hippocampus* 26:1, pp. 3–8.
Scharfman, Helen E (2007). "The CA3 "backprojection" to the dentate gyrus". *Progress in brain research* 163, pp. 627-637.

Schaul, Tom, John Quan, Ioannis Antonoglou, and David Silver (2015).
"Prioritized experience replay". *arXiv preprint arXiv:1511.05952*.

Schmidt-Hieber, Christoph, Peter Jonas, and Josef Bischofberger (2004).
"Enhanced synaptic plasticity in newly generated granule cells of the
adult hippocampus". *Nature* 429:6988, pp. 184-187.

Schnider, Armin and Radek Ptak (1999). "Spontaneous confabulators fail to suppress currently irrelevant memory traces". *Nature neuroscience* 2:7, pp. 677-681.

Schomburg, Erik W, Antonio Fernández-Ruiz, Kenji Mizuseki, Antal Berényi, Costas A Anastassiou, Christof Koch, and György Buzsáki (2014). "Theta phase segregation of input-specific gamma patterns in entorhinal-hippocampal networks". *Neuron* 84:2, pp. 470-485.

Schon, Karin, Randall E Newmark, Robert S Ross, and Chantal E Stern (2016).
"A working memory buffer in parahippocampal regions: evidence from
a load effect during the delay period". *Cerebral Cortex* 26:5, pp. 1965-1974.

Schrauwen, Benjamin, David Verstraeten, and Jan Van Campenhout (2007).
"An overview of reservoir computing: theory, applications and imple-
mentations". In: *Proceedings of the 15th european symposium on artificial
neural networks*. p. 471-482 2007, pp. 471-482.

Scoville, William Beecher and Brenda Milner (1957). "Loss of recent memory after bilateral hippocampal lesions". *Journal of neurology, neurosurgery, and psychiatry* 20:1, p. 11.

Seress, Laszlo (2010). *Hippocampus and seahorse cropped*. URL: https://commons.wikimedia.org/wiki/File:Hippocampus_and_seahorse_cropped.JPG.

Sesack, Susan R, Ariel Y Deutch, Robert H Roth, and Benjamin S Bunney
(1989). "Topographical organization of the efferent projections of the
medial prefrontal cortex in the rat: an anterograde tract-tracing study
with Phaseolus vulgaris leucoagglutinin". *Journal of Comparative Neurol-
ogy* 290:2, pp. 213-242.

Shimamura, Arthur P, Paul J Jurica, Jennifer A Mangels, Felick B Gersh-
berg, and Robert T Knight (1995). "Susceptibility to memory interfer-
ence effects following frontal lobe damage: Findings from tests of paired-associate learning". *Journal of cognitive neuroscience* 7:2, pp. 144–152.

Shors, Tracey J, George Miesegaes, Anna Beylin, Mingrui Zhao, Tracy Rydel, and Elizabeth Gould (2001). "Neurogenesis in the adult is involved in the formation of trace memories". *Nature* 410:6826, pp. 372–376.

Siapas, Athanassios G, Evgueniy V Lubenov, and Matthew A Wilson (2005). "Prefrontal phase locking to hippocampal theta oscillations". *Neuron* 46:1, pp. 141–151.

Skaggs, William E and Bruce L McNaughton (1996). "Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience". *Science* 271:5257, pp. 1870–1873.

Skaggs, William E, Bruce L McNaughton, Matthew A Wilson, and Carol A Barnes (1996). "Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences". *Hippocampus* 6:2, pp. 149–172.

Snow, Mallory A and Jeff Orchard (2022). "Biological softmax: Demonstrated in modern Hopfield networks". In: *Proceedings of the Annual Meeting of the Cognitive Science Society*. Vol. 44. 44.

Solstad, Trygve, Edvard I Moser, and Gaute T Einevoll (2006). "From grid cells to place cells: a mathematical model". *Hippocampus* 16:12, pp. 1026–1031.

Sorscher, Ben, Gabriel Mel, Surya Ganguli, and Samuel Ocko (2019). "A unified theory for the origin of grid cells through the lens of pattern formation". *Advances in neural information processing systems* 32.

Spellman, Timothy, Mattia Rigotti, Susanne E Ahmari, Stefano Fusi, Joseph A Gogos, and Joshua A Gordon (2015). "Hippocampal–prefrontal input supports spatial encoding in working memory". *Nature* 522:7556, pp. 309–314.

Spens, Eleanor and Neil Burgess (2023). "A generative model of memory construction and consolidation". *bioRxiv*, pp. 2023–01.

Spreng, R Nathan, Raymond A Mar, and Alice SN Kim (2009). "The common neural basis of autobiographical memory, prospection, navigation,
theory of mind, and the default mode: a quantitative meta-analysis". *Journal of cognitive neuroscience* 21:3, pp. 489–510.

Squire, Larry R (2009). "Memory and brain systems: 1969–2009". *Journal of Neuroscience* 29:41, pp. 12711–12716.

Squire, Larry R and Pablo Alvarez (1995). "Retrograde amnesia and memory consolidation: a neurobiological perspective". *Current opinion in neurobiology* 5:2, pp. 169–177.

Stachenfeld, Kimberly L, Matthew M Botvinick, and Samuel J Gershman (2017). "The hippocampus as a predictive map". *Nature neuroscience* 20:11, pp. 1643–1653.

Stensola, Hanne, Tor Stensola, Trygve Solstad, Kristian Frøland, May-Britt Moser, and Edvard I Moser (2012). "The entorhinal grid map is discretized". *Nature* 492:7427, pp. 72–78.

Stokes, Mark G (2015). ""Activity-silent" working memory in prefrontal cortex: a dynamic coding framework". *Trends in cognitive sciences* 19:7, pp. 394–405.

Strange, Bryan A, Andrew Duggins, William Penny, Raymond J Dolan, and Karl J Friston (2005). "Information theory, novelty and hippocampal responses: unpredicted or unpredictable?" *Neural Networks* 18:3, pp. 225–230.

Strange, Bryan A, Menno P Witter, Ed S Lein, and Edvard I Moser (2014). "Functional organization of the hippocampal longitudinal axis". *Nature Reviews Neuroscience* 15:10, pp. 655–669.

Sukhbaatar, Sainbayar, Jason Weston, Rob Fergus, et al. (2015). "End-to-end memory networks". *Advances in neural information processing systems* 28.

Sutton, Richard S and Andrew G Barto (2014). *Reinforcement learning: An introduction*. URL: https://inst.eecs.berkeley.edu/~cs188/sp20/assets/files/SuttonBartoIPRLBook2ndEd.pdf.

Szczepanski, Sara M and Robert T Knight (2014). "Insights into human behavior from lesions to the prefrontal cortex". *Neuron* 83:5, pp. 1002–1018.

Tahvildari, Babak, Angel A Alonso, and Charles W Bourque (2008). "Ionic basis of ON and OFF persistent activity in layer III lateral entorhinal cortical principal neurons". *Journal of neurophysiology* 99:4, pp. 2006–2011.
Tang, Wenbo, Justin D Shin, Loren M Frank, and Shantanu P Jadhav (2017). "Hippocampal-prefrontal reactivation during learning is stronger in awake compared with sleep states". *Journal of Neuroscience* 37:49, pp. 11789-11805.

Tayler, Kaycie K, Kazumasa Z Tanaka, Leon G Reijmers, and Brian J Wiltgen (2013). "Reactivation of neural ensembles during the retrieval of recent and remote memory". *Current Biology* 23:2, pp. 99-106.

Teyler, Timothy J and Pascal DiScenna (1986). "The hippocampal memory indexing theory." *Behavioral neuroscience* 100:2, p. 147.

Tolman, Edward C (1948). "Cognitive maps in rats and men." *Psychological review* 55:4, p. 189.

Toni, Nicolas, Diego A Laplagne, Chunmei Zhao, Gabriela Lombardi, Charles E Ribak, Fred H Gage, and Alejandro F Schinder (2008). "Neurons born in the adult dentate gyrus form functional synapses with target cells". *Nature neuroscience* 11:8, pp. 901-907.

Treves, Alessandro and Edmund T Rolls (1992). "Computational constraints suggest the need for two distinct input systems to the hippocampal CA3 network". *Hippocampus* 2:2, pp. 189-199.

- (1994). "Computational analysis of the role of the hippocampus in memory". *Hippocampus* 4:3, pp. 374-391.

Trinchero, Mariela F, Karina A Buttner, Jessica N Sulkes Cuevas, Silvio G Temprana, Paula A Fontanet, M Cristina Monzón-Salinas, Fernanda Ledda, Gustavo Paratcha, and Alejandro F Schinder (2017). "High plasticity of new granule cells in the aging hippocampus". *Cell reports* 21:5, pp. 1129-1139.

Trivers, Robert and Richard Dawkins (1976). *The selfish gene*.

Tse, Dorothy, Tomonori Takeuchi, Masaki Kakeyama, Yasushi Kajii, HI-royuki Okuno, Chiharu Tohyama, Haruhiko Bito, and Richard GM Mor-ris (2011). "Schema-dependent gene activation and memory encoding in neocortex". *Science* 333:6044, pp. 891-895.

Tulving, Endel (1972). "Episodic and Semantic Memory". *Organization of Memory*, p. 381.
Turk-Browne, Nicholas B, Brian J Scholl, Marvin M Chun, and Marcia K Johnson (2009). "Neural evidence of statistical learning: Efficient detection of visual regularities without awareness". *Journal of cognitive neuroscience* 21:10, pp. 1934–1945.

Tyulmankov, Danil, Ching Fang, Annapurna Vadaparty, and Guangyu Robert Yang (2021). "Biological learning in key-value memory networks". *Advances in Neural Information Processing Systems* 34, pp. 22247–22258.

Van Gaal, Simon, K Richard Ridderinkhof, Johannes J Fahrenfort, H Steven Scholte, and Victor AF Lamme (2008). "Frontal cortex mediates unconsciously triggered inhibitory control". *Journal of Neuroscience* 28:32, pp. 8053–8062.

Van Gaal, Simon, K Richard Ridderinkhof, H Steven Scholte, and Victor AF Lamme (2010). "Unconscious activation of the prefrontal no-go network". *Journal of neuroscience* 30:11, pp. 4143–4150.

Van Kesteren, Marlieke TR, Dirk J Ruiter, Guillén Fernández, and Richard N Henson (2012). "How schema and novelty augment memory formation". *Trends in neurosciences* 35:4, pp. 211–219.

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin (2017). "Attention is all you need". *Advances in neural information processing systems* 30.

Vertes, Robert P (2002). "Analysis of projections from the medial prefrontal cortex to the thalamus in the rat, with emphasis on nucleus reuniens". *Journal of comparative neurology* 442:2, pp. 163–187.

- (2006). "Interactions among the medial prefrontal cortex, hippocampus and midline thalamus in emotional and cognitive processing in the rat". *Neuroscience* 142:1, pp. 1–20.

Vertes, Robert P, Walter B Hoover, Klara Szigeti-Buck, and Csaba Leranth (2007). "Nucleus reuniens of the midline thalamus: link between the medial prefrontal cortex and the hippocampus". *Brain research bulletin* 71:6, pp. 601–609.

Vertes, Robert P and James Timothy McKenna (2000). "Collateral projections from the supramammillary nucleus to the medial septum and hippocampus". *Synapse* 38:3, pp. 281–293.
Viskontas, Indre V, Mary Pat McAndrews, and Morris Moscovitch (2002).
"Memory for famous people in patients with unilateral temporal lobe
epilepsy and excisions." *Neuropsychology* 16:4, p. 472.

Vyleta, Nicholas P, Carolina Borges-Merjane, and Peter Jonas (2016). "Plasticity-dependent, full detonation at hippocampal mossy fiber-CA3 pyramidal neuron synapses". *Elife* 5, e17977.

Wallenstein, Gene V and Michael E Hasselmo (1997). "GABAergic modulation of hippocampal population activity: sequence learning, place field development, and the phase precession effect". *Journal of neurophysiology* 78:1, pp. 393-408.

Wallenstein, Gene V, Michael E Hasselmo, and Howard Eichenbaum (1998).
"The hippocampus as an associator of discontinuous events". *Trends in neurosciences* 21:8, pp. 317-323.

Wang, Dong V and Satoshi Ikemoto (2016). "Coordinated interaction between hippocampal sharp-wave ripples and anterior cingulate unit activity". *Journal of Neuroscience* 36:41, pp. 10663-10672.

Wayne, Greg, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, et al. (2018). "Unsupervised predictive memory in a goal-directed agent". *arXiv preprint arXiv:1803.10760*.

Weinrich, M, SP Wise, and K-H Mauritz (1984). "A neurophysiological study of the premotor cortex in the rhesus monkey". *Brain* 107:2, pp. 385-414.

White, Lisa M (1996). *Temporal difference learning: eligibility traces and the successor representation for actions*. Citeseer.

Whittington, James CR, Timothy H Muller, Shirley Mark, Guifen Chen, Caswell Barry, Neil Burgess, and Timothy EJ Behrens (2020). "The Tolman-Eichenbaum machine: unifying space and relational memory through generalization in the hippocampal formation". *Cell* 183:5, pp. 1249-1263.

Whittington, James CR, Joseph Warren, and Timothy EJ Behrens (2021). "Relating transformers to models and neural representations of the hippocampal formation". *arXiv preprint arXiv:2112.04035*.
Wills, Tom J, Colin Lever, Francesca Cacucci, Neil Burgess, and John O'Keefe (2005). "Attractor dynamics in the hippocampal representation of the local environment". *Science* 308:5723, pp. 873-876.

Willshaw, DJ, Peter Dayan, and Richard GM Morris (2015). "Memory, modelling and Marr: a commentary on Marr (1971) "Simple memory: a theory of archicortex"". *Philosophical Transactions of the Royal Society B: Biological Sciences* 370:1666, p. 20140383.

Wilson, Matthew A and Bruce L McNaughton (1993). "Dynamics of the hippocampal ensemble code for space". *Science* 261:5124, pp. 1055-1058.

Witt, Jessica K (2011). "Action's effect on perception". *Current Directions in Psychological Science* 20:3, pp. 201-206.

Witt, Jessica K and Dennis R Proffitt (2005). "See the ball, hit the ball: Apparent ball size is correlated with batting average". *Psychological science* 16:12, pp. 937-938.

Witt, Jessica K, Dennis R Proffitt, and William Epstein (2010). "When and how are spatial perceptions scaled?" *Journal of Experimental Psychology: Human Perception and Performance* 36:5, p. 1153.

Witt, Jessica K and Mila Sugovic (2010). "Performance and ease influence perceived speed". *Perception* 39:10, pp. 1341-1353.

Witter, Menno P, Thanh P Doan, Bente Jacobsen, Eirik S Nilssen, and Shinya Ohara (2017). "Architecture of the entorhinal cortex a review of entorhinal anatomy in rodents with some comparative notes". *Frontiers in systems neuroscience* 11, p. 46.

Witter, Menno P, Floris G Wouterlood, Pieterke A Naber, and Theo Van Haeften (2000). "Anatomical organization of the parahippocampal-hippocampal network". *Annals of the New York Academy of Sciences* 911:1, pp. 1-24.

Wood, Emma R, Paul A Dudchenko, R Jonathan Robitsek, and Howard Eichenbaum (2000). "Hippocampal neurons encode information about different types of memory episodes occurring in the same location". *Neuron* 27:3, pp. 623-633.

Wouterlood, FG, HWM Steinbusch, PGM Luiten, and JGJM Bol (1987). "Projection from the prefrontal cortex to histaminergic cell groups in the posterior hypothalamic region of the rat. Anterograde tracing with
Phaseolus vulgaris leucoagglutinin combined with immunocytochem-
istry of histidine decarboxylase”. *Brain research* 406:1-2, pp. 330–336.

Wyble, Bradley P, Christiane Linster, and Michael E Hasselmo (2000). "Size of CA1-evoked synaptic potentials is related to theta rhythm phase in rat hippocampus". *Journal of Neurophysiology* 83:4, pp. 2138–2144.

Wyss, JM, LW Swanson, and WM Cowan (1979). "A study of subcortical afferents to the hippocampal formation in the rat". Neuroscience 4:4, pp. 463-476.

Xu, Wei and Thomas C Südhof (2013). "A neural circuit for memory speci-
ficity and generalization". *Science* 339:6125, pp. 1290–1295.

Yau, Suk-yu, Ang Li, Kwok-Fai So, et al. (2015). “Involvement of adult hippocampal neurogenesis in learning and forgetting”. *Neural plasticity* 2015.

Young, James J and Matthew L Shapiro (2011). "Dynamic coding of goal-
directed paths by orbital prefrontal cortex". *Journal of Neuroscience* 31:16,
pp. 5989–6000.

Zheng, Chenguang, Kevin Wood Bieri, Yi-Tse Hsiao, and Laura Lee Colgin (2016). “Spatial sequence coding differs during slow and fast gamma rhythms in the hippocampus”. *Neuron* 89:2, pp. 398–408.

Zheng, Yicong, Xiaonan L Liu, Satoru Nishiyama, Charan Ranganath, and Randall C O'Reilly (2022). "Correcting the hebbian mistake: Toward a fully error-driven hippocampus". *PLoS Computational Biology* 18:10, e1010589.

Zielinski, Mark C, Wenbo Tang, and Shantanu P Jadhav (2020). “The role of replay and theta sequences in mediating hippocampal-prefrontal interactions for memory and cognition”. *Hippocampus* 30:1, pp. 60–72.

Zlotnik, Gregorio and Aaron Vansintjan (2019). “Memory: An extended definition”. *Frontiers in psychology* 10, p. 2523.