<figure><img src="image_1.png" /></figure>

HAL
open science

Computational Modeling of the Interactions Between
Episodic Memory and Cognitive Control

Hugo Chateau-Laurent

<figure><img src="image_2.png" /></figure>

To cite this version:

Hugo Chateau-Laurent. Computational Modeling of the Interactions Between Episodic Memory and Cognitive Control. Computer Science [cs]. Université de Bordeaux, 2024. English. NNT: 2024BORD0019 . tel-04573235

HAL Id: tel-04573235

https://theses.hal.science/tel-04573235

Submitted on 13 May 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
université
de BORDEAUX

Inria

THÈSE PRÉSENTÉE

POUR OBTENIR LE GRADE DE

DOCTEUR DE
L'UNIVERSITÉ DE BORDEAUX

ÉCOLE DOCTORALE MATHÉMATIQUES ET INFORMATIQUE
SPÉCIALITÉ INFORMATIQUE

Par Hugo CHATEAU-LAURENT

MODÉLISATION COMPUTATIONNELLE DES
INTERACTIONS ENTRE MÉMOIRE
ÉPISODIQUE ET CONTRÔLE COGNITIF

COMPUTATIONAL MODELING OF THE INTERACTIONS BETWEEN
EPISODIC MEMORY AND COGNITIVE CONTROL

Sous la direction de : Frédéric ALEXANDRE

Soutenue le 8 février 2024

Membres du jury :

M. Frédéric ALEXANDRE
M. Mehdi KHAMASSI
M. Randall O'REILLY
Mme. Anna SCHAPIRO
M. Rufin VANRULLEN
Mme. Emmanuelle
ABISSET-CHAVANNE

Directeur de recherche
Directeur de recherche
Professeur
Professeure adjointe
Directeur de recherche
Professeure

Université de Bordeaux
Sorbonne Université
UC Davis
Université de Pennsylvania
CNRS
Arts et Métiers ParisTech

Directeur de thèse
Rapporteur
Rapporteur
Examinatrice
Examinateur
Examinatrice et
Présidente du Jury
<figure><img src="image_3.png" /></figure>
ACKNOWLEDGEMENTS

I am profoundly grateful to my supervisor, Frédéric Alexandre, whose un-
wavering support and guidance have been crucial in shaping my PhD jour-
ney. Finding a mentor whose philosophy aligns so closely with mine has
been a unique and invaluable experience.

I am also grateful to Dolton Fernandes for conducting preliminary exper-
iments on episodic control and introducing me to the universal Hopfield
networks paper. I want to express appreciation to Thomas Vitry for his on-
going efforts in fine-tuning the model for the emergence of prettier place
cells. I extend my sincere appreciation to Dmitry Krotov for his crucial as-
sistance in understanding modern Hopfield networks. Despite our limited
interactions on Twitter, his expertise has significantly enriched my research,
and I am grateful for his valuable insights. I want to express special grat-
itude to Simon Lecomte, who, in moments when he believed he was not
contributing, played a significant role in helping me understand the intri-
cacies of the hippocampus. I would like to express my sincere gratitude to
all my other colleagues and friends at the Mnemosyne team and Inria for
the wonderful working environment.

I want to extend my appreciation to my family and friends for their unconditional love and support. Their belief in me has been a driving force behind my perseverance. Special thanks to Fandi, Sigrid and Thomas for their patience, encouragement, and unwavering belief in me. Their support has been a constant source of strength. I would also like to acknowledge Xavier Hinaut for his guidance and perpetual flow of project ideas, akin to my own, which proved both frustrating when convergence was essential at the end and exhilarating during other times.

Lastly, my sincere apologies to the thousands of proto-brains I brought
into existence and promptly terminated in my simulations. May they rest
in digital peace! These experiments were carried out using the PlaFRIM
experimental testbed, supported by Inria, CNRS (LABRI and IMB), Uni-
versité de Bordeaux, Bordeaux INP and Conseil Régional d'Aquitaine (see
https://www.plafrim.fr).
Computational Modeling of the Interactions Between
Episodic Memory and Cognitive Control

**Abstract:** Episodic memory is often illustrated with the madeleine de Proust ex-
cerpt as the ability to re-experience a situation from the past following the per-
ception of a stimulus. This simplistic scenario should not lead into thinking that
memory works in isolation from other cognitive functions. On the contrary, mem-
ory operations treat highly processed information and are themselves modulated
by executive functions in order to inform decision-making. This complex inter-
play can give rise to higher-level functions such as the ability to imagine potential
future sequences of events by combining contextually relevant memories. How
the brain implements this construction system is still largely a mystery. The ob-
jective of this thesis is to employ cognitive computational modeling methods to
better understand the interactions between episodic memory, which is supported
by the hippocampus, and cognitive control, which mainly involves the prefrontal
cortex. It provides elements as to how episodic memory can help an agent to
act. It is shown that neural episodic control, a fast and powerful method for re-
inforcement learning, is in fact mathematically close to the traditional Hopfield
network, a model of associative memory that has greatly influenced the under-
standing of the hippocampus. Neural episodic control indeed fits within the uni-
versal Hopfield network framework, and it is demonstrated that it can be used
to store and recall information, and that other kinds of Hopfield networks can be
used for reinforcement learning. The question of how executive functions can con-
trol episodic memory operations is also tackled. A hippocampus-inspired network
is constructed with as little assumption as possible and modulated with contex-
tual information. The evaluation of performance according to the level at which
contextual information is sent provides design principles for controlled episodic
memory. Finally, a new biologically inspired model of one-shot sequence learn-
ing in the hippocampus is proposed. The model performs very well on multiple
datasets while reproducing biological observations. It ascribes a new role to the
recurrent collaterals of area CA3, that is to disambiguate overlapping sequences
by making retrospective splitter cells emerge. Implications for theories of the hip-
 pocampus are discussed and novel experimental predictions are derived.

Research unit

Inria centre at the University of Bordeaux, 200 Avenue de la Vielle Tour,
33405 Talence, France.
# Modélisation Computationnelle des Interactions Entre Mémoire Épisodique et Contrôle Cognitif

**Résumé** : La mémoire épisodique est souvent illustrée par la madeleine de Proust comme la capacité à revivre une situation du passé suite à la perception d'un stigmulus. Ce scénario simpliste ne doit pas mener à penser que la mémoire opère en isolation des autres fonctions cognitives. Au contraire, la mémoire traite des informations hautement transformées et est elle-même modulée par les fonctions exécutives pour informer la prise de décision. Ces interactions complexes donnent lieu à des fonctions cognitives supérieures comme la capacité à imaginer de futures séquences d'événements potentielles en combinant des souvenirs pertinents dans le contexte. Comment le cerveau implémente ce système de construction reste un mystère. L'objectif de cette thèse est donc d'employer des méthodes de modélisation cognitive afin de mieux comprendre les interactions entre mémoire épisodique reposant principalement sur l'hippocampe et contrôle cognitif impliquant majoritairement le cortex préfrontal. Elle propose d'abord des éléments de réponse quant au rôle de la mémoire épisodique dans la sélection de l'action. Il est montré que le contrôle épisodique neuronal, une méthode puissante et rapide d'apprentissage par renforcement, est en fait mathématiquement proche du traditionnel réseau de Hopfield, un modèle de mémoire associative ayant grandement influencé la compréhension de l'hippocampe. Le contrôle épisodique neuronal peut en effet s'inscrire dans le cadre du réseau de Hopfield universel, il est donc montré qu'il peut être utilisé pour stocker et rappeler de l'information et que d'autres types de réseaux de Hopfield peuvent être utilisés pour l'apprentissage par renforcement. La question de comment les fonctions exécutives contrôlent la mémoire épisodique est aussi posée. Un réseau inspiré de l'hippocampe est créé avec le moins d'hypothèses possible et modulé avec de l'information contextuelle. L'évaluation des performances selon le niveau auquel le contexte est envoyé propose des principes de conception de mémoire épisodique contrôlée. Enfin, un nouveau modèle bio-inspiré de l'apprentissage en un coup de séquences dans l'hippocampe est proposé. Le modèle fonctionne bien avec plusieurs jeux de données tout en reproduisant des observations biologiques. Il attribue un nouveau rôle aux connexions récurrentes de la région CA3, qui est celui de distinguer les séquences se chevauchant en faisant émerger des cellules de séparation rétrospective. Les implications pour les théories de l'hippocampe sont discutées et de nouvelles prédictions expérimentales sont dérivées.

Unité de recherche
Centre Inria de l'Université de Bordeaux, 200 Avenue de la Vielle Tour,
33405 Talence, France.
<figure><img src="image_4.png" /></figure>
# CONTENTS

<table><tr><td>1</td><td>INTRODUCTION GÉNÉRALE</td><td>1</td></tr><tr><td>2</td><td>GENERAL INTRODUCTION</td><td>5</td></tr><tr><td>I</td><td>MEMORY & CONTROL</td><td>9</td></tr><tr><td>3</td><td>COGNITION & MEMORY</td><td>11</td></tr><tr><td>3.1</td><td>Associative and Nonassociative Memory</td><td>13</td></tr><tr><td>3.1.1</td><td>Autoassociation and Heteroassociation</td><td>13</td></tr><tr><td>3.1.2</td><td>Associative Space and Forms</td><td>14</td></tr><tr><td>3.2</td><td>Implicit and Explicit Processes</td><td>17</td></tr><tr><td>3.3</td><td>Semantic and Episodic Memory</td><td>18</td></tr><tr><td>3.3.1</td><td>Complementary Learning Systems Theory</td><td>19</td></tr><tr><td>3.3.2</td><td>Implicit Episodic Memory</td><td>21</td></tr><tr><td>3.4</td><td>Short-Term and Long-Term Memory</td><td>22</td></tr><tr><td>3.4.1</td><td>Standard Consolidation Theory</td><td>24</td></tr><tr><td>3.4.2</td><td>Transformative Accounts of Consolidation</td><td>25</td></tr><tr><td>3.5</td><td>Cognitive Control</td><td>26</td></tr><tr><td>3.5.1</td><td>The Need for Cognitive Control</td><td>26</td></tr><tr><td>3.5.2</td><td>The Many Forms of Control</td><td>27</td></tr><tr><td>3.5.3</td><td>Implicit Cognitive Control</td><td>28</td></tr><tr><td>3.6</td><td>Metacognition and Metamemory</td><td>29</td></tr><tr><td>4</td><td>EPISODIC MEMORY</td><td>31</td></tr><tr><td>4.1</td><td>Introduction</td><td>31</td></tr></table>
Contents

<table><tr><td>4.2</td><td>Anatomy</td><td>33</td></tr><tr><td>4.2.1</td><td>Entorhinal Cortex</td><td>34</td></tr><tr><td>4.2.2</td><td>Dentate Gyrus</td><td>36</td></tr><tr><td>4.2.3</td><td>CA3</td><td>37</td></tr><tr><td>4.2.4</td><td>CA1</td><td>38</td></tr></table>

<table><tr><td>4.3</td><td>Representation: the Hippocampus as a Cognitive Map</td><td>38</td></tr><tr><td>4.3.1</td><td>Grid Cells</td><td>38</td></tr><tr><td>4.3.2</td><td>Place Cells</td><td>40</td></tr></table>

<table><tr><td>4.4</td><td>Dynamics: Oscillations as Memory Operations</td><td>43</td></tr><tr><td>4.4.1</td><td>The Entorhinal Buffer</td><td>43</td></tr><tr><td>4.4.2</td><td>Theta-Gamma Code</td><td>46</td></tr><tr><td>4.4.3</td><td>Sharp Wave Ripples</td><td>50</td></tr></table>

<table><tr><td>4.5</td><td>Computation: The Standard Framework, Limits and Alternatives</td><td>52</td></tr><tr><td>4.5.1</td><td>Main Principles</td><td>53</td></tr><tr><td>4.5.2</td><td>Hopfield Network</td><td>54</td></tr><tr><td>4.5.3</td><td>CA3 as a Temporal Associator</td><td>56</td></tr><tr><td>4.5.4</td><td>Evidence Against the Standard Framework</td><td>57</td></tr><tr><td>4.5.5</td><td>Alternative Theories</td><td>60</td></tr></table>

<table><tr><td>4.6</td><td>Discussion</td><td>68</td></tr></table>

<table><tr><td>5</td><td>EPISODIC CONTROL AND CONTROLLED EPISODIC MEMORY</td><td>69</td></tr><tr><td>5.1</td><td>Introduction</td><td>70</td></tr><tr><td>5.2</td><td>Anatomy</td><td>71</td></tr><tr><td>5.3</td><td>Contributions of the Prefrontal Cortex to Memory</td><td>75</td></tr><tr><td>5.4</td><td>Representation: the Prefrontal-Hippocampal System as a Contextual Memory Map</td><td>75</td></tr><tr><td>5.5</td><td>Dynamics: Oscillations as Memory Interactions</td><td>77</td></tr><tr><td>5.6</td><td>Computational Mechanisms and Models</td><td>80</td></tr><tr><td>5.6.1</td><td>Experience Replay</td><td>80</td></tr><tr><td>5.6.2</td><td>Episodic Memory as a Third Way of Control</td><td>81</td></tr><tr><td>5.6.3</td><td>Controlled Episodic Memory</td><td>85</td></tr><tr><td>5.7</td><td>Discussion</td><td>87</td></tr></table>
Contents

<table><tr><td>8.4 Methods</td><td>140</td></tr><tr><td>8.4.1 Datasets</td><td>140</td></tr><tr><td>8.4.2 Hyperparameter Optimization</td><td>141</td></tr><tr><td>8.5 Experiments</td><td>141</td></tr><tr><td>8.5.1 One-shot Sequence Learning</td><td>141</td></tr><tr><td>8.5.2 Place Cells and Remapping</td><td>143</td></tr><tr><td>8.5.3 Disambiguation of Overlapping Sequences</td><td>154</td></tr><tr><td>8.6 Discussion</td><td>156</td></tr><tr><td>9 GENERAL DISCUSSION</td><td>163</td></tr><tr><td>BIBLIOGRAPHY</td><td>173</td></tr></table>
# 1 INTRODUCTION GÉNÉRALE

> Et tout d'un coup le souvenir m'est apparu. Ce goût c'était celui du petit morceau de madeleine que le dimanche matin à Combray (parce que ce jour-là je ne sortais pas avant l'heure de la messe), quand j'allais lui dire bonjour dans sa chambre, ma tante Léonie m'offrait après l'avoir trempé dans son infusion de thé ou de tilleul.

> — Marcel Proust, Du côté de chez Swann (1913)

> Comment la simple saveur d'une madeleine peut-elle nous transporter à travers le temps et l'espace, nous permettant de revivre un moment spécifique de notre passé ? Qu'est-ce qui fait que cet événement particulier se distingue de la myriade d'autres ? En quoi ce souvenir est-il censé guider notre comportement ? Ce sont les questions auxquelles cette thèse vise à répondre.

> La mémoire est une composante essentielle de notre identité. Nos expériences façonnent nos représentations et guident nos actions. Se souvenir d'une situation précise fait appel à une forme de mémoire spécifique : la mémoire épisodique. C'est la forme à laquelle nous pensons spontanément car elle fait partie de la mémoire explicite, en ce sens qu'elle a un accès privilégié à la conscience.

> Un élément crucial de la mémoire épisodique est sa capacité à apprendre très rapidement des informations qui n'ont parfois été perçues qu'une fois. Pensez au jour où vous avez eu votre baccalauréat. Cela ne s'est produit qu'une fois et pourtant le souvenir peut encore être présent. C'est un défi pour les modèles d'apprentissage automatique qui s'appuient souvent sur des exemples répétés pour ajuster méticuleusement les paramètres.
Avec ces méthodes, un élément vu une fois a très peu d'impact sur les paramètres et sera presque certainement oublié. Lorsque les modèles sont ajustés plus rapidement, les nouvelles informations écrasent les souvenirs existants, un problème appelé oubli catastrophique. Cela se produit principalement parce que les modèles d'apprentissage automatique sont conçus pour généraliser leurs connaissances afin de classer, estimer et agir dans de nouvelles situations. Ce n'est pas le cas de la mémoire épisodique qui est distincte d'autres fonctions telles que la mémoire sémantique. La généralisation est le rôle de la mémoire sémantique tandis que la mémoire épisodique peut intégrer rapidement de nouvelles informations en les stockant séparément pour éviter l'interférence avec les anciennes informations (McClelland et al. 1995).

Un autre aspect important de la mémoire épisodique est sa capacité à retenir l'ordre des événements, ce qui est difficile dans le cas de séquences qui se chevauchent. Considérons le scénario suivant. Lorsque vous arrivez sur votre lieu de travail, vous prenez l'ascenseur pour accéder à votre bureau et, à l'heure du déjeuner, vous prenez le même ascenseur pour vous rendre à la cafétéria. Il est utile de se souvenir de l'itinéraire menant à la cafétéria lorsque l'on prend l'ascenseur depuis son bureau. En revanche, lorsque vous arrivez au travail depuis l'extérieur, il est moins utile de s'en souvenir. L'étude de la navigation spatiale est liée à l'étude de la mémoire épisodique, car les représentations spatiales sous-tendent l'encodage des épisodes dans la région de l'hippocampe du cerveau. Certains neurones de cette région, appelés cellules de lieu, codent explicitement des variables spatiales. Dans notre exemple, l'ascenseur est un passage commun vers différentes destinations, ce qui pose problème pour distinguer les souvenirs pertinents. Certaines cellules de lieu représentent en fait les positions spatiales différemment selon l'endroit d'où nous venons (cellules séparatrices rétrospectives) ou l'endroit où nous avons l'intention d'aller (cellules séparatrices prospectives), comme l'explique Duvelle et al. (2023).

La mémoire épisodique nous joue parfois des tours. Nous pouvons nous rappeler malgré nous d'un souvenir intrusif désagréable, oublier des informations utiles ou nous souvenir d'une version déformée de la réal-
ité. Ces effets sont exacerbés dans des pathologies telles que la maladie d'Alzheimer, le stress post-traumatique ou les troubles obsessionnels compulsifs. Le cerveau sain évite cela grâce aux interactions entre la mémoire épisodique et des systèmes qui vérifient la véracité des informations rappelées et contrôlent le moment et le contenu de la récupération en mémoire. Ces systèmes sous-tendent les fonctions exécutives, en ce sens qu'ils contrôlent d'autres fonctions cognitives pour exécuter des plans comportementaux. Le cortex préfrontal en est le principal acteur. Il est soupçonné de provoquer la division prospective des cellules de lieu mentionnée ci-dessus afin d'orienter la recherche de mémoire vers des objectifs. En retour, il utilise les informations envoyées par la mémoire épisodique pour guider le comportement. C'est ainsi, par exemple, que nous pouvons choisir d'acheter un disque d'un artiste dont nous avons apprécié la musique par le passé. Des fonctions encore plus complexes naissent de l'interaction entre la mémoire et les fonctions exécutives. C'est le cas, par exemple, de l'imagination de situations futures, qui semble faire intervenir à la fois les structures impliquées dans la mémoire du passé et les régions dédiées au contrôle cognitif (Schacter, Addis, and Buckner 2007). Cela suggère que l'imagination anticipe le futur en recombinant des souvenirs.

Les progrès récents de l'intelligence artificielle générative ont eu un impact sans précédent sur la société, avec l'utilisation emblématique des transformateurs dans ChatGPT (Vaswani et al. 2017). Comparées à la vitesse d'apprentissage de notre mémoire épisodique, ces méthodes nécessitent des quantités astronomiques de données. Il est intéressant de noter que certains aspects des transformateurs sont similaires à des éléments de modèles de la mémoire épisodique (Ramsauer et al. 2020; Whittington, Warren, et al. 2021). Ce parallèle mérite d'être approfondi pour comprendre leurs besoins divergents en matière de données. En outre, les transformateurs manquent de contrôle cognitif, ce qui les rend susceptibles de produire des informations inexactes.

L'objectif principal de cette thèse est de caractériser les interactions entre la mémoire épisodique et le contrôle cognitif avant de poursuivre le parallèle avec les transformateurs pour développer des modèles computation-
nels biologiquement plausibles de ces interactions. L'objectif est double : d'une part, participer à la compréhension de ces fonctions dont l'altération sous-tend des pathologies sévères, et d'autre part, le développement de meilleurs modèles d'apprentissage automatique. La thèse est structurée comme suit. Elle commence par une revue détaillée de la littérature, fournissant une vue d'ensemble de l'état de l'art sur le sujet. Le chapitre 3 présente la mémoire épisodique et le contrôle cognitif en relation avec d'autres fonctions cognitives. Le chapitre 4 décrit l'anatomie, les représentations, la dynamique et les modèles informatiques de la mémoire épisodique, en mettant l'accent sur l'apprentissage d'une séquence vécue une seule fois. Enfin, le chapitre 5 présente les aspects anatomiques, représentations-nels, dynamiques et informatiques des interactions entre le préfrontal et l'hippocampe, qui permettent un comportement guidé par la mémoire et une mémoire orientée vers un but. L'analyse est suivie de trois chapitres de contribution. Dans le chapitre 6, une équivalence mathématique est trouvée entre les modèles de mémoire épisodique et un module d'un algorithme d'apprentissage profond pour la sélection des actions, renforçant le lien entre cet algorithme et les interactions préfrontal-hippocampique. Il est ensuite démontré que des éléments de l'algorithme améliorent les performances des modèles de mémoire épisodique de pointe dans certaines tâches, et que des éléments des modèles de mémoire épisodique pourraient à leur tour améliorer l'algorithme d'apprentissage profond. Le chapitre 7 explore la modulation contextuelle des modèles de mémoire épisodique. Les réseaux neuronaux émergents comprennent des neurones modulés par le contexte, à l'instar des cellules séparatrices. Les résultats remettent également en question le cadre standard de la mémoire épisodique dans l'hippocampe. À la suite de ces études exploratoires, un nouveau modèle d'apprentissage de séquences en une seule fois dans l'hippocampe est présenté au chapitre 8. Ce modèle surpasse les autres modèles testés et reproduit divers résultats biologiques. Enfin, les contributions sont résumées et discutées au chapitre 9.
# 2 GENERAL INTRODUCTION

> And suddenly the memory returns. The taste was
> that of the little crumb of madeleine which on
> Sunday mornings at Combray (because on those
> mornings I did not go out before church time),
> when I went to say good day to her in her bedroom,
> my aunt Leonie used to give me, dipping it first in
> her own cup of real or of lime-flower tea.

> — Marcel Proust, Swann's Way (1913)

> How can the simple flavor of a madeleine transport us through time and space,
> allowing us to vividly revisit a specific moment from our past? What makes this
> particular event stand out amidst the myriad of others? In what way is this mem-
> ory intended to guide behavior? These are the questions this thesis aims to
> address.

> Memory is a key component of our identity. Experiences shape our rep-
> resentations and guide our actions. Remembering a specific experience
> calls on a very specific form of memory: episodic memory. This is the
> memory we think about spontaneously as it is part of explicit memory, in
> that it has privileged access to consciousness.

> A crucial element of episodic memory is its capacity to learn informa-
> tion very quickly, having sometimes only been perceived once. Think back
> to when you graduated from high school. It only happened once and yet
> the memory can still be vivid. This is a challenge for machine learning
> models, which often rely on millions of repeated examples to meticulously
> adjust parameters. With these methods, an element seen once has very lit-
> tle impact on the parameters and will almost certainly be forgotten. When
models are adjusted more strongly, new information overwrites existing memories, a problem called catastrophic forgetting. This happens mainly because machine learning models are designed to generalize their knowledge to classify, estimate and act in new situations. This is not the case with episodic memory, which is distinct from other functions such as semantic memory. Generalization is the role of the complementary semantic memory, while episodic memory can rapidly integrate new information, storing it separately to avoid interference with old information (McClelland et al. 1995).

Another important aspect of episodic memory is its ability to retain the order of events, which is challenging in the case of overlapping sequences of information. Consider the following scenario. Upon reaching your workplace, you use the elevator to access your office, and during lunchtime, you take the same elevator to reach the cafeteria. It is beneficial to remember the route to the cafeteria when entering the elevator from your office. However, when arriving at work from outside, there is no need to remember this. The study of spatial navigation is closely linked to the study of episodic memory, as spatial representations underpin the encoding of episodes in the hippocampus of the brain. Some neurons in this region, called place cells, explicitly encode spatial variables. In the example scenario, the elevator serves as a common pathway to different destinations, posing a challenge in distinguishing relevant memories. Certain place cells in fact represent spatial positions differently depending on where we come from (retrospective splitter cells) or where we intend to go (prospective splitter cells), as reviewed by Duvelle et al. (2023).

Sometimes episodic memory plays tricks on us. We may recall an unpleasant intrusive memory in spite of ourselves, forget useful information, or remember a distorted version of reality. These effects are exacerbated in conditions such as Alzheimer's disease, post-traumatic stress disorder or obsessive-compulsive disorder. The healthy brain prevents this through interactions between episodic memory and systems that verify the veracity of recalled information and control the timing and content of memory retrieval. These systems underlie executive functions, in that they con-
trol other cognitive functions to execute behavioral plans. The prefrontal cortex is the principal cognitive control region, and is suspected to cause the prospective splitting of place cells mentioned above to direct memory search according to goals. In return, it uses the information sent by episodic memory to guide behavior. This is how, for example, we may choose to buy a record by an artist whose music we have enjoyed in the past. Even more complex functions arise from the interaction of memory and executive functions. This is the case, for example, of imagining future situations, which seems to involve both the structures involved in remembering the past and regions dedicated to cognitive control (Schacter, Addis, and Buckner 2007). This suggests that imagination is a future-oriented construction process based on the recombination of past memories.

Recent advances in generative artificial intelligence have had an unprecedented impact on society, with the emblematic use of transformers in ChatGPT (Vaswani et al. 2017). Compared to the learning speed of our episodic memory, these methods require astronomical amounts of data before they can function properly. Interestingly, some aspects of transformers are similar to elements of episodic memory models (Ramsauer et al. 2020; Whittington, Warren, et al. 2021). This parallel deserves to be explored in greater depth to understand their divergent needs for data. In addition, transformers lack cognitive control, which makes them susceptible to hallucinations, that is to produce inaccurate information.

The main goal of this thesis is to characterize the interactions between episodic memory and cognitive control before extending on the parallel with transformers to develop efficient biologically plausible computational models of these interactions. The objective are twofold: first, to participate in the understanding of these functions whose impairment underlies severe pathologies, and second, the development of better machine learning models. The thesis is structured as follows. It begins with a comprehensive review of the literature, providing an overview of the state of the art on the subject. Chapter 3 presents episodic memory and cognitive control in relation to other cognitive functions. Chapter 4 describes the anatomy, representations, dynamics and computational models of episodic memory,
with a particular focus on one-shot sequence learning. Finally, Chapter 5 introduces the anatomical, representational, dynamical and computational aspects of prefrontal-hippocampal interactions enabling memory-driven behavior and goal-directed memory. The review is followed by three contribution chapters. In Chapter 6, a mathematical equivalence is found between episodic memory models and a module of a deep learning algorithm for action selection, strengthening the link between this algorithm and prefrontal-hippocampal interactions. It is then shown that elements of the algorithm improve the performance of state of the art episodic memory models in some tasks, and that elements of episodic memory models could in turn improve the deep learning algorithm. Chapter 7 explores the contextual modulation of episodic memory models. The emergent neural networks comprise neurons modulated by context similar to splitter cells. The results also question the standard framework of episodic memory in the hippocampus. Following these exploratory studies, a novel model of one-shot sequence learning in the hippocampus is introduced in Chapter 8. It outperforms other models tested and reproduces various biological findings. Finally, the contributions are summarized and discussed in Chapter 9.
# PART I

## MEMORY & CONTROL
<figure><img src="image_5.png" /></figure>
# 3 COGNITION & MEMORY

<figure><img src="image_6.png" /><figcaption>*An agent with cognition and memory*</figcaption></figure>

*Generative AI*

Following fundamental laws of physics and the contingency of events, time naturally leaves a trace. Humans grow out of embryos, become adults, live their life and eventually die, but these events cannot happen in a different order (except perhaps for Benjamin Button). Humans and other animals can leverage information left by a chain of events to make decisions. They can for example hunt by following a track to a prey, as prints reveal its path. Time can also be detrimental. No matter how helpful they can be,
traces of time are sometimes not sufficient to inform the agent as to what to do. Information can be lost when subsequent events further modify the environment. On a more radical note, agents must “fight” against the effect of time for self-preservation. For these reasons, organisms have developed strategies for actively changing their environment to their own benefit. Organisms maintain homeostasis, they build shelters and most importantly to our concern they form memories.

In cognitive science, memory has indeed become an umbrella term refer-
ring to the general ability to learn: encode, store and retrieve information
(Squire 2009). Forming a memory is the action of modifying a substratum
 encodings) in such a way that the modification persists in time (storage)
with the aim of extracting information from that modification (retrieval)
and benefit from it to act in the future. As a result of being general, Zlotnik
and Vansintjan (2019) described how this definition extends the traditional
boundaries of memory to processes outside the brain and mind. For exam-
ple, it can be argued that immunological processes such as the triggering
of a cascade of reactions as a result of the recognition of a previously iden-
tified threat can be interpreted as memory processes. Furthermore, books
and hard drives are devices that are typically used to store information out-
side the body. As a consequence, theories of the extended mind (Francisco et
al. 1991; Clark and Chalmers 1998; Clark 2008) have been proposed to account
for dynamic interactions between the mind, body and environment.

This thesis is largely confined to memory processes operating within the
brain (while being compatible with theories of extended cognition) but this
introduction is meant to emphasize how general the notion of memory
has become. Cognitive scientists have thus tried to refine it over the last
decades by proposing criteria for classifying memory systems and other
cognitive functions. Below are detailed some of these criteria in order to
better lay down the specifics of episodic memory and cognitive control,
which underlie higher-level cognition and are the main focus of this thesis.
**3.1 ASSOCIATIVE AND NONASSOCIATIVE MEMORY**

A common criterion that is used to classify memory forms is whether the
information being stored is an association between elements. Most memo-
ries formed are associative, like when you learn that 2+2 is associated with
the result 4, when you learn the association between food and pleasure (or
disgust) or when you know which song comes next while listening to an
album, to cite a few examples. Canonical forms of associative learning are
classical and operant conditioning in which a first stimulus is paired to a
second stimulus and a consequence respectively. All memories, however,
are not associative. Habituation and sensitization are common forms of
nonassociative learning. Habituation refers to the progressive attenuation
of a response to a stimulus after repetitive presentation without necessar-
ily learning the association between stimulus and outcome (e.g. lack of
punishment). A well known instance of habituation is the desensitization
of hunting dogs to loud sounds. While most dogs innately blench when
hearing loud stimuli, hunting dogs learn to ignore auditive stimuli like
rifle shots. In the case of sensitization, the response strength increases. It
can happen when a stimulus is presented several times above some toler-
ance threshold (which determines whether habituation of sensitization is
induced). Sensitization can be characteristic of pathologies such as post-
traumatic stress disorder and phobias.

**3.1.1 AUTOASSOCIATION AND HETEROASSOCIATION**

A particular case of associative memory is the memorization of individual
elements which might not seem to involve association at first glance. Mem-
orization can indeed be framed in associative terms as learning the inter-
connections between parts and aspects of the whole. The remembrance of
an item thus typically occurs following the perception of an altered version
of that item, whether being incomplete (Figure 3.1b) or noisy (Figure 3.1c),
as remaining bits of information are used as cues for recalling the original
memory. This particular case where the cue enabling recall is part of the
same whole is called auto-association. It contrasts with hetero-association,
the cue of which belongs to a different item.

It is worth noting that in some cases, an autoassociation might be re-
called from a cue that does not really belong to the same item but is not
considered heteroassociative if the cue was not present during encoding.
In the case of pareidolia<fnref n="1" />, the recalled association can be considered spuri-
ous (Figure 3.1e). It is even harming in pathologies with intrusive memory
recalls. However, such remote associations can be helpful for generalising
attributes of an instance to a broader class (Figure 3.1d). Likewise, this pro-
cess can support creative behavior such as the naming of brain structures
(Figure 3.1f).

This way of storing information about individual and multiple things
through associations radically contrasts with the way typically used in a
computer, where a memory is located at an address, the content of which
can be addressed with a pointer to that address. In the computer case,
memories are thus stored locally and very susceptible to data loss. In the
case of associative memory the content is distributed and can be addressed
with cues of many forms which makes the system much more resilient.

3.1.2 ASSOCIATIVE SPACE AND FORMS

Elements of memory can be associated in many dimensions. The space in
which associations are formed includes but is not limited to spatial (e.g.
*X is on top of Y*), semantic (e.g. *X is a duck*), emotional (*X triggers emotion
Y*) and cognitive dimensions (e.g. *X helps me reach goal Y*). Coming back to
a previous example, when learning the contiguity of songs on an album,
an extra dimension is involved: time. Memories concern which songs fol-
low one another. Note how associations can thus be multidimensional, as
songs can be described by semantic, emotional and cognitive dimensions,
and that the order of songs constitutes an additional temporal component.
The cognitive function enabling the learning of event order is called se-

<footnote n="1">¹Pareidolia is an optical illusion involving the perception of an object (e.g. a face) in a totally unrelated object (e.g. a cloud)</footnote>
<figure><img src="image_7.png" /><figcaption>(a) A seahorse.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(b) Part of Figure 3.1a.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(c) Corrupted version of Figure 3.1a.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(d) A different kind of seahorse.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(e) Clouds in the shape of a seahorse (AI generated).</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(f) Preparation of a human hippocampus and fornix. Adapted from Seress (2010).</figcaption></figure>

Figure 3.1: Various visual representations related to the concept of seahorse.
quence memory, which is heteroassociative in essence because it concerns
the temporal relationship between multiple events. An interesting charac-
teristic of music is that time is also inherent to individual songs. Music
is a temporal combination of many sounds and sounds themselves are a
variation of air pressure over time. When practicing, a musician learns to
play each note for a specific duration. Here, it is not only the order of
events that is learned but also the timing of individual elements. A related
example is the acquisition of songs by birds. HVC (used as a proper name)
is the avian brain structure responsible for sequencing songs, the output of
which is associated with downstream muscle activity for song production
(Hahnloser et al. 2003; Long and Fee 2008; Sankar 2022). When this region is
cooled down, the produced song is slowed down, revealing the causal role
of HVC in motor timing (Long and Fee 2008).

In mammals, the hippocampus is a crucial brain region for learning the
temporal unfolding of events. So-called time cells can be found whose
activity is modulated by elapsed time (Itskov et al. 2011; MacDonald et al.
2011). Recently, it has been proposed that the hippocampus generates se-
quences of content-free pointers that get associated with sequences of corti-
cal representations for learning event chronology (Buzsáki and Tingley 2018).
After encoding this associative mapping, replaying sequences in the hip-
 pocampus would replay activity in the cortex corresponding to experienced
events. This supposedly underlies the retrieval of memories and their tem-
poral structure.

Howard Eichenbaum (2017a) proposed that sequential memory is actually
distinct from the memory of associations. In his review, he describes three
forms of memory: associative, sequential and schematic memory. While
“associations” are necessarily empty of semantic meaning in his view, se-
quential and schematic memories involve meaningful relations. Sequential
memory specifically relates to the temporal ordering of events, whereas
what he calls schematic memory stores relations of hierarchy (e.g. A is
greater than B). Temporal dynamics of the hippocampus play a major role
for representing, storing and recalling sequences (see Section 4.4).
### 3.2 IMPLICIT AND EXPLICIT PROCESSES

In everyday language, the term memory typically refers to declarative knowledge related to events, concepts or facts. The type of memory concerned with content amenable to such conscious recall and verbal report is referred to as explicit (or declarative) memory. A lot of what is learned however cannot be accessed and reported so easily. In cognitive science, the notion of memory also encompasses implicit knowledge. For example, procedural memory is the ability to acquire skills without necessarily being able to communicate or reflect on this knowledge (think about how difficult it is to explain how to ride a bike). Another example is semantic priming, the facilitatory effect in the processing of a word following the presentation of a related word. This process occurs automatically without one's awareness.

Historically, the distinction between implicit and explicit memory traces its origins back to the famous case of patient H.M. who suffered from epileptic seizures. A large portion of his left medial temporal lobe was thus surgically removed. As a consequence of that operation, H.M. became severely amnesic, unable to recall events he had experienced in the few years preceding the surgery, as described in Scoville and Milner (1957) (see also Milner et al. 1968). The patient's ability to form lasting explicit memories was also severely impaired. Scoville and Milner (1957) noted that "once he had turned to a new task the nature of the preceding one could no longer be recalled, nor the test recognized if repeated" (p. 108). It was rapidly discovered however that patient H.M. could learn novel information of a certain nature. In the so-called mirror tracing task, H.M. successfully learned to draw an object using the image reflected from a mirror as sole feedback (Milner 1962). His performance improved over time, demonstrating preserved motor skill learning. In a more recent study, it was reported that the acquired skills could be assessed as many as 359 days after exposure to the task (Gabrieli et al. 1993). Follow-up investigations revealed that H.M. and other amnesic patients can acquire other implicit learning tasks, as reviewed by Hannula and Greene (2012).
The case of patient H.M. reveals a marked distinction between acquired
information amenable to conscious access and verbal report (i.e. explicit),
the learning of which involves the medial temporal lobe (where the hip-
pocampus is located), and information learned that cannot be accessed
consciously and reported verbally (i.e. implicit). This thesis is focusing on
episodic memory and cognitive control which are traditionally considered
explicit cognitive functions. Episodic memory indeed concerns the storage
and explicit retrieval of experiences. Similarly, cognitive control was orig-
inally assumed to always employ explicit representations for modulating
action selection contextually. However, the classification of cognitive pro-
cesses on the basis of whether they happen with or without consciousness
has been questioned, especially in the case of episodic memory (Henke 2010;
Hannula and Greene 2012) and cognitive control (Lau and Passingham 2007;
Gaal et al. 2012). In the following sections, these notions will be introduced
and their relationship to consciousness will be discussed.

3.3 SEMANTIC AND EPISODIC MEMORY

Tulving (1972) further divided explicit long-term<fnref n="2" /> memory into episodic and semantic subsystems. Episodic memory is characterized by its fast acquisition and the nature of information it stores. So-called episodes typically comprise multimodal sensory information along with emotional content. Each episode corresponds to a specific experience and refers to the individual experiencing it (i.e. autobiographical). Another key characteristic of episodic memory is the storage of how events unfold through time. It is a temporal associative memory (see Section 3.1).

On the other hand, what is stored in semantic memory is general facts
and concepts about the world, such as *Paris is the capital of France* or 1+1=2.
Characteristically, these pieces of knowledge are invariantly true regard-
less of the sensory, emotional and temporal context. They are indeed not
stored in association with their context of acquisition, as the context may

<footnote n="2">²See Section 3.4 for a discussion on the notion of long-term memory.</footnote>
not be invariantly associated with the semantic information. For example, while the association between *Paris* and *capital of France* is likely to hold in most situations, the association between *Paris is the capital of France* and *My geography teacher is wearing a blue shirt* is far less likely to hold in the future. The web of associative, schematic and sequential relations (see Section 3.1.2) forms so-called schemas, which constitute the scaffolding of semantic memory.

3.3.1 COMPLEMENTARY LEARNING SYSTEMS THEORY

Why are there multiple systems instead of one playing the roles of both
episodic and semantic memory? This is the question addressed by McCle-
land et al. (1995) and the complementary learning systems theory. The main
idea is that the hippocampus and the cortex support episodic and semantic
memory respectively, and that this division of labor avoids the problem of
catastrophic forgetting introduced by McCloskey and N. J. Cohen (1989). They
illustrate this problem in the context of the AB-AC paradigm (J. M. Barnes
and Underwood 1959) in which a first word is presented (stimulus) and the
goal is to produce the paired word (response). The correct answer is given
at the end of each trial. Once a first set of associations (e.g. *locomotive-*
*dishtowel, table-street, and carpet-idea*) has been learned after multiple trials
of each pair, a second list involving the same stimuli but different responses
is presented (e.g. *locomotive-banana, table-basket, and carpet-pencil*). Given
the stimulus and the context of which list to use, participants must provide
the correct association. Rumelhart (1990) trained a connectionist network
to perform the task by providing the stimulus and context (first or second
list) as input and assessing the output. They showed that the ability of the
model to recall the first set of associations drops catastrophically as soon
as the second set is presented, whereas this interference effect is much
weaker in humans. Human subjects are indeed able to acquire the new
set of associations and the drop of performance for the first set is much
slower and less important than in the model. Catastrophic interference can
be attenuated by interleaving trials of the first set and trials involving the
new associations. When such a scheduling is used, the network acquires
the new associations at a slower pace (since old information is presented
along) but the interference effect is greatly mitigated as old associations are
reminded continually (McClelland et al. 1995).

While the interleaved training regime appears to solve the problem of catastrophic interference, an agent cannot always control the order in which information to learn is presented in its environment. The key proposal of the complementary learning systems is that the hippocampus teaches the cortex by replaying old information interleaved with new information to acquire. The hippocampus thus plays the role of an internally simulated environment with a more optimal training regime than the actual environment for learning in the cortex. This raises the question of how the hippocampus realises this function and does not suffer itself from catastrophic interference. McClelland et al. (1995) proposed that the hippocampus stores experiences separately from each other and that the storage of one does consequently not interfere with the retention of another. In order to do so, the hippocampus uses sparse coding (C. A. Barnes et al. 1990; Quirk et al. 1990) such that representations of slightly different events have little overlap (David Marr and Thach 1991; Bruce L McNaughton and Morris 1987; O'Reilly and McClelland 1994). This way of representing and storing information is well suited for fast acquisition without interference, yet comes at the cost of not being able to discover common structure among several examples and generalize across situations. Interestingly, the nature of episodic memory supposedly results from this solution to the catastrophic interference problem: as experiences are stored separately, the multisensory context must not be discarded. In comparison, the cortex employs much denser representations and slowly aggregates information, which makes it much better at finding patterns in data and generalizing over multiple situations (McClelland et al. 1995). In the process of generalization, the context of acquisition is not imprinted as only invariants are kept. Finally, the cortex is much more modular and each cortical region is much more specialised as compared to the hippocampus which virtually integrates information from
the whole cortex. These elements explain the contrast between semantic
and episodic memory.

It is worth to note that novel information can be acquired rapidly in
the neocortex when it is consistent with an existing schema (Tse et al. 2011;
McClelland 2013). The complementary learning systems theory has thus
been refined to account for these findings Kumaran, Hassabis, et al. (2016).

3.3.2 IMPLICIT EPISODIC MEMORY

In Section 3.2, the distinction between explicit and implicit memory sys-
tems was introduced with the notable case of patient H.M. While he re-
tained the ability to acquire skills, his episodic memory was severely im-
paired. Episodic memory is typically considered an explicit memory pro-
cess, as one can be conscious of and verbally report one's episodic mem-
ories. However, there have been reports of hippocampus learning oper-
ations, which are typically associated with episodic memory, that happen
without conscious awareness (Hannula and Greene 2012; Henke 2010). In Han-
nula and Ranganath (2009), hippocampal activity predicted whether partic-
ipants successfully made eye movements to the face that was previously
associated with the current scene. This was also the case in trials in which
participants could not consciously remember which one was the correct
face. By contrast, amnesic patients with hippocampal damage did not ex-
hibit the eye movement effect Hannula, Ryan, et al. (2007). This suggests that
hippocampus-dependent recall of associations can occur without affecting
conscious experience. Similarly, subliminal information can be encoded
in the hippocampus and influence future conscious retrieval (Henke, Mon-
dadori, et al. 2003; Henke, Treyer, et al. 2003). In these studies, pairs of faces
and occupations (e.g. plumber) were presented very briefly (17 ms) with
visual noise preceding and following the stimulus, making it inaccessible
to conscious perception. During both encoding (subliminal presentation)
and retrieval (supraliminal presentation), hippocampal activity changed in
the experimental versus control condition. In a follow-up study (Degonda et
al. 2005), face-occupation pairs were presented supraliminally after present-
ing incongruent pairs subliminally (same faces were paired with different
occupations). The hippocampus was also found to be modulated by the
presentation of stimuli independently of conscious awareness. Most inter-
estingly, implicit pairs impaired the retrieval of incongruent explicit pairs,
suggesting an influence of implicit memory on explicit memory.

These findings challenge the consciousness-based division of memory
systems. An alternative classification was proposed based instead on pro-
cessing modes (Henke 2010), as illustrated in Figure 3.2. Episodic memory
enables rapid encoding of flexible associations (or bindings) such as face-
occupation pairs, and it relies on the hippocampus and parts of the neocor-
tex. Alternatively, familiarity and priming involve rapid encoding of single
or unitized items and do not require the hippocampus but rather surround-
ing medial temporal lobe structures. Slow encoding of rigid associations
supporting procedural memory, classical conditioning and semantic mem-
ory is totally independent from the medial temporal lobe (at least after
consolidation ; see Sections 3.3.1 and 3.4).

## 3.4 SHORT-TERM AND LONG-TERM MEMORY

Another common way to classify memory systems is to distinguish them
on the basis of their temporal capacity. While information stored in work-
ing memory is susceptible to distractions and is typically not kept for
longer than seconds or minutes, the content of semantic, episodic, proce-
dural and other memory systems can last for years or be lifelong. There is
no clear division between short-term and long-term memory which is why
these terms can be misleading. In fact, there is a continuum of temporal ca-
pacities among memory systems, and there can also be differences between
how much time individual items are stored in the same system. However,
there seems to be a relationship between processing stage and system ca-
pacity as suggested by the Multi-Store Model of Atkinson and Shiffrin (1968),
in which each system has a smaller capacity than the preceding one, as it
selects which information to store, but can store selected information for a
<figure><img src="image_13.png" /><figcaption>Figure 3.2: Memory systems classification based on processing modes. Taken from Henke (2010).</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Figure 3.3: Multi-Store Model of Atkinson and Shiffrin (1968). Taken from Reser (2022).</figcaption></figure>

longer time (see Figure 3.3). Sensory memory is at the earliest stage and has biggest capacity but information transiting through it rapidly decays unless it is attended to and stored in working memory.

There is a commonly agreed difference between how short-term (e.g. sensory and working memory) and long-term (e.g. episodic and semantic) memory systems store information. Short-term systems typically rely on the reverberation of neural activation for maintaining information across time. Since activity can be perturbed, this mode of storage is less robust than long-term memory systems that store information by changing the structure of the network (typically the synapses). Stokes (2015) reviewed some data suggesting an alternative synaptic-based and activity-silent way of storing information in working memory. According to Beukers et al. (2021) a more parsimoniously explanation is that episodic memory (synaptic-based) interacts with working memory (activity-based). This illustrates that so-called "long-term" memory systems cannot only play a role in the long term but can also support cognition in the short term.

### 3.4.1 STANDARD CONSOLIDATION THEORY

There is also a disagreement on the time span of hippocampal involvement in memory that further blurs the line between short-term and long-term memory systems. In the hippocampal memory indexing theory of Teyler and DiScenna (1986), the hippocampus flexibly stores novel combinations of
neocortical activations and can recall memories by reinstantiating the acti-
vation of the corresponding cortical areas. Eventually, some of these mem-
ories are transferred to the cortex through a process called consolidation,
while others are forgotten. The standard theory of consolidation (Squire
and Alvarez 1995; McClelland et al. 1995), which is similar to the comple-
mentary learning systems theory mentioned in Section 3.3.1, builds upon
the observation that patients with hippocampal damage can recall remote
memories better than recent ones, a phenomenon called temporally-graded
retrograde amnesia. The idea is that once memories have initially been
stored in the hippocampus, the consolidation process underlies a trans-
fer of memories from the hippocampus to the neocortex for longer term
storage, as originally proposed by Marr (1971), and that memories are no
longer stored in the hippocampus. Then, when information perceived from
the environment is consistent with information consolidated in the cortex
(i.e. with schemas), the hippocampus is inhibited by the prefrontal cortex
to prevent it from learning redundant information (Frankland and Bontempi
2005; Van Kesteren et al. 2012).

3.4.2 TRANSFORMATIVE ACCOUNTS OF CONSOLIDATION

The standard theory of consolidation was later challenged by findings of
remote memory recall impairment following hippocampal lesions when
memory are truly episodic/autobiographical. The hippocampus thus seems
necessary for recalling detailed episodic memories vividly (Viskontas et al.
2002; Rosenbaum et al. 2000; Cipolotti et al. 2001). The multiple trace the-
ory was proposed by Nadel and Moscovitch (1997) as an alternative to the
standard consolidation theory in order to account for these findings. As
its name implies, the theory states that memories are stored with multi-
ple traces (i.e. in both the hippocampus and neocortex) and that the hip-
pocampus is recruited for recalling episodic memories, regardless of how
remote they are. This theory is in agreement with the broader hypoth-
esis that memories are transformed during transfer, such that they take
a more semantic form in the neocortex while remaining truly episodic in
the hippocampus. The recently proposed CRISP theory of the hippocam-
pus (Cheng 2013) also departs from the standard view and emphasizes the
temporal sequence nature of episodic memory and hippocampal process-
ing. The author states that semantic memory stores information similar to
episodic memory, but that the recall of temporal associations necessarily
involves the hippocampus. This rightfully highlights the temporal nature
of episodic memory, however this claim can be contrasted with the observa-
tion that language (and especially syntax) also contains a strong temporal
component that is largely preserved following hippocampal impairment
(but see H.M.'s linguistic impairments reported by MacKay et al. 1998).

The complementary learning systems (see Section 3.3.1) is often asso-
ciated with the standard theory but also suggests a mechanism for the
transformation of episodic memories into semantic ones. Hippocampal
convergence of cortical connections as well as the differing learning rates
and modes of representation are good candidates for explaining why se-
mantic memory forgets the details of individual situations and discards
the context of acquisition. In sum, memory systems have different tem-
poral capacity but the relationship is much more complex than the simple
distinction between short-term and long-term memory, and the temporal
component might not be a very relevant criterion to distinguish them.

3.5 COGNITIVE CONTROL

3.5.1 THE NEED FOR COGNITIVE CONTROL

In Section 3.2, the notion of procedural memory was introduced as the abil-
ity to acquire skills. In the motor domain, this boils down to the selection
and execution of the appropriate motor commands at the right time. A
straightforward strategy for motor control is to learn how rewarded each
action is and to always select the most rewarded candidate. The prob-
lem is that this outcome might be a stochastic function of the action, not
necessarily because the world is intrinsically stochastic but rather because
the agent only has partial information. In that case, individual outcomes
might not be predictable, but an agent could still learn statistics related to
the distribution of possible outcomes. Typically, this is realized by keep-
ing track of intermediate statistics and updating them with the stream of
experience. This implies a trade-off between precision and flexibility: up-
dates should be slow to converge to the real statistics but slowness impairs
adaptation in the case where the distribution is not stationary (i.e. changes
over time). This happens for similar reasons as those justifying comple-
mentary episodic and semantic systems (see Section 3.3.1). A case of non
stationarity is when the action leads to a state that is no longer rewarded,
as captured by reward devaluation paradigms (Dickinson and Balleine 2002).
This can happen if an animal is full after eating or if it has exhausted a food
source. Another case is a modification of state contingency. What changes
in that case is not the reward, but rather the transitions between states. The
travel distance between an agent and a rewarding state might for example
increase as a consequence of the apparition of an obstacle. In these cases,
the commonly chosen actions are likely to result in suboptimal outcomes
and the strategy must therefore be revised.

3.5.2 THE MANY FORMS OF CONTROL

Humans and many other animals are capable of adapting to these situa-
tions. They can change their behavior by integrating information from dif-
ferent sources. An obvious strategy is not to learn absolute action-outcome
associations but rather associations between stimulus, response and out-
come: choosing to eat will result in a much better outcome when facing
food than nothing. Beyond this caricatural example, the relationship be-
tween action and outcome can be arbitrarily complex. It might be that the
right action is conditional on a past observation of the world, on a stim-
ulus that was perceived in the past. For example, foraging honeybees are
thought to store memories of food sources and communicate information
about the direction and distance of these sources to their hive with the
waggle dance (Menzel 2023). It is critical for the survival of the colony that
the dance conveys correct information regarding the position of actual food
sources. Likewise, an agent might decide to take a detour after retrieving
the perception of an obstacle which is changing state contingency.

It might also be that the right thing to do depends on a combination or
sequence of stimuli. This is especially the case with language, note how
different *My cat chased the dog* is from *My dog chased the cat*, and from *chased
cat My the dog*. Similarly, the sequence of actions can be of importance, as
individually sub-optimal actions can be executed in pursuit of a long-term
plan. An agent incapable of executing a coherent sequence of actions to
reach a goal would be analog to Buridan's ass which died midway between
a stack of hay and a pail of water because its hunger equaled its thirst.

The many layers of decision making often appear to coexist within ani-
mals like mammals. While actions generally leading to positive outcomes
are automatized, they can be flexibly inhibited in favor of less salient re-
sponses at will. This is realized through a process called cognitive con-
trol, which monitors incoming outcomes, stores relevant stimuli in working
memory and biases actions accordingly by deploying attention to modulate
salience in the premotor cortex (Posner et al. 2004). This function is mainly
dependent on the integrity of the prefrontal cortex. In contrast to H.M.
whose hippocampal lesion specifically impaired episodic memory, patient
K.M. who had prefrontal damage showed cognitive control deficits.

3.5.3 IMPLICIT COGNITIVE CONTROL

Cognitive control, attention and working memory are intuitively (and his-
torically) associated (and often confused) with consciousness. Once again,
the consensus has shifted away from this view following evidence of these
functions operating in absence of conscious awareness (Lau and Passing-
ham 2007; Van Gaal, Ridderinkhof, Fahrenfort, et al. 2008; Van Gaal, Ridderinkhof,
Scholte, et al. 2010; Gaal et al. 2012, but see Persuh et al. 2018). A taxonomy
of conscious processing was proposed by Dehaene et al. (2006). They em-
phasize on the distinction between consciousness and attention: attending
to a stimulus is a necessary condition for it to access consciousness but a
stimulus can be attended to and remain unconscious if it is too weak. The
stimulus is then referred to as subliminal. In a similar way as for episodic
memory, it seems that cognitive control is intimately linked to explicit rep-
resentations, but that a better defining characteristic is its processing mode:
the contextual modulation of automatic responses.

3.6 METACOGNITION AND METAMEMORY

Both the hippocampus and prefrontal cortex are thought to underlie flexi-
ble cognition, the former binding novel neocortical combinations for mem-
ory storage (episodic memory) and the latter biasing responses when con-
text undermines habitual actions (cognitive control), as described by J.D.
Cohen and O'Reilly (1996). Combining these two structures can enable even
more flexible behavior, each one executing its function on the other. To-
gether, they are thought to support what has been called the construction
system of the brain (Hassabis and Maguire 2009), that is a system that com-
bines information from memory to predict the future through imagination
(Schacter, Addis, and Buckner 2007; D.T. Gilbert and T.D. Wilson 2007; Spreng
et al. 2009; Hassabis and Maguire 2009; Schacter, Addis, Hassabis, et al. 2012).

In Chapter 5, prefrontal-hippocampal interactions will be described (Fig-
ure 3.4). Hippocampus-to-prefrontal projections will be introduced as pro-
viding episodic memory context to cognitive control. The hippocampus
will also be shown to encode prefrontal activity like that of other cortical
areas. Consequently, the hippocampus can also replay prefrontal informa-
tion for consolidating memories and improve the ability of the prefrontal
cortex to perform cognitive control. Conversely, the prefrontal influence on
the hippocampus will be described. In addition to modulating action se-
lection contextually, the prefrontal cortex will be shown to control episodic
memory operations. The control of encoding has been mentioned in Sec-
tion 3.4.1. The contextual control of episodic memory retrieval will also be
described thoroughly.

These functions perhaps lie at the highest levels of our cognition. They
are functions that exert influence on other functions and can thus be re-
<figure><img src="image_15.png" /></figure>

Figure 3.4: Proposed interactions between the prefrontal cortex (cognitive control) and the hippocampus (episodic memory). The prefrontal cortex controls action selection and hippocampal recall. In return, the hippocampus provides context to the prefrontal cortex in addition to augmenting perceptual representations with memory.

ferred to as metacognitive or metamemory. A central goal of this thesis is to argue that neuroscience, cognitive science and AI are now mature enough to tackle these topics, and that these functions could augment machine learning Chateau-Laurent and Alexandre (2021a), account for creative processes Chateau-Laurent and Alexandre (2021b) and underlie the construction system of the brain.
# 4 EPISODIC MEMORY

<figure><img src="image_16.png" /></figure>

*An agent remembering an episodic memory*

*Generative AI*

## 4.1 INTRODUCTION

As presented in the preceding chapter 3, episodic memory can be distinguished from other memory systems using multiple criteria. These criteria can be questioned individually, but collectively shed light on the nature of episodic memory. The content of episodic memory typically influences
explicit processes and can often be reported verbally, which contrasts with
implicit systems like procedural memory. It can potentially be stored for a
lifetime and provide context to other cognitive functions like decision mak-
ing and cognitive control. It can also be consolidated into more perennial
semantic forms.

What perhaps best characterizes episodic memory is its mode of process-
ing, that is its ability to encode and store novel relations between known
objects flexibly and rapidly (Figure 3.2) by binding previously unrelated
cognitive representations. It is a memory form that stores sequences of
multimodal information corresponding to specific situations. In order to
do that, it must have access to information of a wide range of cognitive
modules and encode their conjunction. It must also be able to learn these
conjunctive bindings rapidly, as similar information might not be expe-
rienced multiple times, and also how this information evolves through
time. In addition of disambiguating different memories, it will become
clear in this chapter that the brain regions underlying episodic memory
are also able to quickly extract statistical regularities among similar situa-
tions. When necessary, the memory must be able to retrieve a memory by
filling cognitive modules with information absent at the time of retrieval,
and perhaps replace information of the current situation. Episodic mem-
ory then conducts cognition for the organism to re-experience all aspects of
the episode in the right order. This perhaps idealistic view corresponds to
the current understanding of how episodic memory works, interacts with
other cognitive functions and supports memory consolidation.

It is common to regard the episodic content as a what-where-when con-
junction, but it is hard to apprehend the complete potential of binding the
content of cognitive functions as diverse as those exhibited by humans. The
"what" can indeed include semantic, emotional, social, abstract and even
metacognitive aspects that are encoded at multiple spatiotemporal scales.
Another aspect of episodic content is that it refers to the self, contrasting
with the allocentric nature of semantic memory. It follows from the mul-
timodal contextualization of information and underlies autobiographical
memory (i.e. memory of one's life events) and autonoetic consciousness,
that is the ability to place oneself in the past and future. On the other hand, the allocentric nature of semantic memories can be explained by abstracting away details of individual experiences.

This chapter introduces the anatomy (Section 4.2), representations (Section 4.3) and dynamics (Section 4.4) underlying episodic memory in the medial temporal lobe. Then, computational models for rapid pattern and sequence learning, as well as models for the progressive extraction of statistical regularities, are presented in Section 4.5.

## 4.2 ANATOMY

The medial temporal lobe is the seat of episodic memory (Scoville and Milner 1957). It is mainly composed of the hippocampus and parahippocampal regions like the entorhinal cortex. These regions collectively support the encoding, storage and retrieval of episodes.

<table><caption>Table 4.1: Number of principal neurons in different regions of the rat's medial temporal lobe, as retrieved from Cutsuridis et al. (2019).</caption><thead><tr><td>Region</td><td>Count</td></tr></thead><tbody><tr><td>EC2</td><td>110,000</td></tr><tr><td>EC3</td><td>250,000</td></tr><tr><td>DG granule cells</td><td>1,200,000</td></tr><tr><td>Hilar mossy cells</td><td>50,000</td></tr><tr><td>CA3</td><td>250,000</td></tr><tr><td>CA1</td><td>390,000</td></tr><tr><td>EC5</td><td>330,000</td></tr></tbody></table>

<figure><img src="image_17.png" /><figcaption>Figure 4.1: Schematic illustration of the trisynaptic (blue) and monosynaptic (green) pathways connecting the entorhinal cortex and hippocampus. Adapted from Ketz et al. (2013) with the CA3-to-DG backprojection through the hilus.</figcaption></figure>
4.2.1 ENTORHINAL CORTEX

The entorhinal cortex (EC) is part of the medial temporal lobe and is located next to the hippocampus. In terms of connectivity, the entorhinal cortex is the main interface between the neocortex and the hippocampus. Thus, the entorhinal cortex is the place of convergence of many different sources. It is divided into medial and lateral regions that provide inputs of different kinds to the hippocampus (Howard Eichenbaum, Yonelinas, et al. 2007; Manns and Howard Eichenbaum 2006). The lateral portion connects to the “what” pathway, a neocortical circuit involving mainly the temporal lobe that conveys semantic information about the perceived scene (e.g. features of an object). On the other hand, the medial portion of the entorhinal cortex connects to the “where” pathway that conveys information about the spatial context (e.g. where the object is located) through the parietal lobe. Both entorhinal portions connect to the hippocampus which is thought to bind information from the two streams (Howard Eichenbaum, Yonelinas, et al. 2007; Whittington, T. H. Muller, et al. 2020).

Pretty much orthogonal to the medial-lateral divide of the entorhinal cortex is the longitudinal axis (Figure 4.2), as reviewed by Strange, Menno P Witter, et al. (2014). In rats it extends from the dorsolateral parts to more ventromedial portions. These parts preferentially connect to the dorsal and ventral hippocampus ends respectively (Figure 4.2b). In macaque and humans this connectivity gradient is oriented along the anteroposterior axis of the hippocampus (Figure 4.2a).

Like in other cortical regions, neurons of the entorhinal cortex are grouped
into layers (Figure 4.3). Six layers compose the entorhinal column, layers
II, III, V and VI containing most neurons. Telencephalic inputs converge to
neurons of layers II and III. Layer II then connects to the dentate gyrus and
CA3 of the hippocampus, while neurons from layer III project to CA1. CA1
and the subiculum in turn send the output of the hippocampus to layer V
of the entorhinal cortex, which is then transmitted back to telencephalic
structures. Layers II, III and V of the rat entorhinal cortex are composed
<figure><img src="image_18.png" /><figcaption>(a) Top: The rat hippocampus is oriented such that the longitudinal axis corresponds to the ventral-dorsal direction. In macaque monkeys and humans, the longitudinal axis is described as anteroposterior. Bottom: Position of the entorhinal cortex relative to the hippocampus in these species.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(b) Topographical arrangement of the rat entorhinal-hippocampal connectivity. The dor-solateral portion of the entorhinal cortex (magenta) is preferentially connected to the dorsal hippocampus while more ventromedial parts connect to more ventral ends of the hippocampus.</figcaption></figure>

Figure 4.2: Schematic illustrations of the entorhinal-hippocampal longitudinal axis in different species. Taken from Strange, Menno P Witter, et al. (2014).
<figure><img src="image_20.png" /><figcaption>Figure 4.3: Connectivity of the entorhinal column. Abbreviations: Re, reelin-expressing neurons; RSC, retrosplenial cortex. Adapted from Menno P Witter, Doan, et al. (2017).</figcaption></figure>

of approximately 110,000, 250,000 and 330,000 neurons respectively (Table 4.1).

4.2.2 DENTATE GYRUS

The dentate gyrus (DG) is part of the trisynaptic circuit of the hippocampal formation, that is a pathway with three synapses separating the entorhinal cortex from CA1 (Figure 4.1). The dentate gyrus is mainly populated by granule cells and mossy cells located in the hilus. Granule cells of the dentate gyrus receive projections from the entorhinal cortex through the perforant path, originating mainly from layer II. Granule cells in turn project to both mossy cells of the dentate gyrus and CA3 pyramidal cells. Mossy cells receive inputs from both granule cells and CA3 pyramidal cells, and in turn project back to granule cells. The circuit is thus two-fold recurrent, with a small intra-DG loop, and a bigger loop that involves CA3.

Granule cells have unique properties. First, they are the most abundant neurons of the hippocampal formation, as indicated in Table 4.1. Secondly, their activity appears to be among the most sparse in the brain (M. Jung and B. McNaughton 1993; Diamantaki et al. 2016). In a study of freely mov-
ing rats, M. Jung and B. McNaughton (1993) found only 12% of granule cells that fired faster than 0.5Hz. This sparsity is thought to be mediated by inhibitory basket cells (Andersen 1975). Thirdly, new granule cells are pro- duced throughout life in fish, reptiles, birds, rodents, nonhuman primates and humans (Augusto-Oliveira et al. 2019) while such adult neurogenesis has not been documented elsewhere besides the subventricular zone. Newborn granule cells integrate into the network of preexisting granule cells (Toni et al. 2008). Furthermore, they exhibit high excitability and enhanced synap- tic plasticity (Schmidt-Hieber et al. 2004; Trinchero et al. 2017), unlike mature granule cells which show low excitability and plasticity to the extent that they are sometimes considered functionally retired (as critically reviewed by Lopez-Rojas and Kreutz 2016). The role of newborn neurons in learning is controversial but experiments suggest that neurogenesis is necessary to reduce memory interference between similar events (Yau et al. 2015). Finally, granule cells make contact with CA3 cells with very low probability and uniquely powerful “detonator” synapses, such that a single action poten- tial of a granule cell is sufficient to generate an action potential in the CA3 neurons it targets (Henze et al. 2000; Vyleta et al. 2016). Combined with the steep sigmoidal response of the dentate gyrus to entorhinal inputs (Renata Bartesaghi et al. 1995), the detonator synapse participates in making the hip- pocampus a region with an almost all-or-none response profile (Bartesaghi et al. 2006).

4.2.3 CA3

Properly speaking, the hippocampus is the small curved formation called Cornu Ammonis, which is divided in four portions and does not include the dentate gyrus<fnref n="1" />. The third portion, CA3, is mainly populated by pyramidal neurons receiving inputs from granule cells of the dentate gyrus (second synapse of the trisynaptic circuit). Pyramidal cells of CA3 are known for their axons composed of recurrent collaterals that branch and connect

<footnote n="1">¹However, the dentate gyrus is part of the hippocampal formation and it is very common to include the dentate gyrus when speaking about the hippocampus.</footnote>
to other CA3 pyramidal cells, and Schaffer collaterals that project to CA1 pyramidal cells, completing the trisynaptic pathway (EC →₁ DG →₂ CA3 →₃ CA1). A connection often overlooked is the afference of the second entorhinal layer. An even lesser known connection is the backprojection from CA3 pyramidal cells to hilar mossy cells of the dentate gyrus (Scharfman 2007).

4.2.4 CA1

CA1 is the largest region of the hippocampus proper, it comprises approximately 390,000 pyramidal neurons in rats (Table 4.1). CA1 receives inputs from both CA3 Schaffer collaterals (completing the trisynaptic pathway) and layer III of the entorhinal cortex (monosynaptic pathway). These two major sources can be distinguished in terms of destination. While entorhinal afferents connect to the stratum lacunosum-moleculare of CA1 pyramidal neurons, Schaffer collaterals make contact at the stratum radiatum. CA1 neurons in turn project to layer V of the entorhinal cortex. They also project to the subiculum which is not central to this thesis but potentially plays a role in spatial navigation, memory and stress regulation (see the reviews of S. M. O'Mara et al. 2001; S. O'Mara 2005).

4.3 REPRESENTATION: THE HIPPOCAMPUS AS A COGNITIVE MAP

4.3.1 GRID CELLS

Hafting, Fyhn, Molden, et al. (2005) found that cells of the rat medial entorhinal cortex are tuned to represent a map of the spatial environment. They revealed the hexagonal activity patterns of what they called “grid cells”. These patterns are called grid fields and can be described by three parameters that vary from cell to cell: orientation, spacing and spatial phase. Grid cells are topographically organized such that neighboring neurons are tuned with a similar orientation and spacing (Hafting, Fyhn, Molden, et al.
2005). This is also the case for phase at a much finer scale (Heys et al. 2014). On top of that, spacing increases along the dorsomedial-to-ventrolateral anatomical axis of the medial entorhinal cortex (Hafting, Fyhn, Molden, et al. 2005), with discrete steps suggesting that grid cells are organized in functional modules (Stensola et al. 2012). Grid cells and an axis along which grid spacing increases have also been found in monkeys (Killian et al. 2012).

Perhaps even more interesting is the persistence of their tuning in the absence of visual cues, in total darkness. According to the authors, this property suggests that grid cells play an important role in path integration, which is the capacity of an agent to maintain a representation of its spatial location, and update it with self-motion information. This theory was later supported by the emergence of grid-like cells in artificial recurrent neural networks trained to path integrate (Banino et al. 2018; Cueva and Wei 2018; Sorscher et al. 2019; Whittington, T. H. Muller, et al. 2020; Nayebi et al. 2021). These findings, however, have been reviewed and challenged by Schaeffer et al. (2022) who did not find grid fields unless post-hoc implementation choices were made. They proposed that the path integration objective is perhaps necessary but not sufficient for grid fields to emerge. Beyond this controversy, it is very common for computational models of grid cell formation to assume the medial entorhinal cortex is integrating velocity information. Some of them rely on the notion of continuous attractors (e.g. Burak and Fiete 2009; Fuhs and Touretzky 2006; Bruce L McNaughton, Battaglia, et al. 2006; Guanella et al. 2007) and predict that network dynamics lie on a toroidal manifold. A recent study has provided support for this theory by recording from 7.671 entorhinal neurons simultaneously and showing their activity span a two-dimensional ring that also persists during sleep and across environments (Gardner et al. 2022). Another normative theory of interest is that grid fields emerge as a result of the eigendecomposition of a matrix predicting future state occupancy (Stachenfeld et al. 2017). This relates to the successor representation framework described in Section 4.5.5.
4.3.2 PLACE CELLS

The hippocampus also comprises so-called place cells modulated by spa-
tial location, but whose firing fields (named place fields) do not have the
hexagonal structure of grid cells (O'Keefe and Dostrovsky 1971). The dis-
covery of these place cells supported the old theory that the hippocampus
encodes a cognitive map of the environment (Tolman 1948).

When the environment changes, the activity of place cells changes such
that their field is reallocated to a different position, they remap (R. U. Muller
and Kubie 1987). The continuous morphing of the environment leads to a
discontinuous global remapping, which is reminiscent of attractor dynam-
ics (Wills et al. 2005).

An intuitive explanation as to why these place fields emerge is that hip-
pocampal neurons receive information from grid cells of variable tuning,
and that place fields naturally emerge as the conjunction of their respec-
tive grid. This idea has been formulated and mathematically validated
by Solstad et al. (2006). Another demonstration of this phenomenon was
recently provided by Santos-Pata et al. (2021) who trained a network to sim-
ply output its input in an autoencoder fashion, with inputs resembling
entorhinal activity. Once trained, place cells could be observed in interme-
diate hippocampal layers and exhibited physiological phenomena such as
rate modulation and global remapping. The network however received not
only medial entorhinal input but also a lateral entorhinal source. While
the theory of Solstad et al. (2006) only relies on grid cells and thus does not
ascribe a role to the lateral entorhinal cortex, it has been noted that the
hippocampus is the place of convergence of the “what” and “where” cor-
tical pathways (Howard Eichenbaum, Yonelinas, et al. 2007; Manns and Howard
Eichenbaum 2006; Whittington, T. H. Muller, et al. 2020), and that place fields
might emerge as the conjunction of these two sources of information. The
hippocampus thus would encode items in their context by associating se-
mantic information provided by the lateral entorhinal cortex with position
information encoded by grid cells. A prediction of this alternative theory is
that semantic information exerts an influence on the activity of place cells,
which has been confirmed by the observation of landmark-vector cells that
respond to object manipulations and signal distance to landmarks (Desh-
mukh and Knierim 2013).

Unlike grid cells, place cells do not seem to be topographically arranged
(M. A. Wilson and Bruce L McNaughton 1993; M.-B. Moser et al. 1995 ; but see
Nakamura et al. 2010). However, their scale also increases in the dorsal-to-
ventral direction in rodents (Min W Jung et al. 1994; Maurer et al. 2005; Kjel-
strup et al. 2008). Strange, Menno P Witter, et al. (2014) reviewed a set of results
describing this axis both anatomically and functionally. The multiscale rep-
resentations employed by grid and place cells are thought to support the
hierarchical organization of memory and navigation. A good example is
the task of transitive inference, in which participants must infer that A is
associated to C after being informed that A is associated to B and B is
associated to C. The latter associations are of a lower order than the asso-
ciation between A and C. Studies showed that this inference is dependent
on the ventral hippocampus of rats (Komorowski et al. 2013) and the ante-
rior hippocampus of humans (Heckers et al. 2004; Preston, Shrager, et al. 2004).
Since the size of place fields in these regions is larger, their activity covers
more spatial locations and more items which are potentially not overlap-
ping in dorsal/posterior regions, a mechanism which probably underlies
the aforementioned transitive inference (Strange, Menno P Witter, et al. 2014).

Mehta et al. (1997) showed that place fields are shaped by experienced spa-
tial trajectories. When rats travel across a linear track multiple times in the
same direction, their place fields shift their center of mass in the opposite
direction and increase in size. They thus become asymmetric. Furthermore,
some place cells not only fire according to the position in space, but also
depending on the temporally close past and future trajectory (Wood et al.
2000; L. M. Frank et al. 2000). L. M. Frank et al. (2000) indeed found cells that
only fire when the animal occupies its place field and comes from a spe-
cific part of the environment. Some cells fired when the rat reached their
field and was about to go in a specific direction. Results related to these
so-called splitter cells (their firing is split by a prospective or retrospective
signal) have recently been reviewed by Duvelle et al. (2023).
The name place cell intimately links this category of neurons to spatial
navigation and blurs the relationship with memory. While these cells ob-
viously code for spatial variables, they are modulated by other variables
too as suggested by the remapping and splitter cells phenomena discussed
above. Recently, it has been found that place fields emerge in a model
whose aim is to compress sensory experiences, and that place cells might
simply be memory cells, in that their activity reflects an efficient strategy
for storing correlated patterns (Benna and Fusi 2021). Their model for exam-
ple accounts for history dependent splitter cells which decorrelate similar
representations on the basis of the trajectory. This is in line with the older
proposal of Howard Eichenbaum, Dudchenko, et al. (1999) that the observa-
tion of place cells does not suffice to conclude that hippocampal neurons
primarily encode space.

In the same vein, many studies have found hippocampal cells modu-
lated by nonspatial variables such as sound (Sakurai 2002; Aronov et al. 2017),
odors (Eichenbaum et al. 1987), faces and objects (Fried et al. 1997). These
findings further nourish the long legacy of the cognitive map idea by
Tolman (1948). A modern generalization is the Tolman-Eichenbaum ma-
chine model of Whittington, T.H. Muller, et al. (2020) suggesting that the
entorhinal-hippocampal system integrates any cognitive variable with a
common mechanism, including the most abstract ones. They generalize
spatial path integration to abstract tasks typically associated with relational
memory (e.g. social hierarchies and transitive inference) which have been
found experimentally to rely on the hippocampus (N.J. Cohen et al. 1999;
Konkel and N.J. Cohen 2009; Dusek and Howard Eichenbaum 1997; Kumaran,
H.L. Melo, et al. 2012). In this model, place cells emerge as the conjunc-
tion of sensory and structural information sent by the lateral and medial
entorhinal cortices respectively, as proposed by Manns and Howard Eichen-
baum (2006), giving rise to the landmark cells observed by Deshmukh and
Knierim (2013). A prediction of the Tolman-Eichenbaum machine that the
authors validate with existing experimental data is that place cells do not
remap randomly but rather stay consistent with the entorhinal grid code,
such that a place cell co-firing with a grid cell will still co-fire with it fol-
lowing remapping.

4.4 DYNAMICS: OSCILLATIONS AS MEMORY OPERATIONS

Brain oscillations refer to rhythmic electrical activity generated by pop-
ulations of neurons. They are prominent in the hippocampus and are
thought to reflect and support cognitive operations (Lisman and O. Jensen
2013; Buzsáki and Watson 2012).

4.4.1 THE ENTORHINAL BUFFER

Egorov et al. (2002) described how neurons from layer V of the entorhinal cortex (EC5) in slices of rat brains respond to their input. They demonstrated that EC5 (CA1->EC5) cells could exhibit stable patterns of activity following stimulation for an apparently indefinite period of time (up to 13 min). The likelihood of exhibiting such persisting activity was a nonlinear function of the amplitude (nA) and duration (s) of the stimulation, parameterized differently for each neuron. Furthermore, the authors described multiple stable levels of persistent firing frequency. They showed that up and down transitions in firing frequency could be induced by excitatory and inhibitory stimulations respectively. In a subsequent study, Fransén et al. (2006) intended to reproduce and extend these results. Beyond multiple levels of stability, they observed a continuum of stable states. These results suggest that EC5 neurons implement what is called an integrator, in that they integrate their input over time. Persistent activity has also been observed in the second and third layers of the entorhinal cortex (Tahvildari et al. 2008).

Hasselmo and Stern (2006) reviewed these results and proposed that the
entorhinal cortex uses these mechanisms to implement a working mem-
ory for novel information. In short, a stimulus can cause persistent ac-
tivity in the entorhinal cortex, making stimulus information available to
that region in the short term after stimulus removal. Since mechanisms
for generating persistent activity in the entorhinal cortex occur at the level
of individual cells, this memory does not depend on synaptic connectivity
and can thus store stimuli that were never encountered without synaptic
plasticity. Contrastingly, the prefrontal and parietal cortices also exhibit
persistent activity that is thought to underlie working memory, but this is
thought to depend on the recurrent synaptic excitation of cells. Hasselmo
and Stern (2006) therefore argued that this connectivity-based implementa-
tion of working memory is insufficient for storing novel stimuli that did
not shape the recurrent circuitry beforehand. The authors related their the-
ory to A. Baddeley (2000)'s addition of a multimodal episodic buffer to his
earlier working memory model (A. D. Baddeley and Graham 1974).

A working memory is typically expected to exhibit a load effect, that is a
modulation of its activity depending on the number of stimuli stored. In a
fMRI study, Schon et al. (2016) employed a delayed matching-to-sample task
to probe whether medial temporal lobe regions exhibit such load effect.
Participants were shown a sequence of two (low load) or four (high load)
novel landscape pictures, maintained the stimuli in memory during a delay
period, and were asked to determine whether a test picture matched one
of the previous landscapes of that trial or not. As a result, the activity of
the entorhinal cortex was greater during maintenance of four stimuli than
two, consistent with the theory of Hasselmo and Stern (2006).

A related line of work comes from the study of the role of oscillatory
dynamics in sequence learning (O. Jensen and Lisman 2005; Lisman and Idiart
1995; O. Jensen, Lisman, et al. 1996; O. Jensen and Lisman 1996). In the review
of O. Jensen and Lisman (2005), the authors proposed that the entorhinal
cortex is a short-term memory oscillatory buffer that holds information
about the recent sequence of events and sends it to the hippocampus at
a pace optimal for long-term memory storage. This will be described in
more details in the context of the SOCRATIC model (Section 4.5.5).

We now turn to provide a description of the information sent from the
entorhinal cortex to the hippocampus. These two regions closely interact,
as the entorhinal cortex constitutes both the major input source and ma-
jor output destination of the hippocampus. This loop suggests possible
feedback effects and therefore renders the characterization of hippocampal input challenging. Studies have attempted to study the nature of efferent entorhinal information by describing the entorhinal activity in isolation, under conditions of hippocampal inactivation. In one such study, Hafting, Fyhn, Bonnevie, et al. (2008) reported theta oscillations and phase precession in entorhinal grid cells. A cell exhibits phase precession relative to a population-wide oscillation when it fires earlier and earlier in each successive oscillation cycle. During periods of activity, the hippocampus and the entorhinal cortex are indeed dominated by a strong theta oscillation (4-10Hz). Place cells of the hippocampus typically exhibit phase precession with respect to this rhythm. When entering its place field, a cell will fire at the end of the theta cycle, and as the animal progresses the cell will fire at earlier phases until the animal exits its field and the neuron stops firing. Thus, the population of place cells not only encodes position through the identity of cells firing, but also through the precise timing of their activity with respect to theta. Since Hafting, Fyhn, Bonnevie, et al. (2008) showed that entorhinal cells exhibit phase precession in absence of the hippocampus, phase precession in the hippocampus is at least partially inherited from afferent regions. Interestingly, phase precession was absent in entorhinal neurons of layer III, the layer that connects to CA1, suggesting that the monosynaptic pathway does not convey such timing information. Neurons in entorhinal layer II, CA3 and CA1 do exhibit precession, and CA1 probably inherits it from CA3. Theta sequences are sequences of behavioral states that can be decoded from the activity of hippocampal populations of neurons during individual theta cycles. While this circuit-level phenomenon is probably facilitated by cell-level phase precession, some results suggest that phase precession is already present in the first traversal of a linear track while theta sequences rapidly emerge in subsequent experiences (Feng et al. 2015).
<figure><img src="image_21.png" /><figcaption>Figure 4.4: Theta-gamma code. As the animal traverses its environment, successive places are represented on successive gamma cycles nested within each theta cycle. Theta cycles represent a rolling window of states, such that spatial positions are represented earlier and earlier in successive theta cycles. Adapted from Drieu and Zugaro (2019).</figcaption></figure>

4.4.2 THETA-GAMMA CODE

Nested in theta waves of the hippocampus are oscillations of the gamma band (25-140Hz) that are approximately seven times faster than theta oscillations. In fact, increasing the number of bins dividing theta cycles helps decoding the position of the animal up to six bins (O. Jensen and Lisman 2000). Furthermore, place cells in CA1 typically fire during seven successive theta cycles (Skaggs, Bruce L McNaughton, et al. 1996). From these results emerged the view that hippocampal activity is organized with a theta-gamma code when the animal is active (Lisman and O. Jensen 2013). Information is buffered in the entorhinal cortex and sent to the hippocampus in such a way that each state of the ongoing sequence is represented in each successive gamma cycle of the ongoing theta cycle (Figure 4.4). As the animal moves and perceives new information, the new theta starts with a more advanced state and ends with a previously unrepresented state. This is reminiscent of a rolling window of states, or a queue data structure in computer science. Phase precession naturally emerges from this code, as successive states are represented earlier and earlier in the theta cycle.

Hasselmo, Bodelón, et al. (2002) reviewed a set of results describing the role of theta rhythm in scheduling memory operations. Information com-
ing from the entorhinal cortex is sent to the hippocampus at the peak<fnref n="2" /> of the theta cycle (Brankačk et al. 1993). Meanwhile, input from CA3 to CA1 is weak. In this same phase, CA3 synapses to CA1 can undergo long-term potentiation (LTP) Hölscher et al.; Wyble et al. (1997; 2000). At the trough of theta, inputs from the entorhinal cortex are weaker, while information in CA3 is transmitted to CA1 Brankačk et al. (1993). Meanwhile, LTP is weak and depotentiation can instead be induced Hölscher et al. (1997). Hasselmo, Bodelón, et al. (2002) proposed that the peak is the phase during which cortical information is sent for synaptic storage in the hippocampus, while stored memories are retrieved at the trough (Figure 4.5). This proposal derives from the standard framework of hippocampal function described in Section 4.5 stating that memories are stored in recurrent collaterals of area CA3.

Gamma oscillations are not uniform and their diversity provides key information regarding how CA1 integrates information from CA3 and the entorhinal cortex. In freely moving rats, Colgin et al. (2009) found that CA1 gamma oscillations could be divided into slow (25-50Hz) and fast (65-140Hz) gamma bands. Within each theta cycle, gamma power was usually concentrated in one of the two bands, and theta cycles with the same gamma bands were typically clustered. The EEG activity of CA3 and layer 3 of MEC were recorded simultaneously. During episodes of slow gamma, gamma activity of CA1 was synchronised with CA3, but not with MEC. On the other hand, episodes of fast gamma were characterized by high synchrony between CA1 and MEC, and not between CA1 and CA3. Furthermore, fast gamma oscillations were strongest at the theta trough, while slow gamma oscillations were strongest at the descending phase of theta.

Based on the observation that place cells sometimes employ prospective coding with theta sequences that represent locations ahead of the animal Battaglia et al. (2004), Bieri et al. (2014) analyzed the spatial coding modes at play during slow and fast gamma rhythms. Consistent with the prediction that prospective coding reflects memory retrieval and retrospective coding

<footnote n="2">²As recorded in the LFP of the stratum pyramidale of CA1.</footnote>
<figure><img src="image_22.png" /></figure>

Figure 4.5: Model of Hasselmo, Bodelón, et al. (2002). Left: The peak of theta is the phase of encoding, as input from the entorhinal cortex is strong and LTP is induced. Right: At the trough of theta, input from the entorhinal cortex is weak and CA1 is instead firing in response to CA3, which suggests this phase reflects the recall of past memories. Adapted from Kunec et al. (2005).
reflects encoding, CA3-associated slow gamma increased during prospec-
tive coding and EC-associated fast gamma increased during retrospective
coding. In the linear track, spikes occurred earlier during slow gamma
than during fast gamma, and CA1 cells retrieved upcoming positions dur-
ing slow gamma and encoded past positions during fast gamma.

If gamma cycles represent successive spatial positions as stated by the
theta-gamma code theory, slow gamma should encode shorter trajectories
than fast gamma. C. Zheng et al. (2016) in fact showed that slow gamma en-
codes longer trajectories as multiple locations are encoded in slow (but not
fast) gamma cycles. Fast gamma, on the other hand, represented very short
sequences. This is potentially related to the absence of phase precession in
layer III of the entorhinal cortex (Hafting, Fyhn, Bonnevie, et al. 2008).

While these results shed light on hippocampal information processing,
the identified phase of fast gamma (theta trough) is at odds with Hasselmo,
Bodelón, et al. (2002)'s view on the relationship between theta phases and
memory operations, as pointed out by Aguilera et al. (2022). On top of
that, these results suffer from limitations (Aguilera et al. 2022; Colgin 2015;
Schomburg et al. 2014). First, the tasks performed by the rats in these studies,
open field in Colgin et al. (2009) and linear track in Bieri et al. (2014), do
not require allocentric memory. Secondly, Colgin et al. (2009) used single-
site recording which prevents the characterization of the topology of CA1
oscillations and the identification of their source.

A finer study was performed by Schomburg et al. (2014) that overcame the
limitations of previous investigations. They recorded from up to 256 sites
along the transverse axis of the hippocampus while rats were perform-
ing navigational tasks with varying memory demands. Using independent
component analysis, they found that CA1 oscillations could be divided
into not two, but three distinct gamma bands, each of which supporting
different modes of communication within the hippocampus (Figure 4.6).
At the peak of theta, oscillations were dominated by medium gamma fre-
quencies (60-120Hz) located mainly in the stratum lacunosum-moleculare,
the source of which was the firing of EC3 cells. During the descending
phase of theta, slow gamma (30-80Hz) power increased, as CA3 sent input
<figure><img src="image_23.png" /><figcaption>Figure 4.6: Schematics of the three CA1 gamma bands identified by Schomburg et al. (2014). Adapted from Butler and Paulsen (2014).</figcaption></figure>

to the stratum radiatum. Finally, at the trough of theta, CA1 output firing was phase-locked to fast gamma (> 100Hz), a frequency band dominating the stratum pyramidale. Interestingly, the phases at which CA1 is synchronized with EC and CA3 correspond to the encoding and retrieval phases of Hasselmo, Bodelón, et al. (2002).

Finally, hippocampal dynamics and memory operations are also influenced by neuromodulation. In novel environments, the preferred phase of CA1 firing shifts to the peak of theta (Douchamps et al. 2013; Fernández-Ruiz, Oliva, Nagy, et al. 2017), consistent with Hasselmo, Bodelón, et al. (2002). This effect has been shown to depend on the release of acetylcholine as a novelty response (Douchamps et al. 2013). Dopamine is also thought to play a modulatory role in memory encoding (Lisman and Grace 2005).

### 4.4.3 SHARP WAVE RIPPLES

Another oscillatory pattern characterizing hippocampal activity is the sharp wave and its nested ripples (140-200Hz). While the theta-gamma state is dominant when the animal is active, ripples are characteristic of immobility, consummatory behaviors, and slow-wave sleep. They are generated in CA3 and transmitted to the dentate gyrus and CA1.
Ripples are associated to the phenomenon of replay, which was first observed independently of these oscillations. Pavlides and Winson (1989) recorded CA1 place cells and found those whose field was visited during waking behavior were more active during following sleep episodes. Skaggs and Bruce L McNaughton (1996) later showed that a place cell firing before another was likely to fire before in subsequent sleep, and conversely. Crucially, no such temporal bias correlation was found between awake activity and activity during the preceding sleep session. These results suggest that spatial trajectories experienced during awake behavior are learned and replayed during sleep, the temporal organization of which being preserved. These replay events happen during sharp-wave ripples at 10 to 20 times the speed of real trajectories (Nádasdy et al. 1999).

First, replay is thought to support memory consolidation. Hippocampal replay indeed co-occurs with replay in the cortex (Peyrache et al. 2009; Ji and M. A. Wilson 2007). Furthermore, suppressing ripple events during rest induces a significant memory deficit (Girardeau et al. 2009; Ego-Stengel and M. A. Wilson 2010). Replay-like events can also happen while awake (Kudrimoti et al. 1999), representing path to a goal or the trajectory leading to that goal once reached in reverse order (Foster and M. A. Wilson 2006; Diba and Buzsáki 2007; Pfeiffer and Foster 2013; Ambrose et al. 2016). Furthermore, replay-like events can represent never experienced trajectories (Gupta et al. 2010). Interestingly, the same phenomena can emerge in artificial agents seeking to maximize expected future rewards (Cazé et al. 2018; Mattar and Daw 2018), providing a normative account of their function beyond simple memory consolidation. These results indeed suggest that the hippocampus plays a role in planning, in line with fMRI studies showing the hippocampus is not only necessary for remembering the past, but also to imagine future-oriented novel situations (Hassabis, Kumaran, Vann, et al. 2007; Hassabis, Kumaran, and Maguire 2007).

Pezzulo et al. (2017) proposed a framework for explaining the many modes of hippocampal sequential activity according to the level of task engagement. In their view, prospective theta sequences support online predictive processing and online replays support planning, prospection and informa-
tion seeking. Offline replays either consolidate memory of recent experi-
ence when necessary or reflect a more disconnected sampling process of
an internal model of the world for self-consistency, model pruning and
homeostasis.

4.5 COMPUTATION: THE STANDARD FRAMEWORK, LIMITS
AND ALTERNATIVES

Marr (1971) proposed the first influential theory of hippocampal involve-
ment in memory functions. In essence, Marr proposed that the hippocam-
pus temporarily stores sensory information before sending it to the more
permanent neocortical store. As noted by Willshaw et al. (2015), Marr thus
laid the foundations for the complementary learning systems theory (Sec-
tion 3.3.1) and theories of consolidation (Section 3.4.1). He was very influ-
enced by a learning rule proposed by Hebb (1949). This rule serves as a core
mechanism for storing associative memories in his model. After storing
memories with Hebb's rule, the goal of the hippocampus is to complete
novel information with information stored in memory.

This model dominates the understanding of hippocampal function so
much that it has been termed the standard model (Nadel and Moscovitch
1997; Nadel, Samsonovich, et al. 2000) or standard framework (Cheng 2013).
The term framework is preferred considering that many details of the orig-
inal model of Marr have been discarded. Instead of being used as is, the
work of Marr has influenced the development of many models (Bruce L Mc-
Naughton and Morris 1987; Buzsáki 1989; Treves and Rolls 1992; Treves and Rolls
1994; O'Reilly and McClelland 1994; McClelland et al. 1995; Becker 2005; Kassab
and Alexandre 2018) bringing additional elements, some of which will be
described in the following sections.
### 4.5.1 MAIN PRINCIPLES

The standard framework of the hippocampus derives from the need for an initial memory store postulated by the complementary learning systems theory (Section 3.3.1). In order to avoid catastrophic forgetting in the cortex, novel information is first stored in the hippocampus rapidly and progressively incorporated into cortical networks. For this organisation to be useful, the hippocampus must not suffer itself from catastrophic interference. The root of this problem is that a neural network employing dense distributed representations cannot acquire new information without forgetting old information unless new information is presented along old information. The complementary learning systems states that the cortex solves this problem by being sent old information by the hippocampus to avoid forgetting it. The hippocampus is thought to employ a different solution, that is to avoid using dense distributed representations. If memories are represented and stored orthogonally from each other, with little overlap, their separation makes them resilient to the storage of new memories.

As mentioned in Section 4.2.2, the activity of the dentate gyrus is very sparse in the sense that only a few granule cells are active at a time. Sparsity naturally limits the overlap between activity patterns. Patterns are further sparsified when transmitted to CA3, due to the very low probability of synaptic contact from granule cells. Consequently, the dentate gyrus plays of a pattern separator and forces a different representation in CA3 during learning in the standard framework.

In the framework, CA3 is the locus of episodic memory storage. More precisely, memories are thought to be stored in its recurrent collaterals. Like a Hopfield network (J. J. Hopfield 1982; Section 4.5.2), these recurrent connections can be modified to encode memory patterns. After storage, a pattern can be retrieved from a partial cue represented by the current state of CA3 neurons, once information has been transmitted through the recurrent synapses to modify the state of CA3. Recurrent collaterals are thus thought to encode autoassociations. In Marr (1971) and most future models, the synaptic modification of recurrent collaterals implementing the
associative memory follows a Hebbian rule, which is captured by Hebb's iconic saying "Cells that fire together, wire together", or more formally: "any two cells or systems of cells that are repeatedly active at the same time will tend to become 'associated', so that activity in one facilitates activity in the other" (Hebb 1949, p. 70).

The standard framework does not give CA1 a role as consensual as the dentate gyrus and CA3 (Cheng 2013).

### 4.5.2 HOPFIELD NETWORK

The Hopfield network is the most well-known recurrent autoassociative memory network (J.J. Hopfield 1982) and its description provides a more complete understanding of the role of CA3 in the standard framework. The network is composed of thresholded units that are updated with the following rule:

$$s_i = \operatorname{sgn}\left(\sum_{j \neq i} W_{ji} s_j\right) \qquad (4.1)$$

where $s_i$ is the state of neuron $i$ and $W_{ji}$ is the strength of the connection from neuron $j$ to $i$ (typically $W_{ji} = W_{ij}$). The energy function of a Hopfield network associates a scalar value to each state and is defined as:

$$E = -\frac{1}{2} \sum_{i} \sum_{j \neq i} W_{ji} s_i s_j. \qquad (4.2)$$

As $s_i$ is updated, the energy function monotonically decreases:

$$\Delta E = -\Delta s_i \sum_{j \neq i} W_{ji} s_j. \qquad (4.3)$$

In this formulation of Hopfield networks (i.e. asynchronous update, $W_{ii} = 0$ and symmetric weights), they are proven to converge to local minima of their energy function (Bruck 1990), so-called stable states (or attractors).

What has been described so far is very similar to the earlier spin glass model (Lenz 1920; Ising 1925). The distinguishing characteristic of Hopfield
network is the use of Hebb (1949)'s learning rule to implement an associa-
tive memory, that is the ability to learn the relationship between items, by
manipulating attractors of the network. A Hebbian rule is used to modify
synaptic weights of the Hopfield network when learning N binary patterns:

$$
W_{ji} = \frac{1}{N} \sum_{\mu=1}^{N} \epsilon_{i}^{\mu} \epsilon_{j}^{\mu} \qquad (4.4)
$$

where $\epsilon_i^\mu$ is bit $i$ of pattern $\mu$. The goal with this rule is to make patterns to be stored in a Hopfield network attractor of that network, with the aim of retrieving the patterns when altered versions (e.g. noisy or incomplete) are presented. Intuitively, this can be understood as the formation of attracting and repelling connections: when two bits tend to be positive and negative together ($\epsilon_i^\mu \epsilon_j^\mu = 1$), their neuron will learn to attract each other in state space and conversely, neurons with opposite states will learn to repel each other.

While Hopfield networks have greatly influenced the understanding of
memory in both biological and artificial agents, their initial formulation
suffers from serious limitations. First and perhaps most obvious is that
the threshold activation function does not allow the storage of continuous
patterns (but see the continuous generalization of J. J. Hopfield (1984)). Sec-
ondly, as there is no distinction between sensory and memory units, the
number of patterns that can be stored is thus limited by their dimension-
ality *d* which is equal to the number of neurons. The storage capacity for
retrieval of patterns free of errors is *C* ≈ *d*/(2log₂*d*) and approximately 0.14*d*
with a small percentage of errors. The capacity is drastically decreased
in the case of correlated patterns. Secondly, it is not clear how Hopfield
networks can be adapted to model memory for non-static information. In
principle, metastable states could emerge using a nonsymmetric Hebbian
rule (allowing for weights to be nonsymmetric). Sequences of patterns
could thus be retrieved using the original state update rule, each minimum
being replaced in time by another. However, J. J. Hopfield (1982) reported
failing to faithfully encode and retrieve sequences of four states and more.
4.5.3 CA3 AS A TEMPORAL ASSOCIATOR

The standard framework most often leaves aside the topic of sequence
learning and consequently misses a central aspect of episodic memory.
A strong body of evidence in fact indicates that the hippocampus is in-
volved in rapid learning of sequential information (Agster, Fortin, et al. 2002;
Fortin et al. 2002; Kesner et al. 2002; A. K. Lee and M. A. Wilson 2002; Ergorul
and Howard Eichenbaum 2004) and can disambiguate overlapping sequences
using splitter cells (Duvelle et al. 2023). Sequence memories are expressed
during replays and prospective theta sequences. This cannot be accounted
for by computational models that store static patterns of information.

However, a few models which are otherwise compatible with the stan-
dard framework have tackled the problem of sequence learning in the hip-
pocampus (e.g. Levy 1982; Levy 1996; Wallenstein and Hasselmo 1997; Wallen-
stein, Hasselmo, and Howard Eichenbaum 1998). These models propose that
CA3 does not only store patterns corresponding to the current representa-
tion of the environment, but that it is also capable of learning association
between discontiguous events. The models of Levy (1982) and Levy (1996)
exploit the sparse connectivity between the dentate gyrus and CA3 to pre-
vent certain CA3 neurons to be clamped to external inputs and instead keep
track of past states through recurrent collaterals. As a result, the global
state of CA3 at each time point represents a mixture of environment inputs
provided by dentate gyrus cells, and temporal context formed through re-
currence. Recurrent CA3 connections are also constrained to be sparse in
order to form attractors that are not stable through time and recall multi-
ple items successively. Using CA3 plasticity mechanisms similar to those
of other standard models but with sparse connections grants these mod-
els with sequence learning capabilities. Plasticity indeed stores patterns
representing the conjunction of discontiguous items and sequences can be
recalled with a process similar to standard models applied iteratively.
### 4.5.4 EVIDENCE AGAINST THE STANDARD FRAMEWORK

While very influential and accounting for some biological phenomena, the standard framework suffers from limitations. In fact, Hebb's learning rule which is a central tenet of the standard framework is computationally sub-optimal, as the small capacity of Hopfield networks suggests. Ketz et al. (2013) proposed a model in which the hippocampus does not learn using an Hebbian rule, but rather an error-driven mechanism scheduled by theta phases, like in the model of Hasselmo, Bodelón, et al. (2002). The authors indeed showed that the hippocampus is a much more powerful learner when this type of learning rule is used instead of the classical Hebbian mechanism. In a more recent version of the model (Y. Zheng et al. 2022), the entorhinal-to-CA3 and recurrent CA3 connections are also trained with an error-driven mechanism, further improving memory capacity and learning speed.

After its original proposal, the model of Ketz et al. (2013) has revealed deeper implications. The standard framework states that the engram (i.e. the substrate of memory) is located at the recurrent connections of CA3. This is at odds with the more consensual view that memory engrams are distributed. Studies suggest the hippocampus not only remembers individual episodes but is also involved in the discovery of statistical regularities among them (Mehta et al. 1997; Strange, Duggins, et al. 2005; Turk-Browne et al. 2009; Schapiro, Kustner, et al. 2012; Schapiro, Turk-Browne, Norman, et al. 2016). In the study of Schapiro, Turk-Browne, Norman, et al. (2016), human participants were shown abstract visual stimuli transitioning according to a community structured graph (Figure 4.7a). The activation of the hippocampus assessed by fMRI not only reflected individual transitions, but the higher order community structure too (Figure 4.7c). Activity evoked by stimuli belonging to a common community was more similar than activity evoked by stimuli of different communities. Schapiro, Turk-Browne, M. M. Botvinick, et al. (2017) investigated how can the hippocampus be involved in both the rapid acquisition of episodes composed of a sequence of transitions and a slower statistical form of learning highlighted by the community structure
task. They discovered that the model of Ketz et al. (2013) can in fact ac-
quire both kinds of information while being initially designed to simulate
episodic memory. Like in humans, higher order structure is captured by
the model, such that the response to items of the same community is more
similar than response patterns of different communities (Figure 4.7d). Most
importantly, the structure is better captured by CA1 responses than by the
dentate gyrus and CA3. Properties of the trisynaptic and monosynaptic
pathways in the model, such as their respective fast and slow learning rate,
constrain hippocampal regions to acquire information of different nature.
These results suggest that memories are distributed throughout the hip-
pocampus but do not rule out the possibility that true episodic memories
(i.e. individual items and transitions) are stored in CA3. Schapiro, Turk-
Browne, M. M. Botvinick, et al. (2017) proposed to extend the complementary
learning systems with the monosynaptic pathway as an intermediate learn-
ing system between the truly episodic learner (trisynaptic pathway) and the
truly statistician (cortex).

Cheng (2013) argued for the lack of evidence in favor of CA3 recurrent
plasticity being necessary for rapid learning. Some studies did find deficits
in that ability after experimental suppression of plasticity in CA3 (e.g.
Nakazawa et al. 2003; Cravens et al. 2006) but these deficits were not always
observed. In a Morris water maze experiment involving new platforms
everyday Nakazawa et al. (2003), no deficit was found on days 5-12. In a
short exposure contextual fear conditioning study Cravens et al. (2006), mu-
tant mice performed as well as controls when tested after 24 hours. Since
the deficit could not be assessed in all conditions, Cheng (2013) argued that
CA3 plasticity is not necessary per se. Another study (Kishimoto et al. 2006)
used the eyeblink conditioning paradigm and showed the hippocampus
is necessary to perform the task, but CA3 plasticity is not necessary for
acquisition, only for extinction (gradual decrease in conditioned response
when the reinforcement is removed). Some studies in fact suggest that CA3
responses do not necessarily change significantly during learning (Dupret
et al. 2010; Miyashita et al. 2009), in comparison with area CA1 whose activity
is reorganized. Another prediction of the standard framework is that the
<figure><img src="image_24.png" /></figure>

(a) Graph governing the transition between abstract visual stimuli. The graph is structured with three communities. In each community, only two nodes connect with nodes from other communities. Adapted from Schapiro, Turk-Browne, Norman, et al. (2016).

<figure><img src="image_25.png" /></figure>

(c) Dimensionality reduction applied on the fMRI activation of the human hippocampus following the presentation of the visual stimuli depicted in Figure 4.7a. The representation of stimuli belonging to the same community are clustered together. Adapted from Schapiro, Turk-Browne, Norman, et al. (2016).

<figure><img src="image_26.png" /></figure>

(b) Successor representation reflects the community structure in a simulation of the task. Each state corresponds to a node of the graph represented in Figure 4.7a. Adapted from Stachenfeld et al. (2017).

<figure><img src="image_27.png" /></figure>

(d) Settled response of the model proposed by Ketz et al. (2013) in a simulation of the community structure task performed by Schapiro, Turk-Browne, M. M. Botvinick, et al. (2017). The symmetric heatmaps show the average representational similarity for each node pair of the graph depicted in Figure 4.7a. The diagonal represents the correlation between nodes with themselves. Adapted from Schapiro, Turk-Browne, M. M. Botvinick, et al. (2017).

**Figure 4.7:** Representation of stimuli whose transitions are governed by a community structure, in the human and simulated hippocampus.
storage of novel non-separated patterns in CA3 should interfer with older
memories. The dentate gyrus has indeed been found to separate patterns
and to be necessary for learning new associations that are similar to pre-
viously stored ones (P. E. Gilbert et al. 2001; McHugh et al. 2007). However,
Cheng (2013) noted that trying to learn new similar associations without the
dentate gyrus did not interfere with previous memories in these studies.

Studies also suggest that the role of the dentate gyrus is overlooked by
the standard framework. If the dentate gyrus is a simple pattern separa-
tor that orthogonalizes the patterns to be stored in CA3, it should not be
necessary for memory recall. In accordance with this proposal, studies that
experimentally inactivate the dentate gyrus typically find no impairment
in memory recall (e.g. Lassalle et al. 2000). However, inactivating the dentate
gyrus did impair retrieval in some other studies, suggesting that memory
traces are also located in that region (Denny et al. 2014; S. Park et al. 2016;
Bernier et al. 2017). Furthermore, the dentate gyrus is reactivated during
retrieval of recent (but not remote) memories (Tayler et al. 2013). In a recent
review, Hainmueller and Bartos (2020) proposed that memories are stored in
the EC-to-DG and DG-to-CA3 synapses at an early stage and that granule
cells support the retrieval of CA3 activity patterns minutes to hours after
the experience, while recurrent collaterals of CA3 may still be weak. After
successive sharp wave ripple events involving newborn granule cells and
consolidating CA3 synapses, memories can be retrieved in absence of the
dentate gyrus.

The standard framework is appealing in its simplicity, but this growing
body of evidence depicts a much more complex reality calling for alterna-
tive theories of learning in the hippocampus.

4.5.5 ALTERNATIVE THEORIES

SOCRATIC

A distinct model of the hippocampus was proposed with a focus on the
learning of sequences of items (Lisman 1999). This model was called So-
CRATIC (i.e. Sequences Of Condensed Representations, Autocorrected, Theta-
Gamma Coded, In Context), and was used to account for early symptoms of schizophrenia (Lisman and Otmakhova 2001). In this model, the DG-CA3 loop stores the sequence of items sent by the entorhinal cortex at a theta-nested gamma rhythm. This model is at the origin of the theta-gamma code described in Section 4.4.2. Here, a later revised version of the model is described in which the respective functions of DG and CA3 were swapped (Lisman, Talamini, et al. 2005). This revised model is illustrated in Figure 4.8. One main idea behind this model is that the temporal separation of events happening in the environment is too big for plasticity to learn their associations and how they unfold through time. To resolve this issue which is central to episodic memory and its temporal nature, the entorhinal cortex is thought to act as a short-term memory buffer sending information to the hippocampus at the right cadence for long-term memory storage (O. Jensen and Lisman 2005; Schon et al. 2016). The stream of events is thus chunked, forming a discrete sequence code in which each item is represented in one gamma cycle (Figure 4.4). According to the revised SOCRATIC model, items are sequentially sent to the hippocampus and CA3 recurrent collaterals learn autoassociations like in the standard framework. However, the CA3-to-DG backprojection is too slow to learn the autoassociation and instead the association between one pattern represented in CA3 and the next pattern represented in DG. This division of labor is thought to support robust sequence learning. While the CA3-to-DG heteroassociation process enables the retrieval of (a corrupted version of) the next item in the sequence, the autoassociation CA3 process is important for cleaning the result such that prediction errors do not accumulate (Figure 4.8, inset). As mentioned above, the initial proposal was that CA3 learns heteroassociations and DG learns to correct them but the exchange of their respective function was later suggested in alignment with the standard framework (Lisman, Talamini, et al. 2005).
<figure><img src="image_28.png" /><figcaption>Figure 4.8: Revised SOCRATIC model. Taken from Lisman, Talamini, et al. (2005).</figcaption></figure>

CRISP

CRISP is another hippocampus model that focuses on sequence learning. It was proposed in Cheng (2013) after the review of many arguments against the standard framework. The main idea is that CA3 generates intrinsic sequences that do not necessarily evolve with learning and that sequences of events get mapped to these intrinsic sequences by fast modification of EC-to-CA3 and CA3-to-CA1 synapses. CA1 plays the role of pattern completion such that the monosynaptic pathway is sufficient to learn static patterns while the monosynaptic pathway is necessary to learn sequences. In that model, it is hypothesized that the dentate gyrus can reset the intrinsic CA3 sequence in order to learn new sequences and avoid interference between similar ones, but also to initiate recall.

MODERN HOPFIELD NETWORKS

In the recent years, there has been a resurgence of research related to as-
sociative memory. A family of novel algorithms, dubbed dense associative
memories or modern Hopfield networks, have been proposed to overcome the limitations of the original Hopfield network, as reviewed by Krotov (2023). They do not constitute a model of the hippocampus properly speaking, but their success provides valuable information to develop better hippocampal models.

Krotov and J. J. Hopfield (2016) proposed modifications in the energy function and dynamics of the network leading to a super-linear scaling relationship between pattern dimensionality and capacity. Demircigil et al. (2017) extended the idea of Krotov and J. J. Hopfield (2016) and proved storage capacity can even be exponential with an exponential activation function. Ramsauer et al. (2020) proposed another variant of modern Hopfield network based on previous work (Sukhbaatar et al. 2015) that instead makes use of the softmax function. Interestingly, the introduced update rule is equivalent to the self-attention mechanism of transformers (Vaswani et al. 2017):

$$z = X\mathrm{softmax}(\beta X^{\top} q) \qquad (4.5)$$

where $q$ is the query vector, $X$ is the memory matrix and $\beta$ is an inverse temperature parameter controlling how much memories are separated. Softmax is the normalized exponential function:

$$\mathrm{softmax}(\mathbf{x})_i = \frac{e^{\mathbf{x}_i}}{\sum_j e^{\mathbf{z}_j}} \qquad (4.6)$$

Universal Hopfield networks were introduced as a general framework for associative memory models (Millidge et al. 2022). The idea is that classical and modern Hopfield networks can be decomposed into a similar series of operations: similarity, separation and projection (Figure 4.9). Retrieval from a universal Hopfield network is thus defined as:

$$z = V \cdot \mathrm{sep}(\mathrm{sim}(K, q)), \qquad (4.7)$$

where $q$ is the query vector, $K$ is the matrix of stored keys, $\mathrm{sim}$ is the similarity function, $\mathrm{sep}$ is the separation function and $V$ is the matrix of stored
<figure><img src="image_29.png" /><figcaption>Figure 4.9: Universal Hopfield network operations. The input is compared to stored memories (similarity), the relative comparisons are then ex-aggerated (separation) and used to construct an output (projection). Taken from Millidge et al. (2022).</figcaption></figure>

values. The concepts of key and value are used to account for heteroasso-
ciation where a key element is associated to a distinct value element, but
in most applications of Hopfield networks the value is used to reconstruct
the same key element (i.e. autoassociation with V = K†). In the case of the
original Hopfield network, the similarity function is the dot product and
the separation function is the simple identity function. The lack of a more
elaborate separation function, like the softmax of the more modern model
of Ramsauer et al. (2020), makes it impossible for the classical Hopfield net-
work to properly separate memories.

The main advantage of modern Hopfield networks over classical ones
is that they are not restricted to binary data and are easily integrated into
deep learning architectures. Synaptic connections are then trained using
the back-propagation algorithm (Linnainmaa 1976; Rumelhart et al. 1986). It
is not yet clear to what extent modern Hopfield networks can be trained
under more restrictive regimes without impairing their storage capacity. A
strength of the classical Hopfield network is its biological plausibility: both
the interactions between units and the learning of information are simple
enough to be implemented by real neurons. On top of that, information can
be presented sequentially and memorized after a single presentation. On the other hand, Krotov and J. Hopfield (2020) noted how their modern variants involve many-unit interactions that blur the parallel between their connections and biological synapses that typically involve two neurons only. They proposed to replace many-unit interactions by a network of feature neurons and memory neurons that could be implemented as a traditional artificial neural network, which was later demonstrated by Snow and Orchard (2022). Note, however, that the storage capacity becomes linear in the number of memory neurons (Krotov and J. Hopfield 2020).

The use of back-propagation further undermines the plausibility of modern Hopfield networks and their ability to rapidly memorize novel information. Back-propagation is typically employed under a training regime in which information is presented either many times sequentially or as a single batch (as done by Snow and Orchard 2022). Both cases do not align with the requirements of episodic memory which should rapidly acquire new information presented sequentially. In some works, the patterns to memorize are instead added to the weight matrices manually (e.g. Millidge et al. 2022; Spens and Burgess 2023). Recently, Tyulmankov et al. (2021) proposed a learning rule for storing information in a more plausible way. In order to encode a memory, a neuron in the separation layer (i.e. the memory neurons of Krotov and J. Hopfield 2020) is selected for activation, either randomly or through a sequential process. The input pattern is then associated to the activated neuron using a neoclassical three-factor learning rule (Gerstner et al. 2018), such that the memory is stored in the row of the key matrix corresponding to the selected neuron. A hebbian rule is then used to store the pattern in the column of the value matrix corresponding to the neuron. When using this biologically-driven approach with a sequential selection process, or when patterns are manually added to the matrices, the storage capacity is equal to the number of neurons in the separation layer (Tyulmankov et al. 2021). When selecting a separation neuron randomly, the capacity is sublinear and comparable to the capacity of traditional Hopfield networks. Tyulmankov et al. (2021) also demonstrated that modern Hopfield networks can learn sequences by sending the current pattern for key stor-
age and the next pattern for value storage. The network can then be used
to recall sequences by recurrently using the output as the next input.

Much like the way non-recurrent transformers (Vaswani et al. 2017) led to
the development of performant language models that are now compared
to language processing in biological brains (e.g. Caucheteux and King 2022),
the modern feedforward variants of Hopfield networks will certainly in-
fluence the understanding of memory storage in the hippocampus. In fact,
it has already been demonstrated that the popular model of Whittington,
T.H. Muller, et al. (2020) is performant with no recurrence and that incor-
porating elements of transformers improves its performance (Whittington,
Warren, et al. 2021). Furthermore, it has been proposed that the hippocam-
pus implements a modern associative memory in CA3 and CA1 neurons,
although the precise anatomical mapping is not clear (Krotov and J. Hopfield
2020; Chandra et al. 2023; Spens and Burgess 2023). Their sequence learning
ability having been established (Chaudhry et al. 2023; Tyulmankov et al. 2021;
Millidge et al. 2022), modern Hopfield networks could be used to model
sequence processing in the hippocampus.

## SUCCESSOR REPRESENTATION

The successor representation was introduced by Dayan (1993) in the context
of reinforcement learning, an artificial intelligence paradigm focused on
developing agents that can learn to choose actions for maximizing their
rewards. Later on, it has influenced theories of learning in the hippocam-
pus. Using the formulation of Stachenfeld et al. (2017), one can define<fnref n="3" /> the
value V of a state s as the expected sum of rewards R obtained in future
states s<sub>t</sub>:

$$
V(s) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t) | s_0 = s \right] \quad (4.8)
$$

<footnote n="3">³This is well defined in a Markov process, that is when the transition between states does not depend on past states.</footnote>
where $\gamma \in [0, 1]$ is an exponentially decaying discount factor. Dayan (1993) factorized this value function into the inner product of the reward function and a matrix $M$ predictive of successor states:

$$V(s) = \sum_{s'} M(s, s') R(s'). \qquad (4.9)$$

This successor representation matrix is the expected future occupancy of state $s'$:

$$M(s, s') = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t \delta(s_t, s') | s_0 = s \right], \qquad (4.10)$$

where $\delta(s_t, s')$ is the Kronecker delta function whose value is 1 if $s_t = s'$ and 0 otherwise.

It is possible to learn this predictive matrix using an error-driven temporal difference algorithm that incrementally updates the estimated successor representation to minimize the discrepancy between expected and observed state occupancy (Dayan 1993; White 1996). It was later argued that this algorithm can be used to encode episodic memories and reproduce well-known psychological effects (Gershman, C. D. Moore, et al. 2012). The authors made a connection between their model and the temporal context model, the idea of which is to learn the association between individual items and a vector keeping track of the recent context (Howard and Kahana 2002). More recently, Stachenfeld et al. (2017) showed that when states encode space and a column of the successor representation matrix is interpreted as a neuron, the activation of this neuron resembles a place field, giving rise to the theory that hippocampal neurons encode such a prediction of future states. The cognitive map would thus be a predictive map. Since the matrix is influenced by the behavior of the agent and by transitions between states, these place fields can exhibit the experience-dependent asymmetric backward expansion described by Mehta et al. (1997) and the barrier effect of Alvernhe et al. (2011). Successor representation can also account for the activity of the hippocampus in the task of Schapiro, Turk-Browne, Norman, et al. (2016), as shown in Figure b. Finally, Stachenfeld et al. (2017) found that
the eigenvectors of the matrix resembled the activity of grid cells in various
situations.

Momennejad and Howard (2018) used multiple γ values for multi-scale rep-
resentation, drawing inspiration from the gradient of representational scale
along the longitudinal axis of the entorhinal cortex and the hippocampus.
They showed that this representation can be used for both reconstructing
sequences of experienced states and predicting the distance between states.

While being a promising normative theory, it is not clear how the hip-
ocampus learns such a predictive map, but recent modeling work sug-
gests that theta sequences can support the learning of successor represen-
tation (George, Cothi, Stachenfeld, et al. 2023).

4.6 DISCUSSION

In this chapter, the anatomy, representations and dynamics of the medial temporal lobe have been presented in terms of episodic memory operations. The computational model developed in Chapter 8 will be designed to follow the same architecture. This new model constitutes a viable alternative to the standard framework for sequence learning, and hippocampal-like representations will be shown to emerge.

The present chapter presents a complex image of the hippocampus, with
intertwined processing modes and functions. Although it may function
partially autonomously, the next chapter describes how the hippocampus
is also controlled by executive regions such as the prefrontal cortex.
# 5 EPISODIC CONTROL AND CONTROLLED EPISODIC MEMORY

<figure><img src="image_30.png" /></figure>

An agent remembering an episodic memory and controlling which action to take accordingly

Generative AI
**5.1 INTRODUCTION**

An influential view is that cognition is centered around action, in that it has
evolved as a way of selecting the appropriate motor commands<fnref n="1" />. While the
level of appropriateness of an action might seem difficult to formalize at
first glance, the gene-centered view of evolution (Trivers and Dawkins 1976)
provides an objective criterion: an action is deemed appropriate when the
level of propagation of one's genes is increased as a consequence of that
action. The action-centric view boils down to the proposal that cognition
mainly aims to compute that criterion and execute the actions that are most
appropriate.<fnref n="2" /> The four Fs are often given as primal examples of evolution-
ary adapted behaviors: *fighting*, *fleeing*, *feeding* and *mating*, but higher-level
cognition allows for much richer behaviors with indirect benefits (e.g. lan-
guage and social interactions). This appeals to the behaviorist notions of
primary reinforcers (e.g. food, water, pleasure, sleep), which probably act
as an innate proxy for the evolutionary advantage criterion, and secondary
reinforcers (e.g. money) which are associated with primary ones, as cap-
tured by the model of O'Reilly, M. J. Frank, et al. (2007).

A direct consequence of the action-centered view of cognition is that
cognitive functions are instrumental to action selection. This has mostly
been studied in the context of perception, and there is a lot of evidence
that perception is indeed affected by the ability to act. For example, soft-
ball players with better batting performance tend to perceive the ball as
bigger than players with poorer performance (Witt and Proffitt 2005). In the
same vein, tennis players tend to perceive the ball as slower when they suc-
cessfully return it (Witt and Sugovic 2010). Crucially, only intended actions
appear to have a modulatory effect on perception (Witt, Proffitt, and Epstein
2010). These thrilling results have led to the development of action-specific
theories of perception (Witt 2011; Proffitt 2006) which radically depart from

<footnote n="1">¹Teleological phrasing is used for convenience throughout the text without implying evolution is goal-directed.</footnote>

<footnote n="2">²Since the criterion can be very hard to compute, this does not imply that all actions selected by cognitive functions are appropriate.</footnote>
the early conception that perception is a passive process, the function of
which is to depict reality objectively.

Similarly, it is likely that memory has evolved to inform behavioral con-
trol. Memory of past experience can indeed provide crucial information
about the potential consequences of one's actions and can therefore be
used to make better decisions. Memory systems thus have an incentive
to encode, store and retrieve informative memories. In this chapter, the
goal-directed use of episodic memory is discussed. First, the anatomy of
the frontal lobe and connections with the hippocampus will be described in
Section 5.2. The involvement of the prefrontal cortex in episodic memory
will then be described succinctly (Section 5.3). In Section 5.4, hippocampal
representations will be considered in terms of context formation for the
recall of relevant memories that can be sent to the frontal cortex to guide
decision-making. In Section 5.5, the dynamics of prefrontal-hippocampal
interactions will be discussed, and it will be shown that the prefrontal cor-
tex is not only informed by the hippocampus but also modulates memory
operations actively. Finally, Section 5.6 presents computational models re-
lated to the interactions between memory and control.

5.2 ANATOMY

The frontal lobe is a key structure of motor and cognitive control (Goldman-Rakic 1996). It comprises the prefrontal cortex (Figure 5.1), the premotor cortex and the primary motor cortex. It is hierarchically organized along a rostrocaudal axis, such that more rostral regions represent successively more abstract task information (Koechlin et al. 2003; Badre and D’Esposito 2007; Nee and J. W. Brown 2012; J. R. Reynolds et al. 2012). At the lowest level is the premotor cortex which learns the association between sensory stimuli and motor responses with the recruitment of the basal ganglia (i.e. sensorimotor rules; Weinrich et al. 1984; Brasted and Wise 2004). Higher levels (more rostral regions) lie within the prefrontal cortex and learn to modulate lower levels (more caudal regions) according to more abstract rules. A
<figure><img src="image_31.png" /><figcaption>Figure 5.1: Major brain regions supporting cognitive control. Taken from Purves et al. (2013).</figcaption></figure>

detailed review of the anatomy of the frontal cortex is outside the scope of
this thesis (see Dagar 2023 for an extensive review on the PFC and cognitive
control), but a broad description of the medial and lateral divisions of the
prefrontal cortex is first provided to better detail its interactions with the
hippocampus.

The medial prefrontal cortex (mPFC) comprises the ventromedial PFC (vmPFC), the orbital PFC (OFC) and the dorsomedial PFC (dmPFC). The dmPFC itself comprises the anterior cingulate cortex (ACC). As reviewed by Dagar (2023), the mPFC guides the process of decision making (Kennerley et al. 2011) by evaluating the value of choices (Jocham et al. 2011; Lim et al. 2011) and taking motivation into account (Kouneiher et al. 2009; Holroyd and Yeung 2012). It monitors outcome and error (M. Rushworth et al. 2004; M. F. Rushworth et al. 2007) and the value of the course of action pursued (Hayden and Platt 2010; Noonan et al. 2011). The mPFC is thus thought to monitor behavioral plans by computing prediction errors and initiate rule switching when deemed appropriate.

The lateral prefrontal cortex is divided into dorsal (dlPFC) and ventral (vlPFC) regions. The lateral PFC is thought to support working mem-
ory as suggested by lesion and inactivation studies (M. H. Miller and Orbach 1972; Bauer and Fuster 1976; Funahashi et al. 1993). Neurons of the dlPFC in fact show sustained activity during working memory tasks (Fuster and G. E. Alexander 1971; Kubota and Niki 1971; Niki and Watanabe 1979; Funahashi et al. 1989). The lateral PFC is thus thought to be responsible for maintaining a representation of the current plan and to support the learning of rules that require short-term memory.

In order for the hippocampus to guide cognitive control and for the pre-
frontal cortex to modulate memory operations, these structures must ex-
change information through anatomical circuits. Both the hippocampus
and the prefrontal cortex are extensively connected to cortical and subcor-
tical regions in such a way that these two regions can indirectly interact in
many ways. There are also direct pathways connecting them, most studies
focusing on connections with the medial prefrontal cortex (Howard Eichen-
baum 2017b). Interactions between the hippocampus and the lateral pre-
frontal cortex are likely to be mediated by strong bidirectional connections
between the lateral and medial prefrontal cortices (Anderson et al. 2016). The
most direct hippocampus-to-prefrontal cortex pathway is the monosynap-
tic projection from CA1 of the ventral to intermediate hippocampus (taking
the rodent longitudinal axis as a reference) to the mPFC, including the OFC
(Jay, Glowinski, et al. 1989; Jay and Menno P Witter 1991; Hoover and Vertes 2007).
More ventral portions of the hippocampus are thought to represent broad
information, as exemplified by the big scale of ventral place fields reviewed
in the previous chapter. This pathway is thus well suited to provide con-
textual information to the prefrontal cortex.

A more indirect pathway involves the thalamic nucleus reuniens (Dolleman-
Van Der Weel and M. Witter 1996; Vertes 2002; Vertes 2006; Vertes, Hoover, et al.
2007; Cassel et al. 2013; Dolleman-van der Weel et al. 2019). This structure bidi-
rectionally connects CA1 (all portions) and all medial prefrontal areas. It
also communicates with the perirhinal cortex and the entorhinal cortex.
A lesser known subcortical route involves the small supramammillary nu-
cleus of the hypothalamus, which receives weak projections from the me-
dial prefrontal cortex (F. Wouterlood et al. 1987; Allen and Hopkins 1989; Sesack
et al. 1989; Gonzalo-Ruiz et al. 1992) and projects to the dentate gyrus and area
CA3 of the hippocampus (Pasquier and Reinoso-Suarez 1976; Wyss et al. 1979;
Vertes and McKenna 2000; M. Li et al. 2023).

The medial prefrontal cortex is also directly and reciprocally connected
to the perirhinal and entorhinal cortices (Burwell and Amaral 1998; Menno P
Witter, Floris G Wouterlood, et al. 2000; Apergis-Schoute et al. 2006; Agster and
Burwell 2009). The mPFC projects to superficial layers of the perirhinal cor-
tex and deep layers of the lateral entorhinal cortex (Burwell and Amaral 1998).
It in turn receives projections from these regions that terminate in its lay-
ers I, II and VI (Agster and Burwell 2009). Medial prefrontal projections to
the entorhinal cortex mainly terminate in the lateral compartment (Burwell
and Amaral 1998) which, as reviewed in the previous chapter, connects the
hippocampus to cortical regions specialized in semantic processing. This
suggests that this pathway modulates semantic information to a larger ex-
tent than spatial information.

Finally, it has been reported by Rajasethupathy et al. (2015) and Bian et al. (2019) that ACC neurons project (very sparsely) to CA3 and CA1 neurons. These findings contrast with the previous consensus and have important functional implications since they potentially highlight a straightforward way for the prefrontal cortex to modulate hippocampal activity. Nevertheless, Andrianova et al. (2023) employed multiple complementary methods to validate this discovery and found no evidence of such a direct monosynaptic projection from ACC to the hippocampus.

As reviewed here, the hippocampus and the prefrontal cortex can com-
municate in many ways. However, the precise pathways underlying this
bidirectional communication are difficult to establish due to the involve-
ment of relay structures and inconsistencies between studies. The studies
described in the following sections employ a combination of behavioral,
functional connectivity and selective lesion analyses that point towards a
model of prefrontal-hippocampal interactions.
## 5.3 CONTRIBUTIONS OF THE PREFRONTAL CORTEX TO MEMORY

Before detailing prefrontal-hippocampal interactions, the prefrontal con-
tribution to memory is introduced. Despite early reports indicating that
patient K.M., who had prefrontal damage, had no memory deficits, it is
now well established that the prefrontal cortex plays an important role in
memory (Janowsky et al. 1989; Moscovitch 1992; Shimamura et al. 1995; Schnider
and Ptak 1999; E. K. Miller and J. D. Cohen 2001; Dobbins et al. 2002; Preston and
Howard Eichenbaum 2013; Szczepanski and Knight 2014; Howard Eichenbaum
2017b; S. A. Park et al. 2023). The prefrontal cortex is indeed thought to de-
ploy top-down control over memory processes. Just as it uses attentional
mechanisms to filter out distracting stimuli in the environment, the pre-
frontal cortex selects which memories are relevant to reach current goals.
Many studies indeed show that prefrontal damage leads to intrusions of
irrelevant memories (Schnider and Ptak 1999; Shimamura et al. 1995; Farovik
et al. 2008; Chudasama et al. 2012; Peters et al. 2013; Giustino and Maren 2015;
Anderson et al. 2016). Evidence from rodent and primate studies suggest
that this retrieval control ability is either mediated by connections from the
prefrontal cortex to the entorhinal cortex controlling the input and output
of the hippocampus, or by the nucleus reuniens which could convey mod-
ulatory information directly to the hippocampus (Anderson et al. 2016). The
prefrontal cortex is also thought to control memory encoding by inhibiting
the hippocampus when new information is consistent with pre-existing
knowledge to avoid encoding redundant information (Frankland and Bon-
tempi 2005; Van Kesteren et al. 2012).

5.4 REPRESENTATION: THE PREFRONTAL-HIPPOCAMPAL
SYSTEM AS A CONTEXTUAL MEMORY MAP

In the previous chapter, splitter cells have been introduced as neurons not
only coding for where the animal is located, but also for where it comes
from (retrospective splitter cells) or where it is about to go (prospective
splitter cells). One might wonder how the hippocampus has access to the
latter information. The dentate gyrus and CA3 circuitry is known to un-
derlie CA1 retrospective coding (Keinath et al. 2020), but its contribution to
prospective coding is unknown (Duvelle et al. 2023). Trajectory-dependent
activity can be found in the medial entorhinal cortex but lesion studies
suggest that it is not necessary for CA1 activity to be splitted (Sabariego
et al. 2019). Splitter cells can also be found in the prefrontal cortex (Baeg
et al. 2003; Young and Shapiro 2011; Cowen et al. 2012) and nucleus reuniens
(Ito, Zhang, et al. 2015; Ito, E. I. Moser, et al. 2018). Inactivating the medial pre-
frontal cortex (Guise and Shapiro 2017) or the nucleus reuniens (Ito, Zhang,
et al. 2015) both impact splitter coding in CA1, and this impairment seems
to be specific to prospective splitter cells (Ito, Zhang, et al. 2015). It was also
shown by Ito, E. I. Moser, et al. (2018) that silencing the supramammillary
nucleus reduces splitter coding in CA1 and the nucleus reuniens, but not
in the prefrontal cortex. Hence, a plausible view is that the retrospective
splitting signal is generated internally while the prospective signal is gen-
erated in executive structures like the prefrontal cortex and transmitted to
the hippocampus through relay structures such as the nucleus reuniens of
the thalamus. An alternative theory is that the prospective splitting signal
is conveyed by the entorhinal cortex, as this structure is thought to mediate
prefrontal control over memory processes, and that the nucleus reuniens
orchestrates this communication (Section 5.5; Howard Eichenbaum 2017b).

The hippocampus can thus be seen as representing a temporal and goal-
directed context map for retrieving memories that are most appropriate
and can best inform executive structures like the prefrontal cortex. It in-
deed takes into account the recent past through an internally-generated
retrospective splitting signal, the current sensory observation provided by
the lateral entorhinal cortex, the current structural (e.g. spatial) informa-
tion provided by the medial counterpart, and the ongoing plan through the
prospective splitting signal provided by the prefrontal cortex. This relates
to (and perhaps extends with goal-directedness) the interpretation of hip-
pocampal processing as latent state inference (e.g. Fuhs and Touretzky 2007; Whittington, T. H. Muller, et al. 2020) and echoes to the idea of remapping.

5.5 DYNAMICS: OSCILLATIONS AS MEMORY INTERACTIONS

The prefrontal cortex and hippocampus interact mainly through ipsilat-
eral (i.e. same side) oscillatory synchrony, as reviewed by Howard Eichen-
baum (2017b). Some prefrontal neurons are indeed synchronized to the hip-
pocampal theta rhythm in various tasks (Siapas et al. 2005; Jones and M. A.
Wilson 2005). The strength of synchronization is higher in prefrontal neu-
rons coding for spatial information (Hyman et al. 2005), and is increased in
successful spatial working memory trials (Jones and M. A. Wilson 2005; Hy-
man et al. 2010), specifically when the animal is at a choice point (Jones and
M. A. Wilson 2005; Benchenane et al. 2010). Increased synchronization has also
been found in humans successfully performing a transitive inference task
(Backus et al. 2016). Directional flow analyses reveal that information flows
from the ventral hippocampus to the prefrontal cortex when a memory is
retrieved and maintained (Siapas et al. 2005; O'Neill et al. 2013; Hallock et al.
2016; Place et al. 2016), and from the prefrontal cortex to the hippocampus
when memory retrieval is demanded (Hallock et al. 2016; Place et al. 2016).
The phase shifts between oscillations in these two regions are on the order
of 30ms, exceeding the delay for a monosynaptic projection, which sug-
gests that this synchrony is either mediated by polysynaptic pathways or
supports the transfer of information with segments of about 30ms (approx-
imately a gamma cycle), as proposed by Howard Eichenbaum (2017b).

Spellman et al. (2015) inactivated projections from the ventral hippocam-
pus to the medial prefrontal cortex in mice during a spatial working mem-
ory task. Their manipulation impaired performance and spatial coding
in the prefrontal cortex. A directional flow analysis of theta-gamma os-
cillations indicated that hippocampus-to-prefrontal functional connectivity
was disrupted, but not reciprocal connectivity. This study indicated that
the monosynaptic connection arising from CA1 conveys hippocampal rep-
resentations to the prefrontal cortex and is necessary for spatial working
memory. Sharp-wave ripples, which support spatial working memory too
(Girardeau et al. 2009; Ego-Stengel and M. A. Wilson 2010; Jadhav, Kemere, et al.
2012; Fernández-Ruiz, Oliva, Fermino de Oliveira, et al. 2019; Gridchyn et al. 2020)
are also thought to mediate prefrontal-hippocampal communication (Zielin-
ski et al. 2020). Medial prefrontal neurons are indeed modulated during rip-
ple events (Peyrache et al. 2009; Benchenane et al. 2010; Wang and Ikemoto 2016;
Jadhav, Rothschild, et al. 2016; Tang et al. 2017; Berners-Lee et al. 2021), espe-
cially while awake (Tang et al. 2017). Crucially, the content of hippocampal
replay can be decoded from prefrontal activity (Berners-Lee et al. 2021), and
inhibiting the medial prefrontal cortex during ripple events disrupts spa-
tial alternation learning (Bakker et al. 2023). While online replays provide
hippocampal memory content as context to the prefrontal cortex, offline
replays seem to train the prefrontal cortex like any other cortical region in-
volved in the complementary learning systems theory (Peyrache et al. 2009;
Benchenane et al. 2010).

Other studies investigated the role of reciprocal connections passing through
the nucleus reuniens and found that they are crucial for prefrontal-hippocampal
synchrony and memory specificity (Xu and Südhof 2013; Ito, Zhang, et al. 2015;
Hallock et al. 2016). Xu and Südhof (2013) found that inactivating the medial
prefrontal cortex or the prefrontal-to-reuniens pathway caused an overgen-
eralization in a contextual fear acquisition task, suggesting that the hip-
 pocampus then lacked top-down information. Optogenetic stimulation of
the nucleus reuniens caused increased freezing in novel contexts. However,
as noted by Howard Eichenbaum (2017b), these broad artificial stimulations
probably disrupted the modulatory role of the nucleus reuniens rather than
disrupting the direct exchange of information between the thalamus and
the hippocampus. In fact, unlike neurons of the medial prefrontal cortex
and the hippocampus, nucleus reuniens neurons do not encode detailed
information about spatial trajectories (Ito, Zhang, et al. 2015).

If the nucleus reuniens does not directly convey information from the
prefrontal cortex to the hippocampus, the question remains as to how top-
down signals enabling memory control and prospective coding are con-
veyed. Despite some reports (Rajasethupathy et al. 2015; Bian et al. 2019), it is
very unlikely that a monosynaptic projection is involved (Andrianova et al.
2023). Studies suggest that ipsilateral prefrontal-to-entorhinal connections
play a direct role (Navawongse and Howard Eichenbaum 2013; Jayachandran
et al. 2019). Disconnecting this pathway indeed causes dorsal CA1 neu-
rons to lose their object specificity without affecting their spatial specificity
(Navawongse and Howard Eichenbaum 2013). After reviewing many of the
results presented here, Howard Eichenbaum (2017b) proposed a model of
prefrontal-hippocampal interactions for episodic memory (Figure 5.2), in
which he proposed that these structures interact bidirectionally, and that
the nucleus reuniens synchronizes them and controls in which direction
information flows. When contextual cues are presented, ventral hippocam-
pal information is sent to the medial prefrontal cortex to provide context to
executive functions. Subsequently, when context-guided memory retrieval
is required, the prefrontal cortex biases retrieval of specific memories in
the dorsal hippocampus through the lateral entorhinal and perirhinal cor-
tices. This simple model accounts for many empirical findings and for why
prefrontal projections do not terminate on the same end of the longitudi-
nal axis as where hippocampal projections originate: dorsal hippocampal
neurons encode memory specifics while more ventral neurons represent
broader contextual information. However, this model might underesti-
mate the role of the nucleus reuniens. In a recent study, (Jayachandran et
al. 2019) analyzed the selective contribution of the perirhinal and nucleus
reuniens pathways in a contextual sequence memory task. Both pathways
were necessary for rats to differentiate correct in sequence odors from in-
correct out of sequence ones. In some conditions, silencing the reuniens
pathway impaired performance more than silencing the perirhinal path-
way. More strikingly, the authors found that the two pathways supported
different top-down control strategies. These results are at odds with the
proposal of Howard Eichenbaum (2017b) that nucleus reuniens is just an en-
abling structure of the perirhinal and entorhinal transfer of prefrontal in-
formation to the hippocampus. However, it is possible that the entorhinal
pathway, which was not specifically silenced in the study, is indeed con-
<figure><img src="image_32.png" /><figcaption>Figure 5.2: Model of prefrontal-hippocampal interactions in episodic memory. Adapted from Howard Eichenbaum (2017b).</figcaption></figure>

trolled by the nucleus reuniens, and that the perirhinal pathway operates
more autonomously. The role of the supramammillary nucleus, which also
connects to the prefrontal cortex and the hippocampus, also remains to be
determined in this model.

**5.6 COMPUTATIONAL MECHANISMS AND MODELS**

5.6.1 EXPERIENCE REPLAY

Experience replay can be thought of as the reinforcement learning equiv-
alent of the semantic replay postulated by the complementary learning
systems theory (Section 3.3.1). Beyond specific implementations, what is
referred to as experience replay here is the general machine learning ap-
proach of storing and replaying past experiences to speed up the acqui-
sition process of reinforcement learning agents (Lin 1992; A. W. Moore and
Atkeson 1993; Sutton and Barto 2014; Mnih et al. 2015; Schaul et al. 2015; Mat-
tar and Daw 2018; Cazé et al. 2018). Experience replay is at least partially
inspired by hippocampal replays, which are known to play a role in rein-
forcement learning (De Lavilléon et al. 2015; Girardeau et al. 2009). Trajectories
generated with some experience replay methods are in fact comparable to
hippocampal reactivations (Mattar and Daw 2018; Cazé et al. 2018).

5.6.2 EPISODIC MEMORY AS A THIRD WAY OF CONTROL

Reinforcement learning algorithms are classically divided into two categories. Model-free reinforcement learning algorithms extract the value of individual states or actions from many experiences, by bootstrapping (compiling) information about which states/actions follow individual state/action (i.e. transition dynamics) and how rewarded these are. On the other hand, in model-based algorithms, transition dynamics and rewards are learned separately and only combined during inference, making these algorithms more flexible but more computationally expensive. The hippocampus and episodic memory are thought to contribute to control in a third way (Haselmo and Howard Eichenbaum 2005; Lengyel and Dayan 2007; Gershman and Daw 2017; Cazin 2018; M. Botvinick et al. 2019). Instead of learning by bootstrapping (model-free) or performing expensive inference (model-based), the episodic controller of Lengyel and Dayan (2007) explicitly stores and retrieves experiences leading to unexpectedly high rewards. The authors showed that while such a controller is eventually suboptimal, it can learn tasks more rapidly than a model-based controller, especially when the task is difficult and the model-based controller is noisy (Figure 5.3). Unlike offline experience replay, episodic control thus combines episodes at inference time to compensate the rigidity of model-free bootstrap and the cost of model-based inference. Contrary to classical reinforcement learning algorithms, episodic control is nonparametric in that experiences are stored
<figure><img src="image_33.png" /><figcaption>Figure 5.3: Performance of model-based and episodic controllers in a tree-structured Markov decision problem with varying complexity captured by the branching factor *B*. The noisy model-based controller only has access to noisy versions of the action values it learns. This was done to model the effects of imperfect working memory. Taken from Lengyel and Dayan (2007).</figcaption></figure>

explicitly instead of being transformed and stored as parameters of a transition model (model-based) or value estimates (model-free). In the complementary learning systems framework, this corresponds to experiences being stored separately from each other in episodic memory, as compared to the way semantic memory blends experiences into coherent schemas.

This work has led to the development of differentiable memory-augmented controllers that can be integrated into deep learning architectures (Graves, Wayne, and Danihelka 2014; Graves, Wayne, M. Reynolds, et al. 2016; Blundell et al. 2016; Pritzel et al. 2017; Ritter et al. 2018; Wayne et al. 2018; Fortunato et al. 2019; Lu et al. 2022). Pritzel et al. (2017) for example introduced Neural Episodic Control (NEC), a deep reinforcement learning agent that employs a differentiable neural dictionary (DND) to store state-action-value tuples and act according to retrieval from this DND. More precisely, DND has a memory module $M_a = (K_a, V_a)$ for each action $a$, where $K$ matrices store experienced states as keys and $V$ matrices store the estimated values of these states. Operations of the DND are illustrated in Figure 5.4 and described below.
<figure><img src="image_34.png" /></figure>

(a) A new experience is written in the dictionary with the hidden state of the convolutional neural network as key and Q-value estimate as value. The value of a state is evaluated by looking at the value of similar states in the dictionary.

<figure><img src="image_35.png" /></figure>

(b) Architecture of episodic memory module for a single action *a*. Pixels repre-
senting the current state enter through a convolutional neural network on the
bottom left and an estimate of *Q(s,a)* exits top right. Gradients flow through
the entire architecture.

Figure 5.4: Illustration of operations of a differentiable neural dictionary. Taken
from Pritzel et al. (2017).
WRITING TO A DIFFERENTIABLE NEURAL DICTIONARY

The NEC agent perceives the world through a convolutional neural network (the same as described by Mnih et al. 2015) which processes observation $s_t$ from the environment and outputs an embedding vector $h$. This vector is used as a key for storing a memory in the DND, this means that it is added to the key matrix of the selected action $a$ (i.e. $K_a$). The corresponding value to be added to $V_a$ is computed as a Q-value estimate combining N-step on-policy rewards and off-policy bootstrap:

$$Q^{(N)}(s_t, a) = \sum_{j=0}^{N-1} \gamma^j r_{t+j} + \gamma^N \max_{a'} Q(s_{t+N}, a'). \quad (5.1)$$

The bootstrap term $\max_{a'} Q(s_{t+N}, a')$ is found by querying all memories $M_a$ for each action $a$ and taking the highest estimated Q-value returned. Because a memory cannot be added before waiting $N$ steps to compute the Q-value estimate, memories for time $t$ must be written in the DND at $t' \ge t+N$.

If a state is already present in $K_a$, the corresponding value of $V_a$, $Q_i$, is updated with rate $\alpha$:

$$Q_i \leftarrow Q_i + \alpha(Q^N(s, a) - Q_i). \quad (5.2)$$

When maximum capacity is reached, the item that has least recently shown up as a neighbour in the lookup operation described below is overwritten.

LOOKUP FROM A DIFFERENTIABLE NEURAL DICTIONARY

The lookup operation of a DND is performed as follows:

$$o = \sum_i w_i v_i \quad (5.3)$$
where $v_i$ is the ith value and

$$w_i = k(h, h_i) / \sum_j k(h, h_j) \qquad (5.4)$$

with $k(x, y)$ a kernel between vectors $x$ and $y$ defined as:

$$k(h, h_i) = \frac{1}{||h - h_i||_2^2 + \delta} \qquad (5.5)$$

$$\delta = 10^{-3}. \qquad (5.6)$$

To ensure scalability, the lookup operation is limited to the 50 nearest neighbors with respect to their key vector. This is done using a kd-tree that approximates nearest-neighbor lookup (Bentley 1975). The agent learns to minimize the discrepancy between the output $o$ and a target $Q$-value estimate and selects the best action using the traditional Deep Q-Network method (Mnih et al. 2015). Like other episodic controllers, NEC leverages fast episodic storage to maximize data efficiency. Pritzel et al. (2017) showed that it indeed learns very rapidly and outperforms previous reinforcement learning techniques on Atari games.

### 5.6.3 CONTROLLED EPISODIC MEMORY

While experience replay and episodic control have greatly influenced the field of reinforcement learning by drawing inspiration from hippocampal projections to the prefrontal cortex, the reciprocal connections are still underexplored in this context. Some notable exceptions do not explicitly refer to these anatomical connections but implement elements of top-down control over memory (e.g. Fortunato et al. 2019; Lu et al. 2022; K. T. Jensen et al. 2023). For example, Fortunato et al. (2019) augmented NEC with a long short-term memory (LSTM; Hochreiter and Schmidhuber 1997) which serves as a working memory from which the action is selected. Instead of learning the heteroassociation between sensory states and $Q$-value estimates like in NEC, the DND of Fortunato et al. (2019) stores sensory and working mem-
ory states as keys, and the retrieved value is a combination of past working
memory states that is fed back to the working memory. Crucially, the read-
ing operation not only takes into account the sensory state but also the
state of the working memory, allowing for top-down interactions. This
echoes the prefrontal control over hippocampal operations, and Fortunato
et al. (2019) showed that their architecture outperforms NEC in a variety of
tasks.

Other related works include model-based episodic learning (Le et al. 2021), episodic meta-learning (L. C. Melo 2022; Ritter et al. 2018), separate retrieval mechanisms (Goyal et al. 2022), and so on. Yet, this line of machine learning research is largely disconnected from the biology too.

In computational neuroscience, prefrontal-to-hippocampus interactions
have been modeled more explicitly by Pilly et al. (2018). They extended the
model of Ketz et al. (2013) described in the previous chapter to account for
the contextual modulation of hippocampal memory processing by the pre-
frontal cortex. In order to do that, they used context information to bias
the excitability of granule cells in the dentate gyrus. They thus divided
the population of dentate gyrus cells into four subsets corresponding to
four contexts. When an association was presented to the hippocampus, the
excitability of context-inappropriate dentate gyrus cells was suppressed.
The model performed the task of Peters et al. (2013) in which rats were
presented with eight odor discrimination problems. When presented in a
different context, the reward contingency was reversed, such that the cor-
rect odor of the first context was unrewarded in the second one, and con-
versely. Thanks to the modulation of granule cells, the model successfully
acquired this contextual memory task and reproduced biological findings.
Unlike the proposal of Howard Eichenbaum (2017b), the authors argued that
prefrontal information is potentially conveyed to the dentate gyrus by the
supramammillary nucleus, yet it is not clear whether this small region is
able to support this function.
**5.7 DISCUSSION**

In this chapter, interactions between episodic memory and cognitive con-
trol in the brain have been described. Hippocampus-to-prefrontal connec-
tions have been introduced as a way to provide episodic memory informa-
tion to decision-making regions. Conversely, prefrontal-to-hippocampus
connections have been shown to exert cognitive control on cognitive actions
instead of motor actions, more precisely they contextually control memory
operations. Finally, related computational models have been presented.

We are still a long way from understanding how the simple interactions
revealed by experiments enable complex functions of the construction sys-
tem of the human brain (Hassabis and Maguire 2009), such as imagination.
This is partly due to a lack of computational models related to this topic,
and this thesis aims to help fill this gap. First, the link between episodic
control algorithms and the biological implementation of episodic memory
is unclear. In the next chapter, the differentiable neural dictionary of neural
episodic control will be related to Hopfield networks (Chapter 6). Further-
more, computational models of the prefrontal influence on memory recall
in the hippocampus are rare and involve a direct anatomical connection.
The contextual modulation of modern Hopfield networks through the in-
termediate entorhinal cortex will be investigated in Chapter 7. Finally, a
robust model of one-shot sequence learning in the hippocampus will be
developed in Chapter 8, building upon the results of the previous chap-
ters. Among other things, it will be shown that this model is capable of
disambiguating overlapping sequences by forming contextual representa-
tions.
<figure><img src="image_36.png" /></figure>
# PART II

## COMPUTATIONAL MODELING
<figure><img src="image_37.png" /></figure>
In this part, computational tools are employed to investigate episodic
memory and its relationship with cognitive control. Chapters 6 and 7 are
introductory studies on elements of prefrontal-hippocampal interactions.
The most important contribution is the CONSEQUENCE model of one-shot
sequence learning in the hippocampus, introduced in Chapter 8, that builds
upon the modern Hopfield network studied in Chapters 6 and 7.
<figure><img src="image_38.png" /></figure>
# 6 HOPFIELD EPISODIC CONTROL: RELATING ASSOCIATIVE MEMORY TO REINFORCEMENT LEARNING ALGORITHMS

## 6.1 INTRODUCTION

In this chapter, a novel connection is established between the fields of associative memory and reinforcement learning. It is shown in Section 6.2 that the differentiable neural dictionary (DND; Section 5.6.2) introduced in the context of episodic control (Pritzel et al. 2017) is a Hopfield network. Retrieval from a DND can indeed be decomposed into similarity, separation and projection operations, just as any instance of the recently proposed universal Hopfield network framework (UHN; Millidge et al. 2022). In a DND, similarity scores are computed using the Euclidean distance and separated with a *k*-nearest neighbor algorithm. The projection operation of DND is the same as for all existing UHN instances. DND can thus be thought of as a single-shot associative memory model, just like Hopfield networks and their modern continuous variants. The implications are twofold. The first one is that, while DND was introduced in the context of reinforcement learning, it can be used to store and retrieve information in a wide variety of forms. This ability is illustrated in Section 6.3 using three image datasets. The second implication is that any UHN model can be used to
store experiences in place of the DND for episodic control, as shown in Section 6.4.

## 6.2 DIFFERENTIABLE NEURAL DICTIONARY AS A HOPFIELD NETWORK

In the UHN framework, retrieval is performed by applying similarity and separation functions sep and sim respectively, then projecting the results to the output space with some value matrix V (Equation 4.7):

$$z = V \cdot \text{sep}(\text{sim}(K, q)). \qquad (6.1)$$

Rewriting the reading operation of a DND (Equation 5.3) with different variable names yields:

$$z = \sum_{i} v_{i}w_{i}, \qquad (6.2)$$

where $w$ contains the normalized distances between the query and the keys of the k-nearest neighbors. Using vector notation, the parallel with Hopfield networks becomes more explicit:

$$z = v \cdot w \qquad (6.3)$$

$$z = v \cdot \text{sep}(\text{sim}(K, q), k), \qquad (6.4)$$

where sim is the kernel function:

$$\text{sim}_i(K, q) = \frac{1}{||K_i - q||_2^2 + \delta} \qquad (6.5)$$
which is equal to the Euclidean similarity function defined in Millidge et al. (2022). The separation function finds the top k similarity values (thus implementing the k-nearest neighbor algorithm) and normalizes them:

$$ \mathrm{sep}(\mathbf{x}, k) = \kappa(\mathbf{x}, k) / \sum_{i} \kappa_{i}(\mathbf{x}, k) \qquad (6.6) $$

$$ \kappa_i(\mathbf{x}, k) = \begin{cases} x_i & \text{if } x_i \text{ is among the top } k \text{ values of } \mathbf{x}, \\ 0 & \text{otherwise.} \end{cases} \qquad (6.7) $$

In sum, the kernel function of DND is a similarity function, as it is equivalent to the Euclidean similarity function of Millidge et al. (2022). Furthermore, the k-nearest neighbor algorithm sparsifies the result of the similarity function by cancelling the contribution of the most distant experiences. The output of the algorithm is then normalized before being projected to the value space. This constitutes a novel separation function for the UHN framework. It is worth noting that this separation function is similar to applying a threshold on the similarity function, like what is done in the sparse distributed memory model (Kanerva 1988), which has also been cast as a UHN (Millidge et al. 2022). The only difference is that the threshold for sparse distributed memory is fixed, while it must be dynamic for selecting a constant number k of neighbors. Furthermore, with k = 1, the separation function is equivalent to the max function benchmarked by Millidge et al. (2022). The remaining difference between DND and other UHN instances is that the output of the DND is a scalar value, while UHN models can store vectors. As mentioned in Chapter 5, a DND has been used by Fortunato et al. (2019) to store multidimensional values. In fact, the DND can simply be extended with a matrix of value vectors. Equation 6.3 thus becomes:

$$ z = V \cdot w. \qquad (6.8) $$
6.3 DIFFERENTIABLE NEURAL DICTIONARY AS AN
ASSOCIATIVE MEMORY

In the previous section, the differentiable neural dictionary of neural episodic control has been shown to be a universal Hopfield network. In principle, DND can thus be used as an associative memory. In this section, the MNIST, CIFAR10 and Tiny ImageNet datasets are used to test the robustness and capacity of DND as an associative memory model, using the same methods as in Millidge et al. (2022), unless otherwise specified.

Example reconstructions of MNIST digits by DND are shown in Figure 6.1. Memories are separated by keeping the *k*-nearest neighbors only (or *k*-max). For simplicity, the *kd*-tree of the original neural episodic control implementation is not used. Instead, all similarity scores are computed and those not selected are zeroed out. Using *k* = 50 like in Pritzel et al. (2017) gives an output from which the original image can be recognized, but the model is unable to properly separate memories and the resulting output is blurry. The max separation function is equivalent to selecting the nearest neighbor (*k* = 1) and provides a much clearer output. In fact, max was the best separation function benchmarked by Millidge et al. (2022). Even when the output is blurry, the *k* > 1 models seem to properly capture the statistics of the dataset, such as the fact that central pixels are globally more active than those on the borders. Recall accuracy is typically assessed in absolute terms, by checking that the difference between output and response is below a threshold (Millidge et al. 2022). It is worth exploring whether the statistical modeling capacities of *k* > 1 can consistently improve performance with this criterion, despite the max function having theoretically unbounded capacity with respect to the dimensionality of the query (Millidge et al. 2022). On the other hand, a new criterion will have to be introduced to evaluate recall in relation to other stored patterns, given that the main function of episodic memory is to reconstruct information corresponding to a particular situation rather than to generalize.
## 6.3 Differentiable Neural Dictionary as an Associative Memory

<figure><img src="image_39.png" /><figcaption>Figure 6.1: Example reconstructions of noisy MNIST digits by DND. The top row shows the input cue with increasing amount of noise. The following rows show the reconstruction of the stored memory using $k = 1, 5, 50$.</figcaption></figure>

### 6.3.1 CAPACITY WITH DIFFERENT FUNCTIONS

In DND, the similarity between memories and query is computed using a Euclidean function (Equation 6.5). While rarely used in associative memory models, this function was introduced by Millidge et al. (2022) and found to outperform the more common dot product. An even better performing similarity function was the inverse of the Manhattan distance:

$$ \mathrm{sim}_i(\mathbf{K}, \mathbf{q}) = \frac{1}{\delta + \sum_j \mathrm{abs}(\mathbf{K}_{ij} - \mathbf{q}_j)} \qquad (6.9) $$

The capacity of the model under different similarity and separation functions is assessed by quantifying correctly retrieved data when increasing number of MNIST, CIFAR10 and Tiny ImageNet images is stored (Figure 6.2; Table 6.1). Half-masked images are given as input, and a trial is correct if the sum of squared pixel differences between the output and the actual image is less than a threshold of 50. The Manhattan similarity function out-
<table><caption>Table 6.1: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFAR10 and Tiny ImageNet datasets. Reported are means and standard deviations of the 10 simulations of Figure 6.2. For each dataset and similarity function, the best performance is highlighted in bold.</caption><thead><tr><th>Similarity</th><th>Separation</th><th>MNIST</th><th>CIFAR10</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td rowspan="6">Euclidean</td><td>Max</td><td>0.739 ± 0.14</td><td>0.220 ± 0.18</td><td><b>0.223 ± 0.21</b></td></tr><tr><td>2-Max</td><td>0.826 ± 0.11</td><td><b>0.236 ± 0.18</b></td><td>0.015 ± 0.02</td></tr><tr><td>5-Max</td><td><b>0.851 ± 0.08</b></td><td>0.117 ± 0.09</td><td>0.010 ± 0.02</td></tr><tr><td>10-Max</td><td>0.838 ± 0.08</td><td>0.095 ± 0.08</td><td>0.010 ± 0.02</td></tr><tr><td>50-Max</td><td>0.801 ± 0.09</td><td>0.087 ± 0.09</td><td>0.010 ± 0.02</td></tr><tr><td>Identity</td><td>0.809 ± 0.08</td><td>0.088 ± 0.09</td><td>0.010 ± 0.02</td></tr><tr><td rowspan="6">Manhattan</td><td>Max</td><td>0.835 ± 0.10</td><td><b>0.451 ± 0.21</b></td><td><b>0.669 ± 0.24</b></td></tr><tr><td>2-Max</td><td>0.886 ± 0.08</td><td>0.369 ± 0.20</td><td>0.011 ± 0.02</td></tr><tr><td>5-Max</td><td><b>0.887 ± 0.07</b></td><td>0.106 ± 0.08</td><td>0.010 ± 0.02</td></tr><tr><td>10-Max</td><td>0.826 ± 0.08</td><td>0.075 ± 0.07</td><td>0.010 ± 0.02</td></tr><tr><td>50-Max</td><td>0.775 ± 0.11</td><td>0.067 ± 0.06</td><td>0.010 ± 0.02</td></tr><tr><td>Identity</td><td>0.804 ± 0.09</td><td>0.063 ± 0.06</td><td>0.010 ± 0.02</td></tr></tbody></table>

performs the Euclidean function, especially when *k* is low. Furthermore, the best *k* value is highly dependent on the dataset. In MNIST, the best *k* is 5 for both Euclidean and Manhattan functions. In CIFAR10, the best *k* is 2 with Euclidean similarity and 1 with the Manhattan function. In Tiny ImageNet, max (*k* = 1) outperforms other functions.

### 6.3.2 RETRIEVAL WITH DIFFERENT FUNCTIONS

In order to test robustness of the memory, the ability to recall memories from noisy cues is analyzed. Independent zero-mean Gaussian noise with variance σ is thus added to the query images pixelwise. Like capacity, the best *k* for retrieval depends on the dataset (Figure 6.3 ; Table 6.2). Here again, the performance is better with the Euclidean similarity than with the Manhattan similarity for low *k*, and worse for high *k*. In MNIST, the best *k* is 50 with both Euclidean and Manhattan similarities. In CIFAR10, 2
## 6.3 Differentiable Neural Dictionary as an Associative Memory

<figure><img src="image_40.png" /><figcaption>Figure 6.2: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFAR10 and Tiny ImageNet datasets. Plots represent the means and standard deviations of 10 simulations. A trial is correct when the difference between the output and the actual memory is under a threshold.</figcaption></figure>
<table><caption>Table 6.2: Retrieval capability against noise. Reported are means and standard deviations of the 10 simulations of Figure 6.3. For each dataset and similarity function, the best performance is highlighted in bold.</caption><thead><tr><th>Similarity</th><th>Separation</th><th>MNIST</th><th>CIFAR10</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td rowspan="6">Euclidean</td><td>Max</td><td>0.667 ± 0.44</td><td>0.574 ± 0.45</td><td><strong>0.580 ± 0.44</strong></td></tr><tr><td>2-Max</td><td>0.692 ± 0.41</td><td><strong>0.588 ± 0.44</strong></td><td>0.310 ± 0.43</td></tr><tr><td>5-Max</td><td>0.735 ± 0.37</td><td>0.455 ± 0.41</td><td>0.190 ± 0.35</td></tr><tr><td>10-Max</td><td>0.789 ± 0.31</td><td>0.357 ± 0.39</td><td>0.128 ± 0.33</td></tr><tr><td>50-Max</td><td><strong>0.904 ± 0.09</strong></td><td>0.205 ± 0.30</td><td>0.002 ± 0.01</td></tr><tr><td>Identity</td><td>0.830 ± 0.10</td><td>0.083 ± 0.09</td><td>0.000 ± 0.00</td></tr><tr><td rowspan="6">Manhattan</td><td>Max</td><td>0.672 ± 0.43</td><td>0.627 ± 0.44</td><td><strong>0.620 ± 0.43</strong></td></tr><tr><td>2-Max</td><td>0.699 ± 0.40</td><td><strong>0.628 ± 0.43</strong></td><td>0.223 ± 0.39</td></tr><tr><td>5-Max</td><td>0.741 ± 0.36</td><td>0.424 ± 0.38</td><td>0.009 ± 0.02</td></tr><tr><td>10-Max</td><td>0.794 ± 0.30</td><td>0.282 ± 0.28</td><td>0.002 ± 0.01</td></tr><tr><td>50-Max</td><td><strong>0.891 ± 0.07</strong></td><td>0.085 ± 0.06</td><td>0.000 ± 0.00</td></tr><tr><td>Identity</td><td>0.781 ± 0.05</td><td>0.049 ± 0.02</td><td>0.000 ± 0.00</td></tr></tbody></table>

is the best *k*. Like for capacity, max again outperforms other functions for retrieval of noisy Tiny ImageNet images.

As hypothesized with Figure 6.1, max is not always the best performing function with the absolute accuracy criterion, both for capacity and retrieval. It can indeed be outperformed by higher *k* values, meaning that taking into account more memories than the single most similar one can lead to more precise recall in absolute terms. However, the performance of the identity and 50-Max functions in the MNIST dataset are surprisingly good, even under very high levels of noise. In fact, as pixel values are restricted to lie in the range [0, 1], it is very unlikely that enough information remains in the image to correctly identify it when $\sigma > 1$. Hence, a plausible alternative is that high *k* functions model the dataset such that they output a mixture of many images that is sometimes classified as correct retrieval, although it is not necessarily closer to the query image than any other of the dataset. This is especially true for MNIST which contains simple pictures that are more similar to each other than CIFAR10 and Tiny ImageNet.
## 6.3 Differentiable Neural Dictionary as an Associative Memory

<figure><img src="image_41.png" /><figcaption>Figure 6.3: Retrieval capability against increasing levels of noise. Plots represent the means and standard deviations of 10 simulations with different sets of images. A trial is correct when the difference between the output and the actual memory is under a threshold.</figcaption></figure>
<table><caption>Table 6.3: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFAR10 and Tiny ImageNet datasets. Reported are means and standard deviations of the 10 simulations of Figure 6.4. For each dataset and similarity function, the best performance is highlighted in bold.</caption><thead><tr><th>Similarity</th><th>Separation</th><th>MNIST</th><th>CIFAR10</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td rowspan="6">Euclidean</td><td>Max</td><td><b>0.624 ± 0.22</b></td><td><b>0.223 ± 0.19</b></td><td><b>0.223 ± 0.21</b></td></tr><tr><td>2-Max</td><td><b>0.624 ± 0.22</b></td><td>0.222 ± 0.19</td><td>0.222 ± 0.21</td></tr><tr><td>5-Max</td><td>0.256 ± 0.24</td><td>0.132 ± 0.13</td><td>0.112 ± 0.14</td></tr><tr><td>10-Max</td><td>0.199 ± 0.24</td><td>0.104 ± 0.14</td><td>0.098 ± 0.13</td></tr><tr><td>50-Max</td><td>0.171 ± 0.25</td><td>0.086 ± 0.14</td><td>0.095 ± 0.14</td></tr><tr><td>Identity</td><td>0.169 ± 0.25</td><td>0.085 ± 0.14</td><td>0.095 ± 0.14</td></tr><tr><td rowspan="6">Manhattan</td><td>Max</td><td><b>0.793 ± 0.14</b></td><td><b>0.502 ± 0.24</b></td><td><b>0.669 ± 0.24</b></td></tr><tr><td>2-Max</td><td>0.790 ± 0.14</td><td>0.486 ± 0.25</td><td>0.505 ± 0.36</td></tr><tr><td>5-Max</td><td>0.255 ± 0.22</td><td>0.176 ± 0.19</td><td>0.155 ± 0.20</td></tr><tr><td>10-Max</td><td>0.187 ± 0.22</td><td>0.103 ± 0.15</td><td>0.119 ± 0.18</td></tr><tr><td>50-Max</td><td>0.163 ± 0.23</td><td>0.090 ± 0.15</td><td>0.113 ± 0.18</td></tr><tr><td>Identity</td><td>0.162 ± 0.23</td><td>0.088 ± 0.15</td><td>0.113 ± 0.18</td></tr></tbody></table>

### 6.3.3 PERFORMANCE WITH RELATIVE CRITERION

In order to prevent associative memory models to model the statistics of the dataset rather than focusing on the query image to output the actual memory, a novel criterion is introduced. Instead of the absolute threshold, retrieval must be good relatively to other images. More precisely, the novel criterion is such that a trial is correct if and only if the squared pixel difference between the truth and the output is lower than it is between the output and any other memory. Capacity is again assessed using this new criterion (Figure 6.4 ; Table 6.3). The Manhattan function still outperforms the Euclidean similarity. Most crucially, the best performance is always obtained with the max function.

Retrieval is then tested with the new criterion (Figure 6.5 ; Table 6.4). Once again, the best performance is obtained with the Manhattan similarity. Furthermore, $k = 1$ almost always outperforms other values. Note that performance with $k = 2$ is very similar.
6.3 Differentiable Neural Dictionary as an Associative Memory

<figure><img src="image_42.png" /><figcaption>Figure 6.4: Capacity of associative memory with different similarity and separation functions assessed with MNIST, CIFAR10 and Tiny ImageNet datasets. Plots represent the means and standard deviations of 10 simulations. Here, a trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.</figcaption></figure>
<figure><img src="image_43.png" /><figcaption>Figure 6.5: Retrieval capability against increasing levels of noise. Plots represent the means and standard deviations of 10 simulations with different sets of images. Here, a trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.</figcaption></figure>
<table><caption>Table 6.4: Retrieval capability against noise. Reported are means and standard deviations of the 10 simulations of Figure 6.5. For each dataset and similarity function, the best performance is highlighted in bold.</caption><thead><tr><th>Similarity</th><th>Separation</th><th>MNIST</th><th>CIFAR10</th><th>Tiny ImageNet</th></tr></thead><tbody><tr><td rowspan="6">Euclidean</td><td>Max</td><td><b>0.661 ± 0.44</b></td><td>0.574 ± 0.45</td><td><b>0.580 ± 0.44</b></td></tr><tr><td>2-Max</td><td><b>0.661 ± 0.44</b></td><td><b>0.575 ± 0.44</b></td><td>0.579 ± 0.44</td></tr><tr><td>5-Max</td><td>0.576 ± 0.41</td><td>0.366 ± 0.42</td><td>0.416 ± 0.45</td></tr><tr><td>10-Max</td><td>0.522 ± 0.39</td><td>0.288 ± 0.40</td><td>0.314 ± 0.41</td></tr><tr><td>50-Max</td><td>0.246 ± 0.34</td><td>0.134 ± 0.29</td><td>0.160 ± 0.32</td></tr><tr><td>Identity</td><td>0.150 ± 0.32</td><td>0.029 ± 0.05</td><td>0.101 ± 0.24</td></tr><tr><td rowspan="6">Manhattan</td><td>Max</td><td><b>0.665 ± 0.44</b></td><td><b>0.621 ± 0.44</b></td><td><b>0.622 ± 0.43</b></td></tr><tr><td>2-Max</td><td>0.664 ± 0.43</td><td><b>0.621 ± 0.44</b></td><td><b>0.622 ± 0.43</b></td></tr><tr><td>5-Max</td><td>0.523 ± 0.36</td><td>0.316 ± 0.37</td><td>0.352 ± 0.40</td></tr><tr><td>10-Max</td><td>0.422 ± 0.31</td><td>0.177 ± 0.25</td><td>0.228 ± 0.33</td></tr><tr><td>50-Max</td><td>0.063 ± 0.04</td><td>0.026 ± 0.02</td><td>0.030 ± 0.03</td></tr><tr><td>Identity</td><td>0.022 ± 0.02</td><td>0.012 ± 0.00</td><td>0.012 ± 0.01</td></tr></tbody></table>

### 6.3.4 RELATIONSHIP BETWEEN k-MAX AND SOFTMAX

Like k-Max, the softmax function virtually cancels out the contribution of distant memories, especially when $\beta$, the scaling parameter of its input, is high. It does it by normalizing exponentiated similarity scores (Equation 4.6). While $k$ is a discrete parameter, $\beta$ is continuous, which potentially makes the softmax function more flexible to optimize. Here, the two separation functions are compared. For each dataset, 100 images are encoded. The noise is set to 1 for MNIST and 0.75 for CIFAR10 and Tiny ImageNet. The results are shown in Figure 6.6 with the absolute criterion and in Figure 6.7 with the relative criterion. Except for the condition with CIFAR10, Euclidean similarity and absolute criterion (Figure 6.6c), the softmax can always outperform k-Max.

## 6.4 HOPFIELD NETWORK FOR EPISODIC CONTROL

Neural episodic control greatly improves the sample efficiency of reinforcement learning, as it rapidly assimilates new experience in the dictionary
<figure><img src="image_44.png" /><figcaption>Figure 6.6: Retrieval capability as a function of *k* and *β* parameters of the *k*-Max and softmax separation functions respectively. Plots represent the means and standard deviations of 10 simulations with different sets of images. A trial is correct when the difference between the output and the actual memory is under a threshold.</figcaption></figure>
6.4 Hopfield Network for Episodic Control

<figure><img src="image_45.png" /><figcaption>Figure 6.7: Retrieval capability as a function of *k* and *β* parameters of the *k*-Max and softmax separation functions respectively. Plots represent the means and standard deviations of 10 simulations with different sets of images. Here, a trial is correct when the difference between the output and the actual memory is lower than the difference between the output and any other stored memory.</figcaption></figure>
(Pritzel et al. 2017). Since the differentiable neural dictionary is a Hopfield
network, a question worth exploring is whether other instances of the uni-
versal Hopfield network framework can be used for reinforcement learn-
ing, and perhaps outperform the DND. The strength of this framework is
that it unifies disparate models that sequentially apply a set of operations.
In fact, both the Euclidean kernel and *k*-nearest neighbor of NEC can be
replaced by any differentiable similarity and separation functions, respec-
tively.

The code of Sarrico et al. (2019) is used. It is made more flexible to incorpo-
rate Manhattan and dot product kernels, as well as the softmax separation
function. The models are tested in two classic problems, CartPole and Ac-
robot, implemented in OpenAI Gym (Brockman et al. 2016). When possible,
the original hyperparameters of Pritzel et al. (2017) are used, although some
values are taken from Sarrico et al. (2019) who trained their algorithms in
the same reinforcement learning environments instead of Atari games like
in Pritzel et al. (2017). The number of nearest neighbors is set to $k = 11$ and
the memory can store a total of 10,000 entries. A more exhaustive report of
hyperparameters can be found in Table 6.5. The agents are trained during
500,000 steps and evaluated every 5000 steps.

The results of 3 runs are shown in Figure 6.8. In this (very preliminary)
experiment, the dot product did not perform as good as the other similar-
ity functions. Furthermore, while the performance of the Manhattan and
Euclidean functions is very similar, Manhattan similarity performed better
in the Acrobot environment with *k*-Max separation. Finally, the softmax
function with randomly chosen $\beta = 100$ did not perform as great as *k*-Max.
Overall, the agents did not successfully learn the tasks.

6.5 DISCUSSION

In this chapter, DND, which has initially been introduced in the context of
reinforcement learning (Pritzel et al. 2017), has been shown to be mathemat-
ically related to associative memory models like Hopfield networks. The
<table><caption>Table 6.5: Hyperparameters used for neural episodic control simulations.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Initial policy &epsilon;</td><td>1</td></tr><tr><td>Final policy &epsilon;</td><td>5 &times; 10<sup>-3</sup></td></tr><tr><td>Starting step of policy &epsilon; annealing</td><td>5 &times; 10<sup>3</sup></td></tr><tr><td>Ending step of policy &epsilon; annealing</td><td>25 &times; 10<sup>3</sup></td></tr><tr><td>Discount factor &gamma;</td><td>0.99</td></tr><tr><td>Number of neighbors k</td><td>11</td></tr><tr><td>Kernel &delta;</td><td>10<sup>-3</sup></td></tr><tr><td>Softmax &beta;</td><td>100</td></tr><tr><td>Experience replay size</td><td>10<sup>5</sup></td></tr><tr><td>RMSprop learning rate</td><td>7.92 &times; 10<sup>-6</sup></td></tr><tr><td>RMSprop momentum</td><td>0.95</td></tr><tr><td>RMSprop &epsilon;</td><td>10<sup>-2</sup></td></tr><tr><td>Starting step of training</td><td>10<sup>3</sup></td></tr><tr><td>Batch size</td><td>32</td></tr><tr><td>Memory learning rate</td><td>0.1</td></tr><tr><td>n-step return</td><td>100</td></tr><tr><td>Key size</td><td>64</td></tr></tbody></table>
<figure><img src="image_46.png" /><figcaption>Figure 6.8: Mean and standard deviation learning curves of 3 different runs in Acrobot and Cartpole using different similarity and separation functions.</figcaption></figure>