Article

https://doi.org/10.1038/s41467-024-44877-0

Distinguishing examples while building
concepts in hippocampal and artiﬁcial
networks
Received: 9 March 2023

Louis Kang

1,2

& Taro Toyoizumi

3,4

Check for updates

1234567890():,;

1234567890():,;

Accepted: 3 January 2024

The hippocampal subﬁeld CA3 is thought to function as an auto-associative
network that stores experiences as memories. Information from theseexperiences arrives directly from the entorhinal cortex as well as indirectly
through the dentate gyrus, which performs sparsiﬁcation and decorrelation.
The computational purpose for these dual input pathways has not been ﬁrmly
established. We model CA3 as a Hopﬁeld-like network that stores both dense,
correlated encodings and sparse, decorrelated encodings. As more memories
are stored, the former merge along shared features while the latter remain
distinct. We verify our model’s prediction in rat CA3 place cells, which exhibit
more distinct tuning during theta phases with sparser activity. Finally, we ﬁndthat neural networks trained in multitask learning beneﬁt from a loss term that
promotes both correlated and decorrelated representations. Thus, the complementary encodings we have found in CA3 can provide broad computational
advantages for solving complex tasks.

The hippocampus underlies our ability to form episodic memories,
through which we can recount personally experienced events from ourdaily lives1. In particular, the subﬁeld CA3 is believed to provide thiscapability as an autoassociative network2–4. Its pyramidal cells containabundant recurrent connections exhibiting spike-timing-dependent
plasticity5,6. These features allow networks to perform patterncompletion and recover stored patterns of neural activity from noisycues. Sensory information to be stored as memories arrives to CA3 via
the entorhinal cortex (EC), which serves as the major gateway between
hippocampus and neocortex (Fig. 1A). Neurons from layer II of EC
project to CA3 via two different pathways7. First, they synapse directly
onto the distal dendrites of CA3 pyramidal cells through the perforantpath (PP). Second, before reaching CA3, perforant path axons branch
towards the dentate gyrus (DG) and synapse onto granule cells.
Granule cell axons form the mossy ﬁbers (MF) that also synapse onto
CA3 pyramidal cells, though at more proximal dendrites.
Along these pathways, information is transformed by eachprojection in addition to being simply relayed. DG sparsiﬁes

encodings from EC by maintaining high inhibitory tone
across its numerous neurons8. Sparsiﬁcation in feedforward networks
generally decorrelates activity patterns as well9–13. The sparse,
decorrelated nature of DG encodings is preserved by the MF pathway
because its connectivity is also sparse; each CA3 pyramidal cell
receives input from only ≈50 granule cells14. In contrast, PP
connectivity is dense with each CA3 pyramidal cell receiving input
from ≈4000 EC neurons14, so natural correlations between similarsensory stimuli should be preserved. Thus, CA3 appears to receive two
encodings of the same sensory information with different properties:
one sparse and decorrelated through MF and the other dense andcorrelated through PP. What is the computational purpose of this dualinput architecture? Previous theories have proposed that the MF
pathway is crucial for pattern separation during memory storage, but
retrieval is predominantly mediated by the PP pathway and can evenbe hindered by MF inputs15–17. In these models, MF and PP encodings
merge during storage and one hybrid pattern per memory is recovered
during retrieval.

1

Neural Circuits and Computations Unit, RIKEN Center for Brain Science, 2-1 Hirosawa, Wako-shi, Saitama 351-0198, Japan. 2Graduate School of Informatics,
Kyoto University, 36-1 Yoshida-honmachi, Sakyo-ku, Kyoto 606-8501, Japan. 3Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain
Science, 2-1 Hirosawa, Wako-shi, Saitama 351-0198, Japan. 4Graduate School of Information Science and Technology, University of Tokyo, 7-3-1 Hongo,
e-mail: louis.kang@riken.jp
Bunkyo-ku, Tokyo 113-0033, Japan.

Nature Communications | (2024)15:647

1

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 1 | Overview and motivation. A Entorhinal cortex (EC) projects to CA3 directly
via the perforant path (PP, orange) as well as indirectly through the dentate gyrus(DG) via mossy ﬁbers (MF, purple). Adapted from Rosen GD, Williams AG, Capra JA,
Connolly MT, Cruz B, Lu L, Airey DC, Kulkarni K, Williams RW (2000) The Mouse
Brain Library @ https://www.mbl.org. Int Mouse Genome Conference 14: 166.
https://www.mbl.org. B MF memory encodings are believed to be sparser and lesscorrelated compared to PP encodings. In an autoassociative network, attractor
basins of the former tend to remain separate and those of the latter tend to merge.

C By modeling hippocampal networks, we ﬁrst predict that MF and PP encodings in
CA3 can respectively maintain distinctions between memories and generalize
across them (Figs. 2–4). By analyzing publicly available neural recordings, we thendetect signatures of these encoding properties in rat CA3 place cells (Figs. 5–7). By
training artiﬁcial neural networks, we ﬁnally demonstrate that these encoding types
are suited to perform the complementary tasks of example discrimination andconcept classiﬁcation (Fig. 8).

Instead, we consider the possibility that CA3 can store both MF
and PP encodings for each memory and retrieve either of them.
Inhibitory tone selects between the two; with a higher activity
threshold, sparser MF patterns are more likely to be recovered, and the
opposite holds for denser PP patterns. By encoding the same memoryin two different ways, each can be leveraged for a different computational purpose. Conceptually, in terms of energy landscapes, sparser
patterns have narrower attractor basins than denser patterns because
fewer neurons actively participate (Fig. 1B). Moreover, less correlated
patterns are located farther apart compared to more correlated
patterns. Thus, MF energy basins tend to remain separate with barriers
between them, a property called pattern separation that maintains
distinctions between similar memories and is known to exist in DG18–20.
In contrast, PP energy basins tend to merge, which enables the clustering of similar memories into concepts. This proposed ability for CA3
to recall both individual experiences and generalizations across themwould explain observed features of hippocampal function. For
instance, remembering the details of a recent visit with an acquaintance is an example of hippocampus-dependent episodic memory1,21.
Meanwhile, hippocampal neurons can also generalize over your visits
and respond to many different representations of your acquaintance,
including previously unseen photographs or her name in spoken or
written form22,23.
To instantiate these ideas, we constructed a model for EC, DG,
and CA3 in which CA3 stores both MF and PP encodings of eachmemory (Fig. 1C). We observe that MF encodings remain distinct,
whereas PP encodings perform concept learning by merging similar
memories. Our model predicts relationships between coding
properties and network sparsity across phases of the theta oscillation,
which modulates inhibitory tone in the hippocampal region. We
tested these predictions across two publicly available datasets24,25,

and each analysis reveals that tuning of CA3 neurons is sharper
during sparse theta phases and broader during dense phases. This
supports our model and enriches our understanding of phase
coding in hippocampus. While our model does not include CA1,
we present comparative experimental analyses for this subﬁeld
in various Supplementary Figures. Beyond asserting the presence ofcomplementary encodings in CA3, we demonstrate that they can offer
functional advantages. Applying inspiration from our CA3 model anddata analysis toward machine learning, we introduced a plug-and-playloss function that endows artiﬁcial neural networks with bothcorrelated, PP-like and decorrelated, MF-like representations. These
networks can perform better in multitask learning compared to networks with single representation types, which suggests a promisingstrategy for helping neural networks to solve complex tasks. While theessential components of our networks are explained in the Results
section, their full descriptions and justiﬁcations are provided in the
Methods section with parameter values in Table 1.

Nature Communications | (2024)15:647

Results
MF encodings remain distinct while PP encodings build concepts in our model for CA3
We model how representations of memories are transformed
along the two pathways from EC to CA3 and then how the resultant
encodings are stored and retrieved in CA3. First, we focus on thetransformations between memories and their CA3 encodings.
The sensory inputs whose encodings serve as memories in our model
are FashionMNIST images26, each of which is an example belonging to
one of three concepts: sneakers, trousers, and coats (Fig. 2A). They areconverted to neural activity patterns along each projection from EC to
CA3 (Fig. 2B). Our neurons are binary with activity values of 0 or 1. Each
image iμν representing example ν in concept μ is ﬁrst encoded by EC
2

Articlehttps://doi.org/10.1038/s41467-024-44877-0

using a binary autoencoder, whose middle hidden layer activations
represent the patterns xECμν (Fig. 2C). Only 10% of the neurons areallowed to be active, so the representation is sparse, and there are
more EC neurons than image pixels, so the representation is overcomplete; sparse, overcomplete encoding models and autoencoder

Table 1 | Key hippocampus model parameters and their values
unless otherwise noted
Parameter

Value

See also

number of concepts

3

Fig. 2A

examples stored per concept s

1–100

Fig. 3EC network size NEC

1024

Eqs. (5) and (6),Fig. 2C, D

DG network size NDG

8192

Eq. (6), Fig. 2DCA3 network size NCA3

2048

Eqs. (6) and (8),Figs. 2D and 3

EC pattern density aEC

0.1

Eqs. (5) and (6),Fig. 2C, D

DG pattern density aDG

0.005

Eq. (6), Fig. 2DMF pattern density aMF

0.02

Eq. (6), Fig. 2D

PP pattern density aPP

0.2

Eq. (6), Fig. 2DEC pattern correlation ρEC

0.15

Eqs. (5) and (6),Fig. 2C, D

DG pattern correlation ρDG

0.02

Eq. (6), Fig. 2DMF pattern correlation ρMF

0.01

Eq. (6), Fig. 2D

PP pattern correlation ρPP

0.09

Eq. (6), Fig. 2D

PP pattern strength ζ

0.1

Eq. (8), Fig. 3A

fraction of neurons ﬂipped to form cue

0.01

Fig. 3B

rescaled threshold θ0

0–0.5

Eq. (11), Fig. 3C

rescaled inverse temperature β0

100

Eq. (11)

Fig. 2 | We model the transformation of memory representations along hippocampal pathways; MF and PP encodings of the same memories converge
at CA3. A Memories are FashionMNIST images, each of which is an example of aconcept. B Overview of model pathways. Encoding pathways correspond to the
biological architecture in Fig. 1A. Decoding pathways are used to visualize CA3
activity and are not intended to have biological signiﬁcance. C We use an autoencoder with a binary middle layer to transform each memory iμν into an EC patternxEC
μν . D From EC to CA3, we use random binary connectivity matrices to transform
posteach presynaptic pattern xpre
μν to a postsynaptic pattern xμν . E Enforcing sparser

Nature Communications | (2024)15:647

neural networks are common unsupervised models for natural image
processing27–30.
From EC, we produce DG, MF, and PP encodings with random,binary, and sparse connectivity matrices between presynaptic
and postsynaptic regions (Fig. 2D), i.e., from EC to DG, from DG to
CA3 via MF, and from EC to CA3 via PP. Each matrix transforms
presynaptic patterns xpre
μν into postsynaptic inputs, which are conat a desired density using
verted into postsynaptic patterns xpostμν
a winners-take-all approach. That is, the postsynaptic neurons
receiving the largest inputs are set to 1 and the others are set to 0. We
deﬁne density to be the fraction of active neurons, so lower values
correspond to sparser patterns. Enforcing a desired postsynaptic
pattern density is equivalent to adjusting an activity threshold. At
CA3, two encodings for each image converge: xMF
μν with density 0.02
and xPPμν with density 0.2. Not only are MF patterns sparser, they areless correlated with average correlation 0.01, compared to a corresponding value of 0.09 for PP patterns. Such an association
between sparsiﬁcation and decorrelation has been widely reported
across many theoretical models and brain regions10–13, and it is also
captured by our model. Decreasing postsynaptic pattern density(sparsiﬁcation) correspondingly decreases the postsynaptic correlation (decorrelation) for any presynaptic statistics (Fig. 2E). We contribute further insight by deriving an explicit mathematical formulathat connects densities and correlations of patterns in presynaptic andpostsynaptic networks:

ρpost =

Γhpﬃﬃﬃ
i
2erfc1 ð2apost Þ, apre + ρpre  apre ρpre  a2post

apost ð1  apost Þ
"
#
Z π
1
ϕ2
where Γ½ϕ,σ dψ exp 
1 + cos ψ
2π arccos σ,

ð1Þ

postsynaptic patterns in D promotes decorrelation. Dark gray indicates use of xEC
μν
as presynaptic patterns. Points indicate means and bars indicate standard deviations over 8 random connectivity matrices. Green indicates randomly generated
presynaptic patterns at various densities apre and correlations ρpre . Theoreticalcurves depict Eq. (1). F To visualize CA3 encodings, we pass them through a feedMF
PP
forward network trained to produce the corresponding xEC
μν for each xμν and xμν .
Images are then decoded using the autoencoder in C. Source data are provided as a
Source Data ﬁle.

3

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 3 | We model CA3 to store both MF and PP encodings of the same memories; MF examples remain distinct while PP examples build concept representations. A–C Overview of the Hopﬁeld-like model for CA3. A We store linear
combinations of MF and PP encodings, with greater weight on the former because
MF inputs are stronger. B Retrieval begins by initializing the network to a stored
pattern corrupted by ﬂipping the activity of randomly chosen neurons. C During
retrieval, the network is asynchronously updated with a threshold θ that controls
the desired sparsity of the recalled pattern. D, E Retrieval behavior using MF cues.
Examples from the three concepts depicted in Fig. 2A are stored. D Visualizations ofretrieved patterns. MF encodings, retrieved at high θ, maintain distinct representations of examples. PP encodings, retrieved at low θ, merge into concept
representations as more examples are stored (compare with average image in

Fig. 2A). E Overlap of retrieved patterns with target patterns: MF examples, PP
examples, or PP concepts deﬁned by averaging over PP examples and binarizing(Methods). Solid lines indicate means, shaded regions indicate standard deviations,and the dashed orange line indicates the theoretically estimated maximum value
for concept retrieval (Methods). In all networks, up to 30 cues are tested. F,
G Similar to D, E, but using PP cues. H Network capacities computed using random
MF and PP patterns instead of FashionMNIST encodings. Shaded regions indicate
regimes of high overlap between retrieved patterns and target patterns (Supplementary Information). MF patterns have density 0.01 and correlation 0. PP patterns
have density 0.5. I Similar to H, but overlaying capacities for MF examples and PP
concepts to highlight the existence of regimes in which both can be recovered.
Source data are provided as a Source Data ﬁle.

and erfc1 is the inverse complementary error function. In other words,
given the density apre and correlation ρpre of the presynaptic patterns
and the desired density apost of the postsynaptic patterns, the postsynaptic correlation ρpost is determined. Equation (1) is remarkable inthat only these four quantities are involved, revealing that at least insome classes of feedforward networks, other parameters such as network sizes, synaptic density, and absolute threshold values do notcontribute to decorrelation. It is derived in Supplementary Information,and its behavior is further depicted in Supplementary Fig. 2B, C.
Ultimately, the encoding pathways in Fig. 2C–E provide CA3 with aPPsparse, decorrelated xMF
μν and a dense, correlated xμν for each memory,in accordance with our biological understanding (Fig. 1A, B). Next, we
aim to store these patterns in an autoassociative model of CA3. Before
doing so, we develop visualization pathways that decode CA3

representations back into images, so memory retrieval can be intuitively evaluated. This is accomplished by training a continuous-valued
feedforward network to associate each MF and PP pattern with its
corresponding EC pattern (Fig. 2F). From there, the reconstructed EC
pattern can be fed into the decoding half of the autoencoder in Fig. 2C
to recover the image encoded by CA3. These decoding pathways arefor visualization only and are not designed to mimic biology, although
there may be parallels with the neocortical output pathway from CA3
to CA1 and deep layers of EC7. The neuroanatomical connectivity of
CA1 is more complex and includes temporoammonic inputs from EC as
well as strong secondary outputs through the subiculum, which also
reciprocally connects with EC.
Now, we model memory storage in the CA3 autoassociative network. For each example ν in concept μ, its MF encoding xMF
μν arrives at

Nature Communications | (2024)15:647

4

Articlehttps://doi.org/10.1038/s41467-024-44877-0

the proximal dendrites and its PP encoding xPP
μν arrives at the distal
dentrites of CA3 pyramidal cells (Fig. 3A). The relative strength of PP
inputs is weaker because PP synapses are located more distally and areobserved to be much weaker than MF synapses, which are even called
detonator synapses7,31,32. The inputs are linearly summed and stored ina Hopﬁeld-like network33, with connectivity
W ij ∼

XμνPP
0:9 x MFμνi + 0:1 x μνi




PP
0:9 x MFμνj + 0:1 x μνj ,ð2Þwhere i and j are respectively postsynaptic and presynaptic neurons.
Equation (2) captures the most crucial terms in Wij; see Methods for thefull expression. While we assume linear summation between xMF
μν andxPP
μν for simplicity, integration of inputs across CA3 dendritic compartments is known to be nonlinear17,34,35. Moreover, sublinear summation can also arise from a temporal offset between MF and PP
inputs, in which case changes in synaptic weights across pathways
could be weaker than those within the same pathway according tospike-timing-dependent plasticity5,6. In Supplementary Fig. 3F, weshow that network behavior can be maintained when nonlinearity is
introduced.
In previous models, CA3 would retrieve only MF encodings,
only PP encodings, or only the activity common between MF–PP
pairs15–17. We assess the ability of the network to retrieve either xMF
μν or
xPP
μν using either encoding as a cue (Fig. 3B). Each cue is corrupted byﬂipping randomly chosen neurons between active and inactive and is
set as the initial network activity. During retrieval, the network is
asynchronously updated via Glauber dynamics36. That is, at eachsimulation timestep, one neuron is randomly selected to be updated(Fig. 3C). If its total input from other neurons exceeds a threshold θ,then it is more likely to become active; conversely, subthreshold total
input makes silence more likely. The width of the sigmoid function in
Fig. 3C determines the softness of the threshold. A large width implies
that activation and silence are almost equally likely for recurrent input
near threshold. A small width implies that activation is almost guaranteed for recurrent input above threshold and almost impossible for
input below threshold. See Methods for the full expression of thisupdate rule.
The threshold θ represents the general inhibitory tone of CA3 andplays a key role in retrieval. At high θ, neural activity is disfavored, so
we expect the network to retrieve the sparser, more strongly stored
MF encoding of the cue. Upon lowering θ, more neurons are permitted
to activate, so those participating in the denser, more weakly stored PP
encoding should become active as well. Because our neurons arebinary, active neurons in either the MF or the PP encoding would havethe same activity level of 1, even though their connectivity strengths
differ. This combined activity of both encodings is almost the same as
the PP encoding alone, which contains many more active neurons.
Thus, we expect the network to approximately retrieve the PP
encoding at low θ.
Figure 3 D–G illustrates the central behavior of our CA3 model;
see Supplementary Fig. 3A, B for trouser and coat visualizations, which
behave similarly to the sneaker visualizations shown here. First using
MF encodings as cues, we seek to retrieve either MF or PP encodings by
respectively setting a high or low threshold. As we load the networkwith increasingly more stored examples, distinct MF examples canconsistently be retrieved with high threshold (Fig. 3D). Meanwhile,
retrieval of PP examples with low threshold fails above 1–2 examples
stored per concept. At large example loads, the network again
retrieves a sneaker memory when cued with sneaker examples. However, this memory is the same for all sneaker cues and appears similar
to the average image over all sneaker examples (Fig. 2A), which captures common sneaker features (Supplementary Fig. 2A). Thus, thenetwork is retrieving a representation of the sneaker concept. Notably,
concepts are never directly presented to the network; instead, theNature Communications | (2024)15:647

network builds them through the unsupervised accumulation of correlated examples. The retrieval properties visualized in Fig. 3D arequantiﬁed in Fig. 3E by computing the overlap between retrieved andtarget patterns. Across all example loads shown, retrieved MF patterns
overlap with target examples. As example load increases, retrieved PP
patterns transition from encoding examples to representing concepts.
We deﬁne the target pattern xPP
μ for a PP concept μ by activating themost active neurons across PP examples within that concept until the
PP density is reached, and the dashed line in Fig. 3E coarsely estimates
the largest overlap achievable (Methods).
The network capabilities observed for MF cues are preserved
when we instead use PP cues (Fig. 3F, G) or cues combining the neurons active in either encoding (Supplementary Fig. 3C–E); again, theselatter two are similar because MF encodings are sparse. Thus, retrieval
behavior is driven largely by the level of inhibition rather than theencoding type of cues. This feature implies that our model is agnostic
to whether memory retrieval in the hippocampus is mediated by theMF pathway, the PP pathway, or both. Computationally, it implies that
our model can not only retrieve two encodings for each memory but
also perform heteroassociation between them. Autoassociation andheteroassociation are preserved over large ranges in model parameters (Supplementary Fig. 3F).
To show that concept target patterns xPP
μ and average images
within concepts are indeed valid representations of concepts for ourimage dataset, we plot them in image space after transforming xPPμthrough the visualization pathway (Supplementary Fig. 2A). We
observe that these two representation types appear similar to eachother and lie near the centers of well-separated clusters of examples
for each concept. In machine learning, clustering around central points
is a common paradigm for unsupervised category learning, with kmeans clustering as an example. In cognitive science as well, clusteringhas been used as model for unsupervised category learning37,38, andcentral representations called prototypes can be used for category
assignment39. Thus, we conclude that xPP
μ and average images canindeed serve as concept representations. With more complex image
datasets, such as CIFAR1040, examples may not be clustered in image
space or in encoding space with our model’s simple autoencoder. To
learn concepts, nonlinear decision boundaries can identiﬁed using
supervised algorithms, but these complicated partitions of space may
not admit central prototypes that accurately represent concepts.
Alternatively, we can employ more sophisticated feature extraction
techniques to map examples into an encoding space that exhibits
clusters with simple boundaries between concepts. If that is achieved,
then central features such as averages within clusters in that space canagain serve as concept representations. More powerful feature
extraction can be incorporated in our model by substituting our simple autoencoder with, for example, an unsupervised variational autoencoder or a supervised deep classiﬁer.
We investigate retrieval more comprehensively by randomly
generating MF and PP patterns across a broader range of statistics
instead of propagating images along the hippocampal pathways
in Fig. 2B (Methods). For simplicity, we take MF examples to be
uncorrelated. In Fig. 3H, I, we show regimes for successful retrieval of
MF examples, PP examples, and PP concepts. For MF and PP
examples, the network has a capacity for stored patterns above which
they can no longer be retrieved (Fig. 3H). For PP concepts, the network
requires storage of a minimum number of examples below which
concepts cannot be built. As expected intuitively, fewer examples
are needed if they are more correlated, since common features can be
more easily deduced. Figure 3I overlays retrieval regimes for MF
examples and PP concepts. When the number of concepts is low,
there exists a regime at intermediate numbers of stored examples inwhich both examples and concepts can be retrieved. This
multiscale retrieval regime corresponds to the network behavior
observed in Fig. 3D–G, and it is larger for more correlated PP
5

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 4 | The CA3 model can alternate between MF example and PP concept
representations under an oscillating threshold. Four scenarios are considered: abaseline condition with abrupt threshold changes, sinusoidal threshold changes,
threshold values of 0.55 and 0.25 instead of 0.6 and 0.2, and the weak input of an
MF cue throughout the simulation instead of only at the beginning. A Patternoverlap dynamics. Each panel shows, from top to bottom, the threshold, overlaps
with MF examples, and overlaps with PP concepts. The dashed orange line indicates
the theoretically estimated maximum value for concept retrieval (Methods).
B Visualizations of retrieved patterns show alternation between examples andconcepts. In the baseline case, various examples are explored; in the cuethroughout case, the same cued example persists. C Summary of retrieval behavior
between update cycles 60 to 120. For each scenario, 20 cues are tested in each of 20
networks. Each panel depicts the fractions of simulations demonstrating various
example (left) and concept (right) behaviors. In all networks, 50 randomly chosen
examples from each of the 3 concepts depicted in Fig. 2A are stored. One update
cycle corresponds to the updating of every neuron in the network (Methods).
Source data are provided as a Source Data ﬁle.

encodings. On the other hand, its size does not substantially changewith the sparsity of MF patterns (Supplementary Fig. 3G, H). Our
capacity values agree with theoretical formulas calculated using
techniques from statistical physics41. In all, our networks with randomly generated patterns demonstrate that our results generalize tolarger networks that store more examples in more concepts and are
not idiosyncratic to the pattern generation process in Fig. 2.
To further explore the heteroassociative capability of our network, we cue the network with an MF pattern and apply a time-varying
threshold during retrieval. The network representation can thenalternate between the PP concept of the original cue during oscillation

phases with low threshold and various MF examples of thatconcept during phases with high threshold (Fig. 4A, B). Sharply andsinusoidally varying threshold values both produce this behavior.
From one oscillation cycle to the next, the MF encoding can hop
among different examples because concept information is preferentially preserved over example information during low-thresholdphases. If we weakly apply the MF cue as additional neural
input throughout the simulation (Methods), the network will only
alternate between the target MF example and the target PP concept.
This condition can represent memory retrieval with ongoing
sensory input. If we decrease the amplitude of the oscillation,

Nature Communications | (2024)15:647

6

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 5 | Place ﬁeld data support the model prediction that sparser theta phases
should preferentially encode ﬁner, example-like positions. A Our CA3 model
predicts that single neurons convey more information per spike about example
identity during sparse regimes. Each point represents a neuron, n = 50. B Example
CA3 place cell activity along a linear track. Each spike is represented by two points
at equivalent phases with histograms over position (top) and over theta phase(right). C Activity by theta phase for 5 CA3 place cells. D To test our model, weconstrue CA3 place cells to store ﬁne positions as examples, which can combine
into coarser regions as concepts. Here, we focus on example encoding. E Our model
predicts that CA3 place ﬁelds are more sharply tuned during sparse theta phases.
An alternative hypothesis is sharper tuning during dense phases. F Example phaseprecessing place ﬁeld. G Activity (black), raw position information per spike (blue),and mean null-matched position information (gray) by theta phase for the ﬁeld in F.

Sparsity-corrected position information is the difference between the raw andmean null-matched values. H Null-matched place ﬁeld obtained by replacing spike
positions, but not phases, with uniformly distributed random values. I Shufﬂed
place ﬁeld obtained by permuting spike phases and positions. J, K Similar toF, G, but for a place ﬁeld that is not precessing. L Average difference in position
information between the sparsest and densest halves of theta phases. For all cell
populations, sparse phases convey more position information per spike. Each point
represents a ﬁeld. All and shufﬂed n = 35, precessing n = 21, and other n = 14.
Numbers indicate p-values calculated by two-tailed Wilcoxon signed-rank tests
except for the comparison between precessing and other, which is calculated by
the two-tailed Mann-Whitney U test. For all results, spikes during each traveling
direction are separately analyzed. In A and L, information is sparsity-corrected withhorizontal lines indicating medians. Source data are provided as a Source Data ﬁle.

alternation between examples and concepts is disrupted and the network favors one encoding type over the other. We quantify the distribution of network behaviors during high- and low-threshold phases
in Fig. 4C. The proportion of simulations in which single MF patterns
are retrieved, the persistence of the target PP concept, and other
retrieval properties vary with network parameters. In Supplementary
Fig. 4A, B, we present analogous results for randomly generated MF
and PP patterns demonstrating that these retrieval properties also
depend on MF pattern sparsity. All in all, while our network canrepresent either examples or concepts at each moment in time, an
oscillating threshold provides access to a range of representations
over every oscillation cycle.

Place cell data reveals predicted relationships between encoding properties and theta phase

Nature Communications | (2024)15:647

The central feature of our CA3 model is that an activity thresholddetermines whether the network retrieves example or concept
encodings. We claim that the theta oscillation in CA3 physiologically
implements this threshold and drives changes in memory scale. To be
speciﬁc, our model predicts that single neurons should convey more
information per spike about example identity during epochs of sparser
activity (Fig. 5A). This single-neuron prediction can be tested by analyzing publicly available datasets of CA3 place cells. Figure 5B shows
one example place cell recorded while a rat traverses a linear track24,42.
During locomotion, single-neuron activity in CA3 is strongly
7

Articlemodulated by the theta oscillation (Fig. 5C); we use this activity as anindicator of network sparsity since a relationship between the two has
been observed (Fig. 3 in Skaggs et al.43). We assume an equivalence
between the encoding of images by our CA3 model and the encodingof spatial positions by CA3 place cells (Fig. 5D). Examples are equivalent to ﬁne positions along the linear track. Just as similar examples
merge into concepts, nearby positions can aggregate into coarser
regions of space. Through this equivalence, we can translate the prediction about example information per spike (Fig. 5A) into a prediction
about spatial tuning (Fig. 5E). During denser theta phases, place ﬁelds
should be broader, which corresponds to lower position information
per spike. This prediction relies on our claim that the theta oscillationin CA3 acts as the inhibitory threshold of our model. A priori, thealternative prediction that place ﬁelds are sharper during dense theta
phases is an equally valid hypothesis. Higher activity may result fromstrong drive by external stimuli that the neuron serves to encode, while
lower activity may reﬂect noise unrelated to neural tuning. The sharpening of visual tuning curves by attention is an example of thisalternative prediction44. From a more general perspective, the modeland alternative predictions in Fig. 5E roughly correspond to subtractive and divisive modulation of ﬁring rates, respectively. Bothkinds of inhibitory effects are found in cortical circuits45–47. We will now
test whether experimental data reﬂect our model prediction of sharper
place ﬁeld tuning with higher spatial information during sparser theta
phases, which would support a subtractive role of theta as an oscillating inhibitory threshold over a divisive one.
First, we investigate the encoding of ﬁne, example-like positions
by analyzing phase-dependent tuning within single place ﬁelds. We
use the Collaborative Research in Computational Neuroscience(CRCNS) hc-3 dataset contributed by György Buzsáki andcolleagues24,42. Figure 5F shows one extracted ﬁeld that exhibits phase precession (for others, see Supplementary Fig. 5A). At each phase,
we compute the total activity as well as the information per
spike conveyed about position within the ﬁeld (Fig. 5G and Supplementary Fig. 5B). It is well known that the estimation of information
per spike is strongly biased by sparsity. Consider the null data in
Fig. 5H that is matched in spike phases; spike positions, however, arerandomly chosen from a uniform distribution. In the large spikecount limit, uniformly distributed activity should not convey any
information. Yet, the null data show more position information
per spike during sparser phases (Fig. 5G). To correct for this bias,
we follow previous protocols and subtract averages over many
null-matched samples from position information48. In all of ourcomparisons of information between sparse and dense phases,
including the model prediction in Fig. 5A, we report sparsity-corrected
information. For further validation, we generate a shufﬂed datasetthat disrupts any relationship between spike positions and phases
found in the original data (Fig. 5I). Figure 5J, K illustrates a second
place ﬁeld whose tuning also depends on theta phase but does
not exhibit precession. For each theta-modulated CA3 place ﬁeld, wepartition phases into sparse and dense halves based on activity, andwe average the sparsity-corrected position information per spike
across each partition. CA3 place ﬁelds convey signiﬁcantly more
information during sparse phases than dense phases (Fig. 5L). This
relationship is present in both phase-precessing and other ﬁelds(although slightly non-signiﬁcantly in the latter) and is absent in the
shufﬂed data. Thus, experimental data support our model’s predictionthat CA3 encodes information in a ﬁner, example-like manner during
sparse theta phases. Notably, CA1 place ﬁelds do not convey more
information per spike during sparse phases, which helps to show thatour prediction is nontrivial and demonstrates that the phase behavior
in CA3 is not just simply propagated forward to CA1 (Supplementary Fig. 5C).
To characterize the relationship between information and theta
phase more precisely, we aggregate spikes over phase-precessing

Nature Communications | (2024)15:647

https://doi.org/10.1038/s41467-024-44877-0

ﬁelds in CA3 and in CA1 (Supplementary Fig. 5D–G). This process
implicitly assumes that each phase-precessing ﬁeld is a sample of ageneral distribution characteristic to each region. These aggregateﬁelds recapitulate the single-neuron results that CA3 spikes are
uniquely more informative during sparse phases (Supplementary
Fig. 5H). They also reveal how position information varies with otherﬁeld properties over theta phases (Supplementary Fig. 5I, J). For
example, information is negatively correlated with ﬁeld width, conﬁrming the interpretation that more informative phases have sharper
tuning curves (Fig. 5E). In CA3, information is greatest during earlyprogression through the ﬁeld, which corresponds to future locations,with a smaller peak during late progression, which corresponds to past
locations. In contrast, past locations are more sharply tuned in CA1.
Thus, different hippocampal subﬁelds may differentially encode pastand future positions across the theta cycle; we will return to this topic
in the Discussion.
Next, we turn our attention to the representation of concepts
instead of examples. Our model predicts that single neurons exhibit
more concept information per spike during dense activity regimes(Fig. 6A). To test this prediction using the same CRCNS hc-3 dataset,
we invoke the aforementioned equivalence between concepts in ourmodel and coarser positions along a linear track (Fig. 6B). Thus, single
CA3 neurons should encode more information per spike about coarse
positions during dense theta phases. Previously, to test for ﬁner
position encoding in Fig. 5, we divided single place ﬁelds into multiple
position bins during the computation of information. Here, we analyze
encoding of coarser positions by choosing large position bins acrossthe whole track (Fig. 6C, D). We consider different bin sizes to characterize at which scale the merging of examples into concepts occurs.
When we again compute the average difference in sparsity-corrected
position information per spike between sparse and dense theta phases,
we ﬁnd that dense phases are the most preferentially informative at thecoarsest scales (Fig. 6E). CA1 place cells also exhibit this property(Supplementary Fig. 6B). Crucially, differences between sparse anddense phases are not seen in shufﬂed data, which supports the validityof our analysis methods (Fig. 6F). Our results are further bolstered bytheir preservation under a different binning procedure (Supplementary Fig. 6C–E). Thus, coarse positions along a linear track can be best
distinguished during dense theta phases, in agreement with ourmodel. Note that we always consider 4 bins at a time even for trackscales smaller than 1/4, because changing the number of bins acrossscales introduces a bias in the shufﬂed data (Supplementary
Fig. 6F–H). At ﬁner scales, this process sometimes fails to capture
entire ﬁelds and artiﬁcially splits them (Fig. 6C, left). Here, we adopt aneutral approach and do not adjust our partitions to avoid these cases;
the opposite approach was adopted in Fig. 5 where intact ﬁelds were
explicitly extracted. These different approaches may explain why at
the ﬁnest scales in Fig. 6E, sparse phases do not convey more information like they do in Fig. 5L.
In our model, concepts are formed by merging examples acrossall correlated features. While track position can be one such feature,
we now assess whether our predictions also apply to another one. In
the CRCNS hc-6 dataset contributed by Loren Frank and colleagues,
CA3 place cells are recorded during a W-maze alternation task in whichmice must alternately visit left and right arms between runs along thecenter arm25. It is known that place cells along the center arm canencode the turn direction upon entering or leaving the center arm in
addition to position49,50. Again, our model predicts that sparse theta
phases preferentially encode speciﬁc information (Fig. 7A), so theyshould be more tuned to a particular turn direction (Fig. 7B). During
dense phases, they should generalize over turn directions and solely
encode position.
Figure 7C shows spikes from one CA3 place cell accumulated over
outward runs along the center arm followed by either left or right turns(Supplementary Fig. 7A). For each theta phase, we compute the total
8

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 6 | Place cell data support the model prediction that denser theta phases
should preferentially encode coarser, concept-like positions. A Our CA3 model
predicts that single neurons convey more information per spike about concept
identity during dense regimes. Each point represents a neuron, n = 50. B To test ourmodel, we construe CA3 place cells to store ﬁne positions as examples, which can
combine into coarser regions as concepts. Here, we focus on concept encoding.
C We calculate position information at various track scales over windows of 4
contiguous bins. D Activity (black), raw position information per spike (blue), andmean null-matched position information (gray) by theta phase for the red windows
in C. Sparsity-corrected position information is the difference between the raw andmean null-matched values. E Average difference in position information between
the sparsest and densest halves of theta phases. For coarser scales, dense phases
convey more position information per spike. Each point represents values from aplace cell averaged over all windows. Track scale 1/16 n = 47, 1/8 n = 49, and 1/4
n = 56. Numbers indicate p-values calculated by two-tailed Wilcoxon signed-rank
tests for each scale and by Spearman’s ρ for the trend across scales. F Similar toE, but for shufﬂed data whose spike phases and positions are permuted. For all
results, spikes during each traveling direction are separately analyzed. In A, E, andF, information is sparsity-corrected with horizontal lines indicating medians.
Source data are provided as a Source Data ﬁle.

activity, the turn information per spike (ignoring position), and themean information of null-matched samples used for sparsity correction (Fig. 7D). Figure 7E, F shows similar results for inward runs (forothers, see Supplementary Fig. 7B, C). For both outward and inward
runs, sparsity-corrected turn information per spike is greater during
sparse theta phases compared to dense phases (Fig. 7G). This ﬁnding is
not observed in data in which theta phase and turn direction are
shufﬂed (Fig. 7G, H). Not only do these results support our model, theyalso reveal that in addition to splitter cells that encode turn direction
over all theta phases51, CA3 contains many more place cells that encode
it only at certain phases (Supplementary Fig. 7D). The difference
between sparse and dense phases is signiﬁcantly greater in CA3 than it
is in CA1 (Supplementary Fig. 7E, F). Thus, our subﬁeld-speciﬁc resultsfor example encoding are consistent across position and turn direction. Aggregate neurons, formed by combining spikes from more
active turn directions and those from less active turn directions,demonstrate similar tuning properties to individual neurons (Supplementary Fig. 7G–I).
Beyond the single-neuron results presented above, we seek to test
our predictions at the population level. To do so, we perform phasedependent Bayesian population decoding of turn direction during
runs along the center arm (Fig. 7I). This analysis requires multiple
neurons with sufﬁciently sharp tuning to be simultaneously active
across all theta phases; it can be used to decode left versus right turns,
whereas an analogous decoding of track position, which spans a muchbroader range of values, is intractable with our datasets. We ﬁnd that

Nature Communications | (2024)15:647

9

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 7 | W-maze data support the model prediction that sparser theta phases
should preferentially encode turn direction in addition to position. A Same as
Fig. 5A. B To test our model, we construe CA3 place cells to store turn directions
during the central arm of a W-maze alternation task as examples. By combining
examples, concepts that generalize over turns to solely encode position can beformed. C–H Single-neuron information results. C Example place cell that is active
during outward runs. Each spike is represented by two points at equivalent phaseswith different colors representing different future turn directions. D Activity(black), raw turn information (blue), and mean null-matched turn information(gray) by theta phase for the neuron in C. Sparsity-corrected turn information is thedifference between the raw and mean null-matched values. E, F Similar to C, D, but
for a place cell that is active during inward runs with colors representing past turn
directions. G Average difference in turn information between the sparsest anddensest halves of theta phases. For all cell populations, sparse phases convey more
turn information per spike. Each point represents a place cell. All and shufﬂed
n = 99, outward runs n = 56, and inward runs n = 43. Numbers indicate p-values

calculated by two-tailed Wilcoxon signed-rank tests except for the comparison
between outward and inward runs, which is calculated by the two-tailed MannWhitney U test. H Cumulative distribution functions for values in G. I–L Bayesian
population decoding results. I Likelihood of left (L) or right (R) turns during fourruns along the center arm using spikes from either the sparsest or densest halves oftheta phases. J Sparse encodings exhibit greater conﬁdence about turn direction.
Vertical lines indicate medians with p-value calculated by the two-tailed MannWhitney U test. K Average difference in maximum likelihood estimation accuracy
between the sparsest and densest halves of theta phases. Sparse phases encode
turn direction more accurately. Each point represents one run averaged overdecoded timepoints. All runs and shufﬂed n = 91. Numbers indicate p-values calculated by two-tailed Wilcoxon signed-rank tests. L Cumulative distribution functions for values in K. For all results, spikes during each traveling direction are
separately analyzed. In A, G, and H, information is sparsity-corrected. Source data
are provided as a Source Data ﬁle.

the CA3 population likelihood exhibits greater conﬁdence during
sparse phases (Fig. 7J). From a Bayesian perspective, the population
expresses stronger beliefs about turn direction during sparse phases
and is more agnostic during dense phases. If pressed to choose thedirection with a higher likelihood as its estimate, CA3 is also moreaccurate during sparse phases (Fig. 7K, L). These results match our
predictions in Fig. 7A, B and bolster our single-neuron results.

Moreover, they are speciﬁc to CA3, as similar conclusions cannot bemade about the CA1 place cell population (Supplementary Fig. 7J–L).
In summary, extensive data analysis reveals experimental
support for our CA3 model over two datasets collected by different
research groups, across two encoding modalities, for both example
and concept representations, and at both the single-neuron andpopulation level.

Nature Communications | (2024)15:647

10

Articlehttps://doi.org/10.1038/s41467-024-44877-0

Fig. 8 | Complementary encodings inspired by CA3 can improve machinelearning performance in a complex task. A We extend the MNIST dataset byrandomly assigning an additional set label to each image. B–F We train a multilayer
perceptron to either classify digits or identify sets. B Network architecture. Eachhidden layer contains 50 neurons. C Task structures. Digit classiﬁcation requires
building concepts and is tested with held-out test images. Set identiﬁcation
requires distinguishing examples and is tested with noisy train images. D We apply
the DeCorr loss function (Eq. (3)) to decorrelate encodings in the ﬁnal hidden layer,
in analogy with MF patterns in CA3. Without an encoding loss function, image
correlations are preserved, in analogy with PP patterns. E, F DeCorr decreases
concept learning performance and increases example learning performance. Points
indicate means and bars indicate standard deviations over 32 networks. G–J We
train a multilayer perceptron to simultaneously classify digits and identify sets.
G Network architecture. Each hidden layer contains 100 neurons. The train dataset

contains 1000 images and 10 sets. H We apply the HalfCorr loss function (Eq. (4)) todecorrelate encodings only among the second half of the ﬁnal hidden layer. Correlated and decorrelated encodings are both present, in analogy with MF and PP
patterns across the theta cycle in CA3. I DeCorr networks generally perform betterat example learning but worse at concept learning compared to baseline. HalfCorr
networks exhibit high performance in both tasks. Open symbols represent individual networks and ﬁlled symbols represent means over 64 networks. J Inﬂuence ofeach neuron in HalfCorr networks on concept and example learning, deﬁned as theaverage decrease in accuracy upon clamping its activation to 0. Correlated neurons(orange bars) are more inﬂuential in concept learning, and decorrelated neurons(purple bars) are more inﬂuential in example learning. For all results, p-values arecomputed using unpaired two-tailed t-tests. Source data are provided as a Source
Data ﬁle.

CA3-like complementary encodings improve neural network
performance in multitask machine learning

images based on common features, which resembles concept learningin our CA3 model, and the latter requires discerning differences
between similar images, which resembles example learning in our CA3
model (Fig. 8C). We use a held-out test dataset to evaluate digit classiﬁcation performance and corrupted images from the train dataset toevaluate set identiﬁcation performance.
In our CA3 model, we found that examples were preferentially
encoded by the decorrelated MF pathway and concepts by the correlated PP pathway (Fig. 3). In an analogous fashion, we seek to manipulate the correlation properties within the ﬁnal hidden layer of ourperception, whose activations sα serve as encodings of the input
images iα. In particular, we apply a DeCorr loss function, which penalizes correlations in sα between every pair of items α, β in a training

We have observed how CA3 encodes behaviorally relevant information
at different scales across theta phases. Can these different types of
encodings be useful for solving different types of tasks? Can they evenbeneﬁt neural networks designed for machine learning, abstracting
away from the hippocampus? To address these questions, we turn to aclassic paradigm in machine learning: a multilayer perceptron trainedon MNIST handwritten digit images52. First, we augment the MNIST
dataset by randomly assigning an additional label to each image: a set
number (Fig. 8A). We train the fully connected feedforward network to
perform one of two tasks: classiﬁcation of the written digit or identiﬁcation of the assigned set (Fig. 8B). The former requires clustering of
Nature Communications | (2024)15:647

11

Articlehttps://doi.org/10.1038/s41467-024-44877-0

batch (Fig. 8D):

performance, especially since sparse coding is known to offer certain
computational advantages as well as greater energy efﬁciency27,54,55.
LDeCorr ≈1X
Pearsonðsα ,sβ Þ2 :2 α,β2

ð3Þ

batch

DeCorr mimics the MF pathway; the equation is approximate due to aslight modiﬁcation of the Pearson correlation formula to aid numericalconvergence (Methods). Alternatively, we consider the baseline condition with no loss function on hidden layer activations, which preserves natural correlations between similar images and mimics the PP
pathway. Indeed, we observe that different encoding properties are
suited for different tasks. Baseline networks perform better in concept
learning (Fig. 8E) while DeCorr networks perform better in example
learning (Fig. 8F), and these effects vary consistently with the strengthof the DeCorr loss function (Supplementary Fig. 8A, B). Thus, DeCorr
allows us to tune encoding correlations in neural networks to highlight
input features at either broader or ﬁner scales. Tasks can be solved
more effectively by matching their computational requirements withthe appropriate encoding scale. Note that DeCorr is different from the
DeCov loss function previously developed to reduce overﬁtting53.
DeCorr decorrelates pairs of inputs across all neurons in the speciﬁedlayer, whereas DeCov decorrelates pairs of neurons across all inputs.
As a regularizer that promotes generalization, DeCov improves digitclassiﬁcation and does not substantially improve set identiﬁcation,
which contrasts with the effect of DeCorr (Supplementary Fig. 8C, D).
Complex tasks, including those performed by biological systems,
may require information to be processed at different scales of correlation. In CA3, a spectrum of encodings is available during each theta
cycle. Can neural networks take advantage of multiple encodings? Wetackle this question by asking a perceptron to simultaneously perform
digit classiﬁcation and set identiﬁcation (Fig. 8G). In addition to thebaseline and DeCorr networks, we deﬁne a HalfCorr loss function(Fig. 8H):
LHalfCorr ≈1X
half 2
Pearsonðshalf
α ,sβ Þ ,2 α,β2

ð4Þ

batchrepresents the second half of neurons in the ﬁnal hidden
where shalf
αlayer. After training with this loss function, the neural representationconsists of both a correlated, PP-like component in the ﬁrst half and adecorrelated, MF-like component in the second half. When we evaluate
these networks on both digit classiﬁcation and set identiﬁcation, we
see that baseline and DeCorr networks behave similarly to how they
did on single tasks. Compared to baseline, DeCorr networks performbetter in example learning at the cost of poorer concept learning(Fig. 8I). However, HalfCorr networks do not suffer from this tradeoff
and perform well at both tasks. Their superior performance ismaintained over a variety of network and dataset parameters(Supplementary Fig. 8E). Moreover, HalfCorr networks learn topreferentially use each type of encoding for the task to which it isbetter suited. We use the decrease in task accuracy upon silencing aneuron as a metric for its inﬂuence on the task. Correlated neurons are
more inﬂuential in concept learning and decorrelated neurons inexample learning (Fig. 8J).
Note that we do not manipulate pattern sparsity in these artiﬁcial
networks. Sparsiﬁcation can be useful in the hippocampus because it
provides a biologically tractable means of achieving decorrelation. It
also allows biological networks to access both less and more correlated representations by changing the level of inhibition. Instead, we
can directly manipulate correlation through the DeCorr and HalfCorr
loss functions. Under some conditions, the decorrelated half of the
ﬁnal hidden layer in HalfCorr networks indeed exhibits sparser activation than the correlated half (Supplementary Fig. 8F). It is possiblethat directly diversifying sparsity can also improve machine learning

Nature Communications | (2024)15:647

Discussion
The hippocampus is widely known to produce our ability to recall
speciﬁc vignettes as episodic memories. This process has been
described as indexing every sensory experience with a unique neuralbarcode so that separate memories can be independently
recovered56,57. Recently, research has shown that the hippocampus isalso important in perceiving commonalities and regularities acrossindividual experiences, which contribute to cognitive functions suchas statistical learning58,59, category learning60–63, and semantic
memory64–66. Evidence for this has been obtained largely throughhuman studies, which can present and probe memories in controlled
settings. However, the detailed circuit mechanisms used by the hippocampus to generalize across experiences while also indexing themseparately are not known.
Our analysis of rodent place cell recordings reveals that single CA3
neurons alternate between ﬁner, example-like representations andbroader, concept-like representations of space across the theta cycle(Figs. 5–7). These single-neuron results extend to the network level,
which alternatively encodes more speciﬁc and more general spatial
features in a corresponding manner (Fig. 7). If we accept that place
cells store these features as spatial memories, then our experimental
analysis reveals that CA3 can access memories of different scales atdifferent theta phases. We propose that the computational mechanism
underlying these observations is the multiplexed encoding of eachmemory at different levels of correlation (Figs. 2 and 8). We show thatCA3 can biologically implement this mechanism through the storageof both sparse, decorrelated MF inputs and dense, correlated PP inputs
and their alternating retrieval by the theta oscillation, which acts as anactivity threshold and subtractively modulates neural activity (Fig. 2, 3,
and 4). Our model performs successful pattern completion of bothtypes of encodings, suggesting that patterns across the theta cycle cantruly function as memories that can be recovered from partial cues.
Alone, our secondary analyses of experimental data contribute toa large set of observations on how coding properties vary with theta
phase in the hippocampus. Of note is phase precession, in which different phases preferentially encode different segments within a ﬁring
ﬁeld as it is traversed, with later phases tuned to earlier segments67.
Phase precession is most widely reported for place cells and traversalsof physical space, but it also appears during the experience of othersequences, such as images and tasks68–70. Our analysis implies that the
sharpness of tuning is not constant throughout traversals. In particular, CA3 neurons are more sharply tuned at early positions in placeﬁelds, while CA1 neurons are more sharply tuned at late positions(Supplementary Fig. 5I, J). Transforming these conclusions about
position into those about time through the concept of theta sequences, CA3 represents the future more precisely, while CA1 represents
the future more broadly. The latter is consistent with the idea that CA1
may participate in the exploration of multiple possible futurescenarios71. Furthermore, our W-maze analysis reveals that certain
hippocampal neurons which do not obviously encode an external
modality across all theta phases, such as turn direction, may do so only
during sparse phases (Supplementary Fig. 7D). This observation adds
to the subtleties with which the hippocampus represents theexternal world.
Other groups have investigated the variation of place ﬁeld
sharpness with theta phase in CA1, not CA3, and their results are largelyin agreement with our CA1 analyses. Skaggs et al.43 partitioned theta
phases into halves, one of which with higher activity than the other,and found more information per spike during the less active half. We
observe no difference at the single-neuron level, though our W-mazeresults are only slightly non-signiﬁcant (Supplementary Figs. 5C and
7E). Their partitions differ from ours by 30∘ and they employ a different
12Articlebinning technique, both of which can inﬂuence the results. The lessinformative phases in their work correspond to future positions, which
we also observe (Supplementary Fig. 5J). Note that their computationof sparsity is performed along a different axis compared to ours; using
terms from Willmore and Tolhurst72, they use the lifetime density while
we compute the population density, which fundamentally differ.
Ujfalussy and Orbán73 also found that phases with larger ﬁeld sizes
correspond to future positions. Mehta et al.74 considered phasedependent tuning within CA1 place ﬁelds, but they calculate ﬁeld width
over theta phase as a function of ﬁeld progress, whereas we do the
opposite. Similarly, Souza and Tort75 considered phase tuning at various ﬁeld progresses. Both sets of results appear to be compatible withours, but a direct comparison cannot be made. Overall, our work offers
original insights into hippocampal phase coding not only by focusing
on CA3, which behaves differently from CA1, but also by elucidating aconnection between tuning width and network sparsity. Intriguingly,
Pfeiffer and Foster76 found a relationship between CA1 replay speed
and slow gamma oscillation phase, which modulates network activity
during quiescence. This observation opens the possibility that otheroscillations can leverage the connection between tuning and sparsity
when the dominant theta rhythm is absent.
Better memory performance has been associated with greater
theta power during both encoding77–82 and retrieval79,82–84. Our modelcannot explain the former because it does not contain a theta oscillation during encoding. It does, however, offer a possible explanation
for the latter observation. We simulated retrieval under an oscillating
threshold with lower amplitude and observed that the network stallson MF example encodings instead of alternating with PP concept
encodings (Fig. 4A). Thus, the biological processes that produce larger
theta amplitudes, such as stronger medial septum inputs or changes inneuromodulator concentrations85, may promote memory recall by
granting access to wider ranges of pattern sparsities and, consequently, representational scales.
The temporal coordination between memory storage and retrieval is also biologically signiﬁcant. We make the major simpliﬁcation of
separately simulating memory storage and retrieval. These two operating regimes can represent different tones of a neuromodulator suchas acetylcholine, which is thought to bias the network towards
storage86. Another proposal is that storage and retrieval preferentiallyoccur at different theta phases, motivated by the variation in long-term
potentiation (LTP) strength at CA1 synapses across the theta cycle87–89.
Although this idea focuses on plasticity in CA1, it is possible that storage and retrieval also occur at different phases in CA3. Note that ourexperimental analysis reveals a sharp dip in position information
around the sparsest theta phase in both CA3 and CA1 (Supplementary
Fig. 5E, G). This phase may coincide with the storage of new inputs,
during which the representation of existing memories is momentarily
disrupted; the rest of the theta cycle may correspond to retrieval. This
interpretation could motivate excluding the sparsest theta phase fromfurther analysis, since our model predictions only regard memory
retrieval. However, we take a conservative approach and include allphases. Interestingly, recent work reported that the strength of LTP in
CA1 peaks twice per theta cycle90, suggesting for our model that MF
and PP patterns could have their own storage and retrieval intervals
during each theta cycle.
Our work connects hippocampal anatomy and physiology withfoundational attractor theory. Among others, Tsodyks andFeigel’man91 observed that sparse, decorrelated patterns can be stored
at high capacity, and Fontanari92 found that dense, correlated patterns
can merge into representations of common features. We demonstratethat both types of representations can be stored and retrieved in thesame network, using a threshold to select between them. This capability can be given solid theoretical underpinnings using techniquesfrom statistical mechanics41. The convergence of MF and PP pathways
in CA3 has also been the subject of previous computational

Nature Communications | (2024)15:647

https://doi.org/10.1038/s41467-024-44877-0

investigations15–17. In these models, CA3 stores and retrieves one
encoding per memory, while our model asserts that multiple encodings for the same memory alternate across the theta cycle. Anotherseries of models proposes, like we do, that the hippocampus cansimultaneously maintain both decorrelated, example-like encodings
and correlated, concept-like encodings93,94. These encodings converge
at CA1 and each type is not independently retrieved there, which differs from our model. In a related hippocampal model, the PP pathway
was shown to be crucial for learning cue-target associations in thepresence of additional context inputs to EC that drift over time95.
Successful learning requires the network to perform decontextualization and abstract away the slowly varying context inputs, which is,
like concept learning, a form of generalization. EC has also been
hypothesized to differentially encode inputs upstream of CA3, withspeciﬁc sensory information conveyed by lateral EC and common
structural representations by medial EC96. Further experimental
investigation into the contributions of various subregions would help
to clarify how the hippocampus participates in memory generalization.
Finally, our work addresses how CA3-like complementary
encodings can be computationally leveraged by neural networks tosolve complex tasks. We conceptually extend our results about CA3
and introduce a HalfCorr loss function that diversiﬁes hidden layer
representations to include both correlated and decorrelated components (Fig. 8). HalfCorr networks can better learn tasks that involve
both distinction between similar inputs and generalization acrossthem. They are simultaneously capable of pattern separation andcategorization even based on small datasets, demonstrating a possible
advantage of brain computation over conventional deep learning. Yet,
we deliberately chose a neural architecture that differs from that of the
CA3 network to test the scope over which complementary encodings
can improve learning. Instead of a recurrent neural network storing
patterns of different sparsities through unsupervised Hopﬁeld learning rules, we implemented a feedforward multilayer perceptron, aworkhorse of supervised machine learning. The success of HalfCorr
networks in this scenario supports the possibility that HalfCorr can bebroadly applied as a plug-and-play loss function to improve computational ﬂexibility.
Functional heterogeneity is commonly invoked in the design ofmodern neural networks. It can be implemented in the form of deep or
modular neural networks in which different subnetworks performdifferent computations in series or parallel, respectively97,98. Of note,
Kowadlo et al.99 constructed a deep, modular network inspired by the
hippocampus for one-shot machine learning of both concepts andexamples. As an aside, their architecture shares similarities with our
hippocampus model in Fig. 2, but their Hopﬁeld-like network for CA3
only stores MF patterns and inactivating these recurrent connections
does not affect network performance. In contrast to these networks
with specialized subnetworks, we propose a different paradigm inwhich a loss function applied differentially across neurons promotes
heterogeneity within a single layer. This idea can be extended from thetwo components of HalfCorr networks, correlated and decorrelated,by assigning a different decorrelation strength to each neuron andthereby producing a true spectrum of representations. Furthermore,heterogeneity in other encoding properties such as mean activation,
variance, and sparsity may also improve performance in tasks withvarying or unclear computational requirements. Such tasks are notlimited to multitask learning, but also include continual learning100,
reinforcement learning101, and natural learning by biological brains.

Methods
Transformation of memories along hippocampal pathways
Binary autoencoder from images to EC. Our memories are 256 images from each of the sneaker, trouser, and coat classes in the FashionMNIST dataset26. We train a fully connected linear autoencoder onthese images with three hidden layers of sizes 128, 1024, and 128. Batch
13

Articlehttps://doi.org/10.1038/s41467-024-44877-0

normalization is applied to each hidden layer, followed by a rectiﬁedlinear unit (ReLU) nonlinearity for the ﬁrst and third hidden layers anda sigmoid nonlinearity for the output layer. Activations in the middle
hidden layer are binarized by a Heaviside step function with gradients
backpropagated by the straight-through estimator102. The loss function is

L=

X
μν2
batch

jj^iμν  iμν jj2 + λ!
1 X EC KLx  aEC ,NEC i μνi
μν2

X

ð5Þ

batch

where iμν is the image with pixel values between 0 and 1, ^iμν is its
reconstruction, xECμν represents the binary activations of the middle
hidden layer with NEC = 1024 units indexed by i, and aEC = 0.1 is its
desired density (Fig. 2C). Sparsiﬁcation with strength λ = 10 is achievedby computing the Kullback-Leibler (KL) divergence between thehidden layer density and aEC103. Training is performed over 150 epochswith batch size 64 using the Adam optimizer with learning rate 10−3 andweight decay 10−5.
Binary feedforward networks from EC to CA3. To propagate patterns
from EC to DG, from DG to MF inputs, and from EC to PP inputs, we
compute"x postμνi = ΘX#
W ij x preμνj  θ ,

ð6Þ

j
post
where xpre
μν and xμν are presynaptic and postsynaptic patterns, Wij isthe connectivity matrix, and θ is a threshold. Each postsynaptic neuron
receives l excitatory synapses of equal strength from randomly chosenpresynaptic neurons. θ is implicitly set through a winners-take-allprocess that enforces a desired postsynaptic pattern density apost. Θ isthe Heaviside step function, and N is the network size.
EC patterns have NEC = 1024 and aEC = 0.1. To determine N, a, and lfor each subsequent region, we turn to estimated biological values andloosely follow their trends. Rodents have approximately 5–10 times
more DG granule cells and 2–3 times more CA3 pyramidal cells compared to medial EC layer II principal neurons14,104,105. Thus, we choose
NDG = 8192 and NCA3 = 2048. During locomotion, DG place cells areapproximately 10 times less active than medial EC grid cells106, and MF
inputs are expected to be much sparser than PP inputs15. Thus, wechoose aDG = 0.005, aMF = 0.02, and aPP = 0.2. We do not directly
enforce correlation within concepts, which take values ρEC = 0.15,
ρDG = 0.02, ρMF = 0.01, and ρPP = 0.09. Each DG neuron receives
approximately 4000 synapses from EC and each CA3 neuron receives
approximately 50 MF and 4000 PP synapses14. Thus, we chooselDG = 205, lMF = 8, and lPP = 205. Note that for each feedforward projection, postsynaptic statistics apost and ρpost are not expected todepend on l (Eq. (1)).
EC
In Fig. 2E, for the case of xpreμν = xμν , we use Npost = 2048 and l = 205.
ρpost is obtained by computing correlations between examples within
the same concept and averaging over 3 concepts and 8 connectivity
matrices. For the case of randomly generated xpreμν , we use
N pre = N post = 10000, a single concept, and a single connectivity matrix
with l = 2000. See Supplementary Information for further details,
including the derivation of Eq. (1).

Visualization pathway from CA3 to EC. We train a fully connected
linear feedforward network with one hidden layer of size 4096 to map
EC
PPEC
inputs xMF
μν to targets xμν and inputs xμν also to targets xμν . Batch
normalization and a ReLU nonlinearity is applied to the hidden layer
and a sigmoid nonlinearity is applied to the output layer. The loss

Nature Communications | (2024)15:647

function is
L=

X
μν2
batch

EC

2
^ μν  xECjjx
μν jj :

ð7ÞTraining is performed over 100 epochs with batch size 128 using theAdam optimizer with learning rate 10−4 and weight decay 10−5.

Hopﬁeld-like model for CA3
Pattern storage. Our Hopﬁeld-like model for CA3 stores linear combinations qμν of MF and PP patterns:
PPqμνi = ð1  ζ Þ  ðx MFμνi  aMF Þ + ζ  ðx μνi  aPP Þ,

ð8Þ

where ζ = 0.1 is the relative strength of the PP patterns (Fig. 3A). The
subtraction of densities from each pattern is typical of Hopﬁeld networks with neural states 0 and 191. The synaptic connectivity matrix is
W ij =1 X
q q :N CA3 μν μνi μνjð9ÞPattern retrieval. Cues are formed from target patterns by randomly
ﬂipping the activity of a fraction 0.01 of all neurons (Fig. 3B). Thisquantity is termed cue inaccuracy; in Supplementary Fig. 3F, we also
consider cue incompleteness, which is the fraction of active neurons
that are randomly inactivated to form the cue. During retrieval, neurons are asynchronously updated in cycles during which every neuron
is updated once in random order (Fig. 3C). The total synaptic input toneuron i at time t isg i ðtÞ =X

W ij Sj ðtÞ + hi ðtÞ,

j

ð10Þ

where Sj(t) is the activity of presynaptic neuron j and hi(t) is an external
input. The external input is zero except for the cue-throughout
condition in Fig. 4, in which h(t) = σx for noisy MF cue x andstrength σ = 0.2.
The activity of neuron i at time t is probabilistically updated via
the Glauber dynamics
P½Si ðt + 1Þ = 1 =1,
1 + eβ½g i ðtÞθðtÞ

ð11Þ

where θ is the threshold and β is inverse temperature, with higher β
implying a harder threshold. Motivated by theoretical arguments, we
000deﬁne rescaled variables θ and β such that θ = θ  ð1  ζ Þ2 aMF and0241
β = β =ð1  ζ Þ aMF . Unless otherwise indicated, we run simulations for00
10 update cycles, use β = 100, and use θ = 0:5 to retrieve MF patterns
0and θ = 0 to retrieve PP patterns. The rescaled θ0 is the threshold value
illustrated in Fig. 4A and Supplementary Fig. 4A.
Retrieval evaluation. The overlap between the network activity S anda target pattern x ism=

X
1
S ðx  aÞ,
N CA3 að1  aÞ i i i

ð12Þ

where a is the density of the target pattern. This deﬁnition is also
motivated by theory41. The target pattern xPP
μ for PP concept μ is"
x PP
μi = Θ

Xν

#x PP
μνi  ϕ,ð13Þ

where ϕ is a threshold implicitly set by using winners-take-all to
enforce that xPP
μ has density aPP. The theoretical maximum overlap
14Articlehttps://doi.org/10.1038/s41467-024-44877-0

between the network and xPP
μ is the square root of the correlationpﬃﬃﬃﬃﬃﬃﬃ41
ρPP . Because this estimate is derived for random binary patterns in
the large network limit, it can be exceeded in our simulations.
To visualize S, we ﬁrst recall that the inverse of a dense stored
pattern x with every neuron ﬂipped can also be an equivalent stablestate33. Thus, we invert S if we are retrieving PP patterns at low θ and if
m < 0. Then, we decode its EC representation by passing S through thefeedforward visualization network and binarizing the output withthreshold 0.5. Finally, we pass the EC representation through the
decoding layers of the image autoencoder.
See Supplementary Information for the determination of network
capacity with random binary patterns (Fig. 3H, I and Supplementary
Fig. 3G, H) and the deﬁnition of oscillation behaviors (Fig. 4C andSupplementary Fig. 4B).

Experimental data analysis
General considerations. To calculate activity, we tabulate spikecounts c(r, ϕ) over spatial bins r (position or turn direction) and theta
phase bins ϕ, and we tabulate trajectory occupancy u(r, ϕ) over thesame r and distribute them evenly across ϕ. Activity as a function oftheta phase, the spatial variable, and both variables are respectivelyPPcðr,ϕÞ
ϕ cðr,ϕÞ, f ðrÞ = P, andf ðϕÞ = P r
uðr,ϕÞ
rϕ uðr,ϕÞf ðr,ϕÞ =cðr,ϕÞ
:uðr,ϕÞ

ð14Þ

Information per spike as a function of theta phase is calculated byIðϕÞ =X cðr,ϕÞ
r

cðϕÞ

log2

f ðr,ϕÞ
,
f ðϕÞ

ð15Þ

where c(ϕ) = ∑rc(r, ϕ)107. To perform sparsity correction for each neuron, we generate 100 null-matched neurons in which the spatial bin ofeach spike is replaced by a random value uniformly distributed acrossspatial bins. We subtract the mean I(ϕ) over the null matches from theI(ϕ) for the true data. To calculate the average difference ininformation between sparse and dense phases, we ﬁrst use f(ϕ) topartition ϕ into sparse and dense halves. We then average the sparsitycorrected I(ϕ) over each half, apply a ReLU function to each half toprevent negative information values, and compute the difference
between halves.
See Supplementary Information for dataset preprocessing details.
Model prediction. For the example prediction in Fig. 5A, we choose
one concept from Fig. 2A and ﬁnd 50 neurons that are active in at least
one MF example and one PP example within it. For each neuron, weconvert each active response to one spike and assign equal occupancies across all examples. We calculate the information per spike
across MF examples and across PP examples using example identity ν
as the spatial bin r. These values are sparsity-corrected with 50 nullmatched neurons, and their difference becomes our example prediction, associating MF encodings with sparse phases and PP with dense.
For the concept prediction in Fig. 6A, we ﬁnd 50 neurons that are
active in at least one MF example and one PP example within any
concept. For each neuron, we convert each active response to one
spike and collect MF and PP concept responses by summing spikes
within each concept. We assign equal occupancies across all concepts.
We calculate the information per spike across MF concepts and across
PP concepts using concept identity μ as the spatial bin r. We thenproceed as in the example case to produce our concept prediction.
Linear track data. Single neurons in Fig. 6 are preprocessed from the
CRCNS hc-3 dataset as described in Supplementary Information24. To
identify place cells, we compute the phase-independent position
information per spike using 1 cm-bins across all theta phases, and weselect neurons with values greater than 0.5. For each place cell, we bin

Nature Communications | (2024)15:647

spikes into various position bins as illustrated and phase bins of width30∘. Since our prediction compares sparse and dense information
conveyed by the same neurons, we require at least 8 spikes within eachphase value to allow for accurate estimation of position information
across all theta phases. To ensure theta modulation, we also require
the most active phase to contain at least twice the number of spikes as
the least active phase.
Place ﬁelds in Fig. 5 are extracted as described in Supplementary
Information. Processing occurs similarly to the whole-track caseabove, except we do not enforce a phase-independent information
constraint, we use 5 progress bins, and we require at least 5 spikes
within each phase value. Phase precession is detected by performing
circular–linear regression between spike progresses and phases108,109.
The precession score and precession slope are respectively deﬁned tobe the mean resultant length and regression slope. Precessing neurons
have score greater than 0.3 and negative slope steeper than –72°/ﬁeld.
W-maze data. Single neurons in Fig. 7A–H are preprocessed from the
CRCNS hc-6 dataset as described in Supplementary Information25. For
each neuron, we bin spikes into 2 turn directions and phase bins ofwidth 45∘. Since our prediction compares sparse and dense information conveyed by the same neurons, we require at least 5 spikes withineach phase value to allow for accurate estimation of position information across all theta phases. To ensure theta modulation, we also
require the most active phase to contain at least twice the number of
spikes as the least active phase.
Bayesian population decoding in Fig. 7I–L involves the same binning as in the single-neuron case above, and we enforce a minimum
spike count of 30 across all phases instead of a minimum for eachphase value. We do not ensure theta modulation on a single-neuron
basis. We consider all sessions in which at least 5 neurons are simultaneously recorded; there are 8 valid CA3 sessions and 25 valid
CA1 sessions. For each session, we compute the total activity acrossneurons and turn directions as a function of theta phase to determine
the sparsest and densest half of phases (similarly to Eq. (14)). We thencompute activities fi(r, ψ) over each half, indexed by ψ ∈ {sparse,
dense}, for neurons i and turn directions r. For each neuron, we rectify
all activity values below 0.02 times its maximum.
We decode turn direction during runs along the center arm using
sliding windows of width Δt = 0.5 s and stride 0.25 s. In each window attime t, we tabulate the population spike count c(t, ψ) over sparse anddense phases ψ. The likelihood that it arose from turn direction r ispðcðt,ψÞjrÞ =Yipðci ðt,ψÞjrÞ /Y!
f i ðr,ψÞ

ci ðt,ψÞ

exp ΔtiX!
f i ðr,ψÞ :i

ð16ÞThis formula assumes that spikes are independent across neurons andtime and obey Poisson statistics110. We only decode with at least
2 spikes. By Bayes’s formula and assuming a uniform prior, the likelihood is proportional to the posterior probability p(r∣c(t, ψ)) of turn
direction r decoded from spikes c(t, ψ). Consider one decoding thatyields p(R) as the probability of a right turn. Its conﬁdence is ∣2p(R)−1∣.
Its accuracy is 1 if p(R) > 0.5 and the true turn direction is right or if
p(R) < 0.5 and the true direction is left; otherwise, its accuracy is 0.

Machine learning with multilayer perceptrons
Dataset. We use the MNIST dataset of handwritten digits52. Each imageiα is normalized by subtracting the mean value and dividing by thestandard deviation across all images and pixels. In addition to its digitclass label, we randomly assign a set number. We train networks on asubset of images from the train dataset. To test concept learning
through digit classiﬁcation, we use all held-out images from the test
dataset. To test example learning through set identiﬁcation, we use all15

Articlehttps://doi.org/10.1038/s41467-024-44877-0

train images corrupted by randomly setting 20% of normalized pixel
values to 0.
Single-task learning. We train a fully-connected two-layer perceptron
with a hyperbolic tangent (tanh) activation function applied to eachhidden layer and a softmax activation function applied to the output
layer. Each hidden layer contains 50 neurons, and the output layer
contains 10 neurons for digit classiﬁcation and as many neurons as sets
for set identiﬁcation.
Let sα be the activations of the ﬁnal hidden layer for image α. The
loss is composed of a cross-entropy loss function between recon^ α and true labels yα, which are one-hot encodings ofstructed labels yeither digit class or set number, and the DeCorr loss function:

or HalfCorr loss function:
L= 

N1 
XX
α2 i = 0
batch



N1 
XX
α2 i = 0batch

yαi log ^yαi + ð1  yαi Þ logð1  ^yαi Þ
zαi log z^αi + ð1  zαi Þ logð1  z^αi Þ + λLDeCorr=HalfCorr ,
ð20Þ

where
hP
i2
N1


i = N=2 ðsαi  sα Þðsβi  s β Þ
1X
h
ih
i:
LHalfCorr =2 α6¼β2 PN1 ðs  s Þ2 + Nϵ=2 PN1 ðs  s Þ2 + Nϵ=2αβi = N=2 αii = N=2 βi
batch

L= 

N1 
XX
α2 i = 0batch

yαi log ^yαi + ð1  yαi Þ logð1  ^yαi Þ + λLDeCorr ,

ð17Þ

where

hP
i2
N1


i = 0 ðsαi  s α Þðsβi  s β Þ
1X
h
ih
i:
LDeCorr =2 α6¼β2 PN1 ðs  s Þ2 + Nϵ PN1 ðs  s Þ2 + Nϵαβi = 0 αii = 0 βi

ð18Þ

batch

PN1

ð21Þ

Mean activations are sα = ð2=NÞ i = N=2 sαi . The DeCorr/HalfCorr
strength is λ; we use λ = 1 with a tanh activation function, λ = 0.04 with a
ReLU, λ = 2 with no nonlinearity, and λ = 0 for the baseline case withany nonlinearity.
We train the network using stochastic gradient descent with batch
size 50 and learning rate 10−4. In general, we train until the network
reaches > 99.9% accuracy in both tasks with the train dataset. For
example, we use 100 epochs for the results in Fig. 8I, J.

Reporting summary
Further information on research design is available in the Nature
Portfolio Reporting Summary linked to this article.

We introduce ϵ = 0.001, which is scaled by the number of hidden layer
neurons N, to aid numerical convergence. Mean activations arePsα = ð1=NÞ N1
i = 0 sαi . The DeCorr strength is λ; except for Supplementary Fig. 8A, B, we use λ = 0 for the baseline case and λ = 1 for the
DeCorr case.
We train the network using stochastic gradient descent with batch
size 50, learning rate 10−4, and momentum 0.9. In general, we train until
the network reaches > 99.9% accuracy with the train dataset. For
example, we use 40, 100, and 200 epochs respectively for digit classiﬁcation and set identiﬁcation with 10 and 50 sets.
In contrast to DeCorr, the DeCov loss function formulated toreduce overﬁtting is

ð19Þ

References

batch

2.

Multitask learning. We train a fully-connected two-layer perceptron
with a hyperbolic tangent (tanh) activation function applied to eachhidden layer. In Supplementary Fig. 8E, F, we also consider applying a
ReLU activation function to each hidden layer, or a ReLU to the ﬁrst
hidden layer and no nonlinearity to the second. The ﬁnal hidden layer
is fully connected to two output layers, one for digit classiﬁcation andthe other for set identiﬁcation. A softmax activation function is applied
to each layer. Each hidden layer contains 100 neurons, the concept
output layer contains 10 neurons, and the example output layer contains as many neurons as sets.
The loss is composed of a cross-entropy loss function between^ α and true yα digit labels, a cross-entropy loss function
reconstructed y^α and true zα set labels, and either the DeCorr
between reconstructed z

3.

Nature Communications | (2024)15:647

taken

Code availability

1.

batch

now

All experimental data used in this study are taken from the Collaborative Research in Computational Neuroscience (CRCNS) hc-3
dataset contributed by György Buzsáki and colleagues24 and the hc-6
dataset contributed by Loren Frank and colleagues25. They are publicly
available at https://crcns.org/data-sets/hc. The MNIST dataset52 used inthis study is available at http://yann.lecun.com/exdb/mnist. The FashionMNIST dataset26 used in this study can be found at https://github.
com/zalandoresearch/fashion-mnist. Source data are provided withthis paper.

All network training and simulation code is available at https://github.
com/louiskang-group/kang-2024-examples-concepts.

2
32
N1 X
1 X
4 ðs  si Þðsαj  sj Þ5 ,LDeCov =2 i6¼j = 0 α2 αi

where
mean
activationsareP
items: si = ð1=N batch Þ α2batch sαi 53.

Data availability

over

4.

5.

6.

Scoville, W. B. & Milner, B. Loss of recent memory after bilateralhippocampal lesions. J. Neurol. Neurosurg. Psychiatry 20,
11 (1957).
McNaughton, B. & Morris, R. Hippocampal synaptic enhancement
and information storage within a distributed memory system.
Trends Neurosci. 10, 408–415 (1987).
O’Reilly, R. C. & Rudy, J. W. Conjunctive representations in learning and memory: Principles of cortical and hippocampal function.
Psychol. Rev. 108, 311–345 (2001).
Rolls, E. T. & Kesner, R. P. A computational theory of hippocampal
function, and empirical tests of the theory. Prog. Neurobiol. 79,
1–48 (2006).
Bi, G.-q & Poo, M.-m Synaptic modiﬁcations in cultured hippocampal neurons: Dependence on spike timing, synaptic strength,
and postsynaptic cell type. J. Neurosci. 18, 10464–10472 (1998).
Mishra, R. K., Kim, S., Guzman, S. J. & Jonas, P. Symmetric spiketiming-dependent plasticity at CA3-CA3 synapses optimizes storage and recall in autoassociative networks. Nat. Commun. 7,11552 (2016).

16

Article
7.

8.

9.
10.

11.

12.
13.

14.
15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

Amaral, D. & Pierre, L. Hippocampal neuroanatomy. In
The Hippocampus Book. (eds Andersen, P. et al.). 37–114(Oxford University Press, 2006).
Engin, E. et al. Tonic inhibitory control of dentate gyrus granulecells by α5-containing GABAA receptors reduces memory interference. J. Neurosci. 35, 13698–13712 (2015).
Marr, D. Simple memory: a theory for archicortex. Philos. Trans. R.
Soc. B 262, 23–81 (1971).
O’Reilly, R. C. & McClelland, J. L. Hippocampal conjunctive
encoding, storage, and recall: Avoiding a trade-off. Hippocampus
4, 661–682 (1994).
Vinje, W. E. & Gallant, J. L. Sparse coding and decorrelation inprimary visual cortex during natural vision. Science 287,
1273–1276 (2000).
Pitkow, X. & Meister, M. Decorrelation and efﬁcient coding by
retinal ganglion cells. Nat. Neurosci. 15, 628–635 (2012).
Cayco-Gajic, N. A., Clopath, C. & Silver, R. A. Sparse synaptic
connectivity is required for decorrelation and pattern separation in
feedforward networks. Nat. Commun. 8, 1116 (2017).
Amaral, D. G., Ishizuka, N. & Claiborne, B. Neurons, numbers andthe hippocampal network. Prog. Brain Res. 83, 1–11 (1990).
Treves, A. & Rolls, E. T. Computational constraints suggest theneed for two distinct input systems to the hippocampal CA3
network. Hippocampus 2, 189–199 (1992).
McClelland, J. L. & Goddard, N. H. Considerations arising from acomplementary learning systems perspective on hippocampus
and neocortex. Hippocampus 6, 654–665 (1996).
Kaifosh, P. & Losonczy, A. Mnemonic functions for nonlinear
dendritic integration in hippocampal pyramidal circuits. Neuron
90, 622–634 (2016).
Leutgeb, J. K., Leutgeb, S., Moser, M.-B. & Moser, E. I. Patternseparation in the dentate gyrus and CA3 of the hippocampus.
Science 315, 961–966 (2007).
Aimone, J. B., Deng, W. & Gage, F. H. Resolving new memories: A
critical look at the dentate gyrus, adult neurogenesis, and patternseparation. Neuron 70, 589–596 (2011).
Borzello, M. et al. Assessments of dentate gyrus function:
discoveries and debates. Nat. Rev. Neurosci. 24, 502–517(2023).
Squire, L. R. Memory and the hippocampus: A synthesis fromﬁndings with rats, monkeys, and humans. Psychol. Rev. 99,
195–231 (1992).
Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C. & Fried, I. Invariant
visual representation by single neurons in the human brain. Nature
435, 1102–1107 (2005).
Quian Quiroga, R., Kraskov, A., Koch, C. & Fried, I. Explicit
encoding of multimodal percepts by single neurons in the humanbrain. Curr. Biol. 19, 1308–1313 (2009).
Mizuseki, K., Sirota, A., Pastalkova, E., Diba, K., & Buzsáki, G.
Multiple single unit recordings from different rat hippocampal
and entorhinal regions while the animals were performing
multiple behavioral tasks. CRCNS.org. https://doi.org/10.6080/
K09G5JRZ (2013).
Karlsson, M., Carr, M., & Frank, L. M. Simultaneous extracellular
recordings from hippocampal areas CA1 and CA3 (or MEC andCA1) from rats performing an alternation task in two W-shapped
tracks that are geometrically identically but visually distinct.
CRCNS.org. https://doi.org/10.6080/K0NK3BZJ (2015).
Xiao, H., Rasul, K., & Vollgraf, R. Fashion-MNIST: a novel
image dataset for benchmarking machine learning algorithms. arXiv 1708.07747. https://doi.org/10.48550/arXiv.
1708.07747 (2017).
Olshausen, B. A. & Field, D. J. Emergence of simple-cell receptiveﬁeld properties by learning a sparse code for natural images.
Nature 381, 607–609 (1996).

Nature Communications | (2024)15:647

https://doi.org/10.1038/s41467-024-44877-0
28.
29.
30.

31.

32.

33.

34.

35.
36.
37.
38.

39.
40.
41.

42.

43.

44.

45.
46.
47.

48.

49.

50.

51.

Lee, H., Battle, A., Raina, R., & Ng, A. Y. Efﬁcient sparse coding
algorithms. Adv. Neural Inf. Process. Syst. 19, 801–808 (2006).
Makhzani, A. & Frey, B. k-sparse autoencoders. arXiv 1312.5663.
https://doi.org/10.48550/arXiv.1312.5663 (2014).
Chen, Y., Paiton, D., & Olshausen, B. The Sparse Manifold Transform. Advances in Neural Information Processing Systems, pages
10513–10524. Curran Associates, Inc., (2018).
Henze, D. A., Wittner, L. & Buzsáki, G. Single granule cells reliably
discharge targets in the hippocampal CA3 network in vivo. Nat.
Neurosci. 5, 790–795 (2002).
Vyleta, N. P., Borges-Merjane, C. & Jonas, P. Plasticity-dependent,
full detonation at hippocampal mossy ﬁber-CA3 pyramidal neuron synapses. eLife 5, 3386 (2016).
Hopﬁeld, J. J. Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci.
U.S.A. 79, 2554–2558 (1982).
Kim, S., Guzman, S. J., Hu, H. & Jonas, P. Active dendrites support
efﬁcient initiation of dendritic spikes in hippocampal CA3 pyramidal neurons. Nat. Neurosci. 15, 600–606 (2012).
Makara, J. & Magee, J. Variable dendritic integration in hippocampal CA3 pyramidal neurons. Neuron 80, 1438–1450 (2013).
Amit, D. J., Gutfreund, H. & Sompolinsky, H. Spin-glass models ofneural networks. Phys. Rev. A 32, 1007–1018 (1985).
Anderson, J. R. The adaptive nature of human categorization.
Psychol. Rev. 98, 409–429 (1991).
Love, B. C., Medin, D. L. & Gureckis, T. M. SUSTAIN: A networkmodel of category learning. Psychol. Rev. 111, 309–332(2004).
Ashby, F. G. & Maddox, W. T. Human category learning. Annu. Rev.
Psychol. 56, 149–178 (2005).
Krizhevsky, A. & Hinton, G. Learning Multiple Layers of Featuresfrom Tiny Images. Technical Report 0 (University of Toronto, 2009).
Kang, L. & Toyoizumi, T. Hopﬁeld-like network with complementary encodings of memories. Phys. Rev. E 108,054410 (2023).
Mizuseki, K. et al. Neurosharing: large-scale data sets (spike, LFP)
recorded from the hippocampal-entorhinal system in behaving
rats. F1000Research 3, 98 (2014).
Skaggs, W. E., McNaughton, B. L., Wilson, M. A. & Barnes, C. A.
Theta phase precession in hippocampal neuronal populations andthe compression of temporal sequences. Hippocampus 6,
149–172 (1996).
McAdams, C. J. & Maunsell, J. H. R. Effects of attention onorientation-tuning functions of single neurons in Macaque corticalarea V4. J. Neurosci. 19, 431–441 (1999).
Isaacson, J. & Scanziani, M. How inhibition shapes cortical activity.
Neuron 72, 231–243 (2011).
Carandini, M. & Heeger, D. J. Normalization as a canonical neuralcomputation. Nat. Rev. Neurosci. 13, 51–62 (2012).
Ferguson, K. A. & Cardin, J. A. Mechanisms underlying gain
modulation in the cortex. Nat. Rev. Neurosci. 21, 80–92(2020).
Dotson, N. M. & Yartsev, M. M. Nonlocal spatiotemporal representation in the hippocampus of freely ﬂying bats. Science 373,
242–247 (2021).
Frank, L. M., Brown, E. N. & Wilson, M. Trajectory encoding in
the hippocampus and entorhinal cortex. Neuron 27, 169–178(2000).
Wood, E. R., Dudchenko, P. A., Robitsek, R. & Eichenbaum, H.
Hippocampal neurons encode information about different typesof memory episodes occurring in the same location. Neuron 27,
623–633 (2000).
Duvelle, É., Grieves, R. M. & van der Meer, M. A. Temporal context
and latent state inference in the hippocampal splitter signal. eLife
12, e82357 (2023).
17

Article
52.

53.

54.
55.

56.
57.

58.

59.

60.

61.

62.

63.

64.
65.

66.

67.

68.

69.
70.

71.

72.

73.

LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P. Gradient-basedlearning applied to document recognition. Proc. IEEE 86,
2278–2324 (1998).
Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., and Batra, D.
Reducing overﬁtting in deep networks by decorrelating representations. arXiv 1511.06068. https://doi.org/10.48550/arXiv.1511.
06068 (2015).
Olshausen, B. A. & Field, D. J. Sparse coding of sensory inputs.
Curr. Opin. Neurobiol. 14, 481–487 (2004).
Sze, V., Chen, Y.-H., Yang, T.-J. & Emer, J. S. Efﬁcient processing ofdeep neural networks: A tutorial and survey. Proc. IEEE 105,
2295–2329 (2017).
Teyler, T. J. & DiScenna, P. The hippocampal memory indexing
theory. Behav. Neurosci. 100, 147–154 (1986).
Teyler, T. J. & Rudy, J. W. The hippocampal indexing theory andepisodic memory: Updating the index. Hippocampus 17,
1158–1169 (2007).
Schapiro, A. C., Gregory, E., Landau, B., McCloskey, M. & TurkBrowne, N. B. The necessity of the medial temporal lobe for statistical learning. J. Cognit. Neurosci. 26, 1736–1747 (2014).
Covington, N. V., Brown-Schmidt, S. & Duff, M. C. The necessity of
the hippocampus for statistical learning. J. Cognit. Neurosci. 30,
680–697 (2018).
Knowlton, B. J. & Squire, L. R. The learning of categories: Parallel
brain systems for item memory and category knowledge. Science
262, 1747–1749 (1993).
Zeithamova, D., Maddox, W. T. & Schnyer, D. M. Dissociable prototype learning systems: Evidence from brain imaging and behavior. J. Neurosci. 28, 13194–13201 (2008).
Mack, M. L., Love, B. C. & Preston, A. R. Dynamic updating of
hippocampal object representations reﬂects new conceptual
knowledge. Proc. Natl. Acad. Sci. U.S.A. 113, 13203–13208 (2016).
Bowman, C. R. & Zeithamova, D. Abstract memory representationsin the ventromedial prefrontal cortex and hippocampus support
concept generalization. J. Neurosci. 38, 2605–2614 (2018).
Manns, J. R., Hopkins, R. O. & Squire, L. R. Semantic memory andthe human hippocampus. Neuron 38, 127–133 (2003).
Duff, M. C., Covington, N. V., Hilverman, C. & Cohen, N. J.
Semantic memory and the hippocampus: Revisiting, reafﬁrming,and extending the reach of their critical relationship. Front. Hum.
Neurosci. 13, 471 (2020).
Norman, Y., Raccah, O., Liu, S., Parvizi, J. & Malach, R. Hippocampal ripples and their coordinated dialogue with the default
mode network during recent and remote recollection. Neuron
109, 2767–2780 (2021).
O’Keefe, J. & Recce, M. L. Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3,
317–330 (1993).
Terada, S., Sakurai, Y., Nakahara, H. & Fujisawa, S. Temporal andrate coding for discrete event sequences in the hippocampus.
Neuron 94, 1248–1262 (2017).
Qasim, S. E., Fried, I. & Jacobs, J. Phase precession in the humanhippocampus and entorhinal cortex. Cell 184, 3242–3255 (2021).
Reddy, L. et al. Theta-phase dependent neuronal coding during
sequence learning in human single neurons. Nat. Commun. 12,4839 (2021).
Kay, K. et al. Constant sub-second cycling between representations of possible futures in the hippocampus. Cell 180,
552–567 (2020).
Willmore, B. & Tolhurst, D. J. Characterizing the sparsenessof neural codes. Netw. Comput. Neural Syst. 12, 255–270(2001).
Ujfalussy, B. B. & Orbán, G. Sampling motion trajectories during
hippocampal theta sequences. eLife 11, e74058 (2022).

Nature Communications | (2024)15:647

https://doi.org/10.1038/s41467-024-44877-0
74.

75.

76.

77.

78.

79.

80.

81.

82.

83.

84.

85.
86.
87.

88.

89.

90.

91.

92.
93.

94.

Mehta, M. R., Lee, A. K. & Wilson, M. A. Role of experience andoscillations in transforming a rate code into a temporal code.
Nature 417, 741–746 (2002).
Souza, B. C. & Tort, A. B. L. Asymmetry of the temporal codefor space by hippocampal place cells. Sci. Rep. 7, 8507
(2017).
Pfeiffer, B. E. & Foster, D. J. Autoassociative dynamics in the generation of sequences of hippocampal place cells. Science 349,
180–183 (2015).
Berry, S. D. & Thompson, R. F. Prediction of learning rate from the
hippocampal electroencephalogram. Science 200,
1298–1300 (1978).
Seager, M. A., Johnson, L. D., Chabot, E. S., Asaka, Y. & Berry, S. D.
Oscillatory brain states and learning: Impact of hippocampal
theta-contingent training. Proc. Natl. Acad. Sci. U.S.A. 99,
1616–1620 (2002).
Lega, B. C., Jacobs, J. & Kahana, M. Human hippocampal theta
oscillations and the formation of episodic memories. Hippocampus 22, 748–761 (2012).
Penley, S. C. et al. Novel space alters theta and gamma synchrony
across the longitudinal axis of the hippocampus. Front. Syst.
Neurosci. 7, 20 (2013).
Backus, A. R., Schoffelen, J.-M., Szebényi, S., Hanslmayr, S. &Doeller, C. F. Hippocampal-prefrontal theta oscillations support
memory integration. Curr. Biol. 26, 450–457 (2016).
Herweg, N. A., Solomon, E. A. & Kahana, M. J. Theta oscillations in human memory. Trends in Cognitive Sciences 24,
208–227 (2020).
Jacobs, J., Hwang, G., Curran, T. & Kahana, M. J. EEG oscillations
and recognition memory: Theta correlates of memory retrieval
and decision making. Neuroimage 32, 978–987 (2006).
Zheng, J. et al. Multiplexing of theta and alpha rhythms in the
amygdala-hippocampal circuit supports pattern separation ofemotional information. Neuron 102, 887–898 (2019).
Colgin, L. L. Mechanisms and functions of theta rhythms. Annu.
Rev. Neurosci. 36, 295–312 (2013).
Hasselmo, M. E. The role of acetylcholine in learning and memory.
Curr. Opin. Neurobiol. 16, 710–715 (2006).
Hasselmo, M. E., Bodeln, C. & Wyble, B. P. A proposed function for
hippocampal theta rhythm: Separate phases of encoding andretrieval enhance reversal of prior learning. Neural Comput. 14,
793–817 (2002).
Kunec, S., Hasselmo, M. E. & Kopell, N. Encoding and retrieval in
the CA3 region of the hippocampus: A model of theta-phaseseparation. J. Neurophysiol. 94, 70–82 (2005).
Siegle, J. H. & Wilson, M. A. Enhancement of encoding andretrieval functions through theta phase-speciﬁc manipulation ofhippocampus. eLife 3, e03061 (2014).
Leung, L. S. & Law, C. S. H. Phasic modulation of hippocampal
synaptic plasticity by theta rhythm. Behav. Neurosci. 134,
595–612 (2020).
Tsodyks, M. V. & Feigel’man, M. V. The enhanced storage capacity
in neural networks with low activity level. Europhys. Lett. 6,
101–105 (1988).
Fontanari, J. F. Generalization in a Hopﬁeld network. J. Phys. 51,
2421–2430 (1990).
Schapiro, A. C., Turk-Browne, N. B., Botvinick, M. M. & Norman, K.
A. Complementary learning systems within the hippocampus: aneural network modelling approach to reconciling episodic
memory with statistical learning. Philos. Trans. R. Soc. B 372,
20160049 (2017).
Sučević, J. & Schapiro, A. C. A neural network model ofhippocampal contributions to category learning. eLife 12,
e77185 (2023).

18

Article
95.

96.

97.
98.

99.

100.

101.

102.

103.

104.

105.

106.

107.

108.

109.

110.

Antony, J., Liu, X. L., Zheng, Y., Ranganath, C., & O’Reilly, R. C.
Memory out of context: Spacing effects and decontextualization
in a computational model of the medial temporal lobe. bioRxiv
2022.12.01.518703. https://doi.org/10.1101/2022.12.01.518703(2023).
Whittington, J. C. et al. The Tolman-Eichenbaum Machine: Unifying space and relational memory through generalization in the
hippocampal formation. Cell 183, 1249–1263 (2020).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521,
436–444 (2015).
Amer, M. & Maul, T. A review of modularization techniquesin artiﬁcial neural networks. Artif. Intell. Rev. 52, 527–561(2019).
Kowadlo, G., Ahmed, A., & Rawlinson, D. AHA! an ’Artiﬁcial Hippocampal Algorithm’ for episodic machine learning. arXiv
909.10340. https://doi.org/10.48550/arXiv.1909.10340 (2020).
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C. & Wermter, S. Continual lifelong learning with neural networks: A review. NeuralNetw. 113, 54–71 (2019).
Arulkumaran, K., Deisenroth, M. P., Brundage, M. & Bharath, A. A.
Deep reinforcement learning. IEEE Signal Process. Mag. 34,
26–38 (2017).
Bengio, Y., Léonard, N., & Courville, A. Estimating or propagating gradients through stochastic neurons for conditional
computation. arXiv 1308.3432. https://doi.org/10.48550/
arXiv.1308.3432 (2013).
Le, Q. V., Karpenko, A., Ngiam, J., & Ng, A. Y. ICA with reconstruction cost for efﬁcient overcomplete feature learning. Adv.
Neural Inf. Process. Syst. 24, 1017–1025 (2011).
Murakami, T. C. et al. A three-dimensional single-cell-resolution
whole-brain atlas using CUBIC-X expansion microscopy and tissue
clearing. Nat. Neurosci. 21, 625–637 (2018).
Attili, S. M., Silva, M. F. M., Nguyen, T.-v & Ascoli, G. A.
Cell numbers, distribution, shape, and regional variation throughout the murine hippocampal formation from the adult brain Allen
Reference Atlas. Brain Struct. Funct. 224, 2883–2897 (2019).
Mizuseki, K. & Buzsáki, G. Preconﬁgured, skewed distribution ofﬁring rates in the hippocampus and entorhinal cortex. Cell Rep. 4,1010–1021 (2013).
Skaggs, W. E., McNaughton, B. L., Gothard, K. M., & Markus, E. J. An
information-theoretic approach to deciphering the hippocampal
code. Adv. Neural Inf. Process. Syst. 5, 1030–1037 (1993).
Kempter, R., Leibold, C., Buzsáki, G., Diba, K. & Schmidt, R.
Quantifying circular-linear associations: Hippocampal phase precession. J. Neurosci. Methods 207, 113–124 (2012).
Kang, L. & DeWeese, M. R. Replay as wavefronts and theta
sequences as bump oscillations in a grid cell attractor network.
eLife 8, e46351 (2019).
Zhang, K., Ginzburg, I., McNaughton, B. L. & Sejnowski, T. J.
Interpreting neuronal population activity by reconstruction: Uniﬁed framework with application to hippocampal place cells. J.
Neurophysiol. 79, 1017–1044 (1998).

Nature Communications | (2024)15:647

https://doi.org/10.1038/s41467-024-44877-0

Acknowledgements
We thank Tom McHugh and Łukasz Kuśmierz for helpful ideas. LK is
supported by JSPS KAKENHI for Early-Career Scientists (22K15209) andhas been supported by the Miller Institute for Basic Research in Scienceand a Burroughs Wellcome Fund Collaborative Research Travel Grant.
TT is supported by Brain/MINDS from AMED (JP19dm0207001) and JSPS
KAKENHI (JP18H05432).

Author contributions
L.K. and T.T. conceptualized the study, analyzed the results, and wrote
the manuscript. L.K. trained and simulated the neural networks, performed the mathematical derivations, and analyzed the experimental data.

Competing interests
The authors declare no competing interests.

Additional information
Supplementary information The online version contains
supplementary material available athttps://doi.org/10.1038/s41467-024-44877-0.
Correspondence and requests for materials should be addressed toLouis Kang.
Peer review information Nature Communications thanks Zilong Ji,
Joseph Monaco and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review ﬁle is available.
Reprints and permissions information is available at
http://www.nature.com/reprints
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional afﬁliations.
Open Access This article is licensed under a Creative Commons
Attribution 4.0 International License, which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, aslong as you give appropriate credit to the original author(s) and thesource, provide a link to the Creative Commons licence, and indicate ifchanges were made. The images or other third party material in thisarticle are included in the article’s Creative Commons licence, unlessindicated otherwise in a credit line to the material. If material is notincluded in the article’s Creative Commons licence and your intended
use is not permitted by statutory regulation or exceeds the permitteduse, you will need to obtain permission directly from the copyrightholder. To view a copy of this licence, visit http://creativecommons.org/
licenses/by/4.0/.
© The Author(s) 2024

19

Supplementary Figures and Supplementary Methods for“Distinguishing examples while building concepts
in hippocampal and artificial networks”
Louis Kang∗1 and Taro Toyoizumi2
1

2

Neural Circuits and Computations Unit, RIKEN Center for Brain Science
Laboratory for Neural Computation and Adaptation, RIKEN Center for Brain Science

Contents
Supplementary Figures
Supplementary Figure 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Supplementary Figure 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2
2
2
3
4
5
7
9
11

Supplementary Methods
12
Decorrelation in binary feedforward networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
CA3 model with random binary patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
CA3 model behavior during oscillating threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Experimental data preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Experimental data aggregate analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
References

21

N.B.: All equation numbers below refer to this document and not the main text unless explicitly stated.

∗ louis.kang@riken.jp

1Supplementary Figuressimulation
theorySupplementary Figure 1: Extended results on decorrelation in binary feedforward networks. Joint probability distributionof total inputs Gνi and Gωi on postsynaptic neuron i for two patterns ν ̸= ω (Eq. 12). The theoretically derived probability
density function f (gνi , gωi ) (Eq. 25) agrees with simulation results. Each g is a sample of the corresponding random variable
G. Simulation parameters are 1000 presynaptic neurons, 1000 postsynaptic neurons, synaptic connection probability 0.1, 100
random activity patterns, presynaptic pattern density 0.1, and presynaptic correlation 0.3.

10

0

5

10

image PC 1

ρpre = 0.1

15Cρpre = 0.01

0.1

0.1

0.01

0.01

0.3
0.1

0.03
0.01

0.003

0.001
0.001

0.001
0.001
0.01
0.01
0.1
postsynaptic pattern density apost

0.001

0.1

presynaptic
patterndensity
presynaptic
densityapre apre

presynaptic pattern density apre

sneakers
trousers
coats

0
-5

B

exampleconcept (average image)concept ()

5

postsynaptic correlation ρpost

image PC 2Aapost = 0.1
0.1
0.01
0.001
0.001
0.01
0.1
presynaptic correlation ρpreSupplementary Figure 2: Extended results for Fig. 2 of the main text. (A) Example images and two definitions of concepts
projected along the first two principal components (PCs) of the example dataset. Each concept contains 512 examples. Averaged
images are formed by averaging pixel intensities. PP concept target patterns xPPµ are formed according to Eq. 13 of the main
text and then passed through the visualization pathways in Fig. 2F of the main text. (B, C) Postsynaptic correlation ρpost inbinary feedforward networks in Fig. 2D, E. (B) ρpost as a function of presynaptic pattern density apre and postsynaptic patterndensity apost for two values of presynaptic correlation ρpre . On the left, ρpre = 0.1 corresponds to the red line, and on the right,
ρpre = 0.01 corresponds to the yellow line. Decorrelation occurs in the regions below and to the left of these lines. (C) ρpost as afunction of presynaptic pattern density apre and presynaptic correlation ρpre for postsynaptic pattern density apost = 0.1. Source
data are provided as a Source Data file.

2

MF retrieval (high thresh. θ)APP retrieval (low thresh. θ)

MF cue

1

2

5

10

20

50 100

1

2

5

10

20

50 100

1

2

5

10

20

50 100

1

2

5

10

20

50 100

2

5

10

20

50 100

1

2

5

10

20

50 100

1

2

5

10

20

50 100

1

2

5

10

20

50 100examples stored per concept

B
PP cue1examples stored per concept

combined cue

D

MF retrieval (high thresh. θ)

PP retrieval (low thresh. θ)Epattern overlap

C

1

2

5

10

20

50 100

1

2

5

10

20

target pattern

1.0

MF
examples

0.5
050 100PP
examplesPP
concepts

1
3
10
30
100
examples stored per concept

1.0
0.5

0.04
0.12
0.20
PP strength ζ

0.01 0.02 0.03 0.04
MF density aMF

MF cue, MF example target

examples stored
per concept

G

10

3

10

2

0.1 0.2 0.3 0.4
PP density aPP
PP cue, MF example target

H

101
100

10-3

10-2

10-3

10-2

concepts per neuron

0

10-3

10-2

10

3

10

2

0.1 0.2 0.3
cue inaccuracy0
0.2 0.4 0.6
cue incompletenessMF cue, PP concept target-0.05 00.1 0.5
dendritic nonlinearity

PP cue, PP concept target

MF patterndensity aMF

101
100

10-3

10-2
10-3
10-2
concepts per neuron

10-3

10-2

MF examples
PP examples
PP concepts

0

examples stored
per conceptFpattern overlap

examples stored per concept

0.002
0.01
0.05

targetpattern

Supplementary Figure 3: Extended results for Fig. 3 of the main text. (A) Similar to Fig. 3D, but for trouser and coat concepts.
(B) Similar to Fig. 3F, but for trouser and coat concepts. (C–E) Results for cues that combine active neurons from both MF and
PP encodings. (C) Combined cues formed by the neuron-wise or operation. (D) Similar to Fig. 3D, F, but for combined cues. (E)Similar to Fig. 3E, G, but for combined cues.
(Continued on the next page.)3

Supplementary Figure 3: (Continued from the previous page.)(F) Overlaps of retrieved patterns over a wide range of network parameters. MF examples and PP concepts are retrieved athigh and low threshold, respectively, optimized by grid search. Cue inaccuracy is the fraction of randomly chosen neurons in thetarget pattern whose activity is flipped to form the cue. Cue incompleteness is the fraction of randomly chosen active neurons
in the target pattern which are inactivated to form the cue. Dendritic nonlinearity η introduces nonlinear summation between MF
PP
MF PP
patterns xMFµν and PP patterns xµν by adding a term ηxµνi xµνi to Eq. 8 of the main text. Negative and positive η correspond to
sublinear and superlinear regimes, respectively. For PP pattern strength γ = 0.04, 0.08, 0.12, 0.16, and 0.20, we respectively use
example loads s = 400, 120, 80, 50, and 20; otherwise, s = 100. Points represent means over 4 networks with 15 cues tested ineach. (G, H) Similar to Fig. 3H, I, but for different MF pattern densities aMF . PP patterns have correlation ρPP = 0.04. Source
data are provided as a Source Data file.

MF density aMF = 0.01

0.8

example thresh.
overlapθA

concept
overlap0
1.0

examples withintarget concept

0

0.5
0

other concepts
target concept

0

40
80
time [update cycle]

aMF = 0.002

0
1.0

0

concept
overlap

0.5
40
80
time [update cycle]

proportion

B 1.0

aMF = 0.002

0.5
0

1.0

examples concepts

120

aMF = 0.01

1.0
proportion

0

proportion

concept
overlap

aMF = 0.05

0.8

0
1.0

0

120

example thresh.
overlapθ

0.8

example thresh.
overlapθ

examples within
other concepts (max.)

0.5
0

0.5
0

0

40
80
time [update cycle]

120

aMF = 0.05
examples within other concepts
mixed examples within target concept
single examples within target concept

0.50examples concepts

0

examples concepts

other concepts
target concept

Supplementary Figure 4: Extended results for Fig. 4 of the main text. We use random MF and PP patterns instead of
FashionMNIST encodings. (A) Similar to Fig. 4A. (B) Similar to Fig. 4C. For each scenario in B, 10 cues are tested in each of 10
networks. In all networks, 3 concepts are used, MF patterns have correlation 0, and PP patterns have density 0.5 and correlation
0.16. For MF densities aMF = 0.002, 0.01, and 0.05, we store 5, 10, and 20 patterns per concept, respectively. Source data areprovided as a Source Data file.

4

720°

720°

360°

360°

360°

0°

0

0.5

0°

1.0

progress through eld
63

0
360°
720°
theta phase

540°

0

0°

0
360°
720°
theta phase

0.2

5

0.5

720°

G

540°

0
180° 360° 540° 720°
theta phase

0°

0.4

10

360°

0

0.5

progress through eld0180° 360° 540° 720°
theta phase

0°

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1
0

theta phase

5

10
15
activity [spk/s]

0

0.20

0.22

0.24

eld width

0.26

0

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

position info. [bit/spk]

J

0

0.4

position info. [bit/spk]

I

1.0

0

0
8
12
activity [spk/s]

0

-0.2

CA3

CA1

all place elds

H

0.20 0.22 0.24 0.26
eld width

0

10-8

0.2

10-17
10-11

0.1
0

-0.1
-0.2

0.2

5

180°

position info. [bit/spk]null matched [bit/spk]

F

activity [spk/s]

progress through eld0

1.0

theta phase

0

0.95

0.4

10

180°

0.002

0.2

0.6

15

360°

0°

32

E

720°

0°

1.0

activity [spk/s]

theta phase

0°

0.5

position info. [bit/spk]null matched [bit/spk]

D

0

0.5

0.021

0.4

0

progress through eld

progress through eld

0
360°
720°
theta phase

0°

0°

1.0

0.4 50activity [spk/s]

0

0.5

position info. [bit/spk]null matched [bit/spk]

B

0C
sparse - dense
position info. [bit/spk]

720°

sparse - dense
position info. [bit/spk]

theta phase

A

0.3

0.4

0.3

0.4

0.5

eld center

0.5

eld center

CA3
CA1
bootstrapped aggregate
precessing place elds

0.6

0.6

Supplementary Figure 5: Extended results for Fig. 5 of the main text. (A–C) Additional single-neuron results. (A) Similar to
Fig. 5F, but for three additional precessing fields from CA3. (B) Similar to Fig. 5G, but for the fields in A.
(Continued on the next page.)5Supplementary Figure 5: (Continued from the previous page.)(C) Similar to Fig. 5L, but comparing place fields in CA3 with those in CA1. CA3 n = 35 and CA1 n = 47. Numbers indicate
p-values calculated by two-tailed Wilcoxon signed-rank tests for each population and by the two-tailed Mann-Whitney U test
for the comparison between them. (D–J) Results for aggregate fields formed by accumulating spikes across phase precessing
place fields. (D) Spikes aggregated across 57 CA3 place fields. (E) Activity (black), raw position information per spike (blue), andmean null-matched position information (gray) by theta phase for the aggregate field in D. (F, G) Similar to D, E, but for 55 CA1
place fields. (H) Similar to C, but comparing 100 bootstrap subsamples per region of 1000 spikes from the aggregate fields in D
and F. (I) Parametric plots of features of the aggregate CA3 field in D with respect to theta phase. Field width and field center
are respectively the standard deviation and mean of progress values over spikes. Each point represents one theta phase and its
size is proportional to total activity. Arrows start at the phase with lowest activity and point towards increasing phase. (J) Similar
to I, but for the aggregate CA1 field in F. For all results, spikes during each traveling direction are separately analyzed. In C, H, I,
and J, information is sparsity-corrected with horizontal lines indicating medians. Source data are provided as a Source Data file.

6

scale-dependence p = 10–4

B

720°

sparse - dense
position info. [bit/spk]

720°

0.5

360°

0°

0°

0

theta phaseC

50

100 150 200
position [cm]

250

720°
360°
0°

0

50

1234123412341234
position bin

100 150 200
position [cm]360°
1 2 3 4 1 2 3 4
position bin

track scale: 1/16

D

E

10-3

0.29

0.3

-0.5

720°
360°
1

16

position bin

720°
360°
0°

track scale: 1/16

G

1

position bin

sparse - dense
position info. [bit/spk]

0.3

10

0.022

-0.3

1/16

1/8
track scalebroader
concepts

1/16

1/8
track scale

1/4

720°
360°
0°

1

position bin

4broader
concepts

scale-dependence p = 0.001

10

-4

0.005

0.27

0

-0.3

1/4

4

0.54

0.3

0.004

2
3
position bin

track scale: 1/4

H

0

-0.6

1

track scale: 1/8

scale-dependence p = 0.21
-3

0°

0.26

8

sparse - dense
position info. [bit/spk]

0°

360°

0

-0.3

1/4

1/4

0.66

theta phase

1/8
track scale

theta phasetheta phase

F

1/16

1/8
track scale

scale-dependence p = 0.48

0.6

0

1/16

10-6

0

track scale: 1/4

sparse - dense
position info. [bit/spk]

sparse - dense
position info. [bit/spk]

0.027

10-8

720°

track scale: 1/8

scale-dependence p = 0.032

0.5

250

720°

0°

0.47

-0.5

theta phase360°

theta phasetheta phase

A

1/16

1/8
track scale

1/4

Supplementary Figure 6: Extended results for Fig. 6 of the main text. (A) Two additional CA3 place cells along a linear track.
For each neuron, spikes are represented by two points at equivalent phases and are accumulated over position (top) and phase(right). (B) Similar to Fig. 6E, but for CA1 place cells. Track scale 1/16 n = 122, 1/8 n = 137, and 1/4 n = 144. (C–E) Similar
to Fig. 6C, E, F, but for an alternative method for binning positions across track scales. (C) Four bins are still used for all scales,
but bins cycle across the whole track. (D) For coarser scales, dense phases convey more position information per spike, as inFig. 6E. Each track scale n = 56. (E) Shuffled data exhibit no relationship between position information and theta phase acrosstrack scales, as in Fig. 6F. (F–H) Similar to Fig. 6C, E, F, but for a third method for binning positions across track scales. (F)Different numbers of bins are used across scales.
(Continued on the next page.)7

Supplementary Figure 6: (Continued from the previous page.)(G) At all scales, dense phases convey more position information per spike, which differs from Fig. 6E. Each track scale n = 27.
(H) Shuffled data exhibit a relationship between position information and theta phase for finer scales, which differs from Fig. 6F
and invalidates this binning method. For all results, spikes during each traveling direction are separately analyzed. In B, D, E, G,
and H, information is sparsity-corrected with horizontal lines indicating medians. Source data are provided as a Source Data file.

8

720°

720°

720°

360°

360°

360°

0°

0°

0°

0
0.5
1.0
progress along center arm

E

0.5

0

0

overall turn info. [bit/spk]

0.01
0180° 360° 540° 720°
theta phase

CA1
sparse
dense

0.1
0

0

0.5

0°

1.0

decoding con dence

1.0

0.006

0.50

0.5
0

-0.5
-1.0

CA3

all runs

CA1

0
360°
720°
theta phase

0.5
0
-0.2

I

0.01

0
180° 360° 540° 720°
theta phase

0°

1.0

CA1
all place cells

0.008

K

p = 10–19

0.2

0sparse - dense
decoding accuracy

proportion

J

CA3

5

0.02

0°

F

0.080

activity [spk/s]

activity [spk/s]
0

0.011

0

turn info. [bit/spk]null matched [bit/spk]

5

0
360°
720°
theta phase

H

0.03

turn info. [bit/spk]null matched [bit/spk]

G

0°

0

-0.5

0.5

0

10-4

0.5

sparse - dense
turn info. [bit/spk]

sparse turn info. [bit/spk]

D

16

0.1

0
360°
720°
theta phase

0°

0.4

0
0.2
0.4
sparse - dense
turn info. [bit/spk]

sparse - dense
turn info. [bit/spk]

0

15

0.2

cumulative distr.

activity [spk/s]

13

0
0.5
1.0
progress along center arm
turn info. [bit/spk]null matched [bit/spk]

C

0
0.5
1.0
progress along center arm

0.06
0.03

L

10

-12

CA3
CA1

10-4

0.004

0

-0.03

cumulative distr.

B

theta phase

A

CA3
CA1
bootstrapped aggregate
all place cells

1.0
0.5
CA3

CA1
0
-1.0 -0.5 0 0.5 1.0
sparse - dense
decoding accuracy

Supplementary Figure 7: Extended results for Fig. 7 of the main text. (A–I) Additional single-neuron results. (A) Spikes from
Fig. 7C (top) and Fig. 7E (bottom) superimposed on the animal trajectory (light gray line) and other spikes (dark gray points).
(B) Similar to Fig. 7C, E, but for three additional CA3 place cells. (C) Similar to Fig. 7D, F, but for the fields in B. (D) Averageturn information per spike conveyed by CA3 place cells over sparse theta phases and over all phases. Each point represents
one neuron, n = 99. Note that many neurons convey close to zero overall turn information but convey substantial sparse turninformation. (E) Similar to Fig. 7G, but comparing place cells in CA3 with those in CA1. CA3 n = 99 and CA1 n = 187. Numbers
indicate p-values calculated by two-tailed Wilcoxon signed-rank tests for each population by the two-tailed Mann-Whitney U
test for the comparison between them. (F) Cumulative distribution functions for values in E. (G) Activity (black), raw position
information per spike (blue), and mean null-matched position information (gray) by theta phase for spikes aggregated across
98 CA3 place cells. For each place cell, the turn direction with higher activity across all phases is identified. Aggregation is
performed by collecting spikes corresponding to more active turn directions and those corresponding to less active directions.
(H) Similar to G, but for 187 CA1 place cells.
(Continued on the next page.)9Supplementary Figure 7: (Continued from the previous page.)(I) Similar to E, but comparing 100 bootstrap subsamples per region of 1000 spikes from the aggregates analyzed in G and H.
(J–L) Additional Bayesian population decoding results. (J) Similar to Fig. 7J, but for CA1 place cells. (K) Similar to Fig. 7K,
but comparing runs encoded by CA3 with those encoded by CA1. CA3 n = 91 and CA1 n = 282. (L) Cumulative distributionfunctions for values in K. For all results, spikes during each traveling direction are separately analyzed. In D, E, F, and I,
information is sparsity-corrected with horizontal lines indicating medians. Source data are provided as a Source Data file.

10

10-4
0

0.93
2

concept accuracy

10-17

0.90
10

-20

2

10

10-27

4

10-36

6

-33

baseline
DeCov
DeCorr

8

0.75

10–47

10–6 0.30

concept p-values

F

10–28
10–21

example p-values

10–6

0.6
0.3

10–19 10–51

10–55

0

-1

0
activation

1

10

6

10

0

10 sets

10

-33

0.16

0.6

10-40

10-4

4

50 sets

10-40

10

-33

6

8

dataset size [×102 ]

1.0
0.8

0.10 10-26
0.006

0.6

10-5 0.45

0.52

0.70

0.50

0.49

0.4

10

2

5
activation

10

0

0.59

8

10–38

10

no nonlinearity

0.70

10–9

0.65

0.58 10–12

10–11

10–8

0.02
10–3 10–34

10–37

no nonlinearity
9
6

0.1

0
0.2
0.4
0.6
Treves-Rolls density

0.86
0.88
concept accuracy
10–13

0.75

0.2

3

0

6

0.75

6

-1

0.18

0.80

10–12

10–34

10-42

dataset size [×102 ]

10–45

0

10-3

4

0.905
0.915
concept accuracy

10–13 0.01

10-44

0.005

0.86
0.88
concept accuracy

10–35

10-33

ReLU nonlinearity

0.75

10-2
0.6 0.7 0.8 0.9
Treves-Rolls density

4

DeCorr loss

ReLU nonlinearity

9

3

2

0.54

10–3

tanh nonlinearity

first halfsecond half

0.7

0.80

0.82 0.84 0.86
concept accuracy

10–18

0.8

more neurons

0.80

10–43

0.8

0.9

0.81

2

0.85

0.55

1.0

0.98

10

more setsbaseline
DeCorrHalfCorr

0.60

D

-18

-34 10

dataset size [×103 ]

0.88
0.90
concept accuracy

probability dist.

10

6

DeCorr loss

-20 10-21 10larger dataset

Eexample accuracy

10-22

0.95

4

example accuracy

0.94

C

B

DeCorrstrength1

example accuracy

concept accuracy

A

0

3
-5

0
activation

5

0

0.5
0.6
0.7
0.8
Treves-Rolls density

Supplementary Figure 8: Extended results for Fig. 8 of the main text. (A–D) Additional results for the single-task architecture in
Fig. 8B. (A, B) Concept and example accuracies as functions of DeCorr loss for 192 networks trained with various strengths of the
DeCorr loss function. Increasing DeCorr strength decreases the final DeCorr loss, decreases concept accuracy, and increases
example accuracy. Dataset sizes are respectively 10 000 and 500 in A and B, and 100 sets are used in B. (C, D) Similar to
Fig. 8E, F, but including networks trained with the DeCov loss function developed by Michael Cogswell and colleagues 1 . Unlike
DeCorr, DeCov improves concept accuracy and does not substantially improve example accuracy compared to baseline. (E, F)
Additional results for the multitask architecture in Fig. 8G. (E) Similar to Fig. 8I, but for different conditions. In each condition,
HalfCorr networks exhibit the best combined performance. From left to right: dataset size of 3000 instead of 1000; 50 sets
instead of 10; 500 neurons in each hidden layer instead of 100; ReLU activation function in each hidden layer instead of tanh;
and linear activation in the second hidden layer and ReLU activation function in the first hidden layer, which makes the network
equivalent to a single-layer perception. (F) Activation properties within the final hidden layer of HalfCorr networks with various
activation functions described in E. Except for the linear activation case, the second, decorrelated half of the layer is sparser than
the first, correlated half. Values elicited by 1000 train images in each of 8 trained networks. Treves-Rolls density is the sparsity
defined in Treves and Rolls 2 and is computed with the absolute value of activations as in Willmore and Tolhurst 3 . Source data
are provided as a Source Data file.

11

Supplementary Methods
Decorrelation in binary feedforward networks
Network architecture
We explore how the correlation of binary activity patterns changes when activity is propagated from one
network to another. The two networks are termed presynaptic and postsynaptic. They have sizes Npre andNpost . The presynaptic network exhibits activity patterns xpre
νi ∈ {0, 1}, where ν = 1, . . . , s indexes patterns
and i = 1, . . . , Npre indexes neurons.
W is the connectivity matrix from the presynaptic network to the postsynaptic network. For simplicity, we
consider binary synaptic weights, so Wij ∈ {0, 1}. The postsynaptic patterns xpost∈ {0, 1} are determined
νiby a simple threshold operation:

X
pre
−
θ
,
=
Θ
W
x
xpost
ij νj
νi

(1)

j

where Θ is the Heaviside step function and θ is the activity threshold. We can rewrite Eq. 1 as


xpost
= Θ gνi − θ ,
νi

where

gνi =

X

Wij xpre
νj .

(2)

j

Here, gνi is the synaptic input onto postsynaptic neuron i for pattern ν.
We characterize these patterns, either presynaptic or postsynaptic, by their densitya = ⟨xνi ⟩(3)and the correlation per neuron between two different patterns ν ̸= ωρ=⟨xνi xωi ⟩ − ⟨xνi ⟩ ⟨xωi ⟩⟨xνi xωi ⟩ − a2=.
⟨xνi xνi ⟩ − ⟨xνi ⟩ ⟨xνi ⟩a(1 − a)(4)

The angle brackets indicate averages over patterns and neurons.
We now assume that the activity patterns and the connectivity matrix are generated via random processes.
preTo be explicit, we will write Xνias the random variable for the activity of presynaptic neuron i in patternpost
ν; the same capitalization applies to the postsynaptic activities Xνi
and inputs Gνi . Lowercase lettersrepresent samples of these variables.
preWe assume that each Xνiis an identically distributed random variable. We assume that the Wij ’s are

independent and identically distributed (iid) Bernoulli random variables with parameter u:
Wij ∼ Ber(u).

(5)

post
In this case, each Xνiis also an identically distributed random variable.

We can then write expressions for density and correlation as
a = E[Xνi ]ρ=E[Xνi Xωi ] − a2.
a(1 − a)(6)

These are population values; Eqs. 3 and 4 indicate the sample estimates and should be written as â and ρ̂,12but we will ignore this distinction.
Generating presynaptic activity patterns
For mathematical tractability, we will enforce density on a per-pattern basis; that is, each presynaptic pattern
has the same number of active neurons:n ≡ Npre apre =

X

pre
Xνi
.

(7)

i

We generate correlated patterns obeying this restriction as follows:
1. We create a concept pattern xpre ∈ {0, 1}Npre by randomly choosing n neurons to be 1 and the rest to
be 0. Let S be the set of all active neurons (value 1) and its complement S c be the set of all inactive
neurons (value 0).
2. To create each example pattern xpre
ν , we randomly select a fraction d of neurons in S and set them to
0. We then randomly select the same number nd of neurons in S c and set them to 1.
Now we calculate the correlation ρpre between the patterns Xpre
generated by this method. To do so,ν
we investigate the distribution of M , a random variable for the number of active neurons common to two
different patterns. We note that
M = MS + MS c ,(8)where MS is the number of neurons in S that remain active in both patterns, and MS c is the number ofneurons in S c that are activated in both patterns. These numbers have hypergeometric distributions:MS ∼ Hyp n, n(1 − d), n(1 − d) ,MS c ∼ Hyp Npre − n, nd, nd .

(9)

We can then calculate
1E[M ]pre pre=E[XνiXωi ] =
Npre
Npre



n2 d2
n2 (1 − d)2+nNpre − n= apre (1 − d)2 +a2pre d2.
1 − apre(10)

Substituting this expression into Eq. 6, we obtain2
1 − apre − d,
1 − apre√d = 1 − apre 1 − ρpre .
ρpre =(11)

Thus, we know the number of activity flips required to produce presynaptic patterns with a given correlation.
Identifying the probability distribution of Gνi
Postsynaptic activity patterns are produced by Eq. 2. Written in terms of random variables, it becomes


post
Xνi
= Θ Gνi − θ ,

where

Gνi =

X
j

13

pre
Wij Xνj.

(12)

Thus, the statistics of Xpost
are determined by Gν . To calculate second-order statistics such as ρpost , weνneed to determine the joint distribution of Gνi and Gωi for two patterns ν ̸= ω.
To do so, we define R0 as the set of active neurons that are common to both patterns Xpre
and Xpreνω .
We define Rν and Rω as sets of neurons only active in patterns ν and ω, respectively. We can then write
Gνi =

X

Wij +

X

Wij ≡ G0i + G̃νi

j∈Rν

j∈R0

Gωi =

X

Wij +

j∈R0

X

Wij ≡ G0i + G̃ωi .

(13)

j∈Rω

These sets have cardinalities|R0 | = M,

|Rν | = n − M,

|Rω | = n − M,(14)

where n is given by Eq. 7 and M is given by Eqs. 8 and 9. Since the elements of W are iid Bernoulli randomvariables (Eq. 5),
G0i | M = m ∼ Bin(m, u),G̃νi | M = m ∼ Bin(n − m, u),G̃ωi | M = m ∼ Bin(n − m, u).

(15)

Since R0 , Rν , and Rω are mutually disjoint, G0i , G̃νi , and G̃ωi are mutually independent when conditionedon M .
Calculating the joint probability density function of Gνi and Gωi
We take the Npre → ∞ limit in which the discrete probability distributions for Xνi and Gνi are replaced bytheir continuous limits. First, we present a few points on notation.
1. A continuous random variable A has probability density fA (a) at value A = a. We may simplify it as
f (a).
2. Variables A and B have joint probability density fA,B (a, b) at values A = a and B = b. We may
simplify it as f (a, b).
3. Similarly, the conditional probability density given that the random variable B takes value b is
fA|B=b (a). We may simplify it as f (a|b).
We can now write the joint probability density function (pdf) for Gνi and Gωi . For notational convenience,
we will drop the subscript i for all relevant variables. The pdf isZ
f (gν , gω ) =dg0 f (gν , gω , g0 )Z

dg0 fG̃ν ,G̃ω ,G0(gν − g0 , gω − g0 , g0 )Z
= dm dg0 fG̃ν ,G̃ω ,G0(gν − g0 , gω − g0 , g0 | m)f (m)ZZ= dm dg0 fG̃ν(gν − g0 |m)fG̃ω(gω − g0 |m)f (g0 |m)f (m).

=

Z

14

(16)

The second line is obtained using the change-of-variables formula for a joint pdf.
We have expressions for each pdf in the integrand. In the large Npre limit, the binomial distributions in
Eq. 15 can be approximated by normal distributions (dropping the subscript i for notational convenience):
G0 | M = m ∼ N mu, mu(1 − u) ,G̃ν | M = m ∼ N (n − m)u, (n − m)u(1 − u) ,G̃ω | M = m ∼ N (n − m)u, (n − m)u(1 − u) .

(17)

Thus, we findZ
dg0 fG̃ν(gν − g0 |m)fG̃ω(gω − g0 |m)f (g0 |m)
2 
2 




gω − g0 − (n − m)ugν − g0 − (n − m)u(g0 − mu)2
exp −
exp −
∝ dg0 exp −2(n − m)u(1 − u)2(n − m)u(1 − u)2mu(1 − u)
" 
"
#
2 #



Z
gν +gωgν −gω 2− (n − m)ug0 −(g0 − mu)22
2
= dg0 exp −
exp −
exp −
(n − m)u(1 − u)(n − m)u(1 − u)2mu(1 − u)
"
#
"
2 #

gν −gω 2gν +gω
− nu
2
2
exp −
.
(18)
∝ exp −(n + m)u(1 − u)(n − m)u(1 − u)Z

We can write the terms inside the exponential as
2gν +gω
− nu
2gν −gω 22

(n + m)u(1 − u)h=+(n − m)u(1 − u)i2h(gν −nu)+(gω −nu)2(gν −nu)−(gω −nu)2i2+(n + m)u(1 − u)(n − m)u(1 − u)
"


1
1
1
1
n+m + n−m=gν − nu gω − nu112
2u(1 − u) n+m
− n−m

=

1
1
n+m − n−m1
1
n+m + n−m1
(g − µG )⊤ Σ−1
G (g − µG ).
2!

gν − nu!#gω − nu(19)

The last expression is written in terms of the variable vector, mean vector, and covariance matrix for Gν
and Gω :
g=

µG =
ΣG =

gν

!

gω
!
nu
nu

=

!
Npre apre u
Npre apre unu(1 − u)!
mu(1 − u)mu(1 − u)nu(1 − u)

2
= σG

1

ρG

ρG

1

!
,

(20)

where the covariance and correlation are2
σG= Npre apre u(1 − u)15

and ρG =m.
Npre apre(21)

Combining Eqs. 16, 18, and 19, we obtainZf (gν , gω ) ∝



1
(g
−
µ
)
dm exp − (g − µG )⊤ Σ−1
G f (m).
G
2(22)

Now we consider M = MS + MS c (Eq. 8). From Eq. 9, we see that MS and MS c have means and variancesµS = n(1 − d)2= Npre apre (1 − d)2σS2 = nd2 (1 − d)2= Npre apre d2 (1 − d)2µS c =

n2 d2
Npre − nn2 d2
σS2 c == Nprea2pre d2
1 − apre2
a2pre d2 (1 − apre − apre d)2
Npre − n(1 + d)
= Npre.
3(Npre − n)
(1 − apre )3(23)

Note that the flip fraction d can be expressed in terms of apre and ρpre via Eq. 11.
As Npre → ∞, the hypergeometric random variables MS and MS c approach normal distributions withthe means and variances in Eq. 23. Thus, their distributions become sharply peaked around their means,
and we can approximate the pdf of M by a delta-function at its mean:f (m) → δ(m − µM ),whereµM = Npre apre (1 − d)2 + Npre

a2pre d2
1 − apre= Npre apre (apre + ρpre − apre ρpre ).

(24)

We now have our final expression for the joint pdf of Gνi and Gωi . Reintroducing the neural index i and thenormalization factor, Eq. 22 becomesf (gνi , gωi ) =
where
gi =

gνi
gωi



1
1
exp − (gi − µG )⊤ Σ−1
(g
−
µ
)
i
G ,
G
2
2π det ΣG
√

!
,

µG =

µG

!

µG

,

2
ΣG = σG

1

ρG

ρG

1

(25)

!
(26)

and
µG = Npre apre u,2σG= Npre apre u(1 − u),ρG = apre + ρpre − apre ρpre .

(27)

Supplementary Figure 1 shows a plot of the joint pdf f (gνi , gωi ) along with a histogram obtained throughnumerical simulation. The theoretical formula Eq. 25 agrees very well with the numerical data.
Integrating the joint probability density function to obtain apost and ρpost
With the joint pdf for Gνi and Gωi (Eq. 25), we can compute the postsynaptic pattern density apost andpostcorrelation ρpost using Eqs. 6 and 12. According to Eq. 12, Xνiacts as an indicator random variable forpost post
Gνi > θ, and the product Xνi
Xωi acts as an indicator random variable for Gνi > θ ∩ Gωi > θ. Thus,
post
E[Xνi
] = P (Gνi > θ)andpost post
E[XνiXωi ] = P (Gνi > θ ∩ Gωi > θ).

16

(28)

We can calculate
post
E[Xνi
]=

Z ∞

Z ∞
dgνi

dgωi f (gνi , gωi )


Z ∞
1
(gνi − µG )2dgνi √
=
exp −
2
2σG
2πσG
θ
1
θ − µG= erfc √
.
2
2σG
−∞

θ

(29)

Thus, the postsynaptic pattern density is immediately
post
apost = E[Xνi]=1ϕerfc √ ,
2
2

where

ϕ=

θ − µG.
σG(30)

The rescaled threshold ϕ is the standardized version of θ.
We next need to calculate
Z ∞Z ∞post post
E[Xνi Xωi ] =dgνidgωi f (gνi , gωi )
θ
θ


Z ∞
Z ∞
1
1
√
dgνi
dgωi exp − (gi − µG )⊤ Σ−1
(g
−
µ
)
=
i
G .
G
2
2π det ΣG θ
θ(31)

By standardizing the variables of integration with hνi = (gνi − µG )/σG , this integral can be expressed interms of the standard bivariate normal:
post post
E[XνiXωi ] =

Z ∞

1
2π

p

1 − ρ2G

Z ∞
dhνi

ϕ

ϕ


 2
hνi + h2ωi − 2ρG hνi hωi.
dhωi exp −2(1 − ρ2G )(32)

This double integral cannot be evaluated in closed form, but we can reduce it to a single integral 4 :post post
E[XνiXωi ] = Γ[ϕ, ρG ] ≡

1
2π

Z π


dψ exp −arccos ρGϕ2.
1 + cos ψ(33)

Therefore, the expression for the postsynaptic correlation follows:ρpost =

Γ[ϕ, ρG ] − a2post,
apost (1 − apost )(34)

where apost can be expressed in terms of the standardized threshold ϕ with Eq. 30. On the other hand, we
can stipulate a desired apost and then recover ϕ and ρpost with√

2 erfc−1 (2apost ),√Γ 2 erfc−1 (2apost ), apre + ρpre − apre ρpre − a2post
ρpost =.
apost (1 − apost )
ϕ=

(35)

This is Eq. 1 of the main text. Figure 2E shows that this formula for the postsynaptic correlation agrees
well with values obtained through numerical simulation across a variety of parameter values.

17Exploring ρpost as a function of apre , apost , and ρpreIn Supplementary Fig. 2B, we plot ρpost as a function apre and apost for various ρpre . We see that decorrelation(ρpost < ρpre ) occurs when apost is low. Thus, a downstream (postsynaptic) network with many neurons but
low activity naturally decorrelates patterns of the upstream (presynaptic) network. The low activity can beachieved by low connectivity u or a high threshold θ; Eq. 35 does not differentiate between the two.
Supplementary Figure 2C demonstrates that if presynaptic patterns are sparse and decorrelated (lower
left corner of the plot), postsynaptic patterns also exhibit low correlation even if they are denser. Thus, once
patterns are sparsified and decorrelated, they will remain decorrelated for subsequent feedforward layers.
Note that this panel also shows the symmetry in interchanging apre ↔ ρpre present in Eq. 35.

CA3 model with random binary patterns
We use a network size of NCA3 = 10 000. We generate MF example patterns xMFµν with desired density
aMF and correlation 0 by randomly activating NCA3 aMF neurons. We generate PP concept patterns xPPµ
with desired density 0.5 by randomly activating each neuron with probability 0.5. We then generate PP
example patterns xPPµν with desired correlation ρPP by randomly flipping each concept neuron with probability√(1 − ρPP )/2. Simulations are initiated without cue noise and a sharp activation threshold (β → ∞) toassess the best possible network performance.
To calculate capacities in Fig. 3H, I, we use a higher strength of PP inputs ζ = 0.2 to make the capacity
values more computationally accessible. For a given load of concepts per neuron, we perform a grid searchover the load of examples per concept using 8 networks per load and testing 20 cues in each network. Eachcue is identical to its target pattern. We search for the load at which the average overlap crosses a threshold,√which is 1/2, (1 + ρPP )/2, and (1 + ρPP )/2 for MF examples, PP examples, and PP concepts, respectively,to account for the positive overlap of off-target PP patterns if they are correlated 5 . For MF examples, we
explore activity thresholds θ′ between 0.43 and 0.85 and use the value that maximizes average overlap. For
PP examples and concepts, we set the activity threshold θ′ to 0.

CA3 model behavior during oscillating threshold
The oscillation analysis in Fig. 4C characterizes network behavior between update cycles 60 and 120. Consider
a single oscillation cycle. For example-related behavior, we consider its high-threshold half. Let m1 (t) andm2 (t) be the largest and second-largest overlaps within the target concept at time t, and let m0 (t) be thelargest overlap within other concepts. If m1 (t) > 0.8, m1 (t) > 2 · m2 (t), and m1 (t) > 2 · m0 (t) at any time,
the behavior of the oscillation cycle is categorized as single examples within a target concept. If m1 (t) > 0.8,
m1 (t) < 2 · m2 (t), and m1 (t) > 2 · m0 (t), the behavior is categorized as mixed examples within a target
concept. If at least half of the oscillation cycles receive a certain categorization, it is considered the network
behavior. Otherwise, the network behavior is examples within other concepts. For concept-related behavior,
we consider the low-threshold half of each oscillation cycle. Let m1 (t) be the overlap with the target concept
and m0 (t) be the largest overlap with other concepts. If m1 (t) > 0.1 and m1 (t) > 2 · m0 (t) at any time,
the behavior of the oscillation cycle is categorized as target concept (for the random binary patterns in
Supplementary Fig. 4B, we use m1 (t) > 0.2 instead). If at least half of the oscillation cycles receive thiscategorization, it is considered the network behavior. Otherwise, the network behavior is other concepts.

18Experimental data preprocessing
In all experimental analyses, we consider each traveling direction separately. Thus, each recorded neuron
effectively yields two neurons in our analyses with their own spikes and trajectory occupancies.
Linear track data from the CRCNS hc-3 dataset is used to produce the results in Figs. 5 and 6 6 . For
CA3, we use all linear track sessions from rats ec013, ec016, gor, and vvp with CA3 neurons recorded, andfor CA1, we use all linear track sessions between 761 and 882 from rat ec013. For rats gor and vvp, thesize of the linear track was changed during recording, and we only consider data before the change. Animal
positions are taken to be the mean of the two LED lights on the microdrive. The track axis is taken to be thefirst principal component of sampled positions. Animal velocities are differences between tracked position
samples divided by the sampling rate and smoothed with a Gaussian filter with standard deviation 0.1 s.
We use the recommended quality criteria involving eDist, RefracRatio, and RefracViol when selecting
units. Since we are interested in the theta oscillation, we only consider spikes occurring during locomotion
with speed greater than 10 cm/s. We require units to have at least 50 spikes occurring within the central
70% of the track to avoid neurons whose behavior may be dominated by boundary effects. To determine thetheta signal for each unit, we identify the tetrode from which it was recorded and average the LFP over all
channels on the tetrode. This signal is bandpass-filtered between 6–10 Hz, and the complex argument of its
Hilbert transform is the local theta phase.
To extract place fields for Fig. 5, we first discretize track positions with 1 cm bins. For a given place
cell, we first compute the activity across all theta phases as a function of position and apply a Gaussian
filter whose standard deviation is 0.01 times the track length. We find activity peaks whose maximum is
0.6 standard deviations above the mean; if the activity exhibits multiple peaks while remaining above thisthreshold, the largest is chosen. We then find the closest flanking positions where the activity falls below 0.2
times the peak value. The region in between is the place field. If two place fields overlap, they are divided
at the activity minimum located between their peaks.
W-maze data from the CRCNS hc-6 dataset is used to produce the results in Fig. 7 7 . We use all run
sessions from rats bon, con, dud, fra, mil, and ten. We remove 2 sessions from dud and 3 sessions fromfra without position samples in one of the side arms. Animal positions and velocities are taken directly as
the 30 Hz-interpolated samples from the dataset. We only consider spikes occurring during locomotion withspeed greater than 5 cm/s. We require units to have at least 30 spikes occurring within the center arm. Spiketheta phases are taken directly from the dataset.
To extract arm identity from the position samples, we first linearly rescale both position coordinates tospan from 0 to 1. We define a maze skeleton consisting of lines that represent the left arm from (0.1, 0.1)
to (0.1, 1), the center arm from (0.5, 0.1) to (0.5, 1), the right arm from (0.9, 0.1) to (0.9, 1), and the basefrom (0.1, 0.1) to (0.9, 0.1). We then fit transformations of this skeleton to the position samples, allowing
for stretching along the first coordinate, rotation about its center, and translations. The fit is performed byminimizing the total squared distance between each sampled position and its closest point on the transformed
skeleton. Empirically, this process yields excellent fitting without any need for manual intervention. Then,each position sample is assigned to either the left, center, or right arm based on closest distance.
To extract runs along the central arm, we first smooth arm samples by encoding arm identity as a one-hot
vector, applying a Gaussian filter with standard deviation 0.2 s, and identifying the largest element in eachsample. We then consider each span of central-arm samples. We find the times within the span at which theanimal crosses scaled positions 0.4 and 0.7, where 0 corresponds to the smallest position value at the baseof the maze and 1 corresponds to the largest position value at the far end of the arms. The run duration

19

must be between 0.1 s and 10 s, and we then pad the run by 0.5 s at both ends. Outward runs cross scaled
position 0.7 before 0.4, and the subsequent arm identity is used to determine the future turn direction.
Inward runs cross scaled position 0.4 before 0.7, and the previous arm identity is used to determine the past
turn direction.

Experimental data aggregate analysis
The aggregate fields in Supplementary Fig. 5D–J are formed from phase-precessing place fields. We do not
enforce a minimum spike count or ensure theta modulation on a single-neuron basis. In total, we collect
19 678 spikes from 57 CA3 place fields and 29 664 spikes from 55 CA1 place fields. We perform bootstrappingby sampling with replacement 1000 spikes at a time. For each subsample, we bin spikes into 10 progress bins
and phase bins of width 15◦ .
The aggregate fields in Supplementary Fig. 7G–I are formed from place cells with at least 30 spikes within
the central arm. We do not ensure theta modulation on a single-neuron basis. For each neuron, we identify
the turn directions with higher and lower activities, and collect spikes occurring during each condition. In
total, we collect 47 196 spikes from 331 CA3 place cells and 72 461 spikes from 436 CA1 place cells. We
perform bootstrapping by sampling with replacement 1000 spikes at a time. For each subsample, we bin
spikes into 2 directions (more active and less active) and phase bins of width 30◦ .

20

References[1] M. Cogswell, F. Ahmed, R. Girshick, L. Zitnick, and D. Batra. Reducing overfitting in deep networks
by decorrelating representations. arXiv 1511.06068, 2015.
[2] A. Treves and E. T. Rolls. What determines the capacity of autoassociative memories in the brain?
Netw. Comput. Neural Syst., 2(4):371–397, 1991.
[3] B. Willmore and D. J. Tolhurst. Characterizing the sparseness of neural codes. Netw. Comput. NeuralSyst., 12(3):255–270, 2001.
[4] D. B. Owen. Tables for computing bivariate normal probabilities. Ann. Math. Stat., 27(4):1075–1090,
1956.
[5] L. Kang and T. Toyoizumi. Hopfield-like network with complementary encodings of memories. Phys.
Rev. E, 108(5):054410, 2023.
[6] K. Mizuseki, A. Sirota, E. Pastalkova, K. Diba, and G. Buzsáki. Multiple single unit recordings fromdifferent rat hippocampal and entorhinal regions while the animals were performing multiple behavioral
tasks. CRCNS.org, 2013.
[7] M. Karlsson, M. Carr, and L. M. Frank. Simultaneous extracellular recordings from hippocampal areas
CA1 and CA3 (or MEC and CA1) from rats performing an alternation task in two W-shapped tracks
that are geometrically identically but visually distinct. CRCNS.org, 2015.

21