Binding in hippocampal-entorhinal circuits enables compositionality in cognitive maps
Authors: Christopher Kymn
Sonia Mazelet
Anthony Hitchcock Thomas
Denis Kleyko
Edward Paxon Frady
Friedrich Sommer
Bruno Olshausen
Venue: NeurIPS 2024 poster (2024)
Keywords: cognitive maps
compositionality
hippocampus
entorhinal cortex
Primary area: neuroscience_and_cognitive_science

Abstract: We propose a normative model for spatial representation in the hippocampal formation that combines optimality principles, such as maximizing coding range and spatial information per neuron, with an algebraic framework for computing in distributed representation. Spatial position is encoded in a residue number system, with individual residues represented by high-dimensional, complex-valued vectors. These are composed into a single vector representing position by a similarity-preserving, conjunctive vector-binding operation. Self-consistency between the vectors representing position and the individual residues is enforced by a modular attractor network whose modules correspond to the grid cell modules in entorhinal cortex. The vector binding operation can also be used to bind different contexts to spatial representations, yielding a model for entorhinal cortex and hippocampus. We provide model analysis of scaling, similarity preservation and convergence behavior as well as experiments demonstrating noise robustness, sub-integer resolution in representing position, and path integration. The model formalizes the computations in the cognitive map and makes testable experimental predictions.

Binding in hippocampal-entorhinal circuits enables compositionality in cognitive maps
Christopher J. Kymn", Sonia Mazelet Anthony Thomas Denis Kleyko*s, E. Paxon Frady", Friedrich T. Sommerl,6, and Bruno Olshausen
"Redwood Center for Theoretical Neuroscience, Helen Wills Neuroscience Institute, UC Berkeley, Berkeley,
USA
Université Paris-Saclay, ENS Paris- Maris-Saclay. Gif-sur-Y France
3 Department of Electrical and Computer Engineering, UC Davis, Davis, USA
+ *Centre for Applicd Autonomous Sensor Systems, Örebro University, Örebro, Sweden 'Iteelligent Systems Lab, Research Institutes of Sweden, Kista, Sweden "Intel Labs, Santa Clara, USA
Herberr Wertheim School Optometry Vision Science, Berkeley, Berkeley, USA cjkyym@erkkeeey.ee edu
Abstract
We propose a normative model for spatial representation in the hippocampal formation that combines optimality principles, such as maximizing coding range and spatial information neuron, with an algebraic framework for computing in distributed representation. Spatial position is encoded residue number system, with individual residues represented by high-dimensional, complex-valued vectors. These are composed into single vector representing position by a similarity- preserving, conjunctive vector- bintr-- operation. seff-consistency between the representations the overall position and the individual residues is enforced by modular attractor network whose modules correspond to the grid cell modules in entorhinal cortex. The vector binding operation can also associate different contexts to spatial representations, yielding model for entorhinal cortex and hippocampus. We show that the model achieves normative desiderata including superlinear scaling of patterns with dimension, robust error correction, and hexagonal, carry- -free encoding of spatial position. These propertics turn enable robust path integration and association with sensory inputs. More generally, the model formalizes how compositional computations could occur in the hippocampal formation and leads testable experimental predictions
Introduction
The hippocampal formation (HF), consisting of hippocampus (HC) and the medial and lateral part of the neighboring entorhinal cortex, (MEC) and (LEC), is critical for forming memories and representing variables such as spatial position [T] 2). Recent work has provided evidence of compositional structure HF representations, enabling complex representations be composed by simpler building blocks and their formation rules. Examples include novel recombinations of past experience occurring in replay ( the exponential expressivity of the grid cell code [4 [5]. In particular, compositional representations afford high expressivity with lower dimensional storage requirements (6). less complexity in latent state inference, and generalization to novel scenes with familiar parts.
To gain insight into the possible computational principles and neural mechanisms at play in the HF, we take a normative modeling approach. That is, we seek to construct a model built from set of
Code is available at http:///hhhhg.oooooooaaaaaaaatta
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

neural coding principles that effectively achieves the postulated function of the system. With this approach. we can then explain details about the neuroanatomical and neurophysiological structures in light of their particular contributions to an information processing objective. The resulting model can also lead to new predictions about the neural mechanisms that enable this function.
The postulated function of the HF -as a cognitive map and episodic memory- has a core com- putational requirement, represent and navigate space. Here, space is either the actual physical environment or a more abstract conceptual space. We formulate multiple desiderata for an effective representation of space. We then show that a residue number system, incorporated into a compo- sitional encoding scheme. fulfills these desiderata. It is achieved by modular attractor network that factorizes encoded locations into components of a residue number system. This provides an algorithmic-level hypothesis of huppecmpal-entorhine algorithm is binding, which draws inspiration from work in neuroscience, cognitive science, and artificial intelligence.
interactions. A core mechanism of this
normative model for the hippocampal formation
2.1 Principles for representing space
Our first principle is that space is represented by compositional code that has high spatial-resolutinn, is noise-robust, and in which algebraic operations on the components can be updated in parallel. Prior work 4 [] has proposed the residue number system (RNS) (7) as candidate for fulfilling these requirements. An RNS expresses an integer in terms of its remainder relative to set of co-prime moduli [m, For example, relative to moduli {3, 5, 7] x = 40 is encoded as { 0, 5). The Chinese Remainder Theorem guarantees that all integers in the range [0, M - 1], where M II. m, are assigned unique representation. An RNS provides high spatial resolution, carry-free arithmetic operations, and robust error correction [8]. Experimental observations in entorhinal cortex show a discrete multi-scale organization of spatial grid cells that is compatible with the assumption of discrete RNS modules.
The second principle we adopt is that an individual residue value should be encoded by neural population in a similarity-presevinng fashion. In particular, we require that distinct integer values are represented with nearly orthogonal vectors. To achieve this principle, we use method similar to random Fourier fcatures [IO]. Each modulus, with value m,, is assigned a seed phasor vector, gi E CP, whose elements (g:), are drawn uniformly from the m,-th roots of unity (i.e., (g:); e e--lww with Wij =
2x m, ky. and ky chosen randomly from (0,.., my - 1).. The representation of a particular residue value a; E {0, ...,, m, 1} is then given by rotating the phases of the seed vector according to
g: (a,) = (g:
where we abuse notation slightly to also think of g; as a function that takes a, as input and produces an embedding described above. The complex-values vectors can be mapped to interpretable population vectors via a randomized Fourier transform (Figures 6D and S22.
Our third principle concerns the manner in which unique representation of a particular point in space formed from the individual residue representations. This requires that we somehow combine the residue vectors for each modulus. Combining via concatenation, though straightforward, is not effective because codes that coincide in subsets of their residue representation would be similar, even when the encoded values are very different. Thus, the method of combining residue codes must be conjunctive. Conjunctive composition is often called binding and is of fundamental importance in neuroscience [12), cognitive science [13), ). and machine learning [14] An early proposal for binding is the tensor product of vector representations [15), with the tensor order equal to the number of bound objects.
Here, we implement binding with component-wise vector multiplication, a dimensionality preserving operation that represents a lossy compression of the full tensor product [16] [7]. The resulting compositional vector representation of an integer x E Z using an RNS representation with K moduli, {ai, 02 ... axk is:
K
(
g:(a (a:)).
1=1

---

We prove in Appendix A.1 that this coding scheme represents distinct integer states using nearly orthogonal vectors, and we show that it generalizes in natural way to support representation of arbitrary real numbers in a similarity preserving fashion.
Eq. represents individual points along line. In general, however, spatial representation involves points in 2D or 3D spaces. Conveniently, vector binding can be also used to compose representations of multidimensional lattices from vectors representing individual dimensions. As we will explain, there is still a choice in this composition that determines the resulting lattice structure. Following earlier proposals [18 20 our fourth normative principle is to choose the lattice structure so that spatial information is maximized, as described in Section 3.5
The final principle require is that for computations such as path integration, there should be simple vector manipulation that results in addition of the encoded variables. Again, vector binding provides this functionality with our coding strategy, because of the following property:
g T O g (v) = g(x + y).
2.2 Modular attractor network for spatial representation
A standard model of grid cell circuits is the line attractor, in which states that represent a consistent location lie on a low-energy manifold (4). When initialized from a noisy location pattern, the circuit dynamics will generate a denoised location representation. Rather than forming line attractor model for the entire representational space (. we propose a modular network architecture, so that the compositional structure of a residue number representation can scale towards a large range with fewer memory resources (Section 3.2). in a manner robust to noise (Section 3.3).
A starting point for our attractor network model is the Hopfield network, which acts as an associative memory by storing memory patterns as fixed- prid-point attractors. The Rademacher- Eademacchr-Hopffell network (21) is a dynamical system whose state is a vector x E (-1, +1) that obeys the following dynamics:
x(t + 1) sign xx x(t)
with X the matrix of memorized patterns (column vectors of X). The fixed-point attractor dynamics can be generalized to complex memory patterns z E CD:
z + 1) =
o (ZZ (ZZiz(t))
(5)
where is non- non-linearity normalizing the amplitude of each complex-vallee valued component one [22] and Z the corresponding matrix of memorized patterns. The model can also be discretized, such that each component is often quantized to a r-state phasor (23). The Hedeemmcher-Hooffiee model is the
special case where r = 2 and the phasors happen to be real-valued.
An r-state phasor network of the form of Eq. 5 is well-suited to serve as an attractor network for each of the residue vectors in an RNS representation of position, with r = m, for modulus i, and the matrix Z (which we shall denote G.) storing the (a:) for Og E {0,... .., m, 1}. However, we desire method for representing the whole coding range M == II m, without storing all M patterns in one large associative memory. For this purpose we show that a resonator network, a recently proposed recurrent network for unbinding conjunctive codes (6), lets us represent this range by storing only n == Σx m, & M patterns. Given a vector encoding of position, p(x). as formulated in Eq. (2), resonator network will factorize it into its constituent RNS components by iteratively updating each residue vector estimate, g. This update is similar to the attractor dynamics of Eq. but made to be consistent with p(x) given all other residue estimates g
K
G. 8. G.
(p O (0)) iti
Vi
(6)
Let us now assume that the input p(z)) encodes spatial position 2; using Eq. 2 Given velocity input qi (ve), estimated from self-motion input, path integration is performed by first running attractor dynamics, then updating attractor states by velocity.
K
gt+++ 1) q. (v)) @o)
O
(7)
(f)
3

---

(MEC)
Figure 1: Schematic of proposed attractor model. In MEC, the gi are residue representations in grid modules, and с encodes a context label. Input of velocity estimate q(v) can produce path integra- tion in grid modules via binding, denoted by O. In HC, p represents contextualized place. Binding serves two roles in the MEC/HC interaction (sym- bolized by bidirectional arrows): a) factorizing p into gi and b) generating update of p from the g, is, for example, after path integration. In LEC, S represents sensory input, interacting with p through learned heteroassociative projection.
C
G
(HC)
CP
0 0
((v)
9
G
6
(LEC)
After velocity updates, one can update the input state p(x+)) with the conjunctive representation of the current factor estimates:
p(1(+1) O (t 1).
(8)
Further explanation and detail is provided in Appendix B.3
2.3 Mapping the model to the HF
Although it not obvious how the components of our normative model should map to the anatomical architecture of HE, we make one proposal as shown in Figure [ The memory networks for residue representations g correspond to grid modules in MEC. Similar to the grid modules, a module for context can be added to the architecture, such as tag for the identity of specific environment, with recurrent synapses C storing tags of different environments.
The context neurons could correspond to the non-grid entorhinal cells, which can contain local, non-spatial information about the environment (27]. The vector p(xx) can be linked place cells in hippocampus. Internal HC circuitry can either buffer the input as in Eq. (6) or allow it to be updated dynamically according to the MEC input (Section 4.1p. The mutual interactions between HC and MEC grid modules require projections between these structures. The binding operations that these interactions involve according to Eq. (6) are hypothesized to be implemented by nonlinear interactions between dendritic inputs in HC and MBC neurons.
The model also assumes the ability for sensory cues to provide the initialization signal of the cognitive map, represented by in Figure For completeness, we adopt the assumption of previous models (c.g.. (28) that heteroassociative memories are formed by the brain that link sensory cues to the hippocampal representations p (Section 4.2) This process would require the system to generate new context vector e and initialize the cognitive map to default location in order learn about new environments. We show that through even simple heteroassociative mechanism, our modular attractor network can robustly retrieve sensory memories and even protect its compositional structure.
3
Coding properties of the model
3.1 RNS representations have exponential coding range
The compositional RNS vector representation Eq. can encode coding range of M values using total of n component patterns for representing the residue of individual modules. The scaling of the coding range exponential in the number of moduli, K, since if each module has (m)) patterns, and the co-prime condition is satisfied, the scaling of the coding range is O(m K This recovers the expressivity argued by 29].
More generally, it also exponential in the number of component patterns, n. The optimal coding range is given by the best partition of into set of positive (m.). This optimization is identical to that of finding the maximum order of an element in the group of permutations S,, because the maximum order can be found by finding the longest cycle. The scaling of this value in n is characterized by
Landau's 's function fn)), which is known to converge to exp( Vn In n²) - oo [30]. Figure 2

---

A
C
10
12
K-2
Scaling coefficient ((tx)
Coding range (M)
10
log Cooling range)
10
2.00
K=
1.75
10
K-5
1.50
10
1.25
10
1.00
10
100 200
2.5
5.0
Patterns stored (n)
log(Vector dimension)
Number modules (K)
Figure 2: Residue number systems, combined with modular attractor network (resonator network), result in a new kind of attractor neural network with favorable scaling for a large combinatorial range. A) Number of encoding states, M, grows rapidly in the number of modules, up to maximum established by Landau's 's function (black dots). B) Coefficient of coding range, M, scales roughly O ), depending the number of moduli, K, but with o >> 1. C) Estimation of scaling from slopes of linear regression (fit to log-log scale). Higher values of K require higher dimension to achieve a particular coding range; empirical values are close ak K-I-
illustrates how Landau's function is the upper bound to what is achievable for any fixed number of moduli (K).
Though other kinds representations can achieve an exponential coding range, the advantage of the compositional encoding of Eq. (2) comes from the fact that the binding operation implements carry-free vector addition (our fourth principle). This enables updates the encoded value without requiring further transformations such as decoding, facilitating tasks such as path integration (Section 4.1 Appendix C.3). Binary representations, by contrast, have exponential coding range but require carry-over operations to implement.
3.2 The modular attractor network has superlinear coding range
The exponential scaling of the coding range of the RNS representation is a prerequisite to obtain a large coding range with the attractor network that has to perform computations this representation, such as input denoising, working memory, and path integration. To estimate the scaling the coding range in the proposed attractor network (Eq. (6) study the critical dimension for which the grid modules converge with high probability. Specifically, we empirically estimate the minimum dimension required to retrieve arbitrary RNS representation with high probability, given maximum number of iterations (Figure 3). Remarkably, we find that the number of component patterns n that can be stored is superlinear in the pattern dimension D; empirically O(D)) for some 1. For 2, and moduli, a x= 2.05, 1.45 and 1.23, respectively (Figure 2C).
These empirical scaling laws are consistent with simple cntrmatiionthhoredde calculation (Ap- pendix A.2). The minimal amount of bits be stored for the entire RNS vector encoding scheme is of order (M log M), and the number of synapses in the attractor network O V M). If one makes the cautious assumption of a capacity per synapse of O(1,, the leading order for the coding K
range M is O(D)), with K-T
While the coding range increases with the number of moduli (K) for the RNS representation, the superlinear scaling coefficient o& decreases with K for the modular attractor network, reaching maximum superlinearity at the smallest value = 2. This reversal is caused by the fact that increasing K decreases the number of synapses, i.e.. the memory resource in the attractor network.
3.3 Robust error correction
In addition, we evaluate the robustness of our attractor model to noise. Because the RNS represen- tations are composed phasors, which are circular variables, we sample noise from a von Mises distribution with two parameters: mean (μ 0) and concentration pattern æ (Figure 3 A). Higher values imply less noise; the distribution approximates Gaussian with variance 1/8 K for large K. Further tests of model robustness to dropout, limited precision, and ablation are provided in Figure S6
5

---

A
von Mises PDF
B
Inpul noise
C
Updaate noise
D
Codebook noise
K=
Frobabiity donaly
0.8-
0.8
0.8
Acouracy
0.8
0.8
0.6
0.6
044
0.4
04
0.4
0.2
0.2
x
02-
02
K-44
0.0
0.0
0.0
11/2
0
102
10" 10"
-0" 10
10"
10"
Phase shift
Coding range
Coding range
Cocing range
Figure 3: Recovery of encoded positions is robust to multiple kinds of noise. A) Visualization of the von Mises weight distribution. Note that the magnitude of the noise is inversely proportional K, and that the variance of the phase perturbation is much larger than the distance between the discrete states of phasors. B-D) Visualizations of accuracy as a function of coding range and k for three separate cases: input noise (B), update noise (C), and codebook noise (D), Cases are shown in order of increasing difficulty. The resonator network maintains perfect accuracy up to a point, after which accuracy decays at an earlier point than the noiseless dynamics (black curve).
We consider three cases: noisy input patterns, noise added to each time step, and noisy weights corruptions of patterns G; (Appendix B.2 The empirical accuracy recall varies depending on the type of corruption applied (Figure JA We find that for given dimension D (in this case, 1024), increasing noise decreases the maximum coding range that can be decoded with high accuracy (Figure B-D). For fixed noise level, the high-accuracy coding range is largest for input noise, followed by update noise and codebook noise. is perhaps not surprising that codebook noise has the worst coding range, given that noise added to every stored pattern compounds across the dynamics. Fortunately, the demonstrated robustness to input noise enables sensory patterns to be denoised via heteroassociation (Section 4.2
3.4 Interpolation between patterns enables continuous path integration
In general, there sharp difference between point and line attractors. our attractor model, the RNS representations of integer values are stored as discrete fixed points. Nevertheless, the attractor network also converges to states that represent non-integer values that are not explicitly stored. In other words, the network smoothly interpolates to points integer and non-integer values encoded by ( Figure 4A provides a visualization, showing that the kernel induced by inner product operations retains graded similarity for sub-integer shifts. This kernel enables the modular attractor network to settle to fixed points that correspond to interpolations between integers, and for sub-ing sub-integer positions to be decoded.
manifold of states that represent
The resolution of decoding is fundamentally limited by the signal to noise ratio. Even so, we find that, up to fixed noise level, the accuracy regimes of integer decoding and sub-integer decoding coincide. This property enables sub-integer position shifts to be encoded within the states of the network, which, as we will show, results in stable, error-correctin path integration (Section 4.1). We quantify the gain in precision terms of the bits information that can on average be reconstructed from vector (Figure 4D, Appendix B.2). Notably, even moderate noise level of k 8 results in nearly the same information content as the noiseless case.
3.5 Triangular frames in 2D maximize spatial information
In two- swo-dimensional open field environments, grid cells have firing fields arranged in hexagonal lattice (3I]. Work in theoretical neuroscience shows the optimality this lattice for 2D environments in terms of spatial information [0]. However, the presence of hexagonal firing fields raises puzzle for residue number systems. Although crucial property of RNS is the carry-free property, most implementations of RNS will not perform carry-free updates within module in non-Cartesian coordinate systems. This generally occurs because the updates of different coordinates must interact due to nonoorhhog.nality.

---

B
= 4.0
x 2.0
1.0
20
Ilnner product|
08
Accuracy
0.8
Accuracy
0.8
15
0.6
8x = 0.03
10
04
0.4
04
0.14
Bits
K-8
0.33
0.
Knd
00
0.0
10
10"
10"
Modulo value
Coding range
Coding range
Coding range
Figure 4: Smooth interpolation between integer states enables encoding and decoding of sub- integer values. A) Visualization of interpolation between two integer states. The position of the fractional value can be estimated by fitting periodic sinc function (Appendix [A.I] based on the inner products with integer codebooks (visualized in dots), then finding the location of the peak. B, C) Sub-integer states can be be decoded, up to precision by the noise level. Note that in both cases, sub-integer decoding can be just as accurate as integer decoding for the same range, even though the sub-integer decoding problem is strictly harder. Even K
sufficient achieve accuracy within a precision of Ar = 0.07, but for higher noise (k = 2), the precision is worse. D) The best spatial precision (in bits) that can be decoded for fixed noise level. Representations with less noise achieve both higher coding range and higher information content per vector.
We resolve this issue by showing how to implement version of vector binding of multiple coordinates a triangular 'Mercedes- Benz' frame that enables carry- hexagonal coding. Furthermore, we provide combinatoric argument for the optimality of triangular frames for RR2 (A frame is spanning set for a vector space which the basis vectors need not be linearly independent.) Our argument relies the combinatorics of residue numbers, and so for the first time gives explanation of why the coexistence of RNS and hexagonal codes is optimal.
To form hexagonal tiling of 2D position requires two steps: first, projection into 3-coordinate frame, and second, choosing phases such that simultaneous, equal movements along all three frames cancel out (Appendix A.3).. The resulting Voronoi tessellation for different states is pictured in Figure 5A. This encoding enables higher spatial resolution in terms of the number of discrete states: 3m² 3m for triangular frames, versus m² for Cartesian frames. This increased expressivity results in a higher entropy) code for space (Figure B)) It also results both a periodic hexagonal kernel and the individual grid response fields being arranged in hexagonal lattice (Figure GC))
(bits)
Antropy
hcxagonal
square
ablique
Modulus
Figure 5: Hexagonal coding improves spatial resolution. A) Voronoi tessellation for m 5. Each distinct color corresponds to unique codeword in CP. Black arrows show the coordinate axes of the triangular 'Mercedes-Beee Benz' frame in 2D. B) Hexagonal lattices have higher entropy than square lattices, allowing each state to carry higher resolution in its spatial output.
Prior models achieved hexagonal lattices either by pattern formation from circularly symmetric receptive fields (e.g., [32 33)
arranged on a periodic rectangular sheet or by distorting a square lattice into oblique one (e.g., 28 34). Importantly, oblique lattices have the combinatorial
complexity as the square grid and, unlike construction described above, they not achieve the
same level of spatial resolution (Figure BB).
7

---

Testing functionalities of the model
4.1 Robust path integration
A
Trye trajetory
Total dislance
Decoded postion with attractor dynemiss
Errre no erter conection
Decoded position withour atrroctor dunamics
Ercor giid call emror correction
Distance 15
pootton
Vettor
Time
C
Time (s)
topeciory
Decoded position (ronay irpu))
Sonsory Frp
Time
Figure 6: Attractor dynamics facilitate robust path integration. A) Example of path integration of a 2D trajectory in the case of intrinsic input noise on the place cell representation. The grid cell modules correct the noise that would otherwise induce drift after a short period of time. B) Path integration results averaged over multiple trajectories in the case of intrinsic input noise on the place cell representation. Grid cell modules limit noise accumulation along the trajectory. Solid lines report the median error over 100 trials, with shaded intervals reporting 25th and 75 percentile. C) Simulated trajectory. along which colors represent the similarity between the g, of three different modules and vectors representing each position in the environment. We see hexagonal response fields, similar those obtained from single unit recordings of MEC. D) Sensory patterns (symbolized by red dots), representing visual cues, are associated to positions in the environment. Presentation of visual cues helps correct drifted positions due to extrinsic noise.
Given the ability of the attractor model to update its representation of position from velocity inputs, along with its ability represent continuous space, we evaluate its ability perform path integration in the presence of noise. We simulate trajectories based on a statistical model for generating plausible rodent movements in an arena [35] 36), and we update grid cell and place cell state vectors according to Equations 7 and g respectively.
To evaluate the robustness of the model to error (Appendix B.3
consider both sources of extrinsic
noise (e.g., mis-represennations of velocity information), and intrinsic noise (e.g., due to noise in weight updates). The robustness of model to intrinsic noise is tested by comparing our results to the estimated trajectories obtained without the correction by the MEC modules (Figure 6M and B). We find that our model strongly limits noise accumulation along the trajectory and allows highly accurate integration for a longer period of time (Figure(6A)) Consistent with our previous experiments on noise robustness (Figure 3). we find that the model has strong robustness intrinsic noise, with extrinsic noise resulting in a drift of the estimated position.
We visualize the response fields in different modules and find hexagomal lattices with module dependent scaling (Figure 6C. Appendix [.J).. D. In addition, we show that tethering to external cues (e.g., visual inputs), significantly increase the accuracy of the attractor network. To study this, we associate visual cues to corresponding patches see Section 4.2) and observe that integration of information from sensory visual inputs succeeds in correcting drift due to extrinsic noise (Figure 6D).
4.2 Denoising sensory states via heteroassociative memory
Finally, we describe simple extension to our model, which sensory patterns are fed from the lateral entorhinal cortex (LEC) to update the hippocampal state. This is consistent with theories of

---

A
60 patterns
B
210 patterns
C
1,0
1.0
1.0
Accuracy (solid)
08
Accuracy (solid)
08
Accuracy
0.-
0.6
D-256
0.4
pattoma
pattem
D=512
pattera
0.2
101024
0.2
0.2
pallerns
pattoma
00
D-2048
0.0
0.0
10 patterns
0.0
01
02
c33
04
0.5
0.0
0.1
022 0.3
04
0.5
0.0
0.1
0.2
0.3
0.4
Fraction of bits flipped
Fraction of bits flipped
Fraction of bits flipped
Figure 7: Heteroassociation enables recovery of sensory patterns under corruption and superpo- sition. A) Accuracy for denoising 60 different random binary patterns for different vector dimension D. The dotted line is the average similarity between the decoded and ground truth patterns. B) Same experiment as panel A, but with 210 different possible binary patterns. The accuracy is lower on average. C) Accuracy for denoising multiple patterns from a single input. This task is especially challenging. because sums of patterns combined this way interfere with each other retrieval (a phenomenon known as cross-talk noise). However, the compositional structure of our modular attractor network enables multiple patterns to be decoded with high probability.
memory suggesting that LEC provides the content of experiences to hippocampus [37], as well as with neuroanatomical evidence (38). Although the structure of the representations of those sensory patterns is unknown, it is theorized that HF is critical to sensory pattern completion (39
Consistent with this function, recent work 128 40] has proposed that heteroassociative scaffold connects sensory patterns to hippocampal activity, allowing robust denoising of sensory states. Though the main focus of our normative model is not sensory denoising, we show that a simple extension to our model (Appendix [B.4) robustly retrieves noisy pattern even under high levels of corruption (Figures 7 and B). In Appendix C.3 we also discuss how this capacity for generalization can serve model for sequence retrieval and show some preliminary experiments.
model is also well-equipped to deal with In addition to robust denoising of single patterns, compositions of sensory patterns. Two situations are worth emphasizing: first, we can often unmix multiple sensory states corresponding of binding between grid modules protects the items in summation (Figure This differentiates model from other heteroassociative memories, in which sums of patterns would have multiple equally valid yet incompatible decodings. Second, the context vector module can separate the sensory information corresponding to different environments (Figure S3
of patterns, because the compositional structure
Discussion
There are by now numerous theories the entorhinal cortex and hippocampus, including those that draw upon attractor dynamics and residue number systems. What this paper contributes to the existing body of work is a concrete set of design principles that can be brought together to build a functioning neural system capable of representing space and performing path integration, making the most of limited neural resources and precision. In particular, a core design principle of this model is a compositional representation space that achieves superlinear coding range, which achieved by a compact, multi-module attractor network. The compositional representation, in turn, is achieved via vector binding operation, which enables binding multiple scales (moduli) and spatial dimensions, context, and spatial shifts for path integration. This binding mechanism builds on prior work in the
field of hyperdimensional computing and vector symbolic architectures [T] 17 26 (3) and goes beyond it to develop specific algorithmic hypothesis about structured operations in HF. Our analyses and experiments confirm that model can achieve important functions the hippocampal formation and they explain experimental observations, such as hexagonal grid cells, place cells, and remapping phenomena. The model thus contributes to, and greatly benefits from, existing work in theoretical neuroscience on residue number systems 5]. continuous attractor network models

---

of grid cells 44 32 44 models of compositionality in the hippocampal formation
45
and the
optimality of hexagonal representations in 2D
[T8] 29
That biology organized grid cells into multiple discrete modules, rather than pooling all resources
into single module attractor network, has posed long-standing puzzle to theoreticians: What advantages are conferred by this organization? Our answer is that it provides exponential scaling in dynamic range by combining modules with limited dynamic range multiplicatively. Other recent work has focused on the problem of coordinating representations across multiple modules [28] 34
46 and large scale recordings of the hippocampal formation 49 may provide new opportunities to evaluate their resulting predictions.
Our approach starts from principles of space encoding, in particular, the requirement of composition- ality. This strategy is complementary to investigations of the emergence of place and grid cells in artificial neural networks (e.g.. [36] 57)) These approaches can show the optimality of biological response features under the model assumptions, such as ANN properties, network architecture, train- ing objective and protocol. Here, we emphasize the role of multiplicative binding, computational primitive that is typically difficult to have emerge in an ANN setting. Early suggestions for realizing conjunctive binding already ventured outside the framework of ANNS [3] A simple extension of
ANNs are sigma-pi neurons [58] 59) that can implement vector binding [60]. Recent work amplifies
the view that full conjunctive binding would be a useful inductive bias to augment deep learning architectures [61], and various augmentations of ANNs with dedicated binding mechanisms have been proposed
62, 63).
Our model has clear limitations. The attractor neural network for the cognitive map is still a high-level abstraction of spiking neural circuits in the hippocampal formation. In partieular, the phasor states in the model are one linear transform removed from vectors that describe neural population activity. Thus, the mapping between model and neurobiological mechanisms requires an additional step. This disadvantage can be directly addressed by switching to other encoding schemes, such as sparse real- or compiex-valued vectors, e.g., (64). for which conjunctive binding operations have been proposed 665 66). Although the model is more comprehensive than typical normative models, which usually focus on single computation, it far from covering the many other functional cell types observed in the hippocampal formation or contextual modulations observed during remapping. In addition, the current model includes learning only in the heteroassociative projection to LEC. Most observations regarding plasticity in HF are not captured, i.e., signals from reward, or eligibility traces. Finally, our assumptions about inputs to HF from the sensory pathway are simplifying and primarily intended as proof of concept.
The purpose of the model, express the fundamental principles of a compositional cognitive map, also leads to testable predictions: First, at the biophysical level, the model predicts multiplicative interactions between dendritic imputs providing the conjunctive binding operation. There are several biophysically realistic ways in which neurons can multiply their inputs [67] 68). Contextual gating in dendritic branches of hippocampal neurons is consistent with our theory, hippocampal remapping, and neurophysiology of hippocampal dendrites (27). Our attractor model predicts direct multiplicative interactions between MEC modules, which remains to be tested. Second, the model predicts relatively fixed attractor weights between place and grid cells, with a higher degree of plasticity for the weights between sensory encodings and hippocampal states. Third, we predict that causal perturbations of one grid module can affect the states of other grid modules without involvement of the hippocampus, in direction that is self-consistent with the update of the attractor state.
Finally, we believe that the proposed modeling approach and the specific attractor model presented have broader applications in neuroscience. For example, the problem of factorization is critical to forming compositional representations of visual scenes, and a closely related attractor neural network can find efficient solutions to such problems (69]. In addition, there are promising ways to map complex- complex-valued valued attractor neural networks to spiking neural networks (70 71), which could connect the principles derived here to a concrete implementation on neuromorphic hardware. Such a neuromorphic implementation could yield further quantitative predictions for neuroscience and is an exciting direction for future work.
10

---

Acknowledgments
The work of CJK was supported by the Department of Defense (DoD) through the National Defense Science & Engineering Graduate (NDSEG) Fellowship Program. The work of SM was carried out as part of the ARPE program of ENS Paris-Saclay and supported by the European Union's 's Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement HORIZON- SURRA-2222SSERV-BA
The work of DK and BAO was supported part by Intel's THWAI program. The work of CJK and BAO was supported by the Center for the Co-Design of Cognitive Systems (CoCoSys), one of seven centers JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, well as NSF awards 2147640 and 2493 DK has received funding from the European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cune grant agreement No 839179. FTS discloses support for the research of this work from NIH grant IR01EB026955-0.
References
[1] H. Eichenbaum, "On the integration of space, time, and memory," Neuron, vol. 95, no. 5, pp. 1007-1018, 2017.
[2] E. Moser, M.-B. Moser, and B. L. McNaughton, "Spatial representation in the hippocampal formation: A history," Nature Neuroscience, vol. 20, 11, pp. 1448-1464 2017.
[3] Z. Kurth-Nelson et al., "Replay and compositional computation." Neuron, vol. 111, no.
pp. 454-469, 2023.
[4] I. R. Fiete, Y. Burak, and T. Brookings, "What grid cells convey about rat location," Journal of Neuroscience, vol. 28, no. 27, pp. 6858-6871, 2008.
[5] S. Sreenivasan and I. Fiete, "Grid cells generate an analog error-correcting code for singularly precise neural computation." " Nature Neuroscience, vol. 14, no. 10, pp. 1330-1337, 2011. [6] T. F. Behrens et al., "What is cognitive map? Organizing knowledge for flexible behavior," Newron, vol. 100, no. 2, pp. 490-509, 2018
[7] H. L. Garner, "The residue number system," in Western Joint Computer Conference (WJCC), 1959, pp. 146-153.
[8] O. Goldreich, D. Ron, and M. Sudan, "Chinese remaindering with errors," in Annual ACM symposium on Theory of Computing (STOC), 1999, pp. 225-234.
[9] Stensola, T. Stensola, T. Solstad, K. Frøland, M.-B. Moser, and E. I. Moser, "The entorhinal grid map is discretized," Nature, vol. 492, no. 7427, pp. 72-78, 2012.
[10] A. Rahimi and B. Recht, "Random features for large-scale kernel machines," in Advances in Neural Information Processing Systems (NeurIPS), vol. 20, 2007.
[11] T. A. Plate, "Holographic recurrent networks," in Advances in Neural Information Processing Systems (NeurIPS), vol. 5, 1992.
[12] C. von der Malsburg, "Am thinking assemblies?" Brain Theory, 1986, pp. 161-176.
[13] J. A. Fodor and 7. W. Pylyshyn, "Connectionism and cognitive architecture: A critical analysis," Cognition, vol. 28, no. 1-2. pp. 3-71, 1988.
[14] Greff, Van Steenkiste, and J. Schmidhuber, "On the binding problem in artificial neural networks," arXiv:2012..05208 2020.
[15] P. Smolensky, "Tensor product variable binding and the representation of symbolic structures in connectionist systems," Artificial Intelligence, vol. 46, pp. 159-216, 1990.
[16] T. Plate et al., "Holographic reduced representations: Convolution algebra for compositional distributed representations." International Joint Conference on Artificial Intelligence (IJ- CAI), 1991, pp. 30-35.
[17] P. Kanerva, "Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors," Cognitive Computation, vol. 1, pp. 139- 159, 2009.
[18] A. Mathis, M. B. Stemmler, and A. Herz, "Probable nature of higher-dimensional symme- tries underlying mammalian grid-cell activity patterns," Elife, vol. 4, e05979, 2015.
[19] X.-X. Wei, J. Prentice, and V. Balasubramanian, "A principle of economy predicts the func- tional architecture of grid cells," Elife, vol. 4, e08362, 2015.

---

[20] F. Anselmi, M. M. Murray, and B. Franceschiello, "A computational model for grid maps in neural populations," " Journal of Computational Neuroscience, vol. 48, pp. 149-159, 2020.
[21] J. J. Hopfield, "Neural networks and physical systems with emergent collective computational abilities.." Proceedings of the National Academy of Sciences, vol. 79, no. 8, pp. 2554-2558, 1982.
[22] A. Noest, "Phasor neural networks," in Advances in Neural Information Processing Systems (NeurIPS), 1987. pp. 584-591.
[23]
Noest, "Discrete-state phasor neural networks," " Physical Review A, vol. 38, no. 4, p. 2196, 1988.
[24] E. P. Frady, S. J. Kent, B. A. Olshausen, and F. T. Sommer, "Resonator networks, 1: An efficient solution for factoring high- high-dimensional, distributed representations of data structures," Neural Computation, vol. 32, no. 12, pp. 231 1-2331, 2020.
[25] S. J. Kent, E. Frady, T. Sommer, and B. Olshausen, "Resonator networks, 2: Factoriza- tion performance and capacity compared to optimization-hasee methods," Neural Computation, vol. 32, no. 12, pp. 2332-2388, 2020.
[26] C. J. Kymn et al., "Computing with residue numbers in high-dimensions representation." arXiv 2311.04872, 2023.
[27] P. Latuske, O. Kornienko, L. Kohler, and Allen, "Hippocampal remapping and its entorhinal origin," Frontiers in Behavioral Neuroscience, vol. 11, p. 253, 2018.
(28) S. Chandra, S. Sharma, R. Chaudhuri, and I. Fiete, "High-capacity flexible hippocampal associative and episodic memory enabled by prestructured "spatial" representations, " bioRxiv, 2023.
[29] A. Mathis, A. V. Herz, and M. B. Stemmler, "Resolution of nested neuronal representations can be exponential in the number of neurons," Physical Review Letters, vol. 109, no. 1, p. 018 103, 2012.
[30] E. Landau, "Über die maximalordnung der permutationen gegebenen grades," Archiv der Mathematik und Physik, vol. 3, pp. 92-103, 1903.
[31] T. Hafting, M. Fyhn. S. Molden. M.-B. Moser, and E. I. Moser, "Microstructure of spatial map in the entorhinal cortex," Nature, vol. 436, no. 7052, pp. 801-806, 2005.
[32] M. C. Fuhs and D. S. Touretzky, "A spin glass model of path integration in rat medial entorhinal cortex," " Journal of Neuroscience, vol. 26, no. 16, pp. 4266-422777 2006.
[33] Y. Burak and I. R. Ficte, "Accurate path integration in continuous attractor network models of grid cells," PLoS Computational Biology, vol. 5, no. 2, e1000291, 2009.
[34] N. Mosheiff and Y. Burak, "Velocity coupling of grid cell modules enables stable embedding of a low dimensional variable in a high dimensional neural attractor," Elife, vol. 8, e48494,
2019.
[35] F. Raudies and M. E. Hasselmo, "Modeling boundary vector cell firing given optic flow as cue." PLoS Computational Biology, vol. 8, no. 6, e e1002553, 2012.
[36] A. Banino et al., "Vector-based navigation using grid-like representations in artificial agents," Nature, vol. 557, no. 7705, pp. 429-433, 2018.
[37] J. R. Manns and H. Eichenbaum, "Evolution of declarative memory," Hippocampus, vol. 16, no. 9. pp. 795-808, 2006.
[38] J. J. Knierim, J. P. Neunuebel, and S. S. Deshmukh, "Functional correlates of the lateral and medial entorhinal cortex: Objects, path integration and local-global reference frames," Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 369, no. 1635, p. 20 130 369, 2014.
[39] T. J. Teyler and P. DiScenna, "The hippocampal memory indexing theory." Behavioral Neuro- science, vol. 100, no. 2, pp. 147-154, 1986.
[40] S. Sharma, S. Chandra, and I. Fiete, "Content addressable memory without catastrophic forgetting by heteroassociation with a fixed scaffold," in International Conference on Machine Learning (ICML), 2022, pp. 19 698-199688 682.
[41] R. W. Gayler, "Vector symbolic architectures answer Jackendoff's challenges for cognitive neuroscience," in Joint International Conference on Cognitive Science (ICCS/ASCS), 2003,
pp. 133-138.
12

---

[42] D. Kleyko, D. Rachkovskij, E. Osipov, and A. Rahimi, "A survey on hyperdimensional computing aka vector symbolic architectures, part ii: Applications, cognitive models, and challenges." ACM Computing Surveys, vol. 55, no. 9, pp. 1-52, 2023.
[43] N. S.-Y. Dumont, J. Orchard, and C. Eliasmith, "A model of path integration that connects neural and symbolic representation," " in Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 44, 2022.
[44] K. Zhang. "Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: A theory." Journal of Neuroscience, vol. 16, no. 6, pp. 2112-2126, 1996. [45] D. C. McNamee, K. L. Stachenfeld, M. M. Botvinick, and J. Gershman, "Flexible modulation of sequence generation in the entorhinal hippooampa system." " Nature Neuroscience, vol. 24,
no. 6. pp. 851-862, 2021.
[46] N. Mosheiff, H. Agmon, A. Moriel, and Y. Burak, "An efficient coding theory for dy- namic trajectory predicts non-uniform allocation of entorhinal grid cells to modules," PLoS Computational Biology, vol. 13, no. 6, e e1005597, 2017.
[47] L. Kang and V. Balasubramanian, "A geometric attractor mechanism for self-organization of entorhinal grid modules," Ellife, vol. 8, e46687, 2019.
[48] H. Agmon and Y. Burak, "A theory of joint attractor dynamics in the hippocampus and the entorhinal cortex accounts for artificial remapping and grid cell field-to-field variability," Elife, vol. 9, e56894, 2020.
[49] T. Waaga ef al., "Grid-cell modules remain coordinated when neural activity is dissociated from external sensory cues," " Neuron, vol. 110, no. 11, pp. 1843-1856, 2022.
[50] C. J. Cueva and X.-X. Wei, "Emergence of grid-like representations by training recurrent neural networks to perform spatial localization," in International Conference on Learning Representaations (ICLR), 2018, pp. 1-19.
[51] J. C. Whittington, W. Dorrell, S. Ganguli, and T. Behrens, "Disentanglement with biological constraints: A theory of functional cell types," in International Conference on Learning Representations (ICLR), 2022.
[52] W. Dorrell, P. E. Latham, T. E. Behrens, and J. C. Whittington, "Actionable neural repre- sentations: Grid cells from minimal constraints," in International Conference on Learning Representations (ICLR), 2023.
[53] B. Sorscher, G. C. Mel, S. A. Ocko, L. M. Giocomo, and S. Ganguli, "A unified theory for the computational and mechanistic origins of grid cells," " Neuron, vol. 111, no. 1, pp. 121-137, 2023.
[54] R. Schaeffer, M. Khona, T. Ma, C. Eyzaguirre, S. Koyejo, and I. Fiete, "Self-supervised learning of representations for space generates multi-modular grid cells," in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023.
[55] K. L. Stachenfeld, M. M. Botvinick, and S. J. Gershman, "The hippocampus as a predictive map," Nature Neuroscience, vol. 20, no. 11, pp. 1643-1653, 2017.
[56] J. C. Whittington el al., "The -iiaan-Eihhenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation," " Cell, vol. 183, no. 5, pp. 1249- 1263, 2020.
[57] Y. Chen, H. Zhang, M. Cameron, and T. Sejnowski, "Predictive sequence learning in the hippocampal formation." " bioRxiv, 2022.
[58] J. A. Feldman and D. H. Ballard, "Connectionist models and their properties." " Cognitive Science, vol. 6, no. 3, pp. 205-254, 1982
[59] B. W. Mel and C. Koch, "Sigma-Pi learning: On radial basis functions and cortical associative learning," in Advances in Neural Information Processing Systems (NeurIPS), 1989, pp. 474- 481.
[60] T. A. Plate, "Randomly connected Sigma-Pi neurons can form associator networks," Network:
Computation in Neural Systems, vol. 11, no. 4, p. 321, 2000.
[61] A. Goyal and Y. Bengio, "Inductive biases for deep learning of higher-level cognition." Pro- ceedings of the Royal Society A, vol. 478, no. 2266, 2022.
[62] I. Danihelka, G. Wayne, B. Uria, N. Kalchbrenner, and A. Graves, "Associative long short-term memory." " in International Conference on Machine Learning (ICML), 2016, pp. 1986-1994. [63] Ganesan el al., "Learning with holographic reduced representations," in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021, pp. 25 606-25 620.
13

---

[64] M. Laiho, J. H. Poikonen, P. Kanerva, and E. Lehtonen, "Higg-diimnnioonll computing with sparse vectors," in 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS), IEEE, 2015, pp. 1-4.
[65] D. A. Rachkovskij and E. M. Kussul, "Binding and normalization of binary sparse distributed representations by context-dependen thinning," Neural Computation, vol. 13, no. 2, pp. 411-
452, 2001.
[66] E. Frady, D. Kleyko, and F. T. Sommer. "Variable binding for sparse distributed representa- tions: Theory and applications," IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. pp. 2191-2204, 2023.
[67] C. Koch, Biophysics of computation: information processing in single neurons. Oxford univer- sity press, 2004.
[68] L. N. Groschner, J. G. Malis, B. Zuidinga, and A. Borst, "A biophysical account of multiplica- tion by a single neuron," Nature, vol. 603, no. 7899, pp. 119-123, 2022.
[69] C. J. Kymn, S. Mazelet, A. Ng. D. Kleyko, and B. A. Olshausen, "Compositional factorization of visual scenes with convolutional sparse coding and resonator networks," in 2024 Neuro Inspired Computational Elements Conference (NICE), IEEE, 2024, pp. 1-9.
[70] E. P. Frady and F. T. Sommer, "Robust computation with rhythmic spike patterns," Proceedings of the National Academy of Sciences, vol. 116, no. 36, pp. 18 050-18 059, 2019.
[71] J. Orchard, P. M. Furlong, and K. Simone, "Efficient hyperdimensional computing with spiking phasors," Neural Computation, vol. 36, no. 9, pp. 1886-1911, 2024.
[72] T. A. Plate, Holographic Reduced Representation: Distributed representation for cognitive
structures. CSLI Publications Stanford. 2003, vol. 150.
[73] A. Thomas, S. Dasgupta, and T. Rosing, "A theoretical perspective on hyperdimensional
computing," " Journal of Artificial Intelligence Research, vol. 72, pp. 215-249, 2021.
[74] E. P. Frady, D. Kleyko, C. J. Kymn, B. A. Olshausen, and F. T. Sommer, "Computing on functions using randomized vector representations (in brief)," " in Annual Neuro-Inspired
Computational Elements Conference (NICE), 2022, pp. 115-122.
[75] K. L. Clarkson, S. Ubaru, and E. Yang. "Capacity analysis of vector symbolic architectures," " arXiv preprint arXiv:2301. 10352, 2023.
[76]
J. O. Smith, Spectral Audio Signal Processing. 2011.
[77] B. Komer and C. Eliasmith, "Efficient navigation using a scalable, biologically inspired spatial representation," in Annual Meeting of the Cognitive Science Society (CogSci), 2020, pp. 1532- 1538.
[78] B. Komer, "Biologically inspired spatial representation," Ph.D. dissertation, University of Waterloo, 2020.
[79] E. P. Frady, D. Kleyko, and F. T. Sommer, "A theory of sequence indexing and working memory in recurrent neural networks," Neural Computation, vol. 30, no. 6, pp. 1449-1513, 2018.
[80] D. Kleyko el al., "Efficient decoding of compositional structure in holistic representations," Neural Computation, vol. 35, no. pp. 1159-1186, 2023.
[81] T. Wills, C. Lever, Cacucci, N. Burgess, and O' Oeeeff, "Attractor dynamics in the hippocampal representation of the local environment," Science, vol. 308, no. 5723, pp. 873- 876, 2005.
[82] L. Thompson and P. Best, "Place cells and silent cells in the hippocampus of freely-behaving rats," " Journal of Neuroscience, vol. 9, 7, pp. 2382-2390, 1989.
[83] A. O. Constantinescuu, J. X. O'Reilly, and T. E. Behrens, "Organizing conceptual knowledge in humans with gridlike code," Science, vol. 352, no. 6292, pp. 1464-1468, 2016. [84] J. L.. Bellmund, P. Gärdenfors. E. I. Moser, and C. F. Doeller, "Navigating cognition: Spatial codes for human thinking." " Science, vol. 362, no. 6415, 2018
[85] M. I. Schlesiger et al., "The medial entorhinal cortex is necessary for temporal organization of hippocampal neuronal activity." Nature Neuroscience, vol. 18, no. 8, pp. 1123-1 172--1132 132, 2015.
[86] J. Yamamoto and S. Tonegawa, "Direct medial entorhinal cortex input to hippocampal CAI is crucial for extended quiet awake replay." Neuron, vol. 96, no. 1, pp. 217-227, 2017.
[87] X. Li and P. Li, "Quantization algorithms for random fourier features," in International Conference on Machine Learning, PMLR, 2021, pp. 6369-6380.
14

---

Supplemental material
A Mathematical derivations
A.1 Similarity-preserving properties of embeddings
In the following section, we examine the similarity properties our coding scheme. Recall from Section 2.1 that our crucial desiderata are that: (1) distinct residue values are represented using vectors which are nearly orthogonal, and that (2) the inner product between representations of sub-integer values are reflective of a reasonable notion of similarity between the encoded values. There is robust literature on this topic both within the Vector Symbolic Architectures community 75 and the broader ML community [10), who often study these techniques under the name "random features." " The methods pursued here fall under these traditions.
To briefly recapitulate the construction of Equation fix some positive integer m. and let P(k) denote the uniform distribution over (0...., 1}. Define an embedding - CD using the following procedure: draw ki,.... KD independently from P(k). and set:
g(a), = exp (iwks)) j=
where = 2m/m, and i 1 To simplify amalysis, here assume that is odd, in which case the above is equivalent to shifting the support of P(k) to {-(m 1)/2...., (m - -1)/2}, and defining the embedding g R C component-win siiiee via:
g(a); = exp (iwk,a) DD,j=l,.....
The case that m is even is slightly different, but can be handled using similar techniques and the discrepancy does not affect any of our modeling goals.
Our basic claim is that expectation with respect to randomness in the draw ki, kp inner- products between the embeddings of two numbers a, a' recover the periodic sinc-function (76) of their difference. That is:
(in(T(a d')) mss((Traa - a')/m)
E (lg(a) (a((a)g((')) gg(')"]=
psinc(a a');
This accomplishes goal (1) because, for f integer which is not integer multiple of m. psinc(t) 0. Therefore, distinct integers are represented using vectors which are, in expectation, orthogonal. It also accomplishes goal (2), because psinc(t) Rs for 0 < |t] 1. The following theorem demonstrates this property more formally, and provides an approximation guarantee for a specific instantiation of kj,..., kD-
Theorem 1. Fix any D > and E (0, 1). For any pair a, a' E R such that a a' is not an integer multiple of m, with probability at least 1 over randomness in the draw ki, ... Kn:
sin (a)a) sin( al) a')/m)
2
g(a) g(a')*
ln
8
Proof. Fix any pair a, E R. and denote for concision /. aaa'. Taking an expectation with respect to randomness in k ..., kD and using well-known calculation from the signal processing

---

literature [6):
E ka ka (B(a)) g((')*) DBR [g(a)ig(a')1)
exp (iwkk) (a - a'))
m
k
iwt(m-1)
[wl(m+1]
1
exp
exp
m
exp((wwt)
exp(iwt/22
exp(- -spp exp(Tri)
m mxxx(iiw//2 sin(-nt)
exp( (o(--iil/m) exp(xit/m))
m sin (mTT/mmm)
sin( a'))
m si (ar(a-a')) a()/m)
The third equality follows from the second by noting that the latter is a sum geometric series with common ratio exp(wtt). The fifth line follows from the fourth by recalling the identity sin(2) (2) = (e -
e2) //i. In the limit of - 0, the expression evaluates 1, consistent with the normalized inner product of a vector with itself.
To show concentration around this value, consider:
D Σ 5=1
g(a) T g((")"
expliwky dd)),
and note that since the complex part the sum vanishes expectation, we may consider, without loss of generality, the average the real-valued quantities: (cos(wki (a - dd)))²=1+ which are bounded in the range 1. Therefore, by Hoeffding's inequality:
De
Pr (1B(a) g(a')* Elg(a) g(aa)' 2)) 2exp
whereupon we conclude that, with probability at least 1 over randomness the draw of ki,... kp:
2 In
S D
5'
as claimed.
This result can be readily extended to the binding of multiple residue number values. Let g(a) O (a)), where each (a( instantiated independently. Then, by independence, observe
K
that:
K Il (a) 8i(d')' )
E (g(a) g(d')' [( E
II (a) ga(d')")
The implication is that E[g(a)T g(a')) i and only if all residue values agree, and zero otherwise. To show concentration around this value, we can again use Hoeffding's inequality, which recovers the same bound on the sufficient dimension.
A.2 Informatiion-theoreii estimate of required pattern dimension
In this section, deseribe an information thhorreti estimate the dimension D necessary to retrieve patterns within K modules. The main result we aim to show is that D (n

---

equivalently, the scaling of n for a given D is O(DK)(K-)), This scaling roughly predicts our empirical results of finding the dimension required to achieve high accuracy, suggesting that the attractor network described here performs close to the theoretical bound.
The minimal total amount of information network needs to store for denoising an RNS repre- sentation with coding range M is (M log(M)). This results from the requirement of content addressability, i.e., for serving as a unique pointer one of n patterns, each pattern must at least carry information of the order of Ollog(M)) (M)). For simplicity, we now assume that each module is of size O(M1/K),
). The total capacity of the network is bounded by the number of synapses, which is O(D * K * M = O(D M (assuming K is constant), times the capacity per synapse. Under the conservative assumption that the capacity per synapse is (1), the dimension is of order (e "x log HM)) (log () Thus, the leading order of how D depends on n is O ((M((K11)/ If the capacity per synapse is assumed to be larger, O(log (M)) bits, only the non-leading term cancels and the resulting order of D is still the same.
A.3 Construction of triangular frames
In order to convert a 2D coordinate into 3D frame y, we first multiply it by matrix, v whose rows are the elements of a 3D equiangular frame:
1/ V3
-1 1/3
1/v3
-1 1/3
x
(SI)
0
2/3
(This particular frame is commonly referred to 'Mercedes Benz' frame due to its resemblance to the iconic symbol.) A consequence of working with an overcomplete frame is that there may exist multiple values of y that correspond to the same x. For this frame, the null space of v is the subspace spanned by [1, 1, 1JT grounding the intuition that equal movement in all equiangular directions "cancels out." It therefore might seem that triangular frames require extra operations to determine if two coordinates are equal, but here we show how to avoid this consequence.
The core strategy is to choose seed vectors g 2i gi.3 for each modulus m, that implement this self-cancellation. For a modulus m,, we draw the phasors of seed vectors from the m-th roots of unity. However, further require that, for each vector component, the three selected phases 0 (mod 2Tr.. We then form a hexagonal coordinate vector by binding the three seed vectors:
g: gi,
g gi,3
(22)
By enforcing that the phases sum to (mod 2T), we ensure that positions that have an equivalent x coordinate are mapped to the same g:. Observe that Hadamard product binding of phasors is equivalent to summing their phases, and that binding e corresponds to adding nothing. Hence, a pair of three-dimensional coordinates whose differences are multiple of [1. 1, 1] will be mapped to equivalent vector representations. Finally, we then form the residue number representation for different moduli by binding, as in Eq. 2 The presence of multiple modules and self-cancellatioo properties complement prior work on the efficiency of hexagonal kernels for spatial navigation
tasks 777 78).
The equivalence of certain 3D coordinates also helps us count the number of states. Clearly, the redundancy means that we have less than m² states, but it also shows us that every position in the hexagonal grid can be represented by 3D coordinate which contains at least one coordinate equivalent to O. There is one state where all coordinates are 0, 3(m 1) states where exactly two coordinates are O, and 3(m - 1)² states where exactly one coordinate is zero. Thus, there are 3m² 3m + 1 states for the hexagonal lattice, compared to the m² states for the square lattice.
In the case of square lattices in 2D, all states occupy an equal proportion of space; however, this is not the case for the hexagonal lattice (see Figure (AA). This is because states with more gero-valued coordinates occur slightly more frequently. To estimate the effect of unequal proportions on the entropy, we directly calculate the Shannon entropy of hexagonal lattices for finite size spatial grids of increasing radius I, as an approximation to the infinite lattice. We find that even for / = 1000, m > 7 the hexagonal code has 99 percent of the entropy of a system that divided all possibilities equally, and that this gap decreases as m grows larger. Thus asymptotically, as m - 20, the ratio of entropy for hexagonal vs. square grids tends towards log, (3).
3

---

B Experimental details
All experiments were implemented in Python involving standard packages for scientific computing (including NumPy, SciPy, Matplotlib). We describe here the parameters and training setup of our experiments in further detail.
B.1 Scaling in dimension
For each number of moduli, K, we seek to find the smallest dimension for which our attractor model factorizes its input, into the correct grid states in a fixed time (50 iterations) with high probability (at least percent empirically). In instances where the network states remain similar over time (at least 0.95 cosine similarity), we consider that it converged to a fixed point. If such convergence did not occur, we evaluate the accuracy at the last time step.
To evaluate scaling, we first choose our base moduli to be set of K consecutive primes. We randomly select one of M random numbers serve the input and set the grid states be random. We then evaluate a candidate dimension on the factorization task for a set number of trials (200) and check accuracy. We compare accuracy by considering whether the amplitude of the complex-van -aluued-valued inner products are highest for the true factor. If the accuracy is above our threshold, we then evaluate performance of slightly higher dimension (dimensions evaluated are spaced apart on a logarithmic scale). Once sufficiently high dimension achieves the accuracy threshold, we assume that the scaling non-decreasing and use the last successful dimension as the first try.
Finally, we fit linear regression to all data points on a log-log scale to estimate the scaling between dimension and problem size. We report the slopes to estimate the scaling coefficients.
B.2 Error correction
General experimental setup. We in advance the vector dimension, noise level (determined by 1 1/K), and number of moduli. Given these parameters, we estimate the empirical accuracy of factorization on an arbitrary input known correspond to one of the patterns. We use the same method for checking convergence as above, though we increase the maximum number of iterations to 100. For all experiments in this section, we average over 1,000 trials.
In the case of input noise, the vector p is multiplied by a noise vector. In the case of update noise, after every time step, each module of the attractor network is corrupted by von Mises noise update. In the case of codebook noise, all codebooks are corrupted before the start of any iterations.
Decoding values between integers. In order to test the ability of the modular attractor network to decode at sub-integer resolution, we fix a spatial resolution Ar to decode from. In our experiments, test Ar (1/3, 1 1/7, 1/31), and also report Ax 1 (integer decoding) as control. Then, using as input random integer and random multiple of Ar, we let the modules of the attractor network settle until convergence (as in other experiments). To evaluate accuracy, we test if the resulting output of the attractor network, o.g,<1 (t), is closer to the ground truth RNS representation than to any other value. We test this with a "coarse-to-fine approach: first checking if it within an integer, and then checking all fractional values within one of that integer. We regard the output as correct if both the integer and fraction match, and incorrect otherwise.
Estimation of information content from a vector. To measure the total resolution of our coding scheme in bits, we factor in both the number of states distinguished (7 M and the empirical accuracy (p). To quantitatively estimate this, we report the information decoded in bits according to the following equation 79 80
(--Ta-))
1 [T, =a (Tp) + (1 p) log,
(S3)
A consequence of this equation is that the information decoded is 0 when the empirical accuracy is at chance (1/7).
B.3 Path integration
General experimental setup. We generate paths using a statistical model simulating rodent two-
dimensional trajectories in 50 cm² closed square environment 35 36), with At 100 ms. The

---

path integration method starts from the ground truth first position (zo. yo) which is converted to hexagonal coordinates (an. Cg) (see Section A.3) and encoded as an RNS representation p(0) of dimension D = 3,000 following the method in Section 2.1 for moduli {3. 5, 7). We then factorize p(0) into (0) to produce the estimated representation ((0)
O K (i()).
At each time step 2 O, we estimate the position (Fe+11 3/++1)) We give the modular attractor network as input the previous position vector estimate p((). It is factorized into the residue components
{R)(t) that are then shifted according to the velocity (da,, db,, des) between (a,. be. qt) and (0(+1, b C441). Namely, for each residue module, we build velocity vector (t) = gj.1 (da(t))
())(()(tt) O g333 (g.((d((t) that is binded to each residue component E3 (t). The estimated position
vector is then the binding of the shifted estimated residue components: 1(++1) = O K j=1 8) (4))) The estimated position (. 11+1) is chosen to be the position (x, y) in a grid of 30 x 30 positions mapping the entire environment, corresponding to the highest similarity between p(x, and p(t+
We show the robustness of the path integration dynamics to two different sources of noise. In the case of extrinsic noise (Figure the hexagonal velocity is corrupted by additive Gaussian noise of variance 0. In the case of intrinsic noise (Figures 6/A and B), the position vector P. is corrupted by binding with vector sampled from a von Mises distribution with concentration parameter
2.
Response field visualization. Given moduli m. and a vector gi, we visualize its response field by computing the similarity of the modular attractor output (t) and g; along a trajectory. The periodicity in the distribution of random weights and the hexagonal coordinates produce periodic hexagonal receptive fields whose scale depends on m, Since the inner product between vector states induces a caanlltton-invaariant invariant kernel, the response fields for given moduli are translations of each other.
Connection to sensory cues. Sensory cues are random binary vectors of size N D) that associated with positions along the trajectory. When the true trajectory reaches a sensory cue, the hippocampal state Pe is updated using the heteroassociation method described in Appendix B.4
B.4 Heteroassociation
General experimental setup. We evaluate our model's performance for pattern denoising using a heteroassociative learning rule (28) 40). We consider random binary patterns of size N, D. We corrupt the patterns by randomly flipping bits with probability Pfip E [o, 0.3] and associate them to place cell representations using heteroassociation with pseudo-inverse learning rule. Let E XM be the matrix of M patterns to hook to the scaffold and H E CM the matrix of M position vectors on which to hook the patterns. We associate pattern to place cell representation p- = HSS+s, s, where St is the pseudo-inverse of S. The model returns a denoised place cell representation p from which we can estimate denoised pattern by inverting the heteroassociation projection $ = sgn (SH+ p). Examples of corrupted inputs and reconstructed patterns are shown in Figure $3
Scaling with dimension. We evaluate the impact that the dimension D has on the denoising per- formance in Figure 7 for number of stored patterns M 60 (in this case, 3 x 4 x 5) and 210 (in this case, 5 6x6x7 7). For each dimension D E {256, 512, 1024, 2048]. we show the evolution of accuracy for different levels of corruption. For a given dimension D and noise level Pilpp. we denoise a pattern and consider that the denoising is correct if the denoised pattern is closest to the ground truth pattern (in terms of cosine similarity). We repeat over 500 trials and report the accuracy as well as the average similarity (normalized inner product) between the denoised pattern and its noiseless version.
Superposition of patterns. We show that our model can denoise a superposition of n, patterns one at time, for np E 2,, 3.4 4, 5, 10). We fix the dimension D to 2,000 and for different values of bit flip probability Рmp E [0, .... 0.5], we run the model on a superposition s of random binary patterns {Si...., ..., Sh., } of size N. between each run the denoised pattern is explained away from the superposition [69]. Namely, for run E (1, (1....,?l, Np - 1} we denote S(r) the denoised pattern. The input to run r 1 is then +(r+1) = s(r) - -(()). We find that the more patterns are superposed, the lower the overall denoising accuracy is. This is due to the fact that when a pattern is incorrectly denoised, explaining away
2,000: s = S1 + .... + Sn,, We run the model n. times and
5

---

adds noise or spurious patterns to the representation of the superposition which makes the following denoising steps more difficult.
Comparison to structured patterns. We evaluate our model's ability to denoise structured patterns. We consider the FashionMNIST dataset, from which select 105 images of size 28 x 28 that binarize by setting pixel values to be -1 if below 127, and elsewhere. We compare the denoising performance the performance on random binary patterns of size 28 28 784 for fair comparison (Figure S4).
C Additional results
C.1 Further visualizations of grid cell modules
We further visualize the response fields for path integration by showing response fields from different units taken from the same grid module. We simulate a trajectory that traverses the entire environment and represent the activation of different position vectors along the trajectory. For each modulus my E {3,5, 7]. we show the similarity between different vectors gi from module m, and the position vectors along the trajectory. We show in Figure Sht the different receptive fields of given module are translations one another.
module
module
module
Figure S1: Response field visualization of 4 different g; in 3 different modules m, 3, 5 and For fixed module, the response fields appear as translated versions of one another.
C.2 Remapping via modulation of context
We demonstrate that the context vector can serve as a model of global remapping in hippocampal place fields, which occurs when different environments of cells [27]. The simplest instance of this is when a place field occurs in context but not context B, consistent with the observed sparsity of hippocampal activity [82. To model this kind of remapping phenomenon, we consider an instance where there is a gradation of contexts with some phase transition between them; such instance this end, we model linear combinations of these contexts, where the weights each context is given (z) with x varying from -5 to 5 in equally spaced increments, and with sz) 1/(1 + exp(-x)). To model hippocampal units, we generate units that prefer one of the contexts and have random place field location, using its weight vector, or address, as cO 1 gi and compare output to that of the context/grid system at each location and context. worth noting that the original experiment of (81) also exhibited instances of rate remapping for
encoded with different populations
studied experimentally (8T]. Towards
sigmoid(x))

---

Unit
Context
Figure S2: Remapping of place cells depending context. The global remapping observed in the model response fields is similar to the findings of an experimental study attractor network dynamics in hippocampus (8I).
some units, and there is certainly additional complexity underlying remapping that is not captured by simple model.
C.3 Storing and retracing sequences
We demonstrate that our model can recover sequences by heteroassociation of patterns positions and path integration in conceptual space (Figure (S5A). This is consistent with the postulated role of
the hippocampal formation in performing navigation in conceptual spaces [83,
[84.. and the role of
entorhinal cortex in generating sequences of neural firing in hippocampus [85] 86 To evaluate our
attractor model's s fidelity at sequence memorization and retrieval, we simulate trajectories to form sequences of random binary patterns and recall the sequence using the path integration mechanism following the method in Section 4.1 for = 10,000 and moduli (3, 5). add extrinsic noise to the velocity input, which accumulates along the trajectory and induces drift. This implies that patterns at the end of sequences are less well recovered than ones at the beginning (Figures S5B and C).
C.4 Further tests of model robustness
We find that the proposed modular attractor network also robust other sources of noise. In particular, evaluate robustness to synaptic noise, or dropout, decaying synaptic precision, and weight lesions (Figures S6A-C, respectively).
D Broader impacts
The results presented here primarily addressing fundamental research questions, suggesting computational mechanisms in the brain. These results could lead to experimental design that improves our understanding of circuits in the hippocampal formation, to artificial intelligence models capable of incorporating compositional structure in navigation tasks. Such impacts are typical of computational neuroscience research. On the other hand, point out that the explicit compositionality of our modeling approach provides transparency into its operations, which would reduce the risk of unforeseen consequences.

---

6
3
2
22
Inpul
3
3
8
1
f
4
4
4
0
0
0
4
6
O
0
6
0
6
0
O
6
O
O
10%
7
1
/
1
7
/
/
2
3-
2
2
2
a
2
2
2
2
2 3
2
Recovered
3
3
3
3
3
5
3
3 #
3
//
4
4
4
4
4
4
of
25
8
C
8
6
6
7
3
2
5
Input
0
O
6
O
20%
42
1
/
2
3-
2
22
3
2
2
2
2
2
Recovered
3
3
3
3
7/
3
3 4
3
4/
M
4
4
4
4
4
of
30% corruption
Input
0
6
O
0
8
6
6
0
5
4
3
1
6
1
8
4
7
E
3
%
Z
7
e
Recovered
3
3
3
5
A:
4 R
3
Figure S3: Examples sensory denoising with a heteroassociative memory on a binarized ver- sion of the MNIST Dataset. Here, different contexts are used to index particular digit patterns. The degree of corruption (shown as "Input") influences the success of denoising (shown as "Recovered."
0.8
Accuracy
0.6
0.4
Pocorrect basin) Random patierns
(correct basin) FashionMNIST
0.0
Fraction of bits flipped
Figure $4: A performance comparison for the heteroassociative memory random patterns versus a binarized version of the FashionMNIIS dataset. For different levels of corruption, we denoise flattened binarized FashionMNIST images as well and random binary vectors of the same size. The overall denoising accuracy lower for FashionMNIST, reflecting the difficulty in storing correlated patterns.

---

A
B
C
095
0.85
Aeguraey
Accuracy
0.55
0.85
0.80
noleeless
no soloossss
noley velocity 0.05)
no valocity (var 0.05)
0.75
notey velecity (var 0.06)
0.75
velocily 0.08)
10
20
25
10 15
20
25
30
Sequence position
Sequence position
Figure S5: Flexihle sequence retrieval via path integration in a conceptual space. A) An example of hexagonal lattice with sensory observations associated with different states. Having knowledge of the underlying graph enables generalization to new trajectories in the space [6 [6]. B) Accuracy of random binary pattern retrieval as function of position in the sequence for fixed error rate and one context tag. The noiseless case achieves perfect accuracy, but errors accumulate after incorrect sequence predictions. C) Same B), but with the additional task of inferring the context tag.
A
Robustness dropout
B
Effects of limited precision
Effects of ablation
1.0
1.0
codeword lesioned
0.8
0.8
0.8
module lesioned
Accuracy
0.6
Accuracy
0.6
prediion (D-512)
Accuracy
0.6
0.4
0.05
000000
0.4
bis (D=1004)
bis (D+1024)
(D-1024)
0.4
0.2
0.15
0.100
0.2
Ful prodsion (O-1084)
bat (D-2048)
bas (2-2044)
0.2
0.20
bis (D-2048)
0.0
0.0
Full prezzsin (0-2044)
0.0
102
10
10
10
10 Effective range
10
0.0
0.2
0.4
Effective range
% of weights lesioned
Figure 56: Robustness of attractor network additional sources noise. A) Robustness of the modular attractor network to synaptic failure (dropout). At each time step in the dynamics, each entry (weight value) in the matrices and G has an independent probability, p. of being set to In spite of this synaptic noise, the model empirically converges to the correct solution up slightly smaller coding range. B) Robustness the modular attractor network to limited precision. We use stochastic quantization, technique studied in random feature models in machine learning [87). to round our model down to 6 bits precision. We find empirically that 5 bits of precision (in this regime) performed nearly identically the full precision vectors, indicating diminishing returns for higher precision. On the other hand, increasing vector dimension, which also requires more memory, results in increased capacity without facing the same diminishing returns. C) The effects of lesions (setting weights to 0) on network performance. The curves and data points reflect averages over 1000 trials. The network uses vectors of dimension D) 1024, and has a dynamic range of M = 65231 (= 37 x 41 X 43). The gray curve shows the effect of lesioning random weights, each with independent probability p. Accuracy remains high up to small percentage of lesioned weights (less than 10 percent). The teal square shows performance after lesioning one random column of one column of random module's 's weights G,; the green hexagon shows effects of lesioning one random column for all three modules, and the purple star shows lesions to all codebooks in one module representing a non-zero residue. In each case, the performance worse than random lesions.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction stay within the bounds of what we introduce in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed explicitly in the Discussion (Section 5 as well as implicitly throughout the rest of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?

---

Answer: [Yes]
Justification:
The three technical sections, provided in Appendix A.1 Appendix A.2 and Ap-
pendix A.3 respectively, provide full proofs calculations and cite any required backgroun lemmas.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We explicitly wrote Appendix B to disclose any extra pieces of information required reproduce experimental results. We have also provided implementations in code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Open access the data and code is available at https: com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public//gudde//DdeSSutnsssssss wwt.aaammmiioooooooooooooooeeiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ips.cc/
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, these details are presented Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by error bars and confidence intervals when appropriate for our figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
3

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
performed CPU with local resources. We do have precise estimates of the amount of compute operations required, but each individual simulation took less than days of total compute time.
Justification: All experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ftttps://neuripar https////ppp//ctcccccc...e.eeel
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and confirm that conforms to all standards outlined.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix D
Guidelines:

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model does not involve any of examples listed, and does not require additional safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are owned and created by the authors unless explicitly stated otherwise.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
5

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper involves neither crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper involves neither crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "lz5BfGZidM", "reviewer": "Reviewer_METM", "summary": "This paper introduces a normative model for spatial representation within the hippocampal formation, integrating optimality principles with an algebraic framework. Spatial positions are encoded using a residue number system (RNS) and represented by high-dimensional, complex-valued vectors. These vectors are combined into a single vector representing position through a similarity-preserving, conjunctive vector-binding operation. The model incorporates a modular attractor network, mirroring the grid cell modules in the entorhinal cortex, to ensure self-consistency among these vectors. The paper showcases the model’s robustness, sub-integer resolution, and path integration capabilities through both theoretical analysis and experimental validation.", "questions": "1. How biologically plausible is the RNS as a coding mechanism in the hippocampal-entorhinal circuits? Are there any existing biological structures that directly support this model?\n2. What specific experiments could be designed to empirically test the predictions made by the model? How feasible are these experiments with current technology?\n3. The model suggests encoding contexts as vectors in the entorhinal cortex. How does it manage the vast diversity and complexity of possible contexts in real-world environments?\n4. While the model shows robustness to noise in simulations, how would it perform under the more complex and varied types of noise encountered in biological systems?", "limitations": "The model abstracts away many neurobiological details, focusing on high-level representations and processes. This could overlook important aspects of the hippocampal-entorhinal circuitry, such as specific neuronal firing patterns and synaptic plasticity mechanisms.  While the modular attractor network is theoretically scalable, it is unclear how well this scalability translates to biological systems. The actual implementation of such a network in the brain might face limitations due to resource constraints and other biological factors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of RNS for spatial representation is a novel approach that maximizes coding range and spatial information per neuron. In addition, the model integrates principles from neuroscience, cognitive science, and artificial intelligence, providing a holistic view of spatial representation in the hippocampal-entorhinal circuits. The authors also provide rigorous theoretical analysis and empirical experiments to support the model’s claims, demonstrating noise robustness and precise spatial representation. The model makes several testable predictions about neural mechanisms, which can guide future experimental research.", "weaknesses": "The model’s complexity might pose challenges for practical implementation and experimental validation in biological systems. While the model is comprehensive, it remains a high-level abstraction of spiking neural circuits, potentially overlooking finer neurobiological details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for accurately summarizing our study and capturing the core strengths of the paper. We appreciate your questions, as we think they get at the fundamental context surrounding the paper. We've done our best to address each of them fully and concisely below.\n\n>The model’s complexity might pose challenges for practical implementation and experimental validation in biological systems.\n\n- We provided code, which provides proof of concept for practical implementation and lowers barriers for experimental validation.\n- In addition, we envision that the model can be implemented with spiking neural networks, as recent works (e.g., [A4]) have done for complex-valued vectors. Such implementations could also help bridge the gap to practical implementation.\n\n> 1. How biologically plausible is the RNS as a coding mechanism in the hippocampal-entorhinal circuits? \n\nPrior work [4, 5, A5] suggests that the RNS is not only biologically plausible but indeed realized by the brain. This idea is consistent with the striking organization of grid cells into discrete modules along the dorsoventral axis [9].\n\n> Are there any existing biological structures that directly support this model?\n\nYes. In addition to the partition of multiple grid units into discrete scales [9], there are the complimentary roles of lateral and medial entorhinal cortex in processing spatial and non-spatial relations, respectively) [39], and the role of hippocampus as an index to many possible patterns [40]. Throughout the paper we have also tried to highlight the consistency with biological structures and experiments (e.g., lines 124-6, 241-3, 260-3).\n\n> 2. What specific experiments could be designed to empirically test the predictions made by the model? How feasible are these experiments with current technology?\n\nImportant questions. We address these points in the general rebuttal under suggestions for experimental neuroscience. We believe the experimental methods and analyses outlined there are feasible with current technology.\n\n> 3. The model suggests encoding contexts as vectors in the entorhinal cortex. How does it manage the vast diversity and complexity of possible contexts in real-world environments?\n\nGood question. Assigning contexts to vectors provides a simple but explicit way of measuring the similarity between two contexts (computing their inner product). Indeed, such vector embeddings are widely adopted in machine learning (e.g., word2vec). The situation we consider explicitly in the paper (e.g., in Appendix C.2, Figure S2) are _discrete_ contexts, in which no contexts are similar, since it mirrors global remapping in the hippocampus. \n\nWe should caveat that there is an extensive literature on contextual remapping in the hippocampal formation, and we do not attempt to explain every finding in detail. However, an interesting direction for future work is to consider contexts that have manifold structure (for example, in which environmental boundaries have continuously varying color) and applying our model to such cases.\n\n> 4. While the model shows robustness to noise in simulations, how would it perform under the more complex and varied types of noise encountered in biological systems?\n\nTo more fully answer this question, we have conducted some additional experiments under biologically motivated kinds of noise: synaptic failure (dropout), limited synaptic precision (bounded synapses), and ablations/lesions (cell death). The results are summarized in the PDF attached to the general rebuttal in Figures X1, X2, and X3, respectively. We hope that they demonstrate how the model would fare in the face of \"resource constraints\" and other disturbances in the weights.\n\n[A4] Orchard & Jarvis (2023), _ICON Proceedings_\n\n[A5] Stemmler et al. (2015), _Science Advances_", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r2ZMiqLivS", "reviewer": "Reviewer_ide5", "summary": "The paper proposes a computational model that incorporates a number of properties about encoding of space representation in the system. The mathematical framework appears to be well-justified and carry the desired properties. These properties are related to some of the observations made about the properties of space encoding in hippocampus, however I've found that the paper conflates conceptual similarity (a certain mechanisms seems to have certain properties) and computational/mechanistic similarity (a certain brain mechanisms is _actually_ computing like the model suggest), but more about this later.\n\nThe way I see this work is that is presents an elegant case for encoding of information in mathematical / algebraic sense, but I struggle to connect these properties and the way the system is evaluated to biology. Put another way - I am not sure *why* this particular computational model is a good model for HC? How do we even evaluate if it's close to what the brain is doing? Or maybe this aspect is actually not important to the authors, and the main contribution on this work lies elsewhere? I must admit I might have misunderstood the motivation and the goal of this work, and I will reflect this in my confidence level. To make a better job in the subsequent round of evaluation I would like the authors to explain how do they understand the importance of this work? What is it that main thing that it bring to the table? Apart from mathematical elegance.\n\nThe evaluation of the model is based on simulated trajectories, but the comparison is done \"within\" the proposed model, and does not provide external points of reference to allow the reader to understand if the model is better that some other ones? If what regard? Is it empirically better or worse at explaining known biological quirks of the HF, or the goal was only to model similarity conceptual on an abstract level?", "questions": "(1) The proposed model is based on arithmetic, element-wise operations over vectors, and modulo operations... Is the claim here that computations similar to these ones are happening in HC, or those are just some operations, that satisfy a number of properties? Basically do you want to say that the mechanism of the model is close to HC, or that just some of the observed characteristics of the model are close to HC?\n\n(2) Following up on (1) - the closeness between the model and HC is, as far as I can tell, only conceptual, right? There were no comparison made against actual empirical measurements of a biological HC during some task?\n\n122: Could you please elaborate what you mean by \"grid modules\", I understand that this is different from grid-like spatial activation patterns? Are you referring to grid cells representing different scales, each scale being a \"module\"?\n\n(3) Why the comparison between the trajectories is done using simulated animal trajectories and not actual ones? I understand that it's impossible to model such a chaotic system as a real mouse running in a grid... but if that is the case, what is the benefit of trying to predict trajectories at all, and using simulated trajectories based on some rules of animal behaviour? I guess I am confused about the chosen way to compare the model with biology.\n\nFigure 6A (1): What exactly is the baseline model that is marked as \"without attractor dynamics\"?\n\nFigure 6A (2): How come the decoded trajectory matched the true one so closely? Is this result impressive or is it trivial in the context of how the model works and how true trajectories were generated?\n\n(4) The predictions listed on lines 316-323 - are they true in biological system? (A) multiplicative interactions between dendritic inputs providing conjunctive binding operation - this one seems to be at the core of potential achievements of the model, but it is very superficially explained, I think it would be great to have a more extensive explanation of what \"multiplicative interactions between dendritic inputs providing conjunctive binding operation\" actually is, how can we see it manifest in biology, and, after that, how your model achieves it. (B) \"Binding between MEC modules\" - what is specifically meant by conjunctive composition and binding in neuroscience context as it pertains to your work? Because if we only mean to say that the brain does combines inputs of the modules, then sure, that's trivial and the fact that a model also does that is kind of expected. If you mean some specific mechanism or form of conjunctive composition - then what is it? How does it manifest in biological HF? Does your model do it in the same way? How can we assess that? (C) \"relatively fixed attractor weights, plastic HC->sensory weights\" - while these are properties of the model, are they properties of the brain?", "limitations": "The authors extensively address the limitation of this work and this provides valuable context to understanding the significance.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The math is rigorous and there is a clear sense that the constructed mathematical framework is a good match to desired properties.", "weaknesses": "* The task on which the model is tested (inputs, output, goals) was not clearly defined in the paper, from Section 4 we know that it is about path integrations and there are simulated trajectories (generated according to behavioural rules of animals) that the model is compared against, but \n\n* I think there is a dissonance between the claims of the paper regarding neuroscientific impact of this model and the actual comparisons between the model and biology that are brought forward in the paper. If I am correct that these are actually pretty loose, then from here we logically move onto the next question - if the importance is not in that, then what is it in?\n\n* The evaluation of simulated trajectories is not too informative, because it is unclear how trivial or non-trivial it is to show the match between simulated trajectories and the model following them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your agreement about the rigor and suitability of the mathematical framework. Apologies if responses seem terse, we've tried to give clear explanations within the word count.\n\n> To make a better job ... What is it that main thing that it bring to the table? Apart from mathematical elegance.\n\nOur work gives a theory of computations (representations and algorithms) in the HF. We posit that the HF instantiates our model's recurrent dynamics, population codes, and binding mechanisms.\n\nThese lead to testable predictions for experimental neuroscience and ascribe function to existing observations about neural data. The model's robustness (Figs. 3 & 4) could help explain the brain's robustness. We also list core contributions in the general rebuttal.\n\n> the paper conflates conceptual ... and computational/mechanistic similarity\n\nWe agree that the two are distinct. Our contributions to each are also distinct:\n\n- Our model is conceptually similar because the five principles of spatial representation (outlined in Sec. 2.1) are the same for our model and (we believe) for HF. However, the intention is not to stop at _mere_ conceptual similarity, but rather to motivate our proposed mechanisms.\n\n- More significantly, the model has computational/mechanistic similarity. The variables and computations of the modular attractor network (detailed in Sec. 2.2) map onto specific neural populations and circuit-level dynamics. Sec. 2.3 maps the model's parts to neuroanatomy.\n\nBinding and its realization in HF circuits is \"at the core of potential achievements\". We discuss binding further in the general rebuttal and re: question (4). \n\nThus, we don't think the paper conflates the two, nor is it exhibiting only conceptual similarity.\n\n> The evaluation ... better that some other ones?\n\nPlease refer to our discussion of model comparison in the general rebuttal.\n\n*Weaknesses*:\n> The task ... was not clearly defined\n\nThese were defined in Appendix B: Experimental details. We run four categories of empirical tests; each described in a subsection of the appendix. Path integration is just one evaluation.\n\nStill, we anticipate that the spirit of this comment is to make these clearer in the main text. We will definitely revise the main text accordingly.\n\n> a dissonance ... if the importance is not in that, then what is it in?\n\nWe respectfully disagree that the connections to biology are loose. Please refer to the discussion of computational/mechanistic similarity and response to question (2).\n\n> The evaluation of simulated trajectories...\n\nPlease refer to our responses for Question (3) / Figure 6A.\n\n*Questions*:\nRe (1): The claim is the former: that the computations resemble those in the HF.\n\nRe line 122: A grid module is a population of grid cells in MEC that have approximately the same \"scale\" -- the spacing between the firing field peaks that form the hexagonal lattice. Grid modules are seen as functionally significant since (a) scales appear discretized in experimental data, and (b) neuron's scale correlates strongly with anatomical location [9].\n\nRe (2): There are two comparisons. We give examples of grid response fields from our model that resemble neural data (Fig. 6C, Fig. S1). Our model also recreates experimental data regarding global remapping (Appendix C.2, Figure S2). We would be excited to compare our model to further neural data, but we also believe that it is beyond the scope of this paper.\n\nRe (3): a) It is a standard dataset used in computational neuroscience. Recent examples include [37, A6]. b) There are significant practical advantages: synthetic data gives more control to over room sizes, trial lengths, and numbers of trials. c) The dataset is realistic. The authors used statistics of actual rodent trajectories and validated simulation quality [36].\n\nRe Fig. 6A (1): For the model without attractor dynamics, the update in equation 7 is replaced by $\\hat{\\mathbf{g}}_i(t + 1) = \\mathbf{q}_i(v_t) \\odot \\hat{\\mathbf{g}}_i(t)$. \n\nRe Fig. 6A (2): Performance comes from the model's robustness to noise (Fig. 3, Sec. 3.3) and ability to interpolate between integer values (Fig. 4, Sec. 3.4). The task isn't trivial since noise accumulates over time and since the attractor network denoises sub-integers without using additional resources. \n\nRe (4): \n>The predictions ... true in biological system? \n\nExcitingly, we don't know yet! These predictions are offered as new hypotheses to test model similarity to the brain.\n\n>  it would be great to have a more extensive explanation of what \"multiplicative interactions ...\n\nWe appreciate the opportunity to clarify. Please refer to the general rebuttal section on binding.\n\n> (B) \"Binding between MEC modules\" - what is specifically meant ...\n\nTo clarify, in our paper: binding is just another name for conjunctive composition (please see line 67).\n\nIn our model, the state of each grid module $\\mathbf{\\hat{g}}\\_i$ depends on binding the states of other grid modules ($\\mathbf{\\hat{g}}_{j \\neq i}$) and hippocampus ($\\mathbf{p}$) (per Eqs. 6 and 7). Our claim in Line 319 is that sigma-pi neurons in MEC implement this binding.\n\n> Because if we only mean ... then sure, that's trivial ... If you mean some specific mechanism ... then what is it?\n\nWe mean multiplication, implemented by sigma-pi neurons with nonlinearities in dendritic compartments [57].\n\n>How does it manifest ... How can we assess that?\n\nWe don't know yet, but it's biologically plausible, since nonlinear computations on EC inputs within HC dendrites are important for assigning contexts to place cells [26]. It remains to test if a) these computations implement binding, and b) similar operations occur within MEC grid modules.\n\n> &#40;C) ... are they properties of the brain?\n\nTo test this, there are methods (structural and functional imaging) for measuring the timescale and persistence of synaptic plasticity.\n\n[A6] George et al. (2023), _eLife_", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0aItUX12Ag", "reviewer": "Reviewer_BzEm", "summary": "This paper proposes a model for spatial representations in the hippocampal formation. The model relies on a residue number system for encoding spatial positions and uses complex-valued vectors to represent individual residues. These vectors are then combined into a unified vector representing spatial position through a conjunctive vector-binding operation that preserves similarities. The model ensures consistency between individual residues and overall position representation through a modular attractor network, which corresponds to the grid cell modules observed in the entorhinal cortex.", "questions": "Can you clarify what you mean by \"carry-free\" hexagonal coding?\n\nCan you reflect on scalability in terms of the number of neurons. For example, regarding the statement, \"In particular, we require that distinct integer values are represented with nearly orthogonal vectors.\" how does this requirement affect the scalability of the approach? \n\nWhat is the numerical precision required for numerical stability in terms of the neural activity and the synaptic weights? Can you discuss how realistic this approach is in the context of real neurons with firing raters under 100Hz?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While there has been an ample amount of work addressing the computations in hippocampal formation, this paper introduces several interesting ideas and combines them into a comprehensive framework. This model integrates principles of optimal coding, such as maximizing coding range and spatial information per neuron, with an algebraic framework for computation in distributed representation.", "weaknesses": "While theoretically valuable, the approach remains relatively high-level without a realistic evaluation and comparison with behavioral or neural data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your accurate summary of our work and fair assessment of its strengths, weaknesses, and limitations. We appreciate that you found the ideas to be interesting and comprehensive.\n\nWe agree that our modeling approach is \"relatively high-level\", and that such an approach comes with strengths and weaknesses. We would emphasize that our motivation for this abstraction is to include all of the components of the HF needed to have a minimal working algorithm, and no more. In other words, we can justify the functional relevance of each region and computational step, without introducing \"bells and whistles\" that might dilute that message.\n\nIn the general rebuttal to all reviewers, we have included some additional ways in which our model can be evaluated relative to neural data. These are meant to contextualize the predictions for neuroscience that are outlined in the discussion.\n\nRegarding your specific questions:\n\n> Can you clarify what you mean by \"carry-free\" hexagonal coding?\n\nGood question. Upon reflection, we will revise \"carry-free hexagonal encoding\" (lines 213-4) to \"carry-free implementation of a triangular frame\", in order to make the meaning of this statement clearer. For completeness, we also provide some further clarification below.\n\nIn general, carry-free means that the components of the representation can be updated in parallel, i.e., without dependence on the results of other updates. For example, binary representations are not carry-free with respect to addition, since the final state of a bit depends on two components _and_ the results of computation (\"carry-over\") on lower-order bits.\n\nFor the triangular frame discussed in Section 3.5 and Appendix A.3, each 2-D position is represented with three coordinates. Each coordinate can take up to $m$ integer values from $\\{0, 1, \\dots, m-1 \\}$. To illustrate why these computations are not carry-free in general, suppose we update coordinate [1, 1, 0] by [0, 0, 1] - the resulting coordinate is [1, 1, 1]. This is actually equivalent to the 2D position expressed by the coordinate [0, 0, 0], but we wouldn't know that without further computation. Thus, we'd either need further \"carry-over\" operations to reduce states to one member of an equivalence class, or we make equality testing cumbersome.\n\n> Can you reflect on scalability in terms of the number of neurons. For example, regarding the statement, \"In particular, we require that distinct integer values are represented with nearly orthogonal vectors.\" how does this requirement affect the scalability of the approach?\n\nWe appreciate the question and opportunity to clarify.\n\nIn a $D$-dimensional space, linear algebra dictates there are only up to $D$ exactly orthogonal vectors. Our method relies on the fact that there are many more vectors that are almost orthogonal.. That is, they have a non-zero, but still small, inner-product. This idea is fundamental to the theory of dimensionality reduction in machine learning, and indeed, the kinds of random codes we employ here are commonly used for this purpose (e.g., [A7]).\n\nOur analysis in Theorem 1 (Appendix A.1) implies that, to represent a set of $M$ distinct states using vectors whose inner-product is (with high probability) at most $\\epsilon$, it suffices to take $D = O(\\epsilon^{-2}\\log(M))$. Note that the dependence of the dimension on $M$ (the total size of the universe) is just logarithmic! This result is consistent with a wide body of other work in the machine learning literature that obtains similar rates for coding schemes of this nature [A7, A8]. \n\nFrom a practical perspective, this means that a relatively modest number of neurons (as we use in our experiments) can achieve a large dynamic range.\n\n> What is the numerical precision required for numerical stability in terms of the neural activity and the synaptic weights? Can you discuss how realistic this approach is in the context of real neurons with firing raters under 100Hz?\n\nThis is an important question. We have a few comments:\n\n- The von Mises noise experiments (in Figs. 3 and 4) provide an implicit answer, because adding phasor noise decreases reliability of the precision.\n- To answer your question more explicitly, we have conducted a follow-up experiment testing the effect of quantizing to a small number of bits per synaptic weight. Please see Figure X2 in the PDF attached to the general rebuttal; we will also add it to the supplement). Within this range, we've found that a) 5 bits is nearly as good as full precision, b) even 3 bits still performs well, and c) higher precision faces diminishing returns, in a way that higher dimensionality does not. This last point is consistent with prior work on quantization in theoretical computer science (e.g., [A9, A10]), and with the observation that biological neural networks have high dimension but with low-precision components.\n- Nothing in the model requires high firing rates to function.\n \n[A7] Rahimi & Recht (2007), _NeurIPS_\n\n[A8] Dasgupta & Gupta (2003), _Rand. Struct. Algor._\n\n[A9] Clarkson & Woodruff (2009), _ACM STOC_\n\n[A10] Zhang, May, Dao, Ré (2019), _AISTATS_", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5UJG8HgJR", "reviewer": "Reviewer_aqeG", "summary": "The paper proposes a model for spatial representation in the hippocampal formation using a residue number system (RNS) to encode positions as high-dimensional vectors. These vectors are combined into a single representation through vector binding and maintained by a modular attractor network. The model demonstrates robustness to noise, high-resolution encoding, and effective path integration.", "questions": "- How does the time-scale of unit responses compared to actual neurons? Is the attractor model fast enough to track the changes in the environment?\n- The potential prediction for encoding of episodic memory in this framework is not clear to me.", "limitations": "- Comparison to existing methods, or an ablation study on the proposed method to verify the intended computational role for each component\n- Test of the cognitive map aspect of HF", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Solid theory behind every addition to the RNS model to capture HF functionality\n- Testable hypotheses for experiments\n- Limitation about the bio-plausibility of the proposed model was mentioned in the discussion", "weaknesses": "- There were several mention of compositionally as the motivation, but there is no direct analyses to show the effectiveness of model in compositionally\n- There is virtually no comparison to other models in terms of coding range, robustness to noise and compositionality\n- Code is not provided", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful review and accurate summary of its main points. We concur that the strengths listed: theory (proofs and experimental validation) for every part of the RNS model, in addition to testable hypotheses for experimental neuroscience, are the core results of the paper. We would like to clarify a few points regarding weaknesses/limitations of the paper and answer your questions.\n\n> Code is not provided\n\nWe would like to politely mention that code to replicate experiments was included in the Supplementary Material zip of the submission.\n\n> There were several mention of compositionally as the motivation, but there is no direct analyses to show the effectiveness of model in compositionally\n\nThank you for raising this issue. We'd like to use this opportunity to more clearly define what we mean by compositionality and how the model achieves it.\n\nWe use the term 'compositionality' to denote a design principle of the HF model. Compositionality refers to complex representations that are _composed_ by simpler building blocks and formation rules [A11]. This means that even a small number of primitives can be richly expressive, because they have a large combinatorial range. The representations we posit in the hippocampal formation meet this definition: they are composed by modules representing residue numbers and contextual tags that are formed by binding operations. These design choices result in the large and robust combinatorial range, as we show theoretically and empirically.\n\nAt the same time, we recognize that the term 'compositionality' has different meanings across machine learning, cognitive science, and neuroscience. We are also aware that there are interesting cases of compositionality that our model doesn't speak about. These include compositional _generalization_ to out-of-distribution test datasets, and compositional _generation_ of novel strategies, as studied in program induction.\n\nTo make these points explicitly, in the final version, we will include this definition of compositionality upon introduction of the term (around lines 19-20 in the current PDF), and we will discuss these other related senses of compositionality in the discussion under limitations (around lines 304-315).\n\n> There is virtually no comparison to other models in terms of coding range, robustness to noise and compositionality\n\nPlease see our discussion in the general rebuttal on model comparison. In that section, we have explicitly aimed to address the coding range, robustness to noise, and compositionality points of comparison.\n\nWe should also note that no other related work attempts to measure coding range (of modular attractor networks implementing RNS), robustness, and compositionality for a single model of the hippocampal formation. Consequently, we've done our best to indicate points of comparison when appropriate for individual experiments. We also feel that the theoretical foundations and interpretability of the model provide potential advantages over other models that could be proposed in the future.\n\n> How does the time-scale of unit responses compared to actual neurons? Is the attractor model fast enough to track the changes in the environment?\n\nIn our model, the unit responses are on the order of 100ms. We believe that this is a biophysically reasonable parameter, and it is consistent with the timescale of theta oscillations (4-12 Hz) believed to be important for neural computation in HF. Empirically, we find that this timescale is sufficient to track changes quickly enough for reliable path integration.\n\n> The potential prediction for encoding of episodic memory in this framework is not clear to me.\n\nThank you for raising this point. Page constraints made it a bit difficult to discuss this point in the main text, but we will gladly add this information to Sec. 4.2 and the Discussion in a revised version.\n\nThe larger picture is that many neuroscientists believe that the two functions of the hippocampus -- in episodic memory and in spatial navigation -- are supported by the same neural circuits and principles (e.g., [A12]).  Put more starkly, spatial navigation and episodic memory are two sides of the same coin: one is navigation in real physical space, the other is navigation in a more abstract, conceptual space.\n\nIf this unified picture of navigation and memory is correct, then our model would have implications for how memories are structured and stored in HF. The experiments for sensory recall (Sec. 4.2, Fig. 7) are designed to explain how imperfect memories corresponding to sensory patterns can be denoised by the hippocampal-entorhinal loop. Further, in the Appendix (Sec. C.3 and Fig. S5), we test the model's ability to perform recall sequences of concepts, even in the presence of neural noise.\n\nRegarding limitations:\n>  Comparison to existing methods, or an ablation study on the proposed method to verify the intended computational role for each component\n\nPlease see our discussion of model comparison in the general rebuttal.\n\nThe ablation study that you mention is a helpful suggestion. In the PDF attached to the general rebuttal, we have conducted an additional ablation experiment (Fig. X3) to quantify the impact on performance.\n\n[A11] Szabó, Z.G. (2022). Compositionality. _The Stanford Encyclopedia of Philosophy._\n\n[A12] Buzsáki, G., & Moser, E. I. (2013). Memory, navigation and theta rhythm in the hippocampal-entorhinal system. _Nature neuroscience_, _16_(2), 130-138.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their thorough reviews and critical feedback. In addition to addressing each review individually, we'd like to discuss a few common points.

*Core contributions:* There are many fundamental yet unresolved questions about the function of the hippocampal formation. We propose a computational model that unifies many postulated functions of the HF and formulates neurally plausible mechanisms that achieve them. The resulting normative model of a cognitive map in the HF yields insights into the functional significance of many experimental phenomena. For example, it helps explain:

- Why are grid cells organized into discrete modules? (Because modular attractor networks exhibit superlinear scaling in dimension.)
- Why do hexagonal lattices appear in grid cells? (Because they improve spatial resolution by ~3x.)

Other theoretical results include the scaling of coding range according to Landau's function, the concentration inequality for kernel approximation, and information-theoretic analysis of superlinear scaling with dimension. Overall, reviewers seem to agree that these were strengths, finding the theoretical analysis to be "rigorous" and the mathematical assumptions to be "well-justified."

*The role of abstraction:* A few reviewers commented on the relevance to biology. The art of theoretical neuroscience is to find the right level of abstraction that allows one to elucidate system-level computational function while still making testable predictions about the brain. Our model aims to be concrete enough about postulated computations and associated mechanisms: including population-level representations, dynamics, and instantiations in specific brain regions, while also providing useful suggestions for experimental neuroscience. 

Relative to some other HF models, our model captures the circuits at an abstract level, and we have tried to make the consequent limitations explicit. But this abstraction also has benefits. As reviewer aqEg says, it allows us to provide "[s]olid theory behind *every* addition to the RNS model to capture HF functionality" (emphasis added). It lets us be mathematically precise enough to run full simulations of the system, use results from high-dimensional statistics to provide rigorous theory, and separate assumptions from predictions.

*Connections to experimental neuroscience:* The model's circuit mechanisms lead to testable predictions for experimental neuroscience. Possible evaluations include:

- Representational Similarity Analysis [A1] compares models to neural data by comparing similarity matrices whose entries are the pairwise similarity of representations of two conditions. Model representations (vectors) could then be compared to publicly available experimental datasets, e.g., [46].
- More detailed biophysical models of neurons in HC and MEC, or high-resolution recordings of single units, could help analyze the plausibility of our binding mechanism.
-  Neuroanatomical tracing experiments can determine if the direct connections predicted by our model (e.g., between different modules in the attractor network) exist. 

*Model comparison:* A couple of reviewers asked for comparisons to other models. We would highlight a few points of comparison:

- Coding range (1): The superlinear scaling of the capacity of the modular attractor network with dimension (Sec. 3.2) is better than Hopfield/Noest associative memories [21, 22], for which capacity scales at best linearly in the dimension.
- Coding range (2): The triangular frame results in $3m^2 - 3m + 1$ states per module, vs. other models [e.g., 32, 33] with only $m^2$ states per module (lines 231-234).
- Compositionality: The heteroassociative memory can accurately recover multiple patterns from a single input (Fig. 7C). This improves over other recent heteroassociative models [32, 41] which can recover at most one pattern per input, no matter how large the dimension. We will add a line in Sec. 4.2 to state this comparison explicitly.

Regarding robustness vs. other models, we found it tricky to avoid "apples-to-oranges" comparisons -- in part since the formulation of an attractor neural network for residue number systems is relatively novel. Instead, we focus on experiments showing model robustness to types of noise commonly postulated in neural systems.

*Realization of the binding operation:* In our model, the binding operation $\odot$ is implemented by element-wise vector multiplication. Thus, the model posits that HF has neurons that can multiply their inputs. There are many biologically plausible neuron models of multiplication [A2, sec. 21.1.1]; a recent study gives a concrete example [A3].

In our case, the neuron model for multiplication that makes most sense is the sigma-pi neuron [57, 58]. It is so-called since groups of inputs are first multiplied together (in dendritic compartments), then the outputs of multiplication are summed (in the soma). It maps onto our model: multiplication implements binding, while the sums then compute inner products (similarity).

*Additional experiments:* To strengthen our results, we have run a few more experiments, based on reviewers' suggestions to consider biologically motivated sources of noise and ablation studies. Please see the attached PDF. We find that:

- The model is robust to varying levels of synaptic noise, also commonly known as "dropout" (Fig. X1).
- The model also requires fairly limited synaptic precision (Fig. X2) -- analogous to bounded storage capacity at synapses in real neurons.
- The model can handle some lesions of weights (Fig. X3). Focused lesions of entire columns or modules are more severe than distributed lesions.

We look forward to the discussion period and are happy to clarify further. We will also use each reviewer's feedback to improve the final version of the manuscript.

[A1] Kriegeskorte et al. (2008), _Front. Sys. Neuro._

[A2] Koch, C. (2004), _OUP_

[A3] Groschner et al. (2022), _Nature_

Meta-review: This paper proposes a normative model for spatial representation in the hippocampal formation that combines optimality principles, including maximizing coding range and spatial information per neuron, with an algebraic framework for computations in distributed neural representation. The model uses a residue number system to encode positions as high-dimensional vectors. These vectors are combined into a single representation through vector binding and maintained by a modular attractor network. The model demonstrates robustness to noise, high-resolution encoding, and effective path integration. The results and concepts are well explained and potentially impactful for computational neuroscience.
Decision: Accept (poster)