Not so griddy: Internal representations of RNNs path integrating more than one agent
Authors: William T Redman
Francisco Acosta
Santiago Acosta-Mendoza
Nina Miolane
Venue: NeurIPS 2024 poster (2024)
Keywords: grid cells
path integration
recurrent neural networks
normative models
Primary area: neuroscience_and_cognitive_science
TL;DR: We study the representations that emerge in RNNs trained to path integrate two agents and compare them to the representations that emerge when path integrating a single agent.

Abstract: Success in collaborative and competitive environments, where agents must work with or against each other, requires individuals to encode the position and trajectory of themselves and others. Decades of neurophysiological experiments have shed light on how brain regions [e.g., medial entorhinal cortex (MEC), hippocampus] encode the self's position and trajectory. However, it has only recently been discovered that MEC and hippocampus are modulated by the positions and trajectories of others. To understand how encoding spatial information of multiple agents shapes neural representations, we train a recurrent neural network (RNN) model that captures properties of MEC to path integrate trajectories of two agents simultaneously navigating the same environment. We find significant differences between these RNNs and those trained to path integrate only a single agent. At the individual unit level, RNNs trained to path integrate more than one agent develop weaker grid responses, stronger border responses, and tuning for the relative position of the two agents. At the population level, they develop more distributed and robust representations, with changes in network dynamics and manifold topology. Our results provide testable predictions and open new directions with which to study the neural computations supporting spatial navigation.

Not so griddy: Internal representations of RNNs path integrating more than one agent
William T. Redman" *
Francisco Acosta
Intelligent Systems Center
Department of Physics
Johns Hopkins Applied Physics Laboratory
University of California, Santa Barbara
Santiago Acosta-Moe Mendoza
Nina Miolane
Dynamical Neuroscience
Department of Electrical & Computer Engincering University of California, Santa Barbara
University California, Santa Barbara
Abstract
Success collaborative and competitive environments, where agents must work with against each other, requires individuals encode the position and trajectory themselves and others. Decades of neurophysiological experiments have shed light how brain regions [e.g.. medial entorhinal cortex (MEC), hippocampus] encode the self's position and trajectory. However, has only recently been discov- modulated by the positions and trajectories of
ered that MEC and hippocampus others. To understand how encoding spatial information of multiple agents shapes neural representations,
train recurrent neural network (RNN) model that cap- tures properties of MEC to path integrate trajectories of two agents simultaneously navigating the same environment. We find significant differences between these RNNs and those trained to path integrate only a single agent. At the individual unit level, RNNs trained to path integrate more than one agent develop weaker grid responses, stronger border responses, and tuning for the relative position of the two agents, At the population level, they develop more distributed and robust representations, with changes in network dynamics and manifold topology. Our results provide testable predictions and open new directions with which study the neural computations supporting spatial navigation.
Introduction
When navigating real-world environments, individual agents (e.g., animals) must adjust their jectories based on the observed trajectories of others [1]2). This becomes especially important in competitive and collaborative settings, where the positions, movement directions, and speeds of others contain information necessary for goal-based decision making. For example, an agent competing with others for food must keep track of where "opponent" agents are and where they are headed, order to effectively update its strategy. While decades of research has shed fundamental light on the neural circuitry involved in spatial navigation the absence of others ("single agent" navigation), little is known about how the brain enables spatial navigation in the more general "multi-agent" scenario.
A component of spatial navigation is path integration B, the ability update estimate of position an environment from a sequence of movement directions and speeds (Fig. [T]] Medial entorhinal cortex (MEC) is believed via functional classes of neurons 4. such as grid cells 6), border cells 7 and band cells (8] Recent research has shown that MEC is activated when human subjects observe others navigate
play critical role in supporting path integration
will. www.meeejjjuualeee....
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

environments (9 10], suggesting that it may additionally be involved in computations supporting multi-agent path integration. That neural activity in MEC could be shaped by trajectories of others is further supported by recent work demonstrating that populations of cells in hippocampus and anterior cingulate cortex, brain regions interconnected with MEC, encode spatial information related to the positions of other agents [1 14 [5, 16 17 However, what representations exist in MEC to support multi-agent path integration and how they differ from the representations that have been found in single agent environments, remains largely unexplored. This is of additional importance as it is unclear whether continuous attractor networks (CANs) 18 19 20), which are believed to simultaneossly support the encoding of
underlie the grid code in MEC [21 22 multiple individuals 27).
24 25
To begin to investigate this open research direction, we train a recurrent neural network (RNN) model to perform path integration of two agents ("dual agent" path integration)) and interrogate the representations that emerge. These RNN models [28] 29], when trained on single agent path integration, have been shown to develop properties analogous to MEC. In particular, they exhibit
units with grid, border, and band responses [28 29 30 and population dynamics consistent with
continuous toroiaal attractors
32
While their biological plausibility remains unclear
33 34). we choose to analyze these RNN models for four reasons. First, they contain inter-connectiviti between "hippocampus" --iiipuss""kkieee and "MEC"- like layers, enabling us to model aspects of the experimentally reported multi- aggett hippocampal responses
[2 B 14 16 [77. Second, the lack of neural data from MEC in multi-agent settings makes the outputs of these RNN models unbiased predictions that can be used to further assess their validity. Third, these RNN models have been shown to have response properties beyond grid, border, and band cells that are similar to those in MEC recordings (35). suggesting that the RNNs learn solutions more broadly representative of MEC. And fourth, beyond their connection to neuroscience, these RNN models represent legitimate machine learning systems that exhibit improved goal-based navigation when paired with reinforcement learning (28]. As autonomous agents becoming increasingly deployed in everyday environments, their ability to navigate in multi-agent settings becomes increasingly important (36) and understanding how the representations learned by RNN models trained on single and dual agent path integration compare can provide new insight into how to improve this capability.
We find that, with only a few modifications, we can train existing RNN models 28 29 to achieve
good performance on dual agent path integration ('dual agent RNNs" -ecc 2 These dual agent RNNs are flexible enough to generalize to single agent path integration, whereas RNNs trained on single agent path integration ("single agent RNNs") are not capable of performing dual agent path integration (Sec. 3.I), demonstrating how representations that are optimal in the single agent setting need not be in the dual agent setting. Investigating the learned representations, at the individual unit level, we find that the need to path integrate more than one agent leads to the emergence of stronger border representations and weaker grid representations (Sec. 3.2), result consistent with recent human eeurophysiologioo
experiments ( 100. Mcasuring the representations in a reference frame that captures the joint position of both agents, we find that units in dual agent RNNs develop tuning for "relative space" (Sec. (.3). Collectively, these results provide insight into how the dual agent RNN is able to efficiently solve its task. Utilizing topological and dynamical tools, we characterize differences between single and dual agent RNNs at the population level (Sec. 3.4.. finding no evidence for continuous toroidal attractors underlying the activity of dual agent RNNs which further emphasizes their distinct computational mechanisms. These results question how our existing understanding of MEC can be extended to the ethologically relevant multi-agent setting, and we hope our that they will motivate future experimental and theoretical exploration.
We study, for the first time, how representations that emerge in RNNs trained dual agent path integration differ, at the individual unit and population level, from the representations of single agent RNNs. Our results consistent with recent neurophysiological experiments p 10 and provide testable predictions that can guide future experimental design.
Contributions.
'While there is no direct evidence for simultaneous dual agent path integration, we hypothesize that the brain perform it and that the MEC is involved. See Appendix for our rationale.
2

---

Path integration task
RNN model
Ground truth PC act.
Www
(())
W
Weee
D22)
v(11
O
x0)
Inputs
Recurrent units
Outout units
Figure 1: Overview of RNNs trained to perform single agent path integration. (A) Illustration of path integration task. The agent starts known position z0) and y(0) and makes sequence of movements through space given by the movement directions 0(t) and speeds v(t). From this, ax(f 1) and y(t 1) must be estimated. (B) Schematic single agent RNN model architecture (28 29], which takes input v(t) and ((t), recurrently processes it, and drives activations of an output layer. (C) Example ground truth place cell (PC) activations used as targets for the RNN output units.
2
RNN model
Prior work has considered the setting where a single agent traverses a square environment through
finite length paths (28 29 30). The agent starts given path at position z(0) = [=(0), y()] [-L//2 L/2] x [-, L/2] and subsequently moves to positions z(0) + z(1) - - (()) This path can alternatively be defined by (0) and sequence of movement directions, 0(1) E [-T,, x). and speeds,
ett) E [0, oo), such that z(t + 1) z(t) + At [cos f(t), sin ((t)]" for small At E RR+ (Fig. IA). The accurate updating of z(t + 1) from only u(t) = [0(t), (t)] formally what meant by path integration.
To learn RNN model that perform single agent path integration, prior work has utilized the following architecture (Fig. B). The inputs u(t) are projected, via weights Win E R x2 into recurrent layer of ng units. This recurrent layer has connectivity defined by weights Wree e R xna The units in the recurrent layer used to drive the activation of np output units, via weights Wout E Raexn More explicitly, the RNN model evolves via the dynamics
r(t + 1) = (
r(t) + win
(1)
(t + 1) = Worl + 1),
(2)
where r(f) E R"G and p(t) gur are the activations of the recurrent and output units at time t, and (+)) an activation function.
The RNN model, defined by Eqs. is trained by comparing the output layer activations, ptt), with "ground truth" activations, ptt), using the cross entropy loss,
C(t) = pp()) log p(t) - A/|
(3)
The second term, A|| WT 12, is weight decay regularization term that penalizes the L2 norm of the recurrent weights, by strength À R+ This was found be beneficial for generalization [32]. The form ptt) takes has been shown to determine properties of the representations learned by the single agent RNNs [29]. We follow Sorscher al. (2019) and set the ground truth activations to be "place liie (37]. with the place fields having a difference of Gaussian (DoG) structure 29 (Fig. [C),
(t( = DtG[z(t), c,] = exp [z(()) - cu11/2007) exp [lz(t) c c/12/203)
where (t(t) is the ground truth activation unit i at time t, c; E [-L.22, L/2 x 1-L/2, L/2 the center of its place field, and σ < 02 define the place field's size. The initial activation in the recurrent layer is set using the ground truth place cell activations, r(0) whack ((0). where Aphack E x a learnable set of weights.
To assess the RNN model's performance in single agent path integration, the position of the agent is decoded from the activations of the output layer. Prior work has used a simple top-n, decoder, where the predicted location of the agent is found by averaging the place field centers, C,, corresponding
3

---

to the nd E N units in the output layer with the highest activations. That is, Z(t) where the ar correspond to the ranked order of the output layer activations [ī.e., Pa., (t) Paz (t)
ni EE= Cas (t),
Pamp (t)]. The decoding error is then defined as ||z(t) 2()]l2.
To enable the RNN model perform dual agent path integration, we make four modifications (see Appendix B and Table ST for more details):
Additional inputs: We set ((t) = 0 (L), vi (4), (2()), rz]', where (2(() and vx(t) corre- spond to the 1" agent's movement direction and speed time t.
Summed place cell activations:
We set the ground truth place cell responses to be the sum of their responses to the location of each agent independently. That is, if (t) corresponds to the position of the Th agent at time then (t( DoG [z (t), c,] (tG e.]. This
choice, inspired by "social" place cells
12 13 14
17,. is an important one and we 16
provide our rationale in choosing in Appendix B.1
k-means clustering for decoding: We use k-means clustering to separate the place field centers corresponding to the 2nd output layer units with the highest activations into k = groups. The predicted positions of the two agents, 21 (t) and (t()), are estimated by the center of each cluster.
Reduced weight decay regularization: We reduce the weight regularization strength, A, in the loss function (Eq. 3 as we expect the added complexity of dual agent path integration to require more of Witess 's capacity.
Training this modified RNN on dual agent path integration, we show-for the first time-that it is possible to learn a network that achieves high performance (Figs. ST). While the decoding error of dual agent path integration is greater for dual agent RNNs than the decoding error of single agent path integration is for single agent RNNs (Fig. 2AA B), we find that nearly 50% of all 5000 dual agent trajectories sampled have an error below T. m. which was previously used threshold to determine success [33] (Fig. 2B). In addition, visualizing the RNN's prediction on individual dual agent trajectories shows qualitatively similar behavior to the true trajectories, even when decoding error is >> 0.10 m. (Fig. (2).. Examining the trajectories where the two agents are closest to each other (Fig. (22). we find the dual agent RNN able to maintain its performance. As hypothesized, reducing the strength of weight decay, from À = 10 to À 10-6 enables an increase in performance on dual agent path integration (Fig. 2 compare pink solid and dashed lines).
Results
3.1 Representations in single agent RNNs are not optimal for dual agent path integration
The discovery that recurrent units in RNNs trained on single agent path integration (Sec. 2 develop responses like those found in MEC 228 29 30 35 has been interpreted as normative demonstration of the optimality of these representations. Are these same representations also optimal for dual agent path integration?
To test this, we apply trained single agent RNNs to the dual agent setting (Appendix C). These networks have decoding error of RJ 1.0 m. (Fig, 3 dot), nearly 10x increase in error relative to the trained dual agent RNNs (Fig. A pink solid line). To determine whether this high decoding error is due to poor generalization to the RNN being sensitive to the precise value of the network weights, we "fine-tune" all the trained single agent RNN weights (Wree Win Wour Whack) for 10 epochs on the dual agent path integration task (Appendix C.. We find that this leads to an improvement in performance higher than the decoding error of the trained dual agent RNN (Fig. commprre solid red and pink solid lines). However, this improvement is dependent on updating the trained weights in the recurrent layer, as freezing reee and fine-tuning the other weights again leads to high decoding error (Fig. -edr dashed line). Taken together, these results demonstrate that the representations learned by the recurrent units in trained single agent RNNs are not optimal for dual agent path integration.
dual agent path integration. although the end decoding error remains
In contrast, trained dual agent RNNs can perform single agent path integration with decoding error < 0.10 m. (Fig. 3- green dot). Fine-tuning the weights on single agent path integration further improves performance, even when W is frozen (Fig. 3 green dashed line). This is aligned with

---

A
0t perc. err. = 0.04m
23" perc. err. 0.09 m
Decocing err. (m.)
Single agent
1.2
Single agent X 10"
Dual agent = 10
0.0
Dual agent = 104
Irue pos.
60
00
100
Decoded
B
Epochs
50" perc. 00.12
75" perc. 0.19
Single egent Dual agent
% trajectories
60-
40
30-
20
0.0
0.4
Decoding err. (m.)
Figure 2: RNNs can trained to successfully perform agent path integration. (A) Decoding error, as a function of training epoch, for RNNs trained and tested on single and dual agent path integration. Solid line mean across 5 independently trained networks and shaded area maximum and minimum of all 5 networks. (B) Distribution of median decoding error, across 5000 trajectories (1000 per network). for single and dual agent RNNs. (C) Example ground truth decoded dual agent trajectories. Trajectories were chosen as those closest to the 0th 25th 50th and th percentile of the decoding error distribution.
Single Dual
Dual Single
(converged)
Single: weights:
Single (converged)
weights: Single
(random init.) Single: but RNN weights:
weights: Single
Decoding err. (m.)
Decoding err. (m.)
0.07
1.0
0.6
0.05
02-
0.04
0
2
Fine-tuning epochs
Fine-tuning epochs
Figure 3: Dual agent RNNs can generalize single agent path integration, but vice versa. Decoding error of RNNs trained on single agent path integration and tested dual agent path integration with (red lines) and without (red dot) fine-tuning (FT) on dual agent path integration. Converged performance and performance from random initialization of dual agent RNNs (dashed and solid pink lines. respectively) are shown for comparison. (B) Same (A), but for RNNs trained dual agent path integration and tested single agent path integration. (A)-(B) Lines denote mean and shaded area denotes maximum and minimum, across 5 independently trained RNNs.
recent experimental and computational work showing that fixed MEC connectivity and slow time-scale plasticity between MEC and hippocampus enables generalizable representations new environments 33). These results (which exhibit similar trend when comparing the training loss instead of the decoding error-Figg -ig. (S3) demonstrate that dual agent RNNs have internal representations that are flexible enough to enable generalization to single agent path integration.
3.2 Single and dual agent RNNs develop different representations at the individual unit level
To begin dissecting the representations that emerge dual agent RNNs, we examine functional properties of individual units in the recurrent layer. Previous work has found that single agent RNNs develop responses analogous three functional classes of neurons in MEC: grid cells ( 6, border cells [7]. and band cells [8). The distribution of these properties can be quantified by computing scores

---

A
Single egent agent
recureett units
29-
29
recurrent units
recurent
29-
20-
20
20-
10-
0..5
0.5
-1.0
0.5
0.5
1.0
0
0.4
0.6
0.8
1.0
Grid
Border
Band
Grid grid
Border
0.81
Band band
agent
Grid celle: grid 1.15 1.58
Border cells: border 0.49 0.80
Band cells: band score 0.65 0.84
Single agenti path
Sinlle agen grd soore
Sigle agent border soor
Single agent band score
1.0
Sinlle agent shufte
1.0-
Sengle agent shufle
Bingle agent stuffe
Duel agent gnd score
Dua agent bonder soor
Decoding crerr (m.)
Ducl agent hand soore
Dual agent shuffe
Duel agent shuffe
Dusl agent chufffe
0.8-
0.8-
Decoding
Decoding
0.4-
0.4
0.4-
0.2
0.24
0.2-
0.0
0.0
0.0
10 20
15
25
10
20
grid removed
band
Figure 4: Single and dual agent RNNs differ in their distribution of functional properties. (A) Distribution grid, border, band scores, computed for all units of single and dual agent RNNs. Distribution includes independently trained RNNs. Kolmogorov-Snin -mminov (KS) test used to compare distributions (** : < 0.01). (B) Visualization of rate maps (Appendix D.I) for units, from individual RNNs, with highest grid, border, and band of trained RNNs with the units corresponding the highest grid, border, and band
(C) Decoding score ablated (solid line: mean across independently trained RNNs), compared trained RNNs with random units ablated (dashed lines: mean across randomly sampled choices ablated units for cach independently trained RNN; shaded area: + standard deviation) (Appendix D.3). Mann- Mann-Whitme Whitney test is used to compare distributions (+ < 0.05 and ** p < 0.01 0.01-see see Table for p-values).
for each unit (Appendix D.22). While grid and border scores are standard metrics, to our knowledge no band score exists. We therefore developed metric to quantify "banded-ness ness" (Appendix D.2).
We find statistically significant differences in the distribution of these functional properties between single and dual agent RNNs (Fig. 4A, B). In particular, we find dual agent RNNs develop units with lower grid scores, but higher border and band scores (Fig. [A). Visualizing rate maps associated with the dual agent RNN units that have the highest grid scores, we find that many have responses that are not solely constrained to vertices of triangular lattices, but are instead "honeycomb"-lin (Fig. (4B). This suggests that dual agent path integration leads to a weaker grid code. In contrast, dual agent RNN units with high border and band scores have more "ideal" responses, as compared to units of single agent RNNs, which have additional grid structure (Fig. 4B. This demonstrates that dual agent path integration leads strengthened border and band code. These results are consistently observed across all independently trained RNNs (Fig. S4).

---

Prior work has found all three of these classes to be important for single agent path integration To understand their importance in dual agent path integration, we perform targeted ablations, removing units with the highest grid, border, and band scores (Appendix D.3). We compare the decoding error of these RNNs to RNNs with randomly removed units, as this has been found to be strong baseline 31 B5] Unlike single agent RNNs, which have a significant increase in decoding error relative to the random baseline with the ablation of all functional classes, dual agent RNNs are less sensitive to the ablation of any one functional class (Fig. [C). This suggests that dual agent RNNs develop a more distributed and robust representation. We note that the greater (albeit, small) sensitivity of the dual agent RNNs to ablations of border and band cells are consistent with work arguing their driving role in path integration 30 31 39 40
28 30
39 40).
To determine the extent to which our results are due to the nature of dual agent path integration, as opposed to specific choices of RNN architecture and hyper-pan pperrp,araeetes, we perform several control experiments. First, we consider the possibility that, by having twice as many agents to track, dual agent RNNs have half the effective size of single agent RNNs. In such a case, we might expect a smaller single agent RNN to generate responses similar to those of dual agent RNNs. However, half-sized single agent RNNs have representations that are distinct from the full-sized dual agent RNNs (Fig. SSA) Second, dual agent rate maps are computed using the position of only one of the two agents (the other agent being "averaged out" Appendixx D.I). We re-compute the rate maps of dual agent RNNs, removing one of the agents completely. Thus, while the network dual agent setting, the rate maps are only affected by a single agent. We find this to have little effect on the representations (Fig. [S5 Third, we vary the agent whose position is being used to construct the rate map. We find that the rate maps are nearly identical (Fig. (6)). Given that there is nothing in the loss funetion (Eq. 3 that distinguishes the two agents, it is not surprising that the dual agent RNN treats them the same. Finally, we measure the distribution of grid, border, and band scores for the RNNs trained with greater weight decay regularization (Fig. A dddhed lines). We find that the overall trend of dual agent path integration leading to lower grid scores and higher border scores, relative to single agent path integration, holds (Fig. S7p.
trained in the
Taken together, these results demonstrate that the addition of another dynamic variable (i.c., a second agent) leads to differences in the distribution of functional properties, at the individual unit level, between single and dual agent RNNs.
3.3 Dual agent RNNs develop tuning in relative space
Recent neural recordings have demonstrated that, when mice are trained to navigate to reward sites that move in physical space, grid cells perform path integration in reference frames that are anchored to the locations of these rewards [4).. These results highlight the fact that hippocampus and MEC can perform computations in different coordinate systems (42). Inspired by these results, we examine whether dual agent RNNs develop grid responses in another reference frame. One candidate coordinate system is defined by the relative positions of the two agents ("relative space" -Fig [5A. Appendix E}. We do not find evidence for units having grid responses in relative space (Fig. SB. C). demonstrating that dual agent RNNs are not simply using the same representations as in the single agent setting, but in a different reference frame. However, we find that units classified as border and band cells in the absolute (i.e., allocentric) reference frame have strong tuning in relative space (Fig. S8A), Units with high border score have greatest activations when both agents are near borders (Fig. A8A, B) and units with high band score have greatest activations when both agents are positioned at periodic distances, along parallel lines (Fig. [584, B). The latter could enable easy representation of parallel movement, where the agents are naturally separated. In contrast, units with high grid score have activations that are largely invariant to relative space (Fig. S8A, B).
To further characterize the extent to which dual agent RNNs encode relative position, we compute the spatial information (43) of all recurrent units relative space rate maps (Appendix [E.2]. We find that, while most units contain little spatial information in relative space (corresponding to spatial information x 0), the distribution heavy tailed (Fig. 5D) Visualizing the relative space rate maps with the highest spatial information, we find that many are localized, with greatest activation when the two agents are near each other (Fig. 5 -center pixels). We hypothesize that, because accurate dual agent path integration becomes more challenging as the two agents get closer together, ablating units with high spatial information in the relative space reference frame would lead to an increase in decoding error. In-line with this hypothesis, we find that removing the units with the highest
7

---

A
Rei. calls:
L/2
Allo,
frame
% recurrent units
49-
20
Ay
AX
Ay
29
20-
11/2
10-
-0.5
0.5
1.5
Grid score
High rel. space spatial info. cels
Daal agriinl
recurrntt units
38-
Decoding (m.)
1.0
Daal agent shufi:
20-
TAG
0.6-
0.4
10-
02-
0.5
0.5
1.0
1.5 2.0
25 3.0
5
10 15
20
25
space spatial
96 space encoding cells
Figure 5: Dual agent RNNs develop tuning to "relative space". (A) Schematic illustration of transformation from the allocentric reference frame the relative space reference frame. (B) Distribution of grid scores (across
independently trained RNNs), computed on the relative space rate maps (Appendix) [E.]). (C) Visualization of relative space rate maps, from an individual dual agent RNN, with the highest relative space grid scores. (D) Distribution of spatial information, computed on the relative space rate maps. Distribution includes 5 independently trained RNNs. (E) Visualization of relative space rate maps, from individual dual agent RNN, with the highest relative space spatial information. (F) Decoding
of dual agent RNNs with units having the highest relative space spatial information ablated. Dashed line and shaded area same as Fig. 4C. Mann- Whitney test is used to compute p-values (** < 0.01 -ee Table S3 for statistics).
relative space spatial information leads statistically significant decrease in decoding performance, relative to the random ablation baseline (Fig. F Appendix E.3). That this ablation leads to stronger deficits than the ablation of grid, border, and band cells (Fig. 4C), emphasizes of their importance in supporting dual agent path integration. Further, it demonstrates that the dual agent RNN utilizes an efficient representation (encoding the relative position) to achieve dual agent path integration.
3.4 Single and dual agent RNNs learn different representations at the population level
Our comparison the representations that emerge single and dual agent RNNs has thus far been focused on the activations of individual recurrent units. Our findings that dual agent RNNs develop different distributions of functional classes than single agent RNNs, and that dual agent RNNs encode relative space spatial information, are suggestive that the network structure underlying single and dual agent RNNs are distinct. To further explore this hypothesis, we analyze the recurrent unit activations at the population level.
44 has become widely used tool for studying neural population A core technique in TDA is persistent homology (Appendix F.I.. which be used estimate the topology of the underlying neural manifold. Applying persistent homology, we find evidence that the population activations of single agent RNNs are constrained to manifold having topology consistent with a two-dimensional torus (Fig. 6A. lcfk- arrows), consistent with prior characterizations [31 32). When performing the same analysis on the population activations of dual agent RNNs, find differences in the persistent homology. In particular, the topological signatures of a two-dimensional torus are not clearly present in the persistence diagram (Fig. [6A, right). We see similar patterns in the persistence diagrams across different independently trained single and dual agent RNNs (Fig. S9). This difference in topology can further be seen by fact that the persistence diagrams corresponding to single agent RNNs are closer (with respect to an appropriate metric [51] to the persistence diagrams of an idealized torus, than the persistence diagrams corresponding to the dual agent RNN are (Fig. (STO). This suggests that dual agent RNNs
Topological data analysis (TDA)
activity (25] 45
46
47
48
49
50
do not have the same continuous attractor structure that single agent RNNs have [1 32).

---

A
Single agent topology
Dual agent topology
B
Dynamic simiarity analysis
Single agent
Seed
sm.
Seed
H
Seed
0.20
mose HAMD Aeleenn 00
Sood
0..55
Dual agent
Seed
0..0
0.05
H22
Sood
Sood
0.00
High
Seeel
2
Steel 3
Steed of
Sceed D
Steef
Seed
Stedd
Seed
Filtration value
Filtration value
Single agent
Dual agent
Figure 6: Population level activations of single and dual agent RNNs differ in their topology and dynamics. (A) Persistence diagrams (Appendix[E] [.I] of the population activations example single and dual agent RNNs. Hi bars of high filtration value correspond to existence of loops and H2 bars correspond to existence of two- dimensional cavities. Black arrows denote features of the single agent persistence diagram that are consistent with ovo-dimensional toroidal attractor. (B) DSA (Appendix F2 applied the activations of independently trained single and dual agent RNNs. Color denotes Procrustes analysis over vector fields metric.
The topology and geometry neural manifolds do not capture all properties of the underlying neural circuit. Indeed, RNNs with the same dynamical properties (e.g.. existence of fixed point attractors, limit cycles) can be constructed to have different population activation geometries [22] Similarly, RNNs with the same population activation geometries can be constructed to have different dynamics Recent work has developed a data-driven dynamical systems framework [dynamic similarity analysis (DSA)] for computing compact representations of RNN dynamics [53] Therefore, we
52
complemented our TDA results by applying DSA (Appendix F.2 ) to the population activations of single and dual agent RNNs. We find that independently trained single agent RNNs are dynamically more similar to each other than they are to independently trained dual agent RNNs, and vice versa (Fig. (B. Therefore, both the topological and dynamical level, the representations learned by the single and dual agent RNN are different, suggesting that the addition of a second agent fundamentally changes the network structure that emerges.
4
Discussion
Inspired by recent work demonstrating that brain regions believed to encode the self's position and trajectory (e.g., MEC, hippocampus) modulated by the position and trajectories of others I0 12 13 14,46, we trained a class of RNN models 28 29 to path integrate two agents and investigated the representations that emerged. Given that these RNN models, when trained to path integrate single agents, develop properties that are similar to those found in MEC at the individual unit (c.g., grid, border, and band responses) and population (e.g., network structure with signatures of continuous toroidal attractors) level, this is a natural point for starting to address the larger question of how the MEC supports representation of multiple dynamic variables.
We find that the representations that emerge in dual agent RNNs significantly differ from those of single agent RNNs. Dual agent path integration leads weaker grid codes and stronger border codes (Fig. 4. This is consistent with electrophysiological recordings that have found MEC encodes distance of others boundaries [9] and fMRI recordings that have observed negative correlation between grid cell strength and performance on multi-agent tasks [TO]. Functional class ablations and TDA revealed distributed and robust representation, with no evidence for the population activity being constrained to a woo-dimensional torus (Fig. (6. Despite this, dual agent RNNs are able to generalize perform single agent path integration, even when the weights in their recurrent layer are frozen (Fig. 3B. This demonstrates the flexibility of their representations (38) and is in-line with work finding that neural networks trained perform multiple tasks develop abstract and distributed tuning [54] [3] [56] 57]. Finally, we found that RNNs trained on dual agent path integration develop tuning the coordinate system defined by the relative position the two agents (Figs. 58). This consistent with the tuning of neurons in anterior cingulate cortex to relative position between agents 15), as well as aligned with neurophysiological experiments that have uncovered neurons MEC that encode distance and orientation of self to objects (58).

---

Limitations. The biological plausibility of the RNN models investigated in this work 128, 29 have been questioned [3 (although, see response by Sorscher al. (2022) (59). While our dual agent ground truth place cells were motivated by experimental work measuring hippocampal responses environments with multiple animals [01 12 13
14
16 17], represents only one
possible choice (as discussed in Appendix B). Additionally. it has been found that the RNN model develops weaker modularization in grid properties [28] 29 30), a feature that is believed to be critical for encoding local spatial information [60] 61 62 (although see work suggesting individual grid
modules contain more spatial information than previously thought [63 64 65 (6]) ). Recent use of
loss functions that incorporate more than just path integration have found stronger grid responses and modularization 67 68). Future work should investigate the properties that emerge in these models when tasked with dual agent path integration. In our experiments, we did not account for noise in velocity inputs, which have been found to play a dominant role in human path integration error 699, and are presumably higher for the assessment of others' velocities (given the lack of vestibular and proprioceptive feedback). We hypothesize that, as noise is added to the estimate of one agent's s position and velocity, dual agent RNNs will develop representations more similar single agent RNNs, suggesting a trade-off that should be explored.
Predictions. Recent advances in experimental paradigms, where rodents must update their goal-based planning given the trajectories of others (70, 72), offer the possibility of comparing the in vivo representations of MEC with those generated by the dual agent RNN. Our results make several testable predictions about what responses might be found in MEC. First, we expect grid responses to be weakened, with the cells that do have high grid score exhibiting deviations from standard grid cells (" ("honeycomb" "op"-llie -like responses -Fi.. 4 Second, we expect strengthened boundary and border responses (Fig. 4). with border cells responding most strongly when both agents are near borders (Fig. (S8) Finally, we expect that the relative position between agents will be encoded in the a subpopulation of MEC neurons (Fig. ()) that disjoint from cells with high grid scores (Fig. S8). Testing these predictions will provide insight into the validity of the RNN model, as well as our assumption that MEC can perform simultaneous path integration of two agents.
Future directions. There are several impactful directions that our work can be extended. First, analyzing the emergent properties of RNNs trained to path integrate more than two agents can lead to a better understanding of how the MEC performs computations in the multi-agent setting, Second, reinforcement learning be used probe how the dual agent RNN representations might support higher order behavior 773 74 755 (such as shortcut taking in dynamic chasing (76).. Third, theoretical work, showing- -ane--fie from first principles- what representations are necessary for single agent path integration (8) can be extended to the dual agent path integration setting. This will provide thorough treatment of how the complexity of multi-agent environments changes the computations needed to be performed by MEC. Finally, foundational work on MEC has found that grid cells encode abstract relationships beyond purely spatial variables [79]. These include social hicrarchies [80), sound frequencies [8]], list ordering distances [82). and elapsed time [83] 84 85 Our work
has implications for how the MEC may simultancously integrate information across these different cognitive axes.
are one of the few canonical models in computational neuroscience. That we find RNNs trained to perform dual agent path integration develop structure distinct from CANs is noteworthy. There are several possibilities for reconciling our results with their differences to CANs, two of which we discuss below. First, the relative space encoding that we find could be coupled with CAN so that the resulting network path integrates the other agent by first path integrating the self and then updating the other's position estimate by using the relative spacing encoding. This kind of egocentric framework is consistent with the presence of object vector cells in MEC [58). Second, the grid code could be one of several codes that enables path integration, and scenarios (such as the presence of another) could lead the reduction of the grid code for another, more suitable code. This is in-line with the negative correlation between strength of grid responses and success in a multi-agent task [1].
Outlook. Continuous attractor networks (CANs) for the grid code 18
19 20
We hope that our work inspires new experiments to uncover the representations used by the brain to enable success in multi-agent environments.
10

---

Acknowledgments and Disclosure of Funding
We thank Cash Costello, Andy Alexander, Michael Goard, Kechen Zhang, Will Dorrell, and members of the Goard Lab for helpful discussion surrounding this work. We thank Chris Ratto, Nora Wolcott, Adele Meyers, Anna-Lena Krause, and Luís Pereira for feedback on this manuscript. WTR acknowl- edges funding by an Internal Research and Development grant from JHU/APL. FA acknow funding from the National Science Foundation, grant 2134241. NM acknowledges funding from the National Science Foundation, grant 2313150.
References
[1] Edmund T Hall and Edward T Hall. The hidden dimension, volume 609. Anchor. 1966.
[2] Kazushi Tsutsui, Masahiro Shinya, and Kazutoshi Kudo. Human navigational strategy for intercepting an erratically moving target in chase and escape interactions. Journal of motor behavior, 52(6):750-760 2020.
[3] Bruce L McNaughton, Francesco P Battaglia, Olc Jensen, Edvard Moser, and May- Britt Moser. Path integration and the neural basis of the 'cognitive map'. Nature Reviews Neuroscience, 7(8):663-67) 678, 2006.
[4] Mariana Gil, Mihai Ancau, Magdalene Schlesiger, Angela Neitz, Kevin Allen, Rodrigo J De Marco, and Hannah Monyer. Impaired path integration in mice with disrupted grid cell firing. Nature neuroscience, 21(1):81-91 2018.
[5] Torkel Hafting. Marianne Fyhn, Sturla Molden, MaytBritt Moser, and Edvard Moser. Microstructure of a spatial map in the entorhinal cortex. Nature, 436(7052):801-8000 2005.
[6] Hanne Stensola, Tor Stensola, Trygve Solstad, Kristian Froland, May-Britt Moser, and Edvard Moser. The entorhinal grid map is discretized. Nature, 492(7427):72-78, 2012.
[7] Trygve Solstad, Charlotte Boccara, Emilio Kropff, May-Britt Moser, and Edvard Moser. Representa- tion of geometric borders the entorhinal cortex. Science, 322(5909)) 1865-186,, 2008.
[8] Julija Krupic. Neil Burgess, and John Oeeefee Neural representations of location composed of spatially periodic bands. Science, 337(6096):853-88) 857, 2012.
[9] Matthias Stangl, Uros Topalovic, Cory S Inman, Sonja Hiller, Diane Villaroman, Zahra M Aghajan, Leonardo Christov-Moore, Nicholas R Hasulak, Vikram R Rao, Casey H Halpem, al. Boundary- anchored neural mechanisms location-encoding for self and others. Nature, 58907842):400-425 2021.
[10) Isabella C Wagner, Luise P Graichen, Boryana Todorova, Andre Lüttig, David Omer, Matthias Stangl, and Claus Lamm. Entorhinal grid-like codes and time-locked network dynamics track others navigating through space. Nature Communications, (411:231, 2023.
[11) Xiang Mou and Daoyun Ji. Social observation enhances cross-enviromment activation of hippocampal place cell patterss. Elife, 8022, 2016.
[12] Teruko Danjo, Taro Toyoizumi. and Shigeyoshi Fujisawa. Spatial representations of self and other in the hippocampus. Science, 359(6372):213 2018.
[13] David B Omer, Shir R Maimon, Liora Las, and Nachum Ulanovsky. Social place-cells in the hat hippocampus, Science, 359(6372):218-222) 2018.
[14] Jeroen J Bos, Martin Vinck, Pictro Marchesi, Amos Keestra, Laura A van Mourik Donga, Jadin C Jackson, Paul FMJ Verschure, and Cyriel MA Pennartz. Multiplexing of information about self and others in hippocampal ensembles. Cell Reports, 29(12)3859 1871, 2019.
[15] Seng Bum Michael Yoo, Jiaxin Cindy Tu, and Benjamin Yost Hayden. Multicentric tracking of multiple agents by anterior cingulate cortex during pursuit and evasion. Nature communications, 12(1):1985, 2021.
[16] Angelo Forli and Michael M Yartsev. Hippocampal representation during collective spatial behaviour in bats. Nature, pages 1-8, 2023.
[17] Xiang Zhang, Qichen Cao, Kai Gao, Cong Chen, Sihui Cheng, Ang Li, Yuqian Zhou, Ruojin Liu, Jun Hao, Emilio Kropff, et al. Multiplexed representation of others in the hippocampal cal subfield of female mice. Nature Communications, 15(1):3702, 2024.

---

[18] Mark C Fubs and David S Touretzky. A spin glass model of path integration in rat medial entorhinal cortex. Journal of Neuroscience, 26( (6)1:466664222 4276, 2006.
[19] Alexis Guanella and Paul FMJ Verschure. A model of grid cells based on path integration mechanism. In Artificial Neural Networks-ICAN, 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part 16, pages 740-749. Springer, 2006.
[20] Yoram Burak and Ila R Ficte. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 5(2)=e100029) 2009.
[21] Jonathan Coucy, Aree Witoclar, Sheng-Jia Zhang, Kang Zheng, Jing Ye, Benjamin Dunn, Rafal Czajkowski, May-Britt Moser, Edvard Moser, Yasser Roudi, et al. Recurrent inhibitory circuitry as mechanism for grid formation. Nature neuroxcience, 16(3)3118-244 2013.
[22] KiJung Yoon, Michael A Buice, Caswell Barry. Robin Hayman. Neil Burgess, and Ila R Fiete. Specific evidence low-dimensional continuous attractor dynamics in grid cells. Nature neuroscience, 16(8):0077- 1084, 2013.
[23] Benjamin Dunn, Maria Mørreaunet, and Yasser Roudi. Correlations and functional connections in a population of grid cells. PLoS compittaional biology, 11(2)ee004052, 2015.
[24] Yi Gu, Sam Lewallen, Amina Kinkhabwala, Cristina Domnisoru, Kijung Yoon, Jeffrey L Gauthier, lla R Fiete, and David W Tank. A map-like micro-orgamizattin of grid cells in the medial entorhinal cortex. Cell, 175(3y:33-750,
2018. 2018.
[25] Richard J Gardner, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A Baas, Benjamin A Dunn, May-Britt Moser, and Edvard Moser. Toroidal topology of population activity in grid cells. Nature, 602(7895):123-B5 2022.
[26] Simon M Stringer, Edmund Rolls, and Thomas P Trappenberg. Self-organising continuous attractor networks with multiple activity packets, and the representation of space. Neural networks, 17(1):5-2) 27, 2004.
[27] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the drosophila central brain. Science, 356(63400:849-853, 2017.
[28] Andrea Bamino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(77055:429-433, 2018.
[29] Ben Sorscher, Gabriel Mel, Surya Ganguli, and Samuel Ocko. A unified theory for the origin of grid cells through the lens of pattern formation. Advances in neural information processing systems, 32, 2019.
[30] Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In International Conference Learning Representations, 2018.
[31] Vemund Schøyen, Markus Borud Pettersen, Konstantin Holzhausen, Marianne Fyhn, Anders Malthe- Sørenssen, and Mikkel Elle Lepperød. Coherently remapping toroidal cells but not grid cells responsible for path integration virtual agents. Iscience, 26(11). 2023.
[32] Ben Sorscher, Gabriel C Mel, Samuel A Ocko, Lisa M Giocomo, and Surya Ganguli. A unified theory for the computational and mechanistic origins grid cells. Neuron, 2022.
[33] Rylan Schaeffer, Mikail Khona, and lla Ficte. No free lunch from deep learning in neuroscience: A case study through models of the entorhinal hhppoiall-pppppmpall circuit. Advances in Neural Information Processing Systems, 36:52 16067, 2022.
[34] Rylan Schacffer, Mikail Khona, Adrian Bertagnoli, Sanmi Koyejo, and Ila Rani Ficte. Testing assumptions underlying a unified thcory for the origin of grid cells. arXiv preprint arXiv:2311 1..2223311. 2023.
[35] Aran Nayebi, Alexander Attinger, Malcolm Campbell, Kiah Hardcastle, Isabel Low, Caitlin S Mallory, Gabriel Mel, Ben Sorscher, Alex Williams, Surya Ganguli, et al. Explaining heterogeneity medial entorhinal cortex with task-driven neural networks. Advances in Neural Information Processing Syxtems, 34: 14:12177-22179 2021.
[36] Kapil Katyal, Yuxiang Gao, Jared Markowitz, Sara Pohland, Corban Rivera, I-Jeng Wang, and Chien- Ming Huang. Learning a group-aware policy for robot navigation. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 11328- 11335. IEEE, 2022.
12

---

and [37] John O'Keefe and Lyan Nadel. The Hippocumpus Cognitive Mup. Oxford University Press, 1978.
[38] John Wen, Ben Sorscher, Surya Ganguli, and Lisa Giocomo. One-shot entorhinal maps enable flexible navigation in novel environments. bioRxiv, pages 2023-09, 2023.
[39] Markus Borud Pettersen, Vemund Sigmundson Schøyen, Anders Sellhe-Soreensen, and Mikkel Elle Lepperød. Decoding the cognitive map: Learning place cells and remapping. bioRxiv, pages 2024-03, 2024.
[40] Markus Pettersen, Vemund Sigmundson Schøyen, Mattis Dalsæerr Østby, Anders
Mikkel Elle Lepperød. Self-supervised grid cells without path integration. bioRxiv, pages 2024-05, 2024.
[41] Jing-Jie Peng. Beate Throm, Maryam Najafian Jazi, Ting- Yun Yen, Hannah Monyer. and Kevin Allen. Grid cells perform path integration in multiple reference frames during self-motion-base navigation. bioRxiv, 2023.
[42] Eduard Kelemen and André Fenton. Dynamic grouping of hippocampal neural activity during cognitive control of two spatial frames. PLoS biology. 8(6):1000400) 2010.
[43] William Skaggs, Bruce Menaughton, and Katalin Gothard. An deciphering the hippocampal code. Advances in neural information processing systems, 5, 1992.
approach to
[44] Chad M Topaz. Lori Ziegelmeier. and Tom Halverson. Topological data analysis of biological aggregation models. PloS one, 10(5)e0126383, 2015.
[45] Gurject Singh, Facundo Memoli, Tigran Ishkhanov, Guillermo Sapiro, Gunnar Carlsson, and Dario I. Ringach. Topological analysis of population activity in visual cortex. Journal of vision, 8(8):1 1-11, 2008
[46] Chad Giusti, Eva Pastalkova, Carina Curto, and Vladimir Itskov. Clique topology reveals intrinsic geo- metric structure in neural correlations. Proceedings of the National Academy of Sciences, 133554):3345- 13460, 2015.
[47] Gard Spreemann, Benjamin Dunn, Magnus Bakke Botnan, and Nils A Baas. Using persistent homology reveal hidden information in neural data, arXiv preprint
2015.
[48] Rishidev Chaudhuri, Berk Gerçek, Biraj Pandey, Adrien Peyrache, and Ila Ficte. The intrinsic attractor manifold and population dynamics of canonical cognitive circuit across waking and sleep. Nature neuroscience, 22(9):1512-1520 2019.
[49] Eliott Robert Joseph Levy. Simón Carrillo-Segura Eun Hye Park, William Thomas Redman. José Rafael Hurtado, Yeon Chung. and André Antonio Fenton. A manifold neural population code for space in hippocampal coactivity dynamics independent of place fields. Cell reports, 42(10), 2023.
[50] Erik Hermansen, David A Klindt, and Benjamin A Dunn. Uncovering 2-d toroidal representations in grid cell ensemble activity during 1-d behavior. Nature Communications, 15(1):5429, 2024.
[51] Peter Bubenik et al. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res., 16(1):77-102, 2015.
[52] Niru Maheswaranathan Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing xystems, 32, 2019.
[53] Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and Ila Fiete. Beyond geometry: Comparing the temporal structure of computation in neural circuits with dynamical similarity analysis. arXiv preprint arXiv:2306. 10168, 2023.
[54] Guangyu Robert Yang, Madhura R Joglckar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
[55] Laura Driscoll, Krishma Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. bioRxiv, pages 2022-08, 2022.
[56] Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamies. Nature neuroscience, 25(6)7883-79, 2022.
[57] W Jeffrey Johnston and Stefano Fusi. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nature Communications, 14(1):1040, 2023.
13

---

[58] Øyvind Arne Høydal, Emilie Ranheim Skytoen. Sebastian Ola Andersson, May-Britt Moser, and Edvard I Moser. Object-vector coding in the medial entorhinal cortex. Nature, 568(777)):440004404 2019.
[59] Ben Sorscher, Gabriel C Mel, Aran Nayebi, Lisa Giocomo, Daniel Yamins, and Surya Ganguli. When and why grid cells appear or not trained path integrators. bioRxiv, pages 2022- 11, 2022.
[60] lla R Fiete, Yoram Burak, and Ted Brookings. What grid cells convey about rat location. Journal of Neuroscience, 28(27):6888 6871, 2008.
[61] Xue-Xin Wei, Jason Prentice, and Vijay Balasubramanian. A principle of economy predicts the functional architecture of grid cells. Elife, 4:c08362, 2015.
[62] Martin Stemmler, Alexander Mathis, and Andreas VM Herz. Connecting multiple spatial scales to decode the population activity of grid cells. Science Advances, [(1]xe1500816, 2015.
[63] Geoffrey W Dichl, Olivia J Hon, Stefan Leutgeb, and Jill K Leutgeb. Grid and nongrid cells in medial entorhinal cortex represent spatial location and environmental features with complementary coding schemes. Neuron, 94(1):83 92, 2017.
[64] Revekka Ismakov, Omri Barak, Kate Jeffery, and Dori Derdikman. Grid cells encode local positional information. Current Biology, 27(15):2337-2443 2017.
[65] Benjamin Dunn, Daniel Wennberg. Ziwei Huang, and Yasser Roudi. Grid cells show field-to dele-to-field variabil- ity and this explains the aperiodic response of inhibitory interneurons. arXiv preprint arXiv:1701.04895 2017.
[66] William Redman, Santiago grid cell properties within individual grid modules enhances encoding of local space. bioRxiv, pages 2024-02, 2024.
Xue-Xin Wei, and Michael Goard. Robust variability
[67] Ruiqi Gao, Jianwen Xie, Xue-Xin Wei, Song- Zhu, and Ving Nian Wu. On path integration of grid cells: group representation and isotropic scaling. Advances in Neural Information Processing Systems, 14:28623-28635. 2021.
[68] Rylan Schaeffer, Mikail Khona, Tzuhsuan Ma, Cristobal Eyzaguirre, Sanmi Koyejo, and Ila R Fiete. Self- learning of representations for space generates Conference Neural Information Processing Systems, 2023.
grid cells. In Thirth-seventh
[69] Matthias Stangl, Ingmar Kanitscheider, Martin Riemer, Ila Ficte, and Thomas Wolbers. Sources of path integration error in young and aging humans. Nature communications. 11(1):2626, 2020.
[70] Jisoo Kim, Chaewoo Kim. Hio-Been Han, Cheol Jun Cho. Wooseob Yeom, Sung Q Lee. and Jee Hyun Choi. A bird's-cye view of brain activity in socially interacting mice through mobile edge computing (mec). Science Advances, ccab)944,,,, 2020.
[71] Alexander T Lai, German Espinosa, Gabrielle E Wink, Christopher F Angeloni. Daniel A Dombeck, and Malcolm A MacIver. A robot-rodent interaction arena with adjustahle spatial complexity for ethologically relevant hehavioral studies. Cell reports, 43(2), 2024.
[72] Chunyu A. Duan, Ivana Orsolic, Qianbo Yin, Mchul Rastogi, Tom Hagley, Bruno Cuz, Andre Almeida, and Jeffrey Erlich. Mice dynamically adapt to opponents in multiplayer games. Computational and Systems Neuroscience (Cosyne) Conference, 2024.
[73] Seng Bum Michael Yoo, Jiaxin Cindy Tu, Steven T Piantadosi. and Benjamin Yost Hayden. The neural basis of predictive pursuit. Nature neuroscience, 23(2):25-2259, 2020.
[74] Kazushi Tsutsui, Kazuya Takeda, and Keisuke Fujii. Synergizing deep reinforcement learning and biological pursuit behavioral rule for robust and interpretable navigation. In Ist Workshop on the Synergy of Scientific and Machine Learning Modeling @ ICML2023, 2023.
[75] Kazushi Tsutsui, Ryoya Tanaka, Kazuya Takeda, and Keisuke Fujüi. Collaborative hunting in artificial agents with deep reinforeement learning. Elife, 13:e85694, 2024.
[76] Andrew S Alexander, Janet C Tungg G William Chapman, Allison M Conner, Laura E Shelley, Michael E Hasselmo, and Douglas A Nitz. Adaptive integration of self-motion and goals in posterior parietal cortex. Cell reports, 38(10) (1), 2022.
[77] John B Issa and Kechen Zhang. Universal conditions for exact path integration in neural systems. Proceedings of the National Academy of Sciences, 109(17)6676 6720, 2012.
14

---

[78] Will Dorrell, Peter E Latham, Timothy EJ Behrens. and James CR Whittington. Actionable neural representations: Grid cells from minimal constraints. In The Eleventh International Conference on Learning Representations, 2022.
[79] Jon W Rucckemann, Marielena Sosa, Lisa M Giocomo, and Elizabeth A Buffalo. The grid code for ordered experience. Nature Reviews Neuroscience, 22(10):637-649 2021.
[80] Seongmin A Park, Douglas S Miller, and Erie D Boorman, Inferences multidimensional social hierarchy use a grid-like code. Nature neuroscience, 249)):192-1301, 2021.
[81] Dmitriy Aronov, Rhino Nevers, and David W Tank. Mapping of non-spatial dimension by the sippocampal-entrrhinal circuit. Nature, 543(7647):71-772,, 2017.
[82] Sujaya Neupane, lla Fiete, and Mehrdad Jazayeri. Mental navigation in the primate entorhinal cortex. Nature, pages 1-8, 2024.
183] Benjamin Kraus, Mark P Brandon, Robert Robinson, Michael A Connerne,, Michael E Hasselmo, and Howard Eichenbaum. During running in place, grid cells integrate elapsed time and distance run. Neuron, 88(3):578 589, 2015.
[84] James G Heys and Daniel A Dombeck. Evidence for a subcircuit in medial entorhinal cortex representing clapsed time during immobility. Nature neuroscience, 21(11)1574-1588 2018.
[85] Erin R Bigus, Hyun-Wor Woo Lec, John C Bowler, Jiani Shi, and James G Heys. Medial entorhinal cortex mediates leaming of context dependent interval timing behavior. Nature Neuroscience, pages 1 1-12, 2024.
[86] Yedidyah Dordek, Daniel Soudry. Ron Meir. and Dori Derdikman. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. Elife, 5:e 10094, 2016.
[87] É. Duvelle and KJ Jeffery. Social spaces: place cells represent the locations of others. Current Biology. 2866)2227 2018
[88] Christopher Tralie, Nathaniel Saul, and Rann Bar-On. Ripser. py: A lean persistent homology library for python. Journal of Open Source Software, 3(29):925, 2018.
[89] Ulrich Bauer. Ripser: efficient computation of victoris rips persistence barcodes. Journal of Applied and Computational Topology, 5(3):391-423, 2021.
[90] V Esposito Vinzi, Wynne W Chin, Jörg Henseler, Huiwen Wang, et al. Handbook of partial least squares, volume 201. Springer, 2010.
[91] Omer Hazon, Victor H Minces, David P Tomàs, Surya Ganguli, Mark Schnitzer, and Pablo E Jercog. Noise correlations neural ensemble activity limit the accuracy of hippocampal spatial representations. Nature communications, )::26,,,,, 2022.
[92] Hassan Arbabi and Igor Mezic. Ergodic theory, dynamic mode decomposition. and computation of spectral properties of the koopman operator. SIAM Journal on Applied Dynamical Systems, 16(4):2096 2126, 2017.
[93] Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Eurika Kaiser, and J Nathan Kutz. Chaos as an intermittently forced linear system. Nature communications, 8(1):19, 2017.
[94] 1 Mezić. Data-driven analysis and forecasting of highway traffic dynamics. Nature communications, 11(1):2090, 2020.
[95] Igor Mezić. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics, 41:309-325, 2005.
[96] Marko Budišić, Ryan Mohr, and Igor Mezić. Applied koopmanism. Chaos: An Interdisciplinary Journal of Nonlinear Science, Scienne,, 22(4), 2012.
[97] Stephen Wiggins. Introduction to applied nonlinear dynamical systems and Chaos. Springer, 1996.
[98] William T. Redman, Juan M. Bello-Rivas, Maria Fonoberova, Ryan Mohr, loannis G. Kevrekidis, and Igor Mezić. Identifying equivalent training dynamics. arxiv preprint arXiv: 2302.09103 2024.
[99] William T Redman, Maria Fonoberova, Ryan Mohr, loannis G Kevrekidis, and Igor Mezić. Algorithmic (semi-) conjugacy via koopman operator theory. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 6006-601 IEEE, 2022.
15

---

[100] Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanicy, 656:5-28, 2010.
[101] Clarence W Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan S Henningson. Spectral analysis of nonlinear flows. Journal of fluid mechanics, 641:115-127, 2009.
16

---

A Multi-agent path integration
Elite sports players are often remarkably capable of keeping track of their own movements and the movements of others. For instance, soccer/foothall players are able to accurately pass the ball, while running, to teammate who is also running. While these individuals that have trained in specialized settings for years, the "average" person has every day events that require the navigating in environments with moving agents. As an example. changing lanes on the highway requires maintaining an understanding of where one's 's car is, as well as other nearby cars.
While these experiences highlight that humans are able to accurately keep track of the trajectories of themselves and others, they do not definitively demonstrate that we are simultancously performing dual agent path integration. We know of no such work that has directly addressed this question, however we hypothesize that path integrating multiple agents is possible.
consider two characteristics estimate of location without direct sensory inputs (e.g., maintaining an accurate estimate of position in the dark); 2) the ability to take direct routes. We argue that there is anecdotal and experimental evidence supporting that both of these can occur in dual agent settings. First, if turn around after having passed someone in the hallway several seconds previously, we may surprised where they are (e.g.. they are all the way down the hall, they are almost exactly where they were when we saw them). This suggests that we are maintaining an estimate of where another individual is through an estimation of their movement, all without direct visual inputs. And second, rats trained to perform pursuit of laser pointer showed the ability to take "shortcuts" that were predictive in nature for specific types of laser pointer trajectories 76). This suggests that the rats are able leverage dynamical predictions of where the other is going.
path integration: 1) the ability to update an
dual agent path integration is performed in the brain, what areas might actively supporting the computations involved? We hypothesize that the medial entorhinal cortex (MEC) plays an important role. Both the hippocam-
pus and anterior cingulate have been shown to be modulated by the presence of others 11 13 14] 15 16 17). Given the coneecctntty between MEC and these brain regions, it seems likely that the MEC is also modu- lated. Given that the MEC is believed to perform single agent path integration, we expect that the modulated activity may reflect its role in path integrating multiple agents. support of this, Stangl et al. (2021) 2 found that the MEC had similar border representations for both the self and other. However, because of the lack of experimental work in this direction, we cannot rule out that other brain areas are also (or primarily) involved in dual agent path integration.
Finally, we note that grid cells, and path integration more broadly, have increasingly been recognized as being computational mechanisms that are used for more than purely physical location [79]. The ability to update estimates of location in abstract spaces (e.g.. frequency space (8I) suggests other evidence for being able to perform dual agent path integration. For instance, musicians can simultancously play two melodies. Thus, future work may probe this capability in settings involving experimental variables beyond space.
Because our dual agent RNNs, which assume the existence of dual agent path integration, make predictions that can used test this hypothesis, we believe they will be useful in guiding experiments that probe the nature of agent spatial coding.
B RNN architecture and training details
Code used for training and evaluating RNNs on path integration was pulled from https: hhpp::///thubb com/ httgg//gglllaaa///a aaaab//rrd-pateeeee -attrpaaoootttteeaeeeia-e-----ee set to that reported in Table[ST (unless otherwise noted), were trained on single or dual agent path integration. Code used for this paper is publicly available at https:/ / htt///gttttn tiub.b..... com/ wwwliammiiiii aedeaammrrmmmmaamm Mot-So-Gat Grdd..
Vanilla RNNs, with parameters
In Sec. 2 we listed the four modifications that were made in enabling RNNs to learn dual agent path integration. Here provide more details on each of these modifications:
Each time step is characterized by movement direction, (t), and speed, a(). We expand the architecture of the vanilla RNN to allow for four inputs (agent movement direction, agent 1 speed, agent 2 movement direction, and agent 2 speed) by setting the input variable input_siza size of torch. nn. RNN to 4.
Additional inputs:
Trajectories for both agents were sampled using the same generative function 28 29
Summed place cell activations: The output layer of the RNN model comprises of "place cells" [37]. whose "ground truth" activations are tied to the distance of an agent to specific locations ("place fields"). "). In-line with previous work, we a Sffference-of-Gaussie activations are used as an error signal for training the RNN model ground truth activations for the dual agent setting, we evaluate the distance of each agent to all place fields. This is used to generate activations for each place cell (for each agent, independently). We sum these responses, for each cell, to generate the "ground truth" activations (see Appendix B.I for more rationale on this choice).
model for the place field responses (9). The ground truth supervised fashion. To generate the
17

---

A
B
6:35-
Single agent: A= 10+
12.60
... Dual agent: = 10+
Training loss
Single agent: A 10
Training loss
Dual agent: A= 10
6.30
12:55
6.25
12.50
6.20
12:45
8.15
12:40
40
80
90
100
0
20
80
100
Epochs
Epochs
Figure S1: Training loss for RNNs trained on single and dual agent path integration. (A)-BB Training loss, as a function of training epoch, for RNNs trained on single and dual agent path integration, respectively. Solid line mean across independently trained networks. Shaded area is maximum and minimum of all 5 networks.
k-means clustering for decoding: For single agent path integration. previous work has decoded the location of the agent by considering the Tıd 3 most active place cells and averaging the and y coordinates of their place field centers. To decode positions from place cell activations the dual agent setting, we first consider the 2n = most active place cells. Wc then take the x- and y- y-coordinates field centers and apply k-mcans clustering, using the function kmeans from the package scipy. cluster vq. setting & 2. The centroids of the clusters are used as the predicted positions for the two agents. We compute the decoding eтrоr the Euclidean distance between the true and predicted positions of the agents (the positions considered 1 dimeeniiona vectors). Because the order the centroids is not guaranteed to match arbitrary ordering of the agents' positions, we find the ordering that minimizes the distance. As the &-means clusering incurs additional computational costs, we only perform the decoding twice each epoch (at the beginning and half- way through).
the corresponding place
Reduced weight decay regularization: Previous work has introduced weight regularization to the weights in the recurrent layer, W encourage sparsity and aid generalization (Eq. [2]. Given that expect dual agent path integration to a more challenging task, we hypothesize that the weight decay value used for single agent path integration might lead too much sparsity to perform the dual agent path integration task well, We therefore reduced the value used for weight decay, taken as an input in the Adam optimizer torch. optim. Adam, from À= 10 À 10
Table S1: Parameters used to train RNNs single and dual agent path integration. Unless otherwise noted (in the text), these are values the parameters used to train single dual agent RNNs take. Parameters that take values that are different from the defaults of previous models [29 52 are bolded, Any values not listed are the same as they were in previous work 29 32
Parameter
Value
Epochs
100
Batch size
200
Batches per epoch
1000
Path length (7)
20
Arena length (L)
2.2 m
Learning rate
10
Place cells (np)
512
Grid cells (ng)
4096
σ1
0.12
O2
0.24
Activation
ReLU
Weight decay Optimizer
10-6
Adam
The training loss associated with single and dual agent path integration is presented in Fig. ST there is no equivalent to "test" loss, as the trajectory used to train each iteration was sampled randomly from a distribution 29) No trajectories were separately maintained in a "test set".
note that
All training experiments and analysis were performed using an AWS GovCloud g4dn.xlarge instance with 1 GPU and 4 CPUs.
18

---

B
Error 0
Error 0.09
A
Near trajectories decoding error
Dooodied
29-
near trajectories
21-
Error = 0:17 m
Error 0.11 m
14-
0.00
0.05
0.10
0.15
0.20
0.25
Decoding err. (m.)
0.5
Figure S2: Dual agent RNNs are able to decode trajectories when the two agents are near each other. (A) Distribution of decoding error of dual agent RNN for trajectories where the two agents are on average < 0.10 from each other. (B) Example true and decoded positions for the four trajectories with the smallest average distance between agents.
B.1
Why summed place cells?
When deciding how define the "true" place cell activity, in the dual agent setting, considered two important experimental and model based constraints:
Zero mean input: Previous work considered the DoG functional form for because it maintains
a zero mean across space [29]. This was motivated by earlier work that showed how non-negan negative
PCA could learn grid cell like responses from place cell input that assumed be mean (86). Given that prior observations of the RNN model failing develop grid cells (33) came part because of architectural decisions that were not supported by the original theory (59), we sought to ensure that our definition of p maintained "zero mean-ness" so that the lack of grid-ness we observe could not be duc to choice of Note that this rules out the use of certain non-lincarities, such as taking the maximum value of the place responses evaluated for each agent independently and functions that saturate (e.g., sigmoid).
Neural recordings have identified multiple variables of self other being encoding in the hippocampus and anterior cingulate being represented by individual neurons. instance, Zhang et al. (2024) 17 found that there were more CA1 neurons that were tuned to the location of self and other than there were tuned to the location of only the other. Similarly, Yoo et al. (2021) [15] found the majority of neurons encoded information about self and other's position during virtual pursuit task. Thus. while these results social place cells that respond only to the position of others [87) modelling spatial representations in the multi-agent setting should include mixed selectivity.
Mixed selectivity:
rule out the possibility of subpopulation "pure"
Because these two constraints, zero mean input and mixed selectivity, we chose model p as the linear sum the DoG responses applied to the location of each agent separately. That is, p:(t) = DoGiz (t), ea] + DoG (t).. e:]. Note that this assumption of linearity and complete mixed selectivity is almost certainly an oversimplification, however we believe that it is valuable place to start in this first work on exploring the kinds of representations that useful in dual agent path integration. We believe future work exploring how inclusion of non- con-lincaitties and the existence of both disjoint and mixed place cell responses further shapes representations that emerge in path integrating RNNs in multi-agent agent environments.
C Generalization experiment details
w denote the weights projecting from the inputs units the recurrent layer. Given the difference in input dimension between single and dual agent RNNs modifications to W is necessary to enable an RNN trained on single agent path integration to be tested on dual agent path integration (and vice versa). Going from single dual agent path integration (Fig. 3A), concatenate w with itself, forming new W wit w E R Going from dual single agent path integration (Fig. 3B), we remove the weights corresponding to the third and fourth inputs. That is. if W.. [WT, Win Win, WT]. where W corresponds the weights projecting from input é to all units in the recurrent layer, then generate set of weights
19

---

Single +
Dual
Dual
Single
Train: Dual (converged)
Trai: Single; FT all weights: Dual
Train: Single (converged)
Train: Dual; FT all weights: Single
Irain: Gual frandom ini.) ... tan: Single; + all but RNN waights: Dual
... frain Dual; +1 all but KNN waights: Single
6.22
12.8
Training loss
Training loss
12.7-
6.20
12.8-
6.18
6.16-
12.4
Fine-tuning epochs
Fine-tuning epochs
Figure S3: Generalization trends are maintained when considering training loss instead of decoding error. Same as Fig. B but when plotting training loss. Note the difference in scales is due to the fact that the dual agent RNN has summed ground truth place cell activity.
w = WT, W E x Because all other training and architecture apper-parameters are the same between RNN models trained on single and dual agent path integration, no other modifications were considered.
To freeze the weights in the recurrent layer, we set those parameters to have requires -gaad = False.
D Single unit representation details
D.1 Constructing rate maps
The construction rate maps, which visualize activity of individual neurons, as a function of physical location, is a standard tool used to understand the neural representations underlying spatial navigation. In the context of RNNs trained on single agent path integration, this tool has been used to discover grid, border, and band cells
29 30 3J 32. To do this, n4 trajectories, corresponding to the paths of a single agent, generated and used as inputs to RNN models. The activations of each unit the recurrent layer the position of agent at each time point saved. The average activation value, for a given position, is then computed and plotted. Position is resolved at resolution of m bins in the x and y direectioon leading to m² bins in the rate map.
For the dual agent setting, we similarly sample trajectories, use them as inputs dual agent RNNs, and save the activations. The primary difference for constructing rate maps for dual agent RNNs is that only the position of one of the agents is used construct the maps (and both). By default, take the first agent's position. However, since there no difference between the agents, this choice should not have any major effect.
To better understand extent to which this choice for visualizing the rate maps affects our interpretations, we also compute rate maps where only a single agent's trajectory is sampled. That is, we trained an RNN on dual agent path integration. but then sample only single agent's 's trajectory when constructing the rate maps, setting the imputs corresponding the movement direction and speed of the second agent 0 (we refer this approach as "Dual agent RNN single agent rate map"). find that, general, this leads to similar rate maps (compare Fig. 4 with Fig. (S5
For all rate maps, we set n, = 1000 and m = 20. We note that it is typical for rate maps to be smoothed by applying filter after their construction. However, we chose not to smooth the rate maps (for either single or dual agent path integration) ensure that the interpretations of the representations studied were being driven by filtering.
D.2 Functional class metries
Visual inspection of rate maps have led to the identification of subsets of neurons and hidden units that have distinct properties (c.g., "grid-like"), To quantify these properties and enable unsupervised identification, metries have been developed. These are discussed in detail below:
Grid score: Ideal grid cells are characterized by having firing fields distributed along the vertices a triangular lattice 6). For neural data, the grid signature has been found to be more strongly apparent when considering not the rate map, but the spatial autocorrelogram (SAC) of the rate map. Let X; denote the rate map corresponding unit the recurrent layer, and S, denote SAC. S, has strong triangular lattice structure, rotating it by 60 (which we denote by S:D) should lead to a SAC that is highly correlated with the S, This is similarly true for rotations of 120d and 240". High correlations could also arise when there is a finer symmetry. To identify when this is the case, the correlation between 5:00 900 and S 500 be taken. The grid score, gs,, computed
20

---

Grid cells: grid score 0.91 1.20
Border celis: border score - -0.82
Band cells: band score 0.72 0.83
Dual agent path int seed 0
grid 1.33
Dual agent path int seed
Grid cells: score 0.99 1.44
Border cells: border 0.81 0.88
Band cells: band 0.71 0.88
Dual agent path int seed 2
Grid cells: grid score 1.04 - 1.33
Border cells: border score 0.81 0.82
Band cells: band score 0.73 0.92
Dual agent path Int. seed =3
Ecc
cells: grid score 0.91 1.51
Border cells: border score 0.81 0.82
Band cells: band score 0.72 0.87
Dual apent path int. seed 4
Figure S4: Consistent individual unit level representations independently trained dual agent RNNs.
Same as Fig. 4B (top), for all seeds (each corresponding to an independently trained RNN).
21

---

Grid cells
Border cells
Band colls
Single agent RNN Half sized
agent Single agent ratemap
Figure S5: Control experiment rate maps. (A) Visualization rate maps from an RNN with half the number of recurrent and output units trained on single agent path integration, with the highest grid, border, and band scores. (B) Same as Fig, A (top). but for rate maps computed from the trajectories of single agent (Appendix
D.I.
Note that the RNN was, in the of Fig. 4 (top), trained on dual agent path integration.
Agent 1
Agent 2
Figure S6: Rate maps consistent when using different agent's positions. Top row, Fig. 4B in the main text. Bottom row, same as top row. but using the second agent's position plot the rate maps, instead of the first agent's.
88; (S, + S!200 where (S, ) = (E, St, (. with St,, being the j bin of the SAC of hidden unit with the center peak masked out.
[corr(S,, S + co,r(S,, S co, S1350)
)]
(5)
Ideal border cells characterized by having their activations selectively aligned along horder. To compute the border score 77) the rate map corresponding unit in the recurrent layer, X,, is first thresholded, with any spatial this thresholded rate map are found using scipy ndimage and those with area < 200 em 2 are additionally set 0 77. Let X, X M, correspond to this masked out rate map, where M, E {0, 1] mm is the mask and O the element-wiee wise multiplication operation. The maximum coverage along any of four environment walls, OM is computed by finding the number of non-masked out spatial bins along cach wall (i.c., M2(x, y) = 1) and dividing by m. Finally, the mean firing distance, dv,, is computed for each remaining connected component is by taking the mean of the distance of each spatial bin the connected component the nearest wall, weighted by the normalized activation that spatial bin. dm, then defined by taking the average value across all connected components. The border score, bos,, is the computed as
Border score:
having activation < 0.3 max(X.) being to 0. Connected components of
bos; CM. du.
(6)
CM, dls
22

---

Single agent A 10+
24-
againt
recumentt units
recument units
% recument units
30-
16
18-
16-
-0.5
0.5
1.5
-1.0
-0.5
0.5
1,0
0.2
0.4
0.6
0.8
1.0
Grid
Border
celi:: grid score 0.86 1.47
Border cells: border soore 0.60 0.73
Band cells: band soore 0.78
0.91
Dual agent path int.
7.=10=
Grid cells: grid score 1.24 1.37
Border cells: border score 0.87 0.78
Band cels:: band score 0.85 0..5-0... 0.92
Single agent path int.
2- 2010
Figure S7: Difference in functional classes that emerge in single and dual agent RNNs persists when training using greater weight decay. Same Fig. (A)-BB, but for RNN model trained with weight decay 10 Results computed from independently trained RNNs.
To the best of our knowledge, no metric that explicitly quantifies the "banded-nee ness" of neural activity hidden unit activations exists. We therefore created to enable the exploration of how adding an additional agent affects the development of units with band structure in their rate maps. As with grid and border scores, let X, denote the rate map corresponding to unit the recurrent layer. Let ((kz, ky) be a two-dimensional sinusoid with frequency ki, in the x- direction, and ky, in the y didirection. Finally, let
Band score:
K (0, 0.1......... 2.0} be possible values for lts and ky. The hand score, has:, is computed as
bas:
max corrcoef [X:, X; (ke,, ky)], fx KyEK
where corrcoef is numpy's 's Pearson product- product-moment moment correlation coefficient.
D.3 Functional class ablation experiments
To probe how units
recurrent layer are used the computations underlying ability path integrate one or two agents, we perform functional class targeted ablation experiments (Fig. (C). do this, we rank the units based on their functional properties (grid, border, or band score). We then ablate units with the highest 5% of the scores by setting all incoming and outgoing weights 0. That is, if unit is determined to have sufficiently high score and is to ablated, we Wite Wiee weights in the recurrent layer.
O,
Iy.., nG. Here, Wee are the
E Relative position representation details
E.1 Relative position rate maps
construct the traditional rate maps used visualize grid, border, band cells (Fig. Appendix D.I. x- and y-coordinates of the environment were split into bins and the average activity, per spatial bin, was computed using one of the agent's z- and y-position.
To examine whether there was any tuning in the relative position space, we construct rate maps using not and y but A. 22 and A, y1 y2, where (æ1, y1) and (æ2, y2) the locations of the agents (the
23

---

Table S2: Functional class ablation statistics. HhinnW test p-values for RNN model decoding error (Fig- [P). Statistics, computed using scipy. stats mannwhi tnyy.. correspond testing the null hypothesis that decoding error the RNN model with functional classes ablated not greater than RNN model with random units ablated. Bolded p-values denote those less than 0.05.
% units removed
Functional class
Single agent p-value
Dual agent p-value
Grid cells
0.11
0.99
10
Grid cells
6.4 10-4
0.87
15
Grid cells
2.0-10-
0.98
Grid cells
3 10-6
0.98
25
Grid cells
1.3-10-2
1.0
Border cells
2.9-10-1
0.24
10
Border cells
2.9-10-7
0.98
15
Border cells
2.9-10-7
8.0 10-2
20
Border cells
2.9-10
1.4 10 10-2
25
Border cells
2.9-10
0.90
Band cells
2.9-10
4.6 10-2
10
Band cells
2.9-10-7
2.9 1
15
Band cells
10-7
9.0 10
Band cells
2.9-
0.25
25
Band cells
10
0.82
ordering arbitrary, but fixed). This provides a transformation space reference frame (Fig. (A). Note that this transformation is not one-to-one, as multiple configurations of agents position in allocentric space have the same relative space coordinates. Note that relative space reference frame has twiec the range, in each dimension, allocentric space. This is because, while x, y L-L/2, L/2). Ar, Ay L-L/2, L/2].
allocentric reference frame to relative
Interpreting the rate maps constructed in the relative space slightly un-intuitive. To aid in this, provide illustrations three relative space rate maps, with specific locations of the rate map visualized the allocentric reference frame (Fig. 3). As noted above, because the mapping is not one-to-one, these are not the only possible realizations of the allocentric space. From these examples, we see how unit in the recurrent layer that is identified from its allocentric
map grid cell has the same average activity across wide range of relative positions the two agents (Fig. S left). contrast, example border cell responds only when both agents are along the borders (Fig. (S8A, middle), which can be seen by the activations being restricted to when A, is at its minimal and maximal values. For an example band cell, again there is specificity in the response related to the value of A., with activity occurring only at certain intervals (Fig. S8 intervals, there is little activation.
right). In between those
E.2 Spatial information
To quantify the amount of information neural responses encode about specific variables, the metric spatial information developed [3] In particular, the spatial information, SI,, encoded by the rate map corresponding unit the recurrent layer, X. computed by
1 Σ sI, = X;
Xi v)
(pxx,y)x xx, log,
(8)
X.
where X, the mean value of rate map i over all space, and y correspond to the m spatial bins in each direction that are used to construct the rate map, and p(x, y) is the probability that the agent was the spatial bin (x, y).
We compute the spatial information on the relative rate maps (Figs. [S] Appendix E.T) While a useful tool, the
spatial information can be skewed by units that (spuriously) respond highly to only a single spatial bin (see lead high spatial
Fig. E top left rate map). Additionally, the multiplicative dependence on p(z, y) information in specific bins, when probability skewed. Indeed, find that p(æ, y). in the relative space, is heavily distributed towards the center of the rate maps, corresponding to the two agents being near each other. Thus, while useful tool for enabling to identify new functional class of units, these results do not rule out other interesting responses relative space (or other reference frames).
24

---

Grid cells
Border cells
Band cells
X
Relative Allosettinc
Rel. ref. frame
Rel. ref. frame
Rel. ref. frame
Ay
Figure S8: Border and band cells additionally encode relative space information, but not grid cells. (A) Comparison of rate maps in allocentric space (top) and relative space (bottom), for units, from individual dual agent RNN, that had high grid, border, and band scores. Top row the same Fig. 4 (bottom). (B) Example relative space rate maps (with the functional class their allocentric ratemaps had high scores for), with schematic illustrations of possible configurations for different locations in relative space.
Table S3: Relative space ablation statisties. Mann- Hoan-Whitney test p-values for dual agent RNN model decoding eтror (Fig. (55) for units with the highest relative space spatial information ablated (Appendix E.3 Details same Table S2
% units removed
p-vuuue
5
1.3
10 3
10
6.4 10 -4
15
6.8 10
20
4.6 10
25
2.2 10
E.3 Relative space ablation experiments
To probe the importance of the units in the hidden recurrent layer that encode relative space, we perform the same ablation studies as descrihed in Appendix [33] but ranking units based on the spatial information present in their relative space rate maps, opposed to their grid, border, band scores.
F Population representation details
F.1
Topological data analysis
TDA
In particular, it has been leveraged to provide evidence for the population activity of grid cells (from the same module) being constrained two-dimensional toroidal manifold [23] A core tool in TDA is persistence (co)homology, which is used identify topological structure present in data. Persistence (co)homology, applied neural activity, can loosely be understood by the following overview. First, the activity of all N neurons, at each time point, is viewed as point in the space of all possible population activations. This leads to point cloud in N N-dimensional space. Balls, of dimension N. then centered each point the cloud. Gradually, the radii of these balls
44 has become widely used tool in neuroscience 25 45 46
47
48
49
25

---

Single agent topology applogy-seeee seed
Dual agent ogy seed
Single agent tapology capoloyy-seeed seed
Dual agent lopology- - ssed S
00
02
0.
02
0.6
Filtration value
Filtration value
Filtration value
Filtration value
Figure 89: Difference in topology of population level activations corresponding to single and dual agent RNNs is robustly observed across independently trained networks. Same Fig. (AA: for additional example persistence diagrams, corresponding to two other independently trained RNNs.
PD heat distance to ideal. torus
1200
Heat distance
1000
800
600
400
Single agent
Dual agent
Figure S10: Single agent persistence diagrams are closer to idcalized torus than dual agent persistence diagrams. Box plot corresponding to heat distance between the persistence diagrams of the single and dual agent RNNs, with the persistence diagram an idealized torus.
increased. When balls intersect, they are considered connected component. The topological properties of all connected components then computed. In particular, the existence loops (H1) and two-dimensional cavities (H2) are extracted and the persistence of these features is measured. This can be visualized in format called persistence diagram.
We computed the Vietoris-Ripp Rips persistence diagram corresponding the activations of all recurrent units for both single and dual agent RNNs. To do this computation, we used the Ripser python package [8 g9 However, persistence diagrams can subject noise and are more challenging to compute high- dimensional spaces. Therefore, to ensure that the differences in topology observed between single and dual agent RNNs was due to different properties underlying the population activations, and not noise, we projected the population activations onto lower-dimensional subspace and re-computed the persistence diagrams. There several possible choices for redueing the dimensionality. natural option is principal component analysis (PCA). However, we found that, for the single agent RNN population activations, the subspace identified by PCA does not enable good multi-output regression with the agent's location Using partial least squares (PLS) [90]. which has been successfully applied in neuroscience applications [91]. we found that both single and dual agent RNN population activation, captured in (o-dimensional subspace, achieved high multi-output regression with the agents' positions. Therefore, we use PLS reduce the dimensionality of the population activations, and then computed the persistence diagram on this reduced subspace (Fig. 6). The PLS was computed using PLSRegr ession in from skl earn. cross cross_decomposi tioo. Additional examples of persistence diagrams, corresponding to the activations of single and dual agent RNNs, are shown in Fig. (S9)
To understand how the topology of the population level activations of the single and dual agent RNNs evolves with training, computed the persistence diagram associated with "ideal" two-dimensional torus and measured topological distance [5I] between the observed persistence diagram and the ideal persistence diagram. We find that the topology of the single agent RNN is closer to an idealized torus than the topology of the dual agent RNN (Fig. S1O). further supporting our conclusion that the dual agent RNN does not develop the same attractor structure the single agent RNN.
26

---

F.2 Dynamic similarity analysis
The evolution, in time, of the activations of units in RNNs can be viewed as dynamical system. The features of this dynamical system (e-g.. fixed points, limit cycles, quasi-periodic attractors, chaos) dependent the imputs and structure the RNN. Thus, characterizing corresponding dynamical properties offers the ability to gain insight into the organization of the RNN. Unsurprisingly, use of geometric and topological methods to interrogate RNN population activation can fail to properly identify similar and dis-similar dynamics, as they are sensitive the underlying manifold the activation evolves along (52). Recent work (53) has used data-driven
tools [92] [ [4] from Koopman operator theory [05] [96] extract properties of RNN dynamics that can
he compared using a precise notion of equivalence, called "topological conjugacy" 97 Similar approaches
have been used to study the dynamics associated with training deep neural networks (98] as well as iterative
optimization algorithms. more generally [99]. Therefore. we complimented the geometric and topological analysis performed on the structure of the population level representations (Fig. with dynamical similarity analysis (DSA) (55)
To perform DSA, K E N trajectories physical space, corresponding K trajectories in population activity space, are sampled. Each trajectory has temporal length N. All the activations are collected into tensor, X E I KxxX xng, where ng is the number of units in the recurrett layer. We then perform principal component analysis (PCA) on X, reducing the number of dimensions
that we get new tensor X E R Txxar
where T.M < ng- Next, D N time-delays are then applied, where D < T, and the resulting tensor H [2x x (T-)xaaa D is then flattened along its first dimension. Dynamic mode decomposition [100] is then performed get compact representation of the dynamics in terms of the Koopman operator [10I), which is restricted to have rank nn < NM D. The Koopman- bassed based representation between the dynamics of the single and dual agent RNN are then compared, via the "Procrustes over vector field" metric [X
As can seen the brief overview of DSA, there are several hyper-parameten whose value must he chosen. These include K, D. πıM, and n.. To be consistent with the geometric and topological analysis, we set K 1000. Because the length of the paths (I' = 20) is small. we cannot use too many time-delays Therefore. set D 4. Because small number of dimensions activations in both single and dual agent RNNs, we considered nM 10 and n = (Fig. 6 determine whether this choice affected the conclusion that single and dual agent RNNs have different dynamical properties. we performed the analysis again, with ny = 50 and na = 100. Fig. [11 s shows a similar conclusion to Fig. 6 with the Procrustes analysis over vector field metric being nearly x larger for comparison between single and dual agent RNNs than between single and single agent RNNs dual and dual agent RNNs. However, because more modes are kept (nM = 50), more noise is present, and the difference weaker than when fewer modes are kept.
necessary capture most of the variance of the
Seed 0
Single agent
0.08
Procrusees analysis over weddo 1ooe
5eed 1
Seed
0.06
Seed 3
Seed 4
0.04
Sead
Dual agent
Seed 1
0.02
Seed 2
Seed
Seed 4
0.00
Seed D
Geed Seed 2
Geed
S
Sead
Sead
Seed
Seed 3
Seed
Single agent
Dual agent
Figure S11: Difference of activation dynamics between single and dual agent RNNs is consistent with
different choice of DSA typer-parametert Same Fig. 4 but with different choice of DSA hyper- parameters.
27

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are grounded in experimental results discussed in detail in Sec. Badd Figs. 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The ahstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations inherent in the RNN model explored, and the rationale for using them, are discussed Sec. [ addition, there a Limitations subsection present Sec. where more detailed discussion is presented.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification:
The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
28

---

All the theorems, formulas, and proofs in the paper should be numbered and cross reffseeefereneed.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclosc all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detail on the modifications used enable existing RNN models perform dual agent path integration are presented Sec. _ Full details regarding the parameters used to the RNN models and details on the analysis used to interrogate the experimental results are presented in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other rescarchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer:
[Yes]
Justification:
We provide the link our publicly available Github repository in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... httpp::://iip..ccppbliic.
Siaddes/CooeSSumissiiiee
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
29

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nipa.cc/ppun
Buidee//DoeeSubmiissoooPPPP
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The hyper-parameters used for training the RNN models are presented in Tahle ST The base code is linked to in Appendix B
Guidelines:
The answer NA that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Figures that compare mean values across populations of independently trained RNNs have error bars that have their value reported. When claims of statistical significance are made, the statistical test used to make the conclusion and the p-values are reported (Tables SabblesS$253 S3..
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
96% CL. if the hypothesis of Normality errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Information on the AWS GovCloud instance used for all training and analysis was reported in Appendix B
30

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory
storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp////eerrppp..cpclpptttaabia
Answer: [Yes]
Justification:
Potential harm and negative societal impact were mitigated.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There no significant negative socictal impact this work. The positive socictal impact of this work was discussed in Sccs. 1
Guidelines:
The answer NA means that there is no societal impact of the work performed. the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deplay- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point out. For example, the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
31

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The code base that was used to evalaute single agent RNNs and develop dual agent RNNs is linked in Appendix and the related papers are cited throughout our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are alongside the assets?
introduced in the paper well documented and is the documentation provided
Answer: [Yes]
Justification:
We provide link our publicly available Github repository the Appendix.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
crowdsourcing nor research with human subjects was performed.
Guidelines:
The answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of the paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
32

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No human or animal work was performed in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
33


Reviews:
[{"review_id": "W6Efnxaiz3", "reviewer": "Reviewer_rAUq", "summary": "This paper studies the internal representations of recurrent neural networks that have been trained to path integrate two agents simultaneously, based on the hypothesis (and related experimental evidence) that individual agents account for the positions of others in multi-agent environments. The authors augment existing single agent vanilla discrete-time RNN models of path integration to perform the same for two agents. Through several numerical experiments on these models, the authors show that grid cell responses are weaker in the dual agent case compared to the single agent case, while border and band cell responses are stronger. Then they show that these dual agent RNNs encode information on the relative position of the two agents. Furthermore, they show that networks capable of performing dual agent path integration can generalize to the single agent task, while the reverse does not hold in practice. Finally, they outline testable predictions of their model (weaker grid responses, stronger border and band cell responses, tuning for relative position) and future directions for research on the computations underlying spatial navigation.", "questions": "See the Weaknesses section.", "limitations": "The authors have adequately discussed the limitations of the work, but could potentially elaborate on the effect of and limitations related to certain architectural assumptions (see W3).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing is clear and convincing, and the authors have situated this work well in the context of previous research. The figures and overall presentation are great (**minor nit:** it would be better to use vector images instead of high-res PNGs, if possible). This work provides a novel perspective and is a useful contribution to the rich literature on the emergence of grid cell representations for spatial navigation.\n2. The experimental analysis is straightforward and solid, with several important control experiments and metric analyses performed to test the strength of the results.\n3. The authors outline several clear testable predictions to validate their results using experiments.", "weaknesses": "1. The multi-agent experiments are limited to dual agent path integration. Studying briefly the strengths of grid and border cell responses in the 3-, 4- and perhaps 5-agent cases would strengthen the paper, given that the code framework seems flexible. This is indeed a limitation/direction for future work that the authors themselves have acknowledged.\n2. The RNNs used in the experiments are not noisy and are simple vanilla discrete-time RNNs. It would be good to study the interplay between recurrent noise and the grid and border scores. It might also be interesting to analyze continuous-time RNNs (where noise improves stability and robustness, see Lim et al., 2021 [1]).\n3. Certain architectural assumptions could be justified better:\n    1. The results and predictions of this work are strongly linked to those of Sorscher et al., 2021 [2], which may not be the only method by which grid cell representations emerge in a neural circuit. The underspecification of the problem means that alternative hypotheses [3,4] exist that can explain the emergence of grid cells, and these could be studied in future work.\n    2. Is there any biological evidence that the same neural circuits perform path integration simultaneously for multiple agents? Could a single agent network not be used to estimate the positions of multiple agents separately and sequentially?\n    3. Are there cases where the two agents are present in adjacent/extremely close locations but the $k$-means clustering does not decode the position of one agent correctly (since it always identifies 2 centers from the top-2$n_d$ active output units)? More generally, could the authors elaborate on the accuracy of path integration when the agents are close together?\n    4. Have the authors considered alternatives to the \"social\" summed place cell activations? Perhaps having twice the number of output units instead (although, this is admittedly not as scalable to multi-agent cases)?\n\n**References:**\n1. Lim et al. \"Noisy recurrent neural networks.\" Advances in neural information processing systems 34 (2021).\n2. Sorscher et al. \"A unified theory for the origin of grid cells through the lens of pattern formation.\" Advances in neural information processing systems 32 (2019).\n3. Schaeffer et al. \"Self-supervised learning of representations for space generates multi-modular grid cells.\" Advances in neural information processing systems 36 (2023).\n4. Xu et al. \"Conformal Normalization in Recurrent Neural Network of Grid Cells.\" arXiv preprint arXiv:2310.19192 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. We are encouraged that they found our work well written and our analysis solid. Below we respond to the specific questions the reviewer had.\n\n**\"The RNNs used in the experiments are not noisy and are simple vanilla discrete-time RNNs.\"**\n\nThis is an interesting point. Some of the original work demonstrating that RNNs could learn grid cells through training on path integration [Cueva and Wei (2019)] found it necessary to have recurrent noise in order to see good functional classes. However, we are unaware of any subsequent RNN models (including the Sorscher et al. model) that make use of noise. Since we have focused our work on comparing directly to the Sorscher et al. model, we believe the study of noise is outside the scope of this work. \n\nHowever, we agree that it would be interesting to understand how noise affects border and grid scores. In particular, future work could study whether recurrent noise can lead to the variability in grid orientation and spacing that was recently reported in individual grid modules of rat MEC [Redman et al., (2024)]. We will mention these exciting avenues for future research in the revised version of the manuscript (in the Discussion) and we thank you for this suggestion.\n\n**\"Is there any biological evidence that the same neural circuits perform path integration simultaneously for multiple agents?\"**\n\nThis is a great question and one we realized we did not sufficiently address in the original submission. As all reviewers had similar questions, we have provided a response to this in the general comment (G2). If this remains unclear, we would be happy to discuss it in more detail. \n\n**\"Are there cases where the two agents are present in adjacent/extremely close locations but the k-means clustering does not decode the position of one agent correctly?\"**\n\nThis is a good question and we appreciate you bringing up this possibility. To answer your question, we computed the decoding error for trajectories that had a median distance between the two agents of less than 0.10 m (less than the width of the place fields). We find that the dual agent RNN performs well in this setting (Fig. 1 in the global response pdf). Thus, we do not find any cases where the k-means clustering fails. We believe this is due to the fact that we run the k-means clustering on the place field centers of the top 6 most active hidden units. Thus, as long as the top 6 place fields are near the position where both agents are, the clustering should identify locations that are near the true positions. We will add this figure to the Appendix and discuss it in Sec. 3.2. Additionally, we will note that other choices, besides the k-means clustering, could be used and may achieve better decoding. \n\n**\"Have the authors considered alternatives to the \"social\" summed place cell activations?\"**\n\nThis is an important point. We appreciate this question, as it emphasizes that the rationale for this choice was not clear. We have provided an answer to this in the global response (G1). If this remains unclear, we would be happy to discuss it in more detail. \n\n**\"Studying briefly the strengths of grid and border cell responses in the 3-, 4- and perhaps 5-agent cases would strengthen the paper\"**\n\nWe agree that studying grid and border cells in environments with more agents is an important and interesting avenue to pursue. Our initial experiments on tri-agent RNNs have not yielded strong results on the path integration on three agents. We believe this is because we have run up against the capacity of the RNN, and therefore need to modify the architecture to achieve good results. We have begun to explore the parameter space for tri-agent RNNs, but are still in the early phases. We will further emphasize that this is a limitation of our current work and highlight it as in important future direction. \n\n**References:**\n\nCueva and Wei (2019) ICLR “Emergence of grid-like representations by training recurrent neural networks to perform spatial localization”\n\nRedman et al., (2024) bioRxiv “Robust variability of grid cell properties within individual grid modules enhances encoding of local space”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhNKuQ2Vx6", "reviewer": "Reviewer_bz9a", "summary": "The authors study neural representations of space in artificial agents that perform path integration of the positions of two agents simultaneously. This is motivated by recent neuroscience studies showing place cells that respond to the location of nearby animals. The authors feed the velocity and head direction inputs of both agents into an RNN and train the network to generate the place cells. Similar to previous work with single agent position, they evaluate the spatial tuning of neurons in the RNN, specifically looking for border cells and grid cells. Unlike in the work with a single agent position, they found fewer grid cells. This indicates that representing the positions of two agents relied on a somewhat different neural code than representing the position of a single agent.", "questions": "Do authors consider a possibility biological systems might not really perform dual \"path integration\" (they could infer the position of other agents using visual inputs rather than using path integration)?\n\nIs it possible that populations of the cells encoding each agent's position would be entirely or somewhat separate? Would that lead to emergence of the grid code?", "limitations": "Authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To my knowledge this is a first study that explicityly tries to model activity of the neurons in the hippocampal formation using path integration for two agents. \n\nThe paper is well written, and the authors conducted a set of ablation studies.", "weaknesses": "There is no attempt to analytically derive or explain the obtained result. It remains unclear whether changing some of the hyperparameters could have led to a different result. If the experiments were complemented with derivations, like in some previous work on the emergence of grid cells, that would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and helpful comments. We are encouraged that they found our work novel and well written. Below we respond to the specific questions the reviewer had. \n\n**\"There is no attempt to analytically derive or explain the obtained result\"**\n\nWe agree that the lack of analytical theory is a weakness of our paper. However, we believe that our numerical results, which (as the reviewer noted) are the first to study how multi-agent environments may require different kinds of spatial representations than single agent environments, open up a number of interesting questions for theorists to tackle. Therefore, we hope it wiil have a similar impact as Banino et al., (2018) and Cueva and Wei (2019), which led to a general theoretical framework by Sorscher et al. (2019). \n\n**\"Might biological systems not really perform dual 'path integration' \"?** \n\nThis is a great question. We realized that we did not sufficiently address this point in the original submission. As all reviewers had similar questions, we have provided our response in the global comment (G2). If this remains unclear, we would be happy to discuss it in more detail. \n\n**\"Is it possible that populations of the cells encoding each agent's position would be entirely or somewhat separate?\"**\n\nThis is an important point. We appreciate these questions as they emphasize that the rationale for this choice was not clear. We have provided an answer to this in the global response (G1). If this remains unclear, we would be happy we would be happy to discuss it in more detail. \n\nHere, we answer the specific question of what would happen if the populations of place cells encoding each agent were entirely separate. In this case, we expect that two separate 2-dimensional toroidal attractors would develop, each with grid cells. While we think that this is an interesting possibility (and one that we do not fully discount), we believe that mixed selectivity (i.e., neurons responding to multiple different features - in this case, different agents) is a crucial facet of neural computation. Consequently, we consider fully disjoint place cell representations to be unlikely in the hippocampus. \n\n**References:**\n\nBanino et al (2018) Nature “Vector-based navigation using grid-like representations in artificial agents”\n\nCueva and Wei (2019) ICLR “Emergence of grid-like representations by training recurrent neural networks to perform spatial localization”\n\nSorscher et al. (2019) NeurIPS \"A unified theory for the origin of grid cells through the lens of pattern formation\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bANsl4bbsl", "reviewer": "Reviewer_bmf9", "summary": "This paper studies RNNs trained to path-integrate the position of two agents. They show the network behaves differently to similar networks trained to path-integrate the position of a single agent, and make some neural level predictions.", "questions": "In the 2-agent case you showed the tuning curves to one agent's position. I really wanted to know whether the tuning to the two different agents was similar in the same cell? I don't think this was reported anywhere.", "limitations": "The authors discuss their limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper was very nicely written and presented.\n\nThe question was clear, the steps taken sensible, and the results carefully discussed.", "weaknesses": "One big choice that seems likely to have heavily influenced the learnt representations is the choice of mixed place cell output. Surely when another animal appears in my scene I am still able to report a factorised encoding of my own position? If you make this change, and ask the network to output two factorised codes, do your results change? If so this risks becoming quite a specific point, though it would then be interesting to argue about the causes of the mixing or not mixing.\n\nIf I understood correctly, the position decoding using that k-means approach is never used to train the network, it is merely the visualise the position estimate. If this is correct, it seems a strange metric against which to compare networks, e.g. figure 3, without further comment. Rather, the key metric is the loss that is used to train the network, do the same trends hold there?\n\nLargely, this paper seemed to say 'here's a different setting, in which things are different'. The path-integrating literature is great because we know so much about how this circuit works, and it has such a correspondence with biology. Understanding in more detail how this path-integrator you present works, for example examining the connectivity structure, or the velocity update scheme as people are able to do for traditional CAN models, would significantly improve the impact of this work, as it seems comparisons to existing neural data are not currently possible.\n\nPersonally, I agree with the critiques of the Sorscher framework, there seem to be others that match more grid cell characteristics, but its certainly a good starting point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and helpful comments. We are encouraged that they found our work well written and our results carefully discussed. Below we respond to the specific questions the reviewer had.\n\n**\"One big choice that seems likely to have heavily influenced the learnt representations is the choice of mixed place cell output.\"**\n\nThis is an important point, and one that has been brought up by all reviewers.  We appreciate the questions, as they helped us realize that the rationale for this choice was not clear. We have provided an answer to this in the global response (G1). If this remains unclear, we would be happy we would be happy to discuss it in more detail.  \n\nHere, we specifically answer your question of factorisation. We agree that, when another agent enters the environment, we can represent our location and their location in a separable way. Given the width of the place fields we consider, this separability occurs in many of the trajectories we considered. However, in our dual agent RNN, the place cells themselves are not separable (i.e., there is no place cell for agent 1 and not for agent 2). Therefore, the dual agent RNN has to learn to properly integrate the velocity signals with the correct agent. We believe that this is a reasonable assumption, although certainly a simplification as we have more information about ourselves than others. As noted in the Discussion of our submission, we expect that if we provide more noise to the trajectory of the first agent (“the other”) than the second agent (“the self”), we will find that the dual agent RNN representation approaches that of the single agent (as only one agent’s trajectory is reliable). Exploring this trade-off would be an especially interesting future direction.\n\n**\"Do the same trends hold for generalization when considering loss and not decoding error?\"**\n\nWe chose to focus on the decoding error as it is an easier metric, than the loss, to interpret. However, we agree that, just because there is a low decoding error, it is not guaranteed that there is a low loss. We appreciate your suggestion and have made the same plot as Fig. 3, but with the training loss (Fig. 2 in the global response pdf). We see similar trends, suggesting that our conclusions are not due to the focus on the decoding error. We will add this figure to the Appendix and discuss it in Sec. 3.1. \n\n**\"Largely, this paper seemed to say 'here's a different setting, in which things are different' \"**\n\nWe appreciate this comment, as it emphasizes that we could strengthen the general framing of our work. To re-summarize: motivated by research finding that MEC and hippocampus are modulated by the presence of others, we asked what kinds of representations MEC could have to support multi-agent path integration. \n\nApplying a trained single agent RNN to the dual agent setting, we find that the recurrent layer must be changed in order to support dual agent path integration. This result is itself a contribution, as we are unaware of this being previously shown (although, it is perhaps not so surprising). The failure of trained single agent RNNs to be extended to the dual agent setting motivated us to develop a dual agent RNN. We find that the dual agent RNN is able to not only perform dual agent path integration, but also single agent path integration. Consequently, this argues for representations other than what are found in single agent RNNs as being optimal for multi-agent settings. \n\nTo begin to shed light on what the dual agent RNN is learning, we asked: 1) what are the representations and network structure that emerge in the dual agent RNN? , and 2) how are they different from the single agent RNN? We find that the unit level representations and population level representations are distinct. However, our interest in the dual agent setting comes from it being the *setting where single agent RNNs fail*. \n\nThe fact that we find no evidence for 2-d toroidal topology suggests that there may not be an underlying continuous attractor network. This makes it challenging to pursue some of the analyses that the reviewer rightfully points out as being natural. However, we tackled examining how different the dual agent RNN is from the single agent RNN by looking for grid cells in relative space. Not finding any, we concluded that the dual agent RNN learns distinct computations from the single agent RNN. \n\nWe hope this re-framing of our work better demonstrates that it is not just reporting a different setting where things are different. We will integrate this motivation into our revised manuscript (Introduction and Discussion). \n\n**\"I really wanted to know whether the tuning to the two different agents was similar in the same cell?\"**\n\nWe appreciate this suggestion. Following your comment, we have computed the tuning for all three functional classes when considering the position of the first agent or the second agent, although the ordering is arbitrary (Fig. 3 in the global responses pdf). We find the tuning curves to be almost identical (we did check that they are not exactly the same). This emphasizes that the dual agent RNN units likely do not develop tuning for individual agents. We believe this is due to our choice of summing the place cell activations, leading to no distinguishing between the agents, with respect to the loss. This is, of course, a simplification. As noted in the Discussion section, an important step in future work will be to consider what happens when one agent is made to be more representative of the self (which can be done by: including less noise in the inputs of the “self” vs. the “other”; increasing the importance of the “self’s” loss, as compared to the other, etc.). We will include this figure in the Appendix and discuss its implications in Sec. 3.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time and thoughtful comments. All reviewers found our work clear and novel, which we find encouraging. All reviewers identified similar weaknesses: this makes it clear that there are important ways our work can be strengthened. Here, we provide answers to the two major weaknesses shared by all reviewers. We additionally attach a pdf with figures that answer some specific questions from individual reviewers. A response to each individual reviewer’s comments is provided in the thread of the associated review. 

**G1. Why summed place cell activations?**

The summing of place cell activations is certainly an important aspect of our model and one that we did not sufficiently motivate in our original submission. Mixed selectivity in the hippocampus has been observed in complex multi-agent settings [e.g. Forli and Yartsev, (2023)]. More generally, mixed selectivity is believed to play a broad role in neural computations [e.g. Rigotti et al. (2013)]. Therefore, we wanted a model with some mixing of place cell activity. We chose a sum (linear) for three reasons:

1. For the Sorscher et al. model to have grid cells, the place cell layer is required to have certain properties (e.g., zero spatial mean). Some non-linearities (e.g. sigmoid) that might be expected in the hippocampus - when multiple agents are in the environment - do not satisfy these properties. Therefore, to avoid trivially finding weakened grid cells, we chose summing the place cell activations. This is a simple way of meeting the restrictions, while still providing mixing. 

2. Not having separate place cell populations forces the network to learn to integrate the correct velocities with the correct estimated positions of the agents. If instead, two disjoint populations of place cells are considered, then the RNN can achieve this by partitioning the units in half and developing two toroidal attractors. This is interesting, but not scalable. Additionally, partitioning the network would prevent the use of more flexible computations. For instance, an agent near the wall is restricted in its trajectory and therefore easier to predict. In this case, more units can be used to predict the trajectory of an agent in the middle of the environment, where more resolution may be helpful. However, this can happen only when the place cell population is shared across agents. 

3. Since each place field’s width is relatively small (see Fig. 1C), many of the trajectories have place cell activations that are fully separate for the two agents. This is because the two agents are not close enough to activate the same place cells. Thus, for many trajectories, the summing of place cell activations does not deviate from the separability we might intuitively expect.

We will add these rationales into the main text (in Sec. 2) to help clarify and highlight this important choice. 

**G2. Does MEC path integrate two agents simultaneously?**

It is entirely possible that MEC does not simultaneously path integrate two agents simultaneously. Until extensive neurophysiological and behavioral work is performed on this topic, it will remain an open question. However, there are several reasons why we think simultaneous dual agent path integration may occur in MEC:

1. Anecdotally, sports players often have to perform actions that require the integration of both their own motion and the motion of others. For instance, when a soccer/futbol player kicks the ball to a teammate, they have to understand both the direction and speed they are moving, as well as the direction and speed their teammate is moving. Similar examples can be seen when driving: switching lanes on the highway requires the maintenance of both where the driver’s car is going, as well as where nearby cars are going. Finally, recent experimental work by Alexander et al. (2022) showed that rats can learn to perform shortcuts (i.e., routes that are predictive and take less time) when trained to chase a laser pointer. These shortcuts require predicting where the laser pointer will go and where the self is going. Collectively, these observations suggest that the brain, in certain scenarios, performs path integration of multiple agents with little latency. 

2. As noted in the main text, recent experimental work has found that MEC responds to the presence of other agents [Stangl et al., (2021); Wagner et al., (2023)]. In particular, Stangl et al. (2021) found that MEC had similar border responses whether an individual walked around an environment or watched another person walk around the environment. These results suggest that, if the brain is indeed able to path integrate multiple agents simultaneously, the MEC may be involved. 

In our revised manuscript, we will clarify that simultaneous path integration is a hypothesis. We will include a section in the Appendix where we provide more details on why we think this hypothesis is reasonable, expanding upon points 1-2 above. In addition, we will emphasize in the Discussion section that the predictions of our dual agent RNN actually provide a way to test whether MEC does perform dual agent path integration. If the predictions are not supported by neurophysiological experiments, this could suggest that MEC does not perform dual agent path integration, or that it does so, but in a different reference frame – both of which are interesting alternatives. 

**Additional note about Sorscher et al. RNN model**

As a last note, we mention that Nayebi et al. (2021) found that the Sorscher et al. RNN model outperformed many other models in explaining properties of recorded MEC activity. While the brain score metric used has its own limitations, this provides additional motivation for studying the Sorscher et al. RNN model, as it provides a better fit of MEC, beyond just grid cells. We will add this reference to the main text (Introduction) to further motivate the study of the Sorscher et al. model.

Meta-review: This paper examines the spatial representations that emerge in RNNs trained to do "dual path integration", i.e. to perform path integration for two separate agents moving through space. The authors compare these dual path integrators to single-agent path integration models, and show that they develop distinct spatial representations. Specifically, the dual agents show less "classic" grid cell activity (both in terms of single neurons and population-level torus manifolds), and more relative encoding of spatial location for the two agents. These results have potential implications for the encoding of space in social contexts.

The reviewers raised some concerns about this paper, some of which were fairly consistent across reviewers. In particular, multiple reviewers asked why summed place cell activities should be used and why we should entertain the assumption that the entorhinal cortex may engage in path integration for others. The authors provided some additional reasoning on these questions, and also attended to other comments in the rebuttal. Following the rebuttal period, the final average score was 6, putting this paper in the accept range. The AC felt that the paper makes an interesting contribution, so given this and the final scores, a decision to accept was reached.
Decision: Accept (poster)