Not so griddy: Internal representations of RNNs path integrating more than one agent

Authors: william t redman
francisco  acosta
santiago  acosta-mendoza
nina  miolane
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 00:46:10

Abstract: Success in collaborative and competitive environments, where agents must work with or against each other, requires individuals to encode the position and trajectory of themselves and others. Decades of neurophysiological experiments have shed light on how brain regions [e.g., medial entorhinal cortex (MEC), hippocampus] encode the self's position and trajectory. However, it has only recently been discovered that MEC and hippocampus are modulated by the positions and trajectories of others. To understand how encoding spatial information of multiple agents shapes neural representations, we train a recurrent neural network (RNN) model that captures properties of MEC to path integrate trajectories of two agents simultaneously navigating the same environment. We find significant differences between these RNNs and those trained to path integrate only a single agent. At the individual unit level, RNNs trained to path integrate more than one agent develop weaker grid responses, stronger border responses, and tuning for the relative position of the two agents. At the population level, they develop more distributed and robust representations, with changes in network dynamics and manifold topology. Our results provide testable predictions and open new directions with which to study the neural computations supporting spatial navigation.

Main paper: 
 introduction: 
 When navigating real-world environments, individual agents (e.g., animals) must adjust their trajectories based on the observed trajectories of others [1, 2] . This becomes especially important in competitive and collaborative settings, where the positions, movement directions, and speeds of others contain information necessary for goal-based decision making. For example, an agent competing with others for food must keep track of where "opponent" agents are and where they are headed, in order to effectively update its strategy. While decades of research has shed fundamental light on the neural circuitry involved in spatial navigation in the absence of others ("single agent" navigation), little is known about how the brain enables spatial navigation in the more general "multi-agent" scenario.A core component of spatial navigation is path integration [3] , the ability to update an estimate of position in an environment from a sequence of movement directions and speeds (Fig. 1A ). Medial entorhinal cortex (MEC) is believed to play a critical role in supporting path integration via functional classes of neurons [4] , such as grid cells [5, 6] , border cells [7] , and band cells [8] . Recent research has shown that MEC is activated when human subjects observe others navigate environments [9, 10] , suggesting that it may additionally be involved in computations supporting multi-agent path integration. That neural activity in MEC could be shaped by trajectories of others is further supported by recent work demonstrating that populations of cells in hippocampus and anterior cingulate cortex, brain regions interconnected with MEC, encode spatial information related to the positions of other agents [11, 12, 13, 14, 15, 16, 17] . However, what representations exist in MEC to support multi-agent path integration and how they differ from the representations that have been found in single agent environments, remains largely unexplored. This is of additional importance as it is unclear whether continuous attractor networks (CANs) [18, 19, 20] , which are believed to underlie the grid code in MEC [21, 22, 23, 24, 25] , can simultaneously support the encoding of multiple individuals [26, 27] .To begin to investigate this open research direction, we train a recurrent neural network (RNN) model to perform path integration of two agents ("dual agent" path integration2 ) and interrogate the representations that emerge. These RNN models [28, 29] , when trained on single agent path integration, have been shown to develop properties analogous to MEC. In particular, they exhibit units with grid, border, and band responses [28, 29, 30] and population dynamics consistent with continuous toroidal attractors [31, 32] .While their biological plausibility remains unclear [33, 34] , we choose to analyze these RNN models for four reasons. First, they contain inter-connectivity between "hippocampus"-like and "MEC"-like layers, enabling us to model aspects of the experimentally reported multi-agent hippocampal responses [11, 12, 13, 14, 16, 17] . Second, the lack of neural data from MEC in multi-agent settings makes the outputs of these RNN models unbiased predictions that can be used to further assess their validity. Third, these RNN models have been shown to have response properties beyond grid, border, and band cells that are similar to those in MEC recordings [35] , suggesting that the RNNs learn solutions more broadly representative of MEC. And fourth, beyond their connection to neuroscience, these RNN models represent legitimate machine learning systems that exhibit improved goal-based navigation when paired with reinforcement learning [28] . As autonomous agents becoming increasingly deployed in everyday environments, their ability to navigate in multi-agent settings becomes increasingly important [36] and understanding how the representations learned by RNN models trained on single and dual agent path integration compare can provide new insight into how to improve this capability.We find that, with only a few modifications, we can train existing RNN models [28, 29] to achieve good performance on dual agent path integration ("dual agent RNNs"-Sec. 2). These dual agent RNNs are flexible enough to generalize to single agent path integration, whereas RNNs trained on single agent path integration ("single agent RNNs") are not capable of performing dual agent path integration (Sec. 3.1), demonstrating how representations that are optimal in the single agent setting need not be in the dual agent setting. Investigating the learned representations, at the individual unit level, we find that the need to path integrate more than one agent leads to the emergence of stronger border representations and weaker grid representations (Sec. 3.2), a result consistent with recent human neurophysiological experiments [9, 10] . Measuring the representations in a reference frame that captures the joint position of both agents, we find that units in dual agent RNNs develop tuning for "relative space" (Sec. 3.3) . Collectively, these results provide insight into how the dual agent RNN is able to efficiently solve its task. Utilizing topological and dynamical tools, we characterize differences between single and dual agent RNNs at the population level (Sec. 3.4), finding no evidence for continuous toroidal attractors underlying the activity of dual agent RNNs which further emphasizes their distinct computational mechanisms. These results question how our existing understanding of MEC can be extended to the ethologically relevant multi-agent setting, and we hope our that they will motivate future experimental and theoretical exploration.Contributions. We study, for the first time, how representations that emerge in RNNs trained on dual agent path integration differ, at the individual unit and population level, from the representations of single agent RNNs. Our results are consistent with recent neurophysiological experiments [9, 10] and provide testable predictions that can guide future experimental design.Figure 1 : Overview of RNNs trained to perform single agent path integration. (A) Illustration of path integration task. The agent starts at known position x(0) and y(0) and makes a sequence of movements through space given by the movement directions θ(t) and speeds v(t). From this, x(t + 1) and y(t + 1) must be estimated. (B) Schematic of single agent RNN model architecture [28, 29] , which takes input v(t) and θ(t), recurrently processes it, and drives activations of an output layer. (C) Example ground truth place cell (PC) activations used as targets for the RNN output units. 
rnn model: 
 Prior work has considered the setting where a single agent traverses a square environment through finite length paths [28, 29, 30] . The agent starts a given path at position z(0) = [x(0), y(0)]⊺ ∈ [-L/2, L/2] × [-L/2, L/2] and subsequently moves to positions z(0) → z(1) → ... → z(T ). This path can alternatively be defined by z(0) and a sequence of movement directions, θ(t) ∈ [-π, π], and speeds,v(t) ∈ [0, ∞), such that z(t + 1) = z(t) + v(t)∆t • [cos θ(t), sin θ(t)] ⊺, for small ∆t ∈ R + (Fig. 1A ). The accurate updating of z(t + 1) from onlyu(t) = [θ(t), v(t)] ⊺ is formally what is meant by path integration.To learn an RNN model that can perform single agent path integration, prior work has utilized the following architecture (Fig. 1B ). The inputs u(t) are projected, via weights W in ∈ R n G ×2 , into a recurrent layer of n G units. This recurrent layer has connectivity defined by weights W rec ∈ R n G ×n G . The units in the recurrent layer are used to drive the activation of n P output units, via weights W out ∈ R n P ×n G . More explicitly, the RNN model evolves via the dynamicsEQUATIONEQUATIONwhere r(t) ∈ R n G and p(t) ∈ R n P are the activations of the recurrent and output units at time t, and σ(•) is an activation function.The RNN model, defined by Eqs. 1-2, is trained by comparing the output layer activations, p(t), with "ground truth" activations, p(t), using the cross entropy loss,L(t) = -p(t) log p(t) -λ||W rec || 2 .(EQUATIONThe second term, λ||W rec || 2 , is a weight decay regularization term that penalizes the L 2 norm of the recurrent weights, by strength λ ∈ R + . This was found to be beneficial for generalization [32] . The form p(t) takes has been shown to determine properties of the representations learned by the single agent RNNs [29] . We follow Sorscher et al. ( 2019) and set the ground truth activations to be "place cell"-like [37] , with the place fields having a difference of Gaussian (DoG) structure [29] (Fig. 1C ),EQUATIONwhere p i (t) is the ground truth activation of unit i at time t,c i ∈ [-L/2, L/2] × [-L/2, L/2]is the center of its place field, and σ 1 < σ 2 define the place field's size. The initial activation in the recurrent layer is set using the ground truth place cell activations, r(0) = W back p(0), where W back ∈ R n G ×n P is a learnable set of weights.To assess the RNN model's performance in single agent path integration, the position of the agent is decoded from the activations of the output layer. Prior work has used a simple top-n d decoder, where the predicted location of the agent is found by averaging the place field centers, c i , corresponding to the n d ∈ N units in the output layer with the highest activations. That is,ẑ(t) = 1 n d n d i=1 c αi (t), where the α i correspond to the ranked order of the output layer activations [i.e., pα1 (t) > pα2 (t) > ... > pαn P (t)]. The decoding error is then defined as ||z(t) -ẑ(t)|| 2 .To enable the RNN model to perform dual agent path integration, we make four modifications (see Appendix B and Table S1 , for more details):• Additional inputs: We set u(t) = [θ 1 (t), v 1 (t), θ 2 (t), v 2 ] ⊺, where θ i (t) and v i (t) correspond to the i th agent's movement direction and speed at time t.• Summed place cell activations: We set the ground truth place cell responses to be the sum of their responses to the location of each agent independently. That is, if z i (t) corresponds to the position of the i th agent at time t, thenp i (t) = DoG[z 1 (t), c i ] + DoG[z 2 (t), c i ].This choice, inspired by "social" place cells [11, 12, 13, 14, 16, 17] , is an important one and we provide our rationale in choosing it in Appendix B.1.• k-means clustering for decoding: We use k-means clustering to separate the place field centers corresponding to the 2n d output layer units with the highest activations into k = 2 groups. The predicted positions of the two agents, ẑ1 (t) and ẑ2 (t), are estimated by the center of each cluster.• Reduced weight decay regularization: We reduce the weight regularization strength, λ, in the loss function (Eq. 3), as we expect the added complexity of dual agent path integration to require more of W rec 's capacity.Training this modified RNN on dual agent path integration, we show-for the first time-that it is possible to learn a network that achieves high performance (Figs. 2, S1). While the decoding error of dual agent path integration is greater for dual agent RNNs than the decoding error of single agent path integration is for single agent RNNs (Fig. 2A , B), we find that nearly 50% of all 5000 dual agent trajectories sampled have an error below 0.10 m, which was previously used as a threshold to determine success [33] (Fig. 2B ). In addition, visualizing the RNN's prediction on individual dual agent trajectories shows qualitatively similar behavior to the true trajectories, even when decoding error is > 0.10 m. (Fig. 2C ). Examining the trajectories where the two agents are closest to each other (Fig. S2 ), we find the dual agent RNN is able to maintain its performance. As hypothesized, reducing the strength of weight decay, from λ = 10 -4 to λ = 10 -6 , enables an increase in performance on dual agent path integration (Fig. 2A -compare pink solid and dashed lines).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations inherent in the RNN model explored, and the rationale for using them, are discussed in Sec. 1. In addition, there is a Limitations subsection present in Sec. 4 where more detailed discussion is presented.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
results: 
 3.1 Representations in single agent RNNs are not optimal for dual agent path integrationThe discovery that recurrent units in RNNs trained on single agent path integration (Sec. 2) develop responses like those found in MEC [28, 29, 30, 35] has been interpreted as a normative demonstration of the optimality of these representations. Are these same representations also optimal for dual agent path integration?To test this, we apply trained single agent RNNs to the dual agent setting (Appendix C). These networks have a decoding error of ≈ 1.0 m. (Fig. 3A -red dot), nearly a 10× increase in error relative to the trained dual agent RNNs (Fig. 3A -pink solid line). To determine whether this high decoding error is due to poor generalization or to the RNN being sensitive to the precise value of the network weights, we "fine-tune" all the trained single agent RNN weights (W rec , W in , W out , W back ) for 10 epochs on the dual agent path integration task (Appendix C). We find that this leads to an improvement in performance on dual agent path integration, although the end decoding error remains higher than the decoding error of the trained dual agent RNN (Fig. 3A -compare solid red and pink solid lines). However, this improvement is dependent on updating the trained weights in the recurrent layer, as freezing W rec and fine-tuning the other weights again leads to high decoding error (Fig. 3A -red dashed line). Taken together, these results demonstrate that the representations learned by the recurrent units in trained single agent RNNs are not optimal for dual agent path integration.In contrast, trained dual agent RNNs can perform single agent path integration with decoding error < 0.10 m. (Fig. 3B -green dot). Fine-tuning the weights on single agent path integration further improves performance, even when W rec is frozen (Fig. 3B -green dashed line). This is aligned with recent experimental and computational work showing that fixed MEC connectivity and slow time-scale plasticity between MEC and hippocampus enables generalizable representations of new environments [38] . These results (which exhibit a similar trend when comparing the training loss instead of the decoding error-Fig. S3 ) demonstrate that dual agent RNNs have internal representations that are flexible enough to enable generalization to single agent path integration.To begin dissecting the representations that emerge in dual agent RNNs, we examine the functional properties of individual units in the recurrent layer. Previous work has found that single agent RNNs develop responses analogous to three functional classes of neurons in MEC: grid cells [5, 6] , border cells [7] , and band cells [8] . The distribution of these properties can be quantified by computing scores S2 for p-values).for each unit (Appendix D.2). While grid and border scores are standard metrics, to our knowledge no band score exists. We therefore developed a metric to quantify "banded-ness" (Appendix D.2).We find statistically significant differences in the distribution of these functional properties between single and dual agent RNNs (Fig. 4A, B ). In particular, we find dual agent RNNs develop units with lower grid scores, but higher border and band scores (Fig. 4A ). Visualizing rate maps associated with the dual agent RNN units that have the highest grid scores, we find that many have responses that are not solely constrained to vertices of triangular lattices, but are instead "honeycomb"-like (Fig. 4B ). This suggests that dual agent path integration leads to a weaker grid code. In contrast, dual agent RNN units with high border and band scores have more "ideal" responses, as compared to units of single agent RNNs, which have additional grid structure (Fig. 4B ). This demonstrates that dual agent path integration leads to a strengthened border and band code. These results are consistently observed across all 5 independently trained RNNs (Fig. S4 ).Prior work has found all three of these classes to be important for single agent path integration [28, 30, 31, 32, 39, 40] . To understand their importance in dual agent path integration, we perform targeted ablations, removing units with the highest grid, border, and band scores (Appendix D.3). We compare the decoding error of these RNNs to RNNs with randomly removed units, as this has been found to be a strong baseline [31, 35] . Unlike single agent RNNs, which have a significant increase in decoding error relative to the random baseline with the ablation of all functional classes, dual agent RNNs are less sensitive to the ablation of any one functional class (Fig. 4C ). This suggests that dual agent RNNs develop a more distributed and robust representation. We note that the greater (albeit, small) sensitivity of the dual agent RNNs to ablations of border and band cells are consistent with work arguing their driving role in path integration [30, 31, 39, 40] .To determine the extent to which our results are due to the nature of dual agent path integration, as opposed to specific choices of RNN architecture and hyper-parameters, we perform several control experiments. First, we consider the possibility that, by having twice as many agents to track, dual agent RNNs have half the effective size of single agent RNNs. In such a case, we might expect a smaller single agent RNN to generate responses similar to those of dual agent RNNs. However, half-sized single agent RNNs have representations that are distinct from the full-sized dual agent RNNs (Fig. S5A ). Second, dual agent rate maps are computed using the position of only one of the two agents (the other agent being "averaged out"-Appendix D.1). We re-compute the rate maps of dual agent RNNs, removing one of the agents completely. Thus, while the network was trained in the dual agent setting, the rate maps are only affected by a single agent. We find this to have little effect on the representations (Fig. S5B ). Third, we vary the agent whose position is being used to construct the rate map. We find that the rate maps are nearly identical (Fig. S6 ). Given that there is nothing in the loss function (Eq. 3) that distinguishes the two agents, it is not surprising that the dual agent RNN treats them the same. Finally, we measure the distribution of grid, border, and band scores for the RNNs trained with greater weight decay regularization (Fig. 2A -dashed lines). We find that the overall trend of dual agent path integration leading to lower grid scores and higher border scores, relative to single agent path integration, holds (Fig. S7 ).Taken together, these results demonstrate that the addition of another dynamic variable (i.e., a second agent) leads to differences in the distribution of functional properties, at the individual unit level, between single and dual agent RNNs.Recent neural recordings have demonstrated that, when mice are trained to navigate to reward sites that move in physical space, grid cells perform path integration in reference frames that are anchored to the locations of these rewards [41] . These results highlight the fact that hippocampus and MEC can perform computations in different coordinate systems [42] . Inspired by these results, we examine whether dual agent RNNs develop grid responses in another reference frame. One candidate coordinate system is defined by the relative positions of the two agents ("relative space"-Fig. 5A , Appendix E). We do not find evidence for units having grid responses in relative space (Fig. 5B, C ), demonstrating that dual agent RNNs are not simply using the same representations as in the single agent setting, but in a different reference frame. However, we find that units classified as border and band cells in the absolute (i.e., allocentric) reference frame have strong tuning in relative space (Fig. S8A ). Units with high border score have greatest activations when both agents are near borders (Fig. S8A, B ) and units with high band score have greatest activations when both agents are positioned at periodic distances, along parallel lines (Fig. S8A, B ). The latter could enable easy representation of parallel movement, where the agents are naturally separated. In contrast, units with high grid score have activations that are largely invariant to relative space (Fig. S8A, B ).To further characterize the extent to which dual agent RNNs encode relative position, we compute the spatial information [43] of all recurrent units relative space rate maps (Appendix E.2). We find that, while most units contain little spatial information in relative space (corresponding to spatial information ≈ 0), the distribution is heavy tailed (Fig. 5D ). Visualizing the relative space rate maps with the highest spatial information, we find that many are localized, with greatest activation when the two agents are near each other (Fig. 5E -center pixels). We hypothesize that, because accurate dual agent path integration becomes more challenging as the two agents get closer together, ablating units with high spatial information in the relative space reference frame would lead to an increase in decoding error. In-line with this hypothesis, we find that removing the units with the highest S3 for statistics).relative space spatial information leads to a statistically significant decrease in decoding performance, relative to the random ablation baseline (Fig. 5F , Appendix E.3). That this ablation leads to stronger deficits than the ablation of grid, border, and band cells (Fig. 4C ), emphasizes of their importance in supporting dual agent path integration. Further, it demonstrates that the dual agent RNN utilizes an efficient representation (encoding the relative position) to achieve dual agent path integration.Our comparison of the representations that emerge in single and dual agent RNNs has thus far been focused on the activations of individual recurrent units. Our findings that dual agent RNNs develop different distributions of functional classes than single agent RNNs, and that dual agent RNNs encode relative space spatial information, are suggestive that the network structure underlying single and dual agent RNNs are distinct. To further explore this hypothesis, we analyze the recurrent unit activations at the population level.Topological data analysis (TDA) [44] has become a widely used tool for studying neural population activity [25, 45, 46, 47, 48, 49, 50] . A core technique in TDA is persistent homology (Appendix F.1), which can be used to estimate the topology of the underlying neural manifold. Applying persistent homology, we find evidence that the population activations of single agent RNNs are constrained to a manifold having topology consistent with a two-dimensional torus (Fig. 6A , left-black arrows), consistent with prior characterizations [31, 32] . When performing the same analysis on the population activations of dual agent RNNs, we find differences in the persistent homology. In particular, the topological signatures of a two-dimensional torus are not clearly present in the persistence diagram (Fig. 6A , right). We see similar patterns in the persistence diagrams across different independently trained single and dual agent RNNs (Fig. S9 ). This difference in topology can further be seen by the fact that the persistence diagrams corresponding to single agent RNNs are closer (with respect to an appropriate metric [51] ) to the persistence diagrams of an idealized torus, than the persistence diagrams corresponding to the dual agent RNN are (Fig. S10 ). This suggests that dual agent RNNs do not have the same continuous attractor structure that single agent RNNs have [31, 32] . The topology and geometry of neural manifolds do not capture all properties of the underlying neural circuit. Indeed, RNNs with the same dynamical properties (e.g., existence of fixed point attractors, limit cycles) can be constructed to have different population activation geometries [52] . Similarly, RNNs with the same population activation geometries can be constructed to have different dynamics [52] . Recent work has developed a data-driven dynamical systems framework [dynamic similarity analysis (DSA)] for computing compact representations of RNN dynamics [53] . Therefore, we complemented our TDA results by applying DSA (Appendix F.2) to the population activations of single and dual agent RNNs. We find that independently trained single agent RNNs are dynamically more similar to each other than they are to independently trained dual agent RNNs, and vice versa (Fig. 6B ). Therefore, at both the topological and dynamical level, the representations learned by the single and dual agent RNN are different, suggesting that the addition of a second agent fundamentally changes the network structure that emerges.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include any theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
discussion: 
 Inspired by recent work demonstrating that brain regions believed to encode the self's position and trajectory (e.g., MEC, hippocampus) are modulated by the position and trajectories of others [9, 10, 11, 12, 13, 14, 16] , we trained a class of RNN models [28, 29] to path integrate two agents and investigated the representations that emerged. Given that these RNN models, when trained to path integrate single agents, develop properties that are similar to those found in MEC at the individual unit (e.g., grid, border, and band responses) and population (e.g., network structure with signatures of continuous toroidal attractors) level, this is a natural point for starting to address the larger question of how the MEC supports representation of multiple dynamic variables.We find that the representations that emerge in dual agent RNNs significantly differ from those of single agent RNNs. Dual agent path integration leads to weaker grid codes and stronger border codes (Fig. 4 ). This is consistent with electrophysiological recordings that have found MEC encodes distance of others to boundaries [9] and fMRI recordings that have observed a negative correlation between grid cell strength and performance on multi-agent tasks [10] . Functional class ablations and TDA revealed a distributed and robust representation, with no evidence for the population activity being constrained to a two-dimensional torus (Fig. 6 ). Despite this, dual agent RNNs are able to generalize to perform single agent path integration, even when the weights in their recurrent layer are frozen (Fig. 3B ). This demonstrates the flexibility of their representations [38] and is in-line with work finding that neural networks trained to perform multiple tasks develop abstract and distributed tuning [54, 55, 56, 57] . Finally, we found that RNNs trained on dual agent path integration develop tuning in the coordinate system defined by the relative position of the two agents (Figs. 5, S8 ). This is consistent with the tuning of neurons in anterior cingulate cortex to relative position between agents [15] , as well as is aligned with neurophysiological experiments that have uncovered neurons in MEC that encode distance and orientation of self to objects [58] .Limitations. The biological plausibility of the RNN models investigated in this work [28, 29] have been questioned [33, 34] (although, see response by Sorscher et al. (2022) [59] ). While our dual agent ground truth place cells were motivated by experimental work measuring hippocampal responses in environments with multiple animals [11, 12, 13, 14, 16, 17] , it represents only one possible choice (as discussed in Appendix B). Additionally, it has been found that the RNN model develops weaker modularization in grid properties [28, 29, 30] , a feature that is believed to be critical for encoding local spatial information [60, 61, 62] (although see work suggesting individual grid modules contain more spatial information than previously thought [63, 64, 65, 66] ). Recent use of loss functions that incorporate more than just path integration have found stronger grid responses and modularization [67, 68] . Future work should investigate the properties that emerge in these models when tasked with dual agent path integration. In our experiments, we did not account for noise in velocity inputs, which have been found to play a dominant role in human path integration error [69] , and are presumably higher for the assessment of others' velocities (given the lack of vestibular and proprioceptive feedback). We hypothesize that, as noise is added to the estimate of one agent's position and velocity, dual agent RNNs will develop representations more similar to single agent RNNs, suggesting a trade-off that should be explored.Predictions. Recent advances in experimental paradigms, where rodents must update their goal-based planning given the trajectories of others [70, 71, 72] , offer the possibility of comparing the in vivo representations of MEC with those generated by the dual agent RNN. Our results make several testable predictions about what responses might be found in MEC. First, we expect grid responses to be weakened, with the cells that do have high grid score exhibiting deviations from standard grid cells ("honeycomb"-like responses-Fig. 4 ). Second, we expect strengthened boundary and border responses (Fig. 4 ), with border cells responding most strongly when both agents are near borders (Fig. S8 ). Finally, we expect that the relative position between agents will be encoded in the a subpopulation of MEC neurons (Fig. 5 ) that is disjoint from cells with high grid scores (Fig. S8 ). Testing these predictions will provide insight into the validity of the RNN model, as well as our assumption that MEC can perform simultaneous path integration of two agents.Future directions. There are several impactful directions that our work can be extended. First, analyzing the emergent properties of RNNs trained to path integrate more than two agents can lead to a better understanding of how the MEC performs computations in the multi-agent setting. Second, reinforcement learning can be used to probe how the dual agent RNN representations might support higher order behavior [73, 74, 75] (such as shortcut taking in dynamic chasing [76] ). Third, theoretical work, showing-from first principles-what representations are necessary for single agent path integration [77, 78] can be extended to the dual agent path integration setting. This will provide a thorough treatment of how the complexity of multi-agent environments changes the computations needed to be performed by MEC. Finally, foundational work on MEC has found that grid cells encode abstract relationships beyond purely spatial variables [79] . These include social hierarchies [80] , sound frequencies [81] , list ordering distances [82] , and elapsed time [83, 84, 85] . Our work has implications for how the MEC may simultaneously integrate information across these different cognitive axes.Outlook. Continuous attractor networks (CANs) for the grid code [18, 19, 20] are one of the few canonical models in computational neuroscience. That we find RNNs trained to perform dual agent path integration develop structure distinct from CANs is noteworthy. There are several possibilities for reconciling our results with their differences to CANs, two of which we discuss below. First, the relative space encoding that we find could be coupled with a CAN so that the resulting network path integrates the other agent by first path integrating the self and then updating the other's position estimate by using the relative spacing encoding. This kind of egocentric framework is consistent with the presence of object vector cells in MEC [58] . Second, the grid code could be one of several codes that enables path integration, and scenarios (such as the presence of another) could lead to the reduction of the grid code for another, more suitable code. This is in-line with the negative correlation between strength of grid responses and success in a multi-agent task [10] .We hope that our work inspires new experiments to uncover the representations used by the brain to enable success in multi-agent environments.Elite sports players are often remarkably capable of keeping track of their own movements and the movements of others. For instance, soccer/football players are able to accurately pass the ball, while running, to a teammate who is also running. While these are individuals that have trained in specialized settings for years, the "average" person has every day events that require the navigating in environments with moving agents. As an example, changing lanes on the highway requires maintaining an understanding of where one's car is, as well as other nearby cars.While these experiences highlight that humans are able to accurately keep track of the trajectories of themselves and others, they do not definitively demonstrate that we are simultaneously performing dual agent path integration. We know of no such work that has directly addressed this question, however we hypothesize that path integrating multiple agents is possible. We consider two characteristics of path integration: 1) the ability to update an estimate of location without direct sensory inputs (e.g., maintaining an accurate estimate of position in the dark); 2) the ability to take direct routes. We argue that there is anecdotal and experimental evidence supporting that both of these can occur in dual agent settings. First, if we turn around after having passed someone in the hallway several seconds previously, we may surprised where they are (e.g., they are all the way down the hall, they are almost exactly where they were when we saw them). This suggests that we are maintaining an estimate of where another individual is through an estimation of their movement, all without direct visual inputs. And second, rats trained to perform pursuit of a laser pointer showed the ability to take "shortcuts" that were predictive in nature for specific types of laser pointer trajectories [76] . This suggests that the rats are able to leverage dynamical predictions of where the other is going.If dual agent path integration is performed in the brain, what areas might be actively supporting the computations involved? We hypothesize that the medial entorhinal cortex (MEC) plays an important role. Both the hippocampus and anterior cingulate have been shown to be modulated by the presence of others [11, 12, 13, 14, 15, 16, 17] .Given the inter-connectivity between MEC and these brain regions, it seems likely that the MEC is also modulated. Given that the MEC is believed to perform single agent path integration, we expect that the modulated activity may reflect its role in path integrating multiple agents. In support of this, Stangl et al. (2021) [9] found that the MEC had similar border representations for both the self and other. However, because of the lack of experimental work in this direction, we cannot rule out that other brain areas are also (or primarily) involved in dual agent path integration.Finally, we note that grid cells, and path integration more broadly, have increasingly been recognized as being computational mechanisms that are used for more than purely physical location [79] . The ability to update estimates of location in abstract spaces (e.g., frequency space [81] ) suggests other evidence for being able to perform dual agent path integration. For instance, musicians can simultaneously play two melodies. Thus, future work may probe this capability in settings involving experimental variables beyond space.Because our dual agent RNNs, which assume the existence of dual agent path integration, make predictions that can be used to test this hypothesis, we believe they will be useful in guiding experiments that probe the nature of multi-agent spatial coding. Reduced weight decay regularization: Previous work has introduced weight regularization to the weights in the recurrent layer, W rec , to encourage sparsity and aid generalization (Eq. 3 [32] . Given that we expect dual agent path integration to be a more challenging task, we hypothesize that the weight decay value used for single agent path integration might lead too much sparsity to perform the dual agent path integration task well. We therefore reduced the value used for weight decay, taken as an input in the Adam optimizer torch.optim.Adam, from λ = 10 -4 to λ = 10 -6 .Table S1 : Parameters used to train RNNs on single and dual agent path integration. Unless otherwise noted (in the text), these are the values the parameters used to train single and dual agent RNNs take. Parameters that take values that are different from the defaults of previous models [29, 32] are bolded. Any values not listed are the same as they were in previous work [29, 32] . The training loss associated with single and dual agent path integration is presented in Fig. S1 . We note that there is no equivalent to "test" loss, as the trajectory used to train each iteration was sampled randomly from a distribution [29] . No trajectories were separately maintained in a "test set".All training experiments and analysis were performed using an AWS GovCloud g4dn.xlarge instance with 1 GPU and 4 CPUs. When deciding on how to define p, the "true" place cell activity, in the dual agent setting, we considered two important experimental and model based constraints:• Zero mean input: Previous work considered the DoG functional form for p because it maintains a zero mean across space [29] . This was motivated by earlier work that showed how non-negative PCA could learn grid cell like responses from place cell input that was assumed to be zero mean [86] .Given that prior observations of the RNN model failing to develop grid cells [33] came in part because of architectural decisions that were not supported by the original theory [59] , we sought to ensure that our definition of p maintained "zero mean-ness" so that the lack of grid-ness we observe could not be due to our choice of p. Note that this rules out the use of certain non-linearities, such as taking the maximum value of the place responses evaluated for each agent independently and functions that saturate (e.g., sigmoid).• Mixed selectivity: Neural recordings have identified multiple variables of self and other being encoding in the hippocampus and anterior cingulate being represented by individual neurons. For instance, Zhang et al. (2024) [17] found that there were more CA1 neurons that were tuned to the location of self and other than there were tuned to the location of only the other. Similarly, Yoo et al.(2021) [15] found the majority of neurons encoded information about self and other's position during a virtual pursuit task. Thus, while these results do not rule out the possibility of a subpopulation of "pure" social place cells that respond only to the position of others [87] , modelling spatial representations in the multi-agent setting should include mixed selectivity.Because of these two constraints, zero mean input and mixed selectivity, we chose to model p as the linear sum of the DoG responses applied to the location of each agent separately. That is, pi(t) = DoG[z1(t), ci] + DoG[z2(t), ci].Note that this assumption of linearity and complete mixed selectivity is almost certainly an oversimplification, however we believe that it is a valuable place to start in this first work on exploring the kinds of representations that are useful in dual agent path integration. We believe future work exploring how inclusion of non-linearities and the existence of both disjoint and mixed place cell responses further shapes the representations that emerge in path integrating RNNs in multi-agent environments.Let W in denote the weights projecting from the inputs to the units in the recurrent layer. Given the difference in input dimension between single and dual agent RNNs , modifications to W in is necessary to enable an RNN trained on single agent path integration to be tested on dual agent path integration (and vice versa).Going from single to dual agent path integration (Fig. 3A ), we concatenate W in with itself, forming a newW in = [W in , W in ] ∈ R n G ×4. Going from dual to single agent path integration (Fig. 3B ), we remove the weights corresponding to the third and fourth inputs. That is, ifWin = [W in 1 , W in 2 , W in 3 , W in 4 ], where W in i corresponds to the weights projecting from input i to all units in the recurrent layer, then we generate a new set of weights Same as Fig. 3 , but when plotting training loss. Note the difference in scales is due to the fact that the dual agent RNN has summed ground truth place cell activity.W in = [W in 1 , W in 2 ] ∈ R n G ×2. Because all other training and architecture hyper-parameters are the same between RNN models trained on single and dual agent path integration, no other modifications were considered.To freeze the weights in the recurrent layer, we set those parameters to have requires_grad = False.The construction of rate maps, which visualize activity of individual neurons , as a function of physical location, is a standard tool used to understand the neural representations underlying spatial navigation. In the context of RNNs trained on single agent path integration, this tool has been used to discover grid, border, and band cells [28, 29, 30, 31, 32] . To do this, nt trajectories, corresponding to the paths of a single agent, are generated and used as inputs to RNN models. The activations of each unit in the recurrent layer and the position of the agent at each time point is saved. The average activation value, for a given position, is then computed and plotted.Position is resolved at a resolution of m bins in the xand y-direction, leading to m 2 bins in the rate map.For the dual agent setting, we similarly sample nt trajectories, use them as inputs to dual agent RNNs, and save the activations. The primary difference for constructing rate maps for dual agent RNNs is that only the position of one of the agents is used to construct the maps (and not both). By default, we take the first agent's position. However, since there is no difference between the agents, this choice should not have any major effect.To better understand the extent to which this choice for visualizing the rate maps affects our interpretations, we also compute rate maps where only a single agent's trajectory is sampled. That is, we trained an RNN on dual agent path integration, but then sample only a single agent's trajectory when constructing the rate maps, setting the inputs corresponding to the movement direction and speed of the second agent to 0 (we refer to this approach as "Dual agent RNN single agent rate map"). We find that, in general, this leads to similar rate maps (compare Fig. 4B with Fig. S5B ).For all rate maps, we set nt = 1000 and m = 20. We note that it is typical for rate maps to be smoothed by applying a filter after their construction. However, we chose not to smooth the rate maps (for either single or dual agent path integration) to ensure that the interpretations of the representations we studied were being driven by filtering.Visual inspection of rate maps have led to the identification of subsets of neurons and hidden units that have distinct properties (e.g., "grid-like"). To quantify these properties and enable unsupervised identification, metrics have been developed. These are discussed in detail below: Grid score: Ideal grid cells are characterized by having firing fields distributed along the vertices of a triangular lattice [5, 6] . For neural data, the grid signature has been found to be more strongly apparent when considering not the rate map, but the spatial autocorrelogram (SAC) of the rate map. Let Xi denote the rate map corresponding to unit i in the recurrent layer, and Si denote its SAC. If Si has strong triangular lattice structure, rotating it by 60 • (which we denote by S 60 i ) should lead to a SAC that is highly correlated with the Si. This is similarly true for rotations of 120 • and 240 • . High correlations could also arise when there is a finer symmetry. To identify when this is the case, the correlation between S 30 i , S 90 i , and S 150 i can be taken. The grid score, gsi, is computed asEQUATIONwhere corr(Si, S θ i ) = ( j Si,j Sθ i,j )/ j ( Si,j) 2 , with Si,j being the j th bin of the SAC of hidden unit i, with the center peak masked out.Border score: Ideal border cells are characterized by having their activations selectively aligned along a border. To compute the border score [7] , the rate map corresponding to unit i in the recurrent layer, Xi, is first thresholded, with any spatial bin having activation < 0.3 • max(Xi) being set to 0. Connected components of this thresholded rate map are found using scipy.ndimage and those with area < 200 cm 2 are additionally set to 0 [7] . Let Xi = Xi ⊙ Mi correspond to this masked out rate map, where Mi ∈ {0, 1} m×m is the mask and ⊙ is the element-wise multiplication operation. The maximum coverage along any of the four environment walls, c M i , is computed by finding the number of non-masked out spatial bins along each wall (i.e., Mi(x, y) = 1) and dividing it by m. Finally, the mean firing distance, d M i , is computed for each remaining connected component is by taking the mean of the distance of each spatial bin in the connected component to the nearest wall, weighted by the normalized activation at that spatial bin. d M i is then defined by taking the average value across all connected components. The border score, bosi, is the computed as Band score: To the best of our knowledge, no metric that explicitly quantifies the "banded-ness" of neural activity or hidden unit activations exists. We therefore created one to enable the exploration of how adding an additional agent affects the development of units with band structure in their rate maps. As with grid and border scores, let Xi denote the rate map corresponding to unit i in the recurrent layer. Let X(kx, ky) be a two-dimensional sinusoid with frequency kx, in the x-direction, and ky, in the y-direction. EQUATION)where corrcoef is numpy's Pearson product-moment correlation coefficient.To probe how units in the recurrent layer are used in the computations underlying the ability to path integrate one or two agents, we perform functional class targeted ablation experiments (Fig. 4C ). To do this, we rank the units based on their functional properties (grid, border, or band score). We then ablate units with the highest p% of the scores by setting all incoming and outgoing weights to 0. That is, if unit i is determined to have a sufficiently high score and is to be ablated, we set W rec i,j = W rec j,i = 0, for all j = 1, ..., nG. Here, W rec are the weights in the recurrent layer.To construct the traditional rate maps used to visualize grid, border, and band cells (Fig. 4 , Appendix D.1), the xand y-coordinates of the environment were split into bins and the average activity, per spatial bin, was computed using one of the agent's xand y-position.To examine whether there was any tuning in the relative position space, we construct rate maps using not x and y but ∆x = x1 -x2 and ∆y = y1 -y2, where (x1, y1) and (x2, y2) are the locations of the two agents (the ordering is arbitrary, but fixed). This provides a transformation from the allocentric reference frame to the relative space reference frame (Fig. 5A ). Note that this transformation is not one-to-one, as multiple configurations of agents position in allocentric space have the same relative space coordinates. Note that relative space reference frame has twice the range, in each dimension, as allocentric space. This is because, whilex, y ∈ [-L/2, L/2], ∆x, ∆y ∈ [-L/2, L/2].Interpreting the rate maps constructed in the relative space can be slightly un-intuitive. To aid in this, we provide illustrations of three relative space rate maps, with specific locations of the rate map visualized in the allocentric reference frame (Fig. S8B ). As noted above, because the mapping is not one-to-one, these are not the only possible realizations of the allocentric space. From these examples, we see how a unit in the recurrent layer that is identified from its allocentric rate map as a grid cell has the same average activity across a wide range of relative positions of the two agents (Fig. S8A , left). In contrast, the example border cell responds only when both agents are along the borders (Fig. S8A , middle), which can be seen by the activations being restricted to when ∆x is at its minimal and maximal values. For an example band cell, again there is specificity in the response related to the value of ∆x, with activity occurring only at certain intervals (Fig. S8A , right). In between those intervals, there is little activation.To quantify the amount of information neural responses encode about specific variables, the metric spatial information was developed [43] . In particular, the spatial information, SIi, encoded by the rate map corresponding to unit i in the recurrent layer, Xi, is computed byEQUATIONwhere Xi is the mean value of rate map i over all space, x and y correspond to the m spatial bins in each direction that are used to construct the rate map, and p(x, y) is the probability that the agent was in the spatial bin (x, y).We compute the spatial information on the relative rate maps (Figs. 5, Appendix E.1). While a useful tool, the spatial information can be skewed by units that (spuriously) respond highly to only a single spatial bin (see Fig. 5E , top left rate map). Additionally, the multiplicative dependence on p(x, y) can lead to high spatial information in specific bins, when the probability is skewed. Indeed, we find that p(x, y), in the relative space, is heavily distributed towards the center of the rate maps, corresponding to the two agents being near each other. Thus, while a useful tool for enabling us to identify a new functional class of units, these results do not rule out other interesting responses in relative space (or other reference frames). Table S3 : Relative space ablation statistics. Mann-Whitney test p-values for dual agent RNN model decoding error (Fig. 5E ) for units with the highest relative space spatial information ablated (Appendix E.3). Details same as Table S2 .% units removed p-value To probe the importance of the units in the hidden recurrent layer that encode relative space, we perform the same ablation studies as described in Appendix D.3, but ranking units based on the spatial information present in their relative space rate maps, as opposed to their grid, border, or band scores.TDA [44] has become a widely used tool in neuroscience [25, 45, 46, 47, 48, 49] . In particular, it has been leveraged to provide evidence for the population activity of grid cells (from the same module) being constrained to a two-dimensional toroidal manifold [25] . A core tool in TDA is persistence (co)homology, which is used to identify topological structure present in data. Persistence (co)homology, applied to neural activity, can loosely be understood by the following overview. First, the activity of all N neurons, at each time point, is viewed as a point in the space of all possible population activations. This leads to a point cloud in N -dimensional space. Balls, of dimension N , are then centered on each point in the cloud. Gradually, the radii of these balls is increased. When two balls intersect, they are considered a connected component. The topological properties of all connected components is then computed. In particular, the existence of loops (H1) and two-dimensional cavities (H2) are extracted and the persistence of these features is measured. This can be visualized in a format called a persistence diagram.We computed the Vietoris-Rips persistence diagram corresponding to the activations of all recurrent units for both single and dual agent RNNs. To do this computation, we used the Ripser python package [88, 89] . However, persistence diagrams can be subject to noise and are more challenging to compute for high-dimensional spaces. Therefore, to ensure that the differences in topology observed between single and dual agent RNNs was due to different properties underlying the population activations, and not noise, we projected the population activations onto a lower-dimensional subspace and re-computed the persistence diagrams. There are several possible choices for reducing the dimensionality. A natural option is principal component analysis (PCA). However, we found that, for the single agent RNN population activations, the subspace identified by PCA does not enable good multi-output regression with the agent's location Using partial least squares (PLS) [90] , which has been successfully applied in neuroscience applications [91] , we found that both single and dual agent RNN population activation, captured in a 10-dimensional subspace, can achieved high multi-output regression with the agents' positions. Therefore, we use PLS to reduce the dimensionality of the population activations, and then computed the persistence diagram on this reduced subspace (Fig. 6 ). The PLS was computed using PLSRegression from sklearn.cross_decomposition. Additional examples of persistence diagrams, corresponding to the activations of single and dual agent RNNs, are shown in Fig. S9 .To understand how the topology of the population level activations of the single and dual agent RNNs evolves with training, we computed the persistence diagram associated with an "ideal" two-dimensional torus and measured the topological distance [51] between the observed persistence diagram and the ideal persistence diagram. We find that the topology of the single agent RNN is closer to an idealized torus than the topology of the dual agent RNN (Fig. S10 ), further supporting our conclusion that the dual agent RNN does not develop the same attractor structure as the single agent RNN.The evolution, in time, of the activations of units in RNNs can be viewed as a dynamical system. The features of this dynamical system (e.g., fixed points, limit cycles, quasi-periodic attractors, chaos) are dependent on the inputs and structure of the RNN. Thus, characterizing corresponding dynamical properties offers the ability to gain insight into the organization of the RNN. Unsurprisingly, use of geometric and topological methods to interrogate RNN population activation can fail to properly identify similar and dis-similar dynamics, as they are sensitive to the underlying manifold the activation evolves along [52] . Recent work [53] has used data-driven tools [92, 93, 94] from Koopman operator theory [95, 96] to extract properties of RNN dynamics that can be compared using a precise notion of equivalence, called "topological conjugacy" [97] . Similar approaches have been used to study the dynamics associated with training deep neural networks [98] , as well as iterative optimization algorithms, more generally [99] . Therefore, we complimented the geometric and topological analysis performed on the structure of the population level representations (Fig. F.1) with dynamical similarity analysis (DSA) [53] .To perform DSA, K ∈ N trajectories in physical space, corresponding to K trajectories in population activity space, are sampled. Each trajectory has temporal length T ∈ N. All the activations are collected into a tensor, X ∈ R K×T ×n G , where nG is the number of units in the recurrent layer. We then perform principal component analysis (PCA) on X, reducing the number of dimensions so that we get a new tensor X ∈ R K×T ×n M , where nM < nG. Next, D ∈ N time-delays are then applied, where D < T , and the resulting tensor H ∈ R K×(T -D)×n M D is then flattened along its first dimension. Dynamic mode decomposition [100] is then performed to get a compact representation of the dynamics in terms of the Koopman operator [101] , which is restricted to have rank nR < nM D. The Koopman-based representation between the dynamics of the single and dual agent RNN are then compared, via the "Procrustes over vector field" metric [53] .As can be seen in the brief overview of DSA, there are several hyper-parameters whose value must be chosen. These include K, D, nM , and nR. To be consistent with the geometric and topological analysis, we set K = 1000. Because the length of the paths (T = 20) is small, we cannot use too many time-delays. Therefore, we set D = 4. Because a small number of dimensions were necessary to capture most of the variance of the activations in both single and dual agent RNNs, we considered nM = 10 and nR = 30 (Fig. 6 ). To determine whether this choice affected the conclusion that single and dual agent RNNs have different dynamical properties, we performed the analysis again, with nM = 50 and nR = 100. Fig. S11 shows a similar conclusion to Fig. 6 , with the Procrustes analysis over vector field metric being nearly 2× larger for comparison between single and dual agent RNNs than between single and single agent RNNs or dual and dual agent RNNs. However, because more modes are kept (nM = 50), more noise is present, and the difference is weaker than when fewer modes are kept. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Detail on the modifications used to enable existing RNN models to perform dual agent path integration are presented in Sec. 2. Full details regarding the parameters used to train the RNN models and details on the analysis used to interrogate the experimental results are presented in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the link to our publicly available Github repository in the Appendix. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The hyper-parameters used for training the RNN models are presented in Table S1 . The base code is linked to in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. S2, S3 ).• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Information on the AWS GovCloud instance used for all training and analysis was reported in Appendix B.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The code base that was used to evalaute single agent RNNs and develop dual agent RNNs is linked in Appendix B and the related papers are cited throughout our work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide the link to our publicly available Github repository in the Appendix.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing nor research with human subjects was performed.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No human or animal work was performed in this paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.While there is no direct evidence for simultaneous dual agent path integration, we hypothesize that the brain can perform it and that the MEC is involved. See Appendix A for our rationale. 
 
 Appendix:  


Decision: This paper examines the spatial representations that emerge in RNNs trained to do "dual path integration", i.e. to perform path integration for two separate agents moving through space. The authors compare these dual path integrators to single-agent path integration models, and show that they develop distinct spatial representations. Specifically, the dual agents show less "classic" grid cell activity (both in terms of single neurons and population-level torus manifolds), and more relative encoding of spatial location for the two agents. These results have potential implications for the encoding of space in social contexts.

The reviewers raised some concerns about this paper, some of which were fairly consistent across reviewers. In particular, multiple reviewers asked why summed place cell activities should be used and why we should entertain the assumption that the entorhinal cortex may engage in path integration for others. The authors provided some additional reasoning on these questions, and also attended to other comments in the rebuttal. Following the rebuttal period, the final average score was 6, putting this paper in the accept range. The AC felt that the paper makes an interesting contribution, so given this and the final scores, a decision to accept was reached.

Reviews: review: paper_summary: paper_summary: This paper studies the internal representations of recurrent neural networks that have been trained to path integrate two agents simultaneously, based on the hypothesis (and related experimental evidence) that individual agents account for the positions of others in multi-agent environments. The authors augment existing single agent vanilla discrete-time RNN models of path integration to perform the same for two agents. Through several numerical experiments on these models, the authors show that grid cell responses are weaker in the dual agent case compared to the single agent case, while border and band cell responses are stronger. Then they show that these dual agent RNNs encode information on the relative position of the two agents. Furthermore, they show that networks capable of performing dual agent path integration can generalize to the single agent task, while the reverse does not hold in practice. Finally, they outline testable predictions of their model (weaker grid responses, stronger border and band cell responses, tuning for relative position) and future directions for research on the computations underlying spatial navigation.
strength_weakness: strength_weakness: 1. The multi-agent experiments are limited to dual agent path integration. Studying briefly the strengths of grid and border cell responses in the 3-, 4- and perhaps 5-agent cases would strengthen the paper, given that the code framework seems flexible. This is indeed a limitation/direction for future work that the authors themselves have acknowledged.
2. The RNNs used in the experiments are not noisy and are simple vanilla discrete-time RNNs. It would be good to study the interplay between recurrent noise and the grid and border scores. It might also be interesting to analyze continuous-time RNNs (where noise improves stability and robustness, see Lim et al., 2021 [1]).
3. Certain architectural assumptions could be justified better:
    1. The results and predictions of this work are strongly linked to those of Sorscher et al., 2021 [2], which may not be the only method by which grid cell representations emerge in a neural circuit. The underspecification of the problem means that alternative hypotheses [3,4] exist that can explain the emergence of grid cells, and these could be studied in future work.
    2. Is there any biological evidence that the same neural circuits perform path integration simultaneously for multiple agents? Could a single agent network not be used to estimate the positions of multiple agents separately and sequentially?
    3. Are there cases where the two agents are present in adjacent/extremely close locations but the $k$-means clustering does not decode the position of one agent correctly (since it always identifies 2 centers from the top-2$n_d$ active output units)? More generally, could the authors elaborate on the accuracy of path integration when the agents are close together?
    4. Have the authors considered alternatives to the "social" summed place cell activations? Perhaps having twice the number of output units instead (although, this is admittedly not as scalable to multi-agent cases)?

**References:**
1. Lim et al. "Noisy recurrent neural networks." Advances in neural information processing systems 34 (2021).
2. Sorscher et al. "A unified theory for the origin of grid cells through the lens of pattern formation." Advances in neural information processing systems 32 (2019).
3. Schaeffer et al. "Self-supervised learning of representations for space generates multi-modular grid cells." Advances in neural information processing systems 36 (2023).
4. Xu et al. "Conformal Normalization in Recurrent Neural Network of Grid Cells." arXiv preprint arXiv:2310.19192 (2023).
questions: questions: See the Weaknesses section.
limitations: limitations: The authors have adequately discussed the limitations of the work, but could potentially elaborate on the effect of and limitations related to certain architectural assumptions (see W3).
paper_summary: paper_summary: The authors study neural representations of space in artificial agents that perform path integration of the positions of two agents simultaneously. This is motivated by recent neuroscience studies showing place cells that respond to the location of nearby animals. The authors feed the velocity and head direction inputs of both agents into an RNN and train the network to generate the place cells. Similar to previous work with single agent position, they evaluate the spatial tuning of neurons in the RNN, specifically looking for border cells and grid cells. Unlike in the work with a single agent position, they found fewer grid cells. This indicates that representing the positions of two agents relied on a somewhat different neural code than representing the position of a single agent.
strength_weakness: strength_weakness: There is no attempt to analytically derive or explain the obtained result. It remains unclear whether changing some of the hyperparameters could have led to a different result. If the experiments were complemented with derivations, like in some previous work on the emergence of grid cells, that would strengthen the paper.
questions: questions: Do authors consider a possibility biological systems might not really perform dual "path integration" (they could infer the position of other agents using visual inputs rather than using path integration)?

Is it possible that populations of the cells encoding each agent's position would be entirely or somewhat separate? Would that lead to emergence of the grid code?
limitations: limitations: Authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper studies RNNs trained to path-integrate the position of two agents. They show the network behaves differently to similar networks trained to path-integrate the position of a single agent, and make some neural level predictions.
strength_weakness: strength_weakness: One big choice that seems likely to have heavily influenced the learnt representations is the choice of mixed place cell output. Surely when another animal appears in my scene I am still able to report a factorised encoding of my own position? If you make this change, and ask the network to output two factorised codes, do your results change? If so this risks becoming quite a specific point, though it would then be interesting to argue about the causes of the mixing or not mixing.

If I understood correctly, the position decoding using that k-means approach is never used to train the network, it is merely the visualise the position estimate. If this is correct, it seems a strange metric against which to compare networks, e.g. figure 3, without further comment. Rather, the key metric is the loss that is used to train the network, do the same trends hold there?

Largely, this paper seemed to say 'here's a different setting, in which things are different'. The path-integrating literature is great because we know so much about how this circuit works, and it has such a correspondence with biology. Understanding in more detail how this path-integrator you present works, for example examining the connectivity structure, or the velocity update scheme as people are able to do for traditional CAN models, would significantly improve the impact of this work, as it seems comparisons to existing neural data are not currently possible.

Personally, I agree with the critiques of the Sorscher framework, there seem to be others that match more grid cell characteristics, but its certainly a good starting point.
questions: questions: In the 2-agent case you showed the tuning curves to one agent's position. I really wanted to know whether the tuning to the two different agents was similar in the same cell? I don't think this was reported anywhere.
limitations: limitations: The authors discuss their limitations.

Comments: title: Response Response
Response
comment: That's a fair point. We will work on exploring the impact the readout structure has and discuss this in more detail in the revised manuscript (Sec. 4). 

Thank you for your time and your efforts to improve our work!
Ooooo, your second point is a good point - hadn't thought of that...

Nonetheless, it remains an issue to decide the best readout structure to use, and it worries me the extent it might be determining the claims about different path-integrating mechanisms. Evidence of neurons encoding both agents, e.g. in hippocampus, could either be outputs of a network like yours, or a system that switches which agent it is encoding, or just part of readout that is only partially overlapping in the manner you suggest but elsewhere there might be self vs. other neurons. Perhaps the bat results constrain this more. But, since all reviewers are recommending acceptance it seems like you will get in, if you wanted to increase impact on neuroscientists before publication this would be where I would sure things up - see if you conclusions generalise to completely disjoint output codes, or mixtures.

And I take your third point, that is constraining facts about the computation going on in this RNN in interesting ways.

And the Wagner point is interesting.

Thanks for your engagement
Thank you, I appreciate the clarifications.
We thank the reviewer for their continued engagement with our submission and we respect that they maintain a high bar for novel/interesting work. We believe a few additional comments may help to clarify the questions addressed by the reviewer. 

First, there is no neural evidence of two agents being simultaneously represented. However, we believe that there are anecdotal examples that suggest it possible (e.g., the soccer player that has to make actions based on their and others positions', the driver that has to keep track of their speed and others speeds'). Our rationale for discussing the tuning of MEC for both self and other [Stangl et al., (2021); Wagner et al., (2023)] was to suggest that *if* the brain does simultaneously represent multiple agents, a reasonable candidate for where this would occur is MEC. 

Second, at the start of each trajectory, the true starting position of both agents is encoded into the initial state of the RNN through the learnable weights $\textbf{W}^\text{back}$. From this initial state, the dual agent RNN must correctly integrate agent 1's velocity to update agent 1's position, and integrate agent 2's velocity to update agent 2's position. Thus, while we do not distinguish between self and other in this work, we do enforce that the RNN has some notion of which "bump" is which, as the correct inputs must be integrated to update its position.

Third, in our work we have identified that some units in the dual agent RNN develop tuning for the relative position of the two agents. This enables flexible and efficient computations, such as encoding when both agents are near the borders (Fig. S5), when both agents are moving parallel to each other (Fig. S5), and when both agents are near each other (Fig. 5). We believe that the representation of relative space is useful for tasks beyond dual path integration, such as one agent pursuing another. These are not computations that would emerge in a model that featured two separate CANs.

Finally, we agree that our work would be strengthened by having neural results to compare to. We hope that our results inspire experimentalists to examine MEC in multi-agent settings so that such comparisons can be done in the future. However, we note that work by Wagner et al. (2023) found weakened grid cell responses correlated with better performance in a task where one agent had to track another agent's position. Why this would be is not readily apparent, and we know of no attempts to model this. That our dual RNN model develops weakened grid responses demonstrates an alignment with existing neural data (albeit, neural data that was recorded with fMRI) and suggests that grid cells may not be optimal for representing two agents simultaneously.
Thank you very much for your kind words, time, and consideration. We appreciate your encouraging words!
Thank you for your thorough response and engagement. Thank you for the additional plots in the rebuttal pdf, they are appreciated. 

First, on the evidence of MEC doing two agent integration: the existence of cells tuned to other agents does not imply dual-agent integration. The same single-agent integration system could just switch the agent that it is representing over time. Is there neural evidence of two agents being simultaneously represented?

Second, there's something about the mixed-cell response that I am still not getting. Sure, if the two agents' trajectories are well-separated and correctly tracked then the two place cell outputs will be separated. But, crucially, how does the agent know which place cell bump is its own, and which is the other agent's? This seems vital, but I think the loss does not force it to be true, your decoder is ambiguous to which agent is where, and the cellular responses don't seem to distinguish the agents. If I've not misunderstood, then this is very worrying! We are certainly capable of distinguishing whether it is me or someone else who is in each position, leaving this choice of setting to be of questionable relevance for the brain?

Third, the authors argue that many CANs path-integrating separately is not scalable. Perhaps. But it's not clear to me there are good alternatives? Surely, in some fashion, the same computations must be going on in your network, which you find struggles to train beyond 2 agents perhaps due to lack of scalability. If not, what fundamentally different thing do you think your network is doing allows it to scale (if indeed it can). 

I agree it is interesting how a neural circuit solves this problem, and I apologise for some of my review's harshness. Perhaps I have the wrong threshold for sufficiently interesting, but I think to increase my score I would need some more compelling match to neural data, or understanding of the computation this different network is doing. That is unreasonable to ask for in a rebuttal, leaving us at an impasse I'm afraid.

Please do correct me if I have been stupid.
I thank the authors for their rebuttal, which contains important clarifications. While I do agree with other reviewers that the results are tied to the assumptions made, I believe the authors have provided reasonable justifications for their choices, and further experimental results would be required to truly validate the claims (and so I believe this paper could motivate experimentalists to pursue targeted multi-agent experiments). I maintain my score and overall positive opinion of the paper, and think it is a valuable contribution to NeurIPS.