Stable cognitive maps for Path Integration emerge from fusing visual and proprioceptive sensors
Authors: Arnaud Fanthomme
Rémi Monasson
Venue: ICLR 2022 Submitted (2022)
Keywords: RNNs

Abstract: Spatial navigation in biological agents relies on the interplay between external (visual, olfactory, auditory, $\dots$) and proprioceptive (motor commands, linear and angular velocity, $\dots$) signals. How to combine and exploit these two streams of information, which vastly differ in terms of availability and reliability is a crucial issue. In the context of a new two--dimensional continuous environment we developed, we propose a direct-inverse model of environment dynamics to fuse image and action related signals, allowing reconstruction of the action relating the two successive images, as well as prediction of the new image from its current value and the action. The definition of those models naturally leads to the proposal of a minimalistic recurrent architecture, called Resetting Path Integrator (RPI), that can easily and reliably be trained to keep track of its position relative to its starting point during a sequence of movements. RPI updates its internal state using the (possibly noisy) proprioceptive signal, and occasionally resets it when the image signal is present.  Notably, the internal state of this minimal model exhibits strong correlation with position in the environment due to the direct-inverse models, is stable across long trajectories through resetting, and allows for disambiguation of visually confusing positions in the environment through integration of past movement, making it a prime candidate for a \textbf{cognitive map}. Our architecture is compared to state-of-the-art LSTM networks on identical tasks, and consistently shows better performance while also offering more interpretable internal dynamics and higher-quality representations.

Under review as a conference paper at ICLR 2022
STABLE COGNITIVE MAPS FOR PATH INTEGRATION EMERGE FROM FUSING VISUAL AND PROPRIOCEPTIVE SENSORS
Anonymous authors Paper under double-blind review
ABSTRACT
Spatial navigation in biological agents relies on the interplay between allothetic (visual, olfactory, auditory, ... and idiothetic (proprioception, linear and angular velocity, signals. How to combine and exploit these two streams of informa- tion, which vastly differ in terms of availability and reliability, is a crucial issue. In the context of new two roo-dmmsiiooal continuous environment we developed, we propose a direct-inverse model of environment dynamics to fuse image and action related signals, allowing reconstruction of the action relating the two suc- cessive images, as well as prediction of the new image from its current value and the action. The definition of those models naturally leads to the proposal of min- imalistic recurrent architecture, called Resetting Path Integrator (RPI), that can easily and reliably be trained keep track of its position relative point during a sequence of movements. RPI updates its internal state using the (possibly noisy) self-motion signal, and occasionally resets when the image sig- nal is present. Notably, the internal state of this minimal model exhibits strong correlation with position in the environment due to the direct- sirect-inverse models, is stable across long trajectories through resetting, and allows for disambiguation of visually confusing positions in the environment through integration of past move- ment, making it a prime candidate for cognitive map. Our architecture is com- pared to off-the-shelf LSTM networks on identical tasks, and consistently shows better performance while also offering more interpretable internal dynamics and higher-quality representations.
starting
CONTEXT
The Path Integration task. Path Integration (PI), a task in which an agent has to integrate infor- mation about a sequence of movements to keep track the distance between its current and initial positions, has been extensively studied both in rodents (Etienne & Jeffery 2c
al. 2006). and in artificial systems (Arlco et al. 20000: Banino et al.) 2018 Zhao et al., (2020). and is thought by many be essential ingredient in the elaboration of cognitive maps (Tolman 1948
Redish & Touretzky 1997), that is, internal representations of the spatial structure of an environment capable of supporting navigation tasks (Golledge 2003). Path Integration is particularly relevant from the point of view of representation learning as it relies on the interplay between qualitatively different inputs, subject known as multi-modal learning and recently reviewed by Summaira et al. (202T). Those inputs can be broadly categorized into two groups On the one hand, idiothetic signals, such as velocity (Kropff et al., 2015), head direction (Taube al. 1990), memory of past trajectories (Cooper et al., 2001) reafferent copies of motor signals (lacoboni et al. (000). which are generated by the agent itself. On the other hand, allothetic cues, e.g. provided by vision (Etienne) ct al. 1996), olfaction or "whisking" in mice (Deschénes et al. 2012) are intrinsically related to the external environment.
The simplest solution implement PI would be an integrator network that takes as inputs proprio- ception signals, or, equivalently, the agent's deeendennddnn velocity. While the theory of integrator networks and the representations that emerge have been well studied (Seung 1996] Fanthomme & Monasson 202I). such solution suffers from two major limitations. First. the accumulation of errors across the trajectory, either coming from imperfect sensor information from imperfect

---

Under review as a conference paper at ICLR 2022
integration, would make it unsuitable represent Path Integration on arbitrarily long trajectories. Second, even if integration could be done without any error, representations constructed from pro- prioceptive information only would depend on the sequence of relative movements, and would be inadequate to the establishment of allocentric cognitive maps.
It is therefore crucial to understand how allothetic cues be fused with self-motion information to achieve accurate PI, and to provide appropriate support for representations informative about the ab- solute position of the animal in its environment. This question has already been addressed in several works. Uria et al. 2020) introduced multiple recurrent neural networks (RNN) to build sophisti- cated 3D cognitive maps, with variety of neurons displaying sensory-correle features analogous to the ones encountered in cortical and hippocampal cell populations. [Bicansii & Burgess (2018) proposed a model for the interactions between multiple brain areas concurring to the production of high-level spatial representations. In the field of robotics, Simultaneous Localization And Mapping systems based on Deep Learning are an active topic of research and show promising performance in
key benchmarks of 3D navigation (Gupta et al. 2019 Zhang et al. 2020 Chaplot et al. 2020).
The objective of the present work is to address the issue of PI in the simplest possible setup, from the environment and network points of view, allowing for both good performance and interpretabil- method, e.g. directly applicable to robotics,
ity. While our goal is not to provide a state-of-the- believe that such conceptual and (over)simplified approaches questions about Pl, such as its supervised or unsupervised nature, and its relevance to RL. In addi- tion, the discrepancies between the representations built by our network and its natural biological counterparts may shed light on the additional functional and structural constraints acting on the latter.
valuable to help answer open
The environment, associated sensors, and PI loss. In order study PI in simple and con- trolled setting, we developed a continuous 2 dimensionnl environment, which follows the basis of the OpenAl Gym specification (Brockman et al.,, (216)) to allow other researchers to reuse it in their experiments. This environment, detailed in Appendix A and Figure | colored markers, which will act as landmarks to allow the agent determine its absolute position. also includes walls, which will impede some movements and restrict visibility. Movement and per- ception in this environment corresponds in Pac-Man game, centered on the current position of the agent. This setup is limited compared to real dime-siimesional environments, such as the ones based on Minecraft (Johnson et al.) 2016) or Doom (Wydmuch et al.) 2018). However, the resulting simulations are much faster and easier to run, and our framework is convenient for the study of sensor fusion. In addition, primates could be trained and monitored while performing similar task, with eyes fixated environment and moving via a joystick; this would provide a direct comparison between artificial agents and biological ones and allow for a better understanding of both (Yang & Wang) 2020).
inncludes a certain number of
top-down perspective, similar what could be found
screen displaying the
The two sensors that we want our agent to combine are: 1) a noisy copy of the action a a² te"+eu u, where is unit Gaussian vector, and e the level of noise. This reafferent action represents the proprioceptive signal, in that it does not depend on the state of the environment; 2) a retinal signal 8, which represents the allocentric information, and depends on the position of the agent (Figure 1, Right).
This retinal state mimics, some degree, the activity of biological retina such as the one of our hypothetical monkey: an array"J of Difference of Gaussians retinal cells is centered on the position of the agent; the activity of each cell computed by summing over the currently visible landmarks the activity they each elicit, which depends on their distances to the center of the cell receptive field. More details on the retina, notably on the optimal linear decoding of position from the activity can be found in Appendix B For this sensor, we consider two possible types of errors: (1) the retina receives no information, similar to what would happen the screen flickers or the animal closes its eyes; (2) the retinal state is correct, but at some point along the visual processing pipeline the obtained representation "shuffled"" between neurons. This second type of errors is meant to represent a form of temporal multiplexing (Akam & Kullmann, 2014) in the corresponding population. Depending on time steps, cells participate in the visual processing pipeline, or in other cognitive tasks; in the latter case, we would expect the population activity to have similar distribution across neurons, but no correlation with the visual representation, which is here achieved through reshuffling.
In practice, we use three superimposed arrays, one for each RGB color channel, see Appendix
2

---

Under review as a conference paper at ICLR 2022
Actions
Walls
Receptive field
Agent position
tr ag
Landmarks
T
Retinal array
Environment
Retina
Figure I: Presentation of our top-down perspective, two-dimensional continuous environment. Left: At each time step, the agent moves between positions T; and Tz+1 by performing an action a The image it perceives through its retina, now centered the new position rt+1, is modified accord- ingly, as the "landmarks" now occupy different positions with respect to the center of the retinal array. Right: Each neuron the retinal array has an associated receptive field of the "Difference of Gaussians" type (for clarity, we represent only two); depending on the position of the landmark with respect to its receptive field, each neuron will be more or less activated, generating the "retinal state" that we will consider in the following as the "observation" received from the environment
"Coavolutional" Network y
Visual Representation
Obeervation
Recurrent Neural Network
Total
Displacement
Reafferent
Dease
Encoded
action
Network
action
Figure 2: Shared structure of the models of Path Integration. The signals coming from the allocentric and proprioceptive sensors (respectively, the retinal activity and the reafferent action) are encoded through a first set of Neural Networks, before being used inputs to Recurrent Neural Network, whose output will be the predicted total displacement.
Based on these sensory signals, the agent has estimate the displacements Ar, from its starting We quantify the PI error along
point at all times t. see output of the PI network in Figure 2 trajectory of T steps through the loss
1
Ar:
aj
(I)
k=1
where (:) represents the average over trajectories.
Network architectures. Since PI task requires propagation of information from time step to the next, is not suitable for Multi-Layer Perceptron types of networks, which hold no mem- ory of the previously received inputs, but can be handled with a Recurrent Neural Network (RNN). In the following we will consider two broad categories of RNNs, with variable architectures and training procedures: (1) off-the-shelf Long Short-Term Memory modules (Hochreiter & Schmid- huber 1997); (2) minimal RNN based on the idea of direct-inverse environment models defined Section which call the Resetting Path Integrator (RPI) and introduce Section 3 Precise architectures are presented in Appendix C and a sketch of the common structure shared by all our networks is presented in F
A natural approach to multimodal PI consists in simply concatenating non-recurrent encodings of action and visual inputs, and feeding the resulting joint representation to Recurrent Neural Net- work trained on the PI loss (1). However, reported in the following, these initial attempts yielded unsatisfying solutions that 1) failed to perform resetting (see Section 3) when an image was avail- able, and 2) had internal states in the RNN that correlated only with displacement from the start the trajectory, and not with the absolute position the environment. In order to foster the emergence of allocentric representations of the environment, now introduce the concept of
3

---

Under review as a conference paper at ICLR 2022
invert models, and their associated losses. ships between proprioceptive and visual signals, and as we shall see, lead to a natural approach to performing PI using those two qualitatively distinct streams of information.
models impose strong relation-
DIRECT-INVERSE ENVIRONMENT MODELS
Evidence for internal models of environments has been found both in mammals (Ito 2018 and in humans (Wolpert et al.) 1998). notably within the Purkinje Cells of the Cerebellum, and have been hypothesized be relevant for a wide range of motor (Wolpert & Miall 1996; Kawato, 1999) and reasoning (Merfeld et al. 1999) behaviors. They have also been studied in the field of Reinforcement Learning, notably by Anderson et al. (2013), who showed that prediction of enviromment dynamics is an efficient pretraining step. by Pathak et al. (2017), who used the error in this model as form of "curiosity" signal to encourage exploration, Corneil et al. (2018). who built a tabular model of an environment for use in an explicitly model- based planning algorithm, and Ha & Schmidhuber (2018), who used an internal model to allow the agent to learn from trajectories it "dreams" rather than from direct interaction with the environment, a formalism that could explain the observed coordinated replays of place and grid cells in rats (Ölafsdóttir al. 2016).
These models can be formalized using the vocabulary of Partially Observable Markov Decision Processes (Sutton & Barto, 1998) used in Reinforcement Learning: the "hidden" state of the en- vironment is the agent's 's absolute position; the observation is the retinal signal s (see Section ! for details), the "action" at at time t is the displacement in the environment from time & to + 1 Mod- els of the environment are defined on transition tuples T = (8t, at" 81+1 and aim at predicting one of its components from the other two:
the direct model D estimates the next state from the current one and the action:
D (st, a.) Bc+1.
the inverse model I estimates the action that relates two states:
I (St, $t-1) ++ ai.
(3)
where (.) represents the average over transition tuples. In practice, this approach would be highly in- efficient and noise-sensitive in the case where the observed states are of high dimension but contain little relevant information (e.g. images). It is often preferable to construct these models on represen- fations, obtained for example via Convolutional Network V: similarly, we introduce Multi-Layer Perceptron (MLP) P that will map the two-dimensional action a, to a vector of the same dimen- sion as the representation V(s:); the resulting computation graph is presented in Figure 3 and the detailed architecture of the individual modules can be found in Appendix C
To train these models we introduce two loss functions, computed from transition tuples:
(V,PP,D) (DV(++))) ([V(SH))- - -D(V(s),
(4)
c'(VI))
((o=I(V(ee).
(5)
should be noted that training the direct model D alone using the loss _ of equ (4) results in a trivial representation scheme in which all observations mapped to the null vector 0. The inverse model I can be independently trained, but will generate irregular representations. When training D and I all together, the direct loss acts as regularization, while the inverse loss breaks the symmetry required to converge to the trivial direct model. This yields a spatially structured representation of states, on which the direct operator acts non-trivially. More details on these representations can be found in Appendix D
'The action could be considered a part of the observation, and the "partial observability" comes from the noise on these two as described in Section T
'e do not include a reward signal as these experiments aim at mimicking "free foraging". in which the agent randomly explores an environment without explicit incentive to do so. The influence of a reward on representations will be the subject of follow-up study.
4

---

Under review as a conference paper at ICLR 2022
Internal model
Time t
observation
Old representation
Action
Enviroument
D
Action
V
Time t +
New observation
New representation
Figure 3: Overview of the direct-inverse model architecture, in which operators acting on internal representations aim at reproducing the dynamics of an environment. Dotted arrows indicate that the module they come from is trained to output the quantity they point towards (eqs. 5
RESETTING PATH INTEGRATOR FROM DIRECT-INVERSE MODELS
The direct and inverse models can straightforwardly be combined create a RNN capable of Path Integration, which we will call Resetting Path Integrator (RPI) in the following, and which is sum- marized in Figure The internal state H is initialized with two concatenated copies of the initial observation
Ho (V(8)): V((a))-
(6)
Then, at each time step, the internal state is updated using the direct model
H = (h++1: V(sa)) (D(h., P(a:)); (D(h), V(K0)):
and the displacement Ar, is computed by applying the inverse model I between the updated and non-updated versions of the initial observation:
Ar: I(h;, V(s))).
While this approach suffers a priori from the same accumulation of errors as the direct movement tegration, it also allows for an additional resetting mechanism, which was hypothesized by Prescott (1996) as sufficient mechanism for spatial navigation: at any time-step, the agent could use the current visual observation to "correct" its internal state by disregarding the result of the direct model D. To allow for this, as well as partial resetting we introduce a gating network & that maps the current visual observation s $ to scalar between 0 (no resetting) and (full resetting) that is used to interpolate between the proposed new state and the representation of the current observation, yielding the revised version of the update eqn (7):
H (g V(s:) D(h. P(a:)) + [1 -1-9 oo ((s))))) V(so).
Of course, reliable images were always available, the optimal solution would correspond to at all times, that is, to resetting at every time step and never using the direct model. In standard situations, where reliable visual information may be lacking, the recurrent nature of the network will allow for correct performance in-between resetting steps by keeping the internal state close to what the agent would observe from the environment. We therefore expect the internal state of the network to strongly depend on the current value of the position, but not on the trajectory used to get therd" hence being valid candidate for a cognitive map. More subtly, if the visual information
"The second copy, which will not be modified by the network dynamics, is used as explicit "memory" of the starting point and is essential to observe resetting (see Section Appendix [ more details).
5 'Another possible approach could have been to enforce total resetting by using G as probability choose the visual state, and to train this gate with a Reinforce- algorithm. This setting seems less biologically relevant than ours and we did not investigate further.
FFrrmally, this network could also take the current state H. as input, in practice this made the training more unstable without any noticeable improvement in performance.
II time steps where strong resetting occurs (G 0), this statement is true since the state is exactly the representation of the current observation.
5

---

Under review as a conference paper at ICLR 2022
Old Stase
Encoded reafferent action
Visual regresentation
1
New
State
Estimated displacement
Initial
Representation
Figure 4: Minimal model for a Resetting Path Integrator, based on Direct-Inverse model of envi- ronment dynamics. We assume that the agent is able to see correctly on the first step of the trajectory, and to keep stable memory of this initial observation; this initial state is then updated by either us- ing the direct model and the encoded reafferent action, or the new visual representation (resetting); the choice between those two behaviors determined by the gating module G.
received is ambiguous, e.g. the local set of landmarks seen by the retina is the same as another part of the environment, see Section 4.2 we expect that the internal state should be able to lift the ambiguity in the observations through integration of previous motion, bridging the gap between regions of reliable visual information, phenomenon which we actually observe in experiments.
To combine the direct- and losses, training is done on their weighted sum
Cto (V, P, D,I,9) = apr (V, + tapl
+ aa,(
(10)
computed on random trajectories (see Appendix C for more details on the parameters).
4 RESULTS
In this section, we will analyze both the performance and the representations that emerge in networks trained on Path Integration loss of eqn. ( using the environment layout represented in Figure 5 by looking at five different metrics. First, the average path integration error, computed (1) short trajectories (T = 5), during which no image is presented to the network and (2) on long trajectories (T' = 100) with images available every five steps. We expect short and long-temm errors to be of the same order of magnitude in the case of a network that can perform resetting, while the latter will be much larger than the former if resetting behavior has been learned. Then, the average (over neurons) of the coefficient of determinations R. for the linear regression of the absolute the activity of individual neurons participating to (3) the visual representation or (4) the internal state from the absolute position, and (5) of the (internal-state) neuron from the relative displacement within the trajectory. These individual R² scores are found be close 1, either for the absolute or the relative positions, indicating whether neuron i is carrying allocentric or egocentric representation. Notice that R² ~ 0 would not mean that the neuron state would not convey any positional information, but that the latter would not be accessible to linear decoder.
We will compare these five metrics under different training conditions: a "vanilla" LSTM model and our RPI model, trained on the PI loss in eqn. (1 only; an "improved" LSTM mode and our RPI model, trained end-to-end with the direct/inverse losses
4.1 PERFORMANCE OF PATHI INTEGRATORS AND NATURE OF REPRESENTATIONS
The results on the five metrics obtained in the snake-path environment of Figure are reported in Table similar experiment carried out in more complicated layout is presented in Appendix G
Four conclusions can be drawn from those results: (1) Both recurrent structures (LSTMM and RPI) are
able to learn resetting behaviors, yielding similar short and long-term errors, see Figures and 6 (2) Training with Direct and Inverse losses (as regularization) is necessary for the emergence of resetting: (3) Our RPI model yields internal states with much higher positional tuning than I.STMs;
Several variants of LSTM were considered with varying degrees of success, see Appendix B for details. Using these losses only for pretraining is possible, but leads to catastrophic forgetting, see Appendix

---

Under review as a conference paper at ICLR 2022
Resetting Path Integrator
Long Short Term Memory
All losses
No model losses
Vanilla
Improved
Error (short)
0,021 = 0.017
0.033 + 0,025
0.014 = 0.011
0,027 + 0.018
Error (long)
0.026 0.022
0.43 +: 0.36
0.16 0.15
0.056 0.038
R²² (visual)
0.99 0.048
0.11 = 0.091
0.33 0.18
0.96 + 0.085
I2 (PI, absolute)
0.98 + 0.053
0.33 + 0.08
0.3
0.57 + 0.22
R22 (PI, relative)
0.35 + 0.077
0.82 +: 0.11
0.77 0.21
0.34 + 0.14
Table 1: Comparison between our Resetting Path Integrator model and standard LSTM in the SnakePath environment. When trained without the model losses, both architectures fail to establish proper resetting strategy, leading to higher error rates when tested on long trajectories (T 100) than on short ones = 5), and the internal state during PI linear function of displacement along the trajectory, rather than of absolute position in the environment is the when model losses are used. Errors bars were estimated from 20 realizations the training which differ both by initialization of the network weights, and drawn training trajectories.
Becowczod sodiim
garling
18-1
Vilue
18
Figure 5: Example of Path Integration trajectory Left: Crosses represent the true position of the agent, while stars represent the one evaluated through Path Integration; black circles are placed around the positions which image was provided; time along the trajectory is represented by the color of the symbols. Top right: logarithm of the value of the resetting gate as a function of time along the trajectory. Bottom right: error between the true and reconstructed position. Vertical dashed lines indicate the time-steps at which the image was available to the network and not cor- rupted. In this example, actions are drawn from the "free foraging" random policy but chosen to force exploration of the entire environment to better evaluate generalization at long distances, and reafferent actions are exact, so that errors are due only to the network itself.
(4) Representations depend on absolute position in networks that perform resetting, and relative position along the trajectory in networks that do not (see Appendix H for details).
Despite RPIs being less expressive than their LSTM counterparts, they do not seem to achieve significantly worse performance (although more hyperparameter optimization would be required to confirm this statement). In addition, RPIs converge to solutions that are more easily interpretable, both in terms of tuning to the absolute position (illustrated by the higher R² scores), and of gating dynamics, see Figure In Appendix[] we show that the value of the gate in a trained RPI is directly related to the cognitive mechanism of resetting, with the strength of resetting increasing when the training conditions incorporate more noise the proprioceptive signals, as well as when the visual representations
and more perturbed. In LSTMs, however, the reset and input gates only weakly correlated with resetting, and instead might contribute to the computation of the direct model, see Appendix for details.

---

Under review as a conference paper at ICLR 2022
A
Perfoct reafleretzor [+ m
B
Noiey reafertnos ( - B.075)
0.7
Without imags
Without imoges
With
Wilh
0.6
0.5
0.5
0.4
0.6
0.3
0.2
0.2
0.1
0.0
Tiae
Tian
Figure 6: Path Integration errors achieved by our Resetting Path Integrator, with occasionally avail- able retinal images (orange) and without images (blue). A: the reafferent action (proprioceptive signal) is exactly equal to the true one. B: small amplitude Gaussian noise is added to the reafferent action. Dashed vertical lines indicate steps at which images were presented, kept equal across 512 trajectories for each of the 8 networks used in the averaging. The qualitative agreement between those two plots, as well as results from Appendix suggest that our procedure is robust to small reafference errors, which have the same effect direct model errors.
4.2 DISAMBIGLATION OF AMBIGUOUS ENVIRONMENT BY RPI REPRESENTATIONS
Next, we considered highly ambiguous situation where two rooms, located at the opposite ends of the environment, are designed to provide strictly identical visual cues. In that case, inverse models of the full environment give very large reconstruction errors when either of the images are located within one of these rooms 10
However, training a Resetting Path Integrator on this environment in an end-to-end fashion remains possible, as shown in Appendix K The resulting networks exhibit three important properties: 1) the internal states observed during Path Integration are different in the two ambiguous rooms, as illus- trated in Figure 7 2) the PI етгог does not show any noticeable increase when the agent enters one of the ambiguous rooms, see Supplementary Figure 18 3) the resetting mechanism is not triggered for images coming from the ambiguous rooms, as illustrated in Supplementary Figure 19
The last observation was to be expected: as visual representations are identical between the two rooms, performing resetting would, average, result in loss of spatial information with respect to keeping the state updated through the direct model. The first and second observations are non- trivial. To correctly perform Path Integration in the ambiguous rooms, our RPI network created new states, differing from those coming from the visual cues, that aim at bridging the gaps between "visually informative" regions. These networks have therefore managed to construct a representation of absolute position in the environment that does not rely only on local landmarks, but also draws from proprioceptive information and effectively fuses these sensors; intuitively, the Path Integrator is able to differentiate between two visually identical rooms by remembering how it got there, highly desirable property for cognitive maps.
CONCLUSION
Results. In this study, we have demonstrated how Recurrent Neural Network can be used con- struct a cognitive map of a continuous spatial environment, by fusing unreliable proprioceptive and intermittently available external inputs through the task of Path Integration. We examined several ways of performing this fusion, using either off-the-shelf LSTTM networks, or our proposed Resetting Path Integrator model, based on direct-inverse models of the environment dynamics, and including a single, scalar gating mechanism allowing for resetting (clearing) the internal state of the network and replacing it with an external signal. While all studied architectures and training procedures manage
If we train and test only on images coming from adjacent rooms, an inverse model can perform well as there is no couple of images that would correspond to two different reconstructed positions. The resulting direct- inverse models are however not suitable for Path Integration without retraining, as their long-range performance severely limited by the ambiguous rooms.
8

---

Under review as a conference paper at ICLR 2022
Only visuial
With PI
Figure 7: Comparison between representative neurons in the visual module y (top row) and the internal state h observed during Path Integration (bottom row) as a function of position within our ambiguous environment. The "dynamic" representation constructed during PI lifts the ambiguity between the two opposite rooms of the middle row, which contain the same landmark and are sur- rounded by identical rooms. Each column represents the normalized activation of single neuron.
to perform Path Integration short trajectories, only those regularized through the addition of the direct-inverse losses learned to efficiently use resetting and achieved similar error levels on very long and short trajectories, thus overcoming error accumulation. The idea of incorporating high-level knowledge about desirable aspects of the internal states dynamics through regularization close to the one of Haviv al.] (2019). We observe that the internal neural states qualitatively different between path-integrator networks that learn resetting and those that do not: while the former are very close linear functions of the absolute position in the environment, the latter closer to lin- ear functions of the displacement along the trajectory. see Table 1. This subtle difference is crucial, and implies that only networks capable of resetting have learned a "cognitive map" (Tolman 1948) Spiers & Barry 2015), which could a priori be transferred towards other spatially structured tasks.
Future directions. Contrary to previous works on PI in artificial agents that used highly spatially- structured inputs, relying on hypothesis about the existence of either place cells (for example in Arleo et al.] (2000); Banino et al. (2018)) or grid cells (Zhao et al., 2020). our approach does not make such assumptions, and is, to our knowledge, the first one allow for study of the emergence of these representations during training. Our simplistic environment setup did not result in emergence of either of those types of cells, but instead on a "top-down" map, which accurately depicts the Euclidean (by opposition to topological) structure of the environment. In other words, positions that are close in the layout (viewed from above) are close also in the map, though they could be far from each other in terms of "minimum number of steps", e.g. when separated by wall that would need to be walked around. It is then a logical next step to move towards more realistic environments, using real first fers-pperoon view, and allowing for rotations and translations, e.g. Malmo (Johnson al.) 2016) or VizDoom (Wydmuch et al. 2018), whose interplay might be responsible for the particular coding scheme of place and grid cells (Harsh et al. 2020 Benna & Fusi 2020).
Another major direction of research concerns the role of Path Integration as an end-goal: while is assumed that such task can be used generate high-quality cognitive maps (as confirmed by our study), there is no evidence that this task is ever performed "intentionally".. Preliminary experiments show that the recurrent representations constructed by the networks can be used for Reinforcement Learning tasks, such as goal-oriented navigation (moving towards a specific position in the envi- romment), much more efficiently than through direct training. This result was be expected, since representation learning is known to be limiting factor in RL (Anderson et al. 2015). In follow- up study, we will foeus on incorporating the Path Integration loss (as well as the direct and inverse losses) regularization terms in the policy learning algorithm: this approach is similar to the Intrin- sic Curiosity Module of Pathak et al. (2017), in which the model errors were used as an exploration incentive, well as zero-shot learning through environment models (Ha & Schmidhuber 2018). All those methods can a priori be used at the same time.
Combining those two directions of research, making both tasks and environments more realistie, in particular close to what can be studied in live animals, will be key to understanding the intricacies of cognitive maps and of their relation to behavior.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Thomas Akam and Dimitri M. Kullmann. Oscillatory multiplexing of population codes for selective communication in the mammalian brain. Nature Reviews Neuroscience, 15(2):111-22) Febru- ary 2014. ISSN 1471-0048. doi: 10. 10.1038/mm3668 URL https: www. na ture. com/
articles/nrn36G
Charles W. Anderson, Minwoo Lee, and Daniel L. Elliott. Faster reinforcement learning after pre- training deep networks predict state dynamics. In 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1-7,July 2015. doi: 10. 10.1109UCNN.2015.788000
Angelo Arleo, Fabrizio Smeraldi, Stéphane Hug, and Wulfram Gerstner. Place Cells and Spatial Navigation Based 2D Visual Feature Extraction, Path Integration,
and Reinforcement Learning. Advances Neural Information Processing Systems, 13, 2000. URL
neuripps...e http://ppppccppppp/eeeeeaa http://w......pppa///////////a/ 2ew.2220aa221100000eeeaaaaaaaaa2
Andrea Banino. Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang. Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Char- lie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukeuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid- like representations artificial agents. Nature, 557(7705):429, May 2018. ISSN 1476- 4687. doi: 0005:/44556-011-1000000 841586-018-0102-
URL https: www.:w....ar.......oo htt.://aaaare..caaaattleeeesee
Marcus K. Benna and Stefano Fusi. Are place cells just memory cells? memory compression leads to spatial tuning and history dependence, August 2020. URL/ttps:/ww.bbirriiv..
content/ /10. 1101 62423933
Andrej Bicanski and Neil Burgess. A neural-level model of spatial memory and imagery. eLifee, 7:
e33752, September 2018. ISSN 2050-084X. doi: 10.7554/cLife.33752 URL hnttps:///ol org/10. 71140005554eLiiee 33752
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv:1606.01540 [cs], June 2016. UtLptt::://arat
org/aba/ /106. 01540
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to Explore using Active Neural SLAM. arXiv:2004.05155 [cs], April 2020. URL
httt::/arxiv.cn http///xxiv..g///b..p/a// awtpr////////////e 2204. 05155
Brenton G Cooper, Theodore F Manka, and Sheri JY Mizumori. Finding your way in the dark: The retrosplenial cortex contributes spatial memory and navigation without visual cues. Behavioral neuroscience, 115(5):1012, 2001. doi: 10. 10.103770735-700 70447 1044.115.5.1012
Dane Corneil, Wulfram Gerstner, and Johanni Brea. Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation. arXiv:/802.04335 [cs, stat], June 2018. AtRtttp:
Peter Dayan and Larry Abbott. Theoretical Neuroscience: Computational and Mathematical Mod- eling of Neural Systems, volume 15 of Computational Neuroscience Series. MIT press., January 2001.
Martin Deschênes, Jeffrey Moore, and David Kleinfeld. Sniffing and whisking in rodents. Cur- rent Opinion in Neurobiology, 22(2):233 250, April 2012. ISSN 0959-4388. doi: 10.1016/
1.0ob.2011.111000 URL https: www. sci i rect com/ sci wcienceeerttilee
pi pii pii/509 051/809594388110 0001110001111 69
Ariane S. Etienne and Kathryn J. Jeffery. Path integration in mammals. Hippocampus, 14(2):180- 192, 2004, ISSN 1098- 1063. doi: 10. 10.1102/hipo.10177 10173. URL http::::///niineibbarr.
wi ey. www httm//oo//bbs//inn
1002 1002/hi 0173
10

---

Under review as a conference paper at ICLR 2022
Arianne S Etienne, Roland Maurer, and Valérie Séguinot. Path integration in mammals and its interaction with visual landmarks. Journal of Experimental Biology, 199(1):201-209 January 1996. ISSN 0022-0949. doi: 10. 1..1242fjeb.19.11200 URL https:// D.t.: jeb. 199.1.99..11111
Arnaud Fanthomme and Rémi Monasson. Low-Dimensional Manifolds Support Multiplexed In- legrations in Recurrent Neural Networks, Neural Computation, pp. 1-50. January 2021. ISSN
0899-7667.
doi: 10.1162/neco 01366. URL https: http://ii..rr/1..11111111
01366
Reginald Golledge, G. Human wayfinding and cognitive maps.
The Coloniza-
fion of Unfamiliar Landscapes. pp. 49.54. Routledge edition, 2003.
URL https:
human /ttp://wwwotttaaaa.ccoccoooaaaaa
//
13/
golledge
Saurabh Gupta, Varun Tolani, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive Mapping and Planning for Visual Navigation. arXiv:1702.039900 [cs], February 2019. URL
11002 C3920
David Ha and Jürgen Schmidhuber. World Models. arXiv:1803 10122 [cs, star], March 2018. doi: 10122
10.528 1/zenod.. 1207631. URLphttt://axxiv.coo wtg.vv..rggg.......
Moshir Harsh, Jérôme Tubiana, Simona Cocco, and Rémi Monasson. 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: Breaking and dynamical restora- tion of continuous symmetries the weight space. Journal of Physics Mathematical and Theoretical, 53(17):174002, April 2020. ISSN 1751-8121. doi: 10. 10.1088/1751-81111aa770
URL htt.t::://eeee. http://ooooooog//////////
org. 108 751 11111111100 8121/ab7d00)
Doron Haviv, Alexander Rivkind, and Omri Barak. Understanding and Controlling Memory in Recurrent Neural Networks. In International Conference on Machine Learning, pp. 2663-2671. PMLR, May 2019. Utthttt::/proceeeiiiii
mlr www.m...............mm.mmm http://wwaaailaaaaaa 9htmll
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, doi: 10.1162//ec0..1997999. URL
(8):1735-1780, November 1997. ISSN 0899-7667.
www.o////....../e..o......o 1997 01..9999..887.. 1735
Marco lacoboni, Lisa M. Koski, Marcel Brass, Harold Bekkering, Roger Woods, Marie-Charlotte Dubeau, John C. Mazziotta, and Giacomo Rizzolatti. Reafferent copies of imitated actions in the right superior temporal cortex. Proceedings of the National Academy of Sciences, 98(24): 13995-13999, November 2001. ISSN 0027-8424, 1091-6490. doi: 10. (0.1073/pnas.2414455559
URL hhtpp::/wwwt pras. wwwp:ppp...oooooooeeoottttttmmte http://cooneeet////4433399
Hiroshi T. Ito. Prefromtal-hipooaaneee interactions for spatial navigation. Neuroscience Research, 129:277, April 2018. ISSN 0168-0102. doi: 10.1001//.ucurr22211.0..006 URL http://
www.ecieieennee............ com/ http:///.o...a.oaa...//.////////
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The Malmo platform for artifi- cial intelligence experimentation. In Proceedings the Twenty-Fifth International Joint Confer- ence on Artificial Intelligence, IJCAI' pp. 4246-4247, New York, New York, USA, July 2016. AAAI Press. ISBN 978-1-57735-770-4
Mitsuo Kawato. Internal models for motor control and trajectory planning. Current Opinion in Neurobiology, 96))778877 December 1999. ISSN 0959-4388. doi: 10. (338899995994388999) 00028-8.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13): 3521 3326, March 2017. ISSN 0027-84244 1091-6499 doi: 10.1073/pnas. 1611835114, URL
https: www. wwws.aaa....mmmmm..... org/ ctt.//nttttttccccoo
11

---

Under review as a conference paper at ICLR 2022
Emilio Kropff, James E. Carmichael, May-Britt Moser, and Edvard I. Moser. Speed cells in the medial entorhinal cortex. Nature, 523(7561):419 424, July 2013. ISSN 1476-4687. doi: 10. 1038/nature 14622. URL www. nature com/ articces/naturee45555
Bruce L. McNaughton, Francesco Battaglia, Ole Jensen, Edvard Moser, and May-Britt Moser. Path integration and the neural basis of the 'cognitive map'. Nature Reviews Neuroscience, (8):663-678, 678, August 2006. ISSN 1471-0048. doi: 10.1038/nm1193 URL https:
nature ccm htt.c//aaaaicee///n111....1
Daniel M. Merfeld, Lionel Zupan, and Robert J. Peterka. Humans use internal models estimate gravity and linear acceleration. Nature, (006(667)6661-611111 April 1999. ISSN 1476-4687. doi:
10.10/8/19303. wtt.ttts::////tttt :ttttps::/www/ http:://wwww........tt
9303
H. Freyja Olafsdóttir, Francis Carpenter, and Caswell Barry. Coordinated grid and place cell replay during rest. Nature Neuroscience, 19(6):92-7794, June 2016. ISSN 1546-177 1726. doi: 10. 1038/.. 4291. /tttts::/www....
com/ art c cleeeeeeeeeeee nn. 4291
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Performance Deep Learning Library. arXiv:1912.01703 [es, stat], December 2019. URL abs/
http://arxiv. wttp://wwwww..////////////// 01703
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven Exploratioo by Self-supervised Prediction. arX:v:175..05363 [cs, stat], May 2017. URL http::/arxx:
oxg/abs/1705.c 05363
Tony J. Prescott. Spatial Representation for Navigation in Animats. Adaptive Behavior, 4(2):85
123, January 1996. ISSN 1059-7123. doi: 10.117/1099722366000022 URL https://
A. David Redish and David S. Touretzky. Cognitive maps beyond the hippocampus. Hip- pocampus, 351):15-35, 1997. ISSN 1098-1063. doi: 021002//SIC1D008--11111199771111
AID-HIPO333...0022222 URL https
0. C.C3322-66
H. Seung. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences,
(3323):13339-13444 November 1996. ISSN 0027-8424, 1091-6490. doi: (0.1073/pnas.93.2) 13339. URL srEEps://weee pras 2ttaa/.orr/cooteet333/////11111
Hugo J Spiers and Caswell Barry. Neural systems supporting navigation.
Current Opinion
in Behavioral Sciences, 1:47-55, February 2015. ISSN 2352-1546. doi: 10.1016/j.cobea.
2014.08.005. URL https:// /www. sciencedi rect. rect. com http//siinncccrraee wtc.m//ciennce/arttcleee./ $2352154614000155
Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Songyuan Li, and Jabbar Abdul. Recent Ad- vances and Trends Multimodal Deep Learning: A Review. arXiv:2105.1 1087 [cs], May 2021.
URL http:://aaatte vv org/ abs/ 2105 11087
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive Com- putation and Machine Learning series. MIT press,, Cambridge (Mass.), Etats d' Amérique, Royaume- Uni de Frande-Bretaage
d'Irladde du Nord, 1998. ISBN 978-0-262-19398-6
J. S. Taube, R. U. Muller, and B. Ranck. Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. Journal of Neuroscience, 10(2): 420-435, February 1990. ISSN 0270-6474, 1529-2401. doi: 1990. URL www. jneurosci /wtg/content/iinnnoo /10/2//2000
E. C. Tolman. Cognitive maps in rats and men. Psychological Review, 55(4):189-20) 208, July 1948. ISSN 0033-295X. doi: 10.1037/h001126.
12

---

Under review as a conference paper at ICLR 2022
Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zambaldi, Dharshan Kumaran, Demis Hass- abis, Caswell Barry, and Charles Blundell. The Spatial Memory Pipeline: A model of egocentric to allocentric understanding in mammalian brains. Preprint, Neuroscience, November 2020. URL
htttp://bioxiii .rg/ 1
11 1111/2020.22..11 17.37811
Daniel. M. Wolpert and R. Chris Miall. Forward Models for Physiological Motor Control. Neural Nenworks: The Official Journal of the International Neural Nenwork Sociery, 9(8):1265-127,,
November 1996. ISSN 1879-2782. doi: 101101660099--00009000033333
Daniel M Wolpert, R. Chris Miall, and Mitsuo Kawato. Internal models in the cerebel- lum. Trends Cogmitive Sciences, 2(9):338-33 337, September 1998. ISSN 1364-6613, doi:
URL http::/ttw//t www.s//eeeeeeeeee.ooooeiiiiiiiie
ariicle/s1/S13866638812222
Marek Wydmuch, Michal Kempka, and Wojciech Jaskowski. ViZDoom Competitions: Playing
Doom from Pixels. woXiv:/009003470 [cs, stat], September 2018. URL http://arxiv.
hxg/abs//ee 1aoa/1180...m 03470
Guangyu Robert Yang and Xiao-Jing Wang. Artificial Neural Networks for Neuroscientists: A Primer. Neuron, 1076)):048-107) September 2020. ISSN 0896-6273, doi: 10.1016/j.neurn. 2020.09.005. URL https: /wwwwwwwwwm ce wtt.re/oooooeeeeeeor//rttttaaata S0096 6273 0273666622220000 (20) 30705-4.
Jingwei Zhang, Lei Tai, Ming Liu, Joschka Boedecker, and Wolfram Burgard. Neural SLAM: Learning Explore with External Memory. arXiv:1706.09520 [cs], December 2020. URL
http://arxww..rg//aaaae wttp://wwrg//bb////5.//// 095200
Dongye Zhao, Zheng Zhang, Hong Lu, Sen Cheng, Bailu Si, and Xisheng Feng. Learning Cog- nitive Map Representations for Navigation by Sensory-Moto Integration, IEEE transactions on cybernetics, April 2020. ISSN 2168-2275. doi:
13

---

Under review as a conference paper at ICLR 2022
A THE CONTINUOUS GRIDWORLD ENVIRONMENT
All experiments presented in this article were performed using the GridWorld environment class, defined in the environment.py file. While this environment is neither particularly efficient to run, as it is coded in pure python, nor very expressive, it still allows for a wide variety of interesting situations and is performant enough to not be an unreasonable bottleneck in most situations. also follows the basic specifications of the OpenAi Gym framework, which makes it easy to extend it and test basic Reinforcement Learning tasks such as goal-driven navigation.
While we do not provide level editors or tools to procedurally generate new layouts, they can be added by hand in the environment register, under new key corresponding to the map name, as dictionary containing the following attributes:
the number of rooms in the environment;
a list containing the position of the room centers in the surrounding environment (the envi- ronment can be rescaled as a whole when instantiated, so easier to assume rooms to be of size 1 x 1)
list of room exits, such that the i-th component is a list containing all exits from room i. An exit is defined as room edge (either vertical or horizontal) that connected to another room edge, and the link between coordinates in the two rooms is established by giving the coordinates of the same physical point in the two rooms (the center of the common room edge);
a list of possible item layouts, detailing what items are visible (and at which position from the room center) within each room, allowing us to simulate visual occlusion, or even special effects (such as switching the light off when the agent is in a particular room). All items are point-like light emitters, and only differ through their color.
Given the connection graph of the rooms and the item layout, the GridWorld class can be used to generate trajectories from sequences of actions, well as to generate numan-interpreeen rendering of the arrangement of rooms and items which can be used as basis for more involved plots (most notably, trajectories and value of neuron activity
function of position in the environment).
The inputs to the network, which we refer to simply as images, are obtained from list of rooms and positions within these rooms, by using retina of the type described in Appendix B
We provide several environments and layouts, some of which were used only for preliminary tests but which we retain for the sake of completeness.
B RETINA
Individual retinal fields We begin by introducing the Difference of Gaussians retinal neurons Dayan & Abbott 2001): their receptive fields have a center c, two widths (-), and their activity g when single visual cue is present at distance òr from the center is a difference of Gaus- sians:
2r2
B
2
- cr) V2T0+
exp
exp
(11)
When more visual cues are presented in the image, the activation of the neurons will be the sum of the activations for each individual object.
Retinal array We will consider retina square array of these retinal fields (see Figure B). all with 1 = B for simplicity. The value of A is determined so that, on average over the position of
While this enviromment designed be used for 2-dimensional spatial navigation, consider the centers have three components; the first two are the (z, y) coordinates which can be modified by our two- dimensional actions, while the third one can be used to lift ambiguity between two rooms that would be located at the same position but would differ in another way meaningful for the environment (for example, to change the environment after the agent visits particular room). This functionality is not used in the experiments presented in this study.
14

---

Under review as a conference paper at ICLR 2022
Center of retinal fields
X
"Area of effect" of the retina
Agent position r -xx y)
X
X
Objects
Figure 8: Example of retina: a regular square lattice of Difference of Gaussians fields. We describe the "area of effect" of the retina the zone in which object would contributes the activity vector of the retina as a whole above some arbitrary threshold, e.g. 10-2
single object within a square room of width 1, the average norm of the retina activity vector is equal 1. For experiments, will use an array of 642 4096 cells; for the values of the widths, we choose (a+ = 0.4, a_ = 0.5).
Color retina In order to be able to differentiate between two objects, we associate to each "color", i.e. vector [0, 1]3. This color is perceived by the retina the following way: there are three "copies" of our retina, one for each color "channel": the object activates each "channel" proportionately to the object's value in that color. This makes so that each position in the environments corresponds to image i of size (3, 64, 64) for the 64 64-retinas that we use.
Optimal linear reconstruction of an arbitrary function of position Let us consider the family of functions (r)) (2)), In (r)}, where (r) describes the activation neuron when cue located position A linear model of an arbitrary function (of the position) from the state of our retina can be written as a linear combination of those functions:
f(r) = Σ (T)
(12)
The associated reconstruction error on domain C R2 can then be written:
E(az) = / Σ - - f(r)
/
Σ ( - 25(r) =a)a
(13)
Eajaas / (((r))(0)))2 " aj / f(r)gi(r)+ 5(12
-2
= Eagaayshi - 2 a
ij
The minimum of this loss with respect to the parameters a satisfy:
3P
Vi,
Σ Iyass hy. =
(14)
and therefore the optimal linear decoder is obtained as aj 11
Numerical approximation of the integrals over D are easily obtained by averaging over measures a grid lattice, up to some numerical precision limitations. This decoding is not particularly fast since determining I is computationally expensive.
15

---

Under review as a conference paper at ICLR 2022
A
B
Value of reconstruction error
Value of reconstruction error
1.00
1.00
0.06
0.50
0.02
0.50
0.25
0.25
0.00
0.00
40.55
40.25
-0.02
-40.02
-0.04
.0.75
.0.75
-0.04
40.06
-1.00
-1.0
-0.5
0.5
-1.0
-0.5
0.5
1.0
Value x
Value x
Figure 9: Reconstruction error on a grid with 100 subdivisions on æ and y as function of position, represented heatmap. On the left, the reconstruction error of the optimal linear decoder shows clear geometric patterns related to the geometry of the underlying array of cells. On the right, the reconstruction error for "deep" 3-layer ReLU network. While the highest observed error is higher for this deep network, the error on all points except the corners is much lower than for the linear network. Additionally, the geometric patterns not observed in that case.
Ct-1
ft
h
h
9t
it
Figure 10: Computation diagram for L.TTM cell, implemented in Paszke al. 2019
Expected optimal performance When the environment consists in single room, containing single object, we can easily reconstruct the relative position of the object with respect the center of the retina using either the direct linear solving of the previous paragraph gradient descent a more parametrized structure, e.g. a dense or convolutional neural network. This experiment gives idea of the maximum performance that be expected from any inverse model based retina.
When using 642 array spanning |-0.5, 0.5] and reconstructing the position of object placed arbitrarily in [],,1], the Root Mean Square error for the optimal linear reconstruction is around 10-2, with clear geometric patterns in the errors, while the 3-Layers ReLU network achieves performance closer to 10-3 by seemingly "smoothing" the aforementioned error patterns. These results are presented in Figure
C NETWORK ARCHITECTURES AND TRAINING HYPERPARAMETERS
C. ARCHITECTURES
The exact architectures used in implementations follows:
16

---

Under review as a conference paper at ICLR 2022
the convolutional networks V used to obtain the visual representations
1. Input image with 3 channels, size 64 x 64 (from the retina)
Convolution with filters, kernel size of 5, stride of 3, padding of 2
3. Convolution with 32 filters, kernel size of 5, stride of 5, padding of
4. Flatten layer
Dense layer with 512 outputs
6. Dense layer with 512 outputs
the action encoding networks P are:
1. Input layer of size
2. ReLU layer with 256 outputs
3. Linear layer with 512 outputs
the direct model networks are:
1. Input layer of size 1024 (concatenation of representation and action encoding)
2. ReLU layer with 512 outputs
Linear layer with 512 outputs
the inverse model networks D are:
1. Input layer of size 1024 (concatenation of two representations)
2. ReLU layer with 256 outputs
Linear layer with 2 outputs
the gating networks G are:
1. Input layer of size 512 (visual representation)
ReLU layer with 128 outputs
3. ReLU layer with 64 outputs
4. Sigmoid layer with output
It should be noted that our Recurrent Path Integrator models have number of parameters of the same order magnitude as the off-the- sh-thh--sheff implementations LSTM that we used (see Figure (0).
C.2 TRAINING
In all experiments we present, the PI losses are computed on batches of 32 trajectories of length 40, with actions drawn from a two-dimensional uncorrelated Gaussian of standard deviation 1/2 and starting point chosen randomly at any position in any non-ambiguous room. The direct and inverse losses are computed on batches 512 transitions, for which starting points and actions are chosen in the same way as for PI.
The relative weights in the total loss are times higher for the direct inverse losses than for the PI loss as the former are smaller (these hyperparameters have not been optimized)
Training consists in 4000 steps of computing the losses, and performing one step of Adam opti- mization with uniform learning rates of 10-3 except for the forward model which is at 10 (no hyperparameter optimization was lead on these either). In most cases, losses only evolve marginally after the first hundreds of epochs.
We emphasize that since the environment, starting positions and actions are both continuous and random, trajectory is ever seen twice by the network, so overfitting is not concern (rather, we hope that the network meaningfull interpolates between what it has previously seen). However, single environment, and the question of the capacity-resoluse
train the networks (how the precision of is modified when several environments are learned at the same time) remains unaddressed in the present study and meaningful future direction.
tradeoff
17

---

Under review as a conference paper at ICLR 2022
0.5
0
-0.5
(a) Without forward model
(b) With forward model
Figure 11: Comparison between representations obtained after training the Direct-Inverse Model module in environment. Each panel represents the normalized activation of single neuron in the visual representation V(s), obtained at the end of the visual processing module, represented as function of position within the environment through color code presented on the right scale. The activities are of the same order of magnitude in all cases. When optimizing only the inverse loss (eq. 5), see panel (a), the representations can be spatially irregular; the introduction of the direct loss (eq. smooths out the activities, see panel (b). This effect is quantified in Table
Table 2: Comparison of the inverse model performance and the representation regularity between models trained with or without the direct loss. The addition of the direct loss shifts the distribution of R² scores between neuron activities and spatial position towards one, meaning made some neuron activities closer to linear functions of position, which we argue is desirable property in order to obtain transferable representations. While this shift is noticeable, does not come with any appreciable change in inverse model performance. Means and deviations computed across 8 realizations.
Error (training) 0.017 + 0.01 0.015 0.0091
Error (generalization) 1.6 + 0.77 1.6 0.75
R (visual)
Without direct
0.95 + 0.089
With direct
0.83 0.19
D INTERACTIONS BETWEEN DIRECT AND INVERSE LOSSES
As mentioned in the main text, training the inverse model without the direct one is possible, but the other way around is not as training in that case converges to trivial solution where the representation module V and the direct model D always output O. When training with both losses, we observe slight noticeable smoothing
representations, as illustrated in Figure IT and Table 2 This improvement in regularity does not however translate into any measurable difference in performance of the inverse model: the representations are made simpler by the direct loss (as expected when positional information. It should be
adding regularization term), but they do not carry any noted that while "generalization" errors (computed on any couple of images, even if they can not be reached in single transition and hence large, the difference in position predicted remains qualitatively relevant (just with a lower precision).
part of transition tuple used in training)
E LSTM VARIANTS
In this section, we present our attempts at improving the performance of the "vanilla" LSTM ar- chitecture. Beyond basic hyperparameter tuning (no extensive optimization has been led due to prohibitive computational costs), we mostly considered modifications on initialization, and archi- tecture:
18

---

Under review as a conference paper at ICLR 2022
Visual representation
New State
Encoded LSTM
Estimated displacement
Initial Representation
Action
Figure 12: Computation diagram for the hybrid path integrator structure.
the standard architecture corresponds to simply using a LSTM (see Figure 10 as the RNN in the computational graph of Figure 2 We considered two training schemes with this architecture:
the "vanilla" scheme trains this network from scratch on the PI loss. It yields good integration properties, but fails to learn resetting behaviors (results presented in main paper).
the "pretrained" scheme initializes the "encoders" (both for the image and the action) using the ones of Resetting Path Integrator trained with all losses (since they exhibit the cleanest representations). The results are very similar to the "vanilla" scheme.
the "hybrid" solutions have the computation diagram of Figure 12, which corresponds to using the LSTM only to update the internal state, replacing the combination of the direct model and gating module G. This architecture has the advantage of explicitly using the initial representation converge to resetting behaviors. We considered several training schemes using this archi- tecture:
form of "anchor", which seems in practice help training
the "default" scheme trains this network from scratch on the PI loss, yields similar result to vanilla LSTM.
the "pretrained" initializes the "encoders" (both for the image and the action) using the of a Resetting Path Integrator trained with all losses; it reliably achieves resetting, but also has lower precision on short trajectories.
the "scratch" scheme does not do any initialization, but adds "direct" module (same architecture as the ones our Resetting Path Integrator), defines the direct and inverse losses using it and the I module that outputs the displacement, and uses those losses as regularization. This scheme often manages to find resetting solutions, but requires more care in hyperparameter tuning to converge properly to resetting solution.
the "improved" scheme similar to scratch, but also initializes the encoders. This scheme is included in the main text, and it reliably achieves resetting.
In Table 3 we present results the aforementioned architecture that were not included in the main text, Table [ It should be noted that even schemes that yield solutions that exhibit resetting do not necessarily have higher levels of positional tuning, which we argue still makes them less convincing candidates for transferable cognitive maps than our Resetting Path Integrator model.
F CURRICULUM LEARNING AND CATASTROPHIC FORGETTING
In this section, we consider the case in which the direct-inverse model of the environment is trained first, using transition tuples, without any consideration Path Integration.
After this initial pretraining, we introduce those weights into the full Resetting Path Integrator net- work, and consider three different ways of training the PI task:
A: we optimize only the weights of the resetting gate G, and use only the Path Integration Loss.
we optimize all weights in the network, including those that were initialized from the pretraining, still using only the Path Integration Loss.
19

---

Under review as a conference paper at ICLR 2022
Table Comparison between the different LSTM variants we considered the SnakePath environ- ment, in terms of both errors and representation correlation with position, see main text for details. Means and errors computed on 8 realizations.
Standard
Hybrid
Pretrained
Default
Pretrained
Scratch
Error (short)
0.01 + 0.0064
0,013 0.0086
0.034 + 0.02
0.022 + 0.014
Error (long)
0.091 + 0.088
0.11 + 0.099
0.043 + 0.026
0.042 0.03
R2 (visual)
0.46 + 0.2
0.63 +: 0.24
0.91 + 0.14
0,94 + 0.11
R (PI, absolute)
0.27 + 0.11
0.29 + 0.11
0.63 0.19
0.58 + 0.21
R² (PI, relative)
0.69 + 0.26
0.72 + 0.24
0.24 + 0.092
0.34 + 0.14
Parani
Prees
Buloor
Pars langeatic
Purk lingustai
Parl Scngrets
2300
1500
4000
500
1000
2000
2000
30x
4.00
500
1800
2000
2000
2500
200
400
Figure 13: Evolution of the different losses when training a Resetting Path Integrator, whose visual, direct and inverse modules were pretrained using transition tuples from the environment, in the three different protocols described in the text.
C: we optimize all weights in the network, but do Gradient Descent on sum of all losses (Path Integration, direct and inverse), as we would in end-to-end training.
The results are presented in Figure and show that while the final level of Path Integration eπror is close between the different protocols, retraining parameters of the model on the Path Integration loss only (protocol B) produces noticeable deterioration in the quality of the Direct-Inverse model, an example the catastrophic forgetting phenomenon (Kirkpatrick et al., 2017). Independently of the choice of loss which Gradient Descent is performed, optimizing the parameters of the Direct-Inverse model produces much more chaotic evolution the loss.
G ERRORS AND SPATIAL CORRELATIONS ON THE DOUBLEDONUT ENVIRONMENT
We present in Table T the values of errors and spatial correlations measured in networks trained on second environment layout, DoubleDonut. This environment has the general structure of its ambiguous variant (represented in Figure [8). except that the objects in the left- and right- rooms of the middle row are different in the case of the non-ambiguous version that we consider here. The conclusions of this study are identical the ones presented the main text.
layour, slightly different from the used in the main text, and which we call
H REPRESENTATIONS IN ABSOLUTE AND RELATIVE COORDINATES
In order to complement the RD values presented in main text Table 1 report the value of neuron activations (in the internal state, observed during Path Integration) as a function of absolute posi- tion in the environment (Figure 14] and function of position within the trajectory (Figure 1.5)
20

---

Under review as a conference paper at ICLR 2022
Table Comparison between our Resetting Path Integrator model and standard LSTM in the Dou- bleDonut environment. As was the case in the SnakePath environment, models trained without the direct-inverse losses fail learn how to perform resetting and show lower levels of spatial structure in their representations.
Resetting Path Integrator
Long Short Term Memory
All losses
No model losses
Vanilla
Improved
Error (short)
0.026 + 0.019
0.035 + 0.026
0.015 + 0.0086
0.032 = 0.019
Error (long)
0.032 = 0.023
0,46 0.41
0.15 = 0.14
0.051 = 0.033
R (visual)
0.97 0.068
0.16 0.12
0.36 = 0.16
0.91 0.13
R (PI, absolute)
0.96 + 0.073
0.29 + 0.063
0.27 + 0.086
0.68 + 0.19
R² (PI, relative)
0.34 + 0.058
0.85 0.14
0.74 + 0.22
0.26 0.083
All losses
PI loss only
Figure 14: Activity of four representative neurons in the internal state population of an RPI trained with (left) or without (right) the model losses, as a function of absolute position in the environment. Only the trained with those losses (and performing resetting) close linear function of absolute position.
for our Resetting Path Integrator model, trained with or without the inverse losses (and con- sequently, respectively displaying resetting or not). We find that non-resetting representations are linear functions of displacement, as expected from an integrator network (see) feethommt & Monas- son (202T)), while resetting representations are linear functions of absolute position, which makes them much more relevant as cognitive maps. Interestingly, on the left of Figure [5), points that correspond to "extreme" displacements seem more correlated with trajectory coordinate than points with small trajectory coordinates; this is be expected since extreme displacement points necessarily lie on the edge of our environment (to get a displacement of 3 in the direction, the agent necessarily started the right side of the environment and fin- ished on the left side), so that for those points trajectory coordinates and absolute coordinates are correlated.
note that (for the resetting network
I GATING STRENGTH IN A RESETTING PATH INTEGRATOR
While all training conditions we investigated lead to resetting behaviors, the mean value of the gating obtained from images of the environment was observed to vary drastically, from 10-1 to 10 10-3 while the level of Path Integration errors remained mostly unchanged. Our understanding for this phenomenon is the following: the minimum level achievable error is the same for all training conditions, and related to the limitations of the retinal array detailed in Appendix B therefore, the
21

---

Under review as a conference paper at ICLR 2022
All losses
PI loss only
Figure 15: Activity of four representative neurons in the internal state population an RPI trained with (left) or without (right) the model losses, as a function of position along the trajectories. Only ones trained without those losses (and not performing resetting) are close linear function of position within the trajectory.
A
B
Commiiatio distilaab funntio
10-1
10-2
0.25
x.x
16
0.25
0.4
6.4-
13-1
Logarithm sbe
value
Lageithm of the reaut gate vallon
Figure 16: Cumulative distribution function of the natural logarithm of the reset gate across the environment in different conditions. A: Varying the level of noise e in the reafferent action dur- ing training. As expected, high levels of noise favor strong resettings, hence lower values of g. B: Varying the level of perturbation p. defined as the fraction of neurons in the representation that were randomly reshuffled, at test time. As representations are increasingly perturbed, they become less similar ones that come from the environment, and we expect the network to reset its state less strongly the expected benefit from such resetting decreases. Both panels present results aggregated across 16 different realizations of the PI training.
network can accumulate errors (coming either from imperfect reafference or imperfect integration) for a certain number of time-steps without any noticeable effect. In the limit case where the direct model performs perfectly and reafference exact, no resetting ever necessary. A direct way to limit the accuracy of the direct model is to add noise to the reafferent action during training, and our hypothesis is that as this noise increases the value the resetting gate will get closer to O, meaning that the resettings will be stronger and "keep less memory" of the state before resetting. This hypothesis confirmed by Figure I6A. Additionally, we expect that if, at test time, we present the gating module with increasingly perturbed representations, the value of the reset gate will increase until no resetting happens (g = 1) if the image is completely shuffled [2] This situation is represented in Figure
22

---

Under review as a conference paper at ICLR 2022
Reset gate neurons
0.6
0.5
0.4
0.4
0.3
0.2
0.1
10
80
30
0.63
0.0
0.50
0.4
0.45
62-
0.30
0.25
10
20
10
20
90
40
50
Input gate neurons
0.55
0.50
0.46
0.7
0.25
0.20
0.4
50
0.40
0.5
0.30
0.3
0.20
0.15
20
30
40
50
Figure 17: Activity of four representative neurons in the "reset" and "input" gates, as a function of time, aggregated across 128 trajectories in which images always presented at the same time- steps.
23

---

Under review as a conference paper at ICLR 2022
J INTERNAL GATES IN AN LSTM NETWORK
Averaged across a large number of trajectories, the values the input and reset internal gates at each neuron show, to varying extent, the behavior that to be expected from the gate names: the reset gate neurons are inhibited when an image is presented (meaning that the previous internal state suppressed), while the input gate neurons are activated (meaning that the current input contributes more to the internal state update). We present in Figure [7 a few representative neurons in both populations. Given the high variability that observed in the reset and input gates, we expect that those two subnetworks contribute not only to the resetting, but also to the computation of the direct model.
K RESETTING IN THE CASE OF AMBIGUOUS DOUBLEDONUT
In this section, we consider the same end-to-end training procedure as in the main article, but apply to more complex environment comprised of 16 rooms, two of which (the left-most and right-most of the middle row) provide the agent with identical visual cues, creating an ambiguity where two different positions in the global environment correspond to the same images. This ambiguity is still such that inverse model is unambiguously defined (since there are no positions in the environment that could be reached in single transition from both ambiguous rooms), and the forward model too as long as we choose the start position in ny non-ambiguous rooms (because otherwise, the same initial state and the same action could lead to different new states, for each room). As shown in Figure 18 the Resetting Path Integrator models still manage to perform reasonably well despite this ambiguity by creating new representations, distinct from the visual ones, as shown Figure 7 and not performing resetting when the image comes from one of the ambiguous rooms, see Figure
19
We did not present the networks with partial shufflings of the representations any stage in the training, only unperturbed fully shuffled.
24

---

Under review as a conference paper at ICLR 2022
Recoverol poetsons
Valise gating
Valte the
10-1
Figure 18: Example Path Integration trajectory on the Ambiguous DoubleDonut environment. The network does not exhibit any particular drop performance upon entering either of the am- biguous rooms, suggesting that the internal state it constructed during Path Integration lifted the ambiguity that is present in the visual cues. It still remains notable that resetting is performed the visual cue, even unperturbed, comes from an ambiguous room, a phenomenon further illustrated in Figure
Value the gxing (kg,)
Figure 19: Value of the natural logarithm of the resetting gate g as a function of position, averaged across 8 realizations of the training. As expected, resetting happens at least partially at every position the environment, except within the two rooms that have ambiguous visual cues.
25


Reviews:
[{"review_id": "lEKnMNsuvXu", "reviewer": "Reviewer_CUYu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n- Apart from some confusion with the terminology in the introduction, the paper is clear to read and there is an abundance of analysis performed. \n- The idea of learning state that encode the absolute position using path integration is interesting and potentially of great impact \n\nCons:\n- In the introduction the authors assert that previous memory trajectories are part of proprioception, as currently written this is very inaccurate. Also, vision inputs are not allocentric, but egocentric. Broadly speaking allocentric representations are found in the hippocampal–entorhinal regions after visual stimuli have been converted in the retrosplenial cortex where it’s possible to find egocentric representation. For these reasons, I found the first paragraph poorly written and I would encourage rewriting, maybe after having read some appropriate papers that have not been cited (e.g. Spiers and Barry, 2015. Moser et al., 2018) \n- Again, in the last part of the “Path integration task” section, allocentric information is used in the wrong way.\n- When presenting the environment the authors describe the retinal inputs using a parallel with a monkey; however, most of the previously cited literature is on mice/rodents, this is awkward. To mitigate this I would cite Akam and Kullmann before.\n- When explaining the network architecture the authors claim: \"In practice, naive attempts at performing PI using the concatenation of proprioceptive and visual signals yield very unsatisfying cognitive maps which depend on position along the trajectory, but not on absolute position.\" It’s unclear if this is a statement about what has been tried by the authors or what is present in the literature. In case of the latter, I would recommend the author to look at the citations I'm pointing out below.  Also it is not clear what they mean by naive. \n- Why H_0 is initialized with a concatenation of v_(s_0) with itself? is it one for the state and one for the cell of the LSTM? This should be clarified or motivated.\n- What happens if the images are provided at irregular intervals?\n- The authors used R^2 to determine whether the absolute position of the agent could be predicted from the artificial neurons. If this is applied directly on the embeddings then the authors should adjust it to account for the number of predictors in the model, which in this case  is high-dimensional. It’s not clear from the text  whether this was done or not so it should be clarified. Another option would be to use a linear decoder.\n- I couldn’t find anywhere the details of the hyperparameters used in the work\n- As a general comment there have been several recent papers that used unsupervised signal to learn allocentric representations (e.g. Buria et al, 2020;  Bicanski and Burgess, 2018; Whittington et al, 2018; Whittington et al, 2020) none of these work is actually cited\n\nMinor\nAuthors claim that a follow up direction could be to extend this work in 3D environments. However using inverse dynamic models in 3D could be more tricky due to an increased state aliasing. \n", "summary_of_the_paper": "The authors propose a network that combines propioception information with visual inputs to estimate displacement along a trajectory. The  new proposed network uses an inverse dynamic model and it is trained to do path integration. Such architecture and training regime result in a network that learns to reset and presents an internal state which correlates with absolute position in the environment.", "summary_of_the_review": "The authors claim to have obtained stable cognitive maps in a network trained to do path integration. However this claim is not supported by the results, as they show only a correlation between learnt embeddings and location in the environment. A cognitive map is a spatial knowledge about the environment, that could be used to guide behavior in a flexible manner. From the neuroscience literature when know that this knowledge is based not just on absolute position (place cells), but also on \n- an allocentric sense of direction (head direction cells)\n- boundary position (boundary vector cells)\n- and cells that support coding of metric distances as the animal moves through the world (grid cells). \n\nThis if we only report the most well known. So there is a lot more than just absolute location, but none of these features are present in the paper. For this reason I don’t think the authors can claim to have learnt cognitive maps.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "hKMauw4OAgr", "reviewer": "Reviewer_7DcK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In principle, a study of this type  is well suited to ICLR, and I think some of the analysis of internal dynamics in particular could be interesting. However the manuscript was difficult to follow at parts, and was missing critical details that make it impossible to evaluate. I also have reservations about the significance of problem being studied, as a full discussion of contemporary RL agents that can perform path integration is lacking. I discuss major and minor issues below. I would be happy to take a look at a revised version, as I feel some of the issues with the manuscript are clarity, but my gut feeling is the project may take more time to mature before acceptance is warranted. \n\n\nMajor issues\n(0)\tMy central issue with this work is whether the RPI architecture that is the main focus of the paper is interesting because it reaches state of the art precision on a key benchmark (path integration), or whether it is interesting because it is tractable to analysis. I am not convinced that this is a state of the art network for an important task, or conversely that the results of the network analysis are interesting enough to warrant acceptance on their own. This should be made clear. \n\n(1)\tInsufficient description of past work in reinforcement learning. While there is ample discussion of the problem of path integration in nervous systems, there is very little context on approaches in artificial systems, which is the main focus of the present manuscript. Autonomous naviation is a rich field, and while some references to recent work are given in the discussion, little is given in the introduction. This makes it difficult to ascertain the novelty of the proposed architecture, and assess how the RPI fits into other contemporary models for navigation. \n\n(2)\tMissing critical details about network training and task setup. I found it difficult to evaluate many of the claims in the paper. I could not find critical details about how the network was trained using RL and associated hyperparameters. I am missing an exact definition of the action space – can both the direction and angle be arbitrarily chosen? Relatedly in Figure 5 the network is described as using “specific actions, chosen to force exploration” this process is unclear, making it difficult to evaluate the proposed claims. In Figure 2 the output of this network is unclear. Is it outputting the final action? Is it also trained to output the state? I understand that Figure 3 is demonstrating the use of forward and inverse models, but I am not exactly sure how it fits in with Figures 2 and 4. \n\n\n(3)\tAnalysis of internal network properties. I am a bit confused about the coefficient of determination results in Table 1. I am not actually sure how these are computed. For instance with the activity-position correlation, how are these combined across the x and y dimensions? For the visual correlations, why do those vary in the LSTM cases? Wouldn’t one expect the visual networks to be similar across all the architectures? It is hard to evaluate because we are not given information on how the agents were trained, i.e. does this positional tuning just arise because these agents are heavily overtrained in a single environment? The R2 values are also very high. Even though it is clear that Figure 7 units are spatially tuned, it is not obvious that they would show an R2 of 0.99. \n\nThe link to cognitive maps is also a bit unclear here and overstated. The suggestion is that a global ‘place field’ like cell that is learned is necessary for a cognitive map (i.e. the RPI units with high R2 to position), but many other representations, include those in egocentric coordinates (ie grid cells) are also suitable as cognitive maps. As the authors note, decoding spatial position would be more convincing, as would showing that these representations could be utilized in different tasks, i.e. the classic tolman shortcut experiments. \n\n\n\nMinor points\nPg 19 – table reference in appendix G is missing\nPg 9 – table reference is missing\nI am confused about the term retina and the idea of this as a visual circuit. The information being received is highly local, making this more like a whisking system or some sort of tactile system. \nFigure 7 – how were these examples chosen?\n", "summary_of_the_paper": "This paper introduces several artificial models of path integration and discusses their relationship with biological systems. They train agents that receive noisy proprioceptive information and nearby visual input to minimize their path integration error when navigating in a continuous 2D environment. They compare two different architectures for path integration, based on a vanilla and modified LSTM (the resetting path integrator) that both receive proprioceptive and local visual input processed by two separate MLPs. Both architectures received intermittent visual feedback and used a gating mechanism to replace the internal state when feedback was present. They assessed what loss functions realized improved performance on the task, and found their RPI model showed improvements compared to LSTM variants. Finally, they analyzed the structure of internal representations in the network, showing networks capable of path integration showed high correlation with the absolute position in the environment, indicating that an internal representation of position was learned. ", "summary_of_the_review": "In principle, a study of this type  is well suited to ICLR, and I think some of the analysis of internal dynamics in particular could be interesting. However the manuscript was difficult to follow at parts, and was missing critical details that make it impossible to evaluate. I also have reservations about the significance of problem being studied, as a full discussion of contemporary RL agents that can perform path integration is lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "byCvcWfOCwu", "reviewer": "Reviewer_PuPV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Strengths\n\n- The design and motivation of the proposed method is well grounded in cognitive science, which is quite interesting.\n- The paper is well written, and there are sufficient details about the experiment design in the appendix.\n\n\n#### Weaknesses:\n\n- I am not exactly sure if ICLR is the right venue for this paper. Particularly, the problem that the authors study is a toy problem for navigation, it is 2D navigation with obstacles. Maybe it is common in the cognitive science community to use such environments to study the emergence of meaningful representations of the spatial structure of the environment. It seems in the AI community the focus is on complex 3D navigation problems[1]. I would recommend the authors to run the experiments on [1] so that the contribution of this work can be better evaluated. \n- Secondly, the network design is rather simplistic, it is a recurrent neural network with a single gate for resetting with external signals. It is not clear if this architecture will work beyond simple 2D navigation tasks. \nIn the abstract the author mentioned “state-of-the-art” LSTM for 2D navigation/localization problems, I am not sure where is this coming from. For example in [1], CMP performs better than LSTM. \n- In the embodied AI community, there are a lot of works on cognitive mapping (see [1][2][3]), but this work didn’t mention those, or compare with those. That’s also why I am not sure about the scope and audience of this paper. In other words, this paper might be aiming at understand biological systems, but how they can be used in AI agents or robotics is not immediately clear. \n\n#### References\n\n- [1] Gupta, Saurabh, et al. \"Cognitive mapping and planning for visual navigation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n- [2] Zhang, Jingwei, et al. \"Neural slam: Learning to explore with external memory.\" arXiv preprint arXiv:1706.09520 (2017).\n- [3] Chaplot, Devendra Singh, et al. \"Learning to explore using active neural slam.\" arXiv preprint arXiv:2004.05155 (2020).\n", "summary_of_the_paper": "The paper focuses on fusing external signals and proprioceptive signals for navigation tasks. They propose a direct-inverse model of environment dynamics to fuse image and action related signals. They propose an architecture, namely Resetting Path Integrator (RPI), that can easily and reliably be trained to keep track of its position relative to its starting point during a sequence of movements. RPI updates its internal state using the proprioceptive signal, and resets it when the image signal is present. The architecture outperforms LSTM in performance and interpretability, when benchmarked on a 2D navigation/localization problem. \n", "summary_of_the_review": "This work proposes a simple yet effective way to fuse external signals and proprioceptive signals. It is benchmarked on a 2D maze-like environment and shows better performance than LSTM. I am not able to fully understand the contribution of the paper since it doesn't compare with other state-of-the-art cognitive mapping methods, and it doesn't use standard benchmarks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ZHwJIAgVOv", "reviewer": "Reviewer_hpxs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The system relies on three losses; the path integration loss and the losses for the direct and inverse models respectively. The most interesting contribution is the gating network that selects either the visual state or forward predicted state for integration. In the experiments, it is shown how the gating network helps the system to reset the state when the visual input is unique to a particular location while relying on forward prediction in case of ambiguities. This is similar to the use of so-called keyframes in SLAM that are used for similar purposes.\n\nThe PI loss assumes that the true sum of all actions, or equivalently the location at each point, is available during training, which seems to be a critical assumption in practice. Given the system’s ability to reset, errors cannot accumulate more than what is done between each reset. In a practical scenario, whether it is a biological or an artificial agent, it is hard to see how this would work. In that sense, this is different from what is assumed in SLAM.\n\nInterestingly, the learned state depends not just on the particular location, but also on how the agent ended up there. When there are no ambiguities the gating allows the state to depend more directly on the visual input, while the opposite is true when there are ambiguities. The key component here seems to be the inverse model that is trained both to predict the action between states and the sum of actions back to the starting point, which forces the state to include this information.\n\nThe fact that the inverse model is supposed to work from the current space all the way back to the starting state, suggests that the corresponding neural network should have little opportunities to generalize beyond the starting states used during training, at least as long as there is not enough training data to avoid memorization. Since no information on how much data is used for training, one cannot really tell. However, since not only the PI loss matters, the inverse and direct models seem to introduce regularization that prevents pure memorization.\n\nIt is a bit unclear whether the proposed system is intended to model biological systems or to be used for artificial or robotic ones. Given how the conclusions are written and how noise is added to both the proprioceptive and visual signals to better emulate a real system, the biological connection seems to be most relevant. However, the simulated system is only shortly evaluated with respect to the degree of noise in Figure 6, which is left without much comment. It is also hard to tell whether the added noise really makes the simulation more realistic since no other data is used to support that. \n\nA weakness with the proposed formulation is that both direct and inverse models are deterministic, even if a probabilistic formulation would be more suitable, where the variance could be used as an indication of confidence which could affect gating. However, a probabilistic model would most likely be harder to analyse and require more training data. \n\nSomething that strikes one as odd is why P(a) maps an action a to the same dimension as V(s) does for a state s. Is this really necessary? Is P needed at all, since it is only two-dimensional? After all, P is used in (4), but not in (7). In most cases, the perceptual space is much larger than the action space, and there is nothing in the direct model D that should require the representations to be of the same dimensionality.\n\nA point that needs to be clarified is whether (9) is correct or whether Figure 4 is so. Currently, they do not show the same thing. By the way, there seems to be one circle missing to the left in Figure 5.\n\n", "summary_of_the_paper": "This paper proposes a model for path integration that combines visual and proprioceptive information, i.e. a model that keeps a prediction of where an agent is located, given the past sequence of actions and observations. What is proposed is to learn paired direct (forward) and inverse models (Wolpert, 98), where the direct model predicts perceptual changes due to actions and the inverse model the actions responsible for an observed perceptual change. Through a gating mechanism information is selected from either the current location or from the path towards the location, which in effect resets path integration when unique locations are visited while overcoming ambiguities that might appear elsewhere.", "summary_of_the_review": "The author should make it clearer what the expected target audience is, those who are interested in modelling biological systems, those interested more in artificial systems or those who search for inspiration in biological systems to build artificial ones. The paper does have its merits for each group. It might not be directly applied for e.g. robotics, but the fact that you can learn a state space that combines local and global information as suggested in this paper is worth reading about and could possibly lead to more applicable solutions in areas beyond biological modelling.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BjqhYrD0g7a", "reviewer": "Reviewer_AjJT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Main Review\n\n\nStudying spatial navigation within biological systems is interesting, especially from the computational learning perspective taken in this paper. To make an impactful contribution in this space requires a strong grasp of the cognitive science and the statistical machine learning literature, which the authors clearly have demonstrated. Critically, contributions in this space need to show that they can both fit within an existing conceptual framework of biological research, and provide a computational / statistical benefit.   \n\nThe ideas in this paper are conceptually well-grounded. However, in the manuscript's current form, the empirical support is too weak to sufficiently support its claims. For this reason and several others I expand on below, I believe this paper is not ready for publication. However, I am open to changing my score depending on the responses, and the degree to which my concerns can be addressed. \n\n### Detailed review\n\n\n**The architecture** described in Figure 2 fuses information from two sensory streams. A visual stream is encoded using a convolutional neural network then concatenated with information coming from a proprioceptive stream that has been encoded with a fully-connected network. Architectures that encode then concatenate multiple sensory streams has been extensively studied. See [1] for a recurrent architecture that fuses visual and auditory streams, see [2] for an instance combining two visual streams, see [3] for an architecture that combines visual, tactile, and proprioceptive streams, and [4] for a comprehensive survey of other examples. \n\nThe paper claims that \n> naive attempts at performing PI using the concatenation of proprioceptive and visual signals yield very unsatisfying cognitive maps which depend on position along the trajectory, but not on absolute position. \n\n\nSince the proposed architecture (RPI) concatenates proprioceptive and visual signals, I think the paper is taking issue with the way these signals could be encoded, or rather how they would not be encoded in a naive approach. Can the authors please clarify the point they’re trying to make? Based on prior work mentioned above, any approach that concatenates encoded signals would be natural.  What are the qualities of a cognitive map that would be absent from this approach?\n\nFinally, the joint encoding is used as input to a recurrent neural network then trained to predict total displacement. \n\n**Training the network** is accomplished with two loss terms that enforce a reconstruction of encodings to respect the temporal structure of a Markovian observation process.  A couple things about this confused me:\n\n\n1. The observation process is formalized as a Markov decision process. However, the paper does not mention anything about a reward signal, which is a critical feature of this formalism. Is the reward signal relevant to this work? \n2. The paper describes some shortcomings of seperately training their encoding networks. However, it was never clear how the separate loss terms (4) and (5) were combined so the full network could be trained end-to-end. \n\nOne of the main points of novelty in this work is **a resetting mechanism**—introduced to control the internal recurrent state and allow the system to use predicted encodings from the transition model or the direct encoding of the next state. Resetting is implemented with a gating network that computes a convex combination of the prediction and the next state's encoding. The paper implicitly claims that this mechanism is beneficial in reducing the accumulation of prediction errors, presumably whenever the prediction is more accurate than the direct encoding. A few questions:\n\n\n- Why would the model's prediction of the next state encoding be more accurate than the encoding that is grounded in direct experience? Please correct me here if I’m misunderstanding the significance.\n- How is the gating network trained to impose beneficial resets?\n\nAnother point made in this section gave me pause.\n> We therefore expect the internal state of the network to strongly depend on the current value of the position, but not on the trajectory used to get there\n\n\nHow can you guarantee that the internal recurrent state doesn’t depend on the history of states? Isn’t the purpose of recurrence to account for the past? \n\n\n### Empirical Results\n\n\nThe main hypothesis claims the proposed RPI architecture can integrate long paths with less error than architectures without a resetting mechanism. The experiments consider a two-dimensional domain and compare path integration errors in two data regimes, with over trajectories of 5 and 100 time steps, respectively. Results are tabulated with uncertainty quantified over eight trials.\n\nIn addition, the experiments compare performance of each system trained on different losses. In one case, an inverse model is used, and in the other case the inverse model is absent. \n\nGenerally speaking, the experiments seemed to ask most of the right questions. There appears to be some positive support for the utility of resets within Table 1. However, it’s not clear if the results are entirely positive because most of the confidence intervals overlap. In addition, I struggle to understand the significance of the second experiment (comparison of losses). I also have a few issues involving the evaluation, baselines, and significance of some results which I describe below. \n\nIn general, more information is needed to understand the baselines. The significance of this experiment is less clear to me\n\nThere are two baselines that are necessary to establish the utility of the proposed architecture.\n\n\n1. RIP with [\\mathcal{G}=0](tex://\\mathcal{G}=0). The recurrent state always uses the direct encoding.\n2. RIP with [\\mathcal{G}=1](tex://\\mathcal{G}=1). The recurrent state always uses the prediction.\n\nComparing to the first baseline establishes that the system can benefit from prediction. In the second case—which I believe is similar to the Vanilla LSTM—the comparison implies something about the benefit of resets (i.e. direct experience). To fully support the main hypothesis, the data would need to show the learned RPI system achieves a lower path integration error than these two extremes.  Does the Vanilla LSTM reflect the [\\mathcal{G}=1](tex://\\mathcal{G}=1) case?\n\nIn both cases a new observation is provided every five steps. This is different from the standard Markov environment described in the formalism; where the system gets an observation at each step. I have a feeling like this property of the observation process is critical to the utility of resets. Can the authors comment on this and describe how they believe the system would perform as this hyperparameter varies? \n\nI have several issues with Table 1. and how it’s presented. It is unclear what the errors represent and what sources of randomness are reflected between experimental trials. The experiments only used eight trials, and it is apparent from the way confidence intervals overlap that more should have been used. I suggest this experiment is rerun with thirty or more trials to improve significance. \n\nAdditional claims regarding the significance of the [R^2](tex://R^2) comparisions were less clear. What conclusion should be drawn from these results?\n\nIn Section 4.2 an experiment was performed to in situations with ambiguous observation information. The line of questioning here was not explicit or clear from the writing. Can the authors elaborate on the purpose and significance of these results (Figures 6 and 7.) What does a positive result look like in Figure 7?\n\n### Additional Comments and Questions\n\n- The mathematical notation was effective, even though it was ambiguous at times.\n- Clarity could be improved if variables such as [\\mathbf{\\Delta r}_t](tex://\\mathbf{\\Delta r}_t) were mathematically defined.\n- How is the average over trajectories computed? Are errors first summed then averaged? \n- Consider adding a requirements.txt file to make it easier for others to install code dependencies and run your code.\n- There are many references to the Appendix; it would help if more information was brought into the main text or the additional material was excluded.\n- The related work gave sufficient coverage of the cognitive sciences, but perhaps less to the ML audience members of ICLR.\n\n\n> We insist that our observations, while processed by a convolutional network, are not standard images in a ”human-readable” format, but rather a spatially organized array of sensors.\n\n\nCan you expand on the difference between your “spatially organized” observations and the observations coming from a standard pixel grid?\n\n### References\n\n\n[[1] Crossmodal Attentive Skill Learner](https://arxiv.org/pdf/1711.10314.pdf)\n\n[[2] Multimodal Deep Learning for Robust RGB-D Object Recognition](https://arxiv.org/pdf/1507.06821.pdf)\n\n[[3] Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks](https://arxiv.org/abs/1907.13098)\n\n[[4] Recent Advances and Trends in Multimodal Deep Learning: A Review](https://arxiv.org/pdf/2105.11087.pdf)\n", "summary_of_the_paper": "This paper considers the problem of path integration for the purpose of determining an agent’s position in space. The authors propose a recurrent neural network architecture that fuses information from two sensory streams and integrates the representation to predict total displacement from a starting point. Their main innovation is a gating network that “resets” the hidden state. The paper claims this helps to reduce the accumulation of errors compared with other non-resetting systems. To validate this claim, they compare prediction accuracy in a two-dimensional spatial domain. The paper also makes the following additional claims\n\n\n1. They claim this architecture is a good candidate for a cognitive map.\n2. The proposed architecture consistently shows better performance than baseline LSTM architectures on the same problem.\n3. The proposed architecture’s internal dynamics are more interpretible than a baseline LSTM.\n4. The proposed architecture produces higher quality representations.", "summary_of_the_review": "I am currently **leaning to reject the paper**, because in its current form, the empirical support is too weak to sufficiently support its claims. \n1. I believe the experiments need to include an additional baseline\n2.  The writing should have more details about the domain and methodology. \n3. Some line of empirical questioning seems disconnected from the main claims described in the motivation.  \n4. I cannot be sure the proposed architectural components make a statistical difference, because the reported confidence intervals overlap.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of path integration in cognitive maps, where combining proprioception with visual inputs is required to estimate the displacement.  The paper proposes a small mechanism (a resetting path integrator) that extends a conventional LSTM for this purpose.  The resulting networks demonstrate better performance and interpretability than a conventional LSTM on tested problems.

The reviewers raised many issues with the paper.  One concern was whether the problem was to model biological, artificial, or robot problems (reviewer hpxs, PuPV), which the authors successfully addressed by stating that it is a minimal model. Many other minor concerns were also addressed. However, significant concerns remained.  One is the emphasis on the cognitive map (reviewer CUYu, hpxs) for which path-integration is a small part.  Another major concern is the significance of the results, with reference to the baselines and $R^2$ (AjJt, CUYu).  A third is on the generalizability of the method beyond single small examples (AjJT, hpxs, CUYu).

All reviewers indicate reject due to concerns that the paper is not ready for publication.  The paper is therefore rejected.
Decision: Reject