Prediction and generalisation over directed actions by grid cells
Authors: Changmin Yu
Timothy Behrens
Neil Burgess
Venue: ICLR 2021 Poster (2021)
Keywords: Computational neuroscience
grid cells
normative models

Abstract: Knowing how the effects of directed actions generalise to new situations (e.g. moving North, South, East and West, or turning left, right, etc.) is key to rapid generalisation across new situations. Markovian tasks can be characterised by a state space and a transition matrix and recent work has proposed that neural grid codes provide an efficient representation of the state space, as eigenvectors of a transition matrix reflecting diffusion across states, that allows efficient prediction of future state distributions. Here we extend the eigenbasis prediction model, utilising tools from Fourier analysis, to prediction over arbitrary translation-invariant directed transition structures (i.e. displacement and diffusion), showing that a single set of eigenvectors can support predictions over arbitrary directed actions via action-specific eigenvalues. We show how to define a "sense of direction" to combine actions to reach a target state (ignoring task-specific deviations from translation-invariance), and demonstrate that adding the Fourier representations to a deep Q network aids policy learning in continuous control tasks. We show the equivalence between the generalised prediction framework and traditional models of grid cell firing driven by self-motion to perform path integration, either using oscillatory interference (via Fourier components as velocity-controlled oscillators) or continuous attractor networks (via analysis of the update dynamics). We thus provide a unifying framework for the role of the grid system in predictive planning, sense of direction and path integration: supporting generalisable inference over directed actions across different tasks.

Published as a conference paper at ICLR 2021
PREDICTION AND GENERALISATION OVER DIRECTED ACTIONS BY GRID CELLS
Changmin Yu Timothyy E.J. Behrens Neil Burgess
Institute of Cognitive Neuroscience, UCL. London, UK
"Centre for Artificial Intelligence, UCL, London, UK "Wellcome Centre for Integrative Neuroimaging, University of Oxford, Oxford, UK *Sainsbury Wellcome Centre, UCL, London, UK
ABSTRACT
Knowing how the effects of directed actions generalise to new situations (e.g. moving North, South, East and West, or turning left, right, etc.) is key to rapid generalisation across new situations. Markovian tasks can be characterised by state space and a transition matrix and recent work has proposed that neural grid codes provide efficient representation of the state space, as eigenvectors of transition matrix reflecting diffusion across states, that allows efficient prediction of future state distributions. Here extend the eigenbasis prediction model, utilis- ing tools from Fourier analysis. to prediction over arbitrary translationiinvariit directed transition structures (i.e. displacement and diffusion), showing that a single set of eigenvectors can support predictions over arbitrary directed actions via action-specific eigenvalues. We show how define "sense of direction" " to combine actions to reach target state (ignoring task-specific deviations from wranslationiivaaiinnee and demonstrate that adding the Fourier representations to deep network aids policy learning in continuous control tasks. We show the equivalence between the generalised prediction framework and traditional models of grid cell firing driven by oscillatory interference (via Fourier components as relocity-controlle continuous attractor networks (via analysis of the update dynamics). We thus provide a unifying framework for the role of the grid system in predictive planning, sense of direction and path integration: supporting generalisable inference over directed actions across different tasks.
to perform path integration, either using oscillators)
INTRODUCTION
A "cognitive map" encodes relations between objects and supports flexible planning (Tolman [40] with hippocampal place cells and entorhinal cortical grid cells thought to instantiate such a map (O' Keefe and Dostrovsky B2]; Hafting al. (0)). Each place cell fires when the animal is near a specific location, whereas each grid cell fires periodically when the animal enters a number of locations arranged in triangular grid across the environment. Together, this system could support representation and flexible planning in state spaces where common transition structure is preserved across states and tasks, affording generalisation and inference, e.g., in spatial navigation where Euclidean transition rules are ubiquitous (Whittington et al. 43)).
Recent work suggests that place cell firing provides local representation of state occupancy, while grid cells comprise an eigenbasis of place cell firing covariance (Dordek et al. [15]; Stachenfeld et al. 38] Sorscher et al. [37]: Kropff and Treves (26).. Accordingly, grid cell firing patterns could be learned as eigenvectors of symmetric (diffusive) transition matrix over state space, providing basis set enabling prediction of occupancy distributions over future states. This "intuitive planning" operates by replacing multiplication of state representations by the transition matrix with multiplication of each basis vector by the corresponding eigenvalue (Baram et al. (2]: Corneil and Gerstner [13).. Thus distribution over state space represented weighted sum of eigenvectors re- weighting each eigenvector by its eigenvalue to predict future state occupancy.
be updated by
" Pllasee send any enquiries changmi yu. 198ucl ac. uk and n. burgesssucc .ac.uk

---

Published as a conference paper at ICLR 2021
Fast prediction and inference of the common effects of actions across different environments is important for survival. Intuitive planning. in its original form, supports such ability under a single transition structure, most often corresponding to symmetrical diffusion (Baram et al. (2). Here show that a single (Fourier) eigenbasis allows representation and prediction under the many different directed transition structures corresponding different "translation invariant" actions (whose effects are the same across states, such as moving North or South or left or right in an open environment), with predictions under different actions achieved by action-specific eigenvalues. We define "sense of direction" quantity, i.e., the optimal combinations of directed actions that most likely lead to the goal, based on the underlying connslttioninvariann transition structure (e.g., ignoring local obstacles). We then show how this method could be adapted to support planning in tasks that violate translation invariance (e.g. with local obstacles), and show how adding these Fourier representations to a deep RL network improves performance in continuous control task.
We propose that the medial entorhinal grid cells support this planning function, as linear combinations of Fourier eigenvectors and therefore cigenvectors themselves, and show how traditional models of grid cells performing path integration are consistent with prediction under directed actions. Hence demonstrate that the proposed spectral model acts unifying theoretical framework for understanding grid cell firing.
"INTUITIVE PLANNING" WITH SINGLE TRANSITION STRUCTURE
Intuitive planning represents the occupancy distribution over the state space as weighted sum of the eigenvectors of a single transition matrix (usually corresponding to symmetric diffusion), so that the effect of one step of the transition dynamics the distribution each of the eigenvectors by the corresponding eigenvalue. And this generalises to calculating the cumulative effect of discounted future transitions (Baram et (2).
be predicted by reweighting
Specifically, consider a transition matrix, T' E RNXN, Tas = (( s'|st s) where 9t encodes the state time and N is the number states. Then, 7 the nestep transition matrix, and has the same set of eigenvectors as T. Specifically, the eigendecomposition of T and 7 are:
QAQ
T =AAQQ-
where each column of the matrix Q is an eigenvector of and A = diag(o where (T)) is the set of eigenvalues of T. Similarly, any polynomial in T, p(T), shares the same set of eigenvectors as T and the set of eigenvalues (p())) = p(ar(T)). Hence:
E6TT)* = -I- Qdiag(w)Q where w ( 1 for for E E
The resolvent form (Eq. 2) is infinite discounted summation of transitions, which under a policy and transition structure corresponding to diffusion, equivalent to the successor representation (SR, Fig. [ with discounting factor (Dayan [14; Stachenfeld et al. (8)) See Mahadevan and Maggioni [29] for a related spectral approach using Fourier decomposition of T for estimating the value function. The SR has been shown be useful for navigation via gradient ascent of the future probability of occupying the target state, and has a linear relationship with the true underlying Euclidean distances in spatial tasks (hence "intuitive planning". see Fig. and Fig. D-F)
A
State 'B
C
state
state
Figure 1: Demonstration of intuitive planning diffusive transition task 1D ring track. A: Example transition matrix; B: P( s'|s; 5): C, D: same are shown for T3, showing predicted distribution over the three time steps; the resolvent form/SR (Eq. computed from the eigenbasis of the transition matrix; F: SR values for state 5, which can used for navigation.
The eigenvectors of the diffusion transition matrix generally show grid-like patterns, suggesting close relationship grid cells. However, intuitive planning is restricted to predictions over
2

---

Published as a conference paper at ICLR 2021
single transition structure, hence cannot flexibly adjust its predictions corresponding to the effects of arbitrary directed actions (i.e., variable asymmetric transition structure), hence cannot support the presumed role of grid cells in path inegration.Morooveer predictions over different directed actions would require different eigendecompositions, hence incurring high computational costs that undermines its biological plausibility. In Section unify the prediction and path integration approaches by exploiting translation invariant symmetries to generalise across actions, using single common eigenbasis and cheaply calculated updates via action-dependent eigenvalues.
3 FLEXIBLE PLANNING WITH DIRECTED TRANSITIONS
Updating state representations to predict the consequences of arbitrary directed actions is an important ability of mobile animals, known as path integration and thought to depend on grid cells (McNaughton et al. [30]). To generalise the intuitive planning scheme to simultaneously incorporate arbitrary directed transition structures, we consider the transition dynamics corresponding to translation (drift) and Gaussian diffusion with arbitrary variance (including O, equivalent to plain translation). Our assumption that the transition structure is translation invariant (implying periodic boundary conditions), leads to circulant transition matrices.
Consider 2D rectangular environment with length and width W where each state is node the unit square grid, then the transition matrix can be represented by T R xLW with each row the vectorisation (vec(-) of the matrix of transition probabilities starting from the specified location, i.e., T vec
[IP( ($c+1 (st = j)], where T is constructed by considering the 2D state space ID vector and concatenating the rows (j = xL + y for (z, y) [0, W 1] x [0, L - ID), see Fig 2
The transition matrix is circulant due to the translation invariance of the transition structure (see Appendix Prop. A.I and takes the following form:
To
TEW-1
Ti
Ti
To
TLW-1
T2
T=
Ti
To
TLW-2
TLW-1
TLW-1
TEW-2
Ti
To
The normalised eigenvectors of the circulant matrix T RNXN (N LW) are the vectors of powers of Nth roots of unity (the Fourier modes):
1 VN
[1, wk. w
.u -
qk
where wk = exp 2 k), for k O,.... ......N N 1. and i = VII. Hence the matrix of eigenvectors (as the columns), (Bracewell (4), where Fs₁ = w for 0 < k.j < N 1. The Fourier modes projected back onto the W 2D spatial domain are plane waves, as shown Fig. ZGg with wavevector determined by the value of k that specifies the direction and spatial frequency of each plane wave (see Appendix BB. We can immediately compute the corresponding cigenvalues for the cigenvectors in Eq. 4 (cquivalent to taking the discrete Fourier transform (DFT) of the first row (or column) of T, see Bracewell 4):
(qu q1 q...... qv-1), just the (inverse) discrete Fourier transform matrix
-
Am
Σ Trws": for m = 0, O,...,M. 4=0
(5)
where (To....,Ty_1) are the N unique elements that fully specifies the circulant matrix T' (Eq. 3
We can then utilise tools from Fourier analysis for efficient updating of the eigenvalues whilst leaving the universal eigenbasis unaffected. For transition matrix T corresponding an arbitrary action (translation velocity) v = (Uz, vy), each row of TY is again circulant, but shifted version of the corresponding row vector of the symmetric transition matrix corresponding to zero drift velocity, To Specifically, the first rows of the two matrices are related as follows:
Tv (k) ( Uy T. Uy): for h: -....,N
3

---

Published as a conference paper at ICLR 2021
A
meett
Intuitive distance
10
20
True distance
Figure 2: Demonstration of our method in 2D 50 50 environment with periodic boundary
conditions. A: Transition matrix P(x++1 = (i', (')x; - (i) starting from a randomly chosen state ((8, 10); red star) with drift velocity 10 units southward and Gaussian diffusion (red arrow); B: Usage of the eigenbasis (Eq. 4 and our analysis (Eq.(7 for predicting the distribution over future states given the transition structure given in A. showing successive changes to state occupancy: C: Application of our model to translation-only transition structures (drift velocity v = (3,3)): (D-E): Ground-truth shortest distance state (8, 20) (red star; D) function of the starting location, and distance estimated under the intuitive planning framework (E) using plain diffusion; F: Estimated distance measure (E) v.s. the corresponding ground-truth distance (D) over all pairs of states; G:
Examples of 2D Fourier modes (real parts shown, see Appendix Fig. 7 for the top 100 eigenvectors).
Given the eigenvalues for T A = [X] .i...,,, 2% CN (via the DFT of the first row of T Eq. we can immediately derive the eigenvalues of Tv, AV, via a one-step update based on the Fourier shift theorem (Bracewell [4] without recomputing the eigendeconposition:
2 N
A [k] exp
(v (vxL+ uw)k [k),, for k = O...., --1, 1, for arbitrary v.
[1. w wgw)). No. [.]!
ie.. A = 2 A
2s(v)
where 8(v) = Up + By
This allows path integration by reweighting the common set of eigenvectors at each timestep by the updated eigenvalues corresponding the current drift velocity (Eq. 7). Note that additionally, 7 can include diffusion, thus reweighting by the eigenvalues of the diffusive transition matrix also allows tracking of increasing uncertainty.
Utilising the fixed eigenbasis (Eq, 4 and the respective cigenvalues (Eq. 7 for arbitrary transition structures, we can make efficient prediction for the distribution of future state occupancy with respect transition matrix
to arbitrary action (see Figs. 2 3-C). Adding translation to the translation-invarte does not change the set of eigenvectors allowing one set of eigenvectors (Fourier modes) to support prediction for actions in all directions (or plain diffusion), hence prediction of effects of directed actions can be efficiently generalised across environments.
Sense of Direction. We define "sense of direction", 0", the angle of the transitions (or the linear combinations of the available actions non-spatial setting) that maximise the future probability of reaching the target state given an initial state, which is modelled by the SR matrix.
exp expp2mi((x: - Xx) k D, exp|2rive ky]
l" = arg max Σ
(8)
where is the discounting factor, Dj 1.... LW are the eigenvalues for the symmetric diffusion transition matrix, ky.j = 1,.... LW are the wavevectors for the j-th Fourier components, X XG the coordinates of the start and goal states, and Va (v cos(0), sin(0) represents the velocity (with speed v and head direction 8). We see that the "sense of direction" supports generalisation of predictions of effects of actions across all environments with the same translation-invarian transition
4

---

structure, i... such predicted effects ignore any local deviations from translation invariance. See Appendix [ for the derivation of Eq. Note that here we assume that the goal state So is known priori, e.g., consider a problem where the animal is navigating towards previously visited location. The derived analytical expression for the sense of direction can be retrieved via a lookup table when the state space is small and discrete, whereas in large continuous state spaces, it can he computed either via optimisation algorithms, or modelled by a non-linear function approximator that for neural network approaches to finding goal directions from Published as a conference paper at ICLR 2021
represents Eq. [ See Bush et grid representations.
We thus propose that computational role for the neural grid codes: generating a "sense of direction" (capturing the transition structure of the state space, ignoring the obstacles and boundaries) that reflects a global sense of orientation which allows generalisation to completely new environments.
Flexible Planning & Application Beyond Translation- Invariant Structures. The proposed model can be applied to flexible planning under arbitrary drift velocity as demonstrated in Fig. (A-E). An agent is trying navigate towards goal state in windy grid world. The navigation is performed by following the ascending "gradient" of the SR for occupancy of the target state (the resolvent metric, Eq. 2). The SR computed from the transition matrix including the effects of diffusion and wind
(Fig. B) based on our analysis (eq. 7 leads straight to the target (Fig. 3 C)..
Given the analytical expression of the SR (Eq. 2). we could efficiently adjust the SR matrix to accommodate local changes in the state space, e.g., insertion of a barrier, using the Woodbury inversion formula to update the parts of the SR matrix affected by the local obstacles (see Appendix AA.3 for derivations (54) ); and again in this case, the agent correctly adjusts for the wind as well as taking the shortest path around the inserted wall (Fig. D-E)..
We note, however, that the proposed model is also able to solve tasks without periodic boundary conditions, by considering the original task state space, So. being embedded into larger, periodically bounded pscudo state space Sp, at least twice as large in each dimension as So (Fig. 3 F). We again follow the previous procedures, utilising the Fourier modes, this time computed on Sy, to perform predictions in So (Fig. 3 F-G), and the performance is unaffected. Note that under such formulation, the underlying transition structures can be applied to environments with both periodic and non-periodic boundary conditions allowing sense of direction planning in either case.
Path Integration. We can also use our model for path integration (see also Section 4 in So, by taking velocity inputs (given any path in the grid world) to update the state occupancy distribution (Eq. 7 The path integration performance is strongly correlated with the degree of uncertainty (i.e.. the diffusion strength caused by self-motion noise in addition to translations). This is indeed captured by our model (Fig. H), with perfect path integration when the uncertainty is low up to 1000 time steps (the discretisation of state space means that uncertainty below 0.075 has no effect ), and monotonically increasing path integration error when the uncertainty is higher.
3.1 NEURAL IMPLEMENTATION FOR DEEP REINFORCEMENT LEARNING
Our proposed framework supports prediction and planning under arbitrary direction actions and path integration. To further demonstrate its utility in non-spatial tasks, we propose GridCelI-DQN (gc- DQN), a neural network implementation based on modified version of the classic Deep-Q Network (DQN, Mnih et al. (3I). The architecture of ge-DQN is designed so that quantities corresponding to combinations of Fourier modes weighted by action-dependent values are explicitly available to action-value computation in addition to value estimates based on the state-inputs alone. This should enable the network to predict future state occupancy and thus facilitate the learning of Q values. We evaluated the performance of ge-DQN on the CartPole task (Barto et al. 3 and compared with plain DQN.
We restrict our introduction of the ge-DQN architecture based on the evaluation on the CartPole task. The state space of the CartPole task is 4-dimensional, corresponding to the cart position and velocity, pole angle and angular velocity, and there are 2 possible actions (0 and 1 corresponding to cart rule (Sutton
movement left or right). The Q-values are learnt using the standard and Barto (. The ge- DQN has 2 additional sub-networks (below the standard DQN in Fig. A): the first takes as inputs the n low-frequency Fourier modes over the state space after discretisation into 81 bins and has Pactions 2 outputs allow representation for each action (left right) given
5

---

oscillatory inputs Published as a conference paper at ICLR 2021
G
A
B
C
D
Bounded
Bounded grid
varw0.1
E
F
G
0.4
0.2
0.0
Figure Application spatial navigation in grid worlds. The 10 x 10 windy grid world task environment, with toroidal boundary conditions and constant external force causing two units of displacement southward acting on every state (white arrow: wind; blue arrow: one-step action rightward; green arrow: actual displacement, G: goal state); B: Estimated SR (using Eq 2 and Eq. under diffusion plus the wind effect (color indicates the strength future probabilities of occupying the target states); C: Path following the diffusion SR plus the wind effect; D: Updated diffusion SR plus wind given the insertion of a barrier (dark blocks); E: Path following the updated SR; F: Navigation in the task space (So. yellow) with boundaries (magenta) embedded in pseudo state space (Sp, blue), utilising the Fourier modes computed from Sp: G: Navigation in So with inserted local obstacles (black); H: Path integration error over timesteps under different levels of diffusion.
7
the input Fourier modes; the second takes as inputs the state variables and action and has 2n outputs which serve as action-dependen multipliers to the connection weights from the input layer of the first network. The second sub-network receives state as well as action inputs to mitigate the absence of connslation-ivvaiinnn
The outputs of the first network and of the standard DQN are fully connected
to an output layer learn the updated Q values. We also evaluated a model-based version ge- ge-DQN based on deep Dyna agents (Peng et al. (33] Preliminary results in Fig. 4 show that the ge-DQN and deep ge-Dyna-Q Q greatly accelerates learning comparing to the baseline agents, with relatively minor increase in the model complexity and computational costs. The results support hypothesis that Fourier eigenvectors weighted by action-specific values can aid prediction (in this case, of future value). See Appendix, E for details.
The focus of this paper is proposing theoretical framework for state representation, prediction, planning and path integration via grid-like eigenvectors and setionndeeendent eigenvalues. The proposed ge-DQN is only a preliminary attempt towards a neural network implementation of the proposed approach (see also Mahadevan and Maggioni (2]). more rigorous studies in this direction is left for future work.
4 A UNIFYING FRAMEWORK FOR MODELS OF GRID CELL FIRING
Our focus so far has been on proposing a flexible and efficient extension of the prediction models of grid cells (Baram
[2] Dordek al. [15]: Stachenfeld al. (38)) arbitrary directed transitions. However, many other computational models of grid cells emphasise path integration rather than inputs from place cells, such as continuous attractor network (CAN) models, which grid-like patterns emerge in recurrently connected networks performing path integration (Fuhs and Touretzky [8]: Burak and Ficte (6]; Corneil and Gerstner (13),, and oscillatory interference (OI) models in which grid-like patterns reflect coincidence detection of velocity-dependee during path integration (Hasselmo [22]: Burgess [8]; Welday et al. [42 Here we build upon the previous work of unifying the prediction and path integration models of grid cells firing by Sorscher et al. B7), to show the equivalence of our generalised prediction framework with the CAN models of grid cells; additionally show the equivalence between the proposed model and oscilla-

---

Published as a conference paper at ICLR 2021
gc-DQN
DOW, weder
DON wodes
Output
Performance on the CariPole task
200
DGN
seep Dyna-O
52n d0 uepe bndting
175
5003 go Oynay
gC CCDON
150
Input: modes
Fourier Iiddon 1
Oumput: action
125
wetights
So_ hwddenn 1
Cuupuit:
(stare, settion)
25-
pliers
20
100
Eploodes
Figure 4: Neural network implementation of the proposed grid cell model for reinforcement learning. A: Schematic illustration of the ge-DQN agent. B: Evaluation of ge-DQN and the baseline DQN agents in the CartPole task (Barto et al. (3). The evaluations are computed given random seeds. See Appendix, E for details of the implementation.
tory interference models in terms of their interpretations of path integration and theta phase precession.
4.1 RELATION TO CONTINUOUS ATTRACTOR NETWORK MODELS OF GRID CELLS
One of the most prominent unifying analyses for different grid cell models (Sorscher et al. (77] proves the equivalence between maximising a spatial representation objective function under the prediction models and the pattern formation dynamics of CAN models of path integration. Their analysis, however, does not include non-zero velocity inputs, corresponding to the asymmetric velocity-dependent connectivities in CANs which perform path integration (Fuhs and Touretzky [18]: Burak and Fiete ()). We can explicitly address this equivalence using our framework. Assuming that grid cell firing rate, 9. reflects linear combinations of selected Fourier modes (e.g., 6 Fourier modes at 3 radians increments with the same frequencies):
where fj's 's are the selected Fourier modes with weights wy.
Note that our proposed model is on the theory-level, rather than the implementation level of CAN models, hence we do not assume any specific neural network structure here. Utilising the grid cell firing described by Eq.9) followed by similar analysis as in Sorscher et al. [37], we show that, under non-zero velocities, the differential equations governing the dynamics of the CAN model updates are equivalent to the derivative of the Lagragian equation underlying the optimsation problem of the prediction models (up to scaling factors). Hence we show that under non-stationary transition dynamics, the CAN models and the prediction models should yield identical updates to grid cell firing (up to scaling factors). The complete proof can be found in Appendix
4.2 RELATION TO OSCILLATORY INTERFERENCE MODELS OF GRID CELLS
Another major class of computational models of grid cells is the oscillatory interference model, here show the equivalence between the generalised prediction model and the OI models of grid cells by showing that they perform path integration via similar phase coding.
In Ol models of grid cells, path integration is achieved via the phases of the "velocity controlled oscillators" (VCOs), which encode movement speed and direction by variations in burst firing frequency. The VCOs generate grid-like firing patterns via coincidence detection by the grid cells
7

---

Published as a conference paper at ICLR 2021
-22
A
B
Figure 5: Generated grid-like patterns given our model. A: 6 input Fourier modes with the same wavelength (complex phases shown); B: Simulated grid cell firing given real rat trajectory (gray line) using coincidence detection and baseline modulation given the 6 Fourier modes inputs in A, each spike represented by colored scatter with the color indicates the corresponding "theta" phase at the spiking time; C: Simulated grid cell firing fields given multiple runs in the same direction (black arrow) showing theta phase precession.
Welday et al. (42). The variation of frequency with velocity produces phase code for displacement, enabling the modelled grid cells to perform path integration. Namely, VCOs change their frequencies relative to the baseline according to the projection of current velocity, (f), onto the VCO's "preferred direction". d:
(Burgess [8: Hasselmo [22]
(a( = (b(1) + (v(1)-
(10)
where 3 is a positive constant, and Ss(() is the baseline frequency (the 4 - 11Hz EEG theta rhythm). It follows that VCOs perform linear path integration since the relative phase between VCO and baseline at time t, (t) @ (t) (t), is proportional to the displacement in the preferred direction:
S. 2 (f() fu(r)]dr S x(0)) d
Φus (0)
(11)
where x(f) is the agent's ' location at time t. The interference of VCOs whose preferred directions differ by multiples of 7/3 generates grid-like patterns, provides an explanation of "theta phase precession" in grid cells (in which firing phase relative to the theta rhythm encodes distance travelled through the firing field: Hafting al. [21]: Burgess [8] and complements the attractor dynamics given by symmetrical connections between grid cells (Bush and Burgess (9). We note that the main experimental results held against Ol models (that bats and humans do not have reliable theta frequency oscillations) has recently been resolved: the required phase coding (theta phase precession) can occur relative to a variable baseline frequency (Bush and Burgess [0] and has now been found in both bats and humans (Eliav et al. [17] Qasim et al. 35)..
We simulated the firing its complex phase in the current state, as leaky integrate and fire neuron performing coincidence detection, using a real trajectory of a rat exploring a 50cm x 50cm box. At each time step (corre- sponding one theta cycle), the phase of each Fourier mode is updated according to Eq. given the current velocity, and fires a spike at this phase if it is within the interval [- /44, /4] (modelling modulation by the baseline theta oscillation). The grid cell fires spike at the current location if the integrated inputs reach a threshold. Note that we could also simulate a set of grid cells, with different offsets (depending on the initial phases of the Fourier modes) and different scales and orientations (depending on the choice of Fourier modes), such that the grid cells, like the Fourier modes, comprise a basis for the state space and do so on the basis of path integration (for which environmental inputs are only required to prevent error accumulation S
grid cell with 6 Fourier mode inputs (Fig. AJ--), each firing a spike at
Grid cells show "precession" " in their firing phase relative to theta as the animal moves through the firing field (signaling distance travelled within the field; Hafting et al. [21]: Jeewajee et al. [23 Climer et al. [2].. Our model captures this, like an OI model. The Fourier modes whose wavevectors that are aligned with the current direction of translation advance in phase as the movement progresses. Phase precession results from assuming that the Fourier modes aligned with movement direction are the dominant influence grid cell firing (c.f. those aligned to the reverse direction). The baseline "theta frequency" corresponds to the mean rate of change of phase of all Fourier components and so could vary (e.g., for noise reduction, see Burgess and Burgess 77: Burgess []), without precluding
8

---

Published as a conference paper at ICLR 2021
phase coding (Eliav al. [17]; Bush and Burgess [0)). By simulating straight runs, we clear late-to-early phase precession (Fig. 5 C), as observed in grid cells.
Thus, the OI model and our model perform path integration or prediction in the same way: the phase of each VCO changes corresponding to the component of translation along the VCO's preferred direction, which is exactly analogous to the complex phases of the Fourier modes being updated to reflect transitions along their wavevectors (multiplication by corresponding eigenvalues, Eq.
5 DISCUSSION
Understanding how different actions affect the agent's 's state across environments is essential for generalisation. Existing models are capable of such prediction under single fixed transition matrix, e.g. corresponding to symmetrical diffusion, by using eigenvectors of this transition matrix as a basis for representing state occupancy (Baram al. (2): Corneil and Gerstner [33: Stachenfeld et al. (38)). Here we generalised these models to provide a mathematical framework for predicting the effects of specific actions, so long as their effects (and corresponding transition matrices) are translation invariant. This uses common set of eigenvectors of all such matrices (Fourier modes of the state space) to represent state oceupancy, so that the effects of actions correspond to multiplication by action-specific eigenvectors.
This model explains how grid cells (as superpositions of Fourier modes) could support prediction of the effects of actions across environments that share the same underlying transition structure (see also Whittington et al. [43]). and could, for example, perform sense-of-direction planning in new environments (i.c., finding combinations of actions that most likely lead to the target state by ignoring local obstacles). We assume that other (e.g.. fronto-parietal) brain areas are responsible for detecting and avoiding obstacles following the overall direction provided by the grid cells [16] 28] However, ropology-dependen modifications to grid firing patterns could be used to accommodate local devi- ations from cansllatio-invarianne (e.g. obstacles), utilising the Woodbury inversion formulae, see Fig. E. Appendix A3 and Piray and Daw (4). We also show show our framework corresponds to other computational models of grid cells based on path integration, and provide a functional explanation for theta phase precession.
number of questions and predictions are raised by the proposed model. If basis of neurons with Fourier-mode-ll like firing patterns act as inputs to cells in entorhinal cortex, then grid cell firing patterns are only small proportion of the set of firing patterns that could be synthesised. This consistent with the existence of periodic non-grid cells in entorhinal cortex that resemble combinations of small numbers of Fourier modes (Krupic et al. (27)). We predict the use of the same set of grid cells (superpositions of Fourier modes inputs) for indicating "sense of direction" to goal locations across different environments after a single visit, i.c. showing generalisation across Euclidean environments. Finally, the proposed model predicts future state occupancy from the transition matrix, future work could also consider the reversed direction: inferring the translation between two locations given the phase codes for each (see Bush al. III, as linked discriminative and generative models.
The current model applies to translation invarrinnt transition structures, and use of the Fourier shift theorem calculate eigenvalues also assumes Euclidean state space. We demonstrated ways of generalising planning to bounded or locally non-translation invariant transition structures in Section 3 We note that machine learning methods based on a similar premise (creating a single representation to support planning via multiple different actions) might work even when the transitions are not strictly translation invariant (e.g., family trees, see Whittington al. by giving standard DQN agents the ability to represent the current state as eigenvectors of the state space weighted by state- and action task (Fig. 4. which is not strictly spatial or translation invariant. Hence our approach offers potential generalisation to non-spatial tasks, such as transitive inference (Von Fersen et al. [1I,, Appendix D). Future work will involve more rigorous study of the neural network implementation of the proposed grid cell model and its application to reinforcement learning. A future direction in generalising the current model to non-spatial tasks will be consider Fourier analysis groups of operators utilising group-theoretic knowledge (Kondor and Trivedi 25 Gao et al. (9)).
43 ) Here we showed that,
values, significantly improves learning of the CartPole

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
C.Y. thanks a DeepMind PhD studentship offered by the UCL Centre for Artificial Intelligence. T.E..B and N.B. thank the Wellcome Trust for support. The authors would like thank Daniel Bush, Talfan Evans, Kimberly Stachenfeld, Will de Cothi and Maneesh Sahani for helpful comments and discussions. The authors declare no competing financial interests.
REFERENCES
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp. G. Irving, M. Isard, Y. Jia, R. Joze- fowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, Shlens, Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke,
V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and
X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: www. wwwsorfosoooooooooo org/ Software available from tensorflow.org.
[2] A. B. Baram, T. H. Muller, J. C. Whittington, and E. Behrens. Intuitive planning: global navigation through cognitive maps based grid-like codes. bioRxiv, page 42 421461, 2018.
[3] A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions systems, man, and cybernetics, (5): 834-846, 1983.
[4] R. N. Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
[5] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.01 2016.
[6] Y. Burak and I. R. Fiete. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 5(2), 2009.
[7] C. P. Burgess and N. Burgess. Controlling phase noise in oscillatory interference models of grid cell firing. Journal of Neuroscience, (4(18))6224-0)) 6232, 2014.
[8] N. Burgess. Grid cells and theta as oscillatory interference: theory and predictions. Hippocam- pus, 18512):1157-1174 2008
[9] D. Bush and N. Burgess. hybrid oscillatory interference/contiiuoss attractor network model of grid cell firing. Journal of Neuroscience, 34(14):5065-500 5079, 2014.
[10] D. Bush and N. Burgess. Advantages and detection of phase coding in the absence of rhythmicity. Hippocampus, 2020.
[11] D. Bush, C. Barry, D. Manson, and N. Burgess. Using grid cells for navigation. Neuron, 87(3): 507-520, 2015.
[12] R. Climer, L. Newman, and H. M. E. Phase coding by grid cells unconstrained environments: two-dimensional phase precession. European Journal of Neuroscience, 38(4): 2526-2541, 2013.
[13] D. S. Corneil and W. Gerstner. Attractor network dynamics enable preplay and rapid path planning in maze mazeelike environments. Advances in neural information processing systems, pages 1694-1692,,) 2015.
[14] Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5(4):613-624 1993.
[15] Y. Dordek, D. Soudry, R. Meir, and D. Derdikman. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. Elife, 5:e 10094, 2016.
10

---

Published as a conference paper at ICLR 2021
[16] V. Edvardsen, A. Bicanski, and N. Burgess. Navigating with grid and place cells in cluttered environments. Hippocampus, 2019.
[17] T. Eliav, M. Geva-Sagiv, M. M. Yartsev, A. Finkelstein, A. Rubin, L. Las, and N. Ulanovsky. Nonoseillatory phase coding and synchronization in the bat hippocampal formation. Cell, 175 (4):1119-1130, 2018.
[18] M. C. Fuhs and D. S. Touretzky. A spin glass model of path integration in rat medial entorhinal cortex. Journal of Neuroscience, 26(16):4266442766 2006.
[19] R. Gao, J. Xie, S.-C. Zhu, and N. Wu. A representational model of grid cells based on matrix lie algebras. arXiv preprint arXiv:2006. 10259, 2020.
[20] T. Hafting, M. Fyhn, S. Molden, M.-B. Moser, and E. I. Moser. Microstructure of spatial map in the entorhinal cortex. Nature, 436(7052):801 806, 2005.
[21] T. Hafting, M. Fyhn, T. Bonnevie, M.-B. Moser, and E. I. Moser. Hippocampus-indeenneenn phase precession in entorhinal grid cells. Nature, (1017199):1188-2220 2008.
[22] M. E. Hasselmo. Grid cell mechanisms and function: contributions of entorhinal persistent spiking and phase resetting. Hippocampus, 18(12):1213-122, 2008.
[23] A. Jeewajee, C. Barry, V. Douchamps, D. Manson, C. Lever, and B. N. Theta phase precession of grid and place cell firing in open environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 369(1635)20122532 2014.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
[25] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. arXiv preprint arXiv:/802.03699, 2018.
[26] E. Kropff and A. Treves. The emergence of grid cells: Intelligent design or just adaptation? Hippocampus, 18(12):1256-1269 2008.
[27] J. Krupic, N. Burgess, and J. OeKeeee Neural representations of location composed of spatially periodic bands. Science, 337(6096):553-857,, 2012.
(28) E. A. Maguire, N. Burgess, J. G. Donnett, R. S. Frackowiak, C. D. Frith, and J. Keefe. Knowing where and getting there: a human navigation network. Science, 280(5366) 1998.
[29] S. Mahadevan and M. Maggioni. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research, 0-2169-2231, 2007.
[30] B. L. McNaughton, F. P. Battaglia, O. Jensen, E. I. Moser, and M.-B. Moser. Path integration and the neural basis of the' cognitive map'. Nature Reviews Neuroscience, 7(8):6636778 2006.
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[32] J. O' Keefe and J. Dostrovsky. The hippocampus spatial map: preliminary evidence from unit activity in the freely-moving rat. Brain research, 1971.
[33] B. Peng. X. Li, J. Gao. J. Liu, K.-F Wong, and S.-Y. Su. Deep dyna-q: Integrating planning for task-completion dialogue policy learning, arXiv preprint 1801v.1801.06176 2018.
[34] P. Piray and N. D. Daw. A common model explaining flexible decision making, grid fields and cognitive control. bioRxiv, page 856849, 2019.
[35] S. E. Qasim, I. Fried, and J. Jacobs. Phase precession in the human hippocampus and entorhinal cortex. bioRxiv, 2020.
11

---

Published as a conference paper at ICLR 2021
[36] K. S. Riedel. A mordison-woodbury identity for rank augmenting matrices with application to centering. SIAM Journal on Matrix Analysis and Applications, 13(2):659-622, 1992.
[37] B. Sorscher, G. Mel, S. Ganguli, and S. Ocko. A unified theory for the origin of grid cells through the lens of pattern formation. In Advances in Neural Information Processing Systems, pages 10003-10013, 2019.
[38] K. Stachenfeld, M. M. Botvinick, and Gershman. The hippocampus as a predictive map. Nature neuroscience, 20(11):1644 2017.
[39] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[40] E. C. Tolman. Cognitive maps in rats and men. Psychological review, 55(4):189, 1948.
[41] L. Von Fersen, C. Wynne, J. Delius, and Staddon. Transitive inference formation in pigeons. Journal of Experimental Psychology: Animal Behavior Processes, 17(3):334, 1991.
[42] A. C. Welday, I. G. Shlifer, M. L. Bloom, K. Zhang, and H. T. Blair. Cosine directional tuning of theta cell burst frequencies: evidence for spatial coding by oscillatory interference. Journal of Neuroscience, 31(45):6115711677) 2011.
[43] J. C. Whittington, T. H. Muller, S. Mark, G. Chen, C. Barry, N. Burgess, and T. E. Behrens. The tolman-richenbaum machine: Unifying space and relational memory through generalization in the hippocampal formation. Cell, 2020.
12

---

Published as a conference paper at ICLR 2021
A SOME PROOFS IN SECTIONS 3
Proposition A.1 Given our assumption of periodic boundary condition, the transition matrix, T E CNXN (Eq. 3), is indeed a circulant matrix.
Proof is easy to see that the proposition holds trivially for transition matrices with only one-step translations but without Gaussian spread. Hence here we only show the proof for the case where the transition structure includes both Gaussian spread and one-step translations.
Consider for an arbitrary transition matrix T for 2D rectangular environment with length I. and width W, and the underlying transition velocity e = (M2: fyy), remembering that T is the LW LW 2D matrix formed from concatenating rows from what would be the 4D matrix of transitions between all pairs of states in a L x W 2D state space. An arbitrary entry on the kth lower subdiagonal is T (t: 1) i (t]=(t)== i) for any suitable state given & (i.e., 2 k). If the Gaussian spread radially symmetric with constant variance across states, the value of T25-k only depends on the distance between state i k and the state i, where i, is the translated state of state given the effect of the velocity v. The states - k and i, are equivalent to the states ((i (i-kk//La (i- and (i//L + Ug2 modL + vgy) in the two-dimensional spatial domain respectively (where a/ /b denotes the integer part of a/b). Note that we need to have the velocity E L -W/2] so that the translation leaves the actual distance d unchanged. The distance between the state i k and the
(i-k) k) modL)
expected state io in the 2D state space is then
d ((i
(i -i- i//L v. ((i - k) modL modL vty)
(12)
For any arbitrary i' + such that i' i m, could compute similarly the distance between state and its corresponding expected state (Gaussian center) i' + v. After some algebra, we have that the distance between states k and + equals the distance between states i' k and i' + v.
= V(W - 1////L-i111 vz)2 ((i' - k) modL - modL - v,,)²
= ((i - -o/---k//---C 6)/ v² + ((i + k)) modL - (i + 8) modL - vyy)
(13)
Now if we look the square terms within the square separately, we have
((i + - k)/L- ()++) v,)²
((i (b)//L++ 5//L-i//L- 6 -2))
6//L
(14)
((i
((i ( 8 - k) modL (i + 8) modL - )2y)
= (((i k) modL + 5 mod modl. (i modT. + 5 modL) modL. v2y)
(15)
((i k) modL - å modL - i modL - å modL - v2,)
((i k-) modL. - modL - =,,²
The second equality holds due to the fact that (i - k) mod + 5 mod/. < L, since this is simply the I E-possiiion of state i + - k. which is never larger than L, hence ((i k) modL + modL) modL =
(i 6--A k) modL + 6 modL. Similarly (i modL ó modL) modL modL modL. Hence have
d' (K-
v, ((7 k) modL modL - u2 d
(16)
Hence all entries on the kth lower subdiagonal are identical, i.e. T.,-A Tr.v= for all 1 < k < LW 1. And by similar arguments, we could show that all entries on the Arth upper subdiagonal are identical (for < k < LW - 1), and equals to the corresponding entries the LW kth lower subdiagonals. And the fact that all the main diagonal entries are identical immediate from the problem setting. Hence target transition matrix is indeed circulant matrix. (Note that in simulations the transition matrix will only be approximately circulant due to normalisation and numerical issues.)
13

---

Published as a conference paper at ICLR 2021
Now consider the corresponding (LW k)tth upper subdiagonal (to the kth lower subdiagonal), by similar arguments, we have that for any Tir, r"-* = P(x((t + = k|/(() = x²) for suitable r" (i.e. i" + k < L), the distance between the state i" + k and expected next state z" + 24 the same as d, which is equivalent to Ter, = -- Hence all entries on the (LW - k)th upper subdiagonal are identical and equal to the entries on the kth lower subdiagonal. This holds for
arbitrary 1 S k < LW 1.
Proposition A.2 For any circulant matrix T NNNN as shown in Eq. its kth eigenvector takes the form:
v² 1
[1, wh, -
(17)
VN
where wi exp (2ak() the kth Nth root of unity, and the set of eigenvalues equals the set of DFTs of an arbitrary row/column of T.
Proof Firstly, note that the product between the circulant matrix T and an arbitrary vector v is equivalent convolution.
To
Ty-1
T2
Ti
T
To
Tx-1
T
T2
V1
= v
Ti
To
(18)
:
TN-2
TN-1
LUN-1
TN-1
TN-2
Ti
To
And we immediately have that
N-1
Wk Σ Tz-
(19)
This is true due to the periodicity of the entries given by the circulant structure. Then if we take the dot product of T' and arbitrary vector v of the form shown Eq. 17, the /th entry of the output vector has the following form.
N-1
Σ 1=0
T [40 "w
wi
Σ 1=0
T -w
(20)
where the equality holds since th exp (2 jm) = exp (2 - y'm exp (2 (m))m)) = w/"" w
Note that the last sum in Eq. 20 independent the choice of since both Tj and wj are periodic hence any change in / simply rearranging the terms the summation. Also we have that wt" = wi, the /th entry of the eigenvector Un. Hence have
Tvm Am Vm
(21)
where
N-1 Am Σ Tjws
(22)
1=0
for m = 0, O,..., NN - 1. Hence for an arbitrary N x N circulant matrix 7', the eigenvalues take the form as shown Eq, 22 and the corresponding eigenvectors take the form as shown [7 and the eigenvalues are equivalent to the DFT of the first row of the circulant matrix immediately follows from Eq. and the definition of DFT (Bracewell
The predicted phase change in the eigenvalues over the cigenvalues of the bascline symmetric transition matrix computed with Fourier modes computed via Fourier shift theorem (Eq. 7 under our formulation perfectly captures the actual phase changes caused by the one-step translations in the eigenvalues between the symmetric and and asymmetric transition matrices, as shown in Fig. [6A. However, when the transition dynamics is combination of diffusion and one-step translations, the predicted phase changes in eigenvalues will no longer perfectly match the actual phase changes observed as shown in Fig, 6B, and the oscillation is caused by the diffusion process. Namely, although the expected translation Is indicated by the velocity, the actual translation spans range of states depending the width of the diffusion field.
14

---

Published as a conference paper at ICLR 2021
A
Figure 6: Application of Fourier shift theorem for predicting changes in eigenvalues. We show the ground-truth (blue) and predicted (red) phase shifts of the eigenvalues of transition matrices given arbitrary drift velocity for: A: plain translation (5 units rightward); (B): diffusion with one-step translations (5 states rightward diffusion). The horizontal and vertical axes represent the indices of the eigenvalues and the corresponding phase changes (in radians) respectively.
Proposition A.3 The updated SR given the insertion of barrier is
= So C(I RC)
(23)
where So and S are the initial and updated SR, R = [J and C = Sol:: J] are the J-th rows and columns of Sq respectively, where J is the index set of states adjacent to the inserted barrier.
Proof This derivation is inspired by Piray and Daw
33
Given the definition of the SR, we have
S 5= -()--)>- So = -I-77) E N
(24)
where N is the number of states. Given the insertion of a barrier, S and So only differ in their j-th rows for J where J is the index set of states adjacent to the barrier. Hence we could write:
R = T(J [J R
(25)
Then if we have E E R with zeros everywhere but ones on the j-th rows for j E J, then by setting T and Wo To. we could write:
W Wo + ER
(26)
The Woodbury inversion formula usually use cases whn we are trying to compute the inverse of matrix given low-dimensional perturbution (Riedel (36).
(A + UCV) --1 -----C-- V- VU- -VAA-
(27)
Hence by applying the Woodbury inversion formula, we have:
W = W. -WW REW, RW,
(28)
= S = So C(T + RC) RSD
where C' ESy are the j-th columns of So for j J.
Proposition A.4 The "sense of direction", 8", is given by the form shown in Eq.
Proof Essentially, (u cas(f)) u sin(f)), given the start and target states, and sc, the future discounted occupancy of Se starting from So (or W[s0. sc). where W is the SR matrix), is maximised. Under our formulation. W can be calculated as follows:
wish find value of 0 such that under the drift velocity vo
W- Fdiag(1/(1 ()
(29)
where F is the DFT matrix (Eq. and A is the set of eigenvalues of the transition matrix given velocity Va. From our analysis based on Fourier shift theorem (Eq, 7). for each AY e A we have that:
X Yo Ve-k
(30)
where D; is the ith eigenvalue of the symmetrie (baseline) diffusion transition matrix, and k, is the wavevector for the ith Fourier mode. Then using linear algebra, we immediately arrive at the expression in Eq.
15

---

Published as a conference paper at ICLR 2021
B SOME PROOFS IN SECTION 4
Proposition B.1 The equations governing the dynamics of the prediction model and the CAN model of path integration are equivalent.
Proof We show the proof under the single-cell formulation, which can be immediately generalised the situation with multiple cells.
We firstly note that the prediction model can be mathematically categorised as minimising the following reconstruction objective function.
8(g) = | -
MFE-gwlle gwll.
(31)
where g E TR represents the grid cells firing rates, and w E RIxN represents the linear readout weights. Following Sorscher al. [37] replace w by its optimal value given fixed g. i.e., V = (g_g) ()
Note that any scaling of g can be absorbed into a corresponding reversed scaling into w, hence g is assumed to be of unit modulus (or the matrix G can be taken be orthonormal in the multi-cell case). Additionally, following the non-negativity constraint proposed in Dordek et al.
[5]. the overall optimisation problem becomes.
min {g) = T gullfr subject to g" 1, and 9i >= Ovi
(32)
Hence we can immediately write down the Lagrangian as follows.
T - n g + 1
(33)
where y and a are the multiplicative constant for the additive penalty terms corresponding to the constraints in Eq. 32 The derivate of the Lagrangian with respect to g then takes the following form.
dg ( 79 T 1
g>0 0
(34)
dt 1- ())
11)
g -
where (-) is the rectified linear function. Inserting the grid cell firing representation as a linear summation of the Fourier modes into Eq. 34 we obtain the following.
dg ag 7 2 Aruzftt
g>0 0 aag ag o ( ((E Aywsft) +1),
(35)
de
==0 0
where Ny are the corresponding eigenvalue of f, with respect the (symmetric) transition matrix, To
The dynamics of the grid cells under the CAN models can be written as following.
dg di 9 + (WW b(b))
(36)
where W is the recurrent connectivity matrix, b(v) is the velocity-dependen feedforward input to the grid cell under the CAN model which involves a constant baseline term and a velocity dependent term (Burak and Fiete 66).
Now suppose the agent is moving under non zero velocity, v. Given the grid cell firing represented by the linear summation of Fourier modes (Eq,
9 Eq. 35 can be written following,
dg = dt -ag + N
ag + -
2mi T g + N
1) + p4) X Af,; 0,) 1) + 14)
g>0 0
(37)
2ni
T g + a 2m
N
===
g = 0
where ej is the unit-norm wavevector of the Fourier mode / for all j. Now check with Eq. 36 by setting
a'
2mi 7 T N
W- =
(38)
2mi
6(0)
Agw,fs(le,e)) 1) + p)/a,
N
16

---

Published as a conference paper at ICLR 2021
By checking that when g 0, ( T (i
(()) N N
a > Ži=1 Ayw éÚ) - +)++),, we see that under non-zero velocity inputs. by appropriately adjusting the additive velocity imput term, b(c), the equations governing the dynamics for the normative and mechanistic models are equivalent.
C 2D FOURIER MODES
We know that the Fourier basis vectors from Eq. 4 form plane waves as shown in Fig. 5 From standard Fourier analysis in 2D space, the 2D Fourier modes form an orthonormal basis, and takes the following form.
Vn [x] exp (2miu x)
(39)
where the 2D Fourier basis vectors are encoded by the position vectors u = -w1/L w2/W) E
[0, 1] x [0, 1] (position vectors of each location in the Lx W environment projected onto [o. 1] x [o. 1)). The direction of the encoder position vector represents the direction of the plane wave and the frequency of the plane wave is the unnormalised direction vector ||u']| (where u' = u x (L, W)), note that u' is also the wavevector for the plane wave. This is a slightly different formulation comparing to the formulation given in Eq. 4 which consider the state space as a I-dimensional flattened vector of the 2-dimensional environment, hence the Fourier basis vectors are the corresponding 1 I-dimensional Fourier modes. Though both formulation give us the same set of Fourier basis vectors, under the definition in Eq. 39 could easily track the frequency and direction of the plane wave formed from the 2D Fourier modes. And the phase shift via the Fourier shift theorem Jequivalently applies for this 2D Fourier formulation.
The Fourier modes comprises a basis for representing any distribution over the task state space, so we could use a linearly weighted combination of Fourier modes to reconstruct any firing patterns, such as those observed in place cells (Welday et al. [42]. Fig. 8) However note that the coincidence detection of small number of oscillators with different frequencies will generate periodic patterns, e.g., grid cells, and more oscillators will be needed for those with more local firing fields such as place cells. Note that the total number of Fourier modes equals the number of states in the environment (e.g.. LW for the L x W rectangular environment on square grid), and it could be infeasible and inefficient to compute and store large number of such Fourier modes (or neurons with VCO-like firing patterns) in the brain. Hence here we only use the principal modes (taking the top Fourier basis vectors in terms of the corresponding eigenvalues (frequencies)), within contain the majority of the information is contained, with the number of principal modes depending We utilised the top 100 principal Fourier modes for most of the simulations in the main text (see Fig. 7 for a typical fixed set of Fourier modes). Fig. [ demonstttase that the small number of Fourier modes are able to reconstruct grid cells firing fields with various spacings and orientations, and place cells firing fields.
the desired reconstruction resolution.
D TRANSITIVE INFERENCE
In the main paper we argued that the same set of eigenvectors can be used to predict future occupancy distribution given the transition matrix for symmetrical relations like diffusion between adjacent states and directed transitions (e.g. moving N S W). Here we briefly discuss the generalisation of our model to non-spatial tasks.
We could apply our method to the one-dimensional transitive inference tasks of this type. e-g., given A >>>> B, B > C. >>> D, then infer if A > D) (Von Fersen et al. (1) This would be like having 1D track (and Fourier eigenvectors for 1-step transitions in both directions) corresponding to actions "greater" "smaller". and using "intuitive planning" to see if using cigenvalues for "greater" will take you from A to in the discounted future more likely than eigenvalues for "smaller". In order to deal with the non-peiioyicity of the task, we simulate transitive inference in small subset of the state space of the torus. As shown in Fig. g we see that our framework correctly predicts the transitive relationship between the chosen state 29 and states close to F
Despite the simplicity of ID transitive inference (Fig. 9), our model is still an advance on the original intuitive planning method in being able to predict the effects of both "greater" and "smaller" transitions
17

---

Published as a conference paper at ICLR 2021
Figure 7: Phase plots of 100 chosen low-frequency Fourier basis vectors with different frequencies and wavevectors.
with the same set of eigenvectors, rather than being restricted to prediction with one or the other alone.
E SIMULATIONS
E. FURTHER DETAILS OF THE GC-DQN AGENT
The overall architecture can be found in the graphical illustration in fig. 4 At each timestep, the state values and a specific action value are fed into a neural network for all possible values of actions (Blue box in the bottom left of fig. 4. which outputs Nections output, where Fourier modes inputs to the second network. The output can considered as the specific updates to each Fourier modes corresponding to the action the current location, like the phase shift in the Fourier shift theorem.
the number of
The inputs to the grid cell network are the first n Fourier modes, whose dimensions (D) are determined by the size of the state space. When the state variables are continuous, we compute an approximate size the state space by discretising each state variable. The number of principal Fourier modes (n) chosen arbitrarily as long as the majority of the information can be reconstructed from the chosen set Fourier modes. Higher values of n leads to finer details of the prediction, but also induces higher computational costs.
18

---

Published as a conference paper at ICLR 2021
A
B
Figure 8: Constructed place cell and grid cell firing fields from Fourier modes. A: Place cells firing fields constructed from coincidence detection of selected input Fourier modes (bottom plot shows a place field restricted to small subset of the toroidal state space); B: Grid cell firing fields with various spacings and orientations constructed from principal Fourier modes (Fig. 7
Choser state
Diflerance beiseen Sixs
states
Figure 9: Generalisation of flexible planning on transitive inference task. Given { 250 such that >> 1i+1 for all (and 2259 > .n for ensuring the circulant structure). The bar plot shows that we can correctly infer transitive relations between chosen state 1129 (red star) and nearby target states via computing the difference between the discounted future occupancy of the target state under the action-dependent SRs (Eq.2) corresponding to the "smaller" (left) and "greater" (right) actions. The z-axis denotes the states, and the y-axis denotes the difference between the SRs.
19

---

Published as a conference paper at ICLR 2021
At each timestep, the Fourier modes is fed as the input to the grid cell network (shown in the middle row of fig. [[). Each action multiplier (outputs from the state- action network) is multiplied with the corresponding column of the weight matrix between the input layer and the hidden layer of the grid cell network. The outputs of the hidden layer is then transposed, and forward propagate to the output layer of the second network. The computations of the grid cell network is considered to be equivalent to using the Fourier modes to construct a weight value for choosing each action at a given state that aids navigation/planning.
The outputs from the grid cell network and the standard DQN agent is then combined to output a vector, that acts as the values for each action that guides action choice in the current timestep.
E.2 SIMULATION DETAILS
All simulations were implemented in Python. The simulation details for each task is as follows:
The state space is assumed to be a 1D ring with 20 states, with the transition probabilities F(s++1 = i + 1[s, = i) = [[ = 1|sc i) 0.5, and discounting factory = 0.9 for generating the resolvent (Eq. 2
Fig. D
Fig. 2 Variance of each (Gaussian) firing field (representing the strength of diffusion) is
3; B: (0,5) drift velocity with increasing diffusion (variance increase by 3 per step); C: () drift velocity with O diffusion; E: The successor representation is computed using the Fourier modes and corresponding eigenvalues, with the discounting factor 7 = 0.9.
Fig. 3 A: The wind effect causes (0, 2) (2 units southward) displacement at each timestep; B, D: The successor representation is computed given transition matrix that assumes the variance at each (Gaussian) firing field is 1.5, followed by directed actions under the wind effects (with 0 diffusion), the discounting factor is y 0.9; F, G: The optimal following the ascending values of the successor representation, without any wind effect. The SR is computed given a transition matrix that assumes the variance at each (Gaussian) firing field is 1.5, followed by directed actions, the discounting factor is ς = 0.9; All computations are done by working directly with the Fourier modes instead of the transition matrices.
Fig. 4 The environment is the CartPole task (Barto et al. BB), and is simulated using the OpenAl gym environment (Brockman et al. (5)). The state value consists of variables: (Cart position, Cart velocity, Pole angle, Pole angular velocity), the action value is an integer takes value from (0. 1). where O represents moving left, and 1 represents moving right. For constructing the Fourier modes, we discretised each state variable into 8 bins, hence resulting in 8 number of states, and we chose the top 50 low-frequency Fourier modes as the inputs to the grid cell network. The standard DQN agent consists of two fully connected hidden layers with standard ReLU activations, with 48 and 24 units, respectively. The target network is updated every 500 timesteps. The deep Dyna-Q agent is simplified version of the model proposed in Peng et al. (33), with an additional 2-layer neural network learning the environmental transition dynamics, with 64 and 32 units in each hidden layer followed by ReLU activations. At each timestep, the learnt environment model is called to generate K imaginary trajectories that are used for model-based updates to the DQN agent. The number of model-based updates, K, is taken to be 2. The state-action network in the gc-DQN has one hidden layer, with 32 units followed by ReLu activation. The grid cell network has one hidden layers, with hidden size (n, A) followed by ReLU activation, where n represents the number of input Fourier modes, and A represents the number of possible actions. The deep ge-Dyna-Q has similar architecture as the ge-DQN agent, but with an additional environment network that learns the transition dynamics of the environment that is used for model-basee updates (with same architecture as the standard deep Dyna-Q agent). All models are learnt using the mean squared error loss function and Adam optimiser (Kingma and Ba learning rate 0.001 and no learning rate decay. The exploration strength, c, is set to be 0.8 at the start of each independent run, decreases by 0.05 at each episode, and is bounded below by 0.01. A total of 5 independent runs of 100 episodes are performed for each agent. Note that 100 episodes were simulated for each independent run due to the limited time and computational resources, but the results show that it is sufficient for demonstrating the increase in performance of the ge- DQN agent comparing to the baseline agents. We will, upon acceptance, show simulations with more episodes (up to the points where convergence
with
20

---

Published as a conference paper at ICLR 2021
of the baseline agents observed) in the camera-reate readyy version. All implementations performed the TensorFlow framework (Abadi et M.).
Fig. 5 A: wavevectors chosen input Fourier modes: k - (455, k2
(1/50, 4/50). K3- (3/50, -3/50), K4 (-4/50, 500 -1/50), ks, = (-1500,-4//00) kg (-3/50, 3/50); B: real rat trajectory projected onto x 2D spatial domain, firing phase interval of the input Fourier modes: 1-2. / 12, 2.5 12]. integration time interval: 8, expo- nential decay rate: 0.2, grid cell firing threshold: 2.95, directional bias: within - of the head direction (the range of the relative difference between the direction of the wavevector and the head direction, within which the Fourier modes are allowed to fire); C: running direction: arclan(1/3).
Fig. 2
The discounting factor: 0.3, the number of states: 26, number of effective transitive inference states: 10.
The Python-based implementations can be found at bttp:://glttubb.co///eaat Prediction and den__eeerralsiiiinnnnocmoo over Directed Actions by Sely__rii__eee Cells
21


Reviews:
[{"review_id": "wJvIFlPPUv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows how SR representation theory can be used in a model of grid cells to plan and navigate towards a target. A key insight of the paper is that velocity instructions modify the eigenvalues of the SR but not its eigenvectors, so the eigendecomposition does not need to be recomputed for all candidate velocity instructions, and the eigenbasis can be hard-coded in the neural circuit (by the grid cells). \n\nStrong points:\n- important contribution to our understanding of how grid cells could support navigation towards a goal\n- rigorous mathematical derivation of the results\n\nWeak points:\n- the paper is written for a highly specialized audience, which limits its impact (see below for detailed comments).\n- it is unclear to me how the successive exploration of possible directions could be implemented in practice by the grid cell network (see below for detailed comments). \n\nI recommend acceptance of the paper because of its strengths listed above, although I think the writing could be much improved to target a broader audience.\n\nMain concerns:\n1) Too much prior knowledge is assumed from the reader. I have worked at the intersection of theoretical neuro and ML for 8 years and am familiar with some recent literature on grid cell modeling, but still couldn't make sense of many aspects of the paper because of the many assumptions of prior knowledge. One exception is a fairly good description of the SR. But what is the  \"Woodbury inversion formulae\"? Shouldn't a barrier in the map break the translation invariance property assumed by the authors? What does CAN stand for? What does OI stand for?  What did we learn from comparing these models with the present model?\n2) It is unclear how the \"sense of direction\" can be calculated in practice by a grid cell network. Do the authors assume a successive exploration of all possible directions, followed by an estimation of the corresponding probability of reaching the target? What would be the mechanistic implementation of this algorithm in a grid cell network? It would be helpful to see a concrete description or simulation of a neural network implementing action-directed SR.\n\nTypos:\nfig1 caption: \"of of\"\nsingel-neuron\nframrework\ndiscrinminative\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q9THqOmwA3e", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an efficient method to predict directed transitions in spatial tasks by extending eigenbasis based prediction model. The authors show equivalence of the proposed method to classical models of path integration by grid cells - continuous attractor networks and oscillatory inferencce.\n\n\nStrengths:\n\n1) The proposed method is efficient and doesn't need a decomposition for future state occupancy and can be done directly by re-weighing of eigenvector (by eigenvalues) of the transition matrix.\n\nWeaknesses: \n\n1) Lack of quantitative evaluation and comparison with naive baselines makes it hard to grasp the contribution and value added by the proposed method.\n2) The datasets used for evaluation (and studying different properties of the proposed method) are oversimplistic.\n3) Assumptions made about applicability of the proposed method to tasks without periodic boundary conditions are not substantiated with experimental evidence.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IKDYd7Hfso0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose as extension of the successor-representation approach to Grid cells. The paper shows that this model can generate several experimentally observed properties of grid cells, and can be used in navigation of novel/mutable environments. Overall, the work should be of interest to any ICLR attendees who engage in research surrounding grid cells.\n\nStrong Points:\nOverall, the paper is very well organized and written. The presentation of the proposed model being able to unite previous normative and mechanistic models is elegant. \nExtreme care is taken to show that the model generates experimental findings such as phase precession. \n\nWeak Points: \nIn section 4.2, the authors relate the fourier modes from the previous sections. Given that in the previous section, the authors propose grid cells as being weighted Fourier modes (eq 10), I worry that there might be explicit band cells  in the proposed model. Given that these neurons have been strongly questioned in the electrophysiology literature, this could be a huge impediment to the model, as with previous implementations of OIM. If instead, the fourier component is meant to be distributed among the input population, it the authors should make this clear\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HUG7BiJlp_k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a model of navigation based on grid cells and the successor representation (SR). \n\nQuality: This submission contains some interesting ideas. However, it feels that several key statements in the paper need to be clarified, and some statements need to be toned down. Overall, I feel this is a good submission, but the quality did not quite reach the bar in its current form.\nClarity: The writing requires some improvement. The key idea is not very clear. What does the “sense of direction” mean exactly? Also, it is mentioned that the model is capable of predicting the effect of arbitrary directed transitions, irrespective of local details, e.g., obstacles. This seems to suggest the grid cell responses are not affected by the obstacles. It that’s indeed the case, it would be inconsistent with experimental findings. (Am I misinterpreting this?)  Also, it is not clear exactly what are the inputs and outputs of the grid cell system.\nOriginality: Some key ideas presented in the paper (e.g, eigendecomposition of the transition matrix, successor representation) is fairly standard in previous literature, although the authors show that there is a way to unify some of the previous models with these notions.\nSignificance:  It is difficult to judge how much this work really adds to the previous literature.  \n\nPro: \n- the paper shows that a simple model based on grid cells and SR representation can perform navigation in some simple environments. \n- the paper attempts to unify various kinds of grid cell models, which is interesting. \n- the model is mathematically quite elegant and simple.\n\nCons: \n- Some key ideas in the paper, e.g. eigen-decomposition of transition matrix, SR representation, are not new. \n- The “sense of direction” added to the model is not clearly described. It is unclear how it is implemented in the network model, and how this information can get to the grid cells. \n- It would be useful if the novel contributions could be more clearly stated. For example, Section  2 seems to be pretty standard materials. The paper would benefit by highlighting the true innovations (assuming there are some).\n-Furthermore, the claim of unifying previous models is an over-statement and potentially misleading. Clearly, the proposed model is related to many of the previous models, but to go one step further and say that it unifies these previous models, that would seem to be a over-claim in my view.  For example, the proposed model does not have any recurrent connections- it is difficult to see how this unify the CAN models which crucially study  the role of recurrent computations. \n\nConcerns:\n1) It is claimed that the proposed model generalizes across  different environments. Does that predict that the grid cells are shared across the different environment? Do the local cues such as barriers and reward locations influence the firing pattern of grid cells? Are the predictions of the model consistent with neurophysiological data?\n2) A key quantity in generating “a sense of direction” is the quantity s_G in Eq. (9). However, I didn’t find anything about how it is defined. Sorry if I am missing something. How is S_G encoded in the network? How would the grid cells have access to this information?\n3) The paper claims that this model also unifies the OI models. Notably, various experimental results have argued against the OI models, which now lacks experimental support. Thus, it is unclear how much this really adds.\n4) How does the model update the SR information when environment changes (e.g., inserting a barrier)?\n5) It is stated that  “a computational role for the neural grid codes: generating a \"sense of direction\" (eq. 9) even in new or bounded environments, via utilising a Fourier basis for a larger toroidal pseudo space”. What specific predictions does this model make for neurophysiology? Is there a way to falsify the hypothesis/model?\n\nA few more clarification questions:   \n\nHow does Fig F shows “SR values can be used for gradient-based navigation”?\n\nHow is the barrier implemented using “wind” exactly?\n\nWhat does “intuitive planning” mean?\n\nI am puzzled how theta phase precession could arise from the model. Apologize if this is obvious…Could the authors flesh out the ideas a bit more?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "G7kiFtwYbtn", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the intuitive planning methodology with Fourier analysis to predict state occupancy of directed transitions on a graph-structured state space. The paper uses this idea to develop new connections with path-integration grid cell models for their prediction, illustrating the method in a gridworld and simulated grid cell domain.\n\nI am leaning to reject this paper. On the one hand I agree with the authors on the importance. On the other hand, the contribution feels limited, as it seems to apply almost exclusively to 2D gridworlds and velocity-based actions.  Additionally, many of the insights here about Fourier representations are simple, and the useful connections to reinforcement learning are well-known from the proto-value function work (Mahadevan, 2005). The current paper could use some organizational improvements, and the experiments could be more comprehensive and tuned to demonstrate sharper points.  All taken together, the paper needs a bit more work to provide a significant contribution to the machine learning community. I believe another round of editing, tightening the material, and improving the results could indeed make this a very solid paper.\n\nCurrent justification:\nThe proposed method seems to have limited practical value. The demonstrations in Figures 1, 2, and 3 are made in small gridworlds -- 50x50 is the largest. The paper mentions how results are very sensitive to stochasticity; so it can be assumed it applies only to deterministic gridworlds. Section 3 suggests the transition structure can only derive from a 2D grid space, and that the method requires a periodic boundary condition hold. When it doesn’t, the suggested fix is to inflate the state space to at least twice its size. This feels quite restrictive and potential wasteful in terms of computation. In summary, the method only works on deterministic gridworlds where certain boundary conditions hold or the space is small enough to double its size so it still fits in memory. Can the authors provide examples of more interesting domains where this strategy is feasible? Can they demonstrate that there are sufficiently rich problems in the gridworld space where this method provides significant utility? If there are examples from biology, then explain what those are and show that the Fourier approach provides a significant enhancement in some regard.\n\nCurrently, a reader would not be able to replicate the presented results. Figure 1 shows transition matrices and the histograms of two row slices. The domain details needed to generate these images were not explained. It would be helpful to know: 1. What state space defines the problem, 2. How 1.E was generated, and additionally 3. What the colors represent. The same general comment applies to the other figures. Searching through the appendix did not return any extra information. How were the grid cells simulated? Even if you do not release code, the paper should still contain enough information to replicate the results.  \n\nThe presented data does not test any hypotheses. The first two figures are marked as demonstrative. The last two figures mostly show qualitative information from the successor representation and illustrations of gridworlds. There are no baselines with which to make comparisons. Figure 3.H provides the only piece of quantitative data, and it implies the prediction error increases with transition noise? Can the authors elaborate on what point is being made in Figure 3.H? How was the data gathered and why does it appear to increase monotonically in discrete steps?\n\nThe writing also needs to be tightened in several places. In the discussion, the paper claims “the resulting prediction framework is computationally efficient.” However, the paper provides no evidence for this claim. And along these lines, the data that is presented seems disconnected from any logical argument in the paper -- at least one that I could track. A simple editing pass will reveal many spelling errors and typos, e.g. “theorem to calculaet.”\n\nOverall the paper is too disorganized, unfocused, and limited to merit acceptance. I look forward to the author response so that I can refine both my understanding and assessment of this work.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was controversial amongst the reviewers. There is clear utility to the ICLR community: a new model of grid cells based on well-known technique (SR) used frequently in ML; good science---careful analysis showing the proposed model exhibits key properties and useful in synthetic navigation domains;  such work reminds of the important concerns in natural learning systems which is relevant to those that wish to simulate and build intelligence. Two of the reviewers with subject matter experience in the area advocated for acceptance.

On the other hand, many readers of ICLR may find the paper confusing and unsatisfying as some of the reviewers did. The empirical work was limited to small domains and mostly in the form of demonstrations---a typically ICLR reader would expect a performance improvement claim or a scientific hypothesis tested by each experiment. Presented as a new algorithm for ML the paper might appear too limited and simple (e.g., relying on state aggregation). The reviewers with neuro background found the paper clear and well organized, while the ML reviewers found it confusing. The relevance of the work will be limited to a smaller subset of researchers---but this is true of many ML works also. Finally, ML readers might be more familar with neuro work which propose computational models and then validate those models against real neural activity data from brains. This is work is not like that, rather using synthetic data to demonstrate important properties and explore empirical conjectures about the model.

In the end the paper is boarder line: the subject matter experts both listed issues that should be addressed (e.g., band cells issue), while the reaction of the ML reviewers suggests the impact of the work might be reduced at ICLR (compared to other venues). Additional text clearly articulating the scope and managing reader expectation could mitigate this concern, but it's not a small task to change the tone and pitch this way. Scientific conferences are about insights and understanding, this paper provides both. Please consider the suggested edits to maximize the impact of your work at ICLR this year.   
 
Decision: Accept (Poster)