Implementing Inductive bias for different navigation tasks through diverse RNN attrractors
Authors: Tie XU
Omri Barak
Venue:  (2020)
Keywords: navigation
Recurrent Neural Networks
dynamics
inductive bias
pre-training
reinforcement learning

Abstract: Navigation is crucial for animal behavior and is assumed to require an internal representation of the external environment, termed a cognitive map. The precise form of this representation is often considered to be a metric representation of space. An internal representation, however, is judged by its contribution to performance on a given task, and may thus vary between different types of navigation tasks. Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment. To focus on internal representations, we split learning into a task-agnostic pre-training stage that modifies internal connectivity and a task-specific Q learning stage that controls the network's output. We show that pre-training shapes the attractor landscape of the networks, leading to either a continuous attractor, discrete attractors or a disordered state. These structures induce bias onto the Q-Learning phase, leading to a performance pattern across the tasks corresponding to metric and topological regularities. Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes -- which can be shaped by pre-training and analyzed using dynamical systems methods. Furthermore, we demonstrate that non-metric representations are useful for navigation tasks.  

Published as a conference paper at ICLR 2020
IMPLEMENTING INDUCTIVE BIAS FOR DIFFERENT NAVIGATION TASKS THROUGH DIVERSE RNN ATTR- RACTORS
Tie Xu, Omri Barak
Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion, Israel Institute of Technology
Haifa, 320003, Isracl
com, omri cowr.barakkee........ www..bbrraggaaiicommm
ABSTRACT
Navigation is crucial for animal behavior and is assumed to require an internal rep- resentation of the external environment, termed cognitive map. The precise form this representation is often considered to be a metric representation space. An internal representation, however, is judged by its contribution performance on a given task, and may thus vary between different types of navigation tasks. Here we train a recurrent neural network that controls an agent performing several navigation tasks in a simple environment. To focus on internal representations, we split learning into a task-agnostic pre- pre-training stage that modifies internal connec- tivity and a task musk-specifiic Q learning stage that controls the network's s output. We show that pre-training shapes the attractor landscape of the networks, leading either a continuous attractor, discrete attractors disordered state. These struc- tures induce bias onto the Q-Leerniig phase, leading to performance pattern across the tasks corresponding to metric and topological regularities. By combin- ing two types of networks in a modular structure, we could get better performance for both regularities. Our results show that, in recurrent networks, inductive bias takes the form of attractor landscapes which can be shaped by pre-training and analyzed using dynamical systems methods. Furthermore, we demonstrate that non-metric representations are useful for navigation tasks, and their combination with metric representation leads flexibile multiple-task learning.
INTRODUCTION
Spatial navigation is an important task that requires a correct internal representation of the world, and thus its mechanistic underpinnings have attracted the attention of scientists for long time euclidean map, and this naturally
Keefe & Nadel 1978). A standard tool for navigation leads the hypothesis that internal model is such map. Artificial navigation also relies SLAM (Simultaneous localization and mapping) which is based on maps (Kanitscheider & Fiete) 2017a). On the other hand, both from an ecological view and from a pure machine learning per- spective, navigation is firstly about reward acquisition, while exploiting the statistical regularities of the environment. Different tasks and environments lead different statistical regularities. Thus is unclear which internal representations are optimal for reward acquisition. We take functional approach this question by training recurrent neural networks for navigation tasks with various types of statistical regularities. Because we are interested in internal representations, we opt for two-phase learning scheme instead of end-to-end learning. Inspired by the biological phenomena of evolution and development, first pre-train the networks to emphasize several aspects of their internal representation. Following pre-training, we use Q-learning to modify the network's 's readout weights for specific tasks while maintaining its internal connectivity.
We evaluate the performance for different networks on a battery of simple navigation tasks with different statistical regularities and show that the internal representations of the networks manifest in differential performance according to the nature of tasks. The link between task performance and network structure is understood by probing networks' dynamics, exposing low-dimensional

---

Published as a conference paper at ICLR 2020
manifold of slow dynamics in phase space, which is clustered into three major categories: continuous attractor, discrete attractors, and unstructured chaotic dynamics. The different network attractors encode different priors, inductive bias, for specific tasks which corresponds metric or topology invariances in the tasks. By combining networks with different inductive biases we could build a modular system with improved multiple-task learning.
Overall we offer a paradigm which shows how dynamics of recurrent networks implement different priors for environments. Pre-training, which is agnostic to specific tasks, could lead to dramatic difference in the network's dynamical landscape and affect reinforcement learning of different nav- igation tasks.
2 RELATED WORK
Several recent papers used a functional approach for navigation (Cueva & Wei, 2018 Kanitscheider Fiete 2017b; Banino et al., 2018). These works, however, consider the position as the desired output, by assuming that is the relevant representation for navigation. These works successfully show that the recurrent network agent could solve the neural SLAM problem and that this could re- sult in units of the network exhibiting similar response profiles to those found in neurophysiological experiments (place and grid cells). In our case, the desired behavior was to obtain the reward, and not report the current position.
Another recent approach did define reward acquisition as the goal, by applying deep RL. directly to navigation problems in end to end manner (Mirowski al.) (016). The navigation tasks relied on rich visual cues, that allowed evaluation in state of the art setting. This richness, however, can hinder the greater mechanistic insights that can be obtained from the systematic analysis of toy problems and accordingly, the focus of these works is on performance.
Our work is also related recent works neuroscience that highlight the richness of neural rep- resentations for navigation, beyond Euclidian spatial maps (Hardcastle et al. 2017 Wirth et al. 2017).
Our pre-training is similar to unsupervised, followed by supervised training EEran et al. 2010) In the past few years, end-to-end learning more dominant approach (Graves et al. [2014] Mnih et al. 2013). We highlight the ability of a pre- pre-training framework to manipulate network dynamics and the resulting internal representations and study their effect as inductive bias.
3 RESULTS
3.1 TASK DEFINITION
Navigation can be described taking advantage of spatial regularities of the environment to achieve goals. This view naturally leads to considering a cognitive map as an internal model of the environ- ment, but leaves open the question of precisely which type of map be expected. To answer this question, we systematically study both a space of networks emphasizing different internal models and space of tasks emphasizing different spatial regularities. To allow systematic approach, we design a toy navigation problem, inspired by the Morris water maze (Morris) 1981). An agent placed in a random position in a discretized square arena (size 15). and has to locate the reward location (yellow square, Fig IA), while only receiving input (empty/wallrreward) from the neigh- boring positions. The reward is placed in one of two possible locations in the room according to an external context signal, and the agent can move in one of the four cardinal directions. At every trial, the agent is placed in a random position in the arena, and the network's internal state is randomly initialized as well. The platform location is constant across trials for each context (see Methods).
The agent controlled by a RNN that receives the proximal sensory input, as well as a feedback of its own chosen action (Fig. IB). The network's 's output is value for each of the 4 possible tions, the maximum of which is chosen to update the agent's position. We use a vanilla RNN (see Appendix for LST units) described by:
2

---

Published as a conference paper at ICLR 2020
1441 = 1 (1-
he + tanh (Wh, W:f(z:) +W, W.C,
(1)
Q(4+) Woht + b,
where he is the activity of neurons in the networks(512 neurons default), W is connectivity matrix, T is timescale of update. The sensory input f(2)) ) is fed through connections matrix Wa and action feedback is fed throughW.. The context signal C2 is fed through matrix We. The network outputs a Q function, which is computed by a linear transformation of its hidden state.
Beyond the basic setting (Fig. 1A), design several variants of the task emphasize different statistical regularities (Fig. IC). In all cases, the agent begins from a random position and has to reach the context-dependent reward location in the shortest time using only proximal input. The "Hole" variant introduces a random placement of obstacles (different numbers and positions) in each trial. The "Bar" variant introduces horizontal vertical bar random positions in each trial. The various "Scale" tasks stretch the arena in the horizontal or vertical direction while maintaining the relative position of the rewards. The "Implicit context" task is similar the basic setting, but the external context input is eliminated, and instead, the color of the walls indicates the reward position. For all these tasks, the agent needs to find a strategy that tackles the uncertain elements achieve the goals. Despite the simple setting of the game, the tasks are trivial due to identical visual inputs in most of the locations and various uncertain elements adding to the task difficulty.
Hole (HO)
Bar(BAR)
Basic(BSC)
C
Contect
h(t)
Scale Y(SY)
Scale(SC)
Scale X(SX)
Implicit(IM)
Visual Input
Figure 1: Navigation task and network architecture(A) Basic task setting. The agent begins in random position and has to locate the reward, which is in context-dependent location. Input is only provided from the neighboring cells. (B) The agent controlled by an RNN that receives input from proximal visual stimuli and its action feedback. An external context is provided to indicate which two possible reward locations is active. (C) Tasks used: Basic, Random obstacles placed in each trial (either holes or bars), scaling the arena in either direction or both, implicit context signal (wall color) instead external context.
3.2 TRAINING FRAMEWORK
We aim understand the interaction between internal representation and the statistical regularities of the various tasks. In principle, this could be accomplished by end-to-end reinforcement learning of many tasks, using various hyper-parameters allow different solutions to the same task. We opted for a different approach both due to computation efficiency (see Appendix III) and due to biological motivations. A biological agent acquires navigation ability during evolution and develop- ment, which shapes its elementary cognitive ability such spatial object memory. This shaping provides a scaffold upon which the animal could adapt and learn quickly to perform diverse tasks during life. Similarly, and Q learning phase that sask-specffic (Fig. 2A). During pre-tres pre-training we modify the network's internal and input connectivity, while Q learning only modifies the output.
divide learning into two phases, a pre-traine pre-training phase that is task agnostic
Pre-training is implemented in an environment similar to the basic task, with an arena size chosen randomly between 10 to 20. The agent's actions externally determined as correlated random

---

Published as a conference paper at ICLR 2020
walk, instead of being internally generated by the agent. Inspired by neurophysiological findings, emphasize two different aspects of internal representation landmark memory (Identity of the last encountered wall) and position encoding (O'Keefe & Nadel 1978). We thus pre-train the internal connectivity to generate an ensemble of networks with various hyperparameters that control the relative importance of these aspects, well which parts of the connectivityW, W.: Wa modified. We term networks emphasizing the two aspects respectively MemNet and PosNet, and call the naive random network RandNet (Fig. 2A). This is done by stochastic gradient descent on the following objective function:
== P(z))log((z) B
Σ T.lo
Σ P(At)
(3)
with 2 = (x, y) for position, for landmark memory (identity of the last wall encountered), for action. The term action serves as regularizer. The three probability distributions are estimated from hidden states of the RNN, given by:
exp(Wn wee + bmn)
h(T
= Em (exp(W Wmi hr + dm))
(4)
exp(Wa 1h0- ho] + ba)
P(A
(5)
Σn exp( W. [1 hi0] + ba))
exp((z- - (Wo((z(WWphh) - bp)) /02
P(z]he)
(6)
Σ. expp(== (W bp)))//02)
where Wm, Wp Wa are readout matrices from hidden states and [hg- Ite] denotes the concatena- tion of last and current hidden states. Tables 2 in the Appendix show the hyperparameter choices for all networks. The ratio between a and controls the tradcoff between position and memory. The exact values of the hyperparameters were found through trial and error.
Having obtained this ensemble of networks, we use a Q-learning algorithm with TD-lambda update for the network's outputs, which are Q values. We utilize the fact that only the readout matrix W, is trained to use a recursive least square method which allows a fast update of weights for different tasks (Sussillo & Abboti 2009). This choice leads much better convergence speed when compared to stochastic gradient descent. The update rule used is:
Wn(n + 1)
Wo(n) (()))(()H(())
P(n + 1)
(C(n 1) +a)
(8)
C(n + 1) -AC(n) + (nn)"
(9)
e(n) = WnH(n) - ((n)
(10)
where is a matrix of hidden states over 120 time steps, af is regularizer and À controls forgetting rate of past data.
We then analyze the test performance of all networks on all tasks (Figure 2B and Table 3 in ap- pendix). Figure 2B,C show that there are correlations between different tasks and between different networks. We quantify this correlation structure by performing principal component analysis of the performance matrix. We find that the first two PCs in task space explain 79% of the variance. The first component corresponds to the difficulty (average performance) of each task, while the coeffi- cients of the second component are informative regarding the nature of the tasks (Fig. 2B. right); Bar (-0.49), Hole(-0.25), Basic(-0.21). Implicit context (-0. ScaleX (0.04), ScaleY (0.31), Scale (0.74). We speculate these numbers characterize the importance of two different invariances inherent in the tasks. Negative coefficients correspond to metric invariance. For example, when overcoming dynamic obstacles, the position remains invariant. This type of task was fundamental to establish metric cognitive maps in neuroscience (O Keefe & Nadel 1978). Positive coefficients correspond to topoloiiaal invariance, defined as the relation between landmarks unaffected by the metric infor- mation.
Observing the behavior of networks for the extreme tasks of this axis indeed confirms the specula- tion. Fig. 3A shows that the successful agent overcomes the scaling task by finding set of actions
4

---

Published as a conference paper at ICLR 2020
A
B
0,86
RanNet
Pretrain
MemNet
PosNet
Qlearn
Modify readout W
D
1.00
0.72
0.76
041
0.06
0.40
0.72
1.00
0.75
052
044
016
0.32
0.41
0.52
0:42
1:00
0.20
0.06
0.33
0.20
0.41
0:33
0.20
1.00
0.27
0.34
0.40
0.32
0.25
0.33
0.45
1.00
BSC
PosNet
RanNet
MemNet
Modular
Figure 2: Training scheme and performance analysis. (A) Two-stage learning framework. Task
agnostic pre-training of the internal connectivity is done while emphasizing either position decoding (PosNet) or the identity of the last wall (landmark memory, MemNet). Following pre-training, Q learning of the output performed for each task. This is also done on networks that were not pre- trained (RandNet). (B) Example trajectories of PosNet on the basic task, starting from 9 different initial conditions. The numbers are the scores for each trial (see Appendix). (C) Task performance for all networks on all tasks. The score is average of trials from all starting positions, where each trial is scored by the time relative to the shortest path, or -1 if the agent fails to reach the reward after 120 steps. Bars the left are coefficients of the second principal component, corresponding metric vs. topological tasks. The columns show different realizations of Posnet, RanNet and Mem- Net. The last column is modular network introduced in last section of Results, (D) Correlation between all tasks, showing a clustering into two main groups (metric and topological). Parameters for all networks Appendix Tables 223
that captures the relations between landmarks and reward, thus generalizing larger size arenas. Fig3B shows that the successful agent in the bar task uses very different strategy. An agent that captures the metric invariance could adjust trajectories and reach the reward each time when the obstacle is changed. This ability is often related to the ability to shortcuts (O'Keefe & Nadel, 1978). The other tasks intepolate between the two extremes, due to the presence of both elements in the tasks. For instance, the implicit context task requires the agent to combine landmark memory (color of the wall) with position to locate the reward.
We thus define metrie and topological scores by using weighted average of task performance using negative and positive coefficients respectively. Fig. 3C shows the various networks measured by the scores. We that random networks (blue) can achieve reasonable performance with some
5

---

Published as a conference paper at ICLR 2020
hyperparameter choices, but they are balanced with respect to the metric topological score. On the other hand, PostNet networks are pushed to the metric side and MemNet networks to the topologiaal side. This result indicates that the inductive bias achieved via task agnostic pre-train pre-training manifested in the performance of networks on various navigation tasks.
A Topology
50
C
RanNet MemNet Postlet Modular
0.8
0.4
B
Metric
15
Metric
-0.4
15
15
Topology Score
Figure 3: Different strategies for different regularities. (A) A MemNet network solving the scaling task. The agent uses a sequence
actions, and thus generalizes to larger arenas. (B) A PosNet network solving the bar task. The agent appears understand its metric position, and uses it to move in novel paths towards the reward. (C) Performance of all networks in all tasks, projected onto the metric and topological scores.
3.3 LINKING REPRESENTATION TO DYNAMICS
What are the underlying structures of different networks that encode the bias for different tasks? We approach this question by noting that RNNs are nonlinear dynamical systems. As such, it is informative to detect fixed points and other special areas of phase space to better understand their dynamics. For instance, network that memorizes binary value might be expected contain two discrete fixed points (Fig. 4A). A network that integrates a continuous value might contain a line attracto(Kim al. (0177) Kakaria de Bivort (2017), and network that integrates position might contain plane attractor a 2D manifold of fixed points because this would enable updating æ and y coordinates with actions, and maintaining the current position the absence of action (Burak & Fiete 2009). Trained networks, however, often converge to approximate fixed points (slow points) (Sussillo & Barak] Mante et al., 2013, Maheswaranathan et al. 2019)) as they are not required to maintain the same position for an infinite time. We thus expect the relevant slow points to be some- where between the actual trajectories and true fixed points. We detect these areas phase space using adaptations of existing techniques (Appendix 5.3 (Sussillo & Barak)). Briefly, drive the agent to move in the environment, while recording its position and last seen landmark (wall). This results in collection of hidden states. Then, for each point in this collection, towards approximate fixed points. This procedure results in points with different hidden state veloci- ties for the three networks (Fig. 4B) RandNet does not seem have any slow points, while PosNet and MemNet do, with MemNet's points being slower. The resulting manifold of slow points for typical PosNet depicted Figure 4C, along with the labels of position and stimulus from which relaxation began. is apparent that pretraining has created in PosNet a smooth representation of position along this manifold. The MemNet manifold represents landmark memory points without spatial representation. Note that despite the dominance of position representation in PosNet, landmark memory still modulates this representation (Fig 3A, M) showing that pretraining did not result in perfect plane attractor, but rather in approximate collection of 4 plane attractors (Fig. 3D, MP). This conjunctive representation can also be appreciated by considering the decoding accuracy of trajectories conditioned on the number of wall encounters. As the agent encounters the wall, the decoding of position from the manifold improves, implying the ability to integrate path integration and landmark memory (Fig Appendix 8).
relax the dynamics
distinct fixed

---

Published as a conference paper at ICLR 2020
We thus that the pre-training biases implemented by distinct attractor landscapes, from which we could see both qualitative differences between networks and a trade-off between landmark mem- ory and position encoding. The continuous attractors of PosNet correspond tion of space, albeit modulated by landmark memory. The discrete attractors of MemNet encode the landmark memory in robust manner, while sacrificing position encoding. The untrained RandNet, on the other hand, has no clear structure, and relies on short transient memory of the last landmark.
metric representa-
B
Slow Manifold
5
RanNet
Relaxation Velocity
MemNet PosNet
k(00)
Best ingeetwey
n(13)
PCI
Initial Velocity
X
Y
M
PosNet
MemNet
S
W
15
Figure 4: Diverse attractor landscapes underly diverse agent priors. (A) Velocity of points be- fore and after the relaxation procedure. PosNet and MemNet converged to approximate fixed points (slow points), while RandNet did not. (B) Illustration of possible slow point structures (purple), along with trajectories around them (red). A pair of points can encode discrete memory, a line attractor can integrate a single variable, and plane attractor can integrate two variables (e.g- x, y)- (C) Attractor landscape for PosNet and MemNet projected into the first PCs of the hidden state. Coloring is according to either X, Y coordinates or the identity of the last wall encountered (land- mark memory, M). Note how the position is smoothly encoded on the manifold for PosNet, and memory is encoded by four discrete points for MemNet. The memory panels show a fit of a plane to the X,Y coordinates, conditioned upon given landmark memory showing that PosNet also has memory information, and not just position. The networks used are 1,13.20 from Table 3
The above analysis performed on three typical networks and is somewhat time- time-consuming. In order to get a broader view of internal representations all networks, we use a simple measure of the components of the representation. Specifically, drove the agent move in arena of infinite

---

Published as a conference paper at ICLR 2020
size that was empty except single wall (of different identity in each trial). We then used GLM (generalized linear model) to determine the variance explained by both position and the identity of the wall encountered from the network's 's hidden state. Figure 6A shows these two measures for all the networks. The results echo those measured with the battery of 7 tasks (Fig. 3C), but are orders of magnitude faster compute. Indeed, if we correlate these measures with performance different tasks, we see that they correspond to the metric-topological axis as defined by PCA (Fig. 5B, compare with Fig. 2B, right).
the
A
1.0
stimulus memory path integration
Path Integration
Correlation
0.6
0.2
-0.2
0.0
0.0
0.1
0.5
BSC
HO
BAR
SC
SX
SY
IM
Landmark Memory
Tasks
Figure 5: Components of the internal representation. (A) Variance of the hidden state explained by position and memory for all networks. Note the clear separation between the different pre-training regimes. The crosses denote networks used in Fig. 4. (B) Correlation of the two components with performance on all tasks. The strength of the relevant components in the internal representation are predictive of task performance following Q-learning. Note the similarity with the PCA coefficients in Fig. 2B, right.
3.4 MODULAR SYSTEM THAT COMBINES ADVANTAGES OF BOTH DYNAMICS
Altogether, we showed a differential ability of networks to cope with different environmental reg- ularities via inductive bias encoded in their dynamics. Considering the tradeoff fundamental property of single module RNN, it is natural to ask if could combine advantages from both dy- namics into a modular system. Inspired by Park et al. (2001). We design a hierarchical system composed of the representation layer the bottom and selector layer top. The representation layer concatenates PosNet and MemNet modules together, each evaluating action values according to its own dynamics. The second layer selects the more reliable module based on the combined rep- resentation by assigning value (reliability) to each module. The module with maximal reliability makes the final decision. Thus, the control of the agent shifts between different dynamics according to current input or history. The modular system significantly shifts the metroc-topological balance (Fig2C, Fig3B). The reliability V is learned similarly as Q(Appendix 5.7).
M41 1
1 hi + tanh (Wposlig + (+)( W A, W.C.) 7
(11)
hit1 =
122 + tanh (W wi + WZ/(=,) + W + W.C.)
(12)
Q1(h)) = hh b2
(13)
Q2(h!) Wihh +
(14)
v(h),n)) (eet(h h221) + beel
(15)
DISCUSSION
Our work explores how internal representations for navigation tasks implemented by the dy- namics of recurrent neural networks. We show that pre-training networks in a task-agnostic manner shape their dynamics into discrete fixed points or into a low-D manifold of slow points. These
8

---

Published as a conference paper at ICLR 2020
distinct dynamical objects correspond to landmark memory and spatial memory respectively. When performing Q learning for specific tasks, these dynamical objects serve as priors for the network's s representations and shift its performance on the various navigation tasks. Here show that both plane attractors and discrete attractors are useful. It would be interesting to see whether and how other dynamical objects serve as inductive biases for other domains. In tasks outside of re- inforcement learning, for instance, line attractors were shown to underlic network computations (Mante et al. 2013 Maheswaranathan et 2019).
An agent that has to perform several navigation tasks will require both types of representations. A single recurrent network, however, has a trade-off between adapting to one type of task or to another. The attractor landscape picture provides a possible dynamical reason for the tradeoff. Position requires continuous attractor, whereas stimulus memory requires discrete attractors. While is possible to have four separated plane attractors, it is perhaps easier for learning to converge to one or the other. A different solution to learn multiple tasks is by considering multiple modules, each optimized for different dynamical regime. We showed that such modular system is able to learn multiple tasks, in a manner that is more flexible than any single-module network we could train.
Pre-training alters network connectivity. The resulting connectivity is expected to be between ran- dom networks (Lukoševičius & Jaeger 2009) and designed ones (Burak & Ficte 2009). It perhaps surprising that even the untrained RandNet can perform some of the navigation tasks using only Q- learning of the readout (with appropriate hyperparameters, see Tables 23 and section 4 "Linking dynamics to connectivity" in Appendix). This is consistent with recent work showing that some architectures can perform various tasks without learning (Gaier & Ha 2019). Studying the con- nectivity changes due to pre-training may help understand the statistics from which draw better random networks (Appendix section 4).
Apart from improving the understanding representation and dynamics, is interesting consider the efficiency of our two-stage learning compared to standard approaches. We found that end-to- end training is much slower, cannot learn topological tasks and has weaker transfer between tasks (See Appendix section 5.2). Thus it is interesting to explore whether this approach could be used to accelerate learning in other domains, similar to curriculum learning (Bengio al. 2009).
ACKNOWLEDGMENTS
OB is supported by the Isracli Science Foundation (346/16) and by Rappaport Institute Thematic grant. TX is supported by Key scientific technological innovation research project by Chinese Min- istry of Education and Tsinghua University Initiative Scientific Research Program for computational resources.
REFERENCES
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J. Chadwick, Thomas Degris, Joseph Modayil, Greg Wayne, Hubert Soyer, Fabio Viola, Brian Zhang, Ross Goroshin, Neil Rabinowitz, Razvan Pascanu, Char- lie Beattie, Stig Petersen, Amir Sadik, Stephen Gaffney, Helen King, Koray Kavukcuoglu, Demis Hassabis, Raia Hadsell, and Dharshan Kumaran. Vector-based navigation using grid- like representations in artificial agents. Nature, 557(7705):429445) 5 2018. ISSN 0028- 0836. doi: 10.1000/44115--18--0000 s41 2415866000 01 0110018010000 01022-6
URL http:/ www. nat cow/ artm/ss/tcles/////
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
Yoram Burak and lla R. Fiete. Accurate Path Integration in Continuous Attractor Network Models of Grid Cells. PLoS Computational Biology, 5(2)xe 2 2009. ISSN 1553-7358, doi: 10. 1371/journal. 1000291. URL https: / plos 1ttp://....///..aaaaaaaaaa
1000291
Christopher J. Cueva and Xue-Xin Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. 3 2018. http: org/ abs/
1 88.3. 07770

---

Published as a conference paper at ICLR 2020
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625-6600 2010.
Adam Gaier and David Ha. 2019.
Weight agnostic neural networks. arXiv preprint arXiv: 19Xiv:1906.04358,
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines. 10 2014. URL http: /aaxii..rgg/bb//1411.
Kiah Hardcastle, Niru Maheswaranathan, Surya Ganguli, and Lisa M. Giocomo. A Multiplexed, Heterogeneous, and Adaptive Code for Navigation Medial Entorhinal Cortex. Neuron, 94(2): 375-387, 2017. ISSN 10974199. doi: 10.1016/j. newron.2000000 2017.000011.00. UtLhttpp:://// doi.
www.ooooo....oooooooomooomo neuron 2017 0001777.... 03.025
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-base control with recurrent neural networks. arXiv preprint arXiv:/5/2.0444 2015.
Herbert Jaeger. The &quotıecho state& approach to analysing and training recurrent
neural networks-with Erratum note 1. Technical report, 2010. URL https://pdfs:
pdf
Kyobi S Kakaria and Benjamin L de Bivort. Ring attractor dynamics emerge from spiking model of the entire protocerebral bridge. Frontiers in behavioral neuroscience, 11:8, 2017.
Ingmar Kanitscheider and lla Fiete. Making our way through the world: Towards functional
understanding of the brain's 's spatial circuits. Current Opinion in Systems Biology, 3:186-194, 2017a. ISSN 24523100. doi:
URL https: hwt.ingiiiinnn.n.nn
Flsevier.com/reete
Ingmar to tion problems, 2017b.
Kanitscheider and lla
Fiete.
Training recurrent hard naviga-
networks
generate hypotheses about
how
the
brain
solves
URL
http:://paprrs.nips..coo ccppppperr
Garrett E Katz and James A Reggia. Using directional fibers to locate fixed points of recurrent neural networks. IEEE transactions neural networks and learning systems, 296)):666-33666 2017.
Sung Soo Kim, Herv Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the Drosophila central brain. Science (New York, N.Y.), 3566664) 5 2017. ISSN 1095- 9203. doi: 10. 126/scienne..aa4453. URL http: www. ncbi..ccic.... nlm. nih wttp./obw.pp///////ooo/// 28473639
Mantas Lukoševičius and Herbert Jaeger. Reservoir Computing Approaches Recurrent Neu- ral Network Training. Technical report. URL httt::/ .ittt.issstt. ist pss.psusee edu/
vrlewoo//downooa//200----- 1............... 840.re03reppppp.. repp
Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural
network training. Computer Science Review, 3(3):127-149, 8 2009. ISSN 1574-0137. doi:
00.10160.COSSEE.2000.00002 URL //wppp:://www... wwww. www.ncedirect.ccommmmmmm wttp:/cccccccceeeeeeeeeeeeeee
atticlee/111/1155011170000000000
Niru Maheswaranathan, Alex H Williams, Matthew D Golub, Surya Ganguli, and David Sussillo. Line attractor dynamics in recurrent networks for sentiment classification. 2019.
Valerio Mante, David Sussillo, Krishna V. Shenoy, and William T. Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 5037774))0 84, 11 2013. ISSN 0028-0836. doi: 10. 12742. URL http:/ //www//wwww///.m.o nat ure. com/ articles/
naturel 2742.
10

---

Published as a conference paper at ICLR 2020
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking Connectivity, Dynamics, and Computations in Low-Rank Recurrent Neural Networks. Neuron, 99(3);609 623, 2018. ISSN 10974199. doi:
wwt.:o..eeumon..1.0...000 URL httpa://all.org/1.... http://///........e...oeeeeeeeee http:///nnnuroo..o.0
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu. Dharshan Kumaran, and Raia Hadsell. Learning to Navigate in Complex Environments. 11 2016. URL httt:://axxx
org /bb//ab/// 611 03673
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1121v:131..5602, 2013.
Richard G.M. Morris. Spatial localization does not require the presence of local cues. Learning and Motivation, 12(2):239-266 5 1981. ISSN 0023-9690. doi: 10. 0016/0023-99990881990000 atticile/p1100239991190022
URL https:// www.
ctaeet.cosscccc con/science/
John. O'Keefe and Lynn Nadel. The hippocampus as a cognitive map. Clarendon Press, 1978. ISBN 0198572069.
/ http::/ss///ppootoot....a.ae wwwpeitttr....i.ooo.ooo.... arizona htt.omaaa.dd/aaaae/e111055500 /22/9//0115//6999 620894
Kui-Hong Park, Yong-Jae Kim, and long-Hwan Kim. Modular q-learning based multi-agent coop- eration for robot soccer. Robotics and Autonomous Systems, 35(2):109-122 2001.
David Sussillo and L. F. Abbott. Generating Coherent Patterns of Activity from Chaotic Neural ne0.on..0000..000000....
Networks. Neuron, 63(4))544-557, 2009. ISSN 08966273, neuron. 2009 07.018
URL http: doi
David Sussillo and Omri Barak. Opening the Black Box: Low-dimensional dynamics in high- dimensional recurrent neural networks. Technical report. URL httpss://aaaaaeeeee
barak neco. bar.k-neo...aaao...c pdd
Sylvia Wirth, Pierre Baradue, Aurlie Planté, Serge Pinède, and Jean Ren Duhamel. Gaze-informed, task-situated representation of space in primate hippocampus during virtual navigation. PL.S
Biology, 15(2):1-28, 2017. ISSN 15457885, doi: 10.137 pho.umal.

---

Published as a conference paper at ICLR 2020
5 APPENDIX
5.1 PERFORMANCE MEASURE FOR EACH TASK
When testing the agent on a task, we perform a trial for each possible initial position the agent. Note that the hidden state is randomly initialized in each trial, so this is not an exhaustive search of all possible trial types. We then measure the time it takes the agent to reach the target. This time is normalized by approximate optimal strategy moving from the initial position to corner of the arena (providing x and information), and then heading straight to the reward. If the agent fails to reach the target after 120 steps, the trial score is 1:
T/Topt
T < T man
Score
(16)
T T Tmax
5.2 TWO-STAGE VERSUS END-TO-END LEARNING
To check the effectiveness of our two-stage learning (pre-training followed by Q-Learning). we contrast with an end-to-end approach. We considered two approached to training: a classic deep Q learning version Mnih et al. (2013) and method adapted from RDPG Heess et al, (2015). The naive Q learning is separated into a play phase and training phase. During the play phase the agent collects experience 8t, dg. he T2 into a replay buffer. The action value Q is computed through TD lambda method as Q(h, a) as target function. During the training phase the agent samples the past experience through replay buffer perform gradient descent to minimize the difference between expected (h, a) and target Q. We found that for most tasks the deep Q learning method was better than the adapted RDPG, and thus used it for our benchmarks. We used both LSTM and vanilla RNN for these test.
We found that, for the basic task, all networks achieve optimal performance, but our approach is significantly more data-efficient even with random networks (fig. 6 For all topological tasks, the end-to-end approach fails with both vanilla RNN and LSTM (fig table 3). The end end approach performs relatively well in metric tasks, except for the implicit context task (fig 6B table 3), which converges to a similar performance as PosNet but with much slower convergence speed (10' trials vs 10s trials).
For the end-to-end approaches, critical question whether an internal representation emerges, which enables better performance in similar tasks. For instance, do networks that were successfully end-to-end trained in the basic task develop representation that facilitates learning the bar task? To answer this question, basis for RLS Q-learning of a different task. This allows comparison with the pre-trained networks. Figure 7 shows that pre-training provides better substrate for subsequent Q-learning considering generalization within metric tasks. For the implicit context task, the difference even greater.
networks that were end-to-end trained on one task and use them
when
5.3 EXPLORING THE LOW D NETWORK DYNAMICS
Recurrent neural networks nonlinear dynamical systems. As such, they behave differently in different areas of phase space. It is often informative to locate fixed points of the dynamics, and use their local dynamics as anchors to understand global dynamics. When considering trained RNNs, is reasonable to expect approximate fixed points rather than exact ones. This is because a fixed point corresponds to maintaining the same hidden state for infinite time, whereas trained network is only exposed to a finite time. These slow points (Sussillo & Barak Mante et al., 2013) can be detected in several manners (Sussillo & Barak Katz & Reggial (017)) For the case of stable fixed points (attractors), it is also possible to simulate the dynamics until convergence. In our setting, we opt for the latter option. Because the agent never stays in the same place, we relax the dynamics towards attractors by providing as action feedback the average of all 4 actions. The relevant manifold (e.g. a plane attractor) might contain areas that are more stable than others (for instance a few true fixed points), but want avoid detecting only these areas. We thus search for the relevant manifold in the following manner. We drive the agent to move in the environment, while recording its position and last seen stimulus (wall). This results in a collection of hidden states, labelled by position
12

---

Published as a conference paper at ICLR 2020
basic task
Istr Istm_e22 e2e
lstm_e2e__ee
Performance
0.5
rnn _ee
rnn_e28 @nn_ee##2
0.0
mem
pos
1e4
2e4 trials
3e4
4e4
bar task
scaling task
1.0
0.5
Performance
0.5
Performance
-0.5
-0.5
2.5e4
5e4
7.5e4
10e4
0.5e4
1e4 1.5e4
trials
trials
Figure 6: Learning curves for end-to-end training, compared to the Q-learning phase of the two- stage learning. Each curve represents a single network. Curves shown for Basic, Bar and Scaling tasks.
and stimulus that we the m manifold. For each point on the manifold, we continue simulating the dynamics for m extra steps while providing as input the average of all 4 actions, resulting in further m * manifolds. If these states are the underlying scaffold for dynamics, they should encode the position (or memory). We therefore choose by cross- validatio method decodding new trajectories obtained the basic task by using the 15-nearest neighbors each m-manifold. The red curve in Figure A shows the resulting decoding accuracy for position using PosNet, where the accuracy starts to fall around m 25, indicating that further relaxation leads to irrelevant fixed points.
5.4 ATTRACTOR LANDSCAPE FOR DIFFERENT RECURRENT ARCHITECTURES
We tested the qualitative shape of the slow manifolds that emerge from pre-training other unit types. Specifically, we pre-trained an LSTM network using the same parameters as PosNetl and MemNetl (Table D. Figures / show qualitatively similar behvior that deseribed in the main text. Note that MemNet has slow regions instead of discrete points, and we suspect discrete attractors might appear with longer relaxation times. The differences between the columns also demonstrate that slow points, revealed by relaxation, are generally helpful to analyze dynamics of different types of recurrent networks.
5.5 PRE-TRAINING PROTOCOLS AND PERFORMANCE NETWORKS
As explained in the main text, pre-training emphasizes decoding of either landmark memory the position of the agent. We used several variants of hyperparameters to pre-train the networks. Equation 17 which is written again for convenience, defines the relevant parameters:
13

---

Published as a conference paper at ICLR 2020
Bar
Hole
Performance
Performance
0.5
0.0
FromBasic
FromBasic
-0.5
FromHole FromPos
-0.5
FromBar
FromPos
0.5e4
1c4
1.5e4
0.5e4
1e4
1.5e4
trials
trials
Implicit
Scaling
0.5
FromMem FromBasic
0.5
FromiHale
Performance
Performance
FromBasic
FromBar
FromPas
-0.5
0.5e4
1e4
1.5e4
0.5e4
1e4
trials
trials
Generalization or transfer of end-to-end networks. Q-Learning of various tasks from starting point of either pre-trained networks, or from end-to-end trained networks. The FromPos and FromMem curves denote PosNet and MemNet respecively. Each panel shows learning curve for different task.
Figure 7:
A
B
C
0.8
0.8
0.8
0.6
0.6
0.8
0.2
relextion time
encorcers
The accuracy of decoding position (A) or landmark memory (B) from the attractor manifold as a function of the number of relaxation steps. Panel A shows drop in accuracy around = 25, indicating that this stage the process converges to irrelevant fixed points. (C) Conjunctive coding memory and position. The accuracy of decoding position from the attractor manifold as function of the number of wall encounters. Only PosNet shows improvement in decoding with the added information. This consistent with the joint representation of position and memory in the attractors.
Figure 8:
n Σ
S= P(z.)oog(Zz)
llog
Σ AlogP(AA)
(17)
The agent was driven to explore an empty arena (with walls) using random actions, with a probability p of changing action (direction) any step. Table T shows the protocols (hyperparameters) Table
14

---

Published as a conference paper at ICLR 2020
PosNet et Pre Relaxation
10 Relaxation
20 Relaxation
M
104
42
Cop Pay Ray
X
MemNet
Relaxation
Relaxation
M
X
30
Figure 9: Slow points analysis for LSTM PosNet and MemNet(pretrained for landmark). Similar to Figure 4 in the main text. The three columns show points in phase space during a random walk colored according X, Y and identity of
and after 10 or 20 relaxation steps. The different last wall. The PosNet is on top and MemNet is at bottom
2 shows the random networks hyperparameters, and table 3 shows the performance of the resulting networks all tasks. For all pre-training protocols an 12 regularizer of 10-6 on internal weights
15

---

Published as a conference paper at ICLR 2020
was used, and learning rate of 10-5. All PosNet and MemNet training started from RandNetl (detailed below).
Table I: Pretraining protocols
hyperparameters
protocol name
loss(ar, 7)
Weights adjusted
p
PosNet1
1, 0,
W, Was W;
0.2
PosNet2
1. 3,0,0 0
W
0.2
PosNet3
1,0,0
W
PosNet4
1,
W. W.. W
0.2
PosNet5
0.1.1,
W. W.. W
0.2
MemNetl
0.8. 0.2
W
MemNet2
0, 0.9.9 0.1
W
MemNet3
0, 0.7. 0.3
W
MemNet4
0, 0.8. 0.2
W, Wa. W;
MemNet5
0, 0.0.8 0.2
W
0.2
Different hyper parameter for RandNets:
h41 = (1 - )he + 7
(tanh(Wh, Wif(z)) + W. +.CC,
(18)
Q(he) Wolts bo
(19)
Number neurons used 512, time constant т is taken to be 2, the choice of hyper parameters is according standard reservoir computing litterature (Jaeger 2010 Lukoševičius & Jaeger). The weights are taken from a standard Normal distribution. is crucial to choose an appropriate stan- dard deviation for success of training (Appendix section 5), which is summarized each unit represents N/v/
Table 2: Random Networks hyperparameters
name
Wa
W;
RandNetl
10
RandNet2
1
5
10
RandNet3
0.5
10
RandNet4
0.5
5
10
RandNet5
1.2
10
RandNet6
1.2
5
10
5.6 MODULAR NETWORK PROTOCOL
The results of modular network are obtained from combining the PosNet 1 and MemNet 25 from table. Both the learning of Q function and V function is learned in the the same way main results equ (7,8,9, 10).
5.7 LINKING DYNAMICS TO CONNECTIVITY
Pretraining modified the internal connectivity of the networks. Here, we explore the link between connectivity and and dynamics. We draw inspiration from two observations in the field of reservoir computing (Lukoševičius Jaeger] 2009). On the one hand, the norm of the internal connectivity has large effect network dynamics and performance, with advantage to residing the edge of chaos (Jacger) 2010). On the other hand, restricting learning to the readout weights (which then fed back to the network, Sussillo & Abbott (2009)) results in low-rank perturbation the connectivity, the possible contributions of which were recently explored (Mastrogiuseppe & Ostojic 2018).
We thus analyzed both aspects. Fig. I0A shows the norms of several matrices as they evolve through pre-training, showing an opposite trend for PosNet and MemNet with respect to the internal
16

---

Published as a conference paper at ICLR 2020
Table 3: Performance of all networks all tasks
protocol
BSC
HO
BAR
SC
SX
SY
IM
1ePo
0.91
0.79
0.73
-0.36
0.90
0.19
0.78
2.Pt
0.97
0.78
0.79
-0.47
0.89
0.53
0.11
3. PosNetl
0.95
0.74
-0.04
-0.16
0.70
0.70
0.66
4. PosNetl
0.97
0.82
0.68
0.03
0.89
0.21
0.69
5. S.PosNetl
0.88
0.74
0.73
-0.36
0.88
0.20
0.78
6. I.Posettl
0.91
0.63
0.48
-0.41
0.62
0.04
-0.21
7. PosNetl
0.94
0.64
0.51
0.26
0.64
0.15
-0.31
PosNet2
0.89
0.62
0.14
-0.25
0.76
0.24
0.84
9. PosNett
0.93
0.48
0.41
0.04
0.22
-0.24
0.73
10.PosNet4
0.83
0.58
0.72
-0.07
0.16
0.58
-0.22
PosN
0.86
0.69
0.48
-0.19
0.21
0.65
0.01
12. PosNetó
0,80
0.17
0.11
-0.33
0.07
0.36
-0.10
13. Ret
0.89
0.52
-0.15
0.26
0.79
-0.17
-0.14
14. Ren
0.84
0.25
-0.89
-0.03
0.62
-0.29
-0.10
RandNet3
0.51
0.05
-0.93
-0.39
0.30
-0.35
-0.09
16. RandNet4
0.65
-0.31
-0.94
-0.37
-0.23
-0.04
-0.17
17. RandNet5
0.91
0.22
0.15
-0.16
0.62
0.06
-0.47
18.8 RendNet6
0.88
0.70
-0.36
-0.44
0.63
0.11
-0.33
1..MenNe1
0.65
0.27
-0.84
0.62
0.92
0.62
-0.00
20.MemNet1
0.79
0.15
-0.94
0.48
0.64
0.43
-0.14
21 21.MemNet1
0.82
0.28
-0.30
0.37
0.69
0.45
-0.15
22.MemNetl
0.73
-0.09
-0.51
0.65
0.84
0.58
-0.29
23.MemNetl
0.84
0.54
0.39
-0.07
0.85
0.26
-0.41
24.eemNe22
0.76
0.52
-0.87
0.58
0.47
0.90
-0.99
35.MeNNet3
0.76
-0.11
-0.46
0.65
0.83
0.61
-0.28
26. MemNer4
0.73
0.35
-0.64
0.43
0.91
0.50
-0.10
27. 27.MemNet5
0.75
0.08
-0.86
-0.37
0.12
-0.12
-0.04
28. End2End1
0.89
0.67
0.70
-0.16
-0.09
0.18
0.14
29. End2End22
0.67
0.8
0.73
-0.62
0.51
-0.61
-0.36
30.Modular 0.96
0.76
0.76
0.62
0.92
0.59
0.86
connectivity W. To estimate the low-rank component, we performed singular value decomposition on the change to the internal connectivity induced by pre-training (Fig. [T)B).
W Wo +US
(20)
The singular values of the actual change were compared to a shuffled version, revealing their low- constrained to generate such low- rank perturbation. Furthermore, we show that the low-rank structure is partially correlated to the network's inputs. possibly contributing their effective amplification through the dynamics (Fig. HO). Because we detected both types of connectivity changes (norm and low-rank), we next sought to characterize their relative contributions to network dynamics, representation and behavior.
rank structure (Fig. IOC.D). Note that pretraining
In order to assess the effect of matrix norms, generated a large number of scaled random matrices and use the GLM analyse in figure between landmark memory and path integration is affected by norm (Fig. [O]). But the actual numbers, however, are much lower for the scaled random matrices compared to the pretrained ones indicating the importance of the low-rank component (Fig. I0F). Indeed, when removing only the leading ranks from AW, Network encoding and performance on all tasks approaches that of RandNet.
access its influence on dynamics. We see the trade-off
5.8 BEHAVIOR DIFFERENT NETWORKS
Different networks develop diverse strategies for metric and topological tasks. In this section, we give examples of typical trajectories of PosNet, MemNet, RanNet in the basic, bar and scaling tasks.
17

---

Published as a conference paper at ICLR 2020
A
B
13
rank
rank
WAX
rank
Landmark memory
Path integration
1.5
0.00
0.5
11
13 15
20
mem by path ntegratton
10"1
025
10-2
420
0005
W >
W
rank
Figure 10: Connectivity changes during pre-training. (A) Top, Evolution of norm during pre- training for both PosNet (red) and MemNet (green). Bottom, low rank effect. The network can be decomposed into parts, random part, and learned low-rank structure through SVD. (B) SVD of AW compared to a shuffled version of AW, showing that most of the learned structure concen- trated in the first few ranks for PosNet(top). The same low-rank effect observed for MemNet(bottom). (C-D) Measuring the overlap between action feedback matrix or input matrix and output vector of AW for PosNet (C) and MemNet (D). (E) The variance of hidden states explained by GLM model containing position and landmark memory. Each pixel represents scaled random matrix, with the colored circles showing the norms of the pre-trained networks. The red, blue and green dots correspond to norm of selected PosNet, RandNet, MemNet for dynamics analysis. (F) Effect of gradually removing the leading ranks from A A. measured by the variance explained in the GLM.
18

---

Published as a conference paper at ICLR 2020
0.83
0.87
0.89
0.9
0.55
0.92
Figure 11: PosNet in basic task
19

---

Published as a conference paper at ICLR 2020
1
15
0.41
0.59
0.33
10
15
0.19
0.62
0.66
0.64
0.94
0.66
0.48
0.71
0.94
0.44
0.83
0.74
Figure 12: MemNet in basic task
20

---

Published as a conference paper at ICLR 2020
0.42
0.87
0.66
0.56
0.62
0.81
0.7
0.89
Figure 13: RanNet in basic task
21

---

Published as a conference paper at ICLR 2020
10
15
0.83
15
0.35
Figure 14: PosNet in bar task
22

---

Published as a conference paper at ICLR 2020
10
15
15
Figure 15: MemNet in bar task
23

---

Published as a conference paper at ICLR 2020
10,
15
0.62
15
0.87
0.61
0.79
Figure 16: RanNet in bar task
24

---

Published as a conference paper at ICLR 2020
10 20 30 40 50
10
20
30
40
50
0.64
0.74
Figure 17: PosNet in scale task
25

---

Published as a conference paper at ICLR 2020
10 20 30
40 50
0.49
0.58
10
20
30
40
50
0.3
0.41
0.9
0.9
0.85
0.36
0.58
0.52
Figure 18: MemNet in scale task
26

---

Published as a conference paper at ICLR 2020
10
0..29 40 50
0.13
0.84
0.44
0.78
10
20
30
40
50
0.31
0.22
0.59
0.53
0.42
0.63
0.28
0.25
0.56
0.47
0.32
0.35
0.7
0.52
Figure 19: RanNer in scale task
27


Reviews:
[{"review_id": "rkl_IG0e9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for the navigation task. The proposed method is inspired by the concept of cognitive map in human and animal. \nA recurrent neural network is incorporated and training is divided in two steps of (1) task-agnostic pre-training and (2) task-speciﬁc Q learning.\n\nThe paper is well-written and clear.\n\nWhile the idea of using a representation inspired by cognitive maps is interesting, the paper does not offer much technical novelty. (e.g no technical novelty in eq. 1 and eq. 2)\n\nThe experimental results are weak and only a simple domain is tested.  \n\nIt is not clear how efficient the method would be compared to other approaches.\n\nVisualizations can be improved. As an example, Fig. 4 is not quite self-representative.\n\nI see the paper has a large room for improvement and the current manuscript is not convincing for publication.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1ldu7WAYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the internal representations of recurrent neural networks trained on navigation tasks. By varying the weight of different terms in an objective used for supervised pre-training, RNNs are created that either use path integration or landmark memory for navigation. The paper shows that the pretraining method leads to differential performance when the readout layer of these networks networks is trained using Q-learning on different variants of a navigation task. The main result of the paper is obtained by finding the slow points of the dynamics of the trained RNNs. The paper finds that the RNNs pre-trained to use path integration contain 2D continuous attractors, allowing position memory. On the other hand, the RNNs pre-trained for landmark memory contain discrete attractors corresponding to the different landmarks.\n\nAn interesting implication of the findings for neuroscience is that the same underlying network architecture can learn different dynamics, explaining diverse types of navigation-related signals found in the mammalian brain (place cells, border cells etc.).\n\nI am not entirely sure about the novelty or impact of the presented results. However, the exposition and the results are clear and it is interesting how pre-training can shape the dynamics of a network. I therefore recommend acceptance.\n\nMinor comments:\n- Please describe how the networks 1, 13, 20 used for Figure 4 were selected. Were they selected at random or selected according to some criteria?\n- It may be interesting to study the effect of more modern RNNs, e.g. LSTMs or GRUs, on the dynamics.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJg9M2dCFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Overview\nThis paper explores how pre-training a recurrent network on different navigational objectives confers different benefits when it comes to solving downstream tasks. First, networks are pretrained on an objective that either emphasizes position (path integration) or landmark memory (identity of the last wall encountered). This pretraining generates recurrent networks of two classes, called PosNets and MemNets (in addition to no pre-training, called RandNets). Surprisingly, the authors found that pre-training confers different benefits that manifests as differential performance of PosNets and MemNets across the suite. Some evidence is provided that this difference has to do with the requirements of the task. Moreover, the authors show how the different pretraining manifests as different dynamical structures (measured using fixed point analyses) present in the networks after pre-training. In particular, the PosNets contained a 2D plane attractor (used to readout position), whereas the MemNets contained clusters of fixed points (corresponding to the previously encountered landmark).\n\nOverall, I thought this was a very interesting paper--it is one of the first papers I have seen that demonstrates how different pre-training requirements both change network dynamics (as measured by fixed points), and how those differences can yield different benefits on downstream navigational tasks.\n\n## Major comments/concerns\n- I think the presentation the pretraining objective (eq 3) could be clearer. Is eq 3 what is minimized during pre-training? How are \\alpha, \\beta, and \\gamma chosen? \\alpha is used to separate the two types of networks (MemNet from PosNet), which is the critical difference studied in the paper, so it would helpful to go into more detail about what \\alpha controls and how it was chosen.\n\n- For the first task, I am surprised that the agent is able to navigate the environment using only the eight neighboring locations. What is the size of the arena? What fraction of the states are simply surrounded on all sides by empty space? It would be informative to show some trajectories of agents solving the basic task.\n\n- For Fig 3A and 3B, it would be nice to show the other network's performance (i.e. show the PosNet on the scaling task in 3A, and the MemNet on the bar task in 3B).\n\n- How come there are no networks that are able to solve both sets of tasks? That is, how come there are no networks in the upper right region of Fig 3C? Does this suggest that an agent needs to combine two separate RNNs to solve the whole suite of tasks?\n\n- What happens if you train recurrent networks with more sophisticated cell architectures (e.g. a GRU or an LSTM)? These are typically easier to train (and using automatic differentiation techniques are also amenable to fixed point analysis).\n\n## Minor comments\n- In eq. (1), use `\\left(` and `\\right)` to make the first set of parentheses have an appropriate height.\n- Typo on the first line after eq. (6) (matrices)\n- Relevant reference on comparing networks using dynamics around approximate fixed points: https://arxiv.org/abs/1907.08549.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Navigation is learned in a two-stage process, where the (recurrent) network is first pre-trained in a task-agnostic stage and then fine-tuned using Q-learning. The analysis of the learned network confirms that what has been learned in the task-agnostic pre-training stage takes the form of attractors.

The reviewers generally liked this work, but complained about lack of comparison studies / baselines. The authors then carried out such studies and did a major update of the paper.

Given that the extensive update of the paper seems to have addressed the reviewers' complaints, I think this paper can be accepted.
Decision: Accept (Poster)